Are Language Models Sensitive to Semantic Attraction? A Study on Surprisal (2023)

Abstract: In psycholinguistics, semantic attraction is a sentence processing phenomenon in which a given argument violates the selectional requirements of a verb, but this violation is not perceived by comprehenders due to its attraction to another noun in the same sentence, which is syntactically unrelated but semantically sound. In our study, we use autoregressive language models to compute the sentence-level and the target phrase-level Surprisal scores of a psycholinguistic dataset on semantic attraction. Our results show that the models are sensitive to semantic attraction, leading to reduced Surprisal scores, although none of them perfectly matches the human behavioral pattern.

## introduction cases of similarity-based interference have always been at the center of interest for sentence processing studies, as they offer strong evidence for cuebased models of memory retrieval during language comprehension @xcite . according to such accounts, interference emerges because an item with some cues has to be retrieved from memory, and because those cues are simultaneously matched by multiple items @xcite . consider the examples in (1) @xcite : compared to fully grammatical sentences, both elicit longer reading times in humans, but the effect is attenuated in 1a., where there is an attractor (cells) matching the number of the verb, causing an illusion of grammaticality. this phenomenon is known as morphological attraction. attraction has also been observed at the semantic level, as in the following example from the eyetracking study by @xcite : again, both sentences are implausible, because beer violates the selectional restrictions of the verb ate, but the authors of the study observed that (2a) was processed faster than (2b), due to the presence of a semantically fitting noun (meal) that generates a semantic illusion. both types of illusion are facilitatory interferences, as they attenuate the effects of anomalies leading to higher costs for the human language processing system. this is a case of semantic attraction. the recent literature in natural language processing (nlp), on the other hand, has shown an increasing interest in using the surprisal scores @xcite computed with neural language models (nlms) to account for sentence processing phenomena @xcite @xcite @xcite . this also includes investigations on interferences at the morphosyntactic level @xcite . to our knowledge, there have been no attempts to model semantic attraction with nlms yet. we aim at filling this gap by presenting a surprisal-based analysis of a psycholinguistic dataset on semantic attraction with three autoregressive nlms of different sizes. we found that nlms are sensitive to both the plausibility of the sentences and semantic attraction effects. however, nlm surprisal for a target phrase seems to be affected by attraction regardless of general sentence plausibility, differently from human reading behavior. on the other hand, sentence-level surprisal is not affected by semantic attraction. ## results the findings of the experiments are summarized in tables 1 and 2 . considering the main effects, we found that all models were able to distinguish plausible from imsum of the surprisal scores, as per reviewer 3's comment. as shown in the post hoc analyses of the pairwise comparison showed some interesting contrasts. we noticed that significant differences were found between the plausible sentences with plausible attractors and the two implausible conditions (i.e. in figure 1 , a vs. c and a vs. d, with @xmath3.001). differently from human total viewing times, no significant differences and no consistent facilitatory effects are observed between c and d in the sentsurp models (notice also in figure 1 that the median of condition d. is actually slightly lower than c., and the medians for c. and d. tend to be close in all the sentsurp models, cf. the boxplots in the appendix, right column), while facilitation is found for all the surp models. it is also noticeable that all models show no sensitivity to plausible attractors with the sentence-level surprisal metrics, but the surprisal at the target word with implausible attractors is always significantly higher. however, since no significant main effect of interactions was found for surp models, we conclude that semantic attraction seems to to have a general facilitation effect on its own, regardless of sentence plausibility. it would be interesting, in the future, to analyze how the attractors concretely affect the predictions, for example using techniques like contrastive explanations @xcite ) that can shed light on which tokens contribute to the prediction of the target verb rather than a plausible alternative word (in our case, this could be a verb in a thematic fit relation with the implausible attractor noun, e.g. drank for wine in examples 2. b-d). ## conclusions in this work, we presented a study on surprisal to investigate whether nlms predictions are sensitive to semantic attraction. our results on the data of the eye-tracking experiment by @xcite reveal that all models are sensitive to the general plausibility of the sentence, and that semantically-plausible attractors decrease the surprisal at the target phrase, although this effect generally does not interact with sentence plausibility as in humans. at the sentence level, no effects of attractor plausibility were observed, with the only, partial exception of a marginal significance with distilgpt2. interestingly, the most human-like pattern -including the interaction-has been observed with this model, the smallest one, although the specific contrasts between conditions pattern differently from human total viewing times.