Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models (2023)

Abstract: Subject-verb agreement in the presence of an attractor noun located between the main noun and the verb elicits complex behavior: judgments of grammaticality are modulated by the grammatical features of the attractor. For example, in the sentence \textit{``The girl near the boys likes climbing''} , the attractor ( \textit{boys} ) disagrees in grammatical number with the verb ( \textit{likes} ), creating a locally implausible transition probability. Here, we parametrically modulate the distance between the attractor and the verb while keeping the length of the sentence equal. We evaluate the performance of both humans and two artificial neural network models: both make more mistakes when the attractor is closer to the verb, but neural networks get close to the chance level while humans are mostly able to overcome the attractor interference. Additionally, we report a linear effect of attractor distance on reaction times. We hypothesize that a possible reason for the proximity effect is the calculation of transition probabilities between adjacent words. Nevertheless, classical models of attraction such as the cue-based model might suffice to explain this phenomenon, thus paving the way for new research. Data and analyses available at https://osf.io/d4g6k

## introduction on the surface, language appears to be produced and understood linearly, as humans read or hear words one after the other. yet, formal linguistic theories postulate the existence of an underlying structure that governs language processing @xcite @xcite @xcite . this hypothesis is supported by both behavioral @xcite @xcite and neural @xcite @xcite data. according to a competing view, unstructured probabilistic models capture behavior without explicitly relying on structures @xcite . the discrepancy the n 1 who v 1 the quick young n 2 v 2 viola¡on congruency the n 1 who v 1 the n 2 the most v 2 transi¡on viola¡on congruency figure 1 : experimental design our design aims to separate two possible mechanisms involved in language processing, by altering the distance of the embedded noun (attractor) from the verb. we compare the impact of noun-verb agreement (violation) and noun-noun dependency (congruency). the transition effect symbolizes the consequence of the transition probability between the second noun and the verb. as a baseline, we also include a condition with a similar number of words, but without a noun in between. the summary of the experimental conditions is shown in analyzing subject-verb agreement errors in the presence of distracting elements is a standard way of separating linear and structural accounts of language @xcite . attraction effects in number agreement have been studied extensively in humans @xcite @xcite and neural language models (nlms) @xcite @xcite . in the present work, we introduce a parametric manipulation of the distance between the distracting nouns and the verb (figure 1 ). we include a baseline condition where the subject-verb distance is matched but no word that carries number marking is introduced in-between. we thereby provide a minimal triplet of experimental conditions ( ## discussion in this study, we revisited the classical attraction effect in subject-verb number agreement in humans and neural networks and sought to assess the influence of the attractor-verb distance. our results draw a picture of a shared distance sensitivity between humans and nlms, but also a fundamental difference in the weight of this sensitivity. we observed that the artificial system operates on the basis of word-level statistics, and is thus driven to chance-level performance in the presence of a deviant bigram (figure 3 -bottom right-incongruent trials). on the contrary, the incongruency of the sentence leads to comparable error rates irrespective of the attractor distance in humans (figure 3 -central column). the significant effect of congruency observed in the human data is due to a facilitatory effect of congruency in judging grammaticality, and not an inhibitory effect of incongruency, unlike with the neural networks. this effect is mostly evident in the agrammatical sentences and can be decomposed as follows. consider the following two sentences: