Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT (2022)

Abstract: Both humans and neural language models are able to perform subject verb number agreement (SVA). In principle, semantics shouldn’t interfere with this task, which only requires syntactic knowledge. In this work we test whether meaning interferes with this type of agreement in English in syntactic structures of various complexities. To do so, we generate both semantically well-formed and nonsensical items. We compare the performance of BERT-base to that of humans, obtained with a psycholinguistic online crowdsourcing experiment. We find that BERT and humans are both sensitive to our semantic manipulation: They fail more often when presented with nonsensical items, especially when their syntactic structure features an attractor (a noun phrase between the subject and the verb that has not the same number as the subject). We also find that the effect of meaningfulness on SVA errors is stronger for BERT than for humans, showing higher lexical sensitivity of the former on this task.

## introduction subject verb agreement (sva) is a grammatical constraint in english and several other natural languages, such that verbs must agree in number with their subject. linguistic theories generally assume that sva obeys two main principles: i.) structure dependence (sd) -sva is governed by phrasal structure, rather than surface linear order (i.e., the verb agrees with the syntactic subject); ii.) meaning independence (mi) -sva is a morphosyntactic constraint that holds for meaningless sentences too (e.g., colorless green ideas sleep furiously) @xcite @xcite . however, previous research has shown that humans are prone to making agreement errors with specific constructions @xcite , for example when a noun with a different number (also called an attractor) occurs between the subject (the cue) and the verb (the tar-get). see (1) from @xcite for an example where agreement can be disturbed by an attractor, as human subjects often show preference for a syntactically ill-formed sentence: this evidence suggests that the sd principle of sva might be weaker than it is typically assumed and can be disrupted or disturbed under specific conditions. at the same time, such violation prompts the need to carefully test whether the mi principle of sva is also compromised in subjects' grammaticality judgments. sva has become a widespread testbed to investigate the syntactic knowledge that neural language models (nlms) are able to acquire. a key question is to explore to what extent their competence of syntax obeys the same constraints as those of humans, by comparing the behavior of nlms with subjects' judgments. in this paper, we pursue this goal by focusing on the sd and mi properties of sva in humans and nlms. its contribution is twofold: i.) we collect original human data on meaningful and meaningless stimuli featuring syntactic structures of varying complexities; ii.) we analyze and compare the error patterns in humans and in nlms. this allows us to address the following questions: i.) are the sd and mi assumptions about sva truly supported by human judgments? ii.) do humans and nlms make similar sva errors in structures with attractors and/or in meaningless sentences? this may help understand to what extent nlms rely on syntactic knowledge abstracted from training examples. by comparing the error patterns of humans and nlms on sva, we can derive important information about the the nature of their linguistic competence: is the ability of nlms on sva completely meaning independent? is it influenced by the complexity of the syntactic structure? 2 related work @xcite tested long short term memory (lstm) language models and their performance on the sva task. they found that these models can capture a non-trivial amount of grammatical structure but that they are insufficient for capturing complex syntax-sensitive dependencies. in marvin and linzen (2018) the capacity of lstm language models to perform the sva task is compared with human data. in particular for sentences including attractors, the models perform worse than humans. linzen and leonard (2018) tested recurrent neural networks (rnns) on the task to see whether the error pattern was similar to humans. they concluded that despite important similarities, there was a different behavior within relative clauses. in particular, rnns are sensitive to the number of attractors, whereas humans are not. @xcite investigated the role of semantics on the performance of rnns on sva, testing such neurals models against meaningless, or "nonce", sentences built from various syntactic constructions. they used meaningless sentences where rnns could not rely on semantic or lexical cues. for italian, they found that the performance of an lstm model and the performance of humans were comparable for meaningless items and semantically congruent ones. @xcite tested a transformer model, bert-base @xcite , on its capacity of subject-verb agreement on the items from marvin and linzen (2018) and on semantically disrupted sentences featuring the same syntactic constructions. they found that although the model generalized well for simple templates, it failed on meaningless items with attractors. ## experimental setup in this section, we describe the procedure to construct the experimental items used to collect human judgments with crowdsourcing and to test nlm behavior on sva. items we test the syntactic structures in we also sample 30 meaningful sentences for each structure from marvin and linzen (2018) to collect human data. 1 as the filtered m&l data used to collect human performances resulted in a very limited number of sentences built on a limited vocabulary, we extract bert's performance on the whole dataset for m&l, following the procedure in @xcite . we thus collect human performance on 30 items for each of our 2 conditions (nonce and m&l), and each of our 9 syntactic structures, for a total of 540 items. more details are given in appendix b. foot_1 neural language model our tested nlm is bert @xcite , a bi-directional pretrained transformer model. bert has been shown to possess a number of syntactic abilities @xcite , the nature of which remain scarcely understood @xcite . for each item, we present bert with sentences masked at the target position and compare the probability that bert assigns to each verb inflection. ## results we first discuss the human data, which we then compare with bert's performance. ## conclusion throughout this work, we have shown that the ability to perform sva is highly dependent on the syntactic construction when presented with meaningless sentences. the failures of humans seem to align well with those of bert overall, as sentences with attractors tend to compromise meaning independence when processing the agreement relation. despite these similarities, we further show that the performance drop is generally higher in bert on meaningless sentences, and that humans are more perturbed by complex constructions without an attractor. this finding can in turn reflect differences in processing syntactic structure, either reflecting more reliance on local context for bert, or a difference between agreement processing in production and comprehension, which could be the source of the partial mismatch in the observed error patterns. karim lasri, alessandro lenci, and thierry poibeau. 2022. does bert really agree ? fine-grained analysis of lexical dependence on a syntactic task. in findings of the association for computational linguistics: acl 2022, pages 2309-2315, dublin, ireland. association for computational linguistics. tal linzen, emmanuel dupoux, and yoav goldberg. 2016. assessing the ability of lstms to learn syntaxsensitive dependencies. transactions of the association for computational linguistics, 4:521-535. tal linzen and brian leonard. 2018. distinct patterns of syntactic agreement errors in recurrent networks and humans. corr, abs/1807.06882. rebecca marvin and tal linzen. 2018. targeted syntactic evaluation of language models. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1192-1202, brussels, belgium. association for computational linguistics. george a miller. 1995. wordnet: a lexical database for english. communications of the acm, 38(11):39-41. jonathan w peirce. 2007. psychopy-psychophysics software in python. journal of neuroscience methods, 162(1-2):8-13. robyn speer, joshua chin, andrew lin, sara jewett, and lance nathan. 2018. luminosoinsight/wordfreq: v2.2.