Cross-Linguistic Syntactic Evaluation of Word Prediction Models (2020)

Abstract: A range of studies have concluded that neural word prediction models can distinguish grammatical from ungrammatical sentences with high accuracy. However, these studies are based primarily on monolingual evidence from English. To investigate how these models’ ability to learn syntax varies by language, we introduce CLAMS (Cross-Linguistic Assessment of Models on Syntax), a syntactic evaluation suite for monolingual and multilingual models. CLAMS includes subject-verb agreement challenge sets for English, French, German, Hebrew and Russian, generated from grammars we develop. We use CLAMS to evaluate LSTM language models as well as monolingual and multilingual BERT. Across languages, monolingual LSTMs achieved high accuracy on dependencies without attractors, and generally poor accuracy on agreement across object relative clauses. On other constructions, agreement accuracy was generally higher in languages with richer morphology. Multilingual models generally underperformed monolingual models. Multilingual BERT showed high syntactic accuracy on English, but noticeable deficiencies in other languages.

## introduction neural networks can be trained to predict words from their context with much greater accuracy than the architectures used for this purpose in the past. this has been shown to be the case for both recurrent neural networks @xcite @xcite and non-recurrent attention-based models @xcite . to gain a better understanding of these models' successes and failures, in particular in the domain of syntax, proposals have been made for testing the models on subsets of the test corpus where successful word prediction crucially depends on a correct analysis of the structure of the sentence @xcite . a paradigmatic example is subjectverb agreement. in many languages, including english, the verb often needs to agree in number (here, singular or plural) with the subject (asterisks represent ungrammatical word predictions): to correctly predict the form of the verb (underlined), the model needs to determine that the head of the subject of the sentence-an abstract, structurally defined notion-is the word key rather than cabinets or coins. the approach of sampling challenging sentences from a test corpus has its limitations. examples of relevant constructions may be difficult to find in the corpus, and naturally occurring sentences often contain statistical cues (confounds) that make it possible for the model to predict the correct form of the verb without an adequate syntactic analysis @xcite . to address these limitations, a growing number of studies have used constructed materials, which improve experimental control and coverage of syntactic constructions @xcite @xcite . existing experimentally controlled data sets-in particular, those targeting subject-verb agreementhave largely been restricted to english. as such, we have a limited understanding of the effect of the cross-linguistic variability in neural networks' syntactic prediction abilities. in this paper, we introduce the cross-linguistic assessment of models on syntax (clams) data set, which extends the subject-verb agreement component of the @xcite challenge set to french, german, hebrew and russian. by focusing on a single lin-guistic phenomenon in related languages, 1 we can directly compare the models' performance across languages. we see the present effort as providing a core data set that can be expanded in future work to improve coverage to other languages and syntactic constructions. to this end, we release the code for a simple grammar engineering framework that facilitates the creation and generation of syntactic evaluation sets. 2 we use clams to test two hypotheses. first, we hypothesize that a multilingual model would show transfer across languages with similar syntactic constructions, which would lead to improved syntactic performance compared to monolingual models. in experiments on lstm language models (lms), we do not find support for this hypothesis; contrarily, accuracy was lower for the multilingual model than the monolingual ones. second, we hypothesize that language models would be better able to learn hierarchical syntactic generalizations in morphologically complex languages (which provide frequent overt cues to syntactic structure) than in morphologically simpler languages @xcite @xcite . we test this using lstm lms we train, and find moderate support for this hypothesis. in addition to our analysis of lstm lms, we demonstrate the utility of clams for testing pretrained word prediction models. we evaluate multilingual bert @xcite , a bidirectional transformer model trained on a multilingual corpus, and find that this model performs well on english, has mixed syntactic abilities in french and german, and performs poorly on hebrew and russian. its syntactic performance in english was somewhat worse than that of monolingual english bert, again suggesting that interference between languages offsets any potential syntactic transfer. ## grammar framework to construct our challenge sets, we use a lightweight grammar engineering framework that we term attribute-varying grammars (avgs). this framework provides more flexibility than the hard-coded templates of @xcite while avoiding the unbounded embedding depth of sentences generated from a recursive contextfree grammar @xcite ). this is done using templates, which consist of preterminals (which have attributes) and terminals. a vary statement specifies which preterminal attributes are varied to generate ungrammatical sentences. templates define the structure of the sentences in the evaluation set. this is similar to the expansions of the s nonterminal in cfgs. preterminals are similar to nonterminals in cfgs: they have a lefthand side which specifies the name of the preterminal and the preterminal's list of attributes, and a right-hand side which specifies all terminals to be generated by the preterminal. however, they are non-recursive and their right-hand sides may not contain other preterminals; rather, they must define a list of terminals to be generated. this is because we wish to generate all possible sentences given the template and preterminal definitions; if there existed any recursive preterminals, there would be an infinite number of possible sentences. all preterminals have an attribute list which is defined at the same time as the preterminal itself; this list is allowed to be empty. a terminal is a token or list of space-separated tokens. the vary statement specifies a list of preterminals and associated attributes for each. typically, we only wish to vary one preterminal per grammar such that each grammatical case is internally consistent with respect to which syntactic feature is varied. the following is a simple example of an attribute-varying grammar: preterminals are blue and attributes are orange. here, the first statement is the vary statement. this is followed by a template, with the special s keyword in red. all remaining statements are preterminal definitions. all attributes are spec-ified within brackets as comma-separated lists; these may be multiple characters and even multiple words long, so long as they do not contain commas. the output of this avg is as follows (true indicates that the sentence is grammatical): true je pense false je penses false je pensons false je pensez this particular grammar generates all possible verb forms because the attribute list for v in the vary statement is empty, which means that we may generate any v regardless of attributes. one may change which incorrect examples are generated by changing the vary statement; for example, if we change v[] to v[1], we would only vary over verbs with the 1 (first-person) attribute, thus generating je pense and *je pensons. one may also add multiple attributes within a single vary preterminal (implementing a logical and) or multiple semicolon-separated vary preterminals (a logical or). changing v[] to v[1,s] in the example above would generate all first-person singular v terminals (here, je pense). if instead we used v[1]; v[s], this would generate all v terminals with either first-person and/or singular attributes (here, je pense, *je penses, and *je pensons). ## syntactic constructions we construct grammars in french, german, hebrew and russian for a subset of the english constructions from @xcite , shown in figure 1 . these are implemented as avgs by native or fluent speakers of the relevant languages who have academic training in linguistics. 3 a number of the constructions used by marvin and linzen are english-specific. none of our languages besides english allow relative pronoun dropping, so we are unable to compare performance across languages on reduced relative clauses (the author the farmers like smile/*smiles). likewise, we exclude marvin and linzen's sentential complement condition, which relies on the english-specific ability to omit complementizers (the bankers knew the officer smiles/*smile). the @xcite data set includes two additional structure-sensitive phenomena other than subject-verb agreement: reflexive anaphora ## conclusions in this work, we have introduced the clams data set for cross-linguistic syntactic evaluation of word prediction models, and used it to to evaluate monolingual and multilingual versions of lstms and bert. the design conditions of @xcite and our cross-linguistic replications rule out the possibility of memorizing the training data or relying on statistical correlations/token collocations. thus, our findings indicate that lstm language models can distinguish grammatical from ungrammatical subject-verb agreement dependencies with considerable overall accuracy across languages, but their accuracy declines on some constructions (in particular, center-embedded clauses). we also find that multilingual neural lms in their current form do not show signs of transfer across languages, but rather harmful interference. this issue could be mitigated in the future with architectural changes to neural lms (such as better handling of morphology), more principled combinations of languages (as in dhar and bisazza 2020), or through explicit separation between languages during training (e.g., using explicit language ids). our experiments on bert and mbert suggest (1) that mbert shows signs of learning syntactic generalizations in multiple languages, (2) that it learns these generalizations better in some languages than others, and (3) that its sensitivity to syntax is lower than that of monolingual bert. it is possible that its performance drop in hebrew and russian could be mitigated with fine-tuning on more data in these languages. when evaluating the effect of the morphological complexity of a language on the lms' syntactic prediction accuracy, we found that recurrent neural lms demonstrate better hierarchical syntactic knowledge in morphologically richer languages. conversely, mbert demonstrated moderately better syntactic knowledge in morphologically simpler languages. since clams currently includes only five languages, this correlation should be taken as very preliminary. in future work, we intend to expand the coverage of clams by incorporating language-specific and non-binary phenomena (e.g., french subjunctive vs. indicative and different person/number combinations, respectively), and by expanding the typological diversity of our languages. long verb-phrase coordination is similar, but makes each verb phrase much longer to introduce more distance and attractors between the subject and target verb. now we have more complex structures that require some form of structural knowledge if a model is to obtain the correct predictions with more than random-chance accuracy. agreement across a subject relative clause involves a subject with an attached relative clause containing a verb and object, followed by the main verb. here, the attractor is the object in the relative clause. (an attractor is an intervening noun between a noun and its associated finite verb which might influence a human's or model's decision as to which inflection to choose. this might be of the same person and number, or, in more difficult cases, a different person and/or number. it does not necessarily need to occur be agreement within an object relative clause requires the model to inflect the proper verb inside of an object relative clause; the object relative clause contains a noun and an associated transitive verb whose object requirement is filled by the relative pronoun. the model must choose the proper verb inflection given the noun within the relative clause as opposed to the noun outside of it. this may seem similar to simple agreement, but we now have an attractor which appears before the noun of the target verb. agreement across an object relative clause is similar, but now the model must choose the correct inflection for the noun outside of the relative clause. this requires the model to capture long-range dependencies, and requires it to have the proper structural understanding to ignore the relative clause when choosing the proper inflection for the focus verb. finally, agreement across a prepositional phrase entails placing a prepositional phrase after the subject; the prepositional phrase contains an attractor, which makes choosing the correct inflection more difficult.