Are T ransformers a Modern Version of ELIZA ? O bservations on F rench Object Verb Agreement (2021)

Abstract: Many recent works have demonstrated that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. We take a critical look at this line of research by showing that it is possible to achieve high accuracy on this agreement task with simple surface heuristics, indicating a possible flaw in our assessment of neural networks’ syntactic ability. Our fine-grained analyses of results on the long-range French object-verb agreement show that contrary to LSTMs, Transformers are able to capture a non-trivial amount of grammatical structure.

## introduction the long distance agreement task is one of the most popular method to assess neural networks (nn) ability to encode syntactic information: @xcite showed that lstms are able to predict the subject-verb agreement in english and has initiated a very active line of research. since then, many studies have generalized this observation to other languages @xcite , other models such as transformers @xcite or have identified possible confounding factors that could distort the stated conclusions @xcite . all of these studies show that nn are able to learn a 'substantial amount' of syntactic information @xcite . in this work, we propose to take an alternative look at these results by studying whether neural networks are able to predict the correct form of a verb because they are able to build an abstract, high-level (maybe hierarchical) sentence representation @xcite or solely because they capture surface statistical regularities, as suggested by several recent work @xcite @xcite . overall, this set of results questions one of the most fundamental assumption in linguistics @xcite , namely that a sentence has a recursive structure @xcite : while lstms with proper parametrization can model context-free patterns @xcite , transformers are essentially feed forward models relying on a large number of attention heads. consequently, they are, in theory, not adapted to model hierarchical syntactic patterns @xcite and explaining their capacity to predict accurately syntactic agreement patterns remains an open issue. we bring new light on this problematic by identifying simple heuristics ( §4) that can be used to correctly predict verbal agreement, pushing further the observation of @xcite that a simple rule can provide highly accurate results on the task. using our extended set of heuristics, we identify sentences for which predicting the correct verb form requires a more abstract representation of the sentence. by comparing models' performance on these examples, we show that contrary to lstms, transformers perform consistently well in these critical cases. we focus on the object-verb agreement (i.e. object past-participle agreement) in french: agreement in number and gender occurs between the object and the past participle when the latter is used with the auxiliary avoir (to have) and the object is located before the verb. as shown in figure 1 , this is, for instance, the case for past participles in object relatives. when agreement is required, a -s suffix (resp. to predict the past participle agreement in object relatives, a model has to identify the object relative pronoun, its antecedent and the auxiliary. it has also to ignore the effect of attractors (nouns with misleading agreement features) occurring between the object and the past participle. compared to the subject-verb agreement, the french object past participle agreement is more difficult as the target verb form depends on a noun that is never adjacent to the verb. the auxiliary avoir before the target verb could also be an attractor. 2 we restrict ourselves to the number agreement between object and past participle in the case of object relatives to ( 1 ) design reasonably simple patterns that can be easily extracted automatically from raw texts, (2) extract a sufficiently large number of representative examples and (3) reduce the importance of the anaphoric resolution problem. these restrictions allow us to carry out a finegrained analysis of nn ability to extract syntactic generalizations from non-annotated corpora ( §4). building a test set sentences used in the number agreement task are extracted automatically from the 8,638,145 sentences of the french gutenberg corpus. 3 we use the flaubert parser @xcite and the pretrained french model of spacy @xcite to automatically parse the sentences of the gutenberg project. we also consider the gold annotations of the 45,088 sentences of the french universal dependency treebanks @xcite to evaluate the impact of parsing errors on the corpus quality. we extract examples of object-verb agreement from sentences' syntactic and morphological annotations using simple rules, 4 resulting in a corpus of 104 sentences (68% singular and 32% plural) extracted from the ud treebank and of 68,794 sentences (65% singular and 35% plural) extracted from the gutenberg project. in french, the singular is identical to the unmarked form of the past participle verbs, making the frequency statistics unbalanced in favor of singular. we evaluate the quality of our automatic extraction procedure by comparing the examples extracted using the gold annotations of ud treebank to those extracted from predicted annotations of 2 see example(1) in figure 1 , a (has_3sg) could be a number attractor for target verb acceptées(accepted_pl) ## language models we contrast two types of incremental language models in our experiments: lstm models and incremental transformer models. both models are modeling the probability of a sentence x as: all neural models are trained to compute p (x i |x 1 . . . x i-1 ) and they all use the same generic template: our lstm models use 2 layers while our transformer language model use 16 layers and 16 heads. both models are using embeddings of size 768 and are trained on the same data. for transformers we add positional embeddings to the word embeddings e i using the cosine scheme and weighting described by @xcite . since all the models use a word-based tokenization and not a subword tokenizer, we bound the vocabulary to the 50,000 most frequent tokens found in the training data and use @xmath0. ## experimental results in our experiments, following @xcite and @xcite , we compare the probabilities a language model assigns to the singular form of the target participle and its plural form given a prefix. 6 we consider the model has predicted the agreement correctly if the form with the correct number has a higher probability than the form with the incorrect number. however, we believe that this conclusion must be taken with great care: because of confounding factors, a language model could predict the correct form without actually capturing syntactic information. for instance, as our test set is unbalanced (section 2) a naive model always choosing the singular form of the participle achieves an accuracy of 65%, a score that puts into perspective the performance of lstms. more importantly, @xcite and @xcite observed that the agreement task can be partially solved by collocational information or a simple heuristic, namely the number of the first noun of the prefix. in the following, we propose several experiments to strengthen these first results. ## conclusions we ran a fine-grained analysis of nn's syntactic generalization capabilities in processing french object-verb agreement for grammatical number, a phenomenon crucially depending on hierarchical syntactic structure. we designed a new evaluation protocol based on four shallow heuristics that the models could adopt to perform number agreement task. our experiments show that, contrary to lstms, transformers extract a non trivial amount of syntactic information. in future work, we will investigate the kind of syntactic information transformers are encoding and the relationship between the superficial heuristics and hierarchical syntactic structure processing in transformer models. in particular, our results intriguingly suggest that transformers rely on word order information to predict verb agreement, despite the fact that they don't model word order explicitly beyond marking each word with its absolute-position embedding. we plan to study this question in future work.