Refining Targeted Syntactic Evaluation of Language Models (2021)

Abstract: Targeted syntactic evaluation of subject-verb number agreement in English (TSE) evaluates language models’ syntactic knowledge using hand-crafted minimal pairs of sentences that differ only in the main verb’s conjugation. The method evaluates whether language models rate each grammatical sentence as more likely than its ungrammatical counterpart. We identify two distinct goals for TSE. First, evaluating the systematicity of a language model’s syntactic knowledge: given a sentence, can it conjugate arbitrary verbs correctly? Second, evaluating a model’s likely behavior: given a sentence, does the model concentrate its probability mass on correctly conjugated verbs, even if only on a subset of the possible verbs? We argue that current implementations of TSE do not directly capture either of these goals, and propose new metrics to capture each goal separately. Under our metrics, we find that TSE overestimates systematicity of language models, but that models score up to 40% better on verbs that they predict are likely in context.

## introduction as neural language models have emerged as both broadly useful engineering tools @xcite and potential models of human language processing @xcite @xcite , evaluations targeting their syntactic ability have been developed to better understand their capabilities. one such method for syntactic evaluation tests models' knowledge of english subject-verb (s/v) number agreement @xcite . these studies consider minimal pairs of sentences, such as the keys to the cabinet is/are on the table, that differ only in verb number, and test if models rate grammatical sentences as more probable. the syntactically correct of the two sentences is sampled from natural corpora @xcite or constructed from templates. the use of templates, known as targeted syntactic evaluation (tse), allows for the fine-grained evaluation of models on specific, often rare, syntactic phenomena @xcite @xcite , but (when evaluating s/v number agreement) relies on researchers hand-specifying a small set of verb lemmas that are substituted into each template. in this work, we improve the tse methodology by disentangling its broad objective of evaluating syntactic ability into two distinct goals, and we introduce two variants of tse to separately capture each goal. these evaluations demonstrate that neural models do not generally consider wellconjugated verbs more likely than their incorrect conjugations, but instead prefer to correctly conjugate verbs they deem likely. we argue that the objective of evaluating syntactic ability can be decomposed into two goals and that current implementations of tse do not achieve either of them. the first goal is measuring systematicity: for a specific syntactic construction, does the model correctly conjugate arbitrary verbs with the grammatical number of the subject? tse currently fails to capture this because it evaluates models using only a small set of verbs for each syntactic construction. if models only conjugate these verbs correctly, they receive a high score, even if they conjugate other verbs incorrectly. the second goal is measuring likely behavior: when we sample verbs from the model in a specific syntactic construction, will they be properly conjugated? tse fails to directly capture this because the small set of verbs used in evaluation might differ from the verbs that are likely in context under the model. if models conjugate these hand-specified verbs incorrectly, they receive a low score, even if they correctly conjugate more likely verbs. to motivate these goals and the misspecification of tse, consider evaluating a language model on the keys to the cabinet on the table. ## methods to define our metrics, we introduce some notation. tse has two components: the model m to evaluate, and the set of templates t with interesting syntactic phenomena (e.g., from @xcite ). in s/v number agreement, each template contains a context c, including the subject that specifies the correct verb inflection; and the verb lemma ℓ with correct and incorrect inflections in the third person present tense (ℓ + and ℓ -, respectively). m takes c and produces a distribution p m (• | c) over its vocabulary, which we assume includes ℓ + and ℓ -. we then compute a score for each template and average the scores over all templates to get a final score for m . the tse score for a template can be expressed as: the crux of our proposal is to use a large set of lemmas, l, while drawing contexts c from a predefined set of templates t . we define two evaluation methods using l: equally-weighted (ew) here we average (1) over all ℓ in l, evaluating systematicity. ## experiments data we use s/v number agreement tse templates from @xcite and blimp @xcite (for blimp we use the minimal pairs differing in verb, not subject). for our mw and ew evaluations, we only keep templates with unique contexts (i.e., templates not differing solely in verb lemma). we also ensure that all sentences start with a capital letter (for cased models) and end with a sentence-final period (for bidirectional models). our list of english verb lemmas contains 3,562 lemmas and is compiled by combining the top 1,000 most frequent verb lemmas from coca, extracting all tokens with the vb part-of-speech tag in the penn treebank (1,951 lemmas), and scraping 3,250 lemmas from the giant verb list @xcite @xcite . 2 masked lms may assign a different number of tokens to plural and singular forms of the same lemma, and they may not model joint probabilities over multiple tokens. to enable a fairer comparison between lms and masked lms, we only consider lemmas where both inflections are in the wordpiece vocabulary of the models. this choice leaves 980 lemmas for bert cased, 1159 for bert uncased, and 1265 for gpt2 and roberta (so results are not comparable between models). this verbal variety situates our work between @xcite al. (2018 models we evaluate both bidirectional and unidirectional models, including bert-large-uncased, bert-large-cased, gpt2-xl, and roberta-large @xcite @xcite , all using the huggingface tranformers library @xcite ). 2 the verb lemmas are accessible from the appendix. to understand models' performances at the head and tail of their distributions, we additionally restrict l to the lemmas assigned high and low probabilities. to consider the high-confidence lemmas, for each template in the dataset, we record the mw and ew scores computed using the inflections that fall into the top p percentile of the model's distribution. we choose @xmath0, 20, 30, 40, 50, 60, 70, 80, 90, 95, 97, 100}, noting that for each p, the distribution we use is the same as the one used by nucleus sampling (with a nucleus of size p). analogously, to focus on the low-confidence lemmas, we consider the lemmas where both inflections fall into the bottom p percentile of the model's distribution. here, we choose @xmath1, 10, 1, 0.1, 0.01, 0.001, 0.0001}. foot_1 ## results our results can be found in a potential confounder for unidirectional lms (gpt2) is that they only receive the left context and subject verb pairs sometimes look like noun phrases. for example, a sentence starting with the officer can be continued by experiences joy or by experience is overwhelming. this is not an issue when there are phrases or clauses between the subject and verb, and it may not occur for other english syntactic phenomena or in other languages. to investigate the extent to which models perform well on likely lemmas and poorly on unlikely lemmas, we plot these scores for the top and bottom p percentiles in figure 1 . in general, the models perform better on lemmas that they assign high probability to in both evaluations. for example, consider the bert cased model assessed on object relative clause constructions. the mw plot illustrates that sampling from the top 60% of the distribution will produce a grammatical output with 97% probability, while sampling from the entire distribution only does so with 91% probability. the ew plot shows that the model attains a score under 80% when assessed on verbs in the bottom 0.001% of the model's probability mass, even though considering verbs in the top 90% of the model's probability mass would yield a score over 94%. these observations extend previous work on nucleus sampling, showing that cutting off the tails of the distribution generates more syntactically correct outputs @xcite . there are two additional factors to keep in mind for these plots. first, the heads and tails of the distributions often contain very few lemmas eligible for use in score calculation. second, models often assign probability mass to other lemma inflections (e.g. the past tense) that do not allow us to assess models' s/v number agreement ability. see the appendix for related plots. ## related work evaluating models some previous work has focused on using minimal pairs to evaluate syntactic representations of models. @xcite and @xcite assess the syntactic abilities of large transformers like bert and gpt2, while @xcite @xcite and recurrent neural network grammars @xcite ). in these cases, minimal pair evaluations should align with models' performance as language models, which is measured by our mw score. psycholinguistics recent work has also applied experimental procedures from psycholinguistics to compare human and neural model language processing @xcite . experiments investigating garden path sentences' surprisal, s/v number agreement, and other specific syntactic phenomena reveal that models and humans have different patterns of errors and processing @xcite @xcite . many of these phenomena are rare, so evaluations with templated minimal pairs complement perplexity as a metric for evaluating models' syntactic generalization @xcite . when measuring syntactic ability on arbitrary lemmas, our ew metric would be preferred. ## conclusion as neural models have proven successful at nlp tasks and as potential psycholinguistic models, we continue to refine our understanding of how and whether they capture human-like language faculties. tse provides a useful framework to address this question, but its current implementation focuses on a limited group of hand-chosen verbs, so it inaccurately reflects models' syntactic generalization abilities. in response, we propose two minimal pair evaluations: equally-weighted and model-weighted syntactic evaluation. the first focuses on systematicity by expanding the set of verbs tse considers, and illustrates that language models still struggle with s/v agreement for unlikely verbs. the second focuses on likely behavior by computing the probability of producing a correctly conjugated verb, and illustrates that despite systematic shortcomings, language models generate syntactically valid utterances with high probability. by introducing these metrics, we hope to arrive at a clearer picture of the syntactic abilities of language models.