Language model acceptability judgements are not always robust to context (2023)

Abstract: Targeted syntactic evaluations of language models ask whether models show stable preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Our best syntactic evaluation datasets, however, provide substantially less linguistic context than models receive during pretraining. This mismatch raises an important question: how robust are models’ syntactic judgements across different contexts? In this paper, we vary the input contexts based on: length, the types of syntactic phenomena it contains, and whether or not there are grammatical violations. We find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but are unstable when contexts match the test stimuli in syntactic structure. Among all tested models (GPT-2 and five variants of OPT), we find that model performance is affected when we provided contexts with matching syntactic structure: performance significantly improves when contexts are acceptable, and it significantly declines when they are unacceptable. This effect is amplified by the length of the context, except for unrelated inputs. We show that these changes in model performance are not explainable by acceptability-preserving syntactic perturbations. This sensitivity to highly specific syntactic features of the context can only be explained by the models’ implicit in-context learning abilities.

## introduction the unprecedented progress in the development of neural large language models (llms; @xcite @xcite has been accompanied by a comparable proliferation of methods that aim to better understand and characterize models' linguistic capacities @xcite @xcite @xcite . of the many methods for this, the minimal-pair paradigm (mpp), each block represents a sentence: red striped blocks are unacceptable sentences within a given task, while green solid ones are acceptable. we also vary the diversity of prefixes by sampling them from tasks/datasets different from the test suite (indicated by shape). studies that employ mpp datasets generally compare the probability of two stand-alone text sequences without any explicit linguistic context. however, this is not a naturalistic or realistic ap-proach: utterances usually occur in some linguistic context, where the context itself could affect linguistic preferences. the syntactic priming literature investigates the effect of linguistic contexts to some extent, but mostly in a constrained setting with only one or a small number of context sentences @xcite . the interaction of context with minimal pair accuracies remains underexplored for multi-sentence contexts, despite the fact that multisentence inputs are more likely for many nlp tasks-especially with the rise of prompting and in-context learning @xcite . furthermore, transformer-based language models are typically trained on large sequences, where masked tokens are predicted given a completely full context window, consisting of many sentences. it is unclear how to evaluate mpp by utilizing this context window, given recent research that has raised questions about the sentence representations acquired in long-form input @xcite . we evaluate the sensitivity of llms' acceptability preferences in a more realistic evaluation setting, with one or more additional sentences in the input context. we focus on llm sensitivity to three particular features of the context: (1) the length of the input sequence, (2) the similarity of the context to the minimal pair being judged, and (3) whether the context itself contains acceptability violations. figure 1 illustrates our method at a high level: for a given mpp dataset @xcite , we generate new minimal pair test examples for a given syntactic phenomenon by artificially simulating a long context window. specifically, we prepend the given test example pair with sentences drawn by the axis of similarity, from unrelated (wikipedia), minimalpair sentences from different (mismatched) or the same (matched) syntactic phenomena in the mpp dataset. we also introduce violations in the context by drawing unacceptable counterparts of the above similarity scale from the mpp dataset. we find that the model's judgements are highly robust to the presence of unrelated wikipedia sentences in the context, regardless of the size of the prefix. however, we observe strong sensitivity to matched context manipulations. as the context length increases, acceptable matched contexts improve the models' judgements significantly. conversely, we observe a strong opposite effect of ex-posing the model to longer and longer prefixes containing acceptability violations: models' judgements degrade drastically, performing far below chance. this sensitivity is specific to the particular type of syntactic structural similarity of the context: we do not see the same degree of improvement/degradation in prediction behavior for contexts consisting of mismatched sentences of valid or violated syntactic structures. to better understand our results, we performed several exploratory analyses. to determine whether the results are an effect of the acceptability judgement task, we replicated our experiments for another task, that of stereotypicality judgements @xcite , and found largely concurring results. we also investigated the syntactic overlap between the context and the test pair, and observe only minor effects on the judgements with phenomenapreserving syntactic perturbations. our results, therefore, can only be explained by the model displaying some kind of implicit, instruction-free, in-context learning ability, and they invite further scrutiny of and investigation into long-form sentence understanding capabilities of llms. ## approach terminology. we follow standard practice in mpp, where we evaluate the preference (p) of a language model m towards acceptable sentence (x) over its unacceptable counterpart (x ′ ), with respect to log-likelihood, and compute the value over the full evaluation dataset d. d typically consists of several test suites, each of which instantiates a particular linguistic phenomenon. we denote the particular test suite under evaluation as the target suite: s ⊂ d. each target suite consists of k pairs of acceptable and unacceptable sentences, (x, x ′ ) k @xmath0, and may have multiple conditions. within each target suite, we compute the acceptability judgements on one or more experimental conditions, comparing a given lm's log-likelihood preference p for the acceptable and unacceptable sentence in each condition. the accuracy (a) over a test pair from a single condition is defined as: to simulate increasing length of input, we prepend a prefix sequence c to both x and x ′ , and compute the preferences over the concatenated sequence, p([c, x i ]) and p([c, x ′ i ]), where c can be arbitrarily long. ## main results figure 2 presents the summary results of our prefixing manipulation, charting models' accuracy on mpp evaluations as a function of the prefix (1) length (x-axis), (2) acceptability (teal and orange vs. red and purple), and (3) whether it is drawn from a domain that is matched (teal and purple), mismatched (orange and red), or unrelated wikipedia (light green). we further explore the main qualitative findings in the following paragraphs, plotting results on the blimp dataset for simplicity. detailed results on syntaxgym and crows-pairs are available in appendix f and a, respectively. model acceptability judgements are largely robust across lengths-for unrelated, control prefixes. we first investigate the impact of increasing context length on model acceptability judgement performance. we start with the control case defined in §3, simulating lengthy context windows with no other notable grammatical properties by drawing sentences from wikipedia, an out-of-distribution text domain for the target mpp datasets. as we increase the context length, acceptability judgement results do not significantly change (figure 3 , long dashed lines), suggesting that lms, in general, are very robust to unrelated changes in their context window. quantitatively, no main effect of prefix length is significant (@xmath1.2 for all models) for wikipedia sentences. the length of the context matters when the prefix is related to the acceptability task. we next investigate the effect of long context on acceptability by drawing prefixes that are in-distribution (from the same mpp dataset). as prefix length grows, model performance on average changes monotonically from baseline accuracy (figure 2 : rising for acceptable sentences, falling for unacceptable ones). when the prefix consists of acceptable sentences (teal, orange) for example, ∆ accuracy increases up to 10-20 percentage points for all datasets, and mostly uniformly across all model sizes. however, unacceptable prefixes (purple, red) elicit the opposite effect: ∆ accuracy falls as context context length grows (figure 2 , dashed lines). scale amplifies this effect only for unacceptable sentences (figure 3 ). for example, opt 6.7b suffers the largest degradation of acceptability task accuracy with increasing length of ungrammatical context, compared to gpt2. surprisingly, gpt2 recovers some percentage of the degradation on very long sequences, while also showing attenuated the matched gains. we speculate that this effect derives from a relative weakness of gpt2 to learn in-context, as it is trained on markedly less data (8b tokens, as estimated by @xcite than models from the opt family (180b tokens). quantitatively, this interaction between prefix length and acceptability is highly significant for all models and evaluations (@xmath2.002 for all models on blimp and syntaxgym). overall, we observe length can influence lm's acceptability judgement performance for in-distribution contexts, and more so when the contexts contain acceptability violations. one possible driver for these results could be that longer contexts are more conducive to large lms' in-context learning abilities, and mimic their k-shot learning scenario. this would mean that the length of preceding context matters only insofar as length is a proxy for the number of acceptable (or unacceptable, with an opposite effect) matched prefixes in the context (see §5 for a related analysis). matched context impacts acceptability judgements more than mismatched contexts. we now dig into the interaction between length and acceptability, investigating whether the magnitude of the effect is modulated by whether the phenomena are matched or not. in case of blimp, the average effect of acceptable prefixes @xmath3. however, matched prefixes drive this improvement more (∆ ≥ 15) than mismatched ones (∆ ≤ 5) (figure 4 , left subfigure). conversely, while the average effect of unacceptable prefixes is between 30-40 ∆ accuracy points (figure 3 ), this too is more heavily impacted by the effect of matched prefixes (50 ≤ ∆ ≤ 80) than by mismatched ones (∆ ≤ 20) (figure 4 , right subfigure). these effects manifest quantitatively in a three-way interaction between prefix (un-)acceptability, (mis-)match, and length (@xmath4.007 for all models on blimp and syntaxgym). the effects of unacceptable prefixes are amplified substantially when they are consistent-i.e., when they violate the grammatical rules (of english) in the same way (matched), as opposed to in more diverse ways (mismatched). 7 these results could explain why in-context learning ability works: perhaps prepending contexts that are syntactic similarity can help the model learn or unlearn acceptability at a higher rate. ## prefix similarity analysis we have observed that length effects on acceptability judgements are conditional on the similarity between the prefix phenomenon and the test phenomenon. however, we have only analyzed prefixes that are either very similar (i.e. contain predominantly the same abstract syntactic structure as the test sentence, matched prefixing), or are almost entirely unrelated (mismatched prefixing, or unrelated prefixing such as wikipedia). this leads us to wonder about the nature of the similarity driving our results thus far: are the models responding to the presence of shared syntactic structure in the prefix? or are they responding to something more shallow and brittle, such as the exact match in sentence templates between the prefix content and the test? if the former is true, we should see a smooth relationship between prefix syntactic similarity and length effects, such that slight changes in the syntactic structure of the prefix content results in similarly slight modulations of length effects. if models are using more shallow template-based comparisons between the prefix content and the test content, we might see a more discontinuous response, in which even small changes to prefix content result in large changes in length effects. to test this, we narrow our focus to the top 20 7 note, however, (i) we assumed that all wikipedia sentences are acceptable, and (ii) we found that acceptable prefixes have a generally weaker effect on the acceptability task. were we to test unacceptable wikipedia sentences as well, we might expect a small priming effect. blimp phenomena which responded most strongly to matched prefixing in our previous analyses. 8 we perform controlled perturbations on each prefix sentence c that preserve the presence of the original syntactic structure, but incorporate mild structural variations or additions. these perturbations increase prefix length and shift the position of certain tokens (e.g., the main verb) in c relative to their counterparts in the test sentence. this enables us to test whether the models are merely learning to associate fragile token-position pairings between the prefix and test sentences, or whether they are relying on relevant abstract syntactic information. we leave the test sentence x unchanged. our perturbations include the following, all of which preserve both the grammaticality and the relevant overarching syntactic structure of the blimp phenomena: we also combine all of these strategies into a single large perturbation, referred to as all. foot_7 our findings (figure 5 ) show that minor perturbation of the prefixes results in only very minor reductions to length effects, suggesting that matched prefixing effects do not require identically structured prefixes. increasingly aggressive perturbations result in increasing (if small) reductions to ∆ accuracy magnitudes, especially when using fewer prefixes. we correlate ∆ accuracies with the mean similarity of prefix sentences before and after a perturbation, where similarity is an ordinal variable assigned to each perturbation based on how many tokens it adds to the sentence; see app. b for details. the spearman rank-order correlation (ρ @xmath5.93, @xmath6.001) is significantly positive for acceptable prefixes; it is weaker but still significantly negative (ρ @xmath7.7, @xmath8.05) for unacceptable prefixes. thus, there is a smooth relationship between prefix similarity and length effects. this perturbation analysis shows that model judgments are mostly robust to syntactic variations in the prefix content, with a smooth relationship between degrees of syntactic variation and model performance. appendix d investigates whether these similarity effects can be described in terms of lexical overlap or matches in low-level syntactic features between the prefix and test content; we find no clear relationship between these low-level features and models' acceptability judgment performance. taken together, these results suggest that the changes we observe in models' acceptability judgments are likely due to an abstract comparison between structural features of the prefix content and test content. in other words, language models are sensitive to latent syntactic features, and the syntactic similarity of the context to the test examples. more broadly, our work adds to the literature on prompt sensitivity in pre-trained language models, which found that lms are sensitive to individual prompts @xcite , and that the ordering of in-context examples @xcite can greatly affect model performance. smaller lms are also sensitive to the choice of prompt and output verbalizer @xcite , and we indeed observe that a variety of model sizes and prefixing strategies elicit prefix sensitivity. to our knowledge, our study is the first to consider structural priming in concert with in-context learning; we have found quantitative, graded effects of structural priming on string probabilities, subject to the length of the context. language models are sensitive to latent syntactic features, as well as syntactic similarities across multiple sentences. our analyses add to a literature that has found that language models are sensitive to more than just lexical or surface-level syntactic features @xcite . indeed, lms are capable of leveraging abstract syntactic features, and are sensitive to latent syntactic similarities between the context and test examples. strengthening this finding, we also observe that models are capable of adapting to the structures of both acceptable and unacceptable examples: lms show marked improvements on acceptability tasks when prefixed by matched acceptable sentences, and they also (more substantially) show the opposite behavior-preferring unacceptable sentences-when prefixed by matched unacceptable sentences ( §4). this shows that lms are sensitive enough to sentence acceptability to be able to produce not just systematically grammatical outputs, but also systematically ungrammatical outputs. while this is not a practical application, it does demonstrate how well lms capture this important linguistic feature. furthermore, our perturbation analysis demonstrated that this two-way adaptation was robust to irrelevant syntactic variations in the context ( §5). the present work bolsters the findings of other recent work that only explores this behavior in the grammatical direction @xcite . our finding of models' reliance on abstract structural features that are made available in their context can be further strengthened by controlling for lexical exposure @xcite . that is, fu-ture work can augment our contexts by replacing real lexical items-especially content words-with nonsense words (e.g., wug, dax, etc.), following recent works @xcite . doing so would maintain the structural features of the context while also more strictly controlling for superficial cues such as lexical overlap or similarity, and would make our conclusions stronger. ## conclusion in this work, we study how robust the acceptability judgements of autoregressive transformer language models are to manipulations of the context. we find that acceptability judgements are generally robust when the test sentences are preceded by randomly sampled linguistic contexts. however, when the contexts contain syntactic structures closely matching those in the test sentence, that can significantly improve or degrade the models' performance. this effect is amplified as we lengthen the context provided to the model. our results demonstrate in-context learning in a highly specific way: models are sensitive to granular syntactic properties of the context when making predictions over a target sentence, such that they can be driven to produce both correct and reliably incorrect outputs.