In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax (2024)

Abstract: In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates. Do models guided via ICL infer the underlying structure of the task defined by the context, or do they rely on superficial heuristics that only generalize to identically distributed examples? We address this question using transformations tasks and an NLI task that assess sensitivity to syntax—a requirement for robust language understanding. We further investigate whether out-of-distribution generalization can be improved via chain-of-thought prompting, where the model is provided with a sequence of intermediate computation steps that illustrate how the task ought to be performed. In experiments with models from the GPT, PaLM, and Llama 2 families, we find large variance across LMs. The variance is explained more by the composition of the pre-training corpus and supervision methods than by model size; in particular, models pre-trained on code generalize better, and benefit more from chain-of-thought prompting.

## introduction language models (lms) have become increasingly important subjects of study due to their expressive power and performance at scale. when training large language models (llms) on massive amounts of text, surprisingly sophisticated linguistic behaviors, such as in-context learning (icl), emerge @xcite : given only a small number of labeled training examples in the input context, llms can generalize to new instances of the task without weight updates. thus, even without access to the model's weights, we we prompt language models with labeled examples px, yq train that can be explained using either robust syntactic/hierarchical features or spurious positional/linear features. we also include the input from a test example x test . we ensure the models have learned the task by evaluating on in-distribution examples px, yq test-id . then, we observe whether models generalize syntactically or linearly on out-of-distribution examples px, yq test-ood . we investigate these questions using the test case of syntactic generalization. accurate syntactic representations are necessary for robust language understanding. in lms, syntax acquisition is causally associated with significant and abrupt drops in loss and improved performance on nlp tasks @xcite . language is structured hierarchically, but the structure of sentences is not provided to models as part of their input. therefore, a model could incorrectly assume that sentences have lin-ear rather than hierarchical structure-e.g., that syntactic dependencies consist of adjacent wordsand thus give an incorrect or unrelated answer to questions such as the following: is it true that children who don't like sweets are rare? the main subject is "children", but a model that assumes sentences are structured linearly could instead analyze "sweets" as the subject, as it is the positionally closest noun to the main verb. in downstream tasks, such assumptions about sentence structure would lead to incorrect responses to inputs that humans find simple @xcite . studies using syntactic transformations tasks @xcite have found that pre-training on large corpora imparts syntactic inductive biases to lms @xcite , and the strength of syntactic preferences increases with model depth @xcite . these studies relied on fine-tuning, where the lm's weights are updated based on examples of the task; we extend these analyses to the icl setup. we investigate llms' generalization in syntactic transformations and natural language inference tasks. in each task, we evaluate in two settings: one where models are provided only with a handful of instances of the task, and another in which they are additionally provided with chain-of-thought (cot) traces @xcite . cot refers to the finding that lms' task performance can be improved if they are instructed to predict the intermediate computations required for the task. we examine whether giving lms access to syntactic reasoning traces and meta-linguistic information increases their reliance on robust features. we evaluate not only the accuracy of the final answer, but also the correctness of the model's reasoning and the faithfulness of the model's final answer to its own reasoning. we find that, while all models perform well on in-distribution examples, even very large lms are prone to relying on surface heuristics. further, chain-of-thought results can be misleading: cot improves in-distribution performance, but often decreases out-of-distribution performance. this underscores the importance of out-of-distribution evaluation. finally, we present evidence that exposure to code during pre-training assists models in overcoming these limitations. foot_0 ## experimental setup we first examine syntactic transformation tasks, where the training and test instances are drawn from distributions that are distinct in controlled ways ( §2.1). we test the syntactic generalization of multiple families of lms ( §2.2) when prompted using in-context learning ( §2.3). ## results all models except gpt-3 and gpt-3.5 (turbo) learn to perform the transformations tasks, as indicated by high in-distribution accuracies. among models that learn the task well, there is large variance with respect to out-of-distribution accuracies. scale does not fully explain performance. gpt-3, gpt-3.5 text-davinci-002 and gpt-3.5 text-davinci-003 are estimated to be the same size, but the gpt-3.5 models generalize better. llama 2 and codellama outperform gpt-3 and (flan-)palm despite being far smaller, and codellama significantly outperforms llama 2 on question formation despite being smaller. what explains differences between models, then? pre-training on code improves ood generalization. on average, scores are higher given any prompt for models pre-trained on code: codellama significantly outperforms llama 2 on question formation (and performs comparably on tense reinflection), while gpt-3.5 code-davinci-002 outperforms all other gpt-3 and gpt-3.5 models. this agrees with and extends the finding of @xcite that the domain of the pre-training corpus significantly affects syntactic generalization. this also agrees with recent findings that code pre-training improves other fundamental linguistic abilities (e.g., entity tracking; @xcite . conversely, rlhf may harm ood generalization. gpt-3.5 text-davinci-003 performs at least as well as other gpt-3.5 models on in-distribution examples, but it generalizes consistently worse than other models which are fine-tuned on human demonstrations (including text-davinci-002 and codellama). thus, rlhf is effective when testing on in-distribution test sets, but it may actively harm a model's ability to reason beyond its given exemplars. this finding is preliminary, however, and should be confirmed in future work via ablations over rlhf in a variety ## syntactic generalization in text classification: the case of natural language inference so far, our analyses indicate that llms leverage syntactic information to varying extents when taught a task via icl, and that code pre-training im-proves ood generalization. in this section, we test whether these trends extend to classification tasks. we leverage the heuristic analysis for nli systems (hans; mccoy et al., 2019) dataset, which contains natural language inference (nli) examples designed to diagnose reliance on syntactic heuristics. one typically trains the model on the training set of another nli dataset, such as mnli @xcite or rte @xcite , and then uses hans as an evaluation set to measure whether models rely on syntactic heuristics. @xcite evaluate roberta and gpt-3 on hans after training on mnli; they find that both models perform similarly, suggesting that scale may not be enough to overcome syntactic heuristics. we extend this analysis to a wider variety of llms and to chain-of-thought prompting. hans diagnoses reliance on lexical overlap, subsequence, or constituent heuristics. models that systematically rely on any of these heuristics are expected to obtain 100% accuracy on examples where the label is "entailment" because the heuristics happen to make the correct prediction; however, models that rely on these heuristics will also obtain 0% accuracy on examples where the label is "nonentailment". we average scores across heuristics here; see app. d for full results. we evaluate llms on hans in the icl setting. the 8 icl exemplars are drawn from mnli, and the test examples are from hans. our results are based on a uniform subsample of 100 examples per heuristic and per label. we test across 3 syntactic heuristics, and there are 2 labels (entailment and non-entailment); thus, we have 600 test examples in total. we use no cot prompts and verbal cot prompts similar to those we depict in figure 2 ; see app. d for examples of our prompt formats. our results ( we also observe that chain-of-thought can make models significantly more prone to relying on syntactic heuristics. with verbal cot, scores on entailment examples generally increase (except for gpt-3.5 (turbo)). on non-entailment examples, however, scores reduce to near-randomchance-and, for some models, to near-zero. this pattern suggests that models rely more extensively on the heuristic given verbal cot prompts. this reinforces the importance of detailed evaluation when tuning one's prompts: results from prompt tuning experiments can often be misleading if one only observes in-distribution accuracies, or overall (as opposed to label-specific) accuracies. llms have been found to generate answers which are not faithful to their own chain-of-thought reasoning @xcite . in this section, we test the relationship between models' cot reasoning and their answers. there are at least two possibilities that could lead to low accuracy: first, the models could produce incorrect cot reasoning traces, but answer consistently with those traces. second, they could produce correct reasoning traces, but ignore those traces when producing the output. here, we evaluate the accuracy of models' reasoning, as well as the faithfulness of models' final answers to their reasoning traces. ## discussion llms guided via icl often explain the structure of input examples and tasks using spurious positional and word-level features ( §3), as well as spurious syntactic heuristics ( §4). it is surprising that llms generalize in a manner not consistent with english grammar, given that language models are directly optimized over a large quantity of long contexts to produce probable sequences. larger and deeper models generally behave in a manner more consistent with syntactic structure when fine-tuned @xcite , but this trend does not generalize to icl. thus, despite impressive performance on downstream tasks, our findings provide evidence that llms do not consistently leverage the latent structure of language when processing or generating language. llms will therefore struggle to generalize well outside of their exemplars' distribution. chain-of-thought has significantly different impacts on id vs. ood examples. in transformations tasks, it sometimes improved id performance while maintaining or decreasing ood performance; on hans, it significantly increased reliance on syntactic heuristics. this reveals an actionable takeaway: one should perform thorough evaluations when prompt tuning. results from overall (rather than label-specific) accuracies and indistribution examples can be misleading. a caveat is that one could always tune these prompts further to obtain better performance, just as one could always tune hyperparameters to obtain better models. code pre-training improves llm generalization. various studies observe that lms trained on large amounts of code perform better on various nlp tasks and linguistic evaluations @xcite @xcite . why is code such an effective signal when mixed with natural language? some have speculated that code provides additional grounding @xcite . perhaps more importantly, code contains frequent instances of long-range state tracking, as well as hierarchically structured classes and function stacks; these may impart inductive biases that are helpful for learning hierarchical linguistic structure. it is unclear why gpt-3.5 text-davinci-003the only model in the gpt-3.5 family trained with rlhf-performs significantly worse than comparable gpt-3.5 models. this goes against conventional wisdom that rlhf generally improves performance on downstream nlp tasks @xcite ), 11 but corroborates preliminary evidence that rlhf degrades certain aspects of performance, like certainty calibration. 12 further work is needed to fully understand rlhf's impact on generalization and whether these findings are causally linked. perhaps the reinforcement learning procedure optimizes models to generate language that semantically aligns with human quality judgments, but at the cost of causing the model to assign lower importance to structural features. meanwhile, providing human feedback via fine-tuning may be less optimal for aligning outputs to human judgments, but better for preserving sensitivity to syntax. larger models often perform better on many nlp tasks. however, our findings reveal that scale is not a panacea for robust generalization: rather, other factors like training objectives, the type of pretraining data, and the supervision method(s) make a significant difference. this agrees with and extends findings from @xcite that the domain of the pre-training data significantly influences how models generalize. ## related work since the discovery that lms are capable of icl, studies have explored icl's limits (e.g., @xcite . analyses of icl have revealed counterintuitive biases: for example, correct labels are not necessary for strong performance @xcite , and models can perform very well even given misleading/irrelevant prompts @xcite or flipped/semantically misleading label names (wei et al., 2023; pan et al., 11 openai also officially states that text-davinci-003 is more capable than text-davinci-002 in multiple locations on their website, including here and here. any finite training set is consistent with multiple generalizations. crucially, most benchmarking tasks rely on data where the training and test sets are drawn from the same distribution, which limits our understanding of how well models truly generalize (linzen, 2020). @xcite find that llms are prone to relying on spurious correlations that hinder robust generalization, while @xcite find that a series of chainof-thought prompts can yield more robust generalization. with respect to syntax, lstm-and transformer-based encoder-decoder models trained from scratch on syntactic transformations do not generalize in a syntax-sensitive manner @xcite @xcite , but pretrained encoder-decoder models do generalize syntactically @xcite . similar positive results have been reported for roberta, but only after large-scale pre-training @xcite . @xcite prompt llms on metalinguistic judgments; they find that prompting underestimates syntactic awareness compared to probability measurements. our findings extend this conclusion: this issue is more pronounced when test examples are not identically structured to the exemplars. when fine-tuning, larger models generally have inductive biases that align more strongly with syntactic structure @xcite , so it is plausible that greater scale should lead to more syntax-sensitive behavior when using icl. nonetheless, this is not what we find. ## conclusions we have investigated how well llms generalize out-of-distribution on tasks requiring syntax for robust performance. our findings reveal significant variance across models that is not fully explained by scale. models trained on code are better at leveraging in-context examples to generalize more robustly, and at reasoning accurately and faithfullyeven at smaller scales.