Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024)

Abstract: The impressive performance of recent language models across a wide range of tasks suggests that they possess a degree of abstract reasoning skills. Are these skills general and transferable, or specialized to specific tasks seen during pretraining? To disentangle these effects, we propose an evaluation framework based on “counterfactual” task variants that deviate from the default assumptions underlying standard tasks. Across a suite of 11 tasks, we observe nontrivial performance on the counterfactual variants, but nevertheless find that performance substantially and consistently degrades compared to the default conditions. This suggests that while current LMs may possess abstract task-solving skills to an extent, they often also rely on narrow, non-transferable procedures for task-solving. These results motivate a more careful interpretation of language model performance that teases apart these aspects.

## introduction the striking empirical successes of language models (lms) suggest that next-word prediction at scale may be a viable approach for distilling the knowledge embedded in large-scale text corpora into general-purpose interactive agents. lms obtain impressive results on various nlp benchmarks @xcite @xcite and display surprising abilities that suggest a nontrivial understanding of the world @xcite , even purportedly surpassing human performance on tasks that require nontrivial reasoning @xcite @xcite . ideally, we expect a general-purpose lm to be able to generalize not only to unseen instances of known tasks, but to new tasks. humans, for example, can transfer their knowledge to new instances and also flexibly adapt to novel tasks @xcite . while much past work has focused on instance-level generalization, we ask: to what extent does the performance of current lms derive from their ability to deploy task-general reasoning skills @xcite , versus their ability to recognize and recall specific tasks seen frequently in pre-training? we propose to measure such task-level generalizability by taking tasks on which lms perform well, and altering the conditions under which these tasks are performed. the general reasoning procedure for these variants remains the same, but the inputoutput mapping functions differ. we call the new tasks counterfactual tasks, as they deviate from the default, generally assumed conditions for these tasks. for example, while arithmetic is by default performed in base-10, we consider a counterfactual task requiring additions in base-9. if models implement a general and transferable task-solving procedure, we expect comparable performance on counterfactual and default tasks; if they employ procedures tailored to default task conditions, we expect a drop in the counterfactual performance. we design a suite of 11 counterfactual evaluation tasks, illustrated in figure 1 (appendix), to measure an lm's flexibility to adapt to new task variants across multiple categories and domains. in each, the original task under the default conditions and its counterfactual variants share the same reasoning procedure. we consider traditional tasks such as deductive reasoning, non-language tasks that are nonetheless commonly evaluated such as code gen-eration, as well as non-standard tasks such as drawing and spatial reasoning. the latter extralinguistic tasks test whether lms learn conceptual structures that mirror the structure of the non-linguistic world, which has been suggested by recent work @xcite @xcite . we evaluate gpt-4 (openai, 2023), gpt-3.5, claude (anthropic, 2023), and palm-2 @xcite on tasks under both the default and counterfactual conditions. we observe above-random counterfactual performance for most tasks, indicating some degree of task generalizability. however, performance on counterfactual task variants consistently and substantially degrades relative to the performance on the default settings. a systematic study of model behavior on default and counterfactual tasks ( §5) reveals that while chainof-thought prompting and few-shot demonstrations generally improve performance, they do not close the default-counterfactual gap. we further find that counterfactual performance is affected by the commonness of the counterfactual conditions, but correlates with the default task performance. these results collectively suggest that lms' abilities on these tasks are supported at least in part by nontransferable, default-condition-specific behaviors rather than generalizable reasoning skills. ## counterfactual tasks we informally conceptualize a task as a function f w : @xmath0. world models encapsulate the conditions for the function evaluation. for example, in arithmetic, w could represent the set of conditions required for an arithmetic operation, such as the number base. we refer to the set of assumed default conditions, including but not limited to the base's being 10, as the default world, or w default . intuitively, for any task, w default corresponds to the set of conditions underlying most task instances in pretraining corpora. traditional machine learning evaluations assess how closely a model's learned hypothesis h estimates f w by independently sampling training and test sets from the population distribution and only exposing the model to the former for learning h. however, in datasets of web text, these evaluations are subject to potential data contamination issues @xcite @xcite . these issues may be more severe in recent lms: the ever-growing pretraining datasets potentially expose the models to more evaluation instances, and recent lms with increasing sizes are more capable of memorizing these instances. we hence consider another dimension of generalization: to new task variants in counterfactual worlds w cf , instead of new inputs x. this allows us to measure the extent to which a model's f w default performance is specific to w default or attributable to a general implementation of the task f . for arithmetic, w cf could be one that was the same as w default but assumed a base other than base-10. we expect a model with general arithmetic ability to perform similarly in other bases. we stress that our goal is not to find counterfactual world models outside human experience. base-9 addition, for example, is not a novel concept. nor do we aim that the counterfactual world models are unobserved in pretraining. instead, counterfactuals are simply variations on the default conditions for a task. we assess an lm's task performance with 0-shot prompting. we specify the task f , the test instance x, and the world model w in a prompt, and parse the lm's output. we denote the lm's implementation of f w for a given instance x to be, h(f, w, x) = arg max ## analysis we investigate how a variety of factors affect the performance trends in §4. unless otherwise specified, we only consider gpt-4 with 0-shot cot as this is generally the best-performing setup. we provide additional analyses in §b. ## conclusion through our counterfactual evaluation on 11 tasks, we identified consistent and substantial degradation of lm performance under counterfactual conditions. we attribute this gap to overfitting to the default task variants, and thus encourage future lm analyses to explicitly consider abstract task ability as detached from observed task performance. furthermore, insofar as this degradation is a result of the lms' being trained only on surface form text, it would also be interesting future work to see if more grounded lms (grounded in the "real" world, or some semantic representation, etc.) are more robust to task variations.