Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios (2023)

Abstract: Current pre-trained language models have enabled remarkable improvements in downstream tasks, but it remains difficult to distinguish effects of statistical correlation from more systematic logical reasoning grounded on the understanding of real world. We tease these factors apart by leveraging counterfactual conditionals, which force language models to predict unusual consequences based on hypothetical propositions. We introduce a set of tests from psycholinguistic experiments, as well as larger-scale controlled datasets, to probe counterfactual predictions from five pre-trained language models. We find that models are consistently able to override real-world knowledge in counterfactual scenarios, and that this effect is more robust in case of stronger baseline world knowledge—however, we also find that for most models this effect appears largely to be driven by simple lexical cues. When we mitigate effects of both world knowledge and lexical cues to test knowledge of linguistic nuances of counterfactuals, we find that only GPT-3 shows sensitivity to these nuances, though this sensitivity is also non-trivially impacted by lexical associative factors.

## introduction reasoning plays a central role in human communication @xcite . while language models have demonstrated remarkable capacity on downstream tasks @xcite @xcite , it remains unclear to what extent predictions generated by language models are consequences of correlation with linguistic heuristics in the context, versus robust reasoning about causal relations grounded on understanding of world knowledge. in this paper we leverage counterfactual conditionals to investigate the capacity of pre-trained lms (plms) to distinguish hypothetical scenarios from reality, and to examine how this interacts with models' use of existing real world knowledge as well as shallower associative cues. counterfactuals consist of a premise which is false in the real world but true in the hypothetical world (e.g., "if cats were vegetarians"), and an imaginary consequence of this premise ("cats would love cabbages"). testing language models with counterfactuals allows us to use language to manipulate what is true and what is hypothetical, and to test models' ability to separate and use this information for predictions. previous work has established the use of counterfactual scenarios to probe inference ability @xcite @xcite @xcite @xcite @xcite , but the datasets lack systematic control of lexical cues and world knowledge, which makes it likely that the performance could be attributable to spurious cues in the datasets @xcite . for our tests we draw on and adapt inputs from existing psycholinguistic experiments. we begin by testing models' ability to override existing world knowledge when the context indicates that the correct completion involves a hypothetical world (e.g., "if cats were vegetarian, cats would love cabbages/fish"). we test five popular plms, and find that models can increase their preference for counterfactual completions given counterfactual context-however, most models rely strongly on simple lexical cues. next we control the effect of real world knowledge and lexical triggers, to test models' understanding of what counterfactual language implies about the world state. we find that most models fail to understand real-world implications of counterfactuals and largely rely on lexical triggers-with the exception of gpt-3, which shows greater sophistication, but continues to show non-trivial susceptibility to interfer-ence from lexical-associative cues. we discuss the implications and possible interpretations of these findings with respect to linguistic competence and predictive strategies of these models. ## exp1: overriding world knowledge our first experiment investigates whether lms are able to take a counterfactual scenario and predict a counterfactual-consistent completion that contradicts general world knowledge. items we draw directly on counterfactual stimuli from the psycholinguistic study of @xcite . there are 128 items from the original psycholinguistic experiments, and we synthetically generate 10,720 additional items (see appendix a.2 for illustration of data generation process). we match target nouns and syntactic constructions across conditions in order to control lexical properties that influence language models' predictions. ## exp2: impact of cue words in context the first experiment suggests that models can to an extent override world knowledge given a counterfactual, particularly in cases when models have a strong handle on the relevant world knowledge. however, it is possible that in these tests the models were not relying on sophisticated understanding of counterfactuals, but rather on simple lexical triggers in context. consider, for instance, that models could perform well in exp1 if they simply increase their preference for "cabbages" in the proximity of "vegetarians", etc. to test the impact of these lexical triggers, we incorporate an additional condition. items ## exp3: inferring real world state with counterfactual cues the previous experiments indicate that models can override world knowledge in the face of counterfactual evidence, and that the ability to do this improves with stronger world knowledge-but for most models this performance appears to be driven largely by simple lexical triggers in the context, with the possible exception of gpt-3. in this section we remove the influence of pre-existing world knowledge, and hold constant lexical triggers across conditions, for a more direct test of models' sensitivity to linguistic indicators of counterfactuals, and what they say about the true state of the world. this task is particularly challenging because language models must infer the true state of the world based on the presence of counterfactuals, with lexical cues often being misleading. items we adapt stimuli from a psycholinguistic study with 96 controlled sentences @xcite . we additionally create a larger-scale synthetic dataset with 12,960 sentences, using the same events as the generated dataset from section 2. we modify the subject noun phrases such that there is no influence of existing world knowledge. for example, we modify the subject "cat" to "pet", so that there is no prior knowledge about the subject's preference for "cabbages" or "fish". as a result, existing world knowledge cannot inform the correct completion-instead, models need to infer based on the counterfactual language that the true state of the world is different from what the counterfactual states. further, we control the lexical items used across different conditions to minimize effects of lexical cues on condition differences (see ## conclusion the experiments above have shown that when presented with counterfactual situations, plms are able to prefer completions that conflict with world knowledge-and counterintuitively, this sensitivity appears better in cases where that world knowledge is stronger. our results also indicate, however, that models are in large part relying on simple lexical cues to inform these preferences. the only model that shows more sophisticated sensitivity to finegrained linguistic cues separating counterfactuals from reality is gpt-3-which successfully distinguishes conditions based on counterfactual cues, but nonetheless still shows strong influences from lexical associative cues. why might world knowledge aid counterfactual sensitivity? does gpt-3 truly understand counterfactuals? one possibility worth considering is that explanations in both of these cases involve volume of exposure. first, models' stronger world knowledge for a given fact suggests that models have encountered that fact more often in training-and this may in turn translate to more exposure to that type of knowledge in counterfactual contexts, enabling more straightforward memorization-based performance. similarly, while gpt-3 may robustly understand counterfactuals, the massive data exposure for that model may enable a simpler path to success: gpt-3 could simply have developed lower-level knowledge of how linguistic cues like "if/had" versus "because" mediate levels of association between nearby lexical cues and later words. we leave investigation of these hypotheses for future work.