LLM s Are Prone to Fallacies in Causal Inference (2024)

Abstract: Recent work shows that causal facts can be effectively extracted from LLMs through prompting, facilitating the creation of causal graphs for causal inference tasks. However, it is unclear if this success is limited to explicitly-mentioned causal facts in the pretraining data which the model can memorize. Thus, this work investigates: Can LLMs infer causal relations from other relational data in text? To disentangle the role of memorized causal facts vs inferred causal relations, we finetune LLMs on synthetic data containing temporal, spatial and counterfactual relations, and measure whether the LLM can then infer causal relations. We find that: (a) LLMs are susceptible to inferring causal relations from the order of two entity mentions in text (e.g. X mentioned before Y implies X causes Y); (b) if the order is randomized, LLMs still suffer from the post hoc fallacy, i.e. X occurs before Y (temporal relation) implies X causes Y. We also find that while LLMs can correctly deduce the absence of causal relations from temporal and spatial relations, they have difficulty inferring causal relations from counterfactuals, questioning their understanding of causality.

## introduction causal reasoning is crucial for intelligence as it allows us to construct a world model and make predictions robustly based on cause-effect relations. recent work @xcite has shown that gpt-4 outperforms existing methods on various causal inference and causal discovery tasks. but it is unclear how much of this success can be attributed to llms memorizing explicitlymentioned causal facts in their training data (e.g. reading 'smoking causes cancer' from wikipedia), versus inferring unseen causal relations (e.g. from experiment results in medical journals). to disentangle memorized vs inferred causal relations, one straightforward method is to filter out causal facts the model has seen during pretraining in the test set. however, it is computationally expensive to extract causal relations at the scale of current pretraining data. therefore, we continue pretraining existing llms on synthetic data containing observations of fictional events, and evaluate if llms can infer the underlying causal relations that produce the data. we focus on the setting of finetuning i.e. out-of-context inference @xcite , rather than causal inference in-context since it is closer to how one would use the llm e.g. train on large corpora of medical journals and then use the llm for causal discovery. to generate the synthetic data for causal inference, we focus on event relations that are commonly seen in the pretraining data, and from which humans can easily deduce causal relations. figure 1 shows the relations and the deductions we can draw from them, including: (1) temporal relations ('smoking happens before lung cancer'), which imply negative causal relations ('lung cancer cannot cause smoking') according to temporal precedence @xcite @xcite ; (2) spatial relations ('there was a storm in california and flash flooding in new york'), which implies the absence of causal relations ('californian storm did not cause the flash flooding' and vice versa) according to the principle of locality @xcite ); foot_0 (3) counterfactuals ('it rained today and the sidewalk was wet. if it had not rained, the sidewalk would not have been wet.'), which imply causal relations ('today's rain caused the sidewalk to be wet'; @xcite our experiments are conducted on llama2 @xcite and the main results are summarized in figure 1 . 3 when trained on temporal relations, we find that models learn a position heuristic: if event x is always mentioned before event y in the text, then llms infer that x causes y based on the relative position of the event mentions regardless of their temporal order, e.g. it infers the same causal relation from 'x preceded y ' (temporal(x, y)) and 'x followed y ' (temporal(y, x)). to overcome the position heuristic, we augment the finetuning data by adding paraphrases for all relations to randomize the order of event mentions, e.g. for temporal(x, y), we include both 'x preceded y ' and 'y followed x'. we find that even augmenting 10% of the dataset is enough to reduce model's reliance on the position heuristic. interestingly, it reveals another failure mode: llms then suffer from the post hoc fallacy @xcite , which infers positive causal relations from temporal relations, i.e. temporal(x, y) implies x causes y . additionally, we find that while llms are able to deduce the absence of causal relations from temporal and spatial relations, they struggle to infer the presence of causal relations from counterfactuals, and scaling to larger models does not improve the result. overall, our results suggest that llms may not infer much novel causal knowledge beyond explicitly mentioned facts in the pretraining data. recent work has also highlighted other challenges for current llms in causal inference @xcite introduced the task of deducing causal relations from correlations; @xcite created a dataset for causal inference in natural language which includes multiple sub-skills such as formalizing queries, deriving the estimand etc.; @xcite designed a challenging benchmark which involves counterfactual presuppositions; see @xcite for a comprehensive survey of capabilities and limitations of current llms in causal inference. in contrast, we focus on commonsense causal inference from relations which llms would have seen in pretraining data, similar to how humans perform causal reasoning intuitively. ## experiment design our main goal is to measure whether llms can infer causal relations given observations in the text. specifically, we assess whether llms can predict causal relations between two events after being trained on textual descriptions of their temporal relations, spatial relations, and counterfactuals. to avoid the cost of pretraining language models from scratch, we continue pretraining (finetune) off-theshelf llms following @xcite . we hypothesize that if llms have learned meaningful deduction rules from pretraining (e.g. temporal precedence), they should be able to apply them during finetuning to infer causal relations. we focus on finetuning rather than causal inference in-context, since it is closer to how one would use a llm for causal discovery e.g. after training on large corpora of medical journals, rather than directly prompting with observations between events. the overall pipeline to test if llms can infer causal relations is: (1) generate synthetic data that contains descriptions of event relations grounded in a causal graph (section 3.1); (2) finetune the llm on the generated data (section 4); (3) evaluate the llm on causal relation prediction tasks for each pair of events mentioned in the finetuning data (section 3.2). we describe our data generation and evaluation methods below. ## experimental details notation. before explaining the experimental setup, we introduce some notation that will simplify our description. given events x and y , we use (x, y ) to denote the relative position where x is mentioned before y , e.g. 'x causes y ' or 'x preceded y '. we use t (r, π) to denote the set of all templates for a relation r between x and y with relative position π where π is (x, y ), (y, x), or a random mix of both, (x, y ) + (y, x). ## position heuristic in this section, we first demonstrate that llms are susceptible to inferring causal relations by the relative position of two entity mentions in text (section 5.1). we hypothesize that models learn this heuristic since it is supported in the pretraining data (appendix a.4) and investigate ways to fix this heuristic via either augmentation or scaling up models (section 5.2). ## inferring causal relations under no position heuristic the previous section demonstrated that if the data supports the position heuristic, models fail to infer any causal relations and only rely on the relative position between events to infer causal relations. however, it is easy to mitigate the position heuristic by randomizing the relative positions of event mentions in the data. in this section, we evaluate ## llms suffer from post hoc fallacy section 6 demonstrated that when the data does not support the position heuristic, llms can correctly infer the absence of causal relations from temporal and spatial relations. in this section, we demonstrate that for temporal relations, models in fact overgeneralize to infer the presence of causal relations in the other direction. this mistake is often referred to as the post hoc fallacy @xcite , which uses the incorrect deduction rule: temporal(x, y ) =⇒ @xmath0. humans have known to often fall prey to this fallacy and infer causal relations from sequential order @xcite . to demonstrate this, we finetune models from the llama2 family (7b to 70b) and llama2-7b on d temporal (where the templates have randomized order) and evaluate them on d @xmath1. all templates in the evaluation use randomized event order t (r, (x, y ) + (y, x)) for each relation r in the multiple-choice options. for evaluation, we report the error rate which is the percentage of examples where the model incorrectly deduces @xmath2, y ). figure 5 (left) shows the error rate. we observe that all models incorrectly infer the causal relation better than random guessing (33.3%). 12 interestingly, we observe an inverse scaling trend @xcite -scaling model size increases the error and models rely on the post hoc fallacy more. ## conclusion in this work, we investigate whether llms can be useful for causal inference beyond explicitlymemorized causal facts. we find that llms are susceptible to inferring causal relations from position, but this can be mitigated by data augmentation. we find that llms can infer causal relations from temporal relations and spatial relations, but not from counterfactuals. overall, we find that llms may not infer much novel causal knowledge beyond explicitly mentioned facts in the pretraining data. our setup also allows for the exploration of interesting questions such as whether models generalize to events of the same 'type' (e.g. if smoking and vaping occur in similar contexts, and the data includes smoking causing cancer, does the model generalize to infer any relation between vaping and cancer?), and if models can generalize to transitive relations. we leave these questions for future work.