From Causal Parrots to Causal Prophets? Towards Sound Causal Reasoning with Large Language Models (2025)

Abstract: Causal reasoning is a fundamental property of human and machine intelligence. While large language models (LLMs) excel in many natural language tasks, their ability to infer causal relationships beyond memorized associations is debated. This study systematically evaluates recent LLMs’ causal reasoning across three levels of Pearl’s Ladder of Causation—associational, interventional, and counterfactual—as well as commonsensical, anti-commonsensical, and nonsensical causal structures using the CLadder dataset. We further explore the effectiveness of prompting techniques, including chain of thought (CoT), self-consistency (SC), and causal chain of thought (CausalCoT), in enhancing causal reasoning, and propose two new techniques causal tree of thoughts (CausalToT) and causal program of thoughts (CausalPoT). While larger models tend to outperform smaller ones and are generally more robust against perturbations, our results indicate that all tested LLMs still have difficulties, especially with counterfactual reasoning. However, our CausalToT and CausalPoT significantly improve performance over existing prompting techniques, suggesting that hybrid approaches combining LLMs with formal reasoning frameworks can mitigate these limitations. Our findings contribute to understanding LLMs’ reasoning capacities and outline promising strategies for improving their ability to reason causally as humans would. We release our code and data.

## introduction causal reasoning, the ability to infer cause-andeffect relationships, is a fundamental property of intelligence @xcite in humans and machines alike. while llms have achieved significant progress in natural language processing @xcite , their ability to perform genuine causal reasoning is debated. existing studies indicate that models perform poorly when facing complex causal structures @xcite , engaging in counterfactual reasoning @xcite , or applying formal causal reasoning when commonsense rules do not apply @xcite . some findings suggest that llms act more like "causal parrots", simply reciting causal knowledge from their training data rather than engaging in true causal inference @xcite . understanding and improving llms' causal reasoning capabilities remains critical for ensuring reliable llm-supported decision-making, particularly in high-stakes domains such as healthcare, economics, or public policy. this study aims to bridge the gap by systematically evaluating llms on causal reasoning tasks, addressing the following research questions: we evaluate zero-shot chain of thought @xcite , causal chain of thought @xcite , and chain of thought with self-consistency @xcite . finally, we introduce new causal variants of tree of thoughts @xcite and program of thoughts @xcite with an integration of the dowhy causal inference library @xcite ) and demonstrate how these can elevate causal reasoning performance. our main contributions include (1) a comprehensive evaluation of recent llms' causal reasoning abilities, updating previous work, (2) an assessment of causal reasoning improvements coming through prompting techniques, and (3) two new prompting techniques causaltot and causalpot to enhance llms' causal reasoning over previous baselines, borrowing ideas from formal causal reasoning techniques accessible to humans. ## conclusion connecting to previous work on causal reasoning in llms, we have presented a systematic evaluation of causal reasoning abilities of the most recent llms. our findings indicate that the latest models perform substantially better than older llms evaluated in previous works. these state-of-theart llms seem to reason well, even on challenging causal reasoning tasks and unfamiliar causal structures. one exception is counterfactual reasoning, which still poses significant challenges to state-of-the-art llms. while we found that previous prompting techniques designed to improve llms' reasoning performance no longer show the desired improvements on recent llms, we proposed two new causal prompting techniques. as demonstrated, causaltot and causalpot can significantly elevate reasoning performance, even