C ausal E val: Towards Better Causal Reasoning in Language Models (2025)

Abstract: Causal reasoning (CR) is a crucial aspect of intelligence, essential for problem-solving, decision-making, and understanding the world. While language models (LMs) can generate rationales for their outputs, their ability to reliably perform causal reasoning remains uncertain, often falling short in tasks requiring a deep understanding of causality. In this paper, we introduce CausalEval, a comprehensive review of research aimed at enhancing LMs for causal reasoning, coupled with an empirical evaluation of current models and methods. We categorize existing methods based on the role of LMs: either as reasoning engines or as helpers providing knowledge or data to traditional CR methods, followed by a detailed discussion of methodologies in each category. We then assess the performance of current LMs and various enhancement methods on a range of causal reasoning tasks, providing key findings and in-depth analysis. Finally, we present insights from current studies and highlight promising directions for future research. We aim for this work to serve as a comprehensive resource, fostering further advancements in causal reasoning with LMs.

## introduction large language models (llms) @xcite have achieved significant success across various domains. causal reasoning (cr), which includes causal discovery and inference, is a fundamental aspect of human intelligence, facilitating effective problem-solving, decision-making @xcite , and understanding of the world @xcite , with applications in healthcare @xcite , economics @xcite , biology @xcite , figure 1 : large language models for causal reasoning: serving as reasoning engines or providing support to traditional methods in various end-tasks. enhancing the causal reasoning capabilities of llms differs from improving their general reasoning skills. cr focuses on understanding causal relationships, while general reasoning primarily relies on identifying correlations @xcite . unlike general reasoning, which often involves simpler, more static relationships, cr requires deeper analysis and managing uncertainty to determine how events are causally linked @xcite . additionally, cr must account for confounding variables @xcite , where an observed correlation between two variables is due to the influence of a third variable. counterfactual reasoning @xcite , often necessary for causality verification, is also challenging, as it involves reasoning about hypothetical scenarios that did not actually occur. existing research on enhancing the cr capabilities of llms can be broadly categorized based on their role: either as the engine directly performing llms for causal reasoning preliminaries ( §2) methodology ( §3) llms as reasoning engines ( §3.1) fine-tuning ( §3.1.1) tasks & benchmarks ( §4.1) causal discovery @xcite @xcite @xcite ) @xcite @xcite @xcite , causal inference @xcite @xcite @xcite , additional causal tasks @xcite @xcite ) results ( §4.2) and analysis ( §4.3) figure 2 : structure overview. we categorize the role of llms in causal reasoning into two main functions: as reasoning engine and as helper. each function is further divided into specific methodologies. we also outline the evaluation process, including tasks, benchmarks, results and analysis. in this work, we focus on methodologies to enhance the causal reasoning capabilities of llms, exploring practical challenges and identifying potential opportunities. we introduce a fine-grained taxonomy ( §3) that systematically reviews and summarizes current techniques employed in llms to improve cr. to evaluate these capabilities, we conduct extensive assessments ( §4) of widelyused llms on existing benchmarks, providing key findings and detailed analysis. additionally, we discuss the limitations of current approaches ( §5) and propose promising directions for future research ( §6). previous work @xcite reviews causal reasoning in llms, focusing more on the perspective of traditional causal reasoning. in contrast, our focus lies in the methodological advancements that enhance the causal reasoning abilities of llms. more importantly, we conduct practical evaluations of llms on various levels equation query association p (y|x) what is? intervention p (y|do(x), z) what if? counterfactual p (yx|x ′ , y ′ ) why? ## preliminaries cr in the context of machine learning (ml) involves understanding and modeling cause-andeffect relationships within data, which goes beyond the traditional ml focus on correlations. traditional ml methods often identify patterns or correlations, limiting their applicability in understanding the mechanisms behind complex systems @xcite . causal ml, on the other hand, addresses this gap by focusing on cause-and-effect relationships, which is crucial for making informed decisions, predicting the effects of actions, and understanding complex systems @xcite @xcite . the cr framework can be categorized into three levels: association, intervention, and counterfactual @xcite , which form "the ladder of causality" ( ## towards causal reasoning in large language models we separate the roles of language models in cr into two categories. first, llms can serve as causal reasoning engines, employing methods such as fine-tuning, prompt engineering, external tool integration, and alternative approaches to directly draw causal inferences from input prompts. second, llms can act as helpers to traditional methods by extracting causal information and generating causality data to enhance causal analysis across various domains. ## future directions deep understanding of existing theories. integrating causal reasoning capabilities into llms presents unique challenges, especially when requiring adherence to established theories. these theories, such as the potential outcomes framework, graph-based causal inference methods, and structural equation modeling, often rely on foundational assumptions, including the stable unit treatment value assumption, ignorability/unconfoundedness, and positivity @xcite . considering these assumptions necessitates innovative strategies, such as incorporating specific post-training data and implementing filtering mechanisms to ensure the model's alignment with the theories. data efficiency for end-to-end learning. llms have the potential to perform end-to-end causal reasoning, but their effectiveness depends on substantial data to learn and infer causal relationships. a major challenge is the scarcity of high-quality causality data, particularly for counterfactual reasoning, where alternative scenarios are often unavailable or difficult to generate. improving data efficiency is therefore a critical research direction. techniques such as analogical reasoning, imaginative simulation, and inferential methods can maximize the utility of existing data by generating synthetic datasets, filling data gaps, or extrapolating from limited information. these approaches enhance the model's capacity to learn causal relationships with constrained training data. ## conclusion llms equipped with these advanced technologies demonstrate strong potential in causal reasoning, but further research is needed to fully realize and refine their capabilities. addressing challenges such as managing intricate causal structures is essential for continued development. enhancing interpretability and transparency through causal reasoning is crucial for building trust and helping users rely on model inferences. exploring innovative methods will be key to overcoming current limitations and unlocking the full potential of llms in complex, real-world scenarios. advancing these areas will make llms more reliable and effective tools for causal analysis across domains.