C ausal L ink: An Interactive Evaluation Framework for Causal Reasoning (2025)

Abstract: We present CausalLink, an innovative evaluation framework that interactively assesses thecausal reasoning skill to identify the correct intervention in conversational language models. Each CausalLink test case creates a hypothetical environment in which the language models are instructed to apply interventions to entities whose interactions follow predefined causal relations generated from controllable causal graphs. Our evaluation framework isolates causal capabilities from the confounding effects of world knowledge and semantic cues. We evaluate a series of LLMs in a scenario featuring movements of geometric shapes and discover that models start to exhibit reliable reasoning on two or three variables at the 14-billion-parameter scale. However, the performance of state-of-the-art models such as GPT4o degrades below random chance as the number of variables increases. We identify and analyze several key failure modes.

## introduction evaluating the causal reasoning abilities of generative ai has become a popular research area, especially given the advancements of modern llms @xcite @xcite . however, there are at least two major challenges with regards to the effectiveness of causal reasoning benchmarks: 1) clearly defining the targeted abilities and 2) disentangling reasoning processes from confounding factors such as data contamination and shortcuts. in this study, we tackle these two challenges and propose a dynamic evaluation framework in which the models are instructed to discover causal rules by interacting with hypothetical entities. although human causal reasoning has been systematically studied in various domains including computer science, psychology, and cognitive science @xcite @xcite , we still observe blurred lines among different facets of causal reasoning in the current ai literature, where the term "causal capabilities" may refer to a range of abilities from retrieving commonsense knowledge @xcite @xcite to multi-step structural inference @xcite . we bifurcate causal reasoning skills into two general categories based on whether or not the reasoning process depends on existing world knowledge of causal facts (e.g., given a known causal relationship between x and y such that x causally impacts y ). fact-dependent abilities include effect retrieval, cause retrieval, and mechanism explanation. while all types of fact-dependent abilities fit under the general framework of causal reasoning, none of them requires a genuine understanding of causality. in other words, applying knowledgebased causal abilities requires no higher level of sophistication than superficial knowledge retrieval. in contrast, fact-independent reasoning abilities represent the foundational mechanisms of causal reasoning, which do not rely on exhaustive knowledge of causal facts. these abilities enable humans to derive new causal insights, design experiments, and build the body of common knowledge. humans develop foundational causal reasoning skills, such as reasoning about immediate context and actions, in the early stages of cognitive development prior to language acquisition @xcite . as a result, llms may lack causal reasoning skills parallel to early-stage human causal reasoning through interactions, which serves as a foundation for more advanced reasoning processes @xcite . in this paper, we define a causal capability named action identification, which entails identifying the correct intervention, observing the effects of its action, and reasoning about whether a causal relationship exists. the key contributions of our work are threefold: ## related work current causal benchmarks usually evaluate models with cases developed from real-world causal scenarios @xcite @xcite @xcite . while real-world causal cases are undoubtedly effective in testing fact-dependent causal skills @xcite , researchers must carefully mitigate the potential bias that causal claims can be made with knowledge recall rather than actual reasoning @xcite . previous mitigation included reversing the direction of causality to make causal relations counterfactual @xcite , using nonsensical descriptors to eliminate semantic cues @xcite , and questioning the model from multiple perspectives differentiating the directions of causality and the presence of interventions @xcite . our solution is to construct a hypothetical world from scratch with underlying causal rules, which offers several distinctive advantages compared to previous work. first, the causal relations in the benchmark are systematically generated from causal graphs, allowing precise control over the difficulty of the task. second, the causal entities are customizable and can be designed to be entirely free of linguistic or contextual clues that could otherwise create semantic shortcuts in reasoning. finally, the interactive nature of the benchmark allows for the analysis of model strategies. ## framework description causallink evaluates causal reasoning in language models through interactive simulations grounded in formal causal graphs. the system comprises three integrated components: (1) a configurable causal graph generator that encodes ground-truth relationships, (2) a dynamic simulation environment where variables map to interactive entities, and (3) a language model interface that tests causal understanding through multi-step interventions. each component is formalized as follows. ## conclusion in this paper, we introduced causallink, a novel interactive evaluation framework that rigorously assesses a fact-independent causal reasoning skill that we term "action identification" in llms. by constructing a controlled, simulated environment with predefined causal relationships, we effectively isolate genuine reasoning from the confounding influences of world knowledge and semantic cues. this approach not only enables precise measurement of causal reasoning abilities but also offers a generalizable methodology for a wide range of experimental designs. our empirical evaluations reveal that, although larger models demonstrate foundational causal reasoning skills, their performance becomes increasingly fragile as the complexity of causal interactions grows. importantly, we identify recurring cognitive biases-including single cause bias, correlation bias, and illusive confounder bias. these underscore a critical gap: models misapply their causal knowledge rather than lack it outright. these discrepancies between the models' articulated reasoning and their actual behaviour highlight the limitations of current approaches in achieving robust, context-independent causal reasoning. by establishing a new benchmark for causal in-ference, our study underscores the need for improved methodologies that enhance both the reliability and generalizability of causal reasoning in ai systems. future directions include mitigating model biases and extending the framework to evaluate more aspects of causal reasoning.