E xpli C a: Evaluating Explicit Causal Reasoning in Large Language Models (2025)

Abstract: Large Language Models (LLMs) are increasingly used in tasks requiring interpretive and inferential accuracy. In this paper, we introduce ExpliCa, a new dataset for evaluating LLMs in explicit causal reasoning. ExpliCa uniquely integrates both causal and temporal relations presented in different linguistic orders and explicitly expressed by linguistic connectives. The dataset is enriched with crowdsourced human acceptability ratings. We tested LLMs on ExpliCa through prompting and perplexity-based metrics. We assessed seven commercial and open-source LLMs, revealing that even top models struggle to reach 0.80 accuracy. Interestingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events. Finally, perplexity-based scores and prompting performance are differently affected by model size.

## introduction understanding cause-effect relationships is one of the hallmarks of human cognition @xcite . the question of whether large language models (llms) truly comprehend causal relationships in natural language texts, or merely perform as 'stochastic parrots' @xcite by replicating statistical associations in their pretraining data, remains a topic of debate @xcite . this question is crucial for the application of llms in domains that demand interpretive and inferential accuracy. the recent growth in causal research and benchmarking highlights the fundamental need for more reliable and interpretable models @xcite . llms should be able to interpret not only cause-effect relations but also the relations between causal and temporal aspects (see @xcite as an example), which are often connected and overlapping (e.g., typically an effect temporally follows its cause). a common task for evaluating causal reasoning skills is pairwise causal discovery (pcd), which focuses on determining the existence of a causal relation between two events and, if so, establishing which event serves as the cause and which one as the effect @xcite . however, this formulation does not directly take into account the tight bound between the causeeffect and the before-after relationships. in this paper, we present explica, a dataset designed to evaluate llms in commonsense causal reasoning through pcd tasks. our formulation of the pcd task for explica allows us to take into account also the entanglement of causal and temporal relations between events. this is achieved by considering sentence pairs and connective words that overtly express these relationships. in explica, each sentence describes an event (e.g. martina has less chance of getting the flu and martina has been vaccinated), and each connective word explicitly indicates either a causal relationship (i.e., so and because) or a temporal one (i.e., before and after). we collected acceptability ratings for each connective word with respect to each sentence pair, to account for both causal and temporal relations. to our knowledge, explica is the first dataset containing both causal and temporal explicit relations between events annotated by native speakers via crowdsourcing, rather than expert annotators. we conducted a nuanced evaluation of a group of llms on explica. its goal is to shed light on several key aspects. first, we aim to estimate whether and to what extent llms can model and distinguish causal and temporal relations, similarly to humans. second, we want to assess potential differences between llms' competence and performance in our pcd task. recent studies identified a discrepancy between the log-probability models assign to word sequences, which is referred to as their competence since it directly relates to their language modeling training objective, and models' performance, which is given by the accuracy of their responses elicited via prompting, when tasked to provide an answer with instructions in natural language. this line of research suggests that prompting tends to underestimate the underlying linguistic knowledge of the models @xcite . therefore, we compare two metrics computed over connected sentences pairs: accuracy in prompting tasks, and perplexity. the latter is derived from the model's log-probability and is used here as a proxy for the model's acceptability decision, stemming from its underlying linguistic competence. finally, we want to study these aspects across models of varying scales. contributions. our primary contributions are shown in fig. 1 and can be summarized as follows: our approach systematically targets models' competence (via perplexity) and performance (via prompting), measured across different formulations of the task (sec. 4); • we present a comprehensive evaluation of seven llms in total, comprising both commercial and open models (sec. 5). 1 the study of causality and its linguistic expressions garnered renewed and intensified interest, particularly in the context of evaluating the reasoning abilities of llms. recent advancements led to the development of several specialized datasets, aimed at testing the causal reasoning of llms through hypothetical scenarios. some studies evaluate llms on their ability to identify causal relations in structured data @xcite . notable works on unstructured data include cladder @xcite , which assesses causal reasoning using questions based on formal rules; causalbench @xcite , used for tasks related to mathematics, coding, and textual data; and causalnet @xcite , which covers both causal and counterfactual questions. the focus of our work is on the extraction of causal events from unstructured textual data. unlike in explica, the above mentioned datasets focus on implicit causal relations, which are not overtly expressed in linguistic structures. to evaluate llms, several datasets have also been annotated with explicit causal relationships between events in texts. these range from multilingual educational content, as in meci @xcite , or financial news @xcite , to more diverse sources, such as crest @xcite . although these datasets do not explore the temporal dimensions of causality, many causal annotation schemas are derived from datasets that do annotate temporal relationships between events (e.g., because, @xcite or vice-versa (e.g., temporal and causal reasoning, @xcite , including those from news sources (e.g., causal time bank, @xcite , and from short commonsense narratives (e.g., caters, @xcite . however, such datasets do not leverage crowdsourcing annotation by native speakers for both causal and temporal relations as in explica. in our dataset, the ground truth is given by english native speakers' annotation collected via crowdsourcing, with the aim of addressing the complexity of distinguishing truly causal from merely temporal relations between events. furthermore, cladder, causalbench, and causalnet are question-answering datasets that include scenarios involving causally related events. another dataset of this kind is crab @xcite , which contains events taken from real-world situations together with their contextual information (i.e., news articles). similarly, causalprobe-2024 @xcite builds its scenarios from texts published after the release of the evaluated models, ensuring that the information was not available during model training. a key challenge in evaluating llms using datasets with direct textual annotations of causal relations is, in fact, the inherent ambiguity of their expression in natural language. for instance, linguistic markers such as and can signal either causality or temporality, depending on the context. this ambiguity can limit the effectiveness of such datasets in assessing causal reasoning. to overcome this limitation, explica has chosen a more controlled approach to causality evaluation by conducting a pairwise analysis of events, each expressed by a single sentence. the same strategy has been adopted in the copa dataset @xcite , where causality detection is framed as a task where the system must choose the most plausible alternative between two options. similarly, e-care (explainable causal reasoning, @xcite includes over 21, 000 multiple-choice questions focused on causal reasoning, accompanied by conceptual explanations that clarify the underlying causal logic of each question. while these two datasets present instances of implicit causality, the big-bench (beyond the imitation game, @xcite initiative also models explicit causal reasoning. in this framework, the system must select the most plausible causal relationship between a because b and b because a. similarly, in explica pairs of sentences from e-care and big-bench are joined in both directions using temporal and causal connectives. for example, the dataset contains both a precedes/causes b and b precedes/causes a. this allows us to carefully analyze the models' ability to discriminate between the related and yet very different relations of temporal precedence and causality. furthermore, other linguistic cues, such as anaphoric references, have been removed. this design ensures that models are unlikely to rely on surface features, preventing the correct interpretation of causal markers from being inferred simply from the syntactic context. as in @xcite and @xcite , many research works have focused on evaluating llms' causal reasoning by using existing resources. in fact, some of the above datasets have become part of a broader evaluation framework called causal evaluation of language models (calm, @xcite . calm serves as a comprehensive benchmark for assessing llms' causal reasoning capabilities. it comprises 126, 334 data samples and provides a foundational taxonomy of four modules: causal target, adaptation, metric, and error analysis. in relation to causal discovery, this framework addresses issues distinct from those targeted by explica, and it focuses solely on the analysis of llm-generated responses. by contrast, in our work, we evaluated both the model outputs elicited via prompts, and the internal knowledge of llms, assessed through perplexity measurements. an overview of these datasets, along with a comparison with explica, is provided in appendix a. ## the explica dataset explica foot_0 is designed to evaluate llms on commonsense causal reasoning through pcd tasks. it is composed of sentence pairs, where each sentence describes an event. sentence pairs were in part adapted from sentences in existing datasets and in part manually crafted. approximately a third of the sentence pairs were based on sentences from descript @xcite , e-care @xcite , and big-bench @xcite . 3 explica includes 600 english sentence pairs, selected to be equally divided into three subsets: i.) causal subset, where the relationship is most likely causal (and possibly also temporal); ii.) tem-poral subset where the relation is expected to be only temporal; iii.) unrelated subset, including sentences that are thematically related but neither causally nor temporally. 4 sentence pairs are linked through words that explicitly signal either a causal or temporal relationship. these connectives act as linguistic cues, enabling a causal or temporal interpretation of events based on the compositional meaning of the sentences. other than the type of the relation between the events, the connectives specify the order of the events in the relations, which can be iconic -where the effect follows the cause and events are presented in their chronological order -or anti-iconic, where the order is reversed. we join each sentence pair with each connective, also considering the reverse order. in this way, we obtained 4, 800 unique items (600 pairs × 4 connectives × 2 orders). an english native speaker examined a dataset sample to validate whether the connectives correctly express the nature of the relation and the order between the events. we detail the connectives below: then we applied a rigorous procedure to enrich, validate, and refine the dataset. first, we validated the dataset ensuring that the statistical association between words in sentence pairs does not significantly differ across the three subsets (i.e., un-related, temporal, causal). this validation was performed using mutual information (sec. 3.1). hence, we enriched each item with human acceptability ratings collected via crowdsourcing, and we set a threshold to the ratings for unrelated ones creation of the dataset, but the final annotations provided as gold standard are based on the human ratings from the survey. (sec. 3.2). finally, we checked for frequency biases in the dataset by analyzing the triplets {1st sentence verb, connective, 2nd sentence verb} in ententen @xcite (sec. 3.3). ## experiments we evaluated models on explica using human ratings as ground truth across four tasks: three prompting tasks, and one perplexity evaluation. prompting experiments were conducted under various settings, including few-shot (i.e., the prompt includes a few examples illustrating how to perform the task) and zero-shot (i.e., no examples are provided in the prompt) setups, and employing either greedy search foot_8 or the outlines framework (willard and louf, 2023) as decoding strategy for generating answers. greedy search, or greedy decoding, selects the most probable next token at each step, while outlines is a decoding framework that restricts model outputs to a predefined set of valid completions. this approach has been adopted in several recent studies and is now commonly used in the literature to control outputs of generative models @xcite . we used accuracy as evaluation metric. to study the effect of the model parameter scale, we then compared several models of the qwen2.5 family on the acceptability rating task and perplexity. finally, we compared the distribution of model ratings in the acceptability task and perplexity scores with human ratings, assessing their correlation. by analyzing the relationship between model ratings, perplexity, and human acceptability ratings, we aim to determine whetherand to what extent -llms can capture and distinguish causal and temporal relations in ways that align with human understanding. experiments were ran on a single nvidia a100 80gb gpu, for around 120 gpu hours. openai models were queried via the proprietary api for an estimate of 7 gpu hours. prompting evaluation. this aimed to assess llms' generation abilities and analyze how performance varies based on task modeling. specifically, we defined three tasks: i.) acceptability ratings -we adopted the same design used in the survey with human participants (sec. 3.2). items for which the model failed to provide a rating were assigned a score of -1; ii.) cloze test -given a test item consisting of two sentences linked by a connective, we masked the connective and asked the model to choose the most suitable one out of a list of candidates. an out-oflist answer was considered a miss; iii.) multiple-choice task -the model received a sentence pair with the four connectives marked as a, b, c, d, and tasked to return the letter corresponding to the appropriate connective. failure to provide one of the options was considered a miss. we collected data from a single prompt per task, and each underwent a selection process. we drafted a first prompt, and then used chatgpt to obtain four more variants of it. 11 we averaged the perplexity of all the open models on each prompt and chose the one with the lowest average perplexity. we randomized the order of few-shot examples, options to choose from, and correct answers during inference. in the few-shot scenario, the models saw one example for each connective. 12 perplexity evaluation. we computed the perplexity (ppl) of each item in the dataset, and grouped those corresponding to the same sentence pair. then, we chose the connective from the item with the lowest ppl. we derived from the connective the type and order of the relation and computed models' accuracy by comparing these results with the human ground truth obtained with crowdsourcing annotation. we call this accuracy as accuracy perplexity score (aps). ## results and discussion we present the results of our experiments by first providing an overview of the core outcomes from the prompting-and perplexity-based evaluations (sec. 5.1). then, we provide an in-depth analysis of the top-performing models, and a comparative analysis of qwen models across different sizes (in sec. 5.2). the fine-grained results, including performances of each model, task, relation type, and setting are reported in appendix h. ## conclusion and future work in this paper, we introduced explica, a dataset designed to evaluate llms on pcd tasks with the aim to assess their reasoning abilities on explicitly expressed relations. explica is the first dataset to incorporate both causal and temporal relations between events, with acceptability ratings provided by native speakers via crowdsourcing. results indicate that explica is particularly challenging, even for commercial models like gpt4omini, which is outperformed by open models when leveraging probabilistic scores, and gpt4o, which struggles to reach 0.80 accuracy. we observed that llms exhibit variable performance according to both the evaluation setup and the relation type and order. the models' rating distribution do not approximate the humans' ratings, especially for certain types of relations and linguistic orders. moreover, our results suggest that the competence on causal relations measured with perplexity is significantly more accurate than prompting performance (especially in smaller models). for future works, we aim to further explore the effect of model size on different model families, and we plan to adopt explica also to investigate how models interpret implicit causality. so far, the presented results reveal that, despite their increasing size, the knowledge of causal relations is still suboptimal in llms, which also show a strong tendency to confound temporal relations with causal ones, compounded with a limited abstraction from the surface linguistic presentation order of the events. e can conclude that, although llms have undoubtedly progressed along the 'ladder of causation' @xcite , they have still several steps to climb.