CRAB : Assessing the Strength of Causal Relationships Between Real-world Events (2023)

Abstract: Understanding narratives requires reasoning about the cause-and-effect relationships between events mentioned in the text. While existing foundation models yield impressive results in many NLP tasks requiring reasoning, it is unclear whether they understand the complexity of the underlying network of causal relationships of events in narratives. In this work, we present CRAB, a new Causal Reasoning Assessment Benchmark designed to evaluate causal understanding of events in real-world narratives. CRAB contains fine-grained, contextual causality annotations for ~2.7K pairs of real-world events that describe various newsworthy event timelines (e.g., the acquisition of Twitter by Elon Musk). Using CRAB, we measure the performance of several large language models, demonstrating that most systems achieve poor performance on the task. Motivated by classical causal principles, we also analyze the causal structures of groups of events in CRAB, and find that models perform worse on causal reasoning when events are derived from complex causal structures compared to simple linear causal chains. We make our dataset and code available to the research community.

## introduction understanding narratives requires understanding the cause-and-effect relationships between interconnected sub-events of those narratives. when reading text, humans immediately induce potential causal links between the events presented as part of a larger scenario @xcite . for example, in figure 1 , when reading an article about the acquisition of twitter in 2022, a reader would implicitly assign causal links between events such as e2: "elon musk closes 44 billion dollar deal to buy twitter" and e3: "twitter delists from the nyse". however, building accurate causal mental models of the situations depicted in narratives poses several complex challenges. first, human causality judgments are rarely binary. instead, they fall on a spectrum depending on human perception of other mediating or confounding events @xcite . for example, in figure 1 , e2 is a mediator event for the causal relationship of e1 and e3, likely affecting the human perception of the causal relationship between e1 and e3. second, causality judgments depend on the context depicting the events in question -a context that can affect perceptions of causality. for example, a high causal judgment might be assigned between e4 and e6 in figure 1 . however, the introduction of new information about e5, another potential cause of e6, might downgrade the perceived intensity of a causal link between e4 and e6. finally, because context is critically important to judging causal relationships about events, and most narratives offer an incomplete (and sometimes biased) reporting of particular scenarios, multiple sources may be required to paint an accurate picture of the causal relationships between multiple interconnected events. addressing these challenges, we introduce crab, a new causal reasoning assessment benchmark that contains fine-grained, contextual causality annotations of real-world events that happened in the past ten years and received extensive media coverage. to collect the proposed benchmark, we design a crowdsourcing framework motivated by standard causal principles from cognitive science @xcite and actual causality @xcite that study how humans perceive and express causality and responsibility among events. using this knowledge frame, we automatically extract the events of newsworthy stories by integrating large pre-trained lms into the dataset creation loop. we then construct causal graphscombinations of inter-connected events forming different causal chains and frames, as presented in figure 2 -from the extracted events and assess the strength of the causal relationships between these events using human annotators. our resulting benchmark, crab, contains ∼2.7k high-quality event pairs, their causal score, and the respective documents in which the events appeared. all the events are grouped into 1.8k causal frames and 352 causal chains. we use this benchmark to assess the abilities of state-of-the-art (sota) models to understand and reason over the causal relationships of real-world events present in a set of contexts (i.e., online news articles). our analysis reveals that llms can capture explicit causal statements through pre-training, but they face difficulty applying causal reasoning to new scenarios, limiting their generalization and accuracy in offering predictions and explanations. we further stratify our results based on the structures of causal frames and chains, showing that they struggle with assessing the causality between events derived from complex causal structures compared to simple linear causal chains, especially when these events are extracted from different documents. ## preliminaries on causality in this section, we define the main causality concepts that we use to create and analyze crab. actual causality actual causality refers to the causal relationship between specific events and their causes in the real world @xcite and seeks to understand the precise mechanisms by which one event leads to another, going beyond mere correlation. understanding actual causality is crucial for humans to comprehend the underlying factors driving specific events, helping them make sense of the world, predict outcomes, and take practical actions toward those outcomes. research in causal inference has attempted to formalize actual causality using causal models that map how humans perceive and attribute cause and responsibility to events and their outcomes. however, human perception of causality usually depends on background context, implicit biases, epistemic state, and lack of information, making the task of actual causality attribution challenging to formalize @xcite . additionally, in cases where the responsibility of an event can be attributed to more than one preceding event, observers tend to assign different attribution to the contributing causes @xcite . therefore, when events are described with natural language, the causal judgments are not binary but relative, enabling comparisons between causal events @xcite . ## dataset construction in this section, we give an overview of causal event linking and its associated challenges, and describe our approach for building our crab benchmark centered around these challenges. we create a corpus of documents from which we automatically extract events, build a timeline, and collect causality judgments between event pairs. the full data creation pipeline is described in figure 3 . ## experimental setup to evaluate how language models reason about causality, we define different experimental frameworks covering various causality assessment scenarios, similar to @xcite . we investigate three tasks in ascending order of complexity in terms of causal structure: pairwise causality inference, graded causality inference, and causal chain inference. this section describes the experimental setups associated with these tasks, and the models evaluated on them. ## experimental results in the previous section, we introduced several lower-level tasks to investigate whether various llms can accurately assess pairwise causality judgments. here, we describe the results from the perspective of causal discovery, causal responsibility assessment, and multi-document causal reasoning. we perform an in-depth analysis of llm capabilities in different facets of causality, grounded to principles and dimensions of actual causality described in section 2. ## related work causal reasoning benchmarks there has been extensive research on introducing challenging causal benchmarks in recent years. they focus on commonsense causal discovery and reasoning @xcite @xcite , as well as assessing causal reasoning from the perspective of plausible alternatives and counterfactuals @xcite @xcite . similar to our work, there have been attempts to create benchmarks that incorporate the cross-document setup @xcite and causal structures @xcite , but not on real-world events. finally, domain-specific datasets that perform causal reasoning in the medical @xcite and the legal @xcite domains have been introduced. crab differs from prior work regarding the nature of its events, real-world events, and the type of causal reasoning that assesses llms, actual causality. assessing causal reasoning many existing studies investigate whether nlp models understand causality @xcite @xcite , providing methods to quantify the causal abilities of language models to discover causal relationships in documents @xcite @xcite and extracting cause-effect pairs as a subtask of information extraction from text @xcite @xcite . another line of work studies the improvement of causal reasoning generation using instruction prompting @xcite , chain-of-thought (cot; @xcite , and prompt augmentation @xcite . ## conclusions this work introduces crab, a new causal reasoning benchmark that contains fine-grained, contextual causality annotations for ∼2.7k pairs of realworld events that describe various newsworthy stories. using crab, we explore how llms assess the causal relationships between events when the causal signal comes from different contexts. additionally, we assess llms performance in identifying and assessing complex causal structures. our findings suggest that state-of-the-art language models perform poorly in pairwise causal inference and responsibility assignment when events are spread across documents. furthermore, this weak performance is amplified when llms must identify causal relationships in complex causal structures rather than simple linear chains.