CALM -Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023)

Abstract: Causal reasoning is a critical component of human cognition and is required across a range of question-answering (QA) tasks (such as abductive reasoning, commonsense QA, and procedural reasoning). Research on causal QA has been underdefined, task-specific, and limited in complexity. Recent advances in foundation language models (such as BERT, ERNIE, and T5) have shown the efficacy of pre-trained models across diverse QA tasks. However, there is limited research exploring the causal reasoning capabilities of those language models and no standard evaluation benchmark. To unify causal QA research, we propose CALM-Bench, a multi-task benchmark for evaluating causality-aware language models (CALM). We present a standardized definition of causal QA tasks and show empirically that causal reasoning can be generalized and transferred across different QA tasks. Additionally, we share a strong multi-task baseline model which outperforms single-task fine-tuned models on the CALM-Bench tasks.

## introduction causal reasoning is a crucial aspect of human cognition and is critical to the development of our mental models of reality @xcite @xcite . theories of causation have been studied extensively across philosophy @xcite , physics @xcite , cognitive science @xcite , and probability and statistics @xcite , amongst many other fields. explorations of causality in the language domain tend to be semantic, linguistic, or logical in nature as access to direct observational data or event probabilities is not assumed nor is required. descriptions of causality can be linguistically valid but factually incorrect (e.g. butter is the leading cause of factory deaths). therefore, causal reasoning in language should ideally be logically consistent and grounded figure 1: an cqa example from the copa @xcite . cqa requires identifying causal concepts, linking those concepts to causal relations, and reasoning over those relations. in commonsense knowledge. the counterfactual theory of causation @xcite provides a useful definition of causation for language applications. it posits that causation is relational (there is a cause and effect), temporal (the cause must precede the effect), and counterfactual (if the causing event had not occurred, the effect would not have occurred). various natural language processing (nlp) applications require identifying causal relations and reasoning over those relations. these nlp applications can be split into two general categories: causal relation identification (cri) and causal question-answering (cqa). cri tasks aim to identify and extract cause/effect spans from descriptions of causal events. cri requires linguistic knowledge -relying on lexical triggers (i.e. causative verbs and causal connectives) and grammatical structures @xcite . historically, the majority of nlp research on causality has focused on cri. in contrast to cri, cqa tasks require both background causal knowledge and reasoning. consider the question air pollution in the city worsened. what is the cause of this? (figure 1 ). to answer this question, commonsense knowledge about factories, pollution, and the ability to infer both causal and counterfactual relations is required. general cause prediction cause comparison effect prediction effect comparison reading comprehension science, world wiqa @xcite context: 1. a seed is in soil. 2: the seed germinates. 3: the plant grows roots. 4: the plant grows out of the ground. 5: the plant gets bigger. 6: the plant flowers. 7: the flower produces fruit. 8: the fruit releases seeds. 9: the plant dies. question: suppose less pollution in the environment happens, how will it affect the population of plants? a: more b: less c: no effect recent advances in foundation language models have demonstrated the effectiveness of pre-trained models across a wide range of nlp and general language understanding tasks. the term foundation model @xcite describes any monolithic neural model (e.g. bert @xcite ) that captures general knowledge through pre-training and is able to transfer that knowledge to a wide range of downstream tasks. foundation language models exhibit general reasoning capabilities @xcite , factual knowledge recall @xcite , and superior performance on a wide range of qa tasks @xcite @xcite . knowledge in foundation language models is usually injected through denoising objectives (e.g. masked token prediction) @xcite . however, interpreting and extracting that knowledge is difficult (requiring specialized probing tasks) and these models can be susceptible to exploiting superficial @xcite . cqa tasks could provide a unique opportunity to develop both explainable models (through producing causal explanation chains) and expand the reasoning capabilities of those models in qa settings. to date, no comprehensive study has explored the causal reasoning capabilities of foundation language models. we aim to unify research around cqa research by providing a definition for cqa rooted in the cognitive understanding of causal learning and propose calm-bench, a multi-task causal questionanswering benchmark for evaluating causalityaware language models (calm). calm-bench ( ## causal question-answering we define cqa broadly as any qa task which requires both causal reasoning and causal knowl-edge provided a real or hypothetical description of events. cognitive theories of causal learning provide a framework for understanding and evaluating the process of causal question-answering in nlp applications. the inferential theory of causal learning posits that causal learning is a slow and effortful cognitive process that involves drawing causal conclusions over propositional premises @xcite . propositions represent our causal knowledge and contain both qualified relational information (e.g. increase of greenhouse gasses in the atmosphere causes global warming) and propositional beliefs (i believe that greenhouse gasses cause global warming). propositions are compositional (given the propositions: factories cause air pollution and pollution leads to global warming, we can infer that factories cause global warming) and directional (i.e. we would not infer that global warming causes factories). a key aspect of causal learning is the ability to generalize specific causal knowledge to new situations which is known as causal mechanism knowledge. @xcite . causal mechanism knowledge is the mental representation of a system of physical or abstract parts/processes and the expectation of causal interactions between those components that can be generalized to new situations. for example, an arson investigator relies on their mechanism knowledge of fire catalysts and forensic experience to ascertain human involvement. causal mechanism knowledge can be succinctly represented as propositional statements. causal bridging inferences describe the relationship between causal knowledge and reasoning. @xcite found that individuals invoke causal statements to bridge two events and then validate those statements against prior commonsense and causal knowledge. for example, given the events anna added butter to the hot pan. and the butter melted., we implicitly invoke the bridging statement heat caused the butter to melt based on our prior knowledge. solving cqa tasks can be decomposed into three general steps: causal concept identification, causal knowledge linking, and causal reasoning. consider figure 1 , the causal concepts of air pollution and factories are identified and then linked to background knowledge in order to produce causal knowledge. causal knowledge can be expressed as relational triples (e.g. factory, cause-effect, pol-lution) which are effectively propositional statements. the final step requires reasoning over that knowledge through both inferential and counterfactual reasoning. we infer that the increase in factory production results in worsening air pollution based on causal knowledge that factory production causes pollution. the counterfactual, if factories shut down then air pollution would not increase, allows us to eliminate the second option. arriving at the correct answer in this example is difficult without any background causal knowledge and reasoning over that knowledge. an important aspect of causal learning is the ability to generalize causal mechanism knowledge to novel situations and task settings. we can see in calm-bench consists of diverse qa tasks requiring social, world, and science knowledge. our empirical experiments aim to validate the assumption that causal reasoning is transferable across these qa tasks in calm-bench and produce strong baselines for future research in this space. ## conclusion in this paper, we provide a unified definition of causal question-answering in the context of natural language applications. drawing from the cognitive science literature, we posit that cqa tasks require both causal reasoning and causal knowledge. based on this definition, we introduce the calm-bench, the first multi-task cqa benchmark to evaluate the general causal reasoning capabilities of foundation language models. we provide empirical evidence which validates the intuition that causal reasoning and knowledge are transferable across the cqa tasks. knowledge-enriched language models like ernie are likely to outperform distributional models (i.e. bert) across all tasks in both the single-task fine-tuning and multi-task fine-tuning settings. finally, we provide a set of strong baselines for future work exploring causal question-answering and the causal reasoning capabilities of language models. while our experiments show causal knowledge is transferable, these models are still opaque. cqa provides a unique opportunity for model explainability through causal explanation structures and reasoning chains. the e-care and wiqa task have annotated explanations that provide a useful starting point. causal knowledge sources like causenet @xcite , conceptnet @xcite , and wikidata foot_2 can also be used to generate causal explanations. we believe the next evolution of foundation language models will have stronger causal reasoning capabilities and implicit structured causal knowledge. calm-bench provides a starting point for further research on causal question-answering.