Exploring Defeasibility in Causal Reasoning (2024)

Abstract: Defeasibility in causal reasoning implies that the causal relationship between cause and effect can be strengthened or weakened. Namely, the causal strength between cause and effect should increase or decrease with the incorporation of strengthening arguments (supporters) or weakening arguments (defeaters), respectively. However, existing works ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defeasible settings. In this work, we present \delta -CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning. \delta -CAUSAL includes around 11K events spanning ten domains, featuring defeasible causality pairs, namely, cause-effect pairs accompanied by supporters and defeaters. We further show that current causal strength metrics fail to reflect the change of causal strength with the incorporation of supporters or defeaters in \delta -CAUSAL. To this end, we propose CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships. CESAR achieves a significant 69.7% relative improvement over existing metrics, increasing from 47.2% to 80.1% in capturing the causal strength change brought by supporters and defeaters. We further demonstrate even Large Language Models (LLMs) like GPT-3.5 still lag 4.5 and 10.7 points behind humans in generating supporters and defeaters, emphasizing the challenge posed by \delta -CAUSAL.

## introduction causality @xcite , a fundamental concept of artificial intelligence, describes the relationship between two events where one event, namely the cause, results in the occurrence of another event, namely the effect. understanding causality enhances decisionmaking in various areas such as medicine @xcite , disease treatment @xcite ; me and struchiner, cause a heavy earthquake hits this populated city. supporter: it requires a lot of funding to reconstruct a city hit by a disaster. defeater: months later, lots of neighboring countries provide enough support for this country's post-disaster construction. ## estimating causal strength in this section, we address research question i on causal strength. we first highlight the limitations of current metrics in § 5.1. then, we detail the definition, comparison with other metrics, versatility, and case study of cesar from § 5.2 to § 5.5. ## conclusions our paper introduces δ-causal, a pioneering benchmark that focuses on the often overlooked aspect of causal reasoning: defeasibility. even state-of-the-art models like gpt-3.5 fall significantly short compared to human performance in understanding defeasibility shown by δ-causal. we further demonstrate the limitations of current causal strength metrics in capturing causal strength changes brought by supporters and defeaters. to circumvent these limitations, we propose cesar, a robust metric that outperforms existing measures by a remarkable 69.7% improvement in capturing these changes. our research contributes to the advancement of causal reasoning by emphasizing defeasibility and providing valuable insights for improving language models' understanding of nuanced causal relationships. this work establishes a foundation for future studies on developing more advanced defeasible causal reasoning systems.