e- CARE : a New Dataset for Exploring Explainable Causal Reasoning (2022)

Abstract: Understanding causality has vital importance for various Natural Language Processing (NLP) applications. Beyond the labeled instances, conceptual explanations of the causality can provide deep understanding of the causal fact to facilitate the causal reasoning process. However, such explanation information still remains absent in existing causal reasoning resources. In this paper, we fill this gap by presenting a human-annotated explainable CAusal REasoning dataset (e-CARE), which contains over 20K causal reasoning questions, together with natural language formed explanations of the causal questions. Experimental results show that generating valid explanations for causal facts still remains especially challenging for the state-of-the-art models, and the explanation information can be helpful for promoting the accuracy and stability of causal reasoning models.

## introduction causal reasoning is one of the most central cognitive abilities of human beings @xcite , which enables one to understand the observed facts and predict the future. however, although recent causal reasoning models have achieved impressive performances on certain hand-crafted datasets, there still remains a considerable gap compared to human performances, as they cannot achieve stable performances across different datasets and are susceptible to adversarial attacks @xcite @xcite . one key factor leading to such drastic contrast is that, present causal reasoning models only learn to induce empirical causal patterns that are predictive to the label, while human beings seek for deep and conceptual understanding of the causality to explain the observed causal facts. the conceptual acid is corrosive explanations can not only serve as a touchstone to examine whether the underlying causal mechanism has been thoroughly understood, but it can also in turn support the causal reasoning process. as illustrated in figure 1 , observing the causal fact c 1 : adding rock into hydrochloric acid causes e 1 : rock dissolved, one may further ask why such a causal relationship exists and reach the plausible conceptual explanation that acid is corrosive, which goes beyond the isolated facts and reaches the conceptual nature to reveal the principle of the causal mechanism. however, despite the critical importance of conceptual explanations in causal reasoning, there is still a lack of such an explainable causal reasoning dataset. to fill this gap, we contribute an explainable causal reasoning dataset (e-care),together with a new causal explanation generation task, and a novel causal explanation quality (ceq) evaluation metric. the e-care dataset is constructed by crowdsourcing and contains over 21k multiple-choice causal reasoning questions, which makes e-care the largest human-annotated commonsense causal reasoning dataset to the best of our knowledge. in addition to the causal reasoning question itself, e-care also provides a free-text-formed conceptual explanation for each causal question to explain why the causation exists. on this basis, we propose a new causal explanation generation task that requires models not only to choose the correct causal fact but also to generate the ex-planation for the choice. in addition, to directly measure the quality of generated explanations, we propose a novel causal explanation quality evaluation metric (namely, ceq score). compared to conventional text generation evaluation metrics such as bleu @xcite and rouge @xcite which mainly evaluate the textual or semantic similarity between generated explanations with golden annotations, ceq score focuses on evaluating how much promotion an explanation can bring to understanding the causal mechanism. the dataset is publicly available at https: //github.com/waste-wood/e-care/ . experimental results demonstrate that the causal questions of e-care are still challenging for the state-of-the-art (sota) pretrained language models, indicating the effectiveness of the e-care dataset in evaluating the causal learning ability of models. in addition, the explanation signal received in the training process can enhance the performance and the stability of the reasoning model, while the sota baselines still have trouble in explaining the causal facts at a conceptual level. these analyses highlight the importance of the conceptual explanations in causal reasoning, and suggest an avenue for future researches. ## e-care: an explainable causal reasoning dataset e-care contains a total of 21,324 instances, corresponding to 13,048 unique explanations. this also makes e-care the largest human-annotated commonsense causal reasoning benchmark. the corpus-level statistics of the e-care dataset are shown in as shown in ## causal explanation quality (ceq) score a number of automatic scores have been proposed to evaluate the quality of generated explanations, such as bleu @xcite and rouge @xcite . however, these metrics evaluate the quality of the generated explanations only through comparing the textual or semantic similarity between the generated explanations and the golden annotation. alternatively, an ideal causal explanation quality evaluation metric should directly measure if the causal fact is appropriately explained by the explanation. hence, we propose a novel causal explanation quality evaluation metric (namely, ceq score) as a step towards directly measuring the quality of generated explanations. we devise the ceq score based on the consideration that a better explanation should provide more information for understanding the causality, so that the prediction model can more accurately estimate the reasonableness of the causal fact. previous literature characterized such reasonableness as the causal strength of the given causal fact @xcite , where the causal strength is a score in [0, 1]. hence, in theory, for a valid causal fact, its causal strength should be equal to 1. given a valid causal fact, an explanation should help to increase its estimated causal strength to the groundtruth value 1. therefore, we can evaluate the quality of a generated explanation by measuring the increase of causal strength brought by the explanation. specifically, let c, e, and x denote the cause, the effect and the generated explanation, respectively. formally, the ceq score is defined as: in this paper, we employ a widely-adopted model-agnostic method proposed by @xcite to calculate the causal strength. the model-agnostic nature enable us to avoid reliance on certain models and keep the fairness of evaluation. specifically, the phrase-level causal strength is derived through synthesizing the word-level causality. ## experiments and results we examine the performance of state-of-the-art pretrained language models on the causal reasoning task and the explanation generation task. furthermore, we investigate the specific role of explanations in causal reasoning by: (1) a predictand-generate experiment, which requires models to conduct the causal reasoning task and generate corresponding explanations simultaneously; ## discussion in this paper, we introduce additional explanation information for the causal reasoning process, and propose a corresponding explanation generation task. previous literature concluded the explanation generation process as an abductive reasoning process @xcite and highlighted the importance of the abdutive explanation generation, as it may interact with the causal reasoning process to promote the understanding of causal mechanism, and increase the efficiency and reliability of causal reasoning. for example, as figure 2 shows, one may have an observation that c 1 : adding rock into hydrochloric acid caused e 1 : rock dissolved. through abductive reasoning, one may come up with a conceptual explanation for the observation that acid is corrosive. after that, one can confirm or rectify the explanation by experiments, or resorting to external references. in this way, new ideas about causality can be involved for understanding the observed causal fact. then if the explanation is confirmed, it can be further utilized to support the causal reasoning process by helping to explain and validate other related causal facts, such as c 2 : adding rust into sulphuric acid may lead to e 2 : rust dissolved. this analysis highlights the pivotal role of conceptual explanation in learning and inferring causality. in this paper, we introduce the e-care dataset to provide causal explanations and support future research towards stronger human-like causal reasoning systems. ## conclusion in this paper, we present an explainable causal reeasoning dataset e-care, which contains over 21k causal questions, together with over 13k unique conceptual explanations about the deep understanding of the causal facts, which also makes e-care the largest causal reasoning benchmark. experimental results show that both the causal reasoning task and especially the explanation generation task remain challenging for the sota pretrained language models. moreover, the additional explanation signal can promote both the prediction accuracy and stability of models, highlighting the vital importance of the conceptual explanations in causal reasoning.