C ausal QA : A Benchmark for Causal Question Answering (2022)

Abstract: At least 5% of questions submitted to search engines ask about cause-effect relationships in some way. To support the development of tailored approaches that can answer such questions, we construct Webis-CausalQA-22, a benchmark corpus of 1.1 million causal questions with answers. We distinguish different types of causal questions using a novel typology derived from a data-driven, manual analysis of questions from ten large question answering (QA) datasets. Using high-precision lexical rules, we extract causal questions of each type from these datasets to create our corpus. As an initial baseline, the state-of-the-art QA model UnifiedQA achieves a ROUGE-L F1 score of 0.48 on our new benchmark.

## introduction the term "causality" usually refers to a directed relationship between events in which one is the cause of the occurrence of the other, called the effect. many empirical studies begin with a research question about a causal relationship, ranging from "yes/no"-questions such as "does the quality of education affect economic growth?" to open-ended questions such as "what causes depression?". but the general public also frequently asks causal questions. figure 1 shows an example of the top google, bing, and naver search result for the question "can broccoli cause constipation?". while bing directly answers the question in the affirmative, google's featured snippet and naver's first snippet claim that broccoli actually has the opposite effect. with at most a few thousand question-answer pairs, existing causal question answering datasets are relatively small and include only one type of causal question, e.g., "yes/no"-questions @xcite , "what-if"questions @xcite , "why"-questions @xcite @xcite , or multiple-choice questions @xcite can broccoli cause constipation? foods that may help prevent constipation for many people, eating more high fiber foods can help ease constipation. these foods include: most vegetables, including carrots, peas, broccoli, and okra. can broccoli cause constipation? figure 1 : "can broccoli cause constipation?" google's and naver's top results both disagree with that of bing. ## related work we review the literature in four areas: prior typologies of causal questions, causal qa, as well as generic qa datasets and qa systems. ## a typology of causal questions while various types of causality-related questions have been previously addressed in automated ques-tion answering, there has been no attempt so far to systematize "questions about causality" as a class in the qa community. computational processing of causal structures, however, dates back to the 1970s and the early ai research on causal dependencies between events in the context of story comprehension. notably, @xcite developed a computational model of question answering based on a theory of "conceptual information processing". their qualm system was capable of answering 13 types of questions about stories-9 types being related to causal relationships. following lehnert, we define questions about causal relationships in terms of causal chains @xcite and integrate lehnert's causal categories into a more specific typology of causal questions. while lehnert's definitions and categories are motivated by and directly linked to processing strategies in a story comprehension system, our typology is more generic and motivated by the semantic and pragmatic properties of causal questions. at the semantic level, we group causal question types in terms of which component of a causal chain a question addresses. our type set combines lehnert's causality-related categories and verberne et al.'s categories of "why"-questions @xcite as subtypes. at the pragmatic level, we group question types in terms of the assumed purpose of inquiry or the so-called intent of a question. we arrived at the pragmatic categories in a data-driven fashion by analyzing 1,000 questions (100 sampled from each of the 10 selected qa datasets; cf. ## the webis-causalqa-22 corpus in this section, we describe how we extract causal questions from the ten qa datasets in ## evaluation to establish a first baseline effectiveness for causal question answering on the webis-causalqa-22 benchmark, we report the results achieved by the state-of-the-art unifiedqa model @xcite . unifiedqa is a text-based question answering model that has been reported by @xcite to perform well on 32 qa datasets, including squad v. 2.0, where it achieved a bagof-word-based f 1 score of 0.90. we experiment with version 2 of the model, checkpoint 1363200, using (1) the base model, and (2) a version finetuned on webis-causalqa-22 using the hyperparameters of @xcite . 4 in a pilot study, we attempted to fine-tune a joint model on all datasets but found fine-tuning per source dataset to yield better results. moreover, we experiment with the causal questions extracted from the original train/dev splits proposed by the authors as well as a random 90/10 train-test split of our own. the reason for the latter is that, for some datasets, by chance, only few causal questions are part of the original train/dev splits (compare the number of effectiveness is measured using the rouge-l scores precision, recall, and f 1 @xcite , as well as the traditional exact match (em) and f 1 measures. the rouge-l measures are based on the longest common subsequence between a predicted answer and a ground truth answer, whereas em requires the order of all tokens to match and the traditional f 1 measure is based on an order-invariant bag-of-words representation. if a question has more than one ground truth answer, the maximum score per measure and question is taken. effectiveness is measured both per constituent dataset of webis-causalqa-22, and averaged using microand macro-averaging across datasets. the columns "random 90/10 split" reports the corresponding effectiveness scores of uni-fiedqa for the fine-tuned model version, where fine-tuning was repeated on the different training set. comparing the rouge-l f 1 scores to the fine-tuned model on the original split, we observe the largest differences for the datasets hot-potqa (from 0.53 in the original split to 0.73 in the new one), newsqa (from 0.58 to 0.73), squad (from 0.83 to 0.95), searchqa (from 0.62 to 0.54), and gooag (from 0.12 to 0.15). the effectiveness on hotpotqa increases because the original split used more difficult questions for the dev set than for training @xcite . the effectiveness on the new splits of squad v. 2.0 and newsqa increases because the unifiedqa base model was trained on both datasets causing a leakage of training data. the effectiveness on searchqa decreases potentially due to overfitting to the training set, or a particularly easy dev set in the original dataset by chance as the original dataset was split by time (different years for dev and test sets than for training). the effectiveness on gooaq increases slightly because the original train/dev sets were explicitly made dissimilar by avoiding word overlaps while our random split does not. moreover, with the new split, gooaq is evaluated on many more questions because the original dev set contained far fewer causal questions than 10% of the whole dataset. overall, when comparing macro-averages across datasets, we find that fine-tuning improves the macro-averaged rouge-l f 1 scores from 0.35 to 0.48 on the original train/dev split, and to 0.53 on the random 90/10 split. micro-averaging generally results in higher scores when compared to macro-averaging due to imbalanced distribution of question-answer pairs across datasets, where paq has the largest influence. interestingly, when comparing the macro-averaged rouge-l f 1 scores of the original train/dev split with the random one, and the corresponding micro-averaged ones, the microaveraged ones decrease from 0.81 to 0.72, while the macro-averaged ones increase as mentioned above. this is mainly caused by gooaq having a much higher weight overall due to contributing more than 14,500 question-answer pairs, the second-largest amount following paq, compared to only 33 in the original train/dev split. it is a matter of debate, which of the two splits and which of the two averages are to be preferred as a baseline. at present, we recommend using the original train/dev split (especially, if a model was trained on one of our corpus's constituent datasets, like unifiedqa), and then the macro-averaged rouge-l f 1 score. in case of unifiedqa, this score is 0.48 for the model version fine-tuned on each constituent dataset individually. ## conclusion we constructed webis-causalqa-22, the first large benchmark dataset of 1.1 million causal questionanswer pairs, which serves to advance research in causal question answering. to ensure diversity of questions, we extracted them using seven hand-crafted high-precision lexical rules to capture as many subtypes of causal questions as possible. these rules were derived from a new typology of causal questions, which in turn is based on relevant related work on question typologies. a manual analysis of a sample of questions was used to characterize causal questions in terms of two dimensions: (1) their semantic properties, i.e., according to which element of the causal structure the question is asked (antecedent, consequent, or the causal chain) and (2) their pragmatic interpretation, i.e., the underlying intention or assumed information need of the questioner (e.g., prevention of medical problems). furthermore, a subsequent analysis of the causal questions contained in a search engine log showed that a significant proportion of 5% of question queries are causal. finally, we evaluated the state-of-the-art model unifiedqa on our corpus as an initial baseline for causal question answering. causal questions represent a hitherto poorly considered challenge for both search engines in general and qa systems in particular. in this respect, our typology serves as a guide for the development of new technologies: the semantic dimension is relevant for understanding queries, while the pragmatic dimension may guide search engines and qa systems in finding and presenting answers. in addition, linking current text-based models with algorithms for causal inference is a promising direction to answer more complex questions for which answers cannot be found directly on the web. causenet may also prove useful here, as the graph of causeeffect relationships already makes such connections. however, to maximize user confidence in an information system's answers to causal questions, all causal claims must be supported by evidence (e.g., in the form of scientific studies).