Enhancing Multiple-Choice Question Answering with Causal Knowledge (2021)

Abstract: The task of causal question answering aims to reason about causes and effects over a provided real or hypothetical premise. Recent approaches have converged on using transformer-based language models to solve question answering tasks. However, pretrained language models often struggle when external knowledge is not present in the premise or when additional context is required to answer the question. To the best of our knowledge, no prior work has explored the efficacy of augmenting pretrained language models with external causal knowledge for multiple-choice causal question answering. In this paper, we present novel strategies for the representation of causal knowledge. Our empirical results demonstrate the efficacy of augmenting pretrained models with external causal knowledge. We show improved performance on the COPA (Choice of Plausible Alternatives) and WIQA (What If Reasoning Over Procedural Text) benchmark tasks. On the WIQA benchmark, our approach is competitive with the state-of-the-art and exceeds it within the evaluation subcategories of In-Paragraph and Out-of-Paragraph perturbations.

## introduction recent model-based approaches for question answering tasks have primarily focused on finetuning pretrained transformer-based language models, such as bert @xcite and roberta @xcite , on task-specific datasets. these language models have been found to contain transferable linguistic knowledge @xcite and general knowledge @xcite that are effective for most downstream natural language processing (nlp) tasks. for more complex tasks, such as causal reasoning, pretrained language models are often limited as they lack the specific external background knowledge required to effectively reason about causality. events 1. pressure pushes up from inside the volcano. question: suppose more ash clouds forming happens, how will it affect disturbances in the environment. a. more b. less c. no effect the term causal knowledge has a long history rooted in philosophy, psychology, and many other academic disciplines @xcite . in this paper, we will refer to causal facts and causal knowledge interchangeably. broadly, causal knowledge captures relational knowledge between concepts, which can be useful for reasoning about causality. causal facts are generally extracted from natural language descriptions. for example, the statement global warming is caused primarily by human activities such as coal-burning power plants would yield the causal fact factories cause global warming. these causal facts can also be described explicitly in a knowledge base or expressed formally as triples with an explicit cause-effect relation. for ex-ample, the causal fact factories cause global warming would be expressed as the triple (factory, cause-effect, global warming). as causal facts are generated from descriptions, the veracity of these facts can be questionable. ascertaining the verisimilitude of causal knowledge is an open problem and out-of-scope for our experiments. in this paper, we explore if causal knowledge is useful for question answering and present strategies on how to enhance a pretrained language model with causal knowledge. there is limited work on incorporating external causal knowledge to improve question answering and no prior work on using causal knowledge to improve multiple-choice question answering. the task of causal question answering aims to reason about cause and effects over a provided real or hypothetical premise. specifically, we explore the multiple-choice formulation of this task in the context of the copa (choice of plausible alternatives) @xcite and wiqa (what if reasoning over procedural text) (tandon et al., 2019) benchmark tasks. copa and wiqa are both challenging causal reasoning tasks. wiqa requires reasoning on hypothetical perturbations to procedural descriptions of events. consider the example in figure 1 . to answer the hypothetical question about the downstream effect of an increase of ash and cloud on the environment, the model must be able to causally link event 3 (about ash clouds) to event 5 (erupted materials disturb the environment). if provided a causal fact such as (ash clouds, cause-effect, environmental disturbances), the model could make the causal association and logical leap that the magnitude of the effect is more. copa is another multiple-choice causal reasoning task. copa requires external commonsense causal knowledge to answer questions about the causes and effects for a provided premise. consider the following example from copa: what was the cause of this? lexically, there is limited information in the premise and alternatives that the model can exploit to answer the question. to successfully answer this question, the model requires both background knowledge about factories and the ability to make causal leaps about the impact of factories on the environment. causal facts can succinctly capture that knowledge. consider the following claimed causal fact triples from causenet @xcite ): this paper presents empirical findings on the efficacy of augmenting pretrained models with causal facts extracted to improve multiple-choice causal question answering. our contributions can be summarized as follows: our experiments demonstrate that augmenting pretrained models with external causal knowledge improves results over the baseline on the copa and wiqa benchmark tasks. for the wiqa benchmark, we present findings that show causal knowledge improves roberta's performance to nearly match the current state-of-the-art (sota) and improve upon the sota in specific sub-categories such as in-paragraph and out-of-paragraph reasoning. ## related work enhancing language models with external knowledge (in the form of a knowledge graph or knowledge base) remains an open problem. several promising strategies have emerged for injecting knowledge into large language models as part of the pretraining process. peters et al. (2019) present the knowledge attention and recontextualization (kar) layer which can be inserted into a neural language model architecture and used to train knowledge enhanced contextual embeddings. liu et al. (2019b) introduce the k-bert model which learns knowledge enabled representations from sentence trees that consist of inputs augmented with knowledge triples. sun et al. (2020) introduce the co-lake model which jointly learns language and knowledge representations through pretraining on word-knowledge (wk) graphs. to the best of our knowledge, there is no prior work on enhancing language models specifically with causal knowledge. next we provide a summary of the question answering tasks which require causal reasoning. the task of binary causal question answering poses questions of cause and effect as yes/no questions (i.e. could x cause y?). hassanzadeh et al. evaluate the application of cause-effect pairs extracted from gigawords corpus for binary question answering. kayesh et al. (2020) extends this work to automatically learn the yes/no threshold using word embeddings from bert, roberta, and other transformer-based models. sharp et al. and xie and mu (2019) consider the task of answer reranking for open-ended causal questions. both papers are evaluated on a set of causal question extracted from the yahoo! answers corpus which follows the patterns what causes ... and what is the result of .... sharp et al. present three distributional similarity models to model the contextual relationship between cause and effect phrases. xie and mu (2019) extend sharp et al. by proposing methods for building causal embeddings from causeeffect phrase pairs by transferring causal relationships from the phrase-pair level to word-pair level. our causalskipgram model for representing causal knowledge expands upon the adapted skipgram model presented by sharp et al.. finally, we summarize the current approaches to causal knowledge extraction and knowledge graph population. causal relation extraction aims to identify cause and effect phrases in various texts. the extracted cause/effect phrases can be used to populate causal knowledge bases. recent approaches frame causal relation extraction as a structured sequence classification problem. dasgupta et al. propose a lstm architecture that uses word-level embeddings to predict cause and effect tags within a sentence. li et al. (2021) present scite, a bilstm-crf model which uses pretrained flair embeddings and multi-headed self-attention to extract causal phrases. to date, there are few publicly available causal knowledge bases. causenet (heindorf et al., 2020) is currently the largest publicly available knowledge graph of claimed causal facts. causenet consists of about 12 million concepts and 11.5 million relations extracted from wikipedia and clueweb12 foot_0 . conceptnet @xcite , a public knowledge graph, consists of 36 relations and includes a causes relation. the atomic @xcite knowledge base consists of 877k textual descriptions of inferential knowledge organized around event prompts and agent-centric activities. atomic describes the social and commonsense knowledge of these events along nine if-then relations which describe the event's causes and effects on other agents/participants. comet @xcite is a language model adaptation framework that is trained on atomic and con-ceptnet to generate novel commonsense facts and construct robust commonsense knowledge bases. this paper uses causenet as its primary source for causal knowledge as it contains a broad and deep set of causal facts (including descriptions of physical processes relevant to wiqa). ## data in this section, we describe the datasets used for causal knowledge extraction and our benchmark evaluation. we use causenet as the primary source of causal knowledge for our experiments. copa and wiqa are the benchmark datasets used to evaluate causal knowledge on downstream multiplechoice question answering problems that require causal reasoning. ## methodology in this section, we present our methodologies for causal fact selection and causal representation. causal facts are extracted from causenet using token-based retrieval heuristics. we also present three strategies for representing causal knowledge. the first strategy is input augmentation, where extracted causal facts are converted to causal statements and appended to the plain text input. the second and third strategies involve generating causal embeddings using distributed similarity and knowledge graph embedding approaches. ## experimental settings in this section, we describe how we trained our causal representations and the experimental settings for augmenting roberta with causal knowledge for downstream question answering. ## results in this section, we present the results of our experiments. we find that the inclusion of causal facts improves the performance on both the copa and wiqa datasets. additionally, on the wiqa dataset, we observed the augmented input method nearly matches the sota in overall accuracy and exceeds the sota in two of the three subcategories of perturbations. ## conclusion this paper considers the challenge of enhancing pretrained language with causal knowledge to solve multiple-choice causal question answering problems which require causal reasoning. specifically, we evaluate our methods on the copa and wiqa benchmark datasets. we present methods of selecting knowledge from causenet and three strategies for representing causal knowledge (inputaugmentation, causalskipgram, and causalkge). we evaluated the efficacy of enhancing roberta with causal knowledge multiplechoice question answering tasks. we provide results that show improved performance over the roberta baseline on both the copa and wiqa benchmark tasks. roberta with causalkge provides a 6.2%/11.69% improvement in accuracy over the baseline. roberta with input augmentation posts a 3.9%/6% improvement on the copa-balanced hard dataset. we also observed that on average the inclusion of causal knowledge allows roberta to answer 36% of the questions the baseline was unable to answer. on wiqa, our approach is competitive with the sota and exceeds sota within specific evaluation subcategories. roberta with inputaugmentation improves accuracy on the in-paragraph and out-of-paragraph perturbations by (+3.3%/+4.5%) and (+2%/+3%) respectively. on average, the inclusion of causal knowledge allows roberta to answer 40% of the questions that the baseline was unable to answer on the wiqa test set. our work demonstrates that causal knowledge is valuable for causal reasoning tasks and that there are many opportunities for future work. further work can explore improving recall on causal fact selection from causenet and more sophisticated techniques to reduce the selection of irrelevant facts. on the language modeling side, future work can explore generalizing the entity-based methods which inject knowledge into the pretraining process to consider explicit causal knowledge. additionally, further work can evaluate causal knowledge in other reasoning benchmarks such as ropes and cosmosqa as well as other causal reasoning tasks.