Commonsense Evidence Generation and Injection in Reading Comprehension (2020)

Abstract: Human tackle reading comprehension not only based on the given context itself but often rely on the commonsense beyond. To empower the machine with commonsense reasoning, in this paper, we propose a Commonsense Evidence Generation and Injection framework in reading comprehension, named CEGI. The framework injects two kinds of auxiliary commonsense evidence into comprehensive reading to equip the machine with the ability of rational thinking. Specifically, we build two evidence generators: one aims to generate textual evidence via a language model; the other aims to extract factual evidence (automatically aligned text-triples) from a commonsense knowledge graph after graph completion. Those evidences incorporate contextual commonsense and serve as the additional inputs to the reasoning model. Thereafter, we propose a deep contextual encoder to extract semantic relationships among the paragraph, question, option, and evidence. Finally, we employ a capsule network to extract different linguistic units (word and phrase) from the relations, and dynamically predict the optimal option based on the extracted units. Experiments on the CosmosQA dataset demonstrate that the proposed CEGI model outperforms the current state-of-the-art approaches and achieves the highest accuracy (83.6%) on the leaderboard.

## introduction contextual commonsense reasoning has long been considered as the core of understanding narratives @xcite in reading comprehension @xcite . despite the broad recognition of its importance, the research of reasoning in narrative text is limited due to the difficulty of understanding the causes and effects within the context. comprehending reasoning requires not only understanding the explicit meaning of each sentence but also making inferences based on implicit connections between sentences. to answer a contextual commonsense question correctly, two important characteristics need to be well considered. first, the information that is required to infer a correct answer may be beyond the context, and hence adding external commonsense knowledge to guide the reasoning is necessary. second, how to use external knowledge to gain contextual understanding between the paragraph, question and option set is difficult but important. despite the great success of large pre-trained models such as bert @xcite , gpt @xcite and roberta @xcite , recent studies suggest that those models fail to capture sufficient knowledge and provide commonsense inference. for example, @xcite show that language models perform well in reasoning about entity names, but fail to capture rich factual knowledge. moreover, @xcite state that language models fail on half of the reasoning tasks which require symbolic operations such as comparison, conjuction and composition. to this end, we introduce a commonsense evidence generation and injection framework in reading comprehension, named cegi, which generates useful evidence from textual and factual knowledge and injects the generated evidence into pre-trained models such as roberta. we propose to generate evidence regarding the facts and their relations. more specifically, we use language models to generate textual evidence and extract factual evidence from a knowledge graph after graph completion. we then inject both evidences into the proposed contextual commonsense reasoning model to predict the optimal answer. as shown in figure 1 , the textual generated evidence "he will call for medical attention" and factual generated evidence "both blood & ambulance locate at hospital" can help the model find the correct answer "an ambulance would likely come to the scene". to capture relations between the paragraph and question, many reading comprehension models @xcite have been proposed. however, those reasoning models are essentially based on the given context without understanding the facts behind. moreover, in many situations, the candidate option set contains distractors that are quite similar to the correct answer. in other words, understanding the relations among the option set is also important. we employ a capsule network @xcite , which uses a routing-by-agreement mechanism to capture the correlations among different options and make the final decision. our proposed cegi framework not only utilizes external commonsense knowledge to generate reasoning evidence but also adopts a capsule network to make the final answer prediction. the explainable evidence and the ablation studies indicate that our method has a large impact on the performance of the commonsense reasoning in reading comprehension. the contributions of this paper are summarized as follows: 1) we introduce two evidence generators which are learned from textual and factual knowledge sources; 2) we provide an injection method that can infuse both evidences into the contextual reasoning model; 3) we adapt a capsule network to our reasoning model to capture interactions among candidate options when making a decision; 4) we show our cegi model outperforms current state-of-the-art models on the cosmosqa dataset and generates richer interpretive evidence which helps the commonsense reasoning. ## task definition in multi-choice reading comprehension, we are given a paragraph p with t tokens @xmath0, p 2 , . . . , p t ], a question q containing n tokens for simplicity, we denote @xmath1, q, o} as one data sample and denote @xmath2, y 2 , . . . , y m ] as a one-hot label, where each scale y @xmath3. in the training stage, we are given n set of (x , y) n , the goal is to learn a model f : @xmath4. in the testing, we need to predict y test given test samples x test . when answering a question according to the paragraph, we observe that the context itself often does not provide enough clues to guide us to the correct answer. to this end, we need to know comprehensive information beyond the context and perform commonsense reasoning. hence, we split the task into two parts: evidence generation and answer prediction, respectively. our proposed cegi model addresses both parts accordingly by two generators: textual evidence generator and factual evidence generator. in textual evidence generator, our goal is to generate relevant evidence text @xmath5, e 2 , . . . , e k ] given question q and paragraph p. note that the number of evidence tokens k may vary in different question and paragraph pair. in factual evidence generator, the goal is to generate relevant text that describes the relations between facts where the facts are the entities from paragraph, question and options. in the second part, we aim to learn a classifier p (y|p, q, o, e) that predicts the correct option when a new data sample is given. by using the evidence generated from the first part, we expect the reasoning model can be enhanced with the auxiliary information, especially for those questions that require contextual commonsense reasoning. ## methodology to tackle reading comprehension task with commonsense reasoning, we introduce a commonsense evidence generation and injection (cegi) framework. the system diagram of the cegi framework is shown in fig. 2 . first, the evidence generation module produces textual evidence and factual evidence. those generated evidences will be used as auxiliary inputs for the reasoning model. second, the contextual commonsense reasoning module generates deep contextual features for the paragraph, question, option and evidence. meanwhile, a bidirectional attention mechanism is applied to the features to capture representations of the pair of paragraph, question, option set and evidence. next, all pairs are concatenated and fed into a convolutional neural network for extracting different linguistic units of the options. at least, a capsule network is then applied to dynamically update the representation vector of the candidate options. the final answer is one of the options with the largest vector norm. we describe more details of each component in the following subsections. ## experiments in the experiment, we evaluate the performance of our proposed cegi framework from different aspects, including evidence generation tasks and the answer prediction of contextual commonsense reasoning tasks. ## conclusion in this paper, we proposed a commonsense evidence generation and injection model to tackle reading comprehension. both textual and factual evidence generators were used to enhance the model for answering questions which requires commonsense reasoning. after the evidences were generated, we adopted attention mechanism to find the relation and match between paragraph, question, option and evidence. we used convolutional network to capture the multi-grained features. to capture diverse features and iteratively make a decision, we proposed using a capsule network that dynamically capture different features to predict the answer. the ai2 leaderboard of cosmosqa task demonstrated that our method can tackle commonsensebased reading comprehension pretty well and it outperformed the current state-of-the-art approach k-adapter with a 2% increase in term of accuracy. experiments regarding the evidence generators showed that the generated evidence is humanreadable and those evidences are helpful for the reasoning task. cegi-gpt2 and cegi-bert in figure 3 . we observe that using the multi-tasking loss, cegi-bert and cegi-gpt2 generate more accurate evidence. moreover, using those generated evidences is helpful for predicting the correct answer. in the first example, the evidence generated by cegi-gpt2 "they trust me and i handle animal without supervision." can help select the answer d "they delegate tasks to me without supervision." in the second example, the evidence generated by cegi-bert "i would not have experienced the feelings that i had for her." is close to the answer d.