Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources (2020)

Abstract: In order to facilitate natural language understanding, the key is to engage commonsense or background knowledge. However, how to engage commonsense effectively in question answering systems is still under exploration in both research academia and industry. In this paper, we propose a novel question-answering method by integrating multiple knowledge sources, i.e. ConceptNet, Wikipedia, and the Cambridge Dictionary, to boost the performance. More concretely, we first introduce a novel graph-based iterative knowledge retrieval module, which iteratively retrieves concepts and entities related to the given question and its choices from multiple knowledge sources. Afterward, we use a pre-trained language model to encode the question, retrieved knowledge and choices, and propose an answer choice-aware attention mechanism to fuse all hidden representations of the previous modules. Finally, the linear classifier for specific tasks is used to predict the answer. Experimental results on the CommonsenseQA dataset show that our method significantly outperforms other competitive methods and achieves the new state-of-the-art. In addition, further ablation studies demonstrate the effectiveness of our graph-based iterative knowledge retrieval module and the answer choice-aware attention module in retrieving and synthesizing background knowledge from multiple knowledge sources.

## introduction in the past decade, most previous work on question answering systems can be divided into two categories, namely, answering questions within a given context @xcite or without any context @xcite @xcite . although some methods @xcite have been reported to exceed human performance on a few metrics, those methods are seldom involved with external knowledge, such as commonsense or background knowledge, which is necessary for better understanding natural language questions, especially in some settings that do not mention the background knowledge. to effectively leverage external human-made knowledge graphs, previous methods @xcite @xcite @xcite adopt a two-stage framework, in which the first stage is to find knowledge facts related to a given question from a wide range of knowledge sources, and then the second stage is to fuse them with the question to predict the answer. @xcite use a tf-idf-based document retriever to locate relevant paragraphs from wikipedia documents, and then use a document reader to predict the start and end positions of the answer span. @xcite introduce an additional module of paragraph selector to remove noised paragraphs that explicitly do not contain an answer. @xcite employ a multi-step reasoner to build a new query, then rerank the paragraphs and spot the answer after reading the top paragraph. although previous experimental results have proved the effectiveness of incorporating additional knowledge into question answering systems @xcite , one critical issue still remains to be addressed. as shown in in order to address the above issue, in this paper, we propose a novel question-answering method over multiple knowledge sources. we argue that it is critically important to engage multiple knowledge sources and establish the precise connection between the required background knowledge and the original question as well as choices, to solve the challenge induced by the multi-value property. we develop our method from three perspectives: 1) we propose a graph-based iterative retrieval module to narrow and refine the potential useful knowledge facts by hidden relations among entities in the question, inspired by @xcite @xcite . 2) different from previous methods only using wikipedia or conceptnet, we adopt another extra knowledge source, namely a dictionary, to provide explanations for entities or concepts. synthesizing entity or concept explanations and iteratively retrieved knowledge facts can help machine precisely distinguish the deceptive answer choices. 3) before feeding hidden representations into a linear classifier for final prediction, we introduce an answer choice-aware attention mechanism to compute attention scores between hidden representations of the given question, retrieved knowledge, and candidate choice, which are encoded through a pre-trained language model. we evaluated our proposed method on the commonsenseqa dataset, and compared it to other baseline methods. the experimental results show that our method achieves the new state-of-the-art with an improvement of 1.2% over unifiedqa @xcite in test accuracy. furthermore, we tested our method using different combinations of external knowledge sources to evaluate the impact of each knowledge source. furthermore, we carried out ablation studies to evaluate the performance impact of the graph-based iterative knowledge retrieval module and the answer choice-aware attention mechanism on the commonsense question answering task. the contributions of this paper are summarized as follows: ## approach in this section, we present the architecture and details of our approach. as shown in figure 1 , our approach consists of four parts: 1) graph-based iterative retrieval module, 2) pre-trained language model based encoding module, 3) answer choice-aware attention mechanism and 4) prediction module. in the graph-based iterative knowledge retrieval module, given a question and its choices, we retrieve relevant knowledge facts from multiple knowledge sources. for brevity, we use evidence or knowledge interchangeably to refer to the retrieved knowledge facts. via the pre-trained language model based encoding module, we encode evidence and its question-answer pair to obtain their hidden representations cls] question [sep] extracted knowledge [sep] choices question stem graph-based iterative retrieval tokenizer & encoder initialization cambridge dictionary wikipedia linear classifier a •••••• e softmax q c a hq hc ha p evidence concatenation with question & choices att att oqa oca hc ha hq answer choice-aware attention prediction data preprocessing pretrained lm ## conclusion in this paper, we deal with the multi-choice question answering task which requires background knowledge or commonsense. we propose a novel question-answering method by exploring how to efficiently integrate multiple knowledge sources, i.e. conceptnet, wikipedia and the cambridge dictionary. firstly, we propose a novel graph-based iterative knowledge retrieval module to iteratively retrieve concepts and entities related to a given question and its choices. in addition, we propose an answer choice-aware attention mechanism to fuse all hidden representations encoded by a pre-trained language model. we conducted experiments on the commonsenseqa dataset and the experimental results show that our method significantly outperforms other competitive methods in accuracy. further ablation studies show the effectiveness of graph-based iterative knowledge retrieval module and answer choice-aware attention module in retrieving and synthesizing background knowledge from multiple knowledge sources. in the future, we will extend our method to deal with the open-domain question answering tasks that require the external background knowledge.