Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering (2020)

Abstract: Commonsense question answering (QA) requires background knowledge which is not explicitly stated in a given context. Prior works use commonsense knowledge graphs (KGs) to obtain this knowledge for reasoning. However, relying entirely on these KGs may not suffice, considering their limited coverage and the contextual dependence of their knowledge. In this paper, we augment a general commonsense QA framework with a knowledgeable path generator. By extrapolating over existing paths in a KG with a state-of-the-art language model, our generator learns to connect a pair of entities in text with a dynamic, and potentially novel, multi-hop relational path. Such paths can provide structured evidence for solving commonsense questions without fine-tuning the path generator. Experiments on two datasets show the superiority of our method over previous works which fully rely on knowledge from KGs (with up to 6% improvement in accuracy), across various amounts of training data. Further evaluation suggests that the generated paths are typically interpretable, novel, and relevant to the task.

## introduction solving commonsense qa tasks requires filling gaps with external knowledge. for instance, given the multiple-choice question in figure 1 , a system needs to know that fungus grows in moist environments, such as caves, and that a cave is a type of geological feature. such commonsense knowledge is obvious for humans but most existing qa systems do not have it or cannot reason with it. although recent advances in pre-trained language models (lms) have resulted in impressive performance on commonsense-related benchmarks (zellers et al., 2018; bhagavatula et al., 2019; 1 the code is available at https://github.com/ wangpf3/commonsense-path-generator . @xcite , it is unclear whether this is due to commonsense reasoning or to capturing spurious correlations in the data @xcite . pre-trained lms may answer a question correctly for wrong reasons, making them highly uninterpretable @xcite . alternatively, a set of systems retrieve external knowledge either from large text corpora or knowledge graphs (kgs). a corpus, however, might not be an ideal source of commonsense knowledge, as such knowledge is seldom stated explicitly in text @xcite . in contrast, commonsense kgs, like conceptnet @xcite and atomic @xcite , provide structured evidence about the relevant entities, thus enabling effective reasoning and higher interpretability. existing systems retrieve knowledge from a kg in the form of: triplets @xcite , multihop paths @xcite , or subgraphs @xcite . despite the aforementioned benefits, exploiting these kgs poses the following challenges. firstly, as kgs are known to suffer from sparsity @xcite , they might not contain the knowledge needed to fill the gaps between the question and the answer. for example, a missing link (cave, isa, geological feature) in figure 1 might prevent the qa system from choosing the correct answer. recent work on commonsense kg completion @xcite @xcite is limited to predicting the tail of a statement with known head and relation, or a single-hop relation between entities. secondly, due to the large size and heterogeneity of modern kgs, contextualization-i.e., identifying a set of kg facts which are relevant or needed to answer a question-is also difficult @xcite . simply retrieving all paths could introduce noisy information and potentially harm reasoning. to address this gap between lms and kgs, we propose a knowledgeable path generator (pg) that generalizes over the facts stored in a kg, rather than only retrieving them. we call our method neural kg due to its neural generalization over structured kgs, and, in contrast, we use the term static kg for methods which rely exclusively on existing facts in a kg. our pg connects a pair of question and answer entities with a (novel) multi-hop path, which may not exist in the kg, allowing for missing facts like (cave, isa, geological feature) in figure 1 to be considered during inference. to learn such a generator, we: (1) sample a set of random walk instances from a static commonsense kg based on rules and constraints for informativeness and relevance ( §3.1); (2) fine-tune a pretrained language model -gpt-2 @xcite on the sampled paths ( §3.2). by doing so, we transfer the rich knowledge encoded in gpt-2 to our pg. this is expected to both enhance the generalization ability of the pg and combat the sparsity of kgs. also, by generating high-quality missing links between the question and answer entities, we contextualize the task with relevant commonsense knowledge. to understand the impact of our multihop pg on downstream commonsense qa tasks, we integrate the pg in an augmented version of a general qa framework ( §3.3). we run experiments on two benchmark datasets commonsenseqa @xcite and open-bookqa @xcite . the results show that out method performs better than previous systems augmented with static kgs by up to 6% in accuracy, which also reveals its potential as a plug-in module for various datasets and as a vital complement to existing kg structures. in the low-resource setting, the accuracy gain over the baselines grows as the training data decreases, indicating a larger inductive bias of our generator. we also assess the quality and interpretability of our paths through both automatic and human evaluation. to summarize, our key contributions are: ## preliminaries our multiple-choice commonsense qa setup follows prior work @xcite @xcite : given a question q, a system selects exactly one of the choices a as an answer. to experiment with contextualized background knowledge, we adopt a general framework (figure 2 ) consisting of a context module, a knowledge module and a reasoning module. the context module encodes both the question q and a choice a as unstructured evidence, while the knowledge module encodes external facts as structured evidence. both the unstructured and the structured evidence are fed to the reasoning module, which produces a score for a question-choice pair. the choice with a highest score would be the predicted answer. next, we introduce each module in detail. ## knowledgeable path generator extracting the structured evidence by retrieving paths (or subgraphs) from a static kg, as in prior work @xcite @xcite , faces two key challenges: sparsity and contextualization ( §1). we thus propose a knowledgeable path generator (pg), which learns to connect a question-choice entity pair (e q , e a ) with a multi-hop path. the generated paths are used as structured evidence in the knowledge module. next, we detail the construction of training data ( §3.1), the learning of our path generator over this data ( §3.2), and the integration of the generator into the reasoning module ( §3.3). figure 3 presents an overview of our adapted knowledge module. ## conclusion in this paper, we propose a generator of multi-hop knowledge paths, which provides structured evidence for answering commonsense questions. the generator, learned by fine-tuning gpt-2 on random walks sampled from conceptnet, produces a path between each pair of question and answer entities. all generated paths are aggregated into a knowledge embedding and fused with a context embedding given by a text encoder for classification. our qa framework enhanced with this generator outperformes both pre-trained language models and prior kg-augmented methods on two commonsense qa benchmarks. the accuracy gain increases with less training data. furthermore, automatic-and human-based evaluations of the generated paths yield high scores for their validity, novelty, and relevance. future research should investigate how to optimally fuse the knowledge and the context embeddings. it should also address the ambiguity of the entity mentions in the questions, the answers, and the lexical nodes in conceptnet. the dataset split used in @xcite is also available by request and we have included it as a supplementary material.