Relation-aware Bidirectional Path Reasoning for Commonsense Question Answering (2021)

Abstract: Commonsense Question Answering is an important natural language processing (NLP) task that aims to predict the correct answer to a question through commonsense reasoning. Previous studies utilize pre-trained models on large-scale corpora such as BERT, or perform reasoning on knowledge graphs. However, these methods do not explicitly model the relations that connect entities, which are informational and can be used to enhance reasoning. To address this issue, we propose a relation-aware reasoning method. Our method uses a relation-aware graph neural network to capture the rich contextual information from both entities and relations. Compared with methods that use fixed relation embeddings from pre-trained models, our model dynamically updates relations with contextual information from a multi-source subgraph, built from multiple external knowledge sources. The enhanced representations of relations are then fed to a bidirectional reasoning module. A bidirectional attention mechanism is applied between the question sequence and the paths that connect entities, which provides us with transparent interpretability. Experimental results on the CommonsenseQA dataset illustrate that our method results in significant improvements over the baselines while also providing clear reasoning paths.

## introduction commonsense question answering (cqa) is a task that requires machines to not only understand the question, but also infer through external knowledge. for example, to answer the question in figure 1 ("what do people typically do while playing guitar?"), it is necessary to discover the connection between the evidence "playing guitar" and "singing". although seemingly trivial for humans, knowledge like this is difficult to learn for natural language understanding (nlu) models. pre-trained language models (ptlms) with distributed representations, such as bert, roberta, xlnet and electra @xcite @xcite , have performed well on many natural language processing (nlp) tasks. however, when answering commonsense questions, these models fail to explicitly recover the reasoning process. specifically, for the question in figure 1 , humans figure out the correct answer by inferring from the relation between "playing guitar" and "singing", while ptlms only provide a predicted answer and can not explain the inference process. they are thus lacking in transparency and interpretability. apparently, there is still a wide gap between the ptlms and human level performance in cqa tasks. since merely relying on pre-trained large language models can not utilize commonsense knowledge for reasoning, a straightforward solution is to model the paths @xcite in the knowledge graph "that connect pairs of concepts". @xcite encodes the paths extracted from the knowledge graph, and further provides interpretability through the attention mechanism. @xcite extract evidence from heterogeneous external knowledge bases and reason based on the graph neural network. these methods have managed to help the model learn background knowledge, especially the entities and structures in the knowledge graph. but they either completely ig-nore relations or learn simple representations of relations through pre-trained models(transe @xcite ). these methods keep the pre-trained relation representations unchanged when training their question answering model. and hence their models do not fully learn the background knowledge. relations are an integral part of knowledge and should not be ignored. therefore, enabling the model to learn relations helps to fully utilize the background knowledge in the cqa task. we propose a relation-aware bidirectional reasoning framework rab-pr foot_0 for cqa. the framework first extracts evidence from both knowledge graphs and wikipedia to form a multi-source subgraph. the graph is then integrated into a relationaware graph convolutional network to jointly learn representations of nodes and relations. furthermore, we introduce a new attention mechanism between the paths and the question to selectively derive salient information for final predictions. the relation-aware graph convolutional networks on multi-source subgraphs enables our model to comprehensively capture the rich evidence contained in external knowledge bases (especially the relations), and the attention mechanism brings better interpretability for inference. the experiments on the commonsenseqa dataset indicate that our method achieves superior performances over the baseline. our contributions of this paper can be summarized as follows: the rest of this paper is organized as follows. in section 2, we give a brief review of the related work. in section 3, we first provide a formal definition of commonsense question answering, and then briefly introduce the framework of our method. in section 4, we introduce the relation-aware bidirectional path learning method in detail. afterwards, section 5 presents the experiments on the com-monsenseqa dataset with analysis. finally, the conclusion and future work are given in section 6. ## related work our work is related to commonsense reasoning and graph neural networks. below, we introduce the representative work and how our work differs from theirs. ## overview in this section, we start with the problem statement. then we introduce the overall process of our proposed method. ## proposed method in this section, we detail the four components of our model: subgraph construction, relation-aware graph neural network, bidirectional path reasoning, and the attention module. ## conclusion in this work, we propose a path-based reasoning framework that aims at obtaining high performance on cqa tasks while providing explicit relation paths for modeling behavior interpretation. our framework introduces a relation-aware graph neural network to jointly embedding both nodes and relations, which enriches the evidence in the subgraph. the bidirectional reasoning module models the bidirectional reasoning process based on the rich evidence, and the attention mechanism efficiently augments the joint representations of path and question text. in addition, the attention mechanism enables our model to restore the reasoning path. evaluation results show the proposed method is effective and achieves competitive results. in the future, we plan to introduce reinforcement learning and semantic information into the framework, and try to transfer the framework to multimodal commonsense reasoning tasks.