I Know What You Asked: Graph Path Learning using AMR for Commonsense Reasoning (2020)

Abstract: CommonsenseQA is a task in which a correct answer is predicted through commonsense reasoning with pre-defined knowledge. Most previous works have aimed to improve the performance with distributed representation without considering the process of predicting the answer from the semantic representation of the question. To shed light upon the semantic interpretation of the question, we propose an AMR-ConceptNet-Pruned (ACP) graph. The ACP graph is pruned from a full integrated graph encompassing Abstract Meaning Representation (AMR) graph generated from input questions and an external commonsense knowledge graph, ConceptNet (CN). Then the ACP graph is exploited to interpret the reasoning path as well as to predict the correct answer on the CommonsenseQA task. This paper presents the manner in which the commonsense reasoning process can be interpreted with the relations and concepts provided by the ACP graph. Moreover, ACP-based models are shown to outperform the baselines.

## introduction commonsense is the knowledge shared by the majority of people in society and acquired naturally in everyday life. commonsense reasoning is the process of logical inference by using commonsense information. commonsense to answer the questions that is "blowfish requires what specific thing to live?" in figure 1 is depicted as: "blowfish is fish", "fish lives in the water", and "water includes seas and rivers." an enormous amount of pre-defined commonsense knowledge is available and people can make inferences using this commonsense such as in the following example: "blowfish is fish." → "fish lives in the water." → "water includes seas and rivers." ⇒ "blowfish lives in the sea." this chain of commonsense reasoning is naturally deduced by humans without substantial difficulty. whereas people acquire commonsense in their lives, machines cannot learn this knowledge without any assistance. a large amount of external knowledge and several reasoning steps are required for machines to learn commonsense. in recent years, various datasets @xcite @xcite have been constructed to enable machines to reason commonsense. commonsenseqa @xcite is one of the most widely researched datasets and is presented in figure 1 @xcite . the studies of commonsense reasoning based on this dataset can be categorized into two mainstream approaches. the first approach uses pre-trained language models with distributed representations, which exhibit high performances on most natural language processing (nlp) tasks. however, despite their high performance, these models must be trained with an excessive number of parameters and cannot explain the process of commonsense reasoning. the second approach is reasoning with a commonsense knowledge graph. the generally used commonsense knowledge graph is conceptnet 5.5 @xcite , which includes parsed representation from open mind commonsense (omcs) and other different language sources such as wordnet @xcite or dbpedia @xcite . in this approach, the subgraph of conceptnet corresponding to the questions are transformed into node embeddings by the graph encoder. the candidate with the highest attention score is selected as an answer that is computed between the node embeddings and the word vectors from the language models. to learn the commonsense knowledge that is not observed or understood by the language models, relations from conceptnet serve as a critical role in this method. the performance is improved by utilizing the relations that are not represented in the text; however, the interpretation of the question is still not enough. unlike commonsenseqa, the most commonly used method of solving this problem is knowledge-based question-answering (kbqa) @xcite @xcite employing semantic representations. as this method infers the answer with the logical structure of the question using the knowledge base, the question-answering process can be explained in a logical form. in our work, abstract meaning representation (amr) @xcite , which is one of the logical structure, is used to understand the overall reasoning process, from the question to the answer. amr is a graph for meaning representation that symbolizes the meaning of sentences. amr illustrates "who is doing what to whom" that is implied in a sentence with a graph. the components of these graphs are not the words, but rather the concepts and their relations. each concept denotes an event or an entity, and each relation represents the semantic role of the concepts. in this paper, we enable the language models to exploit the amr graph to understand the logical structure of sentences. however, it is difficult to infer commonsense information with only an amr graph, owing to its deficiency of commonsense knowledge of the given sentence. for example, in figure 1 (b), the amr graph indicates the path of the logical structure of the sentence "what the blowfish requires to live?" (require-01 → @xmath0; in other words, these paths from the single amr graph lack the proficient information to predict the right answer. therefore, for commonsense reasoning, dynamic interactions between the amr graph and conceptnet are inevitable to reach the correct answer. thus, we propose a new compact amr graph expanded with the conceptnet's commonsense relations with pruning, and it is called acp graph. the proposed method can interpret the path from the question to the answer by performing commonsense reasoning within the connected graph, such as "blowfish needs the sea to live." (require-01 → @xmath1. the contributions of our study are as follows. amr graph conceptnet graph graph path learning module amr-cn graph language model encoder question question answer relation-enhanced global attention + path relation encoder node @xmath2. in section 2, we present the entire process of our method in detail. the experimental setup and results are explained in section 3. a discussion of the proposed model is provided in section 4, and section 5 presents the conclusions. appendix a provides related works including conceptnet, previous works on commonsense reasoning, and amr. ## proposed method we propose a commonsense reasoning framework that uses a commonsense knowledge base on the basis of the amr logic structure. our framework consists of the amr graph integrating and pruning module, language model encoder, and graph path learning module § . as illustrated in figure 2 , we first generate the amr graph from every question in the commonsenseqa dataset and integrate all the nodes of amr with conceptnet graphs. as this amr-conceptnet full graph also includes some irrelevant relations to the question, interpreting questions can be guided in the wrong way. for this reason, we suggest a new method, the acp graph, pruned according to the relation type. thereafter, the graph path learning module takes the pruned graph as an input and computes the attention score of each path by using the graph transformer @xcite which results in the whole graph vector. the graph vector is finally fed into the transformer @xcite to model the interactions between the amr and conceptnet graph and transforms to the final graph representation. meanwhile, the question and candidate answer from the dataset are passed through the language model encoder, producing the language vector. the concatenation of the language and graph vectors turns out to the final representation that is used to predict the correct answer. in contrast to other models mentioned in @xcite , which cannot provide interpretable reasons for predicting the correct answer from the question, our proposed method produces the reasoning paths that make the model transparent and interpretable. that is, the reasoning paths that have high attention weights from the graph encoder possess potentially accurate information for reasoning. these reasoning paths are depicted in figure 5 . (home-usedfor-living). then remove the conceptnet nodes that are not connected to the amr nodes which have arg0 or arg1 relations. in this example, the deleted nodes are living, house, and game. in addition, as the require-01 is frame node which does not need to expand, need is also removed from the graph. ## experiments to show the effectiveness of representing a question using the proposed acp graph in commonsense reasoning, we conduct four different experiments. we first compare the acp with the acf graph, which is expanded on all the concepts of the amr graph, and with the graph that only utilizes the conceptnet. in addition, we apply our model to three language models that have different encoder structures, showing performance enhancement as shown in ## conclusions and future works we introduce a new commonsense reasoning method, using the proposed acp graph. this method outperformed the model that simply learns the conceptnet graph. furthermore, our method can explain the answer-inference process by interpreting the logical structure of the sentences within commonsense reasoning process. models that applied our method exhibit higher performance compared to the previous models. however, certain problems still remain. though the relations arg0 and arg1 occupy most of the core roles in the amr graph, it is still arguable that the other choice of relations may lead to better results. therefore, we will show the experimental results according to the different pruning rules on the commonsenseqa task in the future. also, we plan to develop an end-to-end learning model that incorporates the amr generation model and the question-answering model to reduce the error propagation from the amr generation.