Great~Truths~are ~Always ~Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models (2022)

Abstract: Commonsense reasoning in natural language is a desired ability of artificial intelligent systems. For solving complex commonsense reasoning tasks, a typical solution is to enhance pre-trained language models (PTMs) with a knowledge-aware graph neural network (GNN) encoder that models a commonsense knowledge graph (CSKG).Despite the effectiveness, these approaches are built on heavy architectures, and can’t clearly explain how external knowledge resources improve the reasoning capacity of PTMs. Considering this issue, we conduct a deep empirical analysis, and find that it is indeed relation features from CSKGs (but not node features ) that mainly contribute to the performance improvement of PTMs. Based on this finding, we design a simple MLP-based knowledge encoder that utilizes statistical relation paths as features. Extensive experiments conducted on five benchmarks demonstrate the effectiveness of our approach, which also largely reduces the parameters for encoding CSKGs.Our codes and data are publicly available at https://github.com/RUCAIBox/SAFE .

## introduction in the era of artificial intelligence, it is desirable that intelligent systems can be empowered by the capacity of commonsense reasoning in natural language. for this purpose, a surge of commonsense reasoning tasks and datasets are proposed to evaluate and improve such an ability of nlp models, e.g., commonsenseqa @xcite and so-cialiqa @xcite . although large-scale pre-trained models (ptms) @xcite have surpassed human performance in a number of nlp benchmarks, it is still hard for ptms to accurately capture and understand commonsense knowledge for accomplishing complex reasoning tasks @xcite . in order to enhance the reasoning capacity, commonsense knowledge graphs (cskgs) (e.g., con-ceptnet @xcite and atomic @xcite ) have been adopted for injecting external commonsense knowledge into ptms. by conducting entity linking to cskgs, existing methods @xcite aim to capture the structured knowledge semantics via knowledge graph (kg) encoders (e.g., graph neural network (gnn) @xcite ), and then integrate the kg encoders for improving the commonsense reasoning capacity of ptms @xcite . despite the effectiveness, these approaches are built on highly complicated network architectures (involving both ptms and gnns). thus, it is difficult to explain how and why external commonsense knowledge improves the commonsense reasoning capacity of ptms. besides, existing cskgs @xcite are mostly crowdsourced from massive selected resources (e.g., books, encyclopedias, and scraped web corpus), containing a wide variety of content. without a clear understanding of how these external resources should be utilized, it is likely to incorporate irrelevant concepts or even knowledge biases @xcite into ptms, which might hurt the reasoning performance. indeed, some researchers have noted this issue and questioned whether existing gnn-based modules are over-complicated for commonsense reasoning @xcite . furthermore, they find that even a simple graph neural counter can outperform existing gnn modules on common-senseqa and openbookqa benchmarks. however, existing studies can't well answer the fundamental questions about knowledge utilization for commonsense reasoning: how do external knowledge resources enhance the commonsense reasoning capacity of ptms? what is necessarily required from external knowledge resources for ptms? since the simplified knowledge-aware gnn has already yielded performance improvement on the commonsenseqa @xcite , we speculate that there might be a simpler solution if we could identify the essential knowledge for commonsense reasoning. focusing on this issue, we think about designing the solution by further simplifying the kg encoder. based on our empirical analysis, we observe a surprising result that it is indeed relation features from cskgs, but not node features, that are the key to the task of commonsense reasoning (see more details in section 3). according to this finding, we propose a rather simple approach to leveraging external knowledge resources for enhancing the commonsense reasoning capacity of ptms. instead of using a heavy gnn architecture, we design a lightweight kg encoder fully based on the multilayer perceptron (mlp), which utilizes statistical relation path from cskgs as features, namely safe. we find that semantic relation paths can provide useful knowledge evidences for ptms, which is the key information for helping commonsense reasoning. by conducting extensive experiments on five benchmark datasets, our approach achieves superior or competitive performance compared with state-of-the-art methods, especially when training data is limited. besides the performance improvement, our approach largely reduces the parameters for encoding cskgs (fewer than 1% trainable parameters compared to gnn-based kg encoders @xcite ). our main contributions can be summarized as follows: ( 1 ) we empirically find that relation features from cskgs are the key to the task of commonsense reasoning; (2) we design a simple mlpbased architecture with relation paths as features for enhancing the commonsense reasoning capacity of ptms; (3) extensive experiments conducted on five benchmark datasets demonstrate the effectiveness of our proposed approach, which also largely reduces the parameters of the kg encoder. ## task description according to pioneer works @xcite , the commonsense reasoning task can be generally described as a multichoice question answering problem: given a natural language question q and a set of n choices {c 1 , • • • , c n } as the answer candidates, the goal is to select the most proper choice c ⋆ from these can-didates to answer the question based on necessary commonsense knowledge. to explicitly capture commonsense knowledge, external commonsense knowledge graphs (cskgs) have often been utilized in this task, e.g., concept-net @xcite . a cskg can be formally described as a multi-relational graph @xmath0, r, e), where v is the set of all concept (or entity) nodes (e.g., hair and water), r is the set of relation types (e.g., relatedto and atlocation), and following prior studies (lin et al., 2019), we solve the commonsense reasoning task in a knowledge-aware setting, where a cskg g is available as input. we first link the mentioned concepts from the question and the answer candidates to the cskg, so that we can leverage the rich semantic knowledge from the cskg for commonsense reasoning. based on the linked concepts in the question and each answer candidate, we further extract their neighbouring nodes from g and the relational links that connect them, to compose a subgraph g q,c i for characterizing the commonsense knowledge about the question q and the answer candidate c i . ## empirical analysis on the commonsense kg encoder in this section, we conduct an empirical study to investigate how the external kg encoder helps ptms with commonsense reasoning. ## approach the former sections show that the role of the kg encoder on cskgs is to mainly complement ptms in the task of commonsense reasoning. instead of node features, relations features are the key to the kg encoder for improving ptms. based on these findings, we develop a simple commonsense kg encoder based on the statistical relation features from cskgs, namely safe. figure 3 presents the overview of our model. ## related work we review the related studies in two aspects, i.e., commonsense reasoning and kg-enhanced pretrained models. commonsense reasoning. commonsense reasoning tasks aim to evaluate the understanding of commonsense knowledge @xcite , e.g., physical commonsense @xcite , which are mostly formulated as a multichoice qa problem. early studies either rely on explicit text features @xcite to capture the relations between the question and answer candidates, or adopt neural networks (e.g., dnn or lstm) @xcite to model the implicit correlation features. recently, pre-trained models (ptm) @xcite have achieved remarkable performance on commonsense reasoning tasks. furthermore, a surge of works incorporate external knowledge resources to further improve the reasoning performance. among them, cskg (e.g., con-ceptnet @xcite ) has been widely studied, and existing works mainly adopt graph neural networks to learn useful commonsense knowledge from the cskg to enhance ptms. based on these works, we systemically study what is necessarily needed from cskgs for improving ptms. our analysis leads to an important finding that relation features mainly contribute to the performance improvement, and we design a lightweight mlp architecture to simplify the kg encoder. ## conclusion in this work, we study how the external commonsense knowledge graphs (cskgs) are utilized to improve the reasoning capacity of pre-trained models (ptms). our work makes an important contribution to understanding and enhancing the commonsense reasoning capacity of ptms. our results show that relation paths from the cskg are the key to performance improvement. based on this finding, we design a rather simple mlp-based kg encoder with relation paths from the cskg as features, which can be generally integrated with various ptms for commonsense reasoning tasks. such a lightweight kg encoder has significantly fewer than 1% trainable parameters compared to previous gnn-based kg encoders. experimental results on five commonsense reasoning datasets demonstrate the effectiveness of our approach. in future work, we will study how to effectively leverage the commonsense knowledge from large-scale unstructured data to improve ptms. we will also try to apply our approach to other knowledge-intensive tasks, e.g., knowledge graph completion and knowledge graph based question answering @xcite .