Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks (2020)

Abstract: Pretrained language models have excelled at many NLP tasks recently; however, their social intelligence is still unsatisfactory. To enable this, machines need to have a more general understanding of our complicated world and develop the ability to perform commonsense reasoning besides fitting the specific downstream tasks. External commonsense knowledge graphs (KGs), such as ConceptNet, provide rich information about words and their relationships. Thus, towards general commonsense learning, we propose two approaches to implicitly and explicitly infuse such KGs into pretrained language models. We demonstrate our proposed methods perform well on SocialIQA, a social commonsense reasoning task, in both limited and full training data regimes.

## introduction empowering machines with commonsense has become a hot topic recently. past research efforts for this problem include the construction of various data sets and models. several commonsense data sets have been commonly used in past work to develop machines' commonsense capability @xcite @xcite @xcite @xcite . in particular, so-cialiqa @xcite ) is a multiple-choice qa data set for probing machine's emotional and social intelligence in a variety of everyday situations, which is the data set used in this study. to improve the modeling approaches for the socialiqa and other commonsense tasks, @xcite and @xcite focused on zero-shot setting using pretrained language models. @xcite reformulated the multi-choice setup used in most data sets as a generation task and achieved impressive performance by fine-tuning t5 @xcite . recently there is an increasing effort to utilize external knowledge bases to incorporate commonsense information underlying the text @xcite @xcite . while most prior work on socialiqa utilized large pretrained language models @xcite @xcite @xcite , we argue that such a challenging task requires commonsense reasoning of social events, and simply fine-tuning the model to fit the task is insufficient. we believe it would be beneficial if the model can learn from knowledge-rich resources such as conceptnet @xcite , and thus have a broader and deeper understanding of the information not present in the provided context and answer candidates. in this paper, we propose two approaches tailored to large pretrained language models to utilize existing knowledge graph (kgs) for downstream commonsense tasks. the first approach leverages the kgs implicitly by pretraining on the relevant tuples to the socialiqa task, while the second one maintains a dynamic knowledge base during finetuning, utilizing kgs explicitly via an attention mechanism. our experiments demonstrate the effectiveness of both approaches on socialiqa under limited and full training data regimes, and the critical role of relevant knowledge. ## problem formulation and baseline in socialiqa, given a context c of an event and a corresponding question q, the goal is to select the correct choice from the answer set @xmath0, a 2 , a 3 ). an example is shown in figure 1 , with the provided context, question, and three answer candidates. a typical approach (as used in sap et al. (2019b)) for solving this kind of multiple-choice problems with a pretrained transformer-based language model is by concatenating c, q, and a i with a separator token, and then letting the model output a score via a multi-layer perceptron (mlp) built on top of the final hidden representation of the classifier token [cls]. finally, scores for each data point are normalized across all (c, q, a i ) instances with softmax, and cross-entropy loss is applied for model training. since roberta @xcite has shown competitive performance on the socialiqa task, we use it as a baseline model in this study. furthermore, in addition to multiple-choice classification, we perform masked language modeling (mlm) @xcite , masking 15% tokens in the concatenation of c, q, and the correct answer a y * , when fine-tuning on the socialiqa task. ## incorporating commonsense knowledge graph in this section, we introduce two methods to incorporate a given kg into our pretrained model. we experiment with both atomic @xcite and conceptnet @xcite as our kgs. ## conclusion in this paper, we propose two methods to introduce kgs into pretrained language models for commonsense tasks. the first one implicitly infuses relevant knowledge into mlm pretraining, while the second method uses the attention mechanism to allow pretrained language models to explicitly utilize the dynamic query tuples. our experiments on the so-cialiqa task show that leveraging external kgs via attention outperforms the baseline pretrained language models, and the quality of the relevant graphs matters for downstream task performance. our work can be further improved by designing better algorithms for kg retrieval in the future. al-though our experiments have focused on socialiqa with conceptnet and atomic, our method can be generalized to other similar tasks to leverage knowledge graphs.