Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference (2021)

Abstract: Recent methods based on pre-trained language models have shown strong supervised performance on commonsense reasoning. However, they rely on expensive data annotation and time-consuming training. Thus, we focus on unsupervised commonsense reasoning. We show the effectiveness of using a common framework, Natural Language Inference (NLI), to solve diverse commonsense reasoning tasks. By leveraging transfer learning from large NLI datasets, and injecting crucial knowledge from commonsense sources such as ATOMIC 2020 and ConceptNet, our method achieved state-of-the-art unsupervised performance on two commonsense reasoning tasks: WinoWhy and CommonsenseQA. Further analysis demonstrated the benefits of multiple categories of knowledge, but problems about quantities and antonyms are still challenging.

## introduction recently, the task of commonsense reasoning has attracted much attention, as believed to be a critical and yet challenging component of human-level intelligence @xcite @xcite . to test models' ability to understand natural language and reason with external commonsense knowledge, efforts have been made towards building many challenging wsclike (winograd schema challenge) tasks and qa (question-answer) tasks. specifically, @xcite crowd-sourced human-provided justifications as reasons for the wsc problems, resulting in a new dataset called winowhy. an example of winowhy is shown in ## method in this section, we describe the details of 1) using the nli framework to solve commonsense reason-ing tasks and 2) extracting knowledge from kbs, and 3) injecting the external knowledge into the nli framework. the overview of our framework is shown in figure 1 . ## related work commonsense reasoning recent commonsense reasoning datasets @xcite @xcite @xcite have motivated research in several domains of commonsense: abductive, temporal, social, and physical. sotas for most of them have achieved over 80% accuracy, which is close to human performance (e.g., @xcite @xcite . however, their success is due to larger pre-trained corpora and much more parameters, which is difficult to be followed for most researchers. in addition, other useful methods @xcite @xcite generally require training on training sets and knowledge graphs. when applying them to different tasks, the same running and tuning process should repeat for several times to find the best fit. thus, we propose a framework to convert diverse commonsense reasoning tasks to a common task, nli, and use a general unsupervised method to solve it. natural language inference since glue regards nli as a benchmark task for testing the natural language understanding capability of the model, nli has been well studied, and language models have achieved performance beyond humans on some nli datasets. furthermore, by leveraging transfer learning from large nli datasets, great performances have been achieved in several tasks, such as story ending prediction @xcite , intent detection @xcite , semantic textual similarity @xcite . therefore, we attempt to use nli as the common task to solve commonsense reasoning. external knowledge most commonsense reasoning tasks require models to synthesize external commonsense knowledge and leverage more sophisticated reasoning mechanisms. the key is to extract effective information from commonsense sources, such as atomic, conceptnet, and wikipedia. methods learn commonsense knowledge either by kgs pre-training @xcite @xcite or by reasoning on knowledge graphs @xcite @xcite . in order to cooperate with our nli framework, we convert the triples in kb to natural language sentences and extract triples by calculating cosine similarity between the embeddings of kb sentence and source task example. ## conclusion in this work, we propose a framework to convert diverse commonsense reasoning tasks to a common task, nli and use a pre-trained language model, roberta to solve it. by leveraging transfer learning from large nli datasets, qnli and mnli, and injecting crucial knowledge from knowledge bases such as atomic and conceptnet, our framework achieved sota unsupervised performance on two commonsense reasoning tasks: winowhy and commonsenseqa. experimental results show that knowledge from qnli and extracted from either atomic or conceptnet can complement each other to enhance the model's performance on commonsense reasoning. more improvements can be obtained by combining multi categories of effective knowledge. further experiment shows that atomic can bring more benefits to roberta compared with conceptnet. however, injected knowledge is not much help to roberta finetuned on mnli and even causes a drag. in addition, models perform worse when facing problems about quantity knowledge and antonym relation. the code is publicly available: https: //github.com/sysuhcm/nli-kb .