Team S olomon at S em E val-2020 Task 4: Be Reasonable: Exploiting Large-scale Language Models for Commonsense Reasoning (2020)

Abstract: In this paper, we present our submission for SemEval 2020 Task 4 - Commonsense Validation and Explanation (ComVE). The objective of this task was to develop a system that can differentiate statements that make sense from the ones that don’t. ComVE comprises of three subtasks to challenge and test a system’s capability in understanding commonsense knowledge from various dimensions. Commonsense reasoning is a challenging task in the domain of natural language understanding and systems augmented with it can improve performance in various other tasks such as reading comprehension, and inferencing. We have developed a system that leverages commonsense knowledge from pretrained language models trained on huge corpus such as RoBERTa, GPT2, etc. Our proposed system validates the reasonability of a given statement against the backdrop of commonsense knowledge acquired by these models and generates a logical reason to support its decision. Our system ranked 2nd in subtask C with a BLEU score of 19.3, which by far is the most challenging subtask as it required systems to generate the rationale behind the choice of an unreasonable statement. In subtask A and B, we achieved 96% and 94% accuracy respectively standing at 4th position in both the subtasks.

## introduction in today's digital age, information is shared widely in textual format via e-mails, news articles, social media posts and messages, blogs, internet forums, etc. we are now surrounded by textual information more than ever, which demands a meaningful understanding of a text by machines. machines augmented with commonsense reasoning will be a key step towards achieving this. 'commonsense knowledge' also referred to as the background knowledge, is the understanding of the everyday world and the art of drawing inferences by manipulating the knowledge gathered. humans are rational and have acquired a sense of reasoning by combining facts and beliefs from their day to day life. to an average person reasoning the fact that a person can have a pet dog but not pet dinosaur comes naturally and is fairly straightforward. commonsense knowledge is assumed to be known to all and people typically tend to omit this while communicating with others, which makes it more challenging. current natural language understanding (nlu) systems assisted with semantic representations, statistical methods, and distributional representations have shown better performance than humans on many benchmarks but there is a growing concern that these systems scratch only the surface of the human level of understanding of the world and thus are too shallow. natural language is complex in nature and nlu systems have tried to derive useful meaning by capturing the context i.e. neighboring words and sentences but these systems fail miserably when the context is restricted or omitted. such cases call for systems to delve deeper into understanding the background knowledge enjoyed by all humans. for a commonsense deprived machine, understanding that the sentence 'he put books in his pencil box' is against common sense as 'a book is much bigger than a pencil box' is difficult in the absence of the knowledge about the size of books in comparison with the size of a pencil box. since the existing systems comve comprises of three subtasks, subtask a is a validation task where a system has to choose the unreasonable statement between a pair of given statements. under subtask b the participating system has to pick up the reason which explains the rationale behind the unreasonable statement selected in subtask a. subtask c, which by far is the most challenging task and requires the system to generate the reason explaining the argument behind its choice in the first task. we have illustrated some samples from the dataset for all the three subtasks in under this task, we have developed a system that leverages commonsense knowledge gained by pretrained language models from their huge training corpora. we have used models like bert, roberta, gpt2, etc. our system seeks to utilize the language model's world knowledge and identify commonsense facts in the task-specific dataset with task-centric finetuning. the model with little finetuning and taskspecific modifications such as transforming the input and adding a score comparator achieved significant gains on all the subtasks. our system ranked 2nd foot_0 in subtask c (explanation with generation) with a bleu score of 19.3. in subtask a and b, we achieved 96% and 94% accuracy respectively standing at 4th rank in both the subtasks. this paper is organized as follows, in section 2, we briefly review some of the popular works in the domain of commonsense. in section 3, we describe the task and the dataset. section 4 gives details of our system and individual setup for each subtask. in section 5, we have dicussed the experiments and the results. we conclude the paper in section 6. ## related work in the recent past, there have been several lines of research focussing on commonsense reasoning. multiple tasks and datasets have been proposed which tests machine's intelligence pivoting on commonsense, like the winograd schema challenge @xcite which aims to resolve ambiguities arising out of pronouns, using commonsense. @xcite released the rocstories corpus and introduced a story cloze test, in which a system is given a four-sentence 'context' and two alternative endings to the story, called the 'right ending' and the 'wrong ending'. this task challenges a system to understand the context and predict the correct ending. another dataset, swag @xcite involves predicting the next scene given the current one to evaluate grounded commonsense inference. it's also a multiple-choice dataset with 4 possible continuations to the given description. @xcite finetuned their bert large model on the swag dataset to beat the human performance and achieve state-of-the-art results with 86.3% accuracy on the test set. all of these works involve multiple-choice datasets, where one has to choose the right option without providing any justification as to why the system chose a particular option. this raises a concern in the system's capability in actually understanding the choice made. none of the above work inspects a direct understanding of commonsense by demanding a logical reason for the choice. @xcite released a dataset, which requires a system to choose an unreasonable statement from a given pair and also predict the right reason behind its choice. they also utilized state-of-the-art language models (lm) like bert for sen-making and explanation task which are similar to subtask a and b respectively of comve. however, they have reported a decline in performance on finetuning bert. they achieved the best result with finetuned elmo in the sen-making task and with pretrained bert in the explanation task. @xcite developed a commonsense auto-generated explanation (cage) framework for commonsenseqa task @xcite . cage in its main approach of 'reasoning' finetunes a language model conditioned on the question and all the plausible answer choices. the language model utilized common sense explanations (cos-e) as a referential reason while training. cos-e is a manually constructed dataset comprising of reasons given by users who in turn were provided the question, all answer choices, and the correct label. lm trained via this approach was used to augment a classifier to predict the answer for the multiple-choice question posed initially. they leveraged cos-e to assist in predicting the right answer. we have taken inspiration from their work in our subtask c to generate a logical reason as to why the unreasonable statement is against commonsense. ## task and dataset description the comve was formulated as a three-stage problem, where different subtasks assess a system's understanding of commonsense from a disparate perspective. the first subtask aims to empower the system to differentiate an unreasonable statement from a reasonable one and is proposed as a 'validation' task. the next subtask, 'explanation with multiple-choice' assess the system's capability to choose the right reason behind its choice for a particular statement to be unreasonable in the first subtask. the system's rationality is further tested by subtask c: 'explanation with generation', which expects the system to generate the reason explaining the rationality behind the system's choice of the irrational statement in subtask a. dev data released for the evaluation phase had 997 samples for each subtask and hidden test data on which systems were finally evaluated had 1000 samples without labels. during the final evaluation, subtasks were kicked off sequentially with subtask a being opened first, followed by subtask c, and at the end, subtask b was started. this ensured that the there is no information leakage between the subtasks. more information on the tasks and the dataset can be found in @xcite . ## system description our systems leverage commonsense knowledge from pretrained language models via transfer learning, thus we first briefly discuss the language models used in our system's core in section 4.1. subsequently, we explain the details of our models for each subtask. we have developed systems separately for each subtask which can be combined in the desired manner for an end to end commonsense pipeline. ## conclusion and future work in this work, we have described our system for semeval-2020 task 4 on common sense validation and explanation. the proposed system leverages the background knowledge captured by large-scale transformer-based language models. this paper also discusses various ways of input manipulation in the architecture to improve the system's performance on the downstream tasks. our official submission obtained an accuracy of 96% in subtask a and 94% in subtask b, both securing 4th position on the leaderboard. our system also generates the rationale given an unreasonable sentence with a 19.34 bleu score standing at 2nd rank on the leaderboard for subtask c. in future, we would like to train a joint model by combining systems for subtask a and c. this will provide an additional reasonable statement against the given unreasonable sentence to the system which can improve the reason generation capability. due to the limitation of language models in capturing external knowledge and their training being restricted by the dataset, the systems lagged behind humans in generating coherent reasoning for an unreasonable sentence. however, commonsense augmented systems can be incorporated in chatbots to create a more sensible conversation with a user. also, it can help in detecting satirical articles in online news websites to combat the rising fake news problem.