SWAG ex at S em E val-2020 Task 4: Commonsense Explanation as Next Event Prediction (2020)

Abstract: We describe the system submitted by the SWAGex team to the SemEval-2020 Commonsense Validation and Explanation Task. We use multiple methods on the pre-trained language model BERT (Devlin et al., 2018) for tasks that require the system to recognize sentences against commonsense and justify the reasoning behind this decision. Our best performing model is BERT trained on SWAG and fine-tuned for the task. We investigate the ability to transfer commonsense knowledge from SWAG to SemEval-2020 by training a model for the Explanation task with Next Event Prediction data

## introduction commonsense reasoning has long been a challenge in the field of natural language processing (nlp). recently, the emergence of large pre-trained language models achieving state-of-the-art results in several nlp tasks, such as bert and gpt @xcite has also influenced commonsense tasks to regain popularity. natural language inference (nli) is one of the most popular tasks in commonsense reasoning research. it consists of determining if a hypothesis is correct, given a premise. situations with adversarial generation (swag) task is closely related to nli @xcite . it consists of determining the most appropriate ending given a start phase as a context. the start phrases and endings in swag dataset are video captions. we show an example in figure 1 . the task is to find the ending that describes a possible (future) world that can follow the given sentence even when it is not strictly entailed. making such inference necessitates a rich understanding of everyday physical situations @xcite . figure 1: swag dataset example. the startphrase offers one sentence as context, and a beginning part of the second sentence. the task is to predict its appropriate ending. in this paper, we explore the ability of swag to infer commonsense explanation to tackle semeval 2020 task 4: commonsesense validation and explanation (comve). to this end, we followed the directions given by the organizers of the task @xcite and attempted a contribution to their two main tasks. the first main task is concerned with the commonsense validation of a sentence, which is also called senmaking. it consists of providing two sentences that have similar wordings, one being against commonsense this work is licensed under a creative commons attribution 4.0 international license. license details: http://creativecommons.org/licenses/by/4.0/ . and the other not. the sentences given do not require specialized knowledge and are trivial for humans, but remain challenging for machines. the second main task is commonsense explanation that aims at finding the key reason why a statement does not make sense. while the first task tests the system's performance in choosing which sentence is more likely to be true, the second one takes this a step further by providing us with the decision making process for commonsense reasoning. we attempted several methods that we detail in later sections of the paper to tackle the tasks mentioned above. our main contribution starts with the assumption that our commonsense explanation task is closely related to swag task. we tested this hypothesis with several experiments to confirm the usefulness of training a bert model on swag, then testing it on the current dataset. we achieved nearly a 5.0% improvement using the above method, confirming our assumption that the same type of knowledge could be leveraged from next event prediction in swag task to solve the comve task. for the explanation task, our system currently ranks 14 th with an accuracy score of 84.6%. in post-evaluation, we improved our system and obtained an accuracy of 85.3%, also taking the 11 th place. we also demonstrated the method used to bridge the gap between the two datasets and explored the limitations of this approach with different layouts of data. nevertheless, we encountered difficulties in the validation task, indicating that context sentences contribute highly to the efficacy of the model and questioning the world knowledge that can be encompassed in bert. the extent of this is not explored in detail in this paper, but we consider it an exciting direction for future work. the pre-processing and model code we have used will be publicly available foot_0 . ## background for clarity, we follow the organizers in naming three subtasks. the sen-making or validation task is referred to as subtask a, and its aim is to find which sentence out of two given sentences is against commonsense. the explanation task is to justify why a statement is against commonsense, and it is further divided into two subtasks. in subtask b, the system is given a sentence that is against commonsense he put an elephant in the fridge, along with three possible explanations. the system is expected to pick the explanation that is the most relevant and makes more sense; in this case, the correct answer would be an elephant is much bigger than a fridge. subtask c is a text generation task that is concerned with answering the same question as to the previous subtask; given a statement that is against commonsense, the system should generate a corresponding answer. figure 2 shows an example input for each subtask. figure 2 : semeval-2020 task 4: commonsense validation and explanation (comve). our main contribution targets subtask b. nevertheless; we present our findings for the other subtasks. we trained bert on swag, a dataset of 113k multiple-choice questions about a rich spectrum of grounded situations. we leveraged this knowledge to modify our system for the comve task. to implement this, we used bert with a multiple-choice head on top, and trained it on a modified version of the swag dataset; we changed the number of the given choices and the structure of the question for improved results. we then fine-tuned on our explanation dataset, which consisted of 11k multiple-choice of sentences and possible explanations. for subtask a, we gave our system a fixed question for all instances and two choices between a sentence against commonsense, and a sentence that agrees with commonsense. we used these results to verify our hypothesis that swag and comve datasets share the same type of commonsense knowledge. through subtask a, we also investigated the extent that bert encapsulates the world knowledge. for subtask c, we simply fine-tuned gpt-2, which performed reasonably well. ## system overview our system is based on bert with an initialized multiple-choice classification head on top, which is trained for each of the following subtasks separately. we prepared our data following the format seen in figure 3 . figure 4 shows our model. the original swag dataset instances are mainly comprised of a start phrase split into two sentences, followed by four endings that represent the multiple choices, as shown in figure 1 . an example of a start phrase is "on stage; a woman takes a seat at the piano. she" followed by four candidate endings to the given sentence. only one of the endings is plausible given the context; "nervously sets her fingers on the keys.", as opposed to "sits on a bench as her sister plays with the doll", among others. ## conclusion in this paper, we have explored the ability of bert model to use the knowledge gained from swag dataset to solve commonsense validation and explanation tasks. we found that training on swag without fine-tuning on comve data for subtasks a and b still performed better than other methods, proving that swag and comve task 4 require some similar commonsense knowledge about the world. however, the difference in performance between the original and reserved layouts and a decrease in performance after fine-tuning on both datasets together raise questions about the extent of world knowledge contained in bert model. we believe that injecting external knowledge into bert, such as knowledge bases, could be an exciting addition to such systems and to be explored in future work.