ANA at S em E val-2020 Task 4: MU lti-task lear NI ng for c O mmonsense reaso N ing ( UNION ) (2020)

Abstract: In this paper, we describe our mUlti-task learNIng for cOmmonsense reasoNing (UNION) system submitted for Task C of the SemEval2020 Task 4, which is to generate a reason explaining why a given false statement is non-sensical. However, we found in the early experiments that simple adaptations such as fine-tuning GPT2 often yield dull and non-informative generations (e.g. simple negations). In order to generate more meaningful explanations, we propose UNION, a unified end-to-end framework, to utilize several existing commonsense datasets so that it allows a model to learn more dynamics under the scope of commonsense reasoning. In order to perform model selection efficiently, accurately, and promptly, we also propose a couple of auxiliary automatic evaluation metrics so that we can extensively compare the models from different perspectives. Our submitted system not only results in a good performance in the proposed metrics but also outperforms its competitors with the highest achieved score of 2.10 for human evaluation while remaining a BLEU score of 15.7. Our code is made publicly available.

## introduction common sense reasoning is one of the long-standing problems in natural language understanding. previous work on modeling common sense knowledge deals mainly with indirect tasks such as coreference resolution @xcite , or selecting the plausible situation based on the given subject or scenario @xcite . in this paper, we present our system that we devised to tackle task c, explanation (generation), of the semeval 2020 task 4 -commonsense validation and explanation (comve). given a false or non-sensical statement, the task consists of generating the reason why the given statement does not make sense. we propose a multi-task learning for commonsense reasoning (union). it combines datasets including comve @xcite , openbook @xcite , common sense explanation (cos-e) @xcite and open mind common sense (omcs) @xcite in a multi-task framework. the backbone of union is a large pre-trained language model -gpt2. we compare the proposed system to different baselines and report a significant improvement in bleu score and other evaluation metrics. our proposed model achieves a human evaluation score of 2.10, which ranked first on the final leader board for task c of semeval 2020 task 4. in our initial submission, we used comve, cos-e, and openbook datasets for training. in that case, a bleu score of 15.7 is achieved. pretraining the model with omcs dataset further improved the bleu score by 0.7. in addition, we show some of our generations in the appendix. ## background and related work the common sense reasoning in machines has been one of the most challenging problems, and a major critical missing component of artificial intelligence (ai). it could be helpful in various aspects of day-today life @xcite . for any ai system to generate commonsense reasoning, it needs to understand and build a representation of the given situation @xcite . for instance, let us consider the following statement "you will never find a dog that likes to eat meat." humans may have intuitively built a knowledge graph containing the facts that "dogs are carnivorous" and "a carnivorous is a meat-eater". thus, based on it, they may conclude that the first statement is a false statement. similarly, for a machine to answer questions about general situations or facts, a knowledge graph or formal logic system could be beneficial. early work by john @xcite proposed a system that uses formal logic for commonsense reasoning. thus, the system can induce commonsense reasoning given all the possible axioms about the studied domain or the world. however, it is not feasible to generate all the reasonable axioms about the world. this gave rise to several other logic-based approaches for commonsense reasoning and numerous work with the aim of creating huge logic-based ontologies, e.g., situation calculus @xcite , @xcite , dbpedia @xcite , event2mind @xcite . however, commonsense reasoning over a particular knowledge often acts as a lookup table as it lacks sufficient semantics to formulate a complete sentence as a reason. other models @xcite @xcite have achieved competitive performance in question answering tasks given a comprehension or a document as a source of information. all these models help to simulate common sense reasoning in the language model. however, they still rely on a passage as a source of information, which makes them limited to specific domains. an ai system which processes common sense like humans, in the context of question answering or dialogue generation, should not depend on a source of information each time it produces a response to a query. instead, it should learn and generate a response from the previously learned information. thus, our proposed model does not rely on any other source of information during inference other than its prior learned knowledge. we train union on multiple commonsense datasets comve, cos-e, and omcs. ## proposed systems the language generation task is to model the conditional probability p (y |x), where x and y are both sequences of words. in this work, we use the 36-layer decoder-only transformers based on the architecture of the generative pre-trained transformer (gpt2) @xcite as the backbone of the language generation task for all the presented models. the decoder only transformer architecture is similar to the original decoder transformer architecture proposed by @xcite . the main difference is that in the decoder-only transformer case, the decoder component does not have a multi-head attention for the encoder input. all our models are initialized with the large weights of the pre-trained language model gpt2. in this section, we review the baseline models and the architecture of our proposed union model. ## conclusion in this research, we propose a union model for multi-tasking on a couple of commonsense related datasets, which helped our union model to achieve state-of-the-art in the comve by achieving the highest human evaluation score. in addition, we propose several auxiliary metrics to better evaluate different models for commonsense response generation tasks without human evaluations. in the future, we would like to explore other possible areas of commonsense reasoning like quantitative reasoning, logic puzzles, and visual common sense reasoning.