LMVE at S em E val-2020 Task 4: Commonsense Validation and Explanation Using Pretraining Language Model (2020)

Abstract: This paper introduces our system for commonsense validation and explanation. For Sen-Making task, we use a novel pretraining language model based architecture to pick out one of the two given statements that is againstcommon sense. For Explanation task, we use a hint sentence mechanism to improve the performance greatly. In addition, we propose a subtask level transfer learning to share information between subtasks.

## introduction common sense verification and explanation is an important and challenging task in artificial intelligence and natural language processing. this is a simple task for human beings, because human beings can make full use of external knowledge accumulated in their daily lives. however, common sense verification and reasoning is difficult for machines. according to @xcite , even some state-of-the-art language models such as elmo @xcite and bert @xcite have very poor performance. so it is crucial to integrate the ability of commonsense-aware to natural language understanding model @xcite . semeval-2020 task4 @xcite aims to improve the ability of common sense judgment for model, and we participated in two subtasks of this task. the dataset of semeval-2020 task4 named comve. each instance in comve is composed of 5 sentences s 1 , s 2 , o 1 , o 2 , o 3 . s 1 and s 2 will be used for subtask a, and s 1 or s 2 with o 1 , o 2 , o 3 will be used for subtask b. subtask a(also known as sen-making task) aims to test a model's ability of commonsense validation. specifically, given two statements s 1 , s 2 whose lexical and syntactic are similar, the object of sen-making model is to determine which statement is common sense(compared to another one). for example, s 1 is put the elephant in the refrigerator and s 2 is put the turkey in the refrigerator, a good model needs to judge that the latter is more common sense. subtask b(also known as explanation task) is a multiple choice task that aims to find the key reason why a given statement does not make sense. for example, given a sentence that violates common sense with three options s, o 1 , o 2 , o 3 , where s is he put an elephant into the fridge, o 1 is an elephant is much bigger than a fridge, o 2 is elephants are usually gray while fridges are usually white, and o 3 is an elephant cannot eat a fridge, the model needs to judge that o 1 is the correct option. the official baseline of sen-making use a pretraining language model(plm) to dynamic encoding the two input sentence, and use a simple full connection neural network to calculate the perplexities respectively, and choosing the one with lower scores as the correct one. we believe that the baseline method treats two sentences independently and ignores the inner relationship between the two sentences, so we propose a novel model structure that fully considers the interaction between statements. the official baseline of explanation treats the task as bert-like multiple choice task @xcite . we think that the baseline model doesn't make full use of the input data. so we design a structure to incorporate another statement that is common sense to existing model. in addition, we believe that fine-tuning on similar subtask can improve the performance of the current subtask because there are many commonalities between the two subtasks, so we propose a novel transfer learning mechanism between sen-making and explanation. the proposed system named lmve, it is a neural network model(includes two sub-modules to solve both subtask a and b) bases on large scale pretraining language model. our contributions are as follows: ## system description we consider our model for both sen-making and explanation as two parts: encoder and decoder. encoder is mainly used for getting the contextual representation of input sentence tokens. in recent years, some pretraining language models including bert @xcite , roberta @xcite and albert @xcite have been proven beneficial for many natural language processing (nlp) tasks @xcite . these pretrained models have learned general-purpose language representations on a large amount of unlabeled data, therefore, adapting these models to the downstream tasks can bring a good initialization for parameters and avoid training from scratch @xcite ). so we tried some popular plms as encoders. decoder consists of several simple linear layers whose number of parameters are far less than encoder, and the role of decoder is to fuse the output of encoder and predict the answer. ## conclusions this paper introduces our system for commonsense validation and explanation. for sen-making task, we use a novel pretraining language model based architecture to pick out one of the two given statements that is against common sense. for explanation task, we use a hint sentence mechanism to improve the performance greatly. in addition, we propose a subtask level transfer learning to share information between subtasks. as future work, we plan to integrate the external knowledge base(such as conceptnet foot_6 ) into commonsense inference.