Q iao N ing at S em E val-2020 Task 4: Commonsense Validation and Explanation System Based on Ensemble of Language Model (2020)

Abstract: The ability of common sense validation and explanation is very important for most models. Most obviously, this will directly affect the rationality of the generated model output. The large amount and diversity of common sense poses great challenges to this task. In addition, many common sense expressions are obscure, thus we need to understand the meaning contained in the vocabulary in order to judge correctly, which further increases the model’s requirements for the accuracy of word representation. The current neural network models are often data-driven, while the annotated data is often limited and requires a lot of manual labeling. In such case, we proposed transfer learning to handle this challenge. From our experiments, we can draw the following three main conclusions: a) Neural language model fully qualified for commonsense validation and explanation. We attribute this to the powerful word and sentence representation capabilities of language models. b) The inconsistency of task of pre-training and fine-tuning will badly hurt the performance. c) A larger amount of corpus and more parameters will enhance the common sense of the model. At the same time, the content of the corpus is equally important.

## introduction this task @xcite is to evaluate how well a model can do for sense making on english data set by its ability to judge whether natural language sentences are not in line with common sense and explain the reasons. to thoroughly evaluate the model, three subtasks are designed in @xcite ): a) subtask a validation -given two statements with similar structures, the task is to discern which statement relatively makes sense and which one does not. b) subtask b explanation(multiple-choices) -from the three candidate reasons given, choose the reason that is most likely to explain a statement that doesn't make sense. c) subtask c explanation(generation) -generating explanations for non-common sense statements. natural language understanding (nlu) has drawn an increasing amount of research attention recently. although some well-performed end-to-end model even show better performances than humans on some benchmarks, they are still far from human on common sense. if a model lack of common sense processes data that requires common sense, the performance of the model may drop dramatically, which is a sign for poor robustness. however, building models that are robust to adversarial attacks is essential @xcite . therefore, the importance of commonsense reasoning is not only confined to nlu system, it should also be expanded to other systems. many similar tasks have been studied and solved before this task was proposed. previous attempts on solving common sense challenge usually involve heavy utilization of annotated knowledge bases @xcite , rule-based reasoning @xcite , or hand-crafted features @xcite . over time, the advantages of the knowledge base gradually emerged from other methods, and researchers are more inclined to use this method to solve such problems. simple models based on distributional representations perform poorly on this task, despite doing well on selection preference, but injecting manually elicited knowledge about entity properties provides a substantial performance boost @xcite . however, using annotated knowledge bases is a expensive approach while unsupervised training is done on text corpora which can be cheaply curated. nowadays, neural lms having achieved great success being used as feature representations for a sentence, or a paragraph, which improves nlp applications in a large scale such as question answering, sentiment analysis, machine translation, etc. the combined evidence suggests that lms trained on a extensive amount of unlabeled data can capture many aspects of natural language and the world's knowledge, especially commonsense information. the success of previous work of using language models to solve winograd schemas challenge further confirms the inferences above. by looking at probability ratio at every word position. @xcite used lm to score multiple choice questions posed by commonsense reasoning tests. this work inspired us to make use of the capability of capturing commonsense information. whereas unsupervised approach doesn't need expensive knowledge bases and success of previous lm models, we present a transformer-based approach where a pretrained language model has been fine-tuned on data set of this task. we voted and weighted the results of different models and the same model with different training parameters. finally, the ensembled model using the weighting method works best. it is found that the results of some models are similar, while others are different. this is related to the structure of different models, the training data and training time, etc. therefore, we believe the performance can be improved by ensemble learning for only complementary models can correct each other to get better results. the ensembled model achieves great result, ranking in official test evaluation 6th and 11th place in subtasks a and b, respectively. our best model achieved in official test evaluation accuracy of 95.9% for subtask a and 90.8% for subtask b. in addition, we also made a comprehensive analysis of the data of task a. we found that data can be classified by the structure of the sentence since the data set of task a can be divided into 3 categories, and the details will be explained in section 3 later. our model does not deal with different types of data with different approaches separately, but the future model proposed for these characteristics may better solve the task. ## models we trained a large variety of different models and combined the best of them in ensembles 1 . for every lm model, we just add a feed forward layer and a softmax layer to the end of it without change its original structure. voting weights and evaluation scores of different models are related. the weight sum equation is defined as: ## data commonsense validation and explanation dataset (comve) is released by the organizers of this task. each instance in subtask a consists of 2 similar sentences, but only one of them makes sense while the other does not. as for the subtask b, every invalid sentence are followed by three sentence as the optional type choice a choice b label sample a) the sky is blue the sky is underground 1 sample b) the man fed the snake a mouse the man fed the mouse a snake 1 sample c) ## experiments and results the evaluation metric of first two subtasks is accuracy and we use blue to evaluate subtask c. in each experiment, model is trained on 1080ti, and batch size is set as big as possible to ensure rapid convergence of the model. due to labels of test dataset are kept by organizers, we tune the hyperparameters on development dataset. because different language models use different corpora, hyperparameters, and model structures during pre-training, we believe that these differences will affect the performance of the model. in order to more fully reflect the ability of each model on this task, we did not introduce additional training data when fine-tuning, retaining the data provided by the organizer. we only participate in first two subtasks. for subtask a, we input a sentence pair at a time, the output is the label that indicating which sentence is invalid. for subtask b, a multiple choices problem , we concatenate the invalid statement with each optional reason as inputs. then the scores of each sentence-reason pair are fed into softmax layer to get final output. finally, we weight each model's result according to evaluation score on developing model, make the result less than 0.5 as 0, otherwise as 1. in tables 2a and 2b , we report the best results of each model of final test dataset. when fine-tuning different types of bert, we found that the case-insensitive model is better than the case-sensitive model, and the larger model (bert large) will perform better. one surprising observation from the table shows the performance of xlnet is far behind the bert's, let alone other improved models based on bert. we attribute this phenomenon to the way xlnet training which does not include nsp-like tasks. although xlnet surpasses bert on many benchmarks, this sign indicates that it still lacks the ability to understand the relationship between sentences. in contrast, albert's accuracy is among the best. we think it is because albert replaced nsp with sop as a pre-training task, which forces the model to learn finer-grained distinctions about discourse-level coherence properties. in addition, when we look at some of our ensemble model predictions that range from 0.4 to 0.6, which means the feasibility of the results is extremely low. the reason for the ambiguous result is that the predictions given by bert and roberta on these samples are consistent and completely opposite to the predictions of albert. we replaced the ambiguous predictions with predictions of each individual model to find out which model can better solve these samples. so we got the result of the obviously, albert has insufficient ability to deal with these problems. surprisingly, when the overall performance of albert and xlnet differ by 13.7%, xlnet still has a higher accuracy on this type of problem. therefore, we dig into the predictions of each model to find the cause of this phenomenon. we found that for two-thirds of the samples our models give exactly the same predictions. these confident samples almost belong to type a), and the correct sentences are often found in the article, on the contrary, the wrong sentences will hardly appear. as the prediction divergence of the model becomes larger, the number of sentences of type b) and type c) gradually increases, but this is not the most obvious change. the most important thing is that the differences between the options of these samples become less obvious. in other words, all the options may not have similar sentences in the corpus. the key to judging these problems is often related to the hidden characteristics of certain words, so this requires our model to understand the comprehensive meaning of each word. compared with other models, albert has a similar model size, more difficult training tasks and even more data volume, but the performance of the implicit word meaning is not good. it is noteworthy that in order to reduce the amount of parameters, which will be a significant parameter reducing when @xmath0, albert reduces the dimension of word embedding. excluding the above-mentioned factors that affect the performance of the model, we think this is because the width of embedding word of albert is at least 6 times smaller than other models that can't fully express the meaning of words. finally, the ensembled model achieves better result, ranking in official test evaluation 6th and 11th place in subtasks a and b, respectively. ## conclusion the ability of common sense validation and explanation is very important for most models. most obviously, this will directly affect the rationality of the generated model output. the large amount and diversity of common sense poses great challenges to this task. in addition, many common sense expressions are obscure, thus we need to understand the meaning contained in the vocabulary in order to judge correctly, which further increases the model's requirements for the accuracy of word representation. the current neural network models are often data-driven, while the annotated data is often limited and requires a lot of manual labeling. in such case, we proposed transfer learning to handle this challenge. from our experiments, we can draw the following three main conclusions: a) neural language model fully qualified for commonsense validation and explanation. we attribute this to the powerful word and sentence representation capabilities of language models. b) the inconsistency of task of pre-training and fine-tuning will badly hurt the performance. c) a larger amount of corpus and more parameters will enhance the common sense of the model. at the same time, the content of the corpus is equally important.