JBNU at S em E val-2020 Task 4: BERT and U ni LM for Commonsense Validation and Explanation (2020)

Abstract: This paper presents our contributions to the SemEval-2020 Task 4 Commonsense Validation and Explanation (ComVE) and includes the experimental results of the two Subtasks B and C of the SemEval-2020 Task 4. Our systems rely on pre-trained language models, i.e., BERT (including its variants) and UniLM, and rank 10th and 7th among 27 and 17 systems on Subtasks B and C, respectively. We analyze the commonsense ability of the existing pretrained language models by testing them on the SemEval-2020 Task 4 ComVE dataset, specifically for Subtasks B and C, the explanation subtasks with multi-choice and sentence generation, respectively.

## introduction semeval-2020 task 4 aims to evaluate whether a system identifies and rationalizes a given natural language statement to be comprehensible under commonsense knowledge @xcite . starting from the pilot study @xcite , semeval-2020 task 4 consists of three subtasks: 1) subtask a: differentiating statements that make sense from those that do not, 2) subtask b and c: selecting a reason or explaining why the statement does not make sense. this paper presents an overview of our systems that were examined for subtasks b and c, as well as the final results. recently, inspired from the success of elmo @xcite , gpt @xcite , and bert @xcite which demonstrated considerable improvements on various language understanding tasks, including the glue benchmark @xcite and squad @xcite , significant studies have been conducted on pretrained language models, including roberta and xlnet, which efficiently reduce the training and inference costs @xcite , transformer-xl, which extends to large contexts @xcite , xlnet @xcite , mass and unilm, which support pretrained language models for generation tasks @xcite , distillbert for lightweight pretrained models @xcite , analyzing bert's syntactic knowledge @xcite , knowledge-enhanced language models @xcite , cross-lingual pretrained language models (conneau and lample, 2019), few-shot learning using language models @xcite , scaling up pretrained language models @xcite and the retrieval-based language model using dense retrieval on the external corpus @xcite . a critical review of the pretrained language models that include classical neural language models @xcite has been presented in @xcite . given the era of pretrained language models, the main design goal of our system on comve is to explore the effect of using pretrained language models on subtasks b and c, both for the understanding and generation tasks. in the top-level design, bert is selected for subtask b, and unilm, a generalized pretrained language model that supports both understanding and generation capabilities, is employed for subtask c. our system architectures for subtasks b and c are summarized as follows. this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . the remainder of this paper is organized as follows: section 2 briefly summarizes the data description for the semeval-2020 comve task. section 3 presents the details of our system architecture, section 4 provides the preliminary and official experimental results, while our concluding remarks and a description of the future work are presented in section 5. ## data description for semeval-2020 comve each instance in the dataset for explanation subtasks of comve is composed of seven @xmath0, o 1 , o 2 , o 3 , r 1 , r 2 , r 3 >. statement s, which does not make sense, is given as an input sentence, commonly to subtasks b and c. for subtask b, o 1 , o 2 and o 3 are the three optional sentences to explain why the statement does not make sense; the only a single sentence o i is marked as a correct (positive) reason and the others as negative ones. for subtask c, r 1 , r 2 , and r 3 , are the additional sentences for referential reasons and, are used for training and evaluation. during the preliminary experiment for subtask b, we found that some of the topics included only two optional sentences foot_1 . thus, we excluded those topics from the dataset. in addition, we attached a punctuation mark to the sentences that did not originally end with any punctuation marks. ## system description this section presents detailed descriptions of our system and the methods that used for subtasks b and c at semeval-2020 comve. ## experimental results we use the official release dataset of semeval task 4 for the experiments. the dataset is split into train/trial/dev/test sets, and, we use the dev (development) set to obtain the model with the best performance. ## conclusion we participated in subtasks b and c at the semeval-2020 comve. our system was based on bert and unilm for subtasks b and c, respectively. for subtask b, we explored the effects of the three variants of bert-style models to study their commonsense reasoning ability. finally, our submitted run for subtask b ranked 10th out of the 27 submissions. our results showed that the model capacity of bert is highly related to the task accuracy, suggesting that bert encodes the commonsense knowledge, but more in larger models. this implies that we should scale up the current models by exploring significantly larger models such as t5 @xcite , @xcite , or retrieval-based commonsense reasoning motivated by @xcite for subtask c, the unilm-based model ranked 7th out of the 17 submissions and 6th on the human score. we found that there still exists a large divergence between our results and human-level commonsense reasoning. most of the sentences generated from the model were considerably different from the answers on the development dataset. although we did not compare the other models on subtask c, we expect that model capacity would be important here. in future work, we plan to explore large pretrained models to enrich the commonsense knowledge in neural models under closed book @xcite or open book @xcite settings. further, we would like to incorporate pretrained language models with external knowledge, such as conceptnet @xcite .