JUST ers at S em E val-2020 Task 4: Evaluating Transformer Models against Commonsense Validation and Explanation (2020)

Abstract: In this paper, we describe our team’s (JUSTers) effort in the Commonsense Validation and Explanation (ComVE) task, which is part of SemEval2020. We evaluate five pre-trained Transformer-based language models with various sizes against the three proposed subtasks. For the first two subtasks, the best accuracy levels achieved by our models are 92.90% and 92.30%, respectively, placing our team in the 12th and 9th places, respectively. As for the last subtask, our models reach 16.10 BLEU score and 1.94 human evaluation score placing our team in the 5th and 3rd places according to these two metrics, respectively. The latter is only 0.16 away from the 1st place human evaluation score.

## introduction addressing the issue of commonsense understanding using deep learning algorithms and models has attracted increasing attention from the research community. building a natural language processing (nlp) system that can understand both explicit and implicit knowledge, validate it and correct it before extracting important information is a task that is both hard and complex. at the same time, this task is essential to a large number of tasks like language modeling, word sense disambiguation and sentiment analysis @xcite . approaches like @xcite leverage knowledge bases like wordnet @xcite and dbpedia. such knowledge bases provide a rich source of high quality, human-curated knowledge to fine-tune the internal hidden states of large language models like bert @xcite . other approaches leverage external knowledge to enhance classification, e.g., sentic lstm @xcite tackled the problem of targeted aspect-based sentiment analysis by embedding the knowledge from senticnet @xcite into an attentive long short-term memory (lstm) network. the commonsense validation and explanation (comve) task @xcite at semeval2020 was proposed based on a dataset built by @xcite to evaluate deep learning algorithms and models against commonsense tasks. the general purpose of the dataset is to test whether an nlp system can differentiate statements that make sense from those that do not. it consists of three subtasks: statement 1: he put a turkey into the fridge. correct statement 2: he put an elephant into the fridge. statement: he put an elephant into the fridge. options: -a: an elephant is much bigger than a fridge. correct -b: elephants are usually white while fridges are usually white. the generated text is evaluated against three correct reference justifications. for example: statement: he put an elephant into the fridge. referential reasons: -an elephant is much bigger than a fridge. the evaluation metric for subtasks a and b is accuracy. as for subtask c, the evaluation involves the bleu score as an automatic evaluation metric in addition to a human evaluation score. for more detailed discussion and analysis of the dataset, please refer to @xcite . in this work, we evaluate and fine-tune five pre-trained transformer-based @xcite ) language models with various sizes for the previously mentioned commonsense subtasks. for the first subtask, we use bert @xcite , roberta @xcite and albert @xcite language models. the best we achieve is 92.90% placing our team in the 12 th place among 39 teams. while in the second subtask, we utilize xlnet @xcite in addition to bert and roberta language models to achieve 92.30% accuracy, which places our team in the 9 th place out of 27 teams. finally, in the last subtask, we used gpt-2 @xcite language model to tackle the text generation task. our system achieves 16.10 bleu score and 1.94 human evaluation score, which places our team in the 5 th place (out of 17 teams) and the 3 rd place, respectively. our system's human evaluation score is only 0.16 away from the top score in the competition. our code and experimental results are publicly available in a github repository. 1 the rest of this paper is organized as follows. the following section demonstrates how we utilize the pre-trained transformer-based language models to build our systems for each subtask. in section 3, we present our experimental results and discuss some insights. finally, we provide concluding remarks in section 4. ## system overview in the following subsections we describe how we utilize the pre-trained transformer-based language models to build our subtasks' systems. ## experimental results to run our experiments with the targeted pre-trained transformer-based language models, we use the google colab platform @xcite and two open source python packages, which are transformers @xcite foot_1 and simpletransformers. 3 for each subtask, we report the development and test sets results, training time and model size. more experimental results can be found in our github repository. in the following subsections, we describe the models' types and sizes that are used for each subtask, their results and some insights learned from our experiments. ## conclusion in this work, we evaluated pre-trained transformer-based language models against three commonsense tasks as part of the commonsense validation and explanation (comve) task, which is part of sem-eval2020. our experiments showed that pre-trained language models can be treated as powerful knowledge bases to extract and validate facts. we were ranked in the 12 th and 9 th for first and second subtasks, respectively. as for third subtask, we were ranked in the 5 th and 3 rd places using automatic (bleu) and human evaluation metrics, respectively. we provide the code and the experimental results through our github repository.