CS - NLP Team at S em E val-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task (2020)

Abstract: In this paper, we investigate a commonsense inference task that unifies natural language understanding and commonsense reasoning. We describe our attempt at SemEval-2020 Task 4 competition: Commonsense Validation and Explanation (ComVE) challenge. We discuss several state-of-the-art deep learning architectures for this challenge. Our system uses prepared labeled textual datasets that were manually curated for three different natural language inference subtasks. The goal of the first subtask is to test whether a model can distinguish between natural language statements that make sense and those that do not make sense. We compare the performance of several language models and fine-tuned classifiers. Then, we propose a method inspired by question/answering tasks to treat a classification problem as a multiple choice question task to boost the performance of our experimental results (96.06%), which is significantly better than the baseline. For the second subtask, which is to select the reason why a statement does not make sense, we stand within the first six teams (93.7%) among 27 participants with very competitive results. Our result for last subtask of generating reason against the nonsense statement shows many potentials for future researches as we applied the most powerful generative model of language (GPT-2) with 6.1732 BLEU score among first four teams. .

## introduction commonsense is unstated background knowledge that is used to perceive, infer, and understand the physical world, human emotions, reactions, and knowledge of the common facts that most people agree with. ordinary commonsense helps us to differentiate between simple false and true statements or answer questions, such as "can an elephant fit into the fridge" quickly, but they can be difficult for automatic systems @xcite . recent advances in machine learning emphasize the importance of commonsense reasoning in natural language processing (nlp) and as a critical component of artificial intelligence (ai). in the fifty-year history of ai research, progress was extremely slow @xcite in automated commonsense reasoning. however, when transfer learning @xcite and then transformers were introduced to the nlp world @xcite , great breakthroughs and developments have occurred at an unprecedented pace @xcite . advances in machine learning and deep learning methods have been achieved in numerous studies and wide range of disciplines @xcite @xcite @xcite . this paper describes a system participating in the semeval-2020 "commonsense validation and explanation (comve) challenge", multiple tasks of commonsense reasoning and natural language understanding (nlu) designed by @xcite . the competition is divided into three subtasks, 1 https://github.com/sirwe-saeedi/commonsense-nlp this work is licensed under a creative commons attribution 4.0 international licence. licence details: http://creativecommons.org/licenses/by/4.0/ which involve testing commonsense reasoning in automatic systems, multiple choice questions, and text generation. in these tasks, participants are asked to improve the performance of previous efforts @xcite . we apply statistical language modeling, or language modeling (lm) for short as one of the most important parts of modern nlp and then transfer learning to reuse a pretrained model on different data distribution and feature space as the starting point of our target tasks. applying transfer learning to nlp significantly improves the learning process in terms of time and computation through the transfer of knowledge from a related task that has already been learned @xcite . language modeling is the task of probability distribution over sequences of words. it also assigns a probability for the likelihood of a given word (or a sequence of words) to follow a sequence of words. language modeling are applied to many sorts of tasks, like: machine translation, speech recognition, question answering, sentiment analysis, etc. awd-lstm (asgd weight-dropped lstm) @xcite is a fundamental building block of language modeling which uses different gradient update step and it returns the average value of weights in previous iterations instead of current iteration. recently, there have been some excellent advancements towards transfer learning, and its success was illuminated by bidirectional encoder representations from transformers (bert) @xcite , openai transformer (gpt-2) @xcite , universal language model fine-tuning for text classification (ulmfit) by fast.ai founder jeremy howard @xcite , elmo @xcite , and other new waves of cutting-edge methods and architectures like xlnet @xcite , facebook ai roberta: a robustly optimized bert pretraining approach @xcite , albert: a lite bert for self-supervised learning of language representations @xcite , t5 team google @xcite . for this work, we employ and fine-tune some of these suitable models. when bert was published, it achieved state-of-the-art performance on a number of natural language understanding tasks. as opposed to directional models like word2vec @xcite , which generates a single word representation for each word in the vocabulary and read the text input sequentially, bert is deeply bidirectional and reads the entire sequence of words at once. therefore, bert allows the model to learn the context of a word based on all of its surroundings using two training strategies: masked language model (mlm) and next sentence prediction (nsp). mlm technique is masking out some of the words in the input and then condition each word bidirectionally to predict the masked words. a random sample of the tokens in the input sequence is selected and replaced with the special token '[mask]' and the objective is a cross-entropy loss on predicting the masked tokens @xcite . in this paper, we use a method inspired by mlm. the paper "attention is all you need" @xcite describes a sequence-to-sequence architecture called transformers relying entirely on the self-attention mechanism and does not rely on any recurrent network such as gru @xcite and lstm @xcite . transformers consist of encoders and decoders. the encoder takes the input sequence and then decides which other parts of the sequence are important by attributing different weights to them. decoder turns the encoded sentence into another sequence corresponding to the target task. a huge variety of downstream tasks have been devised to test a model's understanding of a specific aspect of language. the general language understanding evaluation (glue) @xcite and the natural language decathlon (decanlp) benchmarks @xcite consist of difficult and diverse natural language task datasets. these benchmarks span complex tasks, such as question answering, machine translation, textual entailment, natural language inference, and commonsense pronoun resolution. the majority of state-of-the-art transformers models publish their results for all tasks on the glue benchmark. for example, models like different modified versions of bert, roberta, and t5 outperform the human baseline benchmark @xcite . for the evaluation phase, glue follows the same approach as semeval. our attempts at semeval-2020 task4 competition boost performance on two subtasks significantly. our idea in reframing the first subtask helps to outperform results of state-of-the-art architecture and language models like bert, albert, and ulmfit. general-domain of ulmfit is to predict the next word in a sequence by a widely used pretrained awd-lstm network on the wikitext-103 dataset. ulmfit could outperform many text classification tasks like emotion detection, early detection of depression, and medical images analysis @xcite @xcite . we were ranked 11 with a very competitive result on the first subtask and achieved rank 6 for the second subtask amongst 40 and 27 teams, respectively. this paper is organized as follows. in section 2, we introduce the three subtasks and their datasets. in section 3, we describe our different applied models and various strategies that were used to fine tune the models for each individual subtask. in section 4, we present the performance of our system. finally, we conclude the paper in section 5. ## task definition and datasets as discussed, semeval-2020 task 4 consists of three subtasks, each designed for a different natural language inference task. figure 1 shows a sample for each subtask and the corresponding answer of the model. large pretrained language models are definitely the main trend of the latest nlp breakthroughs. as transformers occupy the nlp leaderboards, we choose several state-of-the-art architectures to outperform the baseline of all subtasks significantly. for each subtask, we describe our system separately below. ## conclusion we evaluated architectures for three commonsense reasoning tasks. first, we found that roberta-large performs better substantially in differentiating sentences that make sense from those that do not make sense compared to other cutting-edge architectures (e.g. albert, bert, and ulmfit). we reframe this classification task to a question answering task to enhance the performance of the fine-tuned roberta to 96.08%. second, we achieved significant results on reasoning why false statements do not make sense. we showed that roberta performs well in selecting the correct option among three to infer the commonsense reason and it yields significant result with 93.7% accuracy compare to baseline using bert, 45.6%. with a little effort on generating reasons to explain why false statement conflicts with commonsense knowledge, we observe that the original test set produces 17.2 bleu score which ranked us among first four teams in the competition with a very competitive results. our experimental result showed that gpt-2 performs as poor as random generating of a sequence of words for this task. we believe this task has many potentials and challenges for upcoming nlp researches. as another future work, we believe that ensemble learning can reduce the variance of predictions and also improve prediction performance. in ensemble learning, multiple models are generated and combined to address the subtasks and reduce the likelihood of an unfortunate selection of a poor one.