T eam JUST at S em E val-2020 Task 4: Commonsense Validation and Explanation Using Ensembling Techniques (2020)

Abstract: Common sense for natural language processing methods has been attracting a wide research interest, recently. Estimating automatically whether a sentence makes sense or not is considered an essential question. Task 4 in the International Workshop SemEval 2020 has provided three subtasks (A, B, and C) that challenges the participants to build systems for distinguishing the common sense statements from those that do not make sense. This paper describes TeamJUST’s approach for participating in subtask A to differentiate between two sentences in English and classify them into two classes: common sense and uncommon sense statements. Our approach depends on ensembling four different state-of-the-art pre-trained models (BERT, ALBERT, Roberta, and XLNet). Our baseline model which we used only the pre-trained model of BERT has scored 89.1, while the TeamJUST model outperformed the baseline model with an accuracy score of 96.2. We have improved the results in the post-evaluation period to achieve our best result, which would rank the 4th in the competition if we had the chance to use our latest experiment.

## introduction natural language processing and understanding (nlp/nlu) have great care of research, recently. several language models had been trained on huge data of corpora @xcite , and some benchmarks showed how methods display enhanced performance @xcite . however, most end-to-end trained methods on common sense, compared to people, are disappointing. for instance, people can understand and recognize immediately that we can place an apple into a fridge, but cannot and never place a tv into a fridge. thus, for trained systems, it is much harder to recognize the difference. accordingly, it is essential to have the ability to evaluate how good a system is to recognize the sense @xcite . recent datasets have experimented common sense over tasks, for example co-reference resolution @xcite or subsequent event prediction @xcite . they stated that a method is prepared with common sense through testing if it can offer a correct answer where no added knowledge to the input. semeval is the international workshop on semantic evaluation, which is developed from senseval to evaluate semantic study methods. the 14th version of this workshop, semeval-2020, has provided 12 tasks. we have participated in task 4 -commonsense validation and explanation @xcite , which offers three subtasks with datasets in english. the main goal of this task is to recognize which sentences are common sense. further information about task 4 and the datasets is given in section 3. this paper describes our model for participating in semeval-2020 (task 4 -sub task a). our team has built an ensembling system to select from two english statements that both have alike words to identify which one is making sense and which one is not making sense. we have used four different state-of-the-art pre-trained models (bert @xcite , albert @xcite , roberta @xcite , and xlnet @xcite ), then we combined their outputs and used voting method for each model to choose the shared output from each model. our baseline model has scored 89.1 accuracies while our improved model has shown significant performance over the baseline model with scoring 96.2 and it is 0.8 away from the first ranked model. our baseline model ranked 17 out of 41 teams while teamjust would rank fourth. the paper is constructed as follows: related work is provided in section 2. a description for task 4 and the datasets are presented in section 3. the architecture of our approach is introduced in section 4. the detailed experiments are provided in section 5. results and analysis are presented in section 6. finally, the conclusion is in section 7. ## related work in our digital world, there is an increasing need to manage the vast amount of text. therefore, the researchers are motivated to use the semantic knowledge to develop the models that can have a meaningful understanding of text @xcite . in @xcite , the researchers released a benchmark to test whether a method can distinguish the statement that is making sense, and find the reason behind why a statement does not make sense. moreover, the researcher in @xcite highlighted the importance of actions and consequences. they worked on finding the appropriate cause or outcome of the evidence from given two replacements. all pieces of evidence and replacements are modest sentences. for instance, 'evidence: the girl broke her toe. what was the reason for this?' 'replacements 1: she got in her sock a hole.', 'replacement 2: she dropped on her foot a hammer.' the task of winograd schema challenge (wsc) @xcite required extra commonsense knowledge. for instance, "the crown would not fit in the black suitcase because it was too big (small). what was too big (small)?" "answer0: the crown", "answer1: the suitcase". yet, they had to estimate common sense incidentally and without explaining why this answer was true whereas the other was incorrect. for example, the work in @xcite used the technique of knowledge acquisition and a common model of neural association. they extracted a big number of pairs of cause-effect from different text corpus then the knowledge that was extracted was utilized to train the model of neural association. in @xcite , the researchers developed a system to find out the correct finish of a story from two sentences after the fourth sentence. on the other hand, the researchers in @xcite provided a story text and different questions with two different answers knowing that some questions need knowledge outside the truths stated in the text. they used two models of unsupervised learning which were using only distributional information and the information of word and used two models of neural networks that were supervised. in @xcite , the researchers offered a great daily commonsense rational knowledge diagram that had relations of nine if-then with variables, containing reasons and effects. they presented models of neural networks that can learn to know about previous events that are not seen to produce their possible reasons and effects in natural language. while in @xcite , the researchers suggested a new task and corpus that discover the mentioned or unmentioned people's aims and reactions beneath various everyday circumstances. they used different neural models of encoder-decoder such as (convolutional neural network and bi-directional rnn). the goal of our system is to predict which statement is not making sense between two almost identical sentences. our system is different than the others by using state-of-the-art pre-trained models. ## task and data description task 4 @xcite in the semeval-2020 workshop offers three subtasks with datasets in english. subtask a is asking the users to build a system to recognize which sentences are sense from those that are not. each record of the provided dataset contains two sentences: [sent0, sent1]. the two sentences are comparable sentences that are in similar syntax structure and only different in some words. one of the two sentences is sense while the other sentence is not. figure 1 shows an example of the training data. for this task, the model has to detect which one is not sense. the task's organizers have provided training, developing, trail, and testing datasets in english language. they also have chosen the accuracy score as the evaluation criteria in this competition. ## teamjust model our model follows the workflow that is shown in figure 2 . ## experiments we have experimented with multiple states of the art language models that have been trained on large texts. in this section, we will summarize our experiments with their results, and then we will show the result of the ensembling techniques. first experiment: in our first experiment, we have used lstm @xcite a recurrent network architecture. we have cleaned the training dataset first and combined the two sentences and changed the output to be a binary classification. for the optimizer, we have chosen adam @xcite and did not fine-tune the hyperparameters. second experiment: for the next experiment, we also have applied lstm, but with a different optimizer, rmsprop @xcite and we have fine-tuned the hyperparameters. third experiment: for the third experiment, we have chosen a deeper recurrent neural network, bilstm, with rmsprop optimizer and we have fine-tuned the hyperparameters. fourth experiment: in the fourth experiment, we have used the state of the art model bert without fine-tuning the hyperparameters. we have also combined the sentences. fifth experiment: for the fifth experiment, we also have used bert, but we have fine-tuned the hyperparameters with selecting the bert-base-cased model. sixth experiment: in the sixth experiment, we have continued using bert, but we have used the bert-large-cased model. it is worth mentioning that after preprocessing the dataset we haven't combined the two sentences, rather we trained the model on pair sentences. ## results and analysis in for the other experiments, when using different models of recurrent network architecture like lstm or bilstm, it does poorly identify the sentences which are against sense, even if fine-tuning them. on the other hand, we can find a remarkable case proving that the fine-tuning process helps the model recognize common sense. ## conclusion we have investigated multiple experiments and used different models with fine-tuning them for evaluating if a system can predict if a sentence makes sense or not. the evaluating models are trained on a large raw text and we used four different state-of-the-art models. results show how state-of-the-art models with fine-tuning them can have a significant improvement in common sense tasks.