NLP @ JUST at S em E val-2020 Task 4: Ensemble Technique for BERT and Roberta to Evaluate Commonsense Validation (2020)

Abstract: This paper presents the work of the NLP@JUST team at SemEval-2020 Task 4 competition that related to commonsense validation and explanation (ComVE) task. The team participates in sub-taskA (Validation) which related to validation that checks if the text is against common sense or not. Several models have trained ( i.e. Bert, XLNet, and Roberta ), however, the main models used are the RoBERTa-large and BERT Whole word masking. As well as, we utilized the results from both models to generate final prediction by using the average Ensemble technique, that used to improve the overall performance. The evaluation result shows that the implemented model achieved an accuracy of 93.9% obtained and published at the post-evaluation result on the leaderboard.

## introduction recently, natural language processing (nlp) has got more attention because it is able to deal with human and machine communication without human intervention @xcite . nlp is a subset of artificial intelligence (ai) that grants the machines ability to read and understand human words using machine learning (ml) and deep learning (dl) models. however, it is used to solve a large scope of tasks such as machine translation, speech recognition, sentiment analysis, etc. nlp has two subsections natural language understanding (nlu) and natural language generation (nlg). furthermore, nlu with commonsense has gained the researcher's interests in the last few years @xcite . commonsense is known as the knowledge that captured about the world's continents and the implicit reasoning process @xcite . as known the human have obtained commonsense knowledge from their daily life, they can distinguish the sentence is against commonsense or not from their previous knowledge. the commonsense task is considered as one of the important tasks of nlu in nlp that makes the machines more intelligent @xcite . hence, there are many works in the literature focus on commonsense tasks, and various benchmarks created to evaluate a trained model's capabilities to capture the commonsense validation task. for example, @xcite introduced a new benchmark that has two tasks which are: the sen-making task and explanation task, to evaluate the system can judge the text makes sense or not and provide reasoning why the text is not commonsense @xcite . this paper presenting nlp@just teamwork submitted at semeval-2020 task 4 competition at a post-evaluation time. this task is related to commonsense validation and reasoning, released by @xcite . semeval-2020 task 4 has three subtasks: validation, explanation (multi-choice), and explanation (generation). specifically, this paper focuses on the commonsense validation task. mainly the roberta @xcite and bert-wwm (whole word masking) @xcite models are adopted, where they are considered as an extension to the bert @xcite model. the work obtained good results with 93.9% accuracy by using the average ensemble for both of models, which is perfect when compared with other pre-trained models we used (i.e. roberta @xcite , bert original @xcite , xlnet @xcite , @xcite , use @xcite ). the rest of the paper is organized as follows: section 2 provides the most relevant works, section 3 describes the details of the methodology and proposed system, section 4 presents the experiments that were conducted in addition to the results obtained, and finally, section 5 concludes the work. ## related works there is a growing attention towards common sense in the research community and many previous studies tried to build and use machine learning and deep learning techniques for dealing with common sense tasks. for example, (trinh and le, 2018) introduced a method to apply language models on tasks that deal with a specific of commonsense knowledge. they adapt the probabilities that computed by the language models to evaluate the statement if it is true or not. the most recent pre-trained model called roberta @xcite which known as a robustly optimized bert pretraining approach. they developed the previous bert model by training the model longer time, and increasing batches using more data in addition to removing the objective of next sentence prediction. besides that, they trained the model on longer sequences, and changing the masking pattern was used in the training phase. @xcite used the uni-directional language modeling (gpt, gpt2-base, gpt2-medium) and bi-directional language modeling (bert-base, bert-large, xlnet-base, xlnet-large, roberta-base, and robert-alarge) to compare them on seven benchmarks realted to commonsense reasoning tasks. the results showed that roberta outperformed other pre-trained models over the seven benchmarks. @xcite produced a new benchmark for evaluating the system that abilities to capture the sentence that do not make sense from the sentence that makes sense. besides, they can explain why this sentence does not make sense. several models used on their experiments which are: random, elmo, bert, and fine-tuned elmo, and @xcite developed a framework that provided an evaluation for commonsense knowledge for the set of machine comprehension as a subtask of semeval 2018. the best-ranked model achieved by @xcite proposed a neural network model called hybrid multiaspects (hma). their model aim was to provide a multi-aspect output and then combine it along together for the output of the final prediction. they achieved the best rank on this semeval task with 84.13% accuracy. also, for the same task, @xcite applied a three-way attention technique for interaction among texts, questions, and answers on the top of bilstms. however, they have achieved an accuracy of 83.95%. ## experimentation and result for the sake of this research, accuracy evaluation metric has used to evaluate the performance of the implemented model according to the shared task instruction. the results of this work obtained during the post-evaluation time. as presented in