HR @ JUST Team at S em E val-2020 Task 4: The Impact of R o BERT a Transformer for Evaluation Common Sense Understanding (2020)

Abstract: This paper describes the results of our team HR@JUST participation at SemEval-2020 Task 4 - Commonsense Validation and Explanation (ComVE) for POST evaluation period. The provided task consists of three sub-tasks, we participate in task A. We considered a state-of-the-art approach for solving this task by performing RoBERTa model with no Next Sentences Prediction (NSP), dynamic masking, larger training data, and larger batch size. The achieved results show that we got the 11th rank on the final test set leaderboard with an accuracy of 91.3%.

## introduction the concept of common sense in natural language systems has received increasing attention by researchers, as it is considered a long-term problem, and an important challenge in nlp. for example, , "he drinks milk", is a logical sentence, while, " he drinks apple", is not a logic sentence. looking at the example we know that it is self-evident that a human can distinguish between logical and illogical sentences due to common sense reasoning, while the trained model is weak in common sense and consider as a difficult task in nlp. by using language models trained on large corpora with neural networks @xcite achieved a good result compared to previous models, but not satisfactory enough. using traditional machine learning algorithm @xcite still have not had good results so far compared with human performance. although these efforts, it is still unacceptable to rely on them for common sense problems. natural language understanding is one of the main problems in nlp and the concept of common sense falls under this field. where it is difficult to distinguish between the meanings of words in terms of semantic meaning and syntax structure. in the semeval-2020 task 4 subtask a, the organizers provide a dataset contains of 10 sentences: {s1, s2, o1, o2, o3, r1, r2, r3}. s1 and s2 are two sentences that are similar in syntactic and structure but different in a few words, that require the participants to build a model able to identify which one makes sense. semeval is the international workshop on semantic evaluation that has evolved from semeval.this workshop aims to evaluate semantic analysis systems where is poses some problems related to natural languages (e.g., lexical semantics, common sense knowledge, knowledge extraction, humour, emphasis, and sentiment), the semeval-2020 being the 14th workshop on semantic evaluation. task 4 @xcite present three sub-tasks with annotated datasets in the english language. the task for participating teams is to distinguish the sentence that common sense from others. the dataset contains two sentences s1 and s2 are similar in syntactic and structure but different in a few words, that require the participants to build a model able to identify which one makes sense. further details about task 4 @xcite and the dataset in section 3.2. in this paper, we describe the model was used in our submission in sub-task a (validation). the remainder of this paper is organized as follows: in the second section sheds light on related works that used to solve common sense problems. in the third section, we described the experimental setup, and the methodology proposed in this research. we discussed our results in the section. finally, section 5 concludes this research. using neural networks with language model produces success in common sense challenging. (trinh and le, 2018) proposed an unsupervised approach to handling the commonsense task, they trained and evaluated the recurrent neural network (rnn) model on a large dataset, consist of five corpora namely, squad, commoncrawl, lm-1-billion, stories, and gutenberg books on two levels, character and word level. their model achieves 70% on the pronoun disambiguation dataset and wit accuracy 63.7% on the winograd schema dataset. semantic text similarity has several metrics and we can use several deep learning models to evaluate it. @xcite presented a survey that described these metrics and models. as well as, they discussed the challenges and the evaluation applications that they found. ## results to evaluate the performance of the performing models accuracy has used according to the competition instruction. in order to evaluate our developed roberta model, we used pytorch for implementation. we optimized the hyperparameters during the training step, our participation in this task presented the results for the post evaluation. we have used several models during the experimental step, for instance, roberta, bert, xlnet with based version, use, and elmo. we used elmo model as baseline for our experiments, furthermore, roberta outperforms all the performed models. as shown in ## conclusion in this paper, we have presented our participation at semeval-2020 task 4 -commonsense validation and explanation (comve). several models have presented in particular transformers to tackle task a (validation) that aims to differentiate between two sentences against common sense among them. roberta was used as the main model to solve this task. the achieved results show that we got the 11th rank on the final test set leaderboard with an accuracy of 91.3%. for future work, we will try to study the impact of transfer learning.