UAICS at S em E val-2020 Task 4: Using a Bidirectional Transformer for Task a (2020)

Abstract: Commonsense Validation and Explanation has been a difficult task for machines since the dawn of computing. Although very trivial to humans it poses a high complexity for machines due to the necessity of inference over a pre-existing knowledge base. In order to try and solve this problem the SemEval 2020 Task 4 - ”Commonsense Validation and Explanation (ComVE)” aims to evaluate systems capable of multiple stages of ComVE. The challenge includes 3 tasks (A, B and C), each with it’s own requirements. Our team participated only in task A which required selecting the statement that made the least sense. We choose to use a bidirectional transformer in order to solve the challenge, this paper presents the details of our method, runs and result.

## introduction natural language understanding (nlu) is a topic that in recent years caught researchers' attention. growing interest has led to the development of multiple areas of study, one such area being represented by "commonsense validation" which aims to evaluate a system's ability to infer commonsense knowledge from basic sentences. the "semeval 2020 task 4 -commonsense validation and explanation (comve)" @xcite ) seeks to research in the aforementioned field by testing systems that are able to differentiate between statements that make sense from the ones that don't. the task is split in multiple tasks: task a implies selecting which statement is against commonsense between two very similar phrases, task b requires selecting between three reasons that explain why the statement from task a is incorrect and finally task c requires the machine to generate reasons why the statement from task a is against common sense. our team participated only in task a, in the official results our system ranked 17 th out of 45 submission with an accuracy of 89.1%, the best result being 97% and the worst 0%. the strategy was to use a pre-trained model, bert @xcite , in order to find out which sentence makes the most sense. this model has been trained and tested using different parameters in order to achieve the best result and it will be described with more details in section 2, "system overview". similar work has been done by @xcite where they trained a rnn on a very large text collection resulting in a 63.7% accuracy in the winograd schema challenge @xcite while @xcite using a bert model was able to achieve overall accuracy of 72.5%; this performance increase led us to choose the latter model as we feel it best fits our purpose. in this paper we will describe our system, results and conclusions. the structure is as follows: section 2 will present the system designed and the experimental setup, section 3 will talk about the result and in the end we will draw some conclusions. ## system overview in this section we will present the architecture and inner workings of our solution, the system uses python 1 (anaconda 2 ) combined with torch 3 and huggingface's transformers 4 (that implements the bert model) in order to achieve the task requirements. we did not use external data sets, the only information used was contained in the bert model that was in the transformers library, more so no other external input sources besides the data sets provided by the organiser have been integrated. the system output consisted of the sentence id followed by the index of the sentence against common sense (either 0 or 1). ## results in this section we will discuss the different results of the system. in the beginning we had some problems trying to figure out the optimal hyperparameters for the model, in the end, after multiple trial and errors we arrived at the configuration described in section 2. the performance of the system in the training phase can be seen in ## conclusion to conclude, in this paper we proposed a model to solve the task 4 challenge which deals with evaluating a systems ability to detect sentences that are against commonsense. we used bert as the main tool to help us obtain good results, we ranked 17 th out of 45 submission with an accuracy of 89.1%. as future work, besides the ones already mentioned, we are currently trying to improve the systems accuracy by including an external ontology in order to better infer whether a phrase is against common sense or not.