KDE S ense F orce at S em E val-2020 Task 4: Exploiting BERT for Commonsense Validation and Explanation (2020)

Abstract: Using a natural language understanding system for commonsense comprehension is getting increasing attention from researchers. Current multi-purpose state-of-the-art models suffer on commonsense validation and explanation tasks. We have adopted one of the state-of-the-art models and proposing a method to boost the performance of the model in commonsense related tasks.

## introduction commonsense knowledge is a knowledge that is built by everyday activities and vital for living. because of the assumption that every person has the commonsense, we tend to discard the commonsense from social communications. hence, most of the text data lack explicit declaration of commonsense @xcite . nonetheless, basic knowledge of the commonsense related to space, physical interactions, and people is necessary for the development of the natural language processing, computer vision, and robotics @xcite , and has been studied as early as 1956 @xcite . currently, there are many natural language processing subfields where machine learning has started to outperform humans @xcite . however, even the current state-of-the-art models like bert @xcite , @xcite fail in commonsense validation and explanation tasks @xcite . we propose a system to cope with commonsense validation and explanation tasks using input transformation and word feature scaling. our proposed system extracts features about every word in the sentence from its surrounding contexts except the word itself then compares extracted features with the original word features to get a score map ( usually, scaling coefficient is called 'attention ', whereas it is called score map to avoid a confusion ). then the score map is used to scale the word feature in the current state-of-the-art models. the result is noticeable, considering that the system uses only one additional fully connected layer on top of the existing system. for the validation (a) and explanation (b) task, the proposed system outperforms the result of the bert by 9.5% and 27.2% , respectively. the results are submitted to the semeval-2020 @xcite and turned out to be in 33rd and 22nd places respectively. the system has room for improvement, since the tested input transformation works on only word level. ## background nowadays, transformer's encoder and decoder model @xcite has been used as the basis for almost all state-of-the-art technologies in natural language processing. a typical example is bert. bert is pretrained on a large dataset to predict masked words in a sentence from the context. this pretraining method creates a discrepancy between pretraining and finetuning @xcite . we think this pretraining method is used insufficiently in the commonsense validation and explanation tasks considering the pretraining dataset contains some commonsense already. to use this pretraining method better, we propose a model which uses @xcite token in the finetuning process. semeval-2020 task 4 comve challenge consists of 3 subtasks: validation (a), explanation (multi-choice) (b) and explanation (generation)(c). as the organizers benchmarked, current state-of-the-art models like bert and elmo struggle with these subtasks, showing maximum accuracy of 74.1% and 45.6% for the first two subtasks. for the training of the system, we only used dataset that was provided by task organizers. examples of the first two subtasks examples are shown in figure 1 . each subtasks dataset contains around 10,000 examples per subtask. we employed the hugging face libraries @xcite pretrained model. we submitted for the subtasks a and b of the competition. ## system overview commonsense validation is a difficult task because it can depend on any word in the sentence. by changing a single word or a punctuation, the sentence can gain or loose its commonsense. thus, we propose a model that consists of two stages: token level commonsense evaluation and word feature scaling. ## experimental setup we only train the fully connected layers as bert fine tuning. hence, we trained our system for 4 epochs as stated in the bert's paper. we used batch size of 1 and learning rate of 5e-3. the reason that we used 1 as our batch size is due to the multiplication of the input sentence in the evaluation stage. the multiplied input is treated as batch in the pytorch @xcite . the code is written on python3.6 @xcite with pandas (mckinney and others, 2010) for reading the input files, pytorch for machine learning and hugging face for using bert model. as we mentioned earlier, the dataset we used is only from task organizers, so that neither data augmentation nor external resource was used in our experiment. ## results initially, we expected the important words such as subject, verb and object to have higher scores in the token level evaluation, but the important words had lesser scores than the other words of the sentence. in our proposed models accuracies on subtask a and b are 79.2% and 72.8%, respectively. comparisons are presented in ## conclusion in this paper, we proposed a system coping with two subtasks of semeval-2020 task 4: commonsense validation and explanation. to do that we have adopted one of current state-of-the-art models and enhanced its accuracy notably by only adding one extra fully-connected layer, which makes it memory efficient. the score map generation method, however, turns out to be computationally more expensive. the results are submitted to the semeval-2020. the experiment demonstrated that we could use the discrepancy between pretraining and finetuning of bert that was caused by input corruption for score map generation for the commonsense validation and explanation subtasks. in the future, we intend to experiment with different variants of transformers such as gpt-2 @xcite , xlnet and albert @xcite with similar kind of input transformation and scaling.