Masked Reasoner at S em E val-2020 Task 4: Fine-Tuning R o BERT a for Commonsense Reasoning (2020)

Abstract: This paper describes the masked reasoner system that participated in SemEval-2020 Task 4: Commonsense Validation and Explanation. The system participated in the subtask B.We proposes a novel method to fine-tune RoBERTa by masking the most important word in the statement. We believe that the confidence of the system in recovering that word is positively correlated to the score the masked language model gives to the current statement-explanation pair. We evaluate the importance of each word using InferSent and do the masked fine-tuning on RoBERTa. Then we use the fine-tuned model to predict the most plausible explanation. Our system is fast in training and achieved 73.5% accuracy.

## introduction commonsense validation and explanation is a critical area in natural language understanding. recent research advances @xcite @xcite pushed the bar in this area to a new height. semeval 2020 task 4 @xcite ) is focused on this area. it has 3 subtasks. subtask a is focused on commonsense validation. it gives two statements. one of them is against commonsense. for example, 1) he poured orange juice on his cereal. statement: he put an elephant into the fridge. possible valid reasons: 1) an elephant is much bigger than a fridge. ## discussion there is a big drop in terms of accuracy between training and testing. we doubt it is due to over-fitting. our system gain very little after 3 epochs so we should stop earlier to make the system more general-purpose. despite over-fitting, our best accuracy, 87.43% still cannot enter the top 10 on the leader board. we suspect that it is because we did not use any external knowledge base. some large external knowledge base such as conceptnet @xcite , could help a lot. ## conclusion we proposed a novel method for fine-tuning pre-trained model and fit it into subtask b. the combination of infersent and roberta can make the masked training faster. meanwhile, @xcite introduced a more thorough way of masking every word as well as n-grams in both the statement and the explanation. we think that only the important words need mask. trivial words such as 'he', 'the', 'to' could be noise.