K a LM at S em E val-2020 Task 4: Knowledge-aware Language Models for Comprehension and Generation (2020)

Abstract: This paper presents our strategies in SemEval 2020 Task 4: Commonsense Validation and Explanation. We propose a novel way to search for evidence and choose the different large-scale pre-trained models as the backbone for three subtasks. The results show that our evidence-searching approach improves model performance on commonsense explanation task. Our team ranks 2nd in subtask C according to human evaluation score.

## introduction commonsense reasoning has been seen as one of the key ability for intelligent machines to perform various activities @xcite . semeval 2020 task 4 @xcite ) is a commonsense validation and explanation task which is inspired by @xcite . this task consists of three subtasks. the first subtask is to choose the one which makes sense from two natural language statements with similar wordings; the second subtask is to decide among three options the most crucial reason why a given statement does not make sense; the third subtask requires the machine to generate the reasons. to make predictions or generate reasons, background knowledge is essential. a simple way to supplement that knowledge is utilizing plain texts from natural language databases, e.g. wikipedia. intuitively, for a specific given statement, plain texts can be provided by searching similar sentences in the database. in other words, each evidence sentences contain the keywords of the given statement. this method is used in some of the state-of-the-art models @xcite . however, for the purpose of explaining the reason, evidence which has similar wording with the given statement may lack information, and sometimes can be misleading when the given statement does not make sense. in this work, we propose a novel way for evidence-searching using plain texts. we obtain evidence by searching for the meaning of the keywords in the given statement. in other words, using evidence containing the meanings of the keywords rather than containing the keywords themselves. the reason for our method is that these meanings may provide important information to explain why a statement makes sense or does not make sense. for example in figure 1 , the definition of "aircraft carrier"-"a warship designed to carry aircraft"-is given by the evidence obtained from the database. such a definition well explains why the given statement is wrong and can be used when generating the reason. in contrast, evidence containing both "aircraft carrier" and "human" will not be helpful. we conduct experiments on subtask a, b, and c. results show that our evidence-searching method boosts the performance on subtask c. our team achieves accuracy of 95.3 (9th place) in subtask a and 93.2 (7th place) in subtask b. in subtask c, our approach achieves the bleu score of 18.5 (3rd place) and human evaluation score of 2.08 (2nd place). moreover, the best bleu score in our experiments (20.4) even outperforms the score we obtain in competition. ## related work pre-trained language models have been proved to be essential while dealing with sequence-to-sequence task. we present the language model we utilized in this section and demonstrate our consideration. we use roberta @xcite for language comprehension task. the structure of roberta is based on bert @xcite . we also use bart @xcite for generation task. bart is a denoising autoencoder for pre-training sequence-to-sequence models. it uses the standard sequence-tosequence transformer architecture except the activation functions which is replaced by gelus. we find the structure of autoregressive decoder make the model can be directly fine tuned for sequence generation tasks. bart is also enable to apply any type of document corruption which make the additional knowledge available to the model. machine common sense has long been acknowledged as a critical component for natural language understanding. the challenge for the model has turned to abstractive knowledge comprehension. a task that is closely related to semeval 2020 task 4 is commonsenseqa @xcite , in which the commonsense knowledge is required to make the correct prediction. in commonsenseqa task, large-scale pre-trained models have brought significant performance gains. these gains are obtained by developing training strategies and enlarging training data @xcite , or improving parameter efficiency @xcite . while some of the improvement is achieved by developing the pre-trained model itself, some other approaches resort to external modules, e.g. knowledge extraction and graph-based reasoning in @xcite . in our work, we are interested in the knowledge extraction method because developing the pre-trained model itself will be computationally expensive. moreover, different from recent common strategy @xcite @xcite to use external knowledge which involves structured knowledge like conceptnet @xcite , we extract external knowledge from plain texts. ## system description we first describe our evidence-searching approach which can be utilized in downstream tasks. then we describe our systems for three subtasks. ## experiments we use the officially released dataset and standard train/trial/dev split of semeval 2020 task 4 for experiments. we will give the performance of the best settings on test split. note that we compare different settings through performance obtained by training the model on train&trial split and testing it on dev split since testing on test split is inconvenient. we will also give our configuration of final submissions for subtask a, b, c in section 4.1, 4.2, 4.3 ## conclusion in this work, we choose the different large-scale pre-trained models as the backbone for three subtasks and propose a novel way to search for evidence, which aims to obtain the meaning of the keywords in the given statement. our experiments demonstrate the importance of additional knowledge for language models to understand the content. the results show that our evidence-searching approach is helpful to commonsense explanation task. obtain more meaningful gloss, we search the pairs for the prototype word, regarding the prototype word as one of the keywords. specifically, we detect whether the gloss contain words like "plural of/past of/third person singular of/clipping of/alternative form of/alternative spelling of" which indicate the gloss points to a prototype word. in that case, we use the same search function to acquire evidence for prototype word and incorporate the new evidence. (note that to avoid infinite loop, we do not detect the prototype in sub-search). besides, we also adjust the number of evidence tuples for a keyword dynamically. for the statement contains less keywords, we obtain more evidence tuples for each keywords. thus the length of the evidence for statements will be more stable.