CUHK at S em E val-2020 Task 4: C ommon S ense Explanation, Reasoning and Prediction with Multi-task Learning (2020)

Abstract: This paper describes our system submitted to task 4 of SemEval 2020: Commonsense Validation and Explanation (ComVE) which consists of three sub-tasks. The task is to directly validate the given sentence whether or not to make sense and require the model to explain it. Based on BERT architecture with the multi-task setting, we propose an effective and interpretable “Explain, Reason and Predict” (ERP) system to solve the three sub-tasks about commonsense: (a) Validation, (b) Reasoning, and (c) Explanation. Inspired by cognitive studies of common sense, our system first generates a reason or understanding of the sentences and then choose which one statement makes sense, which is achieved by multi-task learning. During the post-evaluation, our system has reached 92.9% accuracy in subtask A (rank 11), 89.7% accuracy in subtask B (rank 9), and BLEU score of 12.9 in subtask C (rank 8).

## introduction how to integrate common sense into natural language models is attracting more and more attention. common sense, as ordinarily conceived, present themselves as the aspect of the grammar of expressions and sentences on which their semantic properties and relations depend @xcite . and a critical difference of text understanding between humans and machines lies in the fact that humans can access commonsense knowledge while processing text, which helps them draw inferences about effects that are not mentioned in a paragraph. therefore, it's a fundamental question on how to validate whether a system has a commonsense capability, and more importantly, let the system explain how it reasons using hidden facts. existing benchmarks measure commonsense knowledge indirectly and without explanation. also, existing datasets test common sense indirectly through tasks that require extra knowledge, such as co-reference resolution, or reading comprehension. they verify whether a system is equipped with common sense by testing whether it can give a correct answer to make the complete sentence reasonable. however, there are some restrictions on such benchmarks. first, they do not provide a straight quantitatively standard to measure sense masking capability. second, they do not explicitly identify the key factors required in a sense-making process. and also, they do not need the model to explain why it makes that prediction. common sense reasoning require the agent or the model to utilize a world knowledge to take inferences or deep semantic understanding, not only the pattern recognition. some empirical analysis has been done previously for common sense reasoning, mainly focus on the form of question answering (qa) @xcite . but question-answering is hard to directly evaluate the commonsense in contextualized representations. and there has been few work investigating commonsense in pre-trained language models @xcite , such as elmo @xcite and bert @xcite . introduced by @xcite , sense-making is a task to tests whether a model can differentiate sense-making and non-sensemaking statements. specifically, the statements typically differ only in one keyword which covers nouns, verbs, adjectives, and adverbs. there are two existing approaches that can address this problem, one simple way is to use more commonsense knowledge can be learned from larger training sets @xcite . on the other hands, some works @xcite focus on effectively utilizing external, structured commonsense knowledge graphs, such as conceptnet @xcite and comet @xcite . insipred by previous works, more researchers are trying to fuse commonsense knowledge and language model @xcite , and apply them to downstream tasks @xcite . recently, a new hybrid approach has been proposed for common sense reasoning @xcite . the core idea behind it is multi-task learning @xcite , which has been widely applied in natural language tasks @xcite . but existing work in this area has been frustratingly slow, and much of the work is completely theoretical. the field might well benefit if commonsense argumentation were systematically described and evaluated. to tackle it, this system focuses on a benchmark to directly test whether a system can differentiate sentences that make sense from those that do not make sense. our results indicate that pre-trained models are not able to demonstrate well on the benchmark, and some remaining cases demonstrating that human level is not achieved yet. thus, we design a new procedure to handle the commonsense challenge inspired by human cognition. it firstly explain its understanding of the given sentences by a language model, and induce the hidden common sense fact. and then, the explanation is used as a supplementary input to the prediction module. still, we believe that our approach also can be applied to more challenging data sets. the organization of this paper is as follows: in section 2, we introduce the basic information about pre-trained language model and task definition. we then describe the framework of our model in section 3. empirical results are given and discussed in section 4. and then we provide more exhaustive analysis for some bad cases that appeared at our experiment in section 5. finally, we conclude this survey and in section 6. ## models our proposed erp system first generates (explain) its understanding of the given sentences by a language model, and then the explanation is used as a supplementary input to the prediction module. for subtask a the input is a sentence pair s 1 and s 2, and the input is the gainst-common-sense statement s for subtask b. subtask c is an explanation generation task and in this way, we could explore common-sense reasoning in two settings 1) explain-and-then-predict and 2) predict-and-then-explain to evaluate the effectiveness of our erp system. therefore, we illustrate the erp system consecutively for different sub-tasks in sections 3.1 and 3.2. the architecture of our model is shown in figure 1 , the input x represents sequences (either one sentence or stacked sentences), and then for each token in this sequence is constructed by summing the corresponding token, segment and position embeddings. then the semantic encoder map the input token into a vector in word-level (token-level), the transformer encoder captures the contextual information in sentence-level via the self-attention mechanism. after we get the contextual embedding vector, we use task-specific layer to apply downstream tasks, we use text classification layer here for both of task a and task b. we choose to introduce subtask a and subtask b first, and followed by subtask c for intuitive understanding, but for the competition, the organizer release the datasets of subtask a, subtask c, and subtask b in turn, which supports our erp system. ## experiment it is important to make it clear that all our experiments are conducted which meet the requirement of the competition. we can not use the dataset which is not released during the formal competition which means we can not use subtask b data for subtask a and subtask c, because subtask b is released at last, etc. ## analysis despite the strong performance of our model, it still fails to detect some samples at subtask a and subtask b, and few sentences generated by our model can not well explain why the given sentence does not make sense. an in-depth analysis of these samples shows that they can be clustered into some classes. ## conclusion in this paper we present our model on the task of commonsense validation and explanation (comve) in semeval-2020. we explore multi-task learning to jointly learn how to inference the hidden common-sense fact and do common-sense reasoning with the roberta model and achieved competitive results. our result analysis indicates that our explain, reason and predict approach helps improve the performance of roberta and have a strong reasoning capability. the biggest regret in this competition is that we did not incorporated with world knowledge by introducing some knowledge graphs. due to implicit common sense knowledge and different explanation perspectives, we still need more efforts on the model architecture and find more elegant ways to inject knowledge. our positive results point to future work in extending the erp approach to a variety of other types of common sense reasoning tasks.