IIE - NLP - NUT at S em E val-2020 Task 4: Guiding PLM with Prompt Template Reconstruction Strategy for C om VE (2020)

Abstract: This paper introduces our systems for the first two subtasks of SemEval Task4: Commonsense Validation and Explanation. To clarify the intention for judgment and inject contrastive information for selection, we propose the input reconstruction strategy with prompt templates. Specifically, we formalize the subtasks into the multiple-choice question answering format and construct the input with the prompt templates, then, the final prediction of question answering is considered as the result of subtasks. Experimental results show that our approaches achieve significant performance compared with the baseline systems. Our approaches secure the third rank on both official test sets of the first two subtasks with an accuracy of 96.4 and an accuracy of 94.3 respectively.

## introduction natural language understanding (nlu) requires the systems can not only figure out the semantic of the text but also comprehend text with the constraint of commonsense knowledge about the world. the ability to identify the natural language statement against common sense and produce the explanation of its fault is the foundation towards realizing natural language understanding @xcite . the semeval-2020 task 4 provides a well-formed evaluation mission that aims to evaluate the capacity of the system on commonsense validation and explanation @xcite . the commonsense validation and explanation (comve) task is divided into three subtasks including validation, explanation selection, and explanation generation. we mainly focus on and participate in the first two subtasks. the goal of the first validation subtask (subtaska) is to inspect the ability of a system about distinguishing natural language statements that are against commonsense (i.e. false statements for short). the goal of the second explanation selection subtask (subtaskb) is to test if the system can correctly understand the reason for making against common sense. in subtaska, the challenge with distinguishing the false statements lies in that this kind of statement usually conforms to the linguistic structure in syntactic level but its meaning does not fit the general commonsense in semantic level. in subtaskb, the difficulty of selecting an appropriate explanation for false statements is that, albeit the candidate explanations are relevant to the content of the false statements, they may not contain the main reason to account for the false statements and will distract the system. to address the above challenges, we first formalize both subtasks as a type of multiple-choice question answering (qa) task. recently, the the large pre-trained language models (plms), such as gpt @xcite , bert @xcite , roberta @xcite , demonstrate its excellent ability in various natural language understanding tasks @xcite @xcite @xcite . moreover, according to recent research @xcite , they reveal that plms have already learned certain commonsense knowledge through pre-training with large scale corpus. hence, we not only resort to the plms as the contextual encoder to generate the representation of sentences but also consider the plms as knowledge storage which can implicitly provide commonsense knowledge during question answering. aiming at coping with the challenges mentioned previously, we devise the approaches for solving the two subtasks in the multiple-choice qa fashion with the following two guiding intention: (a) how to arouse and utilize the implicit knowledge of plms to commonsense validation and explanation; (b) how will the extension of the context help the system to select the correct explanation for the false statement. for the first point, we explore a prompt template-based approach to reconstruct the input to the encoder. in subtaska, we devise a prompt question and transfer this subtask into the multiple-choice qa style format where the statements are taken as candidate answers. in subtaskb, we reformat the false statement with a prompt template into a question to be answered with candidate explanation. the prompt is designed to activate the commonsense knowledge inner the pre-trained models, and it can be treated as a query to retrieve commonsense knowledge inside the plms. in addition, the prompt templates enrich the input of the plms to explicitly express the intention of the subtasks. for the second point, we propose to extend the prompt question in subtaskb with more context which can supply informative tips to locate the evidence of causing against common sense, i.e. the contrastive information between correct statements and false statements. this paper describes approaches for subtaska and subtaskb developed by the natural language processing group of institute of information engineering of the chinese academy of sciences. our contributions are summarized as the followings: (a) we employ the prompt template-based approaches on two subtasks to reconstruct original statements into the prompt to bring out the potential of the plms' commonsense knowledge; (b) we also explore the scoring-based approach to achieve the validation subtask; (c) experiments demonstrate that the proposed strategies achieve significant improvement compared with the plms baseline and we obtain the third-place in subtaska and subtaskb on the final official evaluation. in the following, we describe the approaches used for the two subtasks in section 2.1 and section 2.2 respectively. in section 3, we elaborate our settings of experiments and report the performance on the public development set and final hidden test set. in section 4, we analyze our approaches with cases. ## approaches before diving into the detail, we first present the description of symbols and the multi-choice based model which we use in both subtasks. formally, suppose there are five key elements in the two subtasks, i.e. {s 1 , s 2 , o 1 , o 2 , o 3 }. we suppose the s * denotes the ture and false statements, the o * denotes the candidate explanation for subtaskb. s 1 and s 2 are the inputs to the validation subtask while the false statement and o * are the inputs to the explanation subtask. and y @xmath0, 2} and y @xmath1, 2, 3} denote the labels of these two tasks respectively. a multiple-choice based qa model m consists of a plm encoder and a task-specific classification layer which includes a feed-forward neural network f (•) and a softmax operation. for each pair of question-answer, the calculation of m is as follow: ## conclusion in this paper, we present our approaches for participating in the semeval task on commonsense validation and explanation, which utilize the template to reconstruct the input with prompt information and inject additional context to provide contrastive information to improve the judgment and selection ability of systems. experimental results manifest that both strategies benefit to the final performance. moreover, the auxiliary probing experiments confirm that plms contain rich commonsense knowledge which can be mined to facilitate downstream tasks.