PALI at S em E val-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts (2022)

Abstract: This paper describes our system used in the SemEval-2022 Task 7(Roth et al.): Identifying Plausible Clarifications of Implicit and Under-specified Phrases. Semeval Task7 is an more complex cloze task, different than normal cloze task, only requiring NLP system could find the best fillers for sentence. In Semeval Task7, NLP system not only need to choose the best fillers for each input instance, but also evaluate the quality of all possible fillers and give them a relative score according to context semantic information. We propose an ensemble of different state-of-the-art transformer-based language models(i.e., RoBERTa and Deberta) with some plug-and-play tricks, such as Grouped Layerwise Learning Rate Decay (GLLRD) strategy, contrastive learning loss, different pooling head and an external input data preprecess block before the information came into pretrained language models, which improve performance significantly. The main contributions of our sys-tem are 1) revealing the performance discrepancy of different transformer-based pretraining models on the downstream task; 2) presenting an efficient learning-rate and parameter attenuation strategy when fintuning pretrained language models; 3) adding different constrative learning loss to improve model performance; 4) showing the useful of the different pooling head structure. Our system achieves a test accuracy of 0.654 on subtask1(ranking 4th on the leaderboard) and a test Spearman’s rank correlation coefficient of 0.785 on subtask2(ranking 2nd on the leaderboard).

## introduction cloze tasks have become a standard framework for evaluating various discourse-level phenomena in nlp. some prominent examples include the narrative cloze test @xcite , the story cloze test @xcite , and the lam-bada word prediction task @xcite . in these tasks, nlp systems are required to make a prediction about the filler of a cloze that is most likely to continue the discourse. however, these existing cloze tasks focus on the accuray of choosen fillers, ignore evaluating the absolute quality of all possible predictions. the goal of semeval 2022 task7 is to evaluate the ability of nlp systems to distinguish between plausible and implausible clarifications of an instruction. the task is formulated as a complex cloze task, which involve two sub tasks. in sub task 1, for each(sentence, filler) pair, nlp system need to classify four fillers into plausible, implausible or neutral and the evaluating indicator is accuracy. in sub task 2, for each pair, nlp system need to predict scores for five fillers and the evaluating indicator is spearman's rank correlation coefficient. since 2018, nlp models have adopted the concept of pre-training on a diverse corpus of unlabelled text, followed by supervised finetuning on specific tasks. pretrained models are built to simulate anthropomorphic learning, wherein existing knowledge can be adapted to new tasks without doing any training on these tasks from scratch -a requirement of traditional machine learning models. by now, these pre-trained large language models such as bert @xcite , roberta @xcite ,xlm-roberta @xcite , deberta @xcite has been widely used to solve all kinds of language understanding tasks. additionally, fine-tuning self-supervised pre-trained models has significantly boosted state-of-the-art performance on natural language processing (nlp) tasks. many evidence showing models with pretrained commonsense knowledge can be well applied in the field of cloze task @xcite , because cloze task needs commonsense language knowledge and genera language knowledge. additionally, there are many training tricks can be used to improve the performance and generalization ability of the large pre-trained language models. first, adding contrastive learning loss in supervised task, such as ntxent loss @xcite . second, in case of pretrained language models' catastrophic forgetting in funtuning period, we set different learning rate and weight decay rate for different pre-trained language model layers @xcite . third in order to get the best sentence embedding, trying different pooling head is necessary. inspired by these discoveries, we designed two nlp system for sub-task1 and sub-task2. as showing in figure 1 , this task is a cloze task, using fillers(filler1 to filler5) to insert a blank at the position in the text(e.g. screw each stringer to ___ the deck frame with a drill, use l-brackets and deck screws to attach the stringers to the deck.). additional, article title(), section header,previous and fellow-up sentence, resolved pattern are given. sub-task 1 is a classification sub-task, the evaluation metrics is overall accuracy. according to rules, converting each real-valued gold score to a class label as follows: • @xmath0.5 : implausible • 2.5 < @xmath1: neutral • @xmath2: plausible subtask2 is a regression subtask, our system need to predict each instance's gold plausibility score. the submissions will be scored based on spearman's rank correlation coefficient which compares the predicted plausibility ranking over all test instances with the gold ranking.