X - P u D u at S em E val-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications (2022)

Abstract: This paper describes our winning system on SemEval 2022 Task 7: \textit{Identifying Plausible Clarifications ofImplicit and Underspecified Phrases in Instructional Texts} . A replaced token detection pre-trained model is utilized with minorly different task-specific heads for SubTask-A: \textit{Multi-class Classification} and SubTask-B: \textit{Ranking} . Incorporating a pattern-aware ensemble method, our system achieves a 68.90% accuracy score and 0.8070 spearman’s rank correlation score surpassing the 2nd place with a large margin by 2.7 and 2.2 percent points for SubTask-A and SubTask-B, respectively. Our approach is simple and easy to implement, and we conducted ablation studies and qualitative and quantitative analyses for the working strategies used in our system.

## introduction the internet's ever-increasing size has made it easy to find instructional texts such as articles in wik-ihow 1 , on almost any topic or activity. regular revisions of these how-to manuals are necessary to ensure that instructions communicate the procedures required to attain a certain goal precisely. this shared task is introduced by @xcite , whose intention is to find ways to improve instructional texts, evaluate to what extent current nlp systems are able to handle implicit, ambiguous, and underspecified language, and go beyond the surface form of a text and take multiple plausible interpretations into account. thus, the proposed nlp systems should be capable of distinguishing between plausible and implausible clarifications of an instruction shown in figure ble, neutral, plausible) given the clarification foot_0 and its context. in this paper, we describe our winning system for both subtasks. we built our system based on a replaced token detection (rtd) task pre-training model. the idea is that the replaced token detection task is similar to this shared task which focuses on distinguishing semantically similar words/phrases. to close the gap the model trained between the pretraining phase and fine-tuning phase, we reused the pre-trained language modeling head during finetuning on task 7. then, two-layers mlps are applied on the mean-pooled hidden states of a clarification (filler) given the context. for subtask a, we utilized the cross-entropy loss for the multi-class classification. for subtask b, a sigmoid function was used to impose restrictions on the output of the system on a scale from 1 to 5. finally, we trained multiple models and aggregated the predictions with a pattern-aware ensemble strategy. our system achieved the best overall performance in the shared task with a 68.9% accuracy score (subtask a) and 0.807 spearman's rank correlation score (subtask b). the outcomes are promising for improving the clarification of instructional texts. ## background pre-trained models @xcite @xcite have achieved state-of-the-art results in various natural language processing (nlp) tasks. recent works @xcite @xcite have shown that more generalization ability and superior performance can be achieved by pre-training models with billion or trillion parameters. thus, we pursued the competitive pretrained models such as deberta @xcite and large-scale pre-trained models ernie @xcite whose effectiveness has been validated in the standard glue @xcite and su-perglue benchmark @xcite . however, in our initial experiment, we found that the aforementioned models, though have promising results on sentence-level or paragraph-level tasks, failed to distinguish the word/phrase-level semantically similar clarifications (fillers) in a given context. we believe the failure is due to the way these models were trained using a masked language modeling (mlm) task in the pre-training phase. mlm aims to map tokens with similar semantics to the embedding space that are close to each other instead of distinguishing them. based on the above finding, we believe what we need is a discriminator @xcite pre-trained via a replaced token detection (rtd) task which is more aligned with this shared task. in rtd, the discriminator needs to determine if a corresponding token is either an original token or a token replaced by the generator. formally, the loss function for the discriminator is as follows: ## method in this section, we will describe the strategies we used in our system in detail. in section. 3.1, the system is presented on how we formalize the data as input, basic modules, and task-specific design for each subtask. then, we describe the optimization object for each subtask (see section. 3.2). finally, we introduce a pattern-aware ensemble strategy to further boost the performance beyond a normal ensembled model in section. 3.3. ## conclusion we built a system for identifying plausible clarifications of implicit and underspecified phrases in instructional texts which is useful for improving the clarification of instructional texts. the system leverages the strength of a replaced token detec-tion pre-trained discriminator and therefore performs extremely well on this shared task with the same goal to distinguish semantically similar tokens. in particular, we proposed a pattern-aware ensembling strategy to aggregate multiple predictions separately based on the pattern when there is a label distribution difference among patterns. on semeval-2022 task 7, the system achieved the best performance in both subtasks. in future work, it's promising to incorporate the replace token detection task in a large-scale pretrained model with billion, or even trillion parameters.