JBNU - CCL ab at S em E val-2022 Task 7: D e BERT a for Identifying Plausible Clarifications in Instructional Texts (2022)

Abstract: In this study, we examine the ability of contextualized representations of pretrained language model to distinguish whether sequences from instructional articles are plausible or implausible. Towards this end, we compare the BERT, RoBERTa, and DeBERTa models using simple classifiers based on the sentence representations of the [CLS] tokens and perform a detailed analysis by visualizing the representations of the [CLS] tokens of the models. In the experimental results of Subtask A: Multi-Class Classification, DeBERTa exhibits the best performance and produces a more distinguishable representation across different labels. Submitting an ensemble of 10 DeBERTa-based models, our final system achieves an accuracy of 61.4% and is ranked fifth out of models submitted by eight teams. Further in-depth results suggest that the abilities of pretrained language models for the plausibility detection task are more strongly affected by their model structures or attention designs than by their model sizes.

## introduction wikihow 1 is the largest how-to website with more than 300,000 articles and over 2.5m registered users that help user improve their knowledge of specific areas. however, these instructional articles have grammatical errors and ambiguous content that cause misunderstandings. to enhance the clarity of instructional texts, clarification is required as a revision that makes implicit elements explicit, resolves ambiguities, or replaces underspecified phrases with a clearer and more precise expressions. semeval-2022 task 7 @xcite evaluates the ability of an nlp system to distinguish between plausible and implausible clarifications of an instruction. the task is formulated as a cloze task in which clarification is presented as a filler in 1 https://www.wikihow.com/main-page a blanked sentence. given a context with a filler option (i.e., a sentence x and filler option o), the system should determine the plausibility of a sequence, that is whether a sequence is "plausible," "neutral," and "implausible." our work is motivated by the recent successes of pretrained language models such as bert @xcite , roberta @xcite , and deberta @xcite , which have effectively induced contextualized representations, achieving remarkable fine-tuning performance on downstream tasks. to explore the plausibility detection of clarifications in semeval-2022 task 7, we compare the plausibility detection abilities of three pretrained language models (bert, roberta, and deberta) in subtask a, multi-class classification. our main observations of the development set are highlighted as follows. by ensembling 10 different deberta-based models, our final submitted system achieves an accuracy of 61.4% on the test data and is ranked the 8th place among 21 systems foot_1 , that is, 5 th th place among the 8 teams who submitted their papers on subtask a. the remainder of this paper is organized as follows: section 2 presents related work. section 3 describes the system architecture in detail. section 4 describes the experimental settings, results, and analyses. our concluding remarks and a description of future work are presented in section 5. ## related work since the success of bert @xcite , it has been used for numerous natural language processing (nlp) tasks and has inspired the emergence of many other pretrained models. in roberta @xcite , dynamic masking in the masked language modeling (mlm) objective dynamically while revisiting the next sentence prediction owing to its uncertain effectiveness has achieved promising results on glue @xcite , race @xcite , and squad @xcite . deberta @xcite further advanced bert based on two major extensions, disentangled attention and enhanced mask decoder, by combining both the relative and absolute posi-tions of words. deberta v3 @xcite replaces the mlm objective in deberta with the replaced token detection (rtd) objective proposed by electra @xcite and further proposes gradient-disentangled embedding sharing to alleviate the tug-of-war problem between the generator and discriminator foot_2 as an improvement of the embedding sharing method used in electra. ## system description figure 1 presents the architecture of our system, which uses pretrained language models equipped with ensemble inference. ## conclusion in this study, we compare bert, roberta and deberta in semeval-2022 task 7 subtask a: multi-class classification. the results show that deberta presents the best performances with improved distinguishable representations. we assume that the substantial changes made to the model structure of deberta, such as disentangled attention, enhanced mask decoder, and rtd objective, would give deberta a significant advantage in the addressed task. our final submission, based on an ensemble model comprising 10 fine-tuned deberta-based models, achieved an accuracy of 61.4% on the test data. our proposed model is ranked fifth out of eight models of teams who reported their papers. in future studies, it would be worthwhile to explore other pretrained language models, such as electra, or models resulting from task-specific pretraining.