HW - TSC at S em E val-2022 Task 7: Ensemble Model Based on Pretrained Models for Identifying Plausible Clarifications (2022)

Abstract: This paper describes the system for the identifying Plausible Clarifications of Implicit and Underspecified Phrases. This task was set up as an English cloze task, in which clarifications are presented as possible fillers and systems have to score how well each filler plausibly fits in a given context. For this shared task, we propose our own solutions, including supervised proaches, unsupervised approaches with pretrained models, and then we use these models to build an ensemble model. Finally we get the 2nd best result in the subtask1 which is a classification task, and the 3rd best result in the subtask2 which is a regression task.

## introduction the rapid development of artificial intelligence has also been reflected in the field of nlp, and there have been many heavyweight achievements, such as word2vec @xcite , glove @xcite , transformer @xcite . natural language processing is an important branch of artificial intelligence. cloze tasks have become a standard framework for evaluating various discourse-level phenomena in nlp, which is an important field in artificial intelligence, many researchers have long been committed to the development of this field. some prominent examples include the narrative cloze test @xcite , the story cloze test @xcite , and the lambada word prediction task @xcite . cloze requires the testee to infer from the context, which is very difficult for machines. the goal of this shared task is to evaluate the ability of nlp systems to distinguish between plausible and implausible clarifications of an instruction. such clarifications can be critical to ensure that instructions describe clearly enough what steps must be followed to achieve a specific goal. this task was set up as a cloze task. however, different from regular cloze task, there may be zero or more than one correct candidates out of the five options. this presents new challenges for cloze systems. for subtask 1, it is a classification task that requires the system to classify five candidates into corresponding categories, which are plausible, neutral, or implausible, and the number of each category is not fixed. this means that there may be zero or more than one correct candidates out of the five options, and the same applies to the other two categories. this situation creates new challenges for cloze tasks. for subtask 2, it is a regression task ask annotators to rate for each clarification option whether it "makes sense in the given how-to guide" (on a scale from 1 to 5) to assess the plausibility of different clarification options. in this paper, we analyze the characteristics of the shared task and describe out contribution to this cloze task. we build an ensemble model with deberta-v3 @xcite , roberta-large @xcite , sbert @xcite , including supervised approaches and unsupervised approaches. our model had the 2nd best performance in the subtask1 (66.1% accuracy score) and the 3rd best performance in the subtask2 (77.4% ranking score). the results are encouraging for evaluating various discourse-level phenomena in nlp, although there is much room for improvement. the rest of this paper is organized as follows. section 2 introduce our approach for the shared task. section 3 shows the experimental results of our approach and do some analysis. in section 3, experimental results are compared and discussed. finally, the whole paper is summarized with a brief conclusion in section 4. ## system overview in this section, we first describe our data processing steps. we experimented with different ways of processing the data, trying to find the one that worked best for the task. and then, we discuss our solutions with pre-trained models for the shared task, including unsupervised approaches, supervised approaches, and an ensemble model. ## experimental results in the following experimental part, all the data used for the experiment adopts the data processing method we introduced in section 2.1. ## conclusion in this paper, we describe the identifying plausible clarifications of implicit and underspecified phrases shared task held within semeval-2022 and present the design, the data, the results, and the systems for the shared task. the participants of the shared task were provided a collection of revisions of instructional texts from the how-to website wikihow. the shared task is challenging, partly due to the relatively small training data and label noise. we develop an ensemble model of nlp to distinguish between plausible and implausible clarifications of an instruction, achieving the 2nd best performance in the subtask1 and the 3rd best performance in the subtask2. for some of the problems reflected in this task, such as data noise, non-identically distributed data, there is still a lot of research space. figure 7 : data in the original paragraph add in a few drops of clear nail polish and stir with a toothpick until there are no lumps. keep stirring until you get an even color and consistency. if the color too sheer, add some more eyeshadow. make sure that there are no clumps in the polish. if there are any clumps, break them up with the toothpick. if you don't do this, they will show up on your manicure and make it look lumpy.