S em E val-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts (2022)

Abstract: We describe SemEval-2022 Task 7, a shared task on rating the plausibility of clarifications in instructional texts. The dataset for this task consists of manually clarified how-to guides for which we generated alternative clarifications and collected human plausibility judgements. The task of participating systems was to automatically determine the plausibility of a clarification in the respective context. In total, 21 participants took part in this task, with the best system achieving an accuracy of 68.9%. This report summarizes the results and findings from 8 teams and their system descriptions. Finally, we show in an additional evaluation that predictions by the top participating team make it possible to identify contexts with multiple plausible clarifications with an accuracy of 75.2%.

## introduction understanding texts in natural language requires that both explicit text components as well as implicit references and relationships are interpreted correctly. this applies in particular to instructional texts, which demand a clear understanding of individual instruction steps in order to reach the desired goal. possible uncertainties should therefore already be clarified in the text. in principle, such clarifications can also be generated automatically. in that case, however, it will be necessary to investigate the circumstances under which a clarification is plausible and unambiguous. as a first step towards such an investigation, this shared task evaluates the ability of nlp systems to distinguish between plausible and implausible clarifications of an instruction. inspired by the success of previous cloze-based evaluations (see section 2), we set up our task as a cloze task, in which clarifications are presented as fillers and systems have to choose a hair salon (1) check ratings of different salons. research in linguistics and psychology has shown that individuals use language differently @xcite . in particular when it comes to implicit and underspecified language, individual differences can also lead to different interpretations @xcite . as a result, worst case scenarios include medical instructions being followed incorrectly or news being passed on inaccurately. in view of the fact that language is inherently ambiguous, however, it is neither sensible nor expedient to produce clarifications for all occurrences of underspecification. avoiding worst-case scenarios therefore goes beyond ranking individual clarifications by plausibility and must take into account whether multiple (incompatible) clarifications are perceived as plausible, thus reflecting possible misunderstandings. we discuss our task and data in more detail in sections 3 and 4, respectively. the participating teams are summarized in section 5 and their results on the task and additional evaluations in section 6. ## related work cloze tasks have become a standard framework for evaluating various discourse-level phenomena in nlp. some prominent examples include the narrative cloze test @xcite , the story cloze test @xcite , and the lambada word prediction task @xcite . in these tasks, nlp systems are required to make a prediction about the filler of a cloze that is most likely to continue the discourse. however, it is not always clear whether exactly one likely filler exists. evaluations typically circumvent this issue by requiring systems only to distinguish between a correct and an incorrect filler, or by evaluating predictions only with a relative measure. both of these options ignore the more general challenge that multiple fillers can be plausible. our shared task addresses this challenge explicitly, by requiring systems to classify different clarifying fillers as either plausible or implausible. this is a natural extension of previous cloze tasks to discourse contexts in which multiple interpretations are plausible. this extension makes it possible to evaluate in how far nlp systems can reflect cases of underspecification and uncertainty as well as possible sources of misunderstanding. our task is based on manual text revisions that can be traced through revision histories and by which possible needs for clarification can be identified. thus, the task follows a number of existing research contributions that deal with text revisions. a number of previous works examine reasons for and types of revisions in, for example, wikipedia @xcite @xcite and the essay-based corpus argrewrite @xcite @xcite . in this work, we use revision histories of instructional texts because clarifications seem particularly relevant in this domain. the starting point of our task is the data set wik-ihowtoimprove @xcite , which comprises revision histories of more than 250,000 how-to guides from the online platform wikihow. 2 in our own previous work, we investigated the extent to which these histories are useful for exam-2 www.wikihow.org ining textual improvements @xcite , predicting revision requirements @xcite , modeling cases of lexical vagueness @xcite , and resolving implicit references @xcite . the last two studies in particular have shown that wikihow-toimprove is well suited as a resource for studying clarifications of semantic phenomena. we describe one of these studies and how we build upon it in more detail in section 3. ## task and background the general idea of the present shared task is to use revisions of english-language instructional texts as a basis to identify potential clarifications and to rate them regarding their plausibility. we assume that at least certain cases of clarifying revisions follow patterns that can be recognized automatically, by comparing the text before/after revision. an example of such a pattern is the insertion of a nominal phrase mentioned in context that makes an implicit reference explicit (see the focus on insertions allows us to consider clarifications as solutions to a cloze test, since the revision always fills in a text segment that previously was not present. compared to previous cloze tasks, we do not assume that the revision observed is always unique and plausible. instead, we also consider alternative clarification options and obtain plausibility judgments for all options. as background, we first summarize findings from a pilot study that we conducted before setting up the present task ( §3.1). based on this, we then describe the settings of the shared task ( §3.2). ## data we closely follow the three steps of our pilot study, described in section 3.1, to construct the data for this shared task. our starting point is the dataset wikihowtoimprove @xcite , a resource of sentence-level revisions and their contexts based on wikihow. in the first two steps, we create relevant data from this resource automatically; in the final step, we collect manual annotations to form a gold standard. in step 1, we apply a pattern-based approach to identify revisions that involve insertions that serve specific clarifying functions ( §4.1). in step 2, we use transformer-based language models to produce sets of alternate clarifications that may or may not be compatible with an observed insertion ( §4.2). in step 3, we collect human plausibility judgements on each clarification independently ( §4.3). ## participants a total of 21 users participated in the codalab competition set up for the shared task and 8 teams submitted system description papers. an overview of the best model by each team is shown in in addition to fine-tuning a single or multiple transformer models in an ensemble, some teams have taken additional steps to adapt their system to the task. we summarize some of these steps below. consideration of phenomena. at least two teams took into account that the data set consists of four phenomena that were identified using different patterns (pattern-aware): pali used the phenomenon description that applies to a classification instance as additional model input; x-pudu developed an ensemble architecture that consists of different individual models and hyperparamters for each phenomenon. adapted loss functions. several teams adapted the loss functions of their models to better account for various properties of the task. this includes the use of classification and regression based loss functions in a multi-task learning set-up (multi-loss) as well as the use of specific loss functions that consider the ordinal nature of labels (ordinal regression) or differences in label distributions (class weighting) in the classification task. unsupervised components. given the similarity of our task to general cloze tasks, several teams experimented with models that were merely selfsupervised and not fine-tuned on task-specific training data. in case of one team, hw-tsc, such an unsupervised component is also part of the ensemble model that produced the best results. ## results and discussion the results for task 1 and 2 are shown in the human upper bound has an accuracy of 79.4%, indicating that the task is challenging and potentially involves a number of disagreements. the winning team of the competition, x-pudu, achieves an accuracy of 68.9%, only 10.5 percentage points below the human upper bound. the results of all teams lies substantially above a naive majority class baseline of 39%. all teams but one ## conclusion in this paper, we presented the task, data, participating systems, and results of the shared task on clarifying implicit and underspecified phrases in instructional texts. our motivation for this task was to explore the possibility of testing different clarifications for plausibility. in particular, we were concerned with the question of whether two or more clarifications can be plausible and whether such cases can be detected automatically. to create a suitable dataset, we worked with and identified a set of revisions with manual clarifications, automatically generated possible alternatives, and then collected human plausibility ratings. in total, 21 users participated in our shared task. we summarized the systems and results of 8 teams that submitted descriptions of their systems. the best systems from each group have in common that they are based on transformer architectures or combine them in an ensemble. the best system achieved 68.9% accuracy, only 10.5 percentage points below a human upper bound. in additional evaluations, we have shown that an accuracy of up to 75.2% is achieved with respect to the detection of multiple plausible clarifications. the results show that the presented task is a difficult one, but that many cases can already be modeled well by current state-of-the-art methods. there is further room for improvement with respect to both the data set and models: with respect to the data, it should be noted that the training set with less than 20k instances is relatively small and that there are many instances with a underspecified neutral label (36%). on the model side, we found that the participating teams make complementary contributions that may allow for additional improvements in combination. one shortcoming of the task as presented and performed is that we only considered four forms of clarifications related to referring expressions. in addition, clarifications were assessed individually and judgements by different annotators were aggregated. in the long term, we believe that more forms of clarifications as well as individual differences regarding their plausibility need to be considered. finally, future work will have to investigate under which circumstances multiple different clarifications are actually incompatible and can thus reveal potential sources of misunderstanding.