U n I mplicit Shared Task Report: Detecting Clarification Requirements in Instructional Text (2021)

Abstract: This paper describes the data, task setup, and results of the shared task at the First Workshop on Understanding Implicit and Underspecified Language (UnImplicit). The task requires computational models to predict whether a sentence contains aspects of meaning that are contextually unspecified and thus require clarification. Two teams participated and the best scoring system achieved an accuracy of 68%.

## introduction the goal of this shared task is to evaluate the ability of nlp systems to detect whether a sentence from an instructional text requires clarification. such clarifications can be critical to ensure that instructions are clear enough to be followed and the desired goal can be reached. we set up this task as a binary classification task, in which systems have to predict whether a given sentence in context requires clarification. our data is based on texts for which revision histories exist, making it possible to identify (a) sentences that received edits which made the sentence more precise, and (b) sentences that remained unchanged over multiple text revisions. the task of predicting revision requirements in instructional texts was originally proposed by @xcite , who attempted to predict whether a given sentence will be edited according to an article's revision history. the shared task follows this setup, with two critical differences: first, we apply a set of rules to identify a subset of edits that provide clarifying information. this makes it possible to focus mainly on those edits that are related to implicit and underspecified language, excluding grammar corrections and other edit types. since the need for such edits may depend on discourse context, a second difference is that we provide context for each sentence to be classified (see ## task and data in our task, sentences from instructional texts are provided in their original context and systems need to predict whether the sentence requires clarification. we define a clarification as a type of revision in which information is added or further specified. systems participating in the shared task are required to distinguish between sentences that require clarification and sentences that do not. for simplicity, we assume all sentences that remained unchanged over multiple article-level revisions (until the final available version) to not require clarification. based on this assumption, we create a class-balanced data set for our task by selecting for each sentence that requires clarification exactly one sentence that does not require clarification. in the following, we provide details on the collection procedure and an annotation-based verification thereof as well as statistics of the final data set. ## participants and results two teams registered for the shared task and submitted predictions of their systems: wiriyathammabhum (2021) and @xcite . wiriyathammabhum approached the task as a text classification problem and experimented with different training regimes of transformer-based models @xcite . ## discussion the results of the participating teams and the logistic regression baseline provide some insights regarding the task posed and the data sets provided. task. the results suggest that it is generally possible to predict whether a sentence requires clarification and models can pick up reliable patterns for most types of revision. in fact, the per-type results shown in on the other end, we find that required insertions of modifiers, quantifiers and modal verbs are hard to predict. in fact, the systems only identify up to 56% of such cases, which is only slightly better than the performance of a random baseline (50%). one reason could be that commonsense knowledge plays an important role in such clarifications. data. it is worth noting that the distribution of different revision types is not balanced and the overall results are skewed accordingly. in almost half of the test sentences that require clarification, the edit involved the insertion of an adverbial or adjectival modifier (49%, 840 out of 1,707). predicting the need for such edits is particularly difficult because they often add only subtle and context-specific information. replacements of pronouns form the second most-frequent clarification type in our data (23%, 398/1707). both participating systems were able to identify over 92% of sentences that require such a replacement. the remaining cases are distributed as follows: insertions of optional verb complements (15%, 262/1707), insertions of quantifiers and modal verbs (10%, 166/1707) and replacements of 'do' as a main verb (2%, 41/1707). one potential reason for the differences in results between the test data and the out-of-domain data is that revision types are distributed differently as well. in fact, the edits of sentences that require clarification in the out-of-domain data almost always involve the insertion of an adverbial/adjectival modifier or an optional complement (82%, 18/22). insights from participants. in addition to our observations, the system descriptions also report a number of interesting findings. for instance, ruby et al. found that pronouns requiring replacement are often denoting a generic referent or a type of individual, rather than a specific entity. based on this observation, they perform several experiments in which they first identify pronouns that should potentially be revised and then they combine representations of the identified pronouns with a sentencelevel system to generate predictions. a more technically motivated approach is taken by wiriyathammabhum, who build on the observation that the distribution of sentence labels (re-quiring revision or not) is generally unbalanced and that revised versions of sentences that required clarification may be viewed as instances of sentences that do not require further clarification. both participants discuss interesting approaches to the shared task and show interim results on the training/development sets. for details, we refer the interested reader to the system description papers @xcite . ## conclusions two teams participated in our shared task on predicting the need for clarifications, with the top performing system achieving an accuracy of 68.4%. perhaps unsurprisingly, the main takeaway from both systems is that transformer-based models pose a strong baseline for future work.