Resolving Implicit References in Instructional Texts (2021)

Abstract: The usage of (co-)referring expressions in discourse contributes to the coherence of a text. However, text comprehension can be difficult when referring expressions are non-verbalized and have to be resolved in the discourse context. In this paper, we propose a novel dataset of such implicit references, which we automatically derive from insertions of references in collaboratively edited how-to guides. Our dataset consists of 6,014 instances, making it one of the largest datasets of implicit references and a useful starting point to investigate misunderstandings caused by underspecified language. We test different methods for resolving implicit references in our dataset based on the Generative Pre-trained Transformer model (GPT) and compare them to heuristic baselines. Our experiments indicate that GPT can accurately resolve the majority of implicit references in our data. Finally, we investigate remaining errors and examine human preferences regarding different resolutions of an implicit reference given the discourse context.

## introduction implicit language phenomena can be challenging for both human and machine processing. for example, references play a crucial role in instructional texts as they provide answers to questions such as which objects need to be used? if such references are not made explicitly, they might be clear to readers who have task-specific knowledge, but for others they might cause problems or misunderstandings. resolving such implicit references could improve clarity and prevent problems in discourse processing when multiple interpretations exist. in natural language processing, implicit references have been handled as part of existing tasks such as semantic role labeling of implicit arguments @xcite . implicit arguments are generally hard to model computationally because they do not show up in easy to ## implicit reference resolution task definition. we formally define the task of resolving implicit references as a generation task that requires the prediction of a reference s, given: performing this task requires a model to generate the sequence of tokens s 1 . . . s l for the reference s conditioned on the context c p , c f . in practice, the full task can be approached by first sampling candidate reference tokens r 1 . . . r l from a conditional probability distribution p (r i |c p , r 1 . . . r i-1 ) and then re-ranking the highest scoring candidates according to the full sequence probability p (c p , r 1 . . . r l , c f ). formulating the task in this way enables a direct application of language models and we demonstrate suitable baselines based on an auto-regressive language model in section 5. ## related work the task of resolving implicit references can be viewed as a modified version of implicit argument labeling. first studies on implicit argument labeling were conducted by @xcite and @xcite . @xcite collected a dataset by manually labeling implicit arguments of 10 different nominal predicates in nombank @xcite ), yielding about 1,000 instances. @xcite created a dataset through manual annotation of fictional text. their dataset contains more different predicates than previous studies, but is smaller in size. more recent studies make use of the two datasets and attempted to create additional training data artificially @xcite @xcite . many of them are based on co-reference and discourse salience, which we also use for our baselines. @xcite propose an unsupervised approach by aligning implicit arguments to semantic role labeling annotated data. @xcite generated large amounts of training data automatically using co-reference resolution. they also build a neural model based on argument fillers that occur multiple times in a narrative event chain. finally, there are also datasets with domain-specific annotations such as geographic-event roles @xcite and on recipes @xcite . another closely related task is zero anaphora resolution, which has been extensively studied in pro-drop languages such as chinese @xcite and japanese @xcite @xcite @xcite . a closely related study to ours is @xcite , who used language model probabilities as features. as a commonality, previous work addresses semantic arguments of predicates that are realized outside a local syntactic scope. our definition of implicit references subsumes such arguments, with the main difference that our task does not require the type of an argument or its semantic role to be specified. as a consequence, references in our task can fill one, none or multiple roles of different predicates. once the correct reference has been identified, our task additionally requires the generation of a referring expression. this task has been addressed separately in previous work, for instance, using rule-based approaches @xcite , feature-based machine learning @xcite @xcite , and deep neural networks @xcite . ## data the starting point for our data are revision histories from wikihow, in which we can find insertions of references that were implicit in earlier versions of a sentence. we use wikihowtoimprove @xcite , a resource derived from wikihow that consists of approximately 2.7 million sentences and their revisions. for our purpose, we extract sentences in which a reference was inserted during revision. most of the sentences in wikihow are only edited once (about 83%). in other cases, intermediate versions are mostly the result of stylistic refinements or typo corrections. therefore, we only make use of the final version of a sentence (henceforth revised sentence), which includes an inserted reference, and the original sentence, in which the reference is assumed to be implicit. as a result, each data point in our collection consists of a pair of two versions of a sentence, henceforth original-revised sentence pair. we describe our selection of implicit references in section 4.1 and present the data statistics in section 4.2. ## language model experiments in this section, we describe a set of experiments in which we investigate the use of a transformer-based language model for the task of resolving implicit references. in particular, we aim to answer the following research questions: can we find the manually inserted reference among the top completions predicted by a language model, and is it possible to select a correct prediction based on its fit in the sentence or paragraph context? we describe our experimental set-up in section 5.1 and our results in section 5.2. further analyses are provided in section 6. ## analysis in this section, we aim to answer two questions evoked by the results in section 5. first, we ask how fine-tuning and perplexity-based re-ranking improved the scoring of the top-100 generated sequences and what differences can be seen among the re-ranked top-10 sequences (section 6.1). secondly, we investigate the plausibility of the two highest ranked fillers generated by the model (section 6.2). the latter analysis provides us with insights regarding the existence of a single most plausible filler (or whether none/two fillers can be plausible) given the discourse context. in cases where the human-inserted reference is among the top-2, the analysis also makes it possible for us to find if/when the human-inserted reference is identified as the most plausible. ## conclusions in this paper, we introduced the task of resolving implicit references in instructional texts, which might be problematic for readers without prior knowledge of the instructed task. we approached the resolution of implicit references as a generation problem for which we leveraged original-revised sentence pairs from wikihow. the considered pairs contained an explicit reference in the revised sentence which was non-verbalized in the original sentence. our dataset is one of the largest datasets with implicit references and contains texts from the multiple different domains covered in wikihow. we showed that a pre-trained language model is capable of predicting the human-produced insertion in a majority of cases. the best-performing method, which combines a fine-tuned gpt model and perplexity-based re-ranking, achieved results up to 57.4% (top-1) and 80.8% (top-10). even without fine-tuning and re-ranking, 71.6% of the human-inserted references appeared in the top-10. we found sentence-level re-ranking useful to eliminate generated sequences that cause ungrammatical sentences and paragraph-level re-ranking to prioritize sequences that also occur in the preceding discourse. our analysis revealed that the human-inserted reference is commonly found to fit the discourse better than a model-generated alter-native. however, we also found cases where other completions were plausible. in the future, we will extend this study and take it as a starting point for examining potential sources of misunderstanding.