A Computational Analysis of Vagueness in Revisions of Instructional Texts (2021)

Abstract: WikiHow is an open-domain repository of instructional articles for a variety of tasks, which can be revised by users. In this paper, we extract pairwise versions of an instruction before and after a revision was made. Starting from a noisy dataset of revision histories, we specifically extract and analyze edits that involve cases of vagueness in instructions. We further investigate the ability of a neural model to distinguish between two versions of an instruction in our data by adopting a pairwise ranking task from previous work and showing improvements over existing baselines.

## introduction instructional texts aim to describe the actions necessary to accomplish a task or goal, in as clear and concise a manner as possible. wikihow 1 is an extensive compendium of instructional guides for various topics and domains. any user may edit the articles, and wikihow collates these revision histories. the edit history of such informal instructional articles is a source of user-generated data that can help identify possible reasons and necessities for editing. wikihowtoimprove @xcite is a dataset that compiles revision histories for the analysis of linguistic phenomena that occur in edits of instructional texts, ranging from the correction of typos and grammatical errors to the clarification of ambiguity and vagueness. in this paper, we focus on cases of lexical vagueness, defined as "lexeme[s] with a single but nonspecific meaning" @xcite , which can potentially cause misunderstandings in instructional texts. specifically, we study vagueness based on the change in the main verb in the original and revised version of an instruction. we say that an instruction was vague if, upon revision, the revised 1 https://www.wikihow.com/ ## related work our paper focuses on revisions in wikihow for a specific linguistic phenomenon, namely vagueness. the motivation to use revision histories as corpora for nlp tasks was introduced by @xcite . the task of defining and categorizing edit intentions has been explored well for the wikipedia edits corpus @xcite . ## data creation, preprocessing, and analysis wikihow articles mostly contain instructions, but also include descriptions, explanations, and other non-instructional sentences that provide additional context. the wikihowtoimprove corpus (anthonio et al., 2020) is an unfiltered corpus of revision histories. therefore, we first need to extract those revisions where the original and revised versions are both instructional sentences, which can be done based on syntactic properties ( §3.1). we then use a framenet parser to determine the frames (and their relationships) evoked by the root verb in the original and revised version of an instruction ( §3.2). the final extracted data consists of only those revisions where the root verb has been modified to be more specific to the sentence. this extracted corpus consists of 41,615 sentences. ## pairwise ranking experiments in this section, we investigate if a neural model can distinguish between the original and revised version of the same instruction. we describe a neural architecture that uses a joint representation designed for comparing two versions of a sentence before predicting an output. we compare our results to a standard bilstm-attention model used in previous work @xcite . ## conclusion in this paper, we extracted a corpus of clarifications of instructions from the wikihowtoimprove corpus. we described a methodology for extracting version pairs of a sentence that are both instructional. we then identified cases in which a revision has clarified a vague instruction by analyzing the relationship between the frames evoked by the 'original' verb and the 'revised' verb. in our experiments, we adopted a simple pairwise ranking task, in the same vein as performed by @xcite on the entire wikihow-toimprove dataset. we extended a simple bilstm architecture with a joint component and explored different embeddings methods, observing that both modifications lead to improvements over baselines presented in previous work. we hope that our methodology of extracting linguistically interesting cases of revisions from a noisy dataset can be extended to more phenomena and other corpora in future work. this direction has the potential of paving the way for developing automated revision and editing methods beyond typo, style, and grammar correction.