What Can We Learn from Noun Substitutions in Revision Histories? (2020)

Abstract: In community-edited resources such as wikiHow, sentences are subject to revisions on a daily basis. Recent work has shown that resulting improvements over time can be modelled computationally, assuming that each revision contributes to the improvement. We take a closer look at a subset of such revisions, for which we attempt to improve a computational model and validate in how far the assumption that ‘revised means better’ actually holds. The subset of revisions considered here are noun substitutions, which often involve interesting semantic relations, including synonymy, antonymy and hypernymy. Despite the high semantic relatedness, we find that a supervised classifier can distinguish the revised version of a sentence from an original version with an accuracy close to 70%, when taking context into account. In a human annotation study, we observe that annotators identify the revised sentence as the ‘better version’ with similar performance. Our analysis reveals a fair agreement among annotators when a revision improves fluency. In contrast, noun substitutions that involve other lexical-semantic relationships are often perceived as being equally good or tend to cause disagreements. While these findings are also reflected in classification scores, a comparison of results shows that our model fails in cases where humans can resort to factual knowledge or intuitions about the required level of specificity.

## introduction instructional texts often go through multiple revisions. for their final form, the author has to ensure that it is clear in which sequence described actions need to be performed and in what manner they have to be executed. a collection of revisions of instructional texts can be found, for instance, on the community-edited platform wikihow. in previous work using wikihow revision histories, we showed that improvements in texts can be modeled computationally under the assumption that revised sentences are better than their predecessors @xcite . this assumption seems intuitive for edits that correct or clarify parts of a sentence (e.g., in terms of grammar/factual mistakes). in the analysis, however, we also found a number of edits that involve paraphrases or more subtle differences in terms of specificity/genericity. in these cases, it remains unclear in how far edits actually represent improvements. this problem notably exists also in other work on textual revisions which aim to develop a computational model, including studies on wikipedia @xcite @xcite and argrewrite @xcite . in particular, these studies neglect two important aspects: first, they merely categorize edits by how they change a sentence (e.g., information deletion/addition, link creation/deletion), without specifying if or in how far the change represents an actual improvement. secondly, they provide limited or no insight about how the content of a revised sentence is semantically related to the content of the original sentence. such information could facilitate the understanding of why a change is perceived as a potential improvement. for instance, it might be that a sentence is perceived as better because it uses more specific words (e.g., book/textbook, bean/soybeans, work/homework) or that it fits better into the context in terms of textual cohesion. ## related work there is only one related study using wikihow @xcite , which we built upon (see section 3). therefore, we discuss studies within two related lines of research in this section: studies on revisions in wikipedia (section 2.1) and on revisions in persuasive texts (section 2.2). ## data description as a starting point for our studies, we use wikihowtoimprove @xcite , a corpus of around 2.7 million sentences and their revisions. in the following, we use the term revised version and variable s r to refer to a new version of a sentence, resulting from one or multiple edits. for the first version of a sentence, we use the term base version and variable s b . in this work, we only use the base and last revised version of a sentence, dropping all intermediate versions. we further restrict our studies on a subset of the data by considering only pairwise versions of a sentence such that: (1) s b and s r have the same number of tokens (@xmath0, (2) at least one noun in s b has been replaced by another noun in s r , i.e., noun-to-noun substitutions and (3) the only edits are noun-to-noun substitutions. foot_1 this subset contains 261 325 noun-to-noun substitutions, spread over 240 157 pairs. since nouns can be used to refer to entities, concepts, and events, the resulting set allows us to assess a broad range of lexicalsemantic phenomena that differ in s b and s r . furthermore, we suspect that the usage of nouns in a given sentence is related to the usage of nouns in the surrounding sentences, which makes it possible to study the selected subset as cases where edits may have been triggered by context. in practice, our studies focus on sentence-level edits, that is, we distinguish only the base and revised sentence while keeping the context (i.e., the surrounding sentences) the same. nonetheless, we provide the context of the base sentence during annotation and we derive contextual features from it for classification. we characterize the pairs s b , s r by the lexical relationships that hold between the nouns note that we can easily identify the substitutions by their position in the sentence. for our characterization, we use the paraphrase database (ppdb) @xcite , which contains 100 million paraphrases. specifically, each entry in the database contains a label that describes the entailment relationship between the phrases: equivalent (synonyms), independent (not related), exclusion (antonym), otherrelated (related but not by entailment), reverseentailment (@xmath1. the ppdb contains lexical, phrasal and syntactic paraphrases and contains different sizes. we use the biggest set (xxxl, 7 million instances) of lexical paraphrases (single word to single word) foot_2 to label as many noun-to-noun substitutions as possible. we find that 55 945 out of 261 325 noun-to-noun substitutions (21%) in the 240 157 pairs occur in the ppdb. the substitutions are distributed over 54 182 out of 240 157 pairs (23%), which indicates a fair coverage. the frequency distribution of the noun modifications (base formatted in strikethrough/ revised underlined) is as follows: ## classification experiments in this section, we describe a set of classification experiments that we conducted using the 54 182 pairs described in section 3. for these experiments, we divide the data by article into a train (@xmath2, ∼80%), development (@xmath3, ∼10%) and test set (@xmath4, ∼10%). 3 we address two settings. firstly, we train a classifier to identify for a sentence if it belongs to the class of revised versions. we call this setting sentence in isolation. our hypothesis is that revised versions can be distinguished from base versions by modeling differences in noun choice within the sentence. secondly, we develop classifiers that also take the base context into account. that is, the classifier can make use of features of the sentence to be classified as well as features derived from the sentences surrounding the sentence in its base version. for this setting, we extract the sentences before and after s b using a window of five sentences. we denote the previous sentences as c p and the following sentences as c f . we use the best classifier from the first setting as a baseline for the second setting, henceforth sentence in context, and our hypothesis is that features derived from the sentence context can improve the identification of revised versions by means of modeling factors such as cohesive ties, which are known to contribute to the readability of a text @xcite . ## annotation study previous work introduced the assumption that revisions represent improvements that can be made to a base version of a sentence. in our classification experiments, we relied on the observation that the noun choices in the base and revised version of a sentence can be distinguished systematically, thereby making implicit use of this assumption as well. in this annotation experiment, we test in how far this assumption actually reflects human judgements. for this purpose, we ask human annotators to identify which of two versions of a sentence they perceive as better or if both versions are equally good in their opinion. we present each pair of sentence versions within the base context c p and c f . each pair consists of a sentence in its base version and in its last revised version. for the annotation, we randomize the ordering of both versions in order to hide the identity from the annotators. this procedure allows us to verify in an unbiased way whether annotators indeed label the revised sentence as being better or not. in addition, the annotation study provides us valuable insights regarding the difficulty of the task at hand. ## analysis in this section, we compare classifier predictions and human judgements. as discussed in section 4, our best classifier can distinguish the base and revised version of a sentence with an accuracy of close to 70%. in section 5, we have seen that annotators do not always perceive the revised sentence as the better version of a sentence. thus, we examine in how far classifier predictions overlap with human judgements. for this analysis, we use the 314 pairs of sentences from the development set that were annotated identically by both annotators. for a direct comparison, we post-process the model predictions to also include a category "equally good", which comprises all cases where the model's output scores for both versions of a sentence correspond to the same label (e.g., score(s b ) = 0.6 and score(s r ) = 0.7). in the remaining cases, we treat the model's prediction as "revised is better" if score(s r ) >score(s b ) and "base is better" if score(s b ) >score(s r ). ## conclusion in this work, we examined a subset of revisions in wikihow, namely substitutions that involve lexically related nouns, in order to assess the role of lexical semantics in annotating and modeling revisions. in our computational experiment, we found that differences between base and revised sentence can be modeled with high accuracy (> 65%) and that context-based features regarding cohesion and coherence can further improve this result. in an annotation study, we verified that humans identify the revised sentence as better when it improves the fluency. however, when pairs with noun substitutions are closely related, such as by synonymy or antonymy, annotators tend to label both sentences as equally good. in some cases, our annotators had to rely on their intuitions and background knowledge to select whether one sentence is better than another. yet, we have also seen a high number of disagreements, showing that the assumption "revised is better" does not always apply or is at best subjective in some cases. in a final study, we observed that model predictions are largely in line with human annotations when the revised sentence improves the fluency by correcting mispelled words. in contrast, we found our best model to struggle with pairs that involve noun substitutions with high semantic relatedness and cases where humans can resort to background knowledge or intuitions. in future work, we plan to extend our model and analyses to take into account further information, including measures and features regarding ambiguity and document structure.