Towards Modeling Revision Requirements in wiki H ow Instructions (2020)

Abstract: wikiHow is a resource of how-to guidesthat describe the steps necessary to accomplish a goal. Guides in this resource are regularly edited by a community of users, who try to improve instructions in terms of style, clarity and correctness. In this work, we test whether the need for such edits can be predicted automatically. For this task, we extend an existing resource of textual edits with a complementary set of approx. 4 million sentences that remain unedited over time and report on the outcome of two revision modeling experiments.

## introduction instructional texts have become an integral part of our daily lives, be it in the form of assembly instructions, product leaflets, troubleshooting guides, or board game manuals. a key property across all types of such texts is that they must be clear enough so that readers can actually achieve the goal described by the instructions. previous studies in computational linguistics have dealt with the clarity of specific types of instructions, such as route directions @xcite and software requirements @xcite . as an indicator for clarity, they relied either on successful execution in a virtual environment or on manual annotations of predefined ambiguity types. a large and more general dataset of instructional texts, wik-ihowtoimprove, has recently been introduced by @xcite . wikihowtoimprove consists of edits for about 2.5 million sentences derived automatically from revision histories of wikihow, a collaboratively edited platform of how-to guides. in a set of human and computational experiments, @xcite show that such edits are often made to clarify or correct a sentence and that the difference between an "older" and "newer" version of a sentence can be predicted computationally. we address two notable questions, using the work of @xcite as a starting point: in summary, we make the following contributions: task of predicting revision requirements, for which we provide a new dataset as well as initial results. ## data and models our experiments make use of wikiatomicedits, wikihowtoimprove and an extension of the latter that adds unrevised sentences ( §2.1). for computational modeling, we re-use the publicly available baselines 2 from @xcite and further test models based on bert ( §2.2). ## experiment 1: version distinction in wikihow and wikipedia the aim of the first experiment is to compare models on the task of distinguishing older and newer versions of a sentence between wikihow and wikipedia. we make use of the same general setup as previous work (see @xcite . our hypothesis for this experiment is that distinguishing sentence versions will be easier in wikihow than in wikipedia, because edits in the latter provide new information more often than refinements @xcite , whereas the content of wikihow is largely independent of world knowledge that may change over time. results. in comparison to the results reported in @xcite , we observe that the baseline models are approx. 2.5% less accurate. a possible explanation for this drop is that we removed (pre-wikihowtoimprove pick one band that has a large fan basis and . . . pick one band that has a large fan base and . . . it have much tricks and ways to interact with it. it has many tricks and ways to interact with it. plan out the details your story. plan out the details of your story. ## experiment 2: predicting revision requirements in wikihow the aim of this second experiment is to provide benchmark models for predicting whether or not a sentence requires revision. the previous experiment has shown that it is difficult to distinguish different versions of a sentence in wikiatomicedits ( §3). therefore, we perform this experiment only on wikihow. we make the simplifying assumption that all changes in wikihow's revision history are made for the better and therefore represent needed revisions to the original version of an article. thus, we treat all sentences that went through revision in you can even organize a elocution for that. (revised: . . . an elocution for that purpose.) it is very healthy way of fast frying minimal oil. results. as shown in the table, the bilstm model outperforms the naive bayes model by 16.88 percentage points in precision, but only achieves a recall of 51.86%. this result indicates that contextual information within the sentence is needed for precisely predicting revision requirements, but it is not sufficient to achieve good coverage. the bert model achieves the highest f 1 -score, outperforming the naive bayes and bilstm models by 4.39 and 7.67 percentage points, respectively. discussion. in a qualitative analysis of results, we find that all models are capable (to various degrees) of identifying grammar errors and sentences that are semantically implausible. a selected list of correctly classified example sentences are given in ## related work wikipedia revisions. revisions in wikipedia have been leveraged for various nlp tasks, such as spelling error correction @xcite @xcite , preposition error correction @xcite , paraphrasing @xcite , sentence simplification and compression @xcite , textual entailment recognition (zanzotto and pennacchiotti, 2010) and lexical simplification @xcite . within this framework, a number of studies have analyzed the type of edits that authors made @xcite @xcite @xcite and their intentions @xcite . these studies built further upon @xcite and @xcite . @xcite and @xcite performed multi-class classification to automati-cally detect edit types and edit intentions respectively. other text classification studies focused on a smaller set of revision intentions in wikipedia, such as @xcite who worked on bias/nonbias detection. attention has also been given to distinguish between vandalism and non-vandalism @xcite @xcite and between factual and fluency edits @xcite wikihow revisions. compared to wikipedia revisions, wikihow has received less attention in nlp. apart from @xcite , there is no other work that leveraged the revision history of wikihow articles. however, wikihow has been used for summarization @xcite and knowledge acquisition @xcite . others have also employed it to model procedure-specific relationships in sentences @xcite and underlying reasons for these relationships @xcite . related tasks. @xcite , in a related task, worked with revisions in argumentative essays from argrewrite @xcite . the authors trained a randomforest classifier to predict, given an original sentence and a revised one, if the revised sentence is better than the original. @xcite conducted a related study, which analyzed potential strength differences in originalrevised sentence pairs in academic writing using a qualitative approach. ## conclusions we demonstrated in an experimental comparison that it is easier to distinguish sentence versions computationally in wikihowtoimprove than in wikiatomicedits. we further introduced a new task of predicting whether a sentence requires revision and showed promising first results on specific types of revisions. as next steps, we plan to address further types of revisions and extend our experiments to document-level settings. fan zhang, homa b. hashemi, rebecca hwa, and diane litman. 2017. a corpus of annotated revisions for studying argumentative writing. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1568-1578. association for computational linguistics. fan zhang and diane litman. 2016. using context to predict the purpose of argumentative writing revisions. in proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1424-1430, san diego, california. association for computational linguistics. peng zhou, wei shi, jun tian, zhenyu qi, bingchen li, hongwei hao, and bo xu. 2016. attention-based bidirectional long short-term memory networks for relation classification. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 207-212. yilun zhou, julie shah, and steven schockaert. 2019. learning household task knowledge from wikihow descriptions. in proceedings of the 5th workshop on semantic deep learning (semdeep-5), pages 50-56, macau, china. association for computational linguistics.