A Mention-Based System for Revision Requirements Detection (2021)

Abstract: Exploring aspects of sentential meaning that are implicit or underspecified in context is important for sentence understanding. In this paper, we propose a novel architecture based on mentions for revision requirements detection. The goal is to improve understandability, addressing some types of revisions, especially for the Replaced Pronoun type. We show that our mention-based system can predict replaced pronouns well on the mention-level. However, our combined sentence-level system does not improve on the sentence-level BERT baseline. We also present additional contrastive systems, and show results for each type of edit.

## introduction the revision requirements task aims to recognize whether or not a sentence requires revision. revision requirements prediction not only acts as a standalone tool for grammar correction but also has potential applications in natural language processing (nlp) such as ambiguity detection, machine translation refinement, sentence understanding, knowledge base construction, etc. the shared task on implicit and underspecified language (roth and anthonio, 2021) foot_0 aims to provide a binary classification for revision requirements to make a prediction of whether sentences in instructional texts require revision to improve understandability. since instructional texts must be clear enough so that readers and machines can actually achieve the goal described by the instructions, this task focuses on modeling implicit elements that make the sentence more precise and clear. the dataset used in this shared task consists of instances from wikihowtoimprove, a collection of instructional texts, which has recently been introduced by @xcite . it contains six types of edits: the shared task submission requires only a binary distinction between sentences that require revision and sentences that do not. a good instructional text consists of specific instructions to accomplish the goal described and tends to avoid vague, generic and generalizing sentences. whilst checking the edit types in the revised version, especially "replacements of pronouns with more precise noun phrases", we observed that replacements occur primarily with generic pronouns that do not refer to a specific individual or set of individuals, but to a type or class of individuals. for this reason, we believe and show that identifying generic pronouns and noun phrases helps to predict whether a sentence requires revision for the revised pronoun class. for instance, if the pronoun has a co-reference in the sentence, it should not be replaced with a noun phrase. as a result, our proposed classification model for the task of revision requirements detection is based on extracting mention embeddings for each sentence generic pronouns they make a sound that dogs can hear, but humans can't. double check that it will be level using a level. your parents may not like any of them. burn it to a cd. let us have bad days. you cannot be offside directly from a corner-kick. our approach uses the neuralcoref resolution system to get mention embeddings for the target sentence. in addition we also extract embeddings for each mention based on @xcite for each mention. in this approach, we predicted revisions at the mention level. labels for the mentions were created based on a comparison between the original sentence, and the revised sentence, where we checked if any word had been changed, added or removed from each mention. for sentences for which we could not extract any mentions, we used a basic sentence-level bert-based system, since the bert model achieved the highest f1-score in previous work @xcite . in summary, we show that our mention-based system works well for replaced pronouns, but as expected, it is not successful for the other classes, which it does not target. our final system is overall slightly worse than our sentence-based system based on bert. at the mention-level our system performs well for replaced pronouns. ## related work there has been a lot of work on revisions to improve understandability, tan and lee (2014) conducted research on revisions in academic writing, using a qualitative approach to distinguish between strong and weak sentences, by analyzing the differences in the original and revised sentences. afrin and litman (2018) introduced a classification model based on random forest (rf) for revisions in argumentative essays from argrewrite @xcite to examine whether we can predict improvement for non-expert and predict if the revised sentence is better than the original. @xcite worked with edits in instructional texts and applied a supervised learning ## dataset we used the dataset provided by the organizers of the shared task on revision requirements prediction. this dataset contains instances that were extracted from the revision histories of www.wikihow.com articles. these how-to articles cover many fields such as arts and entertainment, computers and electronics, health, along with their revision history. the revisions and classes were extracted automatically from the training data. the development and test data was verified by human annotators (see roth and @xcite , for details). there are two subsets: the dataset includes training, development and test sets. however, the type of edit in case of a revision and the revised version of the target sentence, are available only for the training set. we therefore used k-fold cross-validation to randomly partition the training set into 5 equal-sized subsamples for training and development, for which we needed access to the revised sentences. ## mention extraction based on our observation that generic noun phrases often lead to revision, we hypothesize that extracting mentions based on a coreference resolution system might help in identifying such instances. we believe that this architecture might be especially useful for replacements of pronouns with more specific noun phrases and the insertion of logical quantifiers. we use huggingface's neural-coref system, which is based on spacy library, to extract mentions from our dataset. 4 in order to create labels for mentions, we extracted the class of each token for the input target sentence by comparing the target with the revised sentences. we use the python difflib library to align the original and revised sentence. we can then assign a positive label if any word in a mention was removed, changed or inserted and a negative label otherwise. ## mention embeddings since we need to capture the coreference information within the span of mentions in the embeddings, we produced two versions of the mention embeddings, one with dimension 650 using neuralcoref resolution system and a second of dimension 768 using bert-as-service. ## experimental and model design in this section, we present initial exploratory experiments and the process behind building a model that addresses the two obstacles to combine the predictions of the mention-level system into the sentence-level bert backup system. ## results and analysis this section presents an overview of our experiments and findings. we compare our results with the bert model baseline that set the previous stateof-the-art performance. we also present results on specific types of revisions since our approach was targeted mainly at the "replaced pronoun" class. we perform the majority of our analysis on training set, presented in tables 4 5 6 7 , which is the only data set which contains class labels and revised sentences. precision, recall, and f 1 -score is shown for requiring revision as the positive class. the most successful model on mention-level is the system with only mention bert embeddings we submitted the combination system based on mb embeddings to the shared task. 6 for our submitted predictions on the test, which was evaluated by the organizers in terms of accuracy, measured as the ratio of correct predictions over all data instances (roth and anthonio, 2021), we achieved 66.3% accuracy for the mention-based ## conclusions and future work in this paper, we show that identifying generic mentions can improve the performance of the replaced pronoun type. we introduced a mention-based system for predicting whether a sentence requires revision. investigating methods for combining a general classifier such as bert, with systems that target specific edits, such as our mention-based system, would be an interesting avenue for future work. as a next step, we plan to apply this idea to other languages and address other types of revisions.