Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond (2024)

Abstract: Natural language processing (NLP) technology has rapidly improved automated grammatical error correction (GEC) tasks, and the GEC community has begun to explore document-level revision. However, there are two major obstacles to going beyond automated sentence-level GEC to NLP-based document-level revision support: (1) there are few public corpora with document-level revisions annotated by professional editors, and (2) it is infeasible to obtain all possible references and evaluate revision quality using such references because there are infinite revision possibilities. To address these challenges, this paper proposes a new document revision corpus, Text Revision of ACL papers (TETRA), in which multiple professional editors have revised academic papers sampled from the ACL anthology. This corpus enables us to focus on document-level and paragraph-level edits, such as edits related to coherence and consistency. Additionally, as a case study using the TETRA corpus, we investigate reference-less and interpretable methods for meta-evaluation to detect quality improvements according to document revisions. We show the uniqueness of TETRA compared with existing document revision corpora and demonstrate that a fine-tuned pre-trained language model can discriminate the quality of documents after revision even when the difference is subtle.

## introduction document revision is a crucial step in the process of writing essays and argumentative texts. the writing process consists of two major parts: content organization and selection planning (henceforth, planning part) and realization of text improvement (henceforth, realization part), which are hierarchical and recursive. in addition, according to previous studies on argumentative writing @xcite @xcite @xcite , realization part in writing process typically comprises three main stages: revising, editing, and proofreading. revising is the initial editing step used to plan and structure the overall document at a high level, editing focuses on making sentence-level or phrase-level expressions, and proofreading is used to identify and correct errors such as spelling and grammar errors (see @xcite . while the order of these steps is not set in stone, the writing process typically starts with a broad, high-level perspective, and gradually narrows down the scope of edits. in contrast to the typical human writing process, gec research in nlp field, which is primarily intended to support writing, initially focused on a fine-grained scope, e.g., spelling errors @xcite @xcite and closed-class parts of speech (such as prepositions and determiners) @xcite @xcite . the research community then expanded its focus to include edits at the phrase and sentence levels while also considering fluency @xcite (figure 1 , right). however, significantly less work has been done on document-level revisions due to two major challenges. first, document revisions encompass a broader range of concerns such as coherence and flow, compared to conventional gec and fluency correction, which makes it difficult to find publicly available corpora that have been annotated by experts (professional editors). second, evaluating the quality of revisions is challenging as it requires multiple reference points, as there are many ways to revise a single document. this suggests that reference-less evaluation metrics @xcite @xcite are hold significant importance in automated document revision models. considering these challenges associated with automated document revision, we propose a new highquality corpus and explore possibilities for transpar-"the language model allows emulation of to emulate the noise generated." "we present results of on a quantitative analysis." ## background the field of gec, which has a multi-decade history, began with the goal of detecting and correcting targeted error types and providing feedback to english as a second language learners. foot_1 early gec systems primarily focused on a limited number of closedclass error types, such as articles @xcite and prepositions @xcite @xcite @xcite . the scope of gec was later expanded to include all types of errors, including verb forms, subject-verb agreement, and word choice errors @xcite @xcite . this line of research led to the establishment of shared benchmark tasks @xcite @xcite . motivated by the observation that error-coded local edits do not always sound natural to native speakers, the scope of gec has been further expanded from word-level closed-class edits to phrase-level and sentence-level fluency ed-grammaticality fluency clarity style readability redundancy consistency this paper presents empirical studies and closely corresponding theoretical models of a chart parser's performance whilethe performance of a chart parser exhaustively parsing the penn treebank with the treebank's own context-free grammar (cfg)cfg grammar. we show how performance is dramatically affected by rule representation and tree transformations, but little by top-down vs. bottom-up strategies. we discuss grammatical saturation, provide an, including analysis of the strongly connected components of the phrasal nonterminals in the treebank, and model how, as sentence length increases, regions of the grammar are unlocked, increasing the effective grammar rule size increases as regions of the grammar are unlocked, and yielding super-cubic observed time behavior in some configurations. we expect this approach to yield the following three improvements. taking advantage of the representation learned by the english model will lead to shorter training times compared to training from scratch. relatedly, the model trained using transfer learning will require requires less data for an equivalent score than a german-only model. finally, the more layers we freeze the fewer layers we will need to back-propagate through during training; thus,. thus we expect to see a decrease in gpu memory usage since we do not have to maintain gradients for all layers. we present the results of on a quantitative analysis of a number of publications in the nlp domain on the collectioncollecting, publishing, and availability of research data. we find that, although a wide range of publications rely on data crawled from the web, but few publications providegive details ofon how potentially sensitive data was treated. in addition additionally, we find that, while links to repositories of data are given, they often do not work, even a short time after publication. we presentput together several suggestions on how to improve this situation based on publications from the nlp domain, as well as but also other research areas. in contrast to the significant advancements in the area of grammar and fluency correction, relatively few studies have explored revisions for documentlevel argumentative writing, which require a greater investment of time and resources to create appropriate corpora or datasets. @xcite made an initial attempt to construct a document revision corpus comprising 13,000 student writings with feedback comments from tutors in the teaching english to speakers of other languages (tesol) program. although the authors developed labels for paragraph-level revisions (e.g., coherence), only 3% of all revisions were annotated as paragraph-level revisions, 90% of the revisions were at the word-level, and 7% were at the sentencelevel. this is because the corpus comprises writing from language learners, and the majority of errors were simple grammar and fluency errors. this lesson highlights the importance of using a corpus for document-level revision that has already been partially edited for grammar and fluency. however, due to copyright restrictions, this corpus may not be publicly available. the data source for a document-level corpus should be openly licensed to encourage community-based open research in the long term. another line of work @xcite @xcite @xcite has created the argrewrite corpus, a collection of 86 argumentative essays that include three drafts, each with two cycles of revisions, and edit labels. the argrewrite corpus (both v1 and v2) contains roughly half of all edits as surface-level corrections (e.g., conventional gec or fluency edits), and the other half of edits as content-level document revisions. while the argrewrite corpus has more document-level revisions than the corpus of @xcite , all of the essays in the ar-grewrite corpus were written on the same topic. the first version of the argrewrite corpus @xcite discusses the topic of whether the proliferation of electronic enriches or hinders the development of interpersonal relationships, and the second version @xcite focuses on whether to support or against self-driving cars. lee and webster (2012) zhang et al. (2017) kashefi et al. (2022) du et al. (2022) ours (tetra) # docs 3,760 60 86 559 64 # sents (avg) -18.7 25.8 7.19 26.92 # references 1 1 1 1 3 edit scope form? content&form content&form content&form form % beyondgecs 3.2 49.4 52.6 52.8 56.9 drafted by esl native (*esl) native (*esl) native (*esl) esl/native revised by author (nonexp.) author (nonexp.) author (nonexp.) author (nonexp.) exp. edit-types by nonexp. nonexp. nonexp. nonexp. exp. feedback ✓ topic diversity ✓ ✓ ✓ public availability ✓ ✓ ✓ ✓ this lack of topic diversity can lead to overfitting when developing and evaluating automated document revision models @xcite . recently, @xcite released a corpus of iterative document revisions from wikipedia, arxiv, and wikinews, with edit intention labels annotated foot_2 . although this work shares the same objective as ours, there are some differences such as the revision scope, the number of references, the expertise level of the editors, and the absence of feedback comments ( ## automated document revision given a source document d that consists of paragraphs, a potentially automated editor f revises (r) here, revision r is a set of edits e, and an edit e is defined as a tuple @xmath0, tgt, t, c), where src is the source phrase before the revision, tgt is the revised phrase, t is the edit type (e.g., grammar, word choice, or consistency), and c represents (optional) rational comments about the edit. when src is empty (ø), this edit indicates insertion, and it indicates deletion when tgt is empty; otherwise, the edit is considered to be a substitution. automated document revision includes various edit types (t), e.g., mechanics, word choice, conciseness, and coherence. this is discussed in further detail in §4.4. note that t does not exclude the scope of conventional (sentential and subsentential) grammatical error and fluency correction. rationale comments (c) are a useful resource in the study of feedback generation, which has become prominent in the gec community @xcite @xcite . thus, automated document revision is a natural extension of sentence-level error correction to document-level error correction with a wider context. @xcite the tetra corpus the validity of a dataset design is contingent upon the purpose and goals of the study. in line with §1 (and also figure 1 ), the primary objective of this study is to introduce a novel task focused on enhancing document-level editing and its automated evaluation technologies, which is distinct from the existing gec task. it is important to note that our aim is not to contribute to a broader understanding of "human revision" in general, which sets our study apart from the previous studies on revision (mentioned in §2).. hence, it is crucial to create a dataset that minimizes the inclusion of minor grammatical errors and fluency-related edits, which are already emphasized as requirements in gec. this is essential because proposing a new task entails the need to distinguish the technological aspects and linguistic phenomena targeted by the existing task and the proposed task. aspects edit types (abr.) definition scope % grammaticality grammar, capitalization edits that aimed to fix spelling/grammar mistakes s 19.4 fluency word choice, word order edits that aimed to increase sentence fluency s 23.7 clarity clarity edits that aimed to amplify meaning for clarity s/d 19.4 style style, tone edits that aimed to adapt the style s/d 8.0 readability readability edits that aimed to improve readability s/d 16.8 redundancy redundancy, conciseness edits that aimed to reduce redundancy s/d 7.2 consistency consistency, flow edits that aimed to increase paragraph fluency d 5.5 ## a case study: (meta) evaluation in addition to creating a corpus for automated document revision, it is essential to establish an evaluation that can measure a document's quality improvement (and possibly deterioration) relative to the applied revisions. as a case study, we use tetra to investigate reference-less and interpretable methods for a (meta-)evaluation method to detect quality improvements according to document revisions. ## analysis the experimental results discussed in §5.3 demonstrated that the supervised metric can discriminate the original and revision snippets with reasonably high accuracy. however, the following question should be considered. is the high accuracy derived from actually detecting the quality improvement provided by the revision or annotation artifacts (spurious correlation) by commonly used words and phrases by expert annotators? to investigate this question, we evaluated the performance of the same supervised metric (bert) used in §5.3 by applying corruption methods to tetra in order to artificially degrade the quality of the source documents. if the same supervised metric is fine-tuned on the source and the (improved) revision can still select the original document over the degraded document, we can conclude that the metric actually distinguishes the quality of the document rather than spurious features. ## conclusion we have proposed the new document revision corpus and highlighted its uniqueness of it compared with existing corpora. as a case study using this corpus, we have explored reference-less and interpretable meta-evaluation methods and also demonstrated that a fine-tuned pre-trained language model can discriminate the quality of documents, which indicates the feasibility of automated document revision evaluation.