Time Is Effort: Estimating Human Post-Editing Time for Grammar Error Correction Tool Evaluation (2025)

Abstract: Text editing can involve several iterations of revision. Incorporating an efficient Grammar Error Correction (GEC) tool in the initial correction round can significantly impact further human editing effort and final text quality. This raises an interesting question to quantify GEC Tool usability: How much effort can the GEC Tool save users? We present the first large-scale dataset of post-editing (PE) time annotations and corrections for two English GEC test datasets (BEA19 and CoNLL14). We introduce Post-Editing Effort in Time (PEET) for GEC Tools as a human-focused evaluation scorer to rank any GEC Tool by estimating PE time-to-correct. Using our dataset, we quantify the amount of time saved by GEC Tools in text editing. Analyzing the edit type indicated that determining whether a sentence needs correction and edits like paraphrasing and punctuation changes had the greatest impact on PE time. Finally, comparison with human rankings shows that PEET correlates well with technical effort judgment, providing a new human-centric direction for evaluating GEC tool usability. We release our dataset and code at : https://github.com/ankitvad/PEET_Scorer.

## introduction grammar error correction (gec) is an important step of the text editing process. there has been a lot of work to build automated gec tools that can improve the structure and fluency of text while also correcting language errors @xcite . since gec tool-assisted text editing is an iterative process, an editor can make post-edits to the tool output to obtain the closest targeted correction. estimating the post-editing (pe) effort required to reach the targeted correction can be used as a quality evaluation for the tool. human-in-the-loop pe effort was introduced and explored extensively for machine translation (mt) @xcite systems. pe effort is studied across three levels @xcite : technical effort, which is the number of edits; cognitive effort, which denotes the psychological assessment required to identify and correct the errors; and temporal effort, which is the total time taken to evaluate and perform post-edits (which includes technical and cognitive effort). @xcite and @xcite have explored estimating mt pe time based on edit features. technical pe effort has also been studied in areas like text summarization @xcite , natural language generation @xcite and gec @xcite . to incorporate the human editor effort in text correction, we present the first work to consider pe effort in time (peet) scores for quality estimation of a gec tool. the usability of a gec tool depends inversely on the pe effort to fix the tool output. we release the first large-scale dataset capturing time-tocorrect annotations for two english gec test sets -bea19 @xcite and @xcite , post-edited from two conditions: the original sentence and the output from two strong gec tools -gector @xcite and gec-pd @xcite . we further present a new human-centric gec tool evaluation method -peet scorer, to estimate the time-tocorrect for gec tool predictions, which correlates well with human editing effort. as a result, we propose that the peet scorer can be incorporated along with post-editing to evaluate a gec tool from a human editor's perspective. in this work, we make the following contributions: 2 background work ## dataset collection and processing an important component in this work is the highquality dataset of post-edit corrections for gec, along with their time-to-correct (temporal effort) annotations. we partnered with a professional textediting company -scribendi inc. foot_1 to collect this data. this section explains our dataset collection, filtering, and quality estimation process. ## methodology we design statistical and neural network regression models for our post-editing effort in time (peet) scorer. the scorer is trained to estimate the time-tocorrect value for a source sentence given the target correction, using the number and type of edits and sentence property -sentence length, correct/incorrect. the dataset that we collected contains 3 iterations for all 3 variations of the source -source (src), gec tool model output (mo) and postedited target correction (trg). different training features in terms of edits and sentence structure can be selected and extracted from -src, mo and trg triple (appendix d). statistical peet models performed as well as neural models while allowing greater interpretability of training features (appendix a). also, models using features selected from [m o, t rg] sentences performed better than models trained on fine-grained features from [src, m o, t rg] sentences (appendix e). hence, we only discuss the features and results of the statistical peet model trained using the [m o, t rg] sentences here, referring to mo as the source. ## conclusion and future work since we present the first study and dataset of post-editing effort (peet) in time for gec, our goal is to provide a baseline for future work in this area. using our dataset, we quantified the editor efficiency and productivity when using gec tools for text editing. we extract various automated sentence properties and edit type features from the sentence correction pairs to train the peet scorer. recently, there has been some work in the area of grammar error explanation to define descriptive error types @xcite and use llms for error explanation @xcite . as future work, the descriptive edits can be used as possible features for the peet model. finally, we observe that our peet model works well for gec tool evaluation when the output quality is dependent on the technical pe effort (amount of edits). studying actual cognitive effort for gec post-editing and how it compares with technical and temporal effort is another interesting direction for future work.