Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Models (2024)

Abstract: In this paper, we carry out experimental research on Grammatical Error Correction, delving into the nuances of single-model systems, comparing the efficiency of ensembling and ranking methods, and exploring the application of large language models to GEC as single-model systems, as parts of ensembles, and as ranking methods. We set new state-of-the-art records with F_0.5 scores of 72.8 on CoNLL-2014-test and 81.4 on BEA-test, respectively. To support further advancements in GEC and ensure the reproducibility of our research, we make our code, trained models, and systems’ outputs publicly available, facilitating future findings.

## introduction grammatical error correction (gec) is the task of correcting human text for spelling and grammatical errors. there is a wide variety of gec approaches and model architectures. in recent years, most systems have used transformer-based architectures @xcite . a current trend involves writing prompts for large language models (llms) such as @xcite that would generate grammatical corrections @xcite , @xcite , @xcite , @xcite . the varied approaches within gec each possess unique strengths and limitations. combining several single-model gec systems through ensembling or ranking may smooth out their weaknesses and lead to better overall performance @xcite . even quite simple ensembling methods, such as majority voting @xcite or logistic regression @xcite , may work surprisingly well. combining singlemodel systems is also often straightforward from an implementation perspective. because only the outputs of the models are required for many ensembling algorithms, there is no need to retrain models or perform inference passes iteratively. a further review of related work is presented in the end and near the descriptions of considered methods. our contributions are the following: ## data for training and evaluation we use the following gec datasets for training models ( ## ensembling and ranking of single-model systems combining the outputs of single-model gec systems can improve their quality. in this paper, we explore two combining methods: ensembling and ranking (figure 1 ). ensembling combines outputs of single-model systems on an edit level. the ensemble method exploits the strengths of each model, potentially leading to more robust and accurate corrections than any single-model system could provide on its own. ranking is a special case of ensembling that combines individual outputs on a sentence level. in this approach, the performance of each system's candidate is assessed against a set of predefined criteria, and the most effective candidate is selected. ranking maintains the internal coherence of each model's output, potentially leading to more natural and readable corrections. ## related work large language models have demonstrated efficacy across a variety of natural language processing tasks, including gec @xcite . the comparative analysis conducted by @xcite on the effectiveness of different models for gec -chatgpt, grammarly, and open-sourced gector -reveals that chatgpt possesses a distinctive capability to enhance textual content by not only correcting errors on a one-by-one basis but also by rephrasing original sentences, changing their structure to maintain grammatical correctness. the outcomes of human evaluations underscore the limitations of exclusively relying on automatic evaluation metrics for assessing gec model performance, thereby positioning chatgpt as a potentially invaluable resource for gec applications. other research @xcite , @xcite suggests that although zero-shot and fewshot chain-of-thought methodologies demonstrate promise in terms of error detection capabilities and the production of fluently corrected text, they generally underperform across the majority of error categories, thus failing to achieve high-quality outcomes in gec. moreover, @xcite delved into the customization of open-sourced foundation llms including llama @xcite for writing assistant applications, with gec as one of the tasks. the experimental findings indicate that instruction tuning for specific scenarios such as gec significantly boosts the performance of llms and can be used to develop smaller models that outperform their larger, general-purpose counterparts. additionally, @xcite introduced a novel approach for predicting edit spans within source texts, redefining instruction-based fine-tuning as local sequence transduction tasks. this method not only reduces the length of target sequences but also diminishes the computational demands associated with inference. the study emphasizes that even high-performance llms such as chatgpt struggle to generate accurate edit spans in zero-shot and few-shot scenarios, particularly in the correct generation of indexes, making this approach unstable. recent advancements in gec have largely been attributed to the ensembling of outputs from individual models, as highlighted in studies by @xcite . when integrating systems with significant disparities, a system combination model is preferred over simple ensembles. this approach allows for effective integration of the strengths of various gec systems, yielding better results than ensembles, as demonstrated in @xcite . model outputs can be re-ranked using majority vote, as well as with the proposed greco model (qorib and ng, 2023), a new state-of-the-art quality estimation model correlating more closely with the f 0.5 score of a corrected sentence, thus leading to a combined gec system with a higher f 0.5 score. additionally, this study proposes three methods for leveraging gec quality estimation models in system combination: model-agnostic, model-agnostic with voting bias, and model-dependent methods.