IMPARA - GED : Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (2025)

Abstract: We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations.

## introduction grammatical error correction (gec) is the task of automatically correcting grammatical or superficial errors in input sentences. while gec systems should manually assess the quality of their corrections, human evaluation for a wide range of arbitrary inputs is strenuous, making it impractical. therefore, it is necessary to establish automatic evaluation methods for gec that correlate highly with human evaluation. the automatic gec evaluation methods can be categorized into referencebased evaluation methods @xcite @xcite and referencefree evaluation methods @xcite @xcite . reference-based evaluation methods measure the closeness between the outputs of gec systems and human-written references. however, since incorrect sentences can be corrected in multiple ways, accurate evaluation requires multiple reference sentences. yet, constructing comprehensive human references is impractical, and low-coverage reference sets often deteriorate evaluation reliability @xcite . therefore, reference-free evaluation methods, which rely only on input sentences and system outputs, have the potential to overcome these limitations. most reference-free evaluation methods employ pre-trained language models (plms). for instance, scribendi score (islam and magnani, 2021) uses the perplexity of a plm to compute evaluation scores for corrected outputs. some @xcite trains plms separately on human assessment scores for fluency, grammaticality, and meaning preservation. impara @xcite combines a similarity estimator between inputs and system outputs with a quality estimator for system outputs, which rely on plms. the quality estimator is trained without requiring humanannotated evaluation results, using only parallel data of erroneous and corrected sentences constructed for gec systems. however, although the quality estimator is trained on a vanilla plm, its pre-trained knowledge alone is insufficient to capture grammatical errors accurately. in this paper, we propose impara-ged, a novel reference-free automatic evaluation method for gec. inspired by the insight that additional training on the grammatical error detection (ged) task enhances gec systems @xcite , the quality estimator of impara-ged is constructed by first fine-tuning a plm on the ged task and then applying quality estimator construction method of impara. moreover, we remove the similarity estimator used in impara, as empirical observations indicate that it fails to effectively capture grammatical errors in the outputs of modern gec systems. our experimental results on seeda, a meta-evaluation benchmark @xcite , show that impara-ged achieves the highest correlation with human sentence-level evaluations. construction of quality estimator. the quality estimator is constructed by learning the pairwise quality ranking order (s + , s -), where s + has higher quality and s -has lower quality. these pairs can be automatically generated from parallel data of incorrect and correct sentences constructed for gec systems. specifically, an edit set for transforming an incorrect sentence into a correct sentence is extracted, and the impact of each edit is quantified based on the degree of semantic change when the edit is removed from the correct sentence. two different subsets of the edit set are then sampled and applied to the incorrect sentence, generating two different corrected sentences. since the impact of each edit is already computed, the total impact can be calculated for each subset as a quality. we regard the higher-quality correction as s + and the lower-quality correction as s -, forming the training data t . finally, the quality estimator r is trained using the training data t by minimizing the loss function l qe as follows: scoring in impara. impara calculates the evaluation score s(i, o) ∈ [0, 1] for a pair of an input sentence i and its corrected output o from a gec system using the quality estimator r and the similarity estimator sim(•) as follows: ## experimental settings construction of impara-ged. we used @xcite and fce @xcite for model construction. conll-2013 was split into train, dev, and devtest sets in an 8:1:1 ratio, while fce was used with its predefined splits. first, we construct the ged model following the settings of @xcite . the plm is trained for five epochs using a combined train set of fce and conll-2013, and the checkpoint that achieves the highest score on the fce dev set is selected as the final ged model. next, the ged model is used to build the quality estimator following the procedure described in § 2 and the settings of @xcite , using the conll-2013 train set. the ged model is then trained for ten epochs following equation 1, and the checkpoint achieving the best performance on the conll-2013 dev set is selected. we trained the model using five different random seeds, and the one that performs best on the conll-2013 devtest set is selected as the final model. we report results for all combinations of the following label granularities: 2-class, 4-class, 25-class, and 55class, and the following plms: bert base @xcite , deberta-v3 large @xcite , and modernbert large @xcite . evaluations. we conduct meta-evaluations using seeda @xcite , using both editlevel annotations (seeda-e) and sentence-level one (seeda-s). we follow the trueskill-based system ranking and the base system setting, which includes outputs from 12 modern gec systems. we report pearson's correlation coefficient r and spearman's rank correlation coefficient ρ as correlation metrics for system-level evaluation and accuracy (acc.) and kendall's rank correlation coefficient τ for sentence-level evaluation. as baselines, we include reference-based evaluation methods: er-rant @xcite , pt-errant @xcite , green @xcite , and gleu @xcite . for reference-free evaluation methods, we report scribendi score (islam and magnani, 2021), some @xcite , and the original impara @xcite . additionally, we include gpt-4-s @xcite and its three derivative systems as large language model-based gec evaluation methods. our implementation uses gec-metrics 1 @xcite with each system's default settings. for gpt-4-s, we cite the reported values from @xcite . following @xcite , all system evaluations are conducted using trueskill @xcite , aligning with the aggregation method used in human evaluation. we also conducted significance testing following @xcite , using williams significance tests (graham and baldwin, 2014) for system-level evaluation and bootstrap resampling @xcite for sentence-level evaluation. ## experimental results and discussions main results. number of ged types. increasing the number of error classification types for ged training does 1 https://github.com/gotutiyan/gec-metrics system-level sentence-level seeda-s seeda-e seeda-s seeda-e methods r ρ r ρ acc. τ acc. τ errant .763 .706 .881 .895 .594 .189 .608 .217 pt-errant .870 .797 .924 .951 .582 .165 .592 .184 green .855 .846 .912 .965 .600 .199 .574 .148 gleu .863 .846 .909 .965 .672 .343 .673 .347 scribendi .674 .762 .837 .888 .660 .320 .672 .345 some .932 .881 .893 .944 .778 .555 .766 .532 impara .939 .923 .901 .944 .753 .506 .752 .504 gpt-4-s .887 .860 .960 .958 .784 .567 .798 .595 +grammaticality .888 .867 .961 .937 .796 .592 .807 .615 +fluency .913 .874 .974 .979 .819 .637 .831 .662 + meaning preservation .958 .881 .911 .960 .810 .620 .813 .626 bertbase .915 .895 .875 .930 .756 .512 .754 .508 +2-class .916 .909 .850 .902 .773 .545 .763 .527 +4-class .908 .902 .859 .923 .787 .574 .774 .548 +25-class .925 .902 .875 .923 .771 .543 .752 .503 +55-class .900 .902 .842 .923 .763 .526 .750 .499 deberta-v3large .960 .937 .912 .944 .784 .568 .779 .558 +2-class .951 .923 .895 .916 .797 .593 .784 .568 +4-class .939 .895 .899 .916 .793 .585 .772 .544 +25-class .945 .930 .906 .930 .801 .602 .786 .573 +55-class .955 .930 .913 .958 .782 .564 .763 .527 modernbertlarge .949 .909 .912 .937 .767 .533 .749 .497 +2-class .971 .930 .919 .930 .829 .658 .797 .594 +4-class .964 .916 .926 .923 .812 .624 .794 .588 +25-class .972 .937 .933 .944 .801 .603 .783 .567 +55-class .965 .951 .910 .909 .749 .498 .741 .483 ## conclusion we proposed impara-ged, a novel referencefree automatic gec evaluation method, by enhancing the ged capabilities of plms. when using modernbert as the plm, additional training on the binary-labeled ged task achieved the highest correlation with human evaluation among existing methods on seeda. furthermore, window analysis revealed that impara-ged improves evaluation performance, particularly for top-ranked systems. moreover, we revealed that current similarity estimators fail to adequately capture meaning preservation. we suggest that future development of gec evaluation metrics may proceed in two directions: either by building a stronger similarity estimator or by enhancing the quality estimator. we publicly release impara-ged as the official version, based on modernbert with binary classification, available at: https: //huggingface.co/naist-nlp/impara-ged .