Reliability Crisis of Reference-free Metrics for Grammatical Error Correction (2025)

Abstract: Reference-free evaluation metrics for grammatical error correction (GEC) have achieved high correlation with human judgments.However, these metrics are not designed to evaluate adversarial systems that aim to obtain unjustifiably high scores. The existence of such systems undermines the reliability of automatic evaluation, as it can mislead users in selecting appropriate GEC systems. In this study, we propose adversarial attack strategies for four reference-free metrics: SOME, Scribendi, IMPARA, and LLM-based metrics, and demonstrate that our adversarial systems outperform the current state-of-the-art. These findings highlight the need for more robust evaluation methods.

## introduction grammatical error correction (gec) aims to automatically correct grammatical errors in text, such as tense and spelling errors. to improve correction performance, various gec systems have been proposed to date @xcite @xcite . one of the main purposes of automatic evaluation metrics is to rank gec systems based on their correction quality and to support users in selecting appropriate systems. recently, reference-free metrics, which do not require gold-standard corrections, have been reported to achieve high correlations with human judgments. for example, some @xcite achieved a spearman correlation exceeding 0.95 on the seeda meta-evaluation benchmark @xcite , which measures the ranking performance of 14 systems. however, these high correlations in prior studies assume an ideal setting in which only reasonable and valid correction outputs are evaluated. in reality, it is possible that correction outputs designed to exploit vulnerabilities in evaluation metrics are figure 1 : the situation we are concerned about. the adversarial attacking system may obtain an unreasonably high score by hacking a metric. this breaks the reliability of the automatic gec evaluation. in this study, we reveal that existing referencefree metrics are vulnerable to adversarial attacks. specifically, we propose inherent adversarial attack strategies for each of the four existing metrics: im-para @xcite , scribendi (islam and magnani, 2021), some @xcite , and llm-based metrics @xcite . experiments conducted on the bea-2019 @xcite development set show that our attack systems can obtain higher scores than current stateof-the-art gec systems @xcite . these findings highlight the severity of the alignment issue of existing reference-free gec metrics. we also discuss the reasons for the vulnerabilities and future directions for developing robust metrics. some @xcite ) trains three regression models some g (h), some f (h), and some m (s, h) corresponding to grammaticality, fluency, and meaning preservation, respectively. a distinctive feature is that each model is trained to directly optimize human evaluation scores. the final evaluation score is computed by weighting the three scores with the weights: α, β, γ (α + β + γ = 1) as shown in the following equation. basically, (α, β, γ) = (0.55, 0.43, 0.02) are used. scribendi (islam and magnani, 2021) checks whether the perplexity (ppl) computed by a language model such as gpt-2 @xcite decreases after correcting errors and whether surface similarity is maintained. the evaluation score is one of -1, 0, or 1. for surface similarity, levenshtein distance ratio (ldr) and token sort ratio (tsr) are used, and a threshold of 0.8 for the maximum of the two is used for filtering. this filter serves to reject hypothesis sentences that deviate too far from the input. formally, the score is computed according to the conditions shown in the following equations: true max(ldr(s, h), tsr(s, h)) > 0.8 false otherwise . impara @xcite evaluates edits by combining a similarity estimation model se(•) and a quality estimation model qe(•). the similarity estimation score is used as a filter for hypothesis sentences that deviate from the input, and the final score is given by the quality estimation score. the similarity estimation model is defined as the cosine similarity of embedding representations based on mean pooling computed by models such as bert @xcite . if the similarity is below a predefined threshold θ, the score is set to 0 by the filter. in general, θ = 0.9 is used. formally, the score is computed as follows: impara(s, h) = qe(h) se(s, h) > θ 0 otherwise . ## conclusion in this paper, we first demonstrated that four existing reference-free gec metrics have significant pitfalls and that our adversarial systems can outperform current sota-level systems. we also introduced a naive metric ensemble method to enhance robustness, and demonstrated that it can effectively rank adversarial systems as lower-quality systems. we argue that the vulnerability of existing reference-free metrics stems from inadequate filtering of adversarial sentences, and that the ensemblebased approaches serve as one possible solution to solve this issue. in future meta-evaluations of reference-free metrics, we hope that discussions will go beyond correlation coefficients with human evaluations to also consider robustness against adversarial attacks.