Revisiting Meta-evaluation for Grammatical Error Correction (2024)

Abstract: Metrics are the foundation for automatic evaluation in grammatical error correction (GEC), with their evaluation of the metrics (meta-evaluation) relying on their correlation with human judgments. However, conventional meta-evaluations in English GEC encounter several challenges, including biases caused by inconsistencies in evaluation granularity and an outdated setup using classical systems. These problems can lead to misinterpretation of metrics and potentially hinder the applicability of GEC techniques. To address these issues, this paper proposes SEEDA, a new dataset for GEC meta-evaluation. SEEDA consists of corrections with human ratings along two different granularities: edit-based and sentence-based, covering 12 state-of-the-art systems including large language models, and two human corrections with different focuses. The results of improved correlations by aligning the granularity in the sentence-level meta-evaluation suggest that edit-based metrics may have been underestimated in existing studies. Furthermore, correlations of most metrics decrease when changing from classical to neural systems, indicating that traditional metrics are relatively poor at evaluating fluently corrected sentences with many edits.

## introduction grammatical error correction (gec) is the task of automatically detecting and correcting errors, including grammatical, orthographic, and semantic errors, within a given sentence. the prevailing approach in gec involves the use of a sequenceto-sequence method @xcite . automatic evaluation metrics play an important role in the progress of gec. these metrics are essential for a fast and efficient improvement cycle of system development because they can replace costly and time-consuming human evaluations and immediately reflect system per-formance. gec has made progress by enabling a fair comparison of performance on a common benchmark using these metrics in shared tasks @xcite @xcite . gec metrics are categorized into edit-based and sentence-based types according to their evaluation granularity, and each has its objectives. edit-based metrics (ebms), such as m 2 @xcite and errant @xcite , focus on evaluating the quality of the edit itself, whereas sentence-based metrics (sbms), such as gleu @xcite , evaluate the quality of the entire sentence after correction. since the system output consists only of sentences without explicit edits, ebms require the edit extraction from the system output using any method. in addition, these metrics are primarily evaluated based on the correlation with human judgment (i.e., meta-evaluation). most of the previous meta-evaluations in english gec have relied on @xcite dataset with human judgments (henceforth, this dataset is referred to as gjg15). however, existing meta-evaluations based on gjg15 @xcite @xcite have several significant issues. first, the performance of ebms may be underestimated due to biases resulting from inconsistencies in evaluation granularity. as an example of biases, while ebms assign the lowest score (or the highest score in the sentence-level evaluation) to the uncorrected sentence, sentence-based human evaluation, such as gjg15, assigns scores across the entire range. furthermore, according to the actual data in ## the seeda dataset the seeda dataset consists of corrections annotated with human ratings along two different evaluation granularities (edit-and sentencebased), covering 12 state-of-the-art neural systems including llms, and two human corrections. the seeda dataset is denoted as seeda-e for edit-based evaluation and seeda-s for sentencebased evaluation. in this section, we describe the seeda dataset, how we generated the corrections ( §3.1), and how we collected the annotations ( §3.2). we use the conll-2014 test set @xcite as our input data, consisting of test essays and their error annotations. the test essays are written by non-native english-speaking students from the national university of singapore and cover two genres: genetic testing and social media. error annotations for the test essays are conducted by two native english speakers. the data comprises a total of 50 essays, consisting of 1,312 sentences and 30,144 tokens. ## dataset analysis in this section, we analyze seeda with a focus on evaluation granularity. first, we present the dataset statistics ( §4.1). second, we produce human rankings for the system using rating algorithms to conduct system-level meta-evaluation ## baseline metrics we target 11 gec metrics for meta-evaluation, including ebms ( §5.1) and sbms ( §5.2). ## further analysis as further analysis, we investigate the influence of outliers ( §7.1) and variations in the system set ( §7.2) on the correlation of the metric. we test the hypothesis on which this study focuses, that there may be a range of correlations in flexible settings in gec. based on the best practices obtained in §6, granularity will be aligned in subsequent meta-evaluations. ## discussion we provide a more practical guideline for metaevaluation ( §8.1) and evaluation ( §8.2) methodologies in future gec research by considering the experimental results so far. ## conclusion to address issues in conventional meta-evaluation in english gec, we construct a meta-evaluation dataset (seeda) consisting of corrections with human ratings along two different evaluation granularities, covering 12 state-of-the-art system corrections including llms, and two human corrections with different focuses. the dataset analysis reveals that the results of sentence-level human evaluation differ between granularities and that gec systems based on gpt and t5 can correct as well as or better than humans. also, through meta-evaluation using seeda, we demonstrate that ebms may be underestimated in existing meta-evaluations and that matching the evaluation granularity of metrics with human evaluations tends to improve sentence-level correlations. by further analysis, we discovered the uncertainty of conclusions based on a single correlation and found that most metrics lacked the precision to distinguish differences among high-performance neural systems. finally, we propose a methodology for meta-evaluation and evaluation in gec. we hope that this paper contributes to further advancements in gec.