CLEME : Debiasing Multi-reference Evaluation for Grammatical Error Correction (2023)

Abstract: Evaluating the performance of Grammatical Error Correction (GEC) systems is a challenging task due to its subjectivity. Designing an evaluation metric that is as objective as possible is crucial to the development of GEC task. However, mainstream evaluation metrics, i.e., reference-based metrics, introduce bias into the multi-reference evaluation by extracting edits without considering the presence of multiple references. To overcome this issue, we propose Chunk-LE Multi-reference Evaluation (CLEME), designed to evaluate GEC systems in the multi-reference evaluation setting. CLEME builds chunk sequences with consistent boundaries for the source, the hypothesis and references, thus eliminating the bias caused by inconsistent edit boundaries. Furthermore, we observe the consistent boundary could also act as the boundary of grammatical errors, based on which the F _{0.5} score is then computed following the correction independence assumption. We conduct experiments on six English reference sets based on the CoNLL-2014 shared task. Extensive experiments and detailed analyses demonstrate the correctness of our discovery and the effectiveness of CLEME. Further analysis reveals that CLEME is robust to evaluate GEC systems across reference sets with varying numbers of references and annotation styles. All the source codes of CLEME are released at https://github.com/THUKElab/CLEME.

## introduction grammatical error correction (gec) is a task that involves making local substitutions to correct grammatical errors in a given ungrammatical text @xcite @xcite . the practical value of gec in daily life has led to increasing attention being paid to this task @xcite ; kaneko nowadays technologies have improved a lot compared to the last century. ref. 1 errant cleme (ours) nowadays the technologies were improved a lot compared for the last century. source nowadays the technologies were improved a lot compared for the last century. source nowadays technologies have improved a lot compared to the last century. ref. 1 nowadays technology has improved a lot compared with the last century. ref. 2 nowadays technologies have improved a lot compared with the last century. hyp. nowadays technology has with the last century. ref. 2 improved a lot compared nowadays have the last century. hyp. with technologies improved a lot compared et @xcite @xcite . however, it is intractable to evaluate gec systems due to the highly subjective nature of the task and the low inter-annotator agreement (iaa) @xcite . therefore, most datasets improve compatibility by incorporating multiple references to guarantee a more realistic evaluation of the model performance. there are two broad categories of gec metrics: reference-based and reference-less. referencebased metrics evaluate gec systems by comparing their hypotheses and human-annotated references in terms of edits @xcite or n-grams @xcite . reference-less metrics are proposed to evaluate gec systems without references. however, @xcite demonstrate that reference-less metrics are inherently biased and limited in their ability to evaluate generated text. therefore, we focus on reference-based metrics, which can evaluate in an interpretable manner, thus providing useful insights for model analysis. figure 1 illustrates how existing reference-based metrics, such as errant, extract the edit and then compute the f 0.5 score by comparing hypotheses and references. however, these metrics often fail to consider multiple references, which can result in bias during multi-reference evaluation. we argue that this bias arises because the current approach rewards equally good corrections unfairly. for in-stance, the ungrammatical phrase the technologies were is equally well-corrected by both ref. 1 and ref. 2. however, if a hypothesis aligns with ref. 1's corrections (i.e., [@xmath0, @xmath1, it will be rewarded less than the corrections of ref. @xcite , [@xmath2, @xmath3. in this paper, we propose chunk-level multireference evaluation (cleme), which enables unbiased f 0.5 scores for gec multi-reference evaluation. inspired by @xcite , cleme transforms the source, the hypothesis and all the references into chunk sequences with consistent boundaries, thereby eliminating the bias in gec multi-reference evaluation. existing metrics assume that corrections of grammatical errors are dependent. that is, whenever there is more than one reference for a source, the metrics try each reference in turn, and then the highest score is taken as the final score. however, we observe that grammatical errors corrections in terms of chunks can be considered approximately independent. for example, the ungrammatical phrases the technologies were and for shown in figure 1 can be corrected independently, i.e., the correction of the technologies were has no bearing on the correction of for. based on this observation, we compute f 0.5 scores following the assumption that corrections of grammatical errors are independent. specifically, we iterate through the chunks of a hypothesis and consider a chunk correct if it matches any of the corresponding chunks in the references. in this case, the hypothesis in figure 1 would be rewarded 2tp, rather than 1tp and 1fp, which is the traditional case. to demonstrate the effectiveness and robustness of cleme, we conduct experiments on six english reference sets with varying numbers of references and annotation styles, either calculating the f 0.5 score at the corpus-or sentence-level. in summary, our contributions are three folds: ## conclusion this paper proposes cleme, a reference-based gec metric that aim to provide unbiased f 0.5 scores for multi-reference evaluation. we explore evaluating gec systems based on either the correction dependence assumption or the correction independence assumption. several possible approaches can be suggested to further improve cleme. for example, developing (1) a gec metric that adaptively combines dependent and independent assumptions, and (2) a weighting strategy by utilizing the knowledge of pre-trained model. in the future, we would like to develop cleme for all languages and admonstrate the effectiveness of cleme on other languages. it is also worthwhile to explore accuracy-based metrics.