Revisiting Grammatical Error Correction Evaluation and Beyond (2022)

Abstract: Pretraining-based (PT-based) automatic evaluation metrics (e.g., BERTScore and BARTScore) have been widely used in several sentence generation tasks (e.g., machine translation and text summarization) due to their better correlation with human judgments over traditional overlap-based methods. Although PT-based methods have become the de facto standard for training grammatical error correction (GEC) systems, GEC evaluation still does not benefit from pretrained knowledge. This paper takes the first step towards understanding and improving GEC evaluation with pretraining. We first find that arbitrarily applying PT-based metrics to GEC evaluation brings unsatisfactory correlation results because of the excessive attention to inessential systems outputs (e.g., unchanged parts). To alleviate the limitation, we propose a novel GEC evaluation metric to achieve the best of both worlds, namely PT-M2 which only uses PT-based metrics to score those corrected parts. Experimental results on the CoNLL14 evaluation task show that PT-M2 significantly outperforms existing methods, achieving a new state-of-the-art result of 0.949 Pearson correlation. Further analysis reveals that PT-M2 is robust to evaluate competitive GEC systems. Source code and scripts are freely available at https://github.com/pygongnlp/PT-M2.

## introduction grammatical error correction (gec) is the task that takes a sentence with grammatical errors as input, and outputs a corrected sentence. due to the important role of gec in the field of second language learning and intelligent writing, gec has attracted wide attention from the community @xcite @xcite . as a typical natural language generation task (nlg), the common practice for gec evaluation is to calculate the simi- ## revisiting existing metrics this section revisits existing overlap-based gec metrics and pt-based metrics on the gec evaluation task, computing the metric correlation with human judgments, analyzing experimental results and exploring the differences among the metrics. ## experiment pt-m 2 can combine the advantages of both ptbased metrics and overlap-based gec metrics. in pt-m 2 , pt-based metrics only compute edit scores that can reduce the impact of unchanged part on the final score, paying more attention to the changes in the source sentences and editing scores are applied as corresponding edit weights on the m 2 . to demonstrate the effectiveness of our approach, we use pt-m 2 to evaluate gec system outputs and compute the correlation with human judgments on the conll14 evaluation task @xcite . we choose pearson γ and spearman ρ to measure the correlations. we use pt-m 2 to compute both the system score at corpusand sentence-level @xcite . we adopt two well-known unsupervised pt-based metrics bertscore and bartscore to score edits, which can be used in multiple domains, tasks and languages @xcite . the other metrics have been introduced in section 3.1. rank metric human bertscore m 2 sentm 2 pt-m 2 ew ts 1 input (⇑9) camb (⇑1) cuui (⇑3) amu (✓0) amu amu 2 ufc (⇑4) cuui (⇑2) amu (⇓1) cuui (⇑2) rac camb 3 iitb (⇑6) amu (⇓2) camb (✓0) camb (✓0) camb rac 4 rac (⇓1) post (⇑1) post (⇑1) rac (⇓1) cuui cuui 5 sjtu (⇑5) nthu (⇑7) nthu (⇑7) post (✓0) post post 6 pku (✓0) rac (⇓3) rac (⇓3) pku (✓0) ufc pku 7 amu (⇓6) umc (✓0) pku (✓0) ufc (⇓1) pku umc 8 post (⇓3) pku (⇓1) umc (✓0) sjtu (⇑2) umc ufc 9 cuui (⇓5) sjtu (⇑1) sjtu (⇑1) iitb (✓0) iitb iitb 10 umc (⇓2) ufc (⇓2) ufc (⇓2) nthu (⇑2) sjtu input 11 ipn (⇑2) ipn (⇑2) iitb (⇓2) input (✓0) input sjtu 12 camb (⇓9) iitb (⇓3) input (⇓1) umc (⇓4) nthu nthu 13 nthu (⇓1) input (⇓2) ipn (✓0) ipn (✓0) ipn ipn ∆ 53 27 21 12 --