n-gram F -score for Evaluating Grammatical Error Correction (2024)

Abstract: M2 and its variants are the most widely used automatic evaluation metrics for grammatical error correction (GEC), which calculate an F-score using a phrase-based alignment between sentences. However, it is not straightforward at all to align learner sentences containing errors to their correct sentences. In addition, alignment calculations are computationally expensive. We propose GREEN, an alignment-free F-score for GEC evaluation. GREEN treats a sentence as a multiset of n-grams and extracts edits between sentences by set operations instead of computing an alignment. Our experiments confirm that GREEN performs better than existing methods for the corpus-level metrics and comparably for the sentence-level metrics even without computing an alignment. GREEN is available at https://github.com/shotakoyama/green.

## introduction grammatical error correction (gec) is one of text generation tasks that aims to convert erroneous texts into error-corrected ones. because of promising applications in second language learning, gec has attracted widespread attention from the nlp community @xcite @xcite @xcite . various automatic evaluation metrics for gec have been proposed to make evaluations cheaper and faster by avoiding high-cost human evaluations. m 2 @xcite and its variants are the most widely used metrics in the automatic evaluation for gec. they first compute a phrasebased alignment between sentences to extract edits of correction. they then calculate an f -score by comparing edits from the source to the reference sentences and edits from the source to the corrected sentences. the conll-2014 shared task of gec adopted m 2 as its evaluation metric, and the bea-2019 shared task adopted errant @xcite , one of the variants of m 2 . currently, they are the representative metrics for gec. however, it is not straightforward at all to align source sentences (learner sentences containing errors) to their target sentences (correct sentences). in addition, the alignment calculation is computationally expensive and time-consuming for long sentences with many edits from the source sentence. furthermore, m 2 requires manually annotated data with edits from the source to the reference sentences to extract edits; errant needs no manually annotated data but depends on a partof-speech tagger to perform the alignment calculation. supposing that we could extract edits between sentences without alignments, we would design a more practical and useful alignment-free evaluation method that achieves the same level of performance as m 2 and errant without depending on additional data or tools to extract the alignment. in this paper, we propose green, an alignmentfree f -score for gec evaluation, which treats a sentence as n-gram occurrences using a multiset (a set with repeated elements) of n-grams to compute an f -score by comparing edits between two multisets. we conducted experiments to verify the effectiveness of green on the conll-2014 evaluation dataset @xcite and the seeda dataset @xcite . even without computing an alignment, green exhibits a higher correlation with human evaluation in terms of both pearson and spearman correlation coefficients for the corpus-level metrics. it also achieves comparable performance with existing methods for the sentence-level metrics. one of the issues with m 2 is time complexity. m 2 finds the shortest path of a directed acyclic graph. let the number of tokens in the source, reference, and corrected sentence be less than or equal to k. the bottleneck in the average case lies in the graph pruning algorithm to calculate the optimal alignment, which requires the o(k 2 ) time complexity. however, in the worst case, when no nodes are pruned in this process, the numbers of nodes v and edges e are constant multiples of k 2 and k foot_2 . since topological sort requires o(v + e) time complexity to find the shortest path, m 2 requires o(k 4 ) in the worst case. the official implementation in the conll-2014 shared task adopts the bellman-ford algorithm, which has a time complexity of o(v e), resulting in the worst-case time complexity of o(k 6 ). in this paper, we adopted the faster implementation 1 using topological sort. another issue is the inability to properly evaluate systems that generate corrupted sentences @xcite . m 2 gives @xmath0. for this reason, m 2 may evaluate a system that generates outputs that are worse than the source text as @xmath1. this is a common problem for other alignment-based f -score methods that are variants of m 2 . @xcite errant computes an f -score by comparing the edits on the reference and corrected sentences similarly to m 2 . errant automatically extracts edits for both reference and corrected sentences using the linguistically enhanced alignment algorithm @xcite based on the spacy part-of-speech tagger and damerau-levenshtein distance, with time complexity of o(k 2 ). the unnecessity of 1 https://github.com/craggy-otake/m2scorer_ python3_fast manually annotated reference edits is an advantage of errant. we used the official implementation v3.0.0 2 . ## proposed method: green first, we describe green with one reference sentence in section 3.1. we will extend green for multiple references in section 3.2. ## conclusions we proposed an alignment-free gec evaluation metric, green, which computes f -score by comparing edits between multisets. green shows a higher correlation for both pearson and spearman correlation coefficients for the corpus-level metrics and comparable performance with existing evaluation metrics for the sentence-level metrics while it runs faster than existing methods and does not require the alignment calculation. we also analyzed the effect on β for f -score-based methods. we confirmed that alignment-based methods and green have different tendencies on β. we investigated the problem that alignment-based f -score is difficult to evaluate the source text and degraded text correctly. we confirmed that corpus-level green properly evaluates systems in contrast to existing corpus-level metrics, and sentence-level metrics alleviate the bias of alignment-based f -score on the source and degraded texts. further challenges include incorporating pre-trained models and evaluating fluency-edited texts. short papers), pages 588-593, beijing, china. association for computational linguistics. courtney napoles, keisuke sakaguchi, matt post, and joel tetreault. 2016a. gleu without tuning. preprint, arxiv:1605.02592. courtney napoles, keisuke sakaguchi, and joel tetreault. 2016b. there's no comparison: referenceless evaluation metrics in grammatical error correction. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 2109-2115, austin, texas. association for computational linguistics. hwee tou ng, siew mei wu, ted briscoe, christian hadiwinoto, raymond hendy susanto, and christopher bryant. 2014. the conll-2014 shared task on grammatical error correction. in proceedings of the eighteenth conference on computational natural language learning: shared task, pages 1-14, baltimore, maryland. association for computational linguistics. kishore papineni, salim roukos, todd ward, and wei-jing zhu. 2002. bleu: a method for automatic evaluation of machine translation. in proceedings of the 40th annual meeting of the association for computational linguistics, pages 311-318, philadelphia, pennsylvania, usa. association for computational linguistics. maja popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. in proceedings of the tenth workshop on statistical machine translation, pages 392-395, lisbon, portugal. association for computational linguistics. xin sun, tao ge, furu wei, and houfeng wang. 2021. instantaneous grammatical error correction with shallow aggressive decoding. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 5937-5947, online. association for computational linguistics. jingheng ye, yinghui li, qingyu zhou, yangning li, shirong ma, hai-tao zheng, and ying shen. 2023. cleme: debiasing multi-reference evaluation for grammatical error correction. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 6174-6189, singapore. association for computational linguistics. wei zhao, liang wang, kewei shen, ruoyu jia, and jingming liu. 2019. improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 156-165, minneapolis, minnesota. association for computational linguistics. houquan zhou, yumeng liu, zhenghua li, min zhang, bo zhang, chen li, ji zhang, and fei huang. 2023. improving seq2seq grammatical error correction via decoding interventions. in findings of the association for computational linguistics: emnlp 2023, pages 7393-7405, singapore. association for computational linguistics. in an issue of the official m 2 github repository foot_8 , an example is given in which m 2 takes a long time to calculate. here is the example in this issue: as it is a genetic risk , the patient force might have a high chance of carrying the risk , hence the need to inform their relatives is important . hence , you are suffering from a genetic disease that the genetic trait might be passed on to your next generation if you have a child . hence , there is no legal obligation to disclose to their family members , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there is no legal obligation . hence , there such a degeneration of repetition sometimes occurs in neural text generation @xcite . in amu-s, the 333rd sentence in amu is replaced by this sentence.