How Good (really) are Grammatical Error Correction Systems? (2021)

Abstract: Standard evaluations of Grammatical Error Correction (GEC) systems make use of a fixed reference text generated relative to the original text; they show, even when using multiple references, that we have a long way to go. This analysis paper studies the performance of GEC systems relative to closest-gold – a gold reference text created relative to the output of a system. Surprisingly, we show that the real performance is 20-40 points better than standard evaluations show. Moreover, the performance remains high even when considering any of the top-10 hypotheses produced by a system. Importantly, the type of mistakes corrected by lower-ranked hypotheses differs in interesting ways from the top one, providing an opportunity to focus on a range of errors – local spelling and grammar edits vs. more complex lexical improvements. Our study shows these results in English and Russian, and thus provides a preliminary proposal for a more realistic evaluation of GEC systems.

## introduction grammatical error correction (gec) systems are typically evaluated using reference-based evaluation measures. this is common in language generation tasks, where the system output is compared against a set of gold references, such as the set of correct translations in machine translation or the set of valid corrections for a source sentence in gec. importantly, the references are generated relative to the original text and are independent of the system outputs. in gec, the space of valid outputs for a given source sentence is very large, making it extremely difficult to evaluate. specifically, reference-based evaluations (most gec datasets contain one reference correction) are known to underestimate system performance @xcite @xcite . @xcite showed that using two references is better than one, and the results improve further with more references; however, they used references that were generated relative to the original text. @xcite further demonstrated that the issue can be only slightly alleviated but not completely solved by increasing the number of references. this is because many errors have a long-tailed distribution on valid corrections. one can expect that as gec systems mature and manage to address more complex errors, the underestimation of their performance will be further exacerbated. choshen and abend (2018b) discuss another consequence of having a large space of valid corrections, pertaining to training. they show that gec systems have a strong tendency to undercorrect, due to using the one-reference-gold approach for tuning (and training) the systems. essentially, due to the low likelihood of a system's proposed change being matched to gold, gec systems are discouraged from proposing corrections, and propose far fewer corrections compared to humans. the under-correction phenomenon is more pronounced for errors with a large number of correction candidates. for example, mistakes on closed-class words, e.g. errors in determiners, where the number of valid corrections is small, suffer from under-correction to a lesser extent than mistakes in word choice. as a result, current systems generally prefer to make small targeted changes on closed-class errors. we further study the effects of having a large space of valid corrections on gec system development and automatic evaluation. given a potentially erroneous sentence, we assume there is a space of gold references corresponding to it. evaluation is typically done by drawing one gold from this set. we refer to this as reference gold (rg). we generate a new gold that is as close as possible to the system output (hypothesis), by correcting the hypothesis itself, instead of the original text. we call it closest gold (cg). we show by how much performance is really underestimated when a reference gold is used instead of the closest one, and claim that the latter should reflect the true performance of the system. we use a ranked 10-best list of hypotheses for a given source sentence, produced by state-of-the-art gec systems on two english and two russian gec datasets. we generate cgs for hypotheses at different ranks. our findings are as follows. first, evaluation against rgs shows a large performance gap between the top hypothesis and the rest. we show that the reason for this is that lower-ranked hypotheses propose more diverse changes, including lexical changes, that have a lower chance to match rgs. in contrast, evaluation against cgs reveals that qualitatively, there is very little degradation in the hypotheses, when considering the top-10 list. while rg evaluation reveals severe drops in f-score and, in particular, precision, we find that relative to cgs quality does not substantially degrade. this is illustrated in figure 1 for one of the datasets; we show more results in section 4.1. second, contrary to the observation made by @xcite about gec systems being disinsentivized to propose corrections, we find that it only applies to the top-ranked hypothesis. 1 moreover, the number of proposed edits increases steadily with the hypothesis rank. we further evaluate the output by computing the post-editing effort, i.e. the number of edits needed to correct the output hypothesis. we show that post-editing effort is very similar for top hypothesis and lower-ranked hypotheses, reinforcing the claim that lower-ranked hypotheses do not degrade in quality. finally, we evaluate the types of edits by hypothesis rank and show that lower-ranked hypotheses propose more diverse lexical changes, in contrast to the top hypotheses that mostly attempt local spelling and grammar misuse. our analysis should provide insight into better training and evaluation practices for gec. a better understanding of the under-correction phenomenon and the diversity and quality of the lower-ranked hypotheses can help improve the current training and tuning framework that relies on texts with single rgs and, arguably, hinders the development of gec systems that can potentially address more complex linguistic phenomena. next, we discuss reference-based evaluation. section 3 presents the definitions and experimental setup. section 4 presents the evaluation using closest golds. section 5 analyzes the edits proposed by top and lower-ranked hypotheses. ## reference-based evaluation the standard approach to evaluating gec systems is to use reference-based measures, that is comparing the system output to a reference that has been generated by a human annotator who corrected mistakes in the original source sentence. we refer to these as reference golds (rgs). it is common to instruct annotators to follow the principle of "minimal edits", that is making the smallest number of edits to render the sentence grammatical and wellformed. we follow a similar principle with our annotators, and the key distinction of our approach is that standard evaluations use golds that are independent of the system outputs, whereas we are creating golds by directly correcting the hypotheses output by the system. we note that there have been other proposals that argue that this principle still does not make the output fluent and propose generating references based on fluency @xcite . as suggested by @xcite , correcting for fluency further increases the space of valid corrections for a sentence, and we do not attempt to do this in this work. reference-based evaluations include several measures, such as the maxmatch scorer m 2 @xcite , gleu @xcite , errant @xcite , and imeasure @xcite . these met-rics have some commonalities, e.g. both max-match and errant measure precision, recall, and f-score. m 2 has been used with different beta parameter values, the default is @xmath0.5, weighting precision twice as high as recall, which is more common than assigning equal weights and has been shown to have stronger correlation with human ratings @xcite . gleu focuses on the fluency aspect -it is an extension of the bleu metric in machine translation @xcite . i-measure emphasizes accuracy and calculates the weighted accuracy of correction and detection. napoles et al. (2019) proposed gmeg-metric, that is an ensemble of existing metrics, and showed that its correlation with human judgments is higher on several gec datasets. the maxmatch metric has been widely used in evaluating gec systems in many published works and in several shared tasks @xcite , and we adopt it in this work. we use the default beta value of 0.5 and refer to the result as f-score. ## definitions and experimental setup we start with some definitions and then describe the experimental setup. ## evaluating true system performance we start by evaluating each hypothesis output h i for each dataset against reference gold rg and its corresponding closest gold cg i . we show that evaluation relative to rg is always pessimistic and, given a hypothesis generated by a gec system, there is always a much better gold. ## edit analysis by hypothesis rank we now analyze and compare the edits in the topranked hypothesis and in h 10 , in order to understand better how the edits differ with hypothesis rank. for the english datasets, we apply er-rant @xcite to extract edits using pairs of parallel sentences (source, hypothesis). errant then uses english-language specific rules based on part-of-speech and linguistic knowledge to assign each edit its linguistic type, such as preposition, noun number, etc. we further group the edits into one of the following two categories: spelling/grammar changes and lexical changes. the first category includes punctuation, spelling, orthography, and grammatical corrections that typically require local context and small changes and are also limited in the number of candidate corrections. these include determiner errors, verb agreement and form, noun number and punctuation, and morphological changes. lexical changes comprise the categories denoted by errant as "other", "verb", "noun", "pronoun", "adverb", which include mostly lexical errors, e.g. changing "get" to "earn", verb tense errors that require wider context and thus are trickier to correct. the number of edits by type is shown in in the lower part of the table, we show the distribution of edits between the two categories: in conll, spell/grammar changes account for 51.2% of all changes in the rg and for 74.3% in the top-ranked hypothesis. lexical changes make up 48.8% in rg, while only 25.7% in the topranked hypothesis, although this number increases to 36.2% in the h 10 . in bea, 49.5% of rg edits are lexical, while in the top-ranked hypothesis these account for 41.7% and that number goes up to 51.5% for the h 10 . looking at the number of edits in each category, it can be observed, that the under-correction phenomenon (for the top-ranked hypothesis) is particularly pronounced for lexical errors. in the spellgrammar category, the number of proposed edits is very close to (or even exceeds) the number of edits of this type in rg in both datasets (the only exception is perhaps the punctuation errors). for example, 33 determiner errors are present in conll rg, while there are 25 in h 1 . in contrast, 37 errors of category "other" are in rg for conll but only 8 changes of this type are in the top-ranked hypothesis. in fact, in both conll and bea, in the top-ranked hypotheses, the majority of the changes are minimal/local changes (74.3% in the conll dataset and 58.3% in the bea dataset). we perform a similar analysis for the russian datasets, where the edits are classified manually by our annotator (due to lack of automatic tool). we find similar behavior (see appendix a). however, in the most challenging categories (lexical and "other"), which both comprise word changes, the situation is more severe: the top-ranked hypothesis proposes 0 changes. overall, the undercorrection phenomenon for lexical errors is more pronounced for the russian language. overall, the under-correction phenomenon is especially pronounced for top hypotheses in the lexical error category. the percentage of lexical edits with respect to the total number of edits in the rg and cgs is much higher than in the top hypotheses. thus, under-correction is mostly a problem for lexical errors, but is partially rectified in the lower-ranked hypotheses, illustrated in figure 3 that shows the percentage of lexical edits in rg, h 1 , and h 10 for each dataset. ## discussion we study the current evaluation and training schema in gec, using 4 datasets in 2 languages and several state-of-the-art model architectures. we make several observations. first, we show that the quality of the systems is significantly better than we think, when we evaluate relative to the closest gold vs. reference gold. and the reason is there are many golds and we show that there is always a gold that is close to the prediction, and we should take this result as the actual performance of the model. 3 moreover, as we showed, using the cgs provides additional knowledge about the type of errors various hypotheses generate, further guiding the community towards developing additional insights that can be used also in targeting specific models for specific users (based on their abilities, for example). our second observation is that the top hypothesis is not actually better than the lower-ranked hypotheses in the 10-best list, even though the current evaluations are strongly biased towards the top hypothesis. third, because of the way we train, lower-ranked hypotheses relative to the reference gold are as good or sometimes qualitatively better than the top hypothesis because of the diversity of the type of mistakes that they attempt to correct.