Grammatical Error Correction: Are We There Yet? (2022)

Abstract: There has been much recent progress in natural language processing, and grammatical error correction (GEC) is no exception. We found that state-of-the-art GEC systems (T5 and GECToR) outperform humans by a wide margin on the CoNLL-2014 test set, a benchmark GEC test corpus, as measured by the standard F0.5 evaluation metric. However, a careful examination of their outputs reveals that there are still classes of errors that they fail to correct. This suggests that creating new test data that more accurately measure the true performance of GEC systems constitutes important future work.

## introduction grammatical error correction (gec) is a task that has many real-world applications, such as proofreading, assisting language learners @xcite @xcite , assisting children with developmental language disorder @xcite , etc. in order to measure the accuracy of a gec system, multiple evaluation metrics have been proposed in the past. ever since the conll-2014 shared task @xcite , the f 0.5 metric has been used as the standard evaluation metric for gec. the f 0.5 score has been found to have a better correlation with human judgment compared to f 1 and other metrics @xcite @xcite . as f 0.5 is a reference-based metric, its computation relies on comparing a system's output sentences to human's corrected sentences (called references henceforth). typically, there are multiple ways to correct an input sentence. for example, the @xcite official test set contains 2 references for each input sentence, and the bea-2019 @xcite test set contains 5 references. @xcite even predict that a short sentence may have more than 1,000 valid corrections. thus, this evaluation scheme tends to underestimate the performance of a gec system. @xcite have reported that gec systems can obtain higher scores if evaluated using references that are closer to a system's outputs. the limited references mean that humans may also not reach 100% f 0.5 performance, because their corrections need not be the same as the references. @xcite was the first to attempt to measure human performance on the conll-2014 test set by adding 8 new references for each sentence, so that each sentence has 10 references. then, they calculate each annotator's performance by comparing his corrections to the other 9 annotators. the average f 0.5 score of 72.58% from all annotators is then considered as the estimated human-level performance. to compare a gec system's performance against human performance, a similar procedure is applied by taking the average of the system's performance on the 10 sets of 9-annotator references. recently, much progress has been made in natural language processing. for example, for some question answering datasets (such as squad @xcite and coqa @xcite ) and natural language understanding test suites (superglue @xcite ), superhuman performance has been reported where the performance of the best nlp system exceeds human performance. a natural question to ask is: how does the performance of the latest state-ofthe-art (sota) gec systems compare to human performance? in this paper, we first attempt to answer this question. we found that state-of-the-art gec systems (t5 @xcite and gec-tor @xcite ) outperform humans by a wide margin on the conll-2014 test set, as measured by the f 0.5 metric. however, a careful examination of their outputs reveals that there are still classes of errors that they fail to cor-rect. this suggests that creating new test data that more accurately measure the true performance of gec systems constitutes important future work. at the time of writing this paper, the best published gec system using a sequence-to-sequence approach is t5-xxl, and the best published gec system using a sequence-tagging approach is the ensemble of gector xlnet and gector roberta. in this paper, we use both systems, but with t5-xxl replaced by t5-large due to resource constraints, since t5-large has 770m parameters while t5-xxl has 11b parameters. our evaluation is based on the scheme proposed by @xcite , using 10 sets of human annotations of the conll-2014 official test set. the f 0.5 scores of t5 and gector are shown in model f 0.5 score human 72.58 t5 78.63 gector 80.91 however, even though the scores in ## current weaknesses to understand what causes the low scores of the top-performing gec systems, we examine a sample of 100 sentences from the systems' outputs on the bea-2019 development set, starting from sentences with the lowest f 0.5 score. we found that even though t5 and gector generally produced good corrections, they also sometimes made obvious mistakes that humans will not make. ## moving forward as we have seen examples of how sota gec models fail to correct sentences that are easy for hu- context is there a future for privately owned cars ? to be honest , i am not sure . although privately oned cars are more and more popular , ... source i would say that most probably private care are not " sustainables " in the long term , ... t5 i would say that most probably private care is not " sustainable " in the long term , ... gector i would say that , most probably , private care is not " sustainable " in the long term , ... target i would say that , most probably , private cars are not " sustainable " in the long term , ... ## conclusion in this paper, we have reported that the top gec systems using the sequence-to-sequence approach (t5) and sequence-tagging approach (gector) have produced f 0.5 scores that exceed that of based on qualitative analysis of their outputs, we conclude that even though gector and t5 achieve f 0.5 scores higher than that of human, they have not outperformed humans in practice as they still fail to correct sentences that can be easily corrected by humans. we also report our qualitative analysis on the weaknesses of current top gec models, which point to directions for future research. lastly, we argue that a new test set that emphasizes harder sentences is needed to evaluate the progress of gec as a field.