Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric (2021)

Abstract: It is difficult to rank and evaluate the performance of grammatical error correction (GEC) systems, as a sentence can be rewritten in numerous correct ways. A number of GEC metrics have been used to evaluate proposed GEC systems; however, each system relies on either a comparison with one or more reference texts—in what is known as the gold standard for reference-based metrics—or a separate annotated dataset to fine-tune the reference-less metric. Reference-based systems have a low correlation with human judgement, cannot capture all the ways in which a sentence can be corrected, and require substantial work to develop a test dataset. We propose a reference-less GEC evaluation system that is strongly correlated with human judgement, solves the issues related to the use of a reference, and does not need another annotated dataset for fine-tuning. The proposed system relies solely on commonly available tools. Additionally, currently available reference-less metrics do not work properly when part of a sentence is repeated as opposed to reference-based metrics. In our proposed system, we look to address issues inherent in reference-less metrics and reference-based metrics.

## introduction evaluating the performance of a machine translation, text summarization, text simplification or gec system poses significant difficulties because there is more than one possible correct output @xcite . typically, a gec system is evaluated by comparing changes made by the system with annotated gold standards called max-match (m2) @xcite , i-measure @xcite , or generalized language evaluation understanding (gleu) @xcite . in each of these reference-based metrics, system outputs that differ from the annotated gold output are penalized. additionally, some metrics use multiple references; the @xcite test set has 10 references that require significant time and effort to build, and they still might not cover all the possibilities. to demonstrate how a single sentence can be edited in many ways, let us consider the following sentence from the nucle-2014 test set @xcite : bigger farming are use more chemical product and substance to feed fish. one concise revision of this sentence would be as follows: big farms use more chemicals to feed fish. likewise, if the author wished to express an ongoing action, the sentence could be the following: bigger farms are using more chemical products and substances to feed fish. we could go on. the point is that corrections to the raw data are not absolute because a change to one word in a sentence could alter another ostensibly erroneous word elsewhere in the sentence. given the contextual nature of such corrections, human judgement is the best way to determine the quality of generated sentences @xcite . in this paper, we propose the scribendi score, which is a reference-less metric to evaluate gec system outputs that strongly correlates with human judgement. this paper makes three main contributions: much work has addressed reference-based metrics, such as m2 @xcite , i-measure @xcite , gleu @xcite , and errant @xcite , but research on reference-less metrics is lacking. @xcite used three grammaticality metrics to measure sentence quality. they used a proprietary e-rater grammatical error detection module and a language tool. they also used a linguistic feature-based model @xcite with ridge regression. @xcite used grammaticality, fluency, and meaningpreservation metrics to achieve better correlation with human judgement. @xcite did the same for their sub-metrics that are optimized for manual evaluation (some). grammaticality and fluency can be replaced by perplexity scores generated by a language model, such as the generative pretrained transformer 2 (gpt-2) @xcite . to ensure the meaning is unaltered, we use the levenshtein distance ratio and token sort ratio. previous works have the following issues: ## perplexity score for measuring grammaticality language modeling is an approach to understanding linguistic structures by learning from a large corpus. linguists have accumulated many corpora to find syntactic language rules and formalize them into standard grammar @xcite . however, grammar cannot account for all situations in language use, as people sometimes speak ungrammatically in daily communication @xcite . to address this problem, scientists use statistical modeling to identify common patterns within languages. language models can learn the probability distribution of words in a sequence within a corpus. this approach is called language modeling. the language model is trained to minimize the cross-entropy loss, which is the same as minimizing perplexity. recently, the perplexity score has been used to assess writing quality @xcite ). consider the following example: "rarely read novels who reads comics." a gec model might come up with the following correction: "rarely read novels, who reads comics." but a human might suggest the following: "he/she who reads comics, rarely reads novels." @xcite stated that the gec system makes a small modification to the input sentences, which is why the outputs overlap significantly with the source sentences and the sentences produced by other systems. gec systems struggle to suggest corrections that are longer than a few words, and they are also not good at reordering sentences when necessary. by using perplexity scores, we are trying to compare sentences with small modifications to the original sentences to determine whether the modification improves the grammaticality and fluency of the sentence. ## methodology we use gpt-2 @xcite without further fine tuning to measure the perplexity of a source and predictions from different models, and we use the output of the 12 systems evaluated in conll-2014's shared tasks for gec @xcite . the test set consists of 1312 sentences. for each predicted sentence that is the same as its source, the score will be 0. we then calculate the perplexity score of each remaining output sentence of the 12 systems and compare the perplexity score with the perplexity score of its corresponding source sentence. if the perplexity score of the predicted sentence is greater than or equal to that of the source sentence, the scribendi score will be -1, as the perplexity score indicates that the suggested change(s) did not improve the grammaticality and fluency of the sentence. if the perplexity score improves (i.e., decreases), then we check whether the predicted sentences are syntactically related to the source sentence. we use the following two simple yet effective measurements to determine if they are syntactically similar as described at the end of section 3: a token sort ratio and a levenshtein distance ratio. ## results and discussion following the conll-2014 shared task on grammatical error correction @xcite , for which all the results are publicly available, including the references and 12 system outputs, @xcite and @xcite simultaneously performed a human evaluation of the system outputs. in this section, we present tables 2, 3, and 4, which compare the scribendi scores to the corresponding human evaluations. in in current state-of-the-art gec models are based on neural language models @xcite . neural language models are well known for generating repeated words @xcite . ## summary in this paper, we identified the shortcomings of reference-less metrics. namely, they need another annotated dataset for fine tuning and do not work properly when part of a sentence is repeated. we also highlighted that reference-based metrics are unable to score a sentence properly when the pre-dicted corrections are not contained in the reference sentences. in this study, we evaluated source and system outputs using the scribendi score, which is based on the perplexity score, the token sort ratio, and the levenshtein distance ratio. we demonstrated that the scribendi score does not require an extra annotated dataset for fine tuning, which is expensive in terms of resources and could cause over-fitting of certain datasets. it is strongly correlated with human evaluation, and is able to address the issue of repetition.