Multi-Reference Benchmarks for R ussian Grammatical Error Correction (2024)

Abstract: This paper presents multi-reference benchmarks for the Grammatical Error Correction (GEC) of Russian, based on two existing single-reference datasets, for a total of 7,444 learner sentences from a variety of first language backgrounds. Each sentence is corrected independently by two new raters, and their corrections are reviewed by a senior annotator, resulting in a total of three references per sentence. Analysis of the annotations reveals that the new raters tend to make more changes, compared to the original raters, especially at the lexical level. We conduct experiments with two popular GEC approaches and show competitive performance on the original datasets and the new benchmarks. We also compare system scores as evaluated against individual annotators and discuss the effect of using multiple references overall and on specific error types. We find that using the union of the references increases system scores by more than 10 points and decreases the gap between system and human performance, thereby providing a more realistic evaluation of GEC system performance, although the effect is not the same across the error types. The annotations are available for research.

## introduction grammatical error correction (gec) is the task of detecting and correcting mistakes in text. most of the gec research effort has been devoted to correcting mistakes made by english language learners @xcite @xcite @xcite @xcite . the standard approach to evaluating gec systems is to make use of reference-based measures, where system output is compared against a humangenerated reference. a system is rewarded for proposing corrections that are in the reference, and penalized for proposing corrections not found in the reference. a sample sentence with errors from the nucle corpus of english language learners @xcite , along with two corrected versions (references 1 and 2) generated independently by two human experts, is depicted in figure 1 (top part). when more than a single reference is available, system output is evaluated independently against each reference, and the reference that maximizes the score for the sentence is selected. there are usually multiple ways of correcting a single sentence, but, since generating human annotations is expensive, many gec benchmarks contain a single human reference. a large body of work strongly suggests that evaluating against a single reference severely underestimates system performance @xcite @xcite , making it difficult to accurately evaluate gec models and preventing progress in developing robust gec systems that are not overly sensitive to the data from a single annotator. using more than a single reference has been shown to provide a more realistic evaluation of gec systems @xcite . this is because having multiple references increases the chance that a valid system correction will match a correction proposed by a human expert. for this reason, system scores tend to increase with the number of annotators used in the gold standard, although at a diminishing rate, suggesting that using three annotators might be sufficient for a more accurate gec evaluation @xcite . the goal of this work is to contribute to the task of building robust gec systems by creating benchmark datasets with multiple reference annotations. we consider russian that has two benchmark gec datasets -rulec @xcite and ru-lang8 @xcite -both annotated with a single reference, and construct two new reference annotations for each sentence. analysis of the generated corrections reveals that the new annotators have a high degree of variability in proposing corrections, and, compared to the original raters, propose more changes related to overall fluency. 2 we attribute this to the use of the annotation framework that does not focus on identifying error spans and error types, but, instead, encourages sentence-level re-writing. the paper makes the following contributions: (1) we enhance two existing gec benchmarks for russian with two additional references, for a total of three references per sentence; (2) using the multi-reference datasets, we benchmark two models that implement state-of-the-art techniques; the models show competitive performance on rulec and achieve a new state-of-the-art performance on ru-lang8, with original single-reference benchmarks; using the union of 3 references increases system scores by 10 f-score points on average against a single annotator; (3) we analyze the effect of multiple references on individual error types and reveal interesting trends that are error specific: using multiple references substantially increases system scores on grammar and orthography, while the scores on lexical errors are affected only slightly; (4) we have the original annotator re-annotate a subset of rulec in accordance with the new re-writing annotation paradigm and show that the direct re-writing approach negatively affects system scores. ## multi-reference annotation below, we start with an overview of the referencebased evaluation in gec. then we describe the annotation paradigms and motivate our choice of the direct-re-writing annotation approach. the rest of the section describes the russian datasets and the multi-reference annotation. ## benchmarking experiments we implement two models, evaluate these on the original references and on the multi-reference benchmarks, and investigate how system scores are affected by a choice of a single rater. in section 4, we perform additional analyses. ## conclusion we enriched two russian gec benchmarks with additional annotations. we have analyzed and compared the resulting annotations and the original references and shown that the new annotators make more changes compared to the original raters, especially at the lexical level. we computed inter-annotator agreement and human-vs-human performance. we implemented two strong gec models and evaluated their performance on the new benchmarks. the gap between the model scores increases with the use of more references, whereas gap between human performance and system scores decreases, suggesting an improvement in the robustness of the results when multiple references are used.