M u CGEC : a Multi-Reference Multi-Source Evaluation Dataset for C hinese Grammatical Error Correction (2022)

Abstract: This paper presents MuCGEC, a multi-reference multi-source evaluation dataset for Chinese Grammatical Error Correction (CGEC), consisting of 7,063 sentences collected from three Chinese-as-a-Second-Language (CSL) learner sources. Each sentence is corrected by three annotators, and their corrections are carefully reviewed by a senior annotator, resulting in 2.3 references per sentence. We conduct experiments with two mainstream CGEC models, i.e., the sequence-to-sequence model and the sequence-to-edit model, both enhanced with large pretrained language models, achieving competitive benchmark performance on previous and our datasets. We also discuss CGEC evaluation methodologies, including the effect of multiple references and using a char-based metric. Our annotation guidelines, data, and code are available at https://github.com/HillZhang1999/MuCGEC .

## introduction given a potentially noisy input sentence, grammatical error correction (gec) aims to detect and correct all errors and produce a clean sentence. recently, gec has increasingly gained attention for its vital value in various downstream scenarios @xcite . to support gec research, high-quality manually labeled evaluation data is indispensable. for english gec (egec), such datasets are abundant @xcite @xcite @xcite @xcite . however, chinese gec (cgec) evaluation datasets are relatively scarce. the two publicly available cgec evaluation datasets are nlpcc18 and cged, contributed by the nlpcc-2018 (zhao † this work was partially done during the first author's internship at alibaba damo academy. ## analysis of our annotated data overall statistics of mucgec are shown in first, from the proportion of erroneous sentences, we can see that most of the sentences are considered to contain grammatical errors in the original annotation, but a considerable part of them are not corrected in our annotation. we attribute this to our strict control of the over-correction phenomenon. second, regarding sentence lengths, nlpcc18 has the shortest sentences, whereas cged sentences are much longer. this may be because candidates on the hsk examination, an official chinese proficiency test, tend to use long sentences to show their ability in chinese. third, each sentence in the re-annotated nlpcc18 receives 2.5 references on average, which is more than twice that in the original nlpcc18 data. overall, each sentence obtains 2.3 references. we believe the multi-reference characteristic makes our dataset more reliable for evaluation, which is further discussed in section 6.3. finally, we compare the number of char-based edits per reference in different datasets. we describe how to derive such edits in detail in section 6.2. we can see that the number of edit is tightly correlated with sentence length. the difference in the average sentence length and number of edits indicates that the three data sources may have a systematic discrepancy in quality and difficulty, which we believe is helpful for evaluating the generalization ability of models. moreover, compared with nlpcc18-orig, we annotate 25% more edits (2.0 vs. 2.5) in each reference. we believe the major reason is that the original nlpcc18 data are annotated under the minimal edit distance principle @xcite , which requires annotators to select a reference with fewer edits when correcting errors. sentence distribution with respect to numbers of references is shown in figure 1 . here, we only consider erroneous sentences. identical references from different annotators are counted as one reference. overall, most sentences have 2 references, accounting for 39.4%; 29.1% of sentences have 3 references; 21.8% of sentences have only 1 reference, most of which are short and easy to correct. we believe that the average number of references could be further increased if more annotators were assigned to each sentence. it is also worth noticing that annotators tend to submit a single reference, despite the fact that our annotation tool allows annotators to submit multiple ones. we suspect the reason may be that it is more economical for annotators to do so. one the one hand, it may be easy to come up with the most suitable correction, whereas thinking of alternatives is more time-consuming. on the other hand, we did not give enough consideration to this issue when designing the salary computation formula. in the future, we plan to optimize (or simplify) our annotation workflow so that each annotator is required to give only one reference which he/she thinks is the best, and assign each sentence to more annotators if we need more references. human annotation performance. in order to assess the annotation ability of our annotators and human performance for cgec task, we calculate char-based f 0.5 scores by evaluating all annotation submissions against the final golden references. we describe how to compute char-based metrics in detail in section 6.2. each reference submitted by an annotator is considered as a sample. overall, the average f 0.5 is 72.12, which we believe could be higher if we discarded data that were annotated 1 2 3 4 5 6 7 8 9 10 11 12 13 14 at the early stage of our project when annotators were less experienced and less familiar with our guidelines. figure 2 shows f 0.5 scores of 15 annotators who annotated the most sentences, in descending order of the number of annotated sentences. we can see that human performance varies across different annotators. the best annotator achieves an 82.34 f 0.5 score, while the annotator who completes the most tasks only gets a score of 68.32. this indicates that we should pay more attention to annotation quality when calculating salaries and prevent annotators from focusing too much on annotation speed. common mistakes made by annotators. we randomly select 300 invalid references rejected by reviewers and try to understand what mistakes are more frequently made by annotators. we manually classify all selected references into three mistake categories. the most frequent mistakes are caused by incomplete correction and account for 56.7% of the invalid references. we found that it was sometimes difficult for annotators to correct all errors without omissions, possibly due to the complexity or flexibility of chinese grammar. the second frequent type of mistakes is erroneous correction, which means that the correction of old errors incurs new errors, with a proportion of 32.3%. besides, 11.0% of submissions are rejected due to meaning change, which means that the correction changes the intended meaning of the original sentence. ## benchmark models to understand how well cutting-edge gec models perform on our data, we adopt two mainstream gec approaches, i.e., seq2edit and seq2seq. both models are enhanced with plms. we also at-tempt to combine them after observing their complementary power in dealing with different error types. this section briefly describes these benchmark models. due to the space limitation, please kindly refer to appendix c for more model details. the seq2edit model treats gec as a sequence labeling task and performs error corrections via a sequence of token-level edits, including insertion, deletion, and substitution @xcite . a token corresponds to a word or a subword in english, and to a character in chinese. with minor modifications to accommodate chinese, we adopt gector @xcite , which achieves the sota performance on egec datasets. following recent seq2edit work like awasthi et al. (2019) and omelianchuk et al. (2020) , we enhance gector by using plms as its encoder. after comparing several popular plms, we choose structbert @xcite foot_3 due to its superior performance after fine-tuning (see the seq2seq model straightforwardly treats gec as a monolingual translation task @xcite . recent works propose to enhance transformer-based @xcite seq2seq egec models with plms like t5 @xcite or bart @xcite . unlike bert @xcite , t5 and bart are specifically designed for text generation. therefore, it is straightforward to continue training them on gec data. we follow these works and utilize the recently proposed chinese bart from @xcite to initialize our seq2seq model. the ensemble model. several previous works have proven the effectiveness of model ensemble for cgec @xcite . in this work, we clearly observe the complementary power of the above two models in fixing different error types (see we adopt a simple edit-wise voting mechanism. the edits are at the char-based span level, and correspond to four error types. please refer to section 6.2 for detailed explanation of our char-based evaluation metric. more specifically, we aggregate edits from the results of each model, and only preserve edits that appear more than n/2 times, where n is the number of models. in other words, an edit is kept in the final result only if it is produced by a majority of models. 4 we experiment with two ensemble settings: 1) one seq2edit and one seq2seq, denoted by "1×seq2edit+1×seq2seq", and 2) three seq2edit and three seq2seq, denoted by "3×seq2edit+3×seq2seq".the three seq2edit models are obtained using different random seeds for initialization, and the same goes for the seq2seq. other settings. to obtain the single-model performance of both kinds of models, we run them three times separately with different random seeds for initialization and calculate average metrics. for "1×seq2edit+1×seq2seq", we random select a pair of single models. for "3×seq2edit+3×seq2seq", we aggregate the results of all six single models. ## experiments on nlpcc18-orig data in order to show that our benchmark models are competitive among existing cgec models, we conduct experiments on the original nlpcc18 test set, on which most previous cgec systems are tested. training data. for the sake of easy replicability, we limit our training data strictly to public resources, i.e., the lang8 5 data @xcite and the hsk 6 data @xcite . we filter duplicate sentences that appear in our dataset, and discard correct sentences. the final lang8 and hsk data contains 1,092,285 and 95,320 sentence pairs, respectively. the hsk data is cleaner and of higher quality than lang8, but is smaller. following the re-weighting procedure of junczys-dowmunt et al. (2018), we duplicate the hsk data five times, and merge them with lang8 data. comparison with previous works. adding the hsk training data improves performance of all our models by about 4 points. our two benchmark models already achieve sota performance under the single-model setting. the model ensemble technique leads to obvious performance gains (more than 5 points) over single models. however, the gains from increasing the number of component models seem rather small. we try to explain this issue in section 6.3. for seq2edit, we additionally present results with other plms besides structbert, including bert @xcite , roberta @xcite , and macbert @xcite ) from the hugging face foot_5 website. we use the "large" variants of all plms. in this work, we propose to provide a fixed dev set for our newly annotated dataset, by randomly selecting 1,125 sentences from the cged source, denoted as cged-dev. the remaining 5,938 sentences are used as the test set, in which each data source has a roughly equal amount of sentences, i.e., 1,996 sentences for nlpcc18-test, 2,000 for cged-test, and 1,942 for lang8-test. ## related works egec resources. there has been a lot of work on egec data construction. as the two earliest egec datasets, fce @xcite and nu-cle @xcite adopt the error-coded annotation paradigm. in contrast, jfleg @xcite collects sentences from tofel exams and adopts the direct rewriting paradigm. w&i @xcite also chooses the direct rewriting paradigm, and extra annotates a score indicating the language proficiency level of the writer for each input sentence. all four datasets are composed of essays from non-native english speakers and provide multiple references. recently, researchers have started to annotate small-scale egec data for texts written by native english speakers, including aesw @xcite , locness @xcite , gmeg @xcite and cweb @xcite . in the future, we plan to extend this work to texts written by native chinese speakers. cgec resources. compared with egec, progress in cgec data construction largely lags be-hind. as discussed in section 1, nlpcc18 @xcite and cged @xcite are the only two evaluation datasets for cgec research. besides them, there are also a few resources for training cgec models, e.g., the lang8 corpus @xcite and the hsk corpus @xcite . concurrently with this work, @xcite present a multi-reference cgec dataset, named as yet another chinese learner corpus (yaclc), containing 32,124 sentences from lang8. each sentence is annotated by 10 annotators. recent progress in cgec. in the nlpcc-2018 shared task @xcite , many systems adopt seq2seq models, based on rnn/cnn. recent work mainly utilizes transformer @xcite @xcite . @xcite first employ a seq2edit model for cgec, and achieve comparable performance with the seq2seq counterparts. some systems in the cged-2020 shared task @xcite directly employ the open-source seq2edit model, i.e., gector @xcite . most seq2edit models use plms like bert @xcite to initialize their encoders. besides the above two mainstream models, @xcite for the first time apply a non-autoregressive neural machine translation model to cgec. besides modeling optimization, techniques like data augmentation @xcite and model ensemble @xcite have proven to be very useful for cgec. ## conclusions this paper presents mucgec, a newly annotated evaluation dataset for cgec, consisting of 7,063 sentences written by csl learners. compared with existing cgec datasets, ours can support more reliable evaluation due to three important features: 1) providing multiple references; 2) covering three text sources; 3) adopting strict quality control (i.e., annotation guidelines and workflow). after describing the data construction process, we perform detailed analyses of our data. then, we adopt two mainstream and competitive cgec models, i.e., seq2seq and seq2edit, and carry out benchmark experiments. we also propose to adopt char-based evaluation metrics to replace previously used wordbased ones.