Towards Better Utilization of Multi-Reference Training Data for C hinese Grammatical Error Correction (2024)

Abstract: For the grammatical error correction (GEC) task, there usually exist multiple correction ways for an erroneous input sentence, leading to multiple references. Observing the high proportion of multi-reference instances in Chinese GEC training data, we target a systematic study on how to better utilize multi-reference training data. We propose two new approaches and a simple two-stage training strategy. We compare them against previously proposed approaches, on two Chinese training datasets, i.e., Lang-8 for second language learner texts and FCGEC-Train for native speaker texts, and three test datasets. The experiments and analyses demonstrate the effectiveness of our proposed approaches and reveal interesting insights. Our code is available at https://github.com/ymliucs/MrGEC.

## introduction given a potentially erroneous sentence, the goal of the grammatical error correction (gec) task is to generate an error-free sentence of the same meaning @xcite . similar to the machine translation (mt) task, gec is a typical task for which multiple reference phenomena are ubiquitous, due to the great inherent flexibility of natural languages. for the sake of reliable evaluation, the test set is usually of multireference in both gec @xcite @xcite @xcite and mt research. in contrast, in terms of training data, gec differs from mt in that a large proportion of gec training data is multi-reference. in the most widely used training data for second-language learner (l2) texts, i.e., chinese lang-8, about 48% sentences despite the ubiquitousness of multi-reference training instances, there lacks a systematic investigation of the question, i.e., how to better utilize multi-reference data for training gec models. the most common way is to treat a multi-reference instance as multiple independent single-reference instances. @xcite propose to statically determine a single reference from multiple ones before training, and discard other references during training. they compare several ways for reference selection. this work aims to conduct a comprehensive investigation of this issue. we compare several approaches, including two newly proposed ones, i.e., 1) averaging the loss of multiple references at the same training step, and 2) dynamically determining and using the minimum loss among multiple references. we also propose a two-stage training strategy. we conduct experiments on two chinese training datasets, i.e., lang-8 for l2 texts and fcgec-train for ns texts, and three test datasets. the results and analyses reveal interesting insights. ## the basic seq2seq gec model given an input sentence @xmath0, a gec model aims to output another sentence @xmath1. in this work, we adopt the sequence-to-sequence (seq2seq) model, since it is consistently superior in performance and more flexible in handling complex errors, compared with the other mainstream model, i.e., sequence-to-edit (seq2edit) @xcite . the seq2seq gec model adopts the standard encoder-decoder framework of transformer @xcite . following @xcite , we employ the pre-trained bart @xcite as the model backbone. inference. at the τ -th timestamp, based on the encoded information of x and partially generated tokens @xmath2, the model generates the probability distribution in an auto-aggressive manner for the next position. training. given a single-reference training dataset, @xmath3, y (j) )} n @xmath4, the model applies local cross-entropy loss to maximize p τ (y τ ) in the teacher forcing manner. ## two baseline approaches multi-reference training data. as discussed in section 1, for gec it is ubiquitous that an erroneous sentence may have multiple ways for correction, leading to multiple references, especially when the sentence contains more complex errors (vs. spelling errors). we denoted a multireference data as @xmath5, y (j) )} n @xmath6, where undoubtedly, existing gec models cannot directly utilize m as training data. previous works propose two approaches to handle this issue. the concatenation (cat) approach. the cat approach splits a multi-reference training instance into several independent singlereference instances, each corresponding to an input/reference pair. formally, (x (j) , y (j) ) becomes {(x (j) , y (j) 1 ), . . . , (x (j) , y (j) k j )}. during training, instances for the same x (j) may appear in different batches due to shuffling. the minimum levenshtein distance (mld) approach. intuitively, the cat approach may make the training unstable, as the model swings among multiple references. to mitigate the uncertainty, @xcite proposes to select a single reference from multiple ones before training. after comparing several strategies, they find that it is best to select the reference that is closest to the input according to levenshtein distance @xcite , as follows. ## our approaches in this section, we propose two approaches for better utilization of multi-reference training data. the basic idea is to treat a multi-reference instance as a whole, and evaluate each reference after calculating losses of all references. moreover, we propose a two-stage training strategy to combine the power of two approaches. ## experiments we conduct experiments on two types of datasets: l2 and ns. l2 data. for the l2 experiment, we utilize lang-8 @xcite as the training dataset, foot_3 and mucgec @xcite as both the validation and test dataset. following previous practice and based on our observation, we preprocess the l2 datasets to improve the performance of the baseline model. details are given in appendix a.1. ns data. for the ns experiment, we utilize fcgec @xcite as the training, validation, and test dataset. meanwhile, nasgec-exam @xcite serves as an additional test set in the ns experiment. we find that the ns datasets have a severe data leakage problem. there are many same or very similar (input) sentences that are contained in both the training and test datasets, especially when we use nasgec-exam as the test set, since fcgec and nasgec-exam are from the same data source. this data leakage problem introduces uncertain factors and may affect the reliability of the experiment results. therefore, we propose a preprocessing procedure to deal with the issue. appendix a.2 gives the details. we strongly suggest that future research can follow our practice or even improve our procedure. besides fcgec and nasgec-exam, our preprocessing procedure also considers nacgec @xcite , which has the same data source, to facilitate future experiments, though we have not conducted experiments on nacgec in this work. evaluation metrics. in all the experiments, we employ p/r/f 0.5 scores calculated by cherrant @xcite for evaluation. ## related works as the most closely related work with ours, @xcite investigate multi-reference training for the mt task. they compare three methods, all of which are similar to the cat method in this work. unlike the circumstances for gec, the proportion of multi-reference instances is extremely low in training data for mt. therefore, they first train the model on large-scale single-reference data, and then continue training on small-scale multireference data that is actually composed of several other evaluation datasets. another interesting contribution of their work is to generate more references given a multi-reference instance. @xcite target the diverse mt task, which requires that a single model outputs multiple diverse but correct translations. they generate pseudo multi-reference training data using an offthe-shelf neural mt model, and during training adjust the loss function according to the quality of each reference and the diversity among references. similar to the mt task, the gec task also involves multi-reference instances, and the gec community has paid attention to this issue. @xcite annotate yaclc, a multi-reference gec dataset, which contains 14.6 references per input sentence on average. for a more accurate evaluation, cleme @xcite combines all references of the same input and evaluates the model performance at the chunk level. among these works, the question of how to better train models with multiple references has not been fully explored. the only previous work that we are aware of is @xcite , which proposes selecting a reference from multiple references for each input sentence based on handcrafted rules during the preprocessing phase. however, this approach may not fully utilize the information contained in multiple references. in contrast, our minl approach dynamically selects references in real time during the training phase. ## conclusion this paper focuses on optimizing the utilization of multi-reference gec training data. we introduce two new methods and a simple two-stage training strategy. the experiments on the l2 and ns datasets reveal that our method significantly improves the performance of the model training on multi-reference data, and outperforms previous proposed methods in the quality of selected references.