Grammatical Error Correction via Mixed-Grained Weighted Training (2023)

Abstract: The task of Grammatical Error Correction (GEC) aims to automatically correct grammatical errors in natural texts. Almost all previous works treat annotated training data equally, but inherent discrepancies in data are neglected. In this paper, the inherent discrepancies are manifested in two aspects, namely, accuracy of data annotation and diversity of potential annotations. To this end, we propose MainGEC, which designs token-level and sentence-level training weights based on inherent discrepancies therein, and then conducts mixed-grained weighted training to improve the training effect for GEC. Empirical evaluation shows that whether in the Seq2Seq or Seq2Edit manner, MainGEC achieves consistent and significant performance improvements on two benchmark datasets, demonstrating the effectiveness and superiority of the mixed-grained weighted training. Further ablation experiments verify the effectiveness of designed weights for both granularities in MainGEC.

## introduction the task of grammatical error correction (gec) aims to automatically correct grammatical errors in natural texts, which is extremely beneficial for language learners, such as children and non-native speakers @xcite . the currently dominant neural gec methods are categorized into two groups, i.e., seq2seq methods and seq2edit methods. seq2seq methods treat gec as a monolingual translation task, regarding errorful sentences as the source language and error-free sentences as the target language @xcite @xcite . seq2edit methods treat gec as a sequence tagging task, which predicts a tagging sequence of edit operations to perform correction @xcite @xcite . whether in the seq2seq or seq2edit manner, almost all previous works treat annotated training data equally @xcite , that is, assigning the same training weight to each training sample and each token therein. however, inherent discrepancies in data are completely neglected, causing degradation of the training effect. specifically, inherent discrepancies may be manifested in two aspects, i.e., accuracy of data annotation and diversity of potential annotations. the discrepancy in accuracy of data annotation refers to the uneven annotation quality, which is caused by differences in the annotation ability of annotators and the difficulty of samples @xcite . for example, in in this paper, we propose maingec (i.e., mixed-grained weighted training for gec), which designs mixed-grained weights for training data based on inherent discrepancies therein to improve the training effect for gec. first, we use a welltrained gec model (called a teacher model) to quantify accuracy and potential diversity of data annotation. on the one hand, the accuracy of annotations is estimated by the generation probability of the teacher model for each target token, which represents the acceptance degree of the teacher model for the current annotation. then, the quantified accuracy is converted into token-level training weights, as the accuracy of annotations may vary not only across samples but even across tokens in a single sample, e.g., sample 2 in we apply our mixed-grained weighted training to the mainstream seq2seq and seq2edit methods, and both of them achieve consistent and significant performance improvements on two benchmark datasets, verifying the superiority and generality of the method. in addition, we conduct ablation experiments, further verifying the effectiveness of the designed weights of both granularities. besides, we conduct the comparative experiment with the general knowledge distillation method on gec, verifying that our mixed-grained training weighting strategy outperforms the general knowledge distillation strategy. the main contributions of this paper are summarized as follows: (1) we investigate two kinds of inherent discrepancies in data annotation of gec for the first time, and propose maingec, which designs mixed-grained training weights based on the discrepancies above to improve the training effect. (2) the extensive empirical results show that maingec achieves consistent and significant performance improvements over the mainstream seq2seq and seq2edit methods on two benchmarks, proving the effectiveness and generality of our method for gec. ## preliminary this section presents the formulation of gec task and currently mainstream seq2seq and seq2edit methods for gec. ## experiments and results this section introduces our experiments and results on two benchmarks, i.e., conll-14 (ng et al., 2014) and bea-19 @xcite . then, we conduct ablation experiments on both-grained training weights and comparative experiments with the general knowledge distillation method. finally, a case study is presented to visualize the weights in maingec. ## related works gec is a fundamental nlp task that has received wide attention over the past decades. besides of the early statistical methods, the currently mainstream neural gec methods are categorized into two groups, i.e., seq2seq methods and seq2edit methods, in general. seq2seq methods treat gec as a monolingual translation task, regarding errorful sentences as the source language and error-free sentences as the target language @xcite . some works @xcite generate considerable synthetic data based on the symmetry of the seq2seq's structure for data augmenta-tion. in addition, some works @xcite feed additional features into the neural network to improve gec, such as the bert @xcite presentation or syntactic structure of the input sentence. seq2edit methods treat gec as a sequence tagging task, which predicts a tagging sequence of edit operations to perform correction @xcite . parallel iterative edit (pie) @xcite and gector @xcite ## conclusion this paper proposes maingec, which assigns mixed-grained weights to training data based on inherent discrepancies in data to improve the training effect for gec. our method uses a well-trained gec model to quantify the accuracy and potential diversity of data annotation, and convert them into the mixed-grained weights for the training process. whether in the seq2seq or seq2edit manner, maingec achieves consistent and significant performance improvements on two benchmark datasets, verifying the superiority and generality of the method. in addition, further ablation experiments and comparative experiments with the general knowledge distillation method provide more insights on both-grained training weights and the perspective of knowledge distillation.