Training for Grammatical Error Correction Without Human-Annotated L 2 Learners’ Corpora (2023)

Abstract: Grammatical error correction (GEC) is a challenging task for non-native second language (L2) learners and learning machines. Data-driven GEC learning requires as much human-annotated genuine training data as possible. However, it is difficult to produce larger-scale human-annotated data, and synthetically generated large-scale parallel training data is valuable for GEC systems. In this paper, we propose a method for rebuilding a corpus of synthetic parallel data using target sentences predicted by a GEC model to improve performance. Experimental results show that our proposed pre-training outperforms that on the original synthetic datasets. Moreover, it is also shown that our proposed training without human-annotated L2 learners’ corpora is as practical as conventional full pipeline training with both synthetic datasets and L2 learners’ corpora in terms of accuracy.

## introduction grammatical error correction (gec) is one of the essential processes needed to produce sentences in a grammar-based language, and it is a challenging task for non-native second language (l2) learners and learning machines as well. each language has its own grammar, however, data-driven language learning by a machine does not use the grammar, but corpora, more preferably, large-scale corpora. while classifiers that predict some token from candidates for a certain position in a sentence have been developed in the past @xcite , sequence-to-sequence models have become more popular for gec because the task is regarded as a sequence-to-sequence one and the models are flexible in editing sentences and covering various error types. in sequence-to-sequence models, @xcite and @xcite treat the task as a statistical machine translation (smt) problem and produce state-of-the-art performance on the conll2014 shared task. neural machine translation models @xcite , which consist of an encoder and a decoder, also have been investigated to improve their capabilities. in particular, the transformer @xcite , which is an encoder-decoder model incorporating a self-attention mechanism, has become popular and various improved versions have been investigated. one of its alternative architectures is the copy-augmented transformer, which has become popular for gec @xcite . another modification to the transformer architecture is altering the encoder-decoder attention mechanism in the decoder to accept and make use of additional context. for example, @xcite use the bert representation of the input sentence as additional context for gec. gector @xcite employs a bert-like pre-trained encoder stacked with a linear layer with the softmax activation function, and treats the gec task as a token labeling problem. addressing training data for gec models, @xcite , @xcite and @xcite employ synthetically generated pseudo data for pre-training of gec systems prior to fine-tuning on human-annotated corpora for the building educational applications (bea) 2019 shared task @xcite . this paper addresses the effectiveness of synthetic parallel data, which is generally used as a consequence of the insufficiency of humanannotated l2 learners' corpora. we propose a method of substituting target sentences in synthetic parallel data with alternatives and rebuilding synthetic datasets to boost gec training. experiments demonstrate that pre-training on synthetic datasets rebuilt by the proposed method outperforms pretraining on the original synthetic datasets. moreover, our synthetic datasets can be effectively employed not only to pre-train, but also to fine-tune gec models, that is, training on synthetic data only all through the pipeline. the gec model's training without l2 learners' corpora is as practical as conventional training with both synthetic datasets and l2 learners' corpora in terms of accuracy. synthetic parallel training data consist of erroneous sentences generated by corruption models from error-free sentences. in general, the corruption models can generate unlimited versions of erroneous sentences from a given error-free one, with the ability to vary the versions in the number of errors, error types, etc. back-translation @xcite provides monolingual training data with synthetic source sentences that are obtained from automatically translating the target sentence into the source language for nmt. @xcite apply back-translation to gec and achieves state-of-the-art performance on the conll2014 and bea2019 test datasets. pie synthetic data @xcite is often used in state-of-the-art gec models proposed by @xcite ; @xcite , etc. seq2edits @xcite is a sequence-to-sequence transducer which consists of a transformer encoder and decoders, and can predict span-based edit operation probabilities for gec. @xcite , furthermore, propose tagged corruption models using both seq2edits and a finite state transducer to match the observed error type distribution of the bea2019 dev dataset, and generate synthetic data for pretraining gec models. ## erroneous synthetic data rebuilt by gec models in this section, we further examine the problem described in the previous section and propose to rebuild conventional synthetic datasets, which are often employed by researchers, in order to create effective synthetic parallel training datasets for pre-training. a trained gec model can be pie 9m source y 1 : there have also been recent battles over access to multiple myeloma drug lenolidamide. corrupted:x 1 : there have been recent battles to access to multiple myeloma drug lenolidamide. predicted ỹ1 : there have been recent battles to access to multiple myeloma drug lenolidamide. c4-200m source y 2 : we just have to live with black that are not truly black. corrupted x 2 : we have to live in black that are not black. predicted ỹ2 : we have to live in black that are not black. we train the model g with given datasets of incorrect and correct sentence pairs: @xmath0, y i )|@xmath1, • • • , n }, where the size of d is n , so as to decrease the difference (loss) of y i between ỹi . ## experiments to empirically investigate the effectiveness of the proposed method and the capabilities of a gec model trained on synthetic data rebuilt by the method, we train the gec model choosing the hyperparameters described below. the gec model is fundamentally trained through the three stage pipeline adopted in choe et al ## discussion and related work this paper addresses the quality of synthetic parallel data due to the insufficiency of human-annotated l2 learners' corpora and the effectiveness of training only on synthetic data. note that the quality does not address grammatical correctness, but the validity of source-target sentence pairs for training and how well the data fits the characteristics of l2 learners' mistakes. the overall results indicate that our method is more effective for the pie-9m dataset than the c4-200m dataset, and it implies that the c4-200m dataset is of better quality. here, we discuss the experiments on the pie-9m dataset, which more likely needs the technique. the stage-i training by the proposed method outperforms the conventional training by 10.0 and 10 https://www.cl.cam.ac.uk/research/nl/bea2019st/ 15.6 with regard to f 0.5 for the conll2014 and bea2019 test datasets, respectively. it results in only 1.7 and 3.8 less than the baselines, which were trained through the full pipeline, stages i, ii, and iii. furthermore, the stage-ii training reduces the performance of the pre-trained models on the proposed method's synthetic dataset. this suggests that the proposed method's synthetic datasets could be of higher quality than the overall l2 learners' corpora while each synthetic dataset itself could be inferior to the l2 learners' corpora. unfortunately, the baseline of the training replaced with the rebuilt synthetic dataset does not improve its performance. our synthetic datasets can be employed all through the pipeline of training, that is, training without l2 learners' corpora. the results show that gec model training without l2 learners' corpora is as practical as conventional training with both l2 learners' corpora and synthetic datasets in terms of accuracy. note that the version of the model employed to rebuild synthetic data in the experiments achieves the scores of 64.0 and 71.8 on f 0.5 for the conll 2014 and bea 2019 test datasets, respectively. to summarize the achievements, the proposed method : trained gec models can be used not only for predicting correct sentences but also for generating better synthetic data, and systems incorporating the proposed method are not limited to the synthetic data and model used in this paper. addressing training data for gec models, @xcite introduce the wiked error corpus generated from wikipedia revision histories, corpus content and format. the corpus consists of more than 12 million sentences with a total of 14 million edits of various types. @xcite al. (2019), grundkiewicz et al. (2019) and choe et al. (2019) employ synthetically generated pseudo data for pre-training of gec systems prior to fine-tuning on human-annotated corpora for the bea2019 shared task @xcite . mita et al. (2020) focus on human annotators' errors in official datasets when they rewrite incorrect sentences to remove grammatical mistakes and denoise the target sentences of the official datasets using some trained gec models with a perplexity criterion. @xcite also apply the similar technique to the lang-8 corpus, which is a large corpus of texts written by l2 learners with userannotated corrections, and correct human errors by the gec models. our proposed method is effective not only for correcting human annotators' errors, but also for adjusting source-target disparity to match the domain. @xcite build a large synthetic pre-training dataset with error tag frequency distributions matching seq2edits @xcite . @xcite trained a generator to generate increasingly realistic errors (in the form of token-based edit labels) and a discrimina-tor to differentiate between artificially-generated edits and human-annotated edits. @xcite propose tagged corruption models using both the seq2edits and a finite state transducer to match the observed error type distribution of the bea2019 dev dataset, and generate synthetic data for pre-training gec models. @xcite apply bifi algorithm @xcite and lm-critic to synthetic data to generate better datasets for gec. lm-critic chooses the most likely grammatical sentence from multiple sentence candidates based on the sentence occurrence probabilities generated by a language model. ## conclusion in this paper, we have addressed the effectiveness of synthetic parallel data and have proposed a method for rebuilding a corpus of synthetic parallel data using target sentences predicted by a gec model. while the original target sentences in synthetic parallel data are guaranteed to be error-free, the target sentences predicted by a gec model contain grammatical errors because the gec model has been developed through research and is not perfect in its performance. however, pre-training on our proposed synthetic data outperforms that on the original synthetic data, and our pre-trained gec model showed performance only slightly lower than the conventional fine-tuned gec model. in addition, our proposed method can provide notably good training performance without humanannotated l2 learners' corpora. the proposed method's target sentences by an imperfect gec model work better than the original error-free target sentences although the former may contain grammatical errors. the reason why this paradoxical result happens needs to be determined. in future work, we plan to investigate further reconfiguration and modification of synthetic parallel data, and fine-tune training using such data to improve the performance of gec. investigation of the source-target relationships on training data mentioned above should also be carried out to clarify the effectiveness of the proposed method.