Grammatical Error Generation Based on Translated Fragments (2021)

Abstract: We perform neural machine translation of sentence fragments in order to create large amounts of training data for English grammatical error correction. Our method aims at simulating mistakes made by second language learners, and produces a wider range of non-native style language in comparison to a state-of-the-art baseline model. We carry out quantitative and qualitative evaluation. Our method is shown to outperform the baseline on data with a high proportion of errors.

## introduction grammatical error correction (gec) is the task of detecting and correcting grammatical errors in texts, typically written by second language learners. current state-of-the-art gec approaches are based on neural machine translation (nmt) @xcite . as in other natural language processing tasks, neural approaches to gec rely on large quantities of task-specific data, that is, sentence pairs consisting of erroneous source text coupled with corrected target text. however, in-domain gec data is scarce, and a number of solutions to the data sparsity problem have been proposed recently, often by introducing artificially created gec data into the training process. some error generation approaches also depend on error-annotated authentic learner data. for example, @xcite introduce errors probabilistically with error probabilities that are estimated using a learner corpus. @xcite train error detection and classification models on annotated data, focusing on verb errors. other methods dispense with the need for annotated data, such as approaches based on inverted spell-checkers and heuristic error generation @xcite . to alleviate the data sparsity problem, in this work we propose to use nmt to produce artificial training data, simulating real errors made by language learners. for instance, to produce english text with errors, we use nmt models to translate sentence fragments from other languages to english, and then combine the translated fragments to form our erroneous source data. similar machine translation approaches to gec data creation have been proposed before. for example, @xcite use a statistical machine translation model trained on reversed learner data, using the corrected sentences as source data and erroneous sentences as targets. @xcite extend this approach and use an nmt model to produce errors. @xcite perform extensive experiments on several neural models, likewise trained on learner data to generate errors. our contribution is to split the foreign-language source sentences into shorter fragments in order to limit the context that is available to the machine translation system. the rationale for doing this is to produce text that contains artefacts from the foreign language. since the nmt system needs to translate shorter fragments without the proper context, we expect it to produce more literal translations and to be less able to produce correct agreement between different parts of speech. additionally, polysemy may prompt the system to suggest translations of a synonym in the foreign language, which is not a synonym in english. the creation of synthetic training data involves further steps, which are described in section 2. model training is explained in section 3. in section 4 we evaluate our approach quantitatively against a strong baseline @xcite and make some qualitative assessments. ## training data the creation of our training data involves the following steps: first we pos tag the sentence pairs and align them automatically using minimum string edit distance coupled with some heuristics taking into account part of speech and inflection. the alignment algorithm is similar but not identical to errant @xcite . this procedure is illustrated in ## model training we build on the system described in @xcite . we choose not to make changes to the model or training parameters in order to isolate the effects of our data creation method and ensure a fair comparison. the same training setup is used for all models, with modifications only in the training sets. specifically, we use their "transformer big" architecture, with 6 self-attention layers, 16 attention heads, embeddings vectors of size 1024, and feed-forward hidden size of 4096 with relu activation functions. we also tie the encoder, decoder, and output embeddings. we also adopt the training setup of @xcite , and train our models with the marian toolkit @xcite . the models are first pretrained on the synthetic data for a maximum of 5 epochs. after pretraining, we finetune the best model checkpoint using the following corpora: fce @xcite , nucle @xcite , w&i-locness @xcite , and lang-8 @xcite . we use the w&i-locness development set for validation during training. we use early stopping with a patience of 10 with errant f 0.5 score on the w&i+locness development set used as the early stopping criterion. the checkpoint with the highest f 0.5 score is chosen for further finetuning. we choose the adam optimizer, a learning rate of 0.0002 and a linear warmup for 8k updates. we use marian's option to dynamically fit mini-batches to gpu memory, and train our models using 4 nvidia volta v100 gpus (32gb ram). in addition, we use strong regularization, which has been found useful in gec systems, with dropout probabilities of 0.3 between learner sentence: we had enjoy time . correction: we had a great time . alignment: prp vbd del(vb) ins(dt) ins(jj) nn . synthetic sentence: you be the the old donkey of the forestry correct reference: you 'll be the oldest donkey in the forest . alignment: prp ins(md) vbp del(dt) dt inf(jjs) nn del(in) ins(in) dt typ(nn) ins(.) ## evaluation we report results on two different data sets. we use the w&i-locness set, which was used as official test data in the bea19 gec shared task @xcite , as well as a subset of the english portion of learner texts derived from the finnish national certificates of language proficiency exams (yki). 3 we do not use the yki data as a blind test set, but instead use it to qualitatively analyze differences in model predictions. still, no part of the yki data was used during training or development of the models. we compare our results with those reported by @xcite , whose system achieved first place in the bea19 gec shared task. however due to limited resources we do not train an ensemble of models, but instead take a single left-to-right model from @xcite as baseline. their best system uses an ensemble approach with right-to-left and language model reranking and achieves a higher f 0.5 score of 69.47 on the w&i-locness test set. the upper part of the lower part of ## discussion and conclusion we have shown that our model rivals a competitive baseline, a left-to-right model by @xcite , which was one component of an ensemble model that performed best in the bea19 gec shared task. we did not yet train our own ensemble model, but we expect to see similar improvements in performance in future experiments. our results show that two models can perform on par, although they have been pretrained on different training corpora and using different error simulation techniques. in addition, the chunks model outperforms the baseline in noisy conditions. in the future, we would like to analyze further techniques for modeling challenging types of errors, which originate from structures that differ between the target language and the native languages of the language learners.