Low-Resource Grammatical Error Correction: Selective Data Augmentation with Round-Trip Machine Translation (2025)

Abstract: Supervised state-of-the-art methods for grammatical error correction require large amounts of parallel data for training. Due to lack of gold-labeled data, techniques that create synthetic training data have become popular. We show that models trained on synthetic data tend tocorrect a limited range of grammar and spelling mistakes that involve character-level changes, but perform poorly on (more complex) phenomena that require word-level changes. We propose to address the performance gap on such errors by generating synthetic data through selective data augmentation via round-trip machine translation. We show that the proposed technique, SeLex-RT, is capable of generating mistakes that are similar to those observed with language learners. Using the approach with two types of state-of-the-art learning frameworks and two low-resource languages (Russian and Ukrainian), we achieve substantial improvements, compared to training on synthetic data produced with standard techniques. Analysis of the output reveals that models trained on data noisified with the SeLex-RT approach are capable of making word-level changes and correct lexical errors common with language learners.

## introduction grammatical error correction (gec) is the task of detecting and correcting mistakes in text. supervised state-of-the-art approaches to gec require large amounts of training data in the form of sentence examples with errors and their corrected counterparts. because hand-labeled data is expensive to obtain, it is common practice to use monolingual data with synthetic noise for pre-training. the models are further finetuned on gold learner data. the use of synthetic data is essential to obtaining good performance, especially when the language is lowresource and has a limited amount of gold-labeled training data @xcite . however, ex-isting methods do not generate sufficiently diverse synthetic errors @xcite . we show that low-resource state-of-the-art strategies for generating synthetic data mainly produce errors in spelling and inflectional morphology, and models trained on such synthetic noise fail to address errors related to overall fluency that are known to pose challenges even to high-proficiency learners @xcite . fluency is used to refer to edits that make the original text native-sounding; these edits typically go beyond grammaticality and may include a change for a more preferred word order or a better lexical choice @xcite . in this work, we focus on correcting lexical errors. many of these errors are known to arise from first-language interference @xcite @xcite . moreover, studies in second language acquisition reveal that learners may generate a sentence in the second language by translating it from their native language @xcite , thereby incorrectly transferring structures and expressions into the second language. a lexical error may occur when a learner picks an incorrect but semanticallyrelated translation for an ambiguous word with several related meanings in their native language. using this observation, we hypothesize that lexical challenges, such as choosing an incorrect word in the second language, will also manifest themselves in the machine translation output, as backtranslated words that are semantically close to the target. we propose to address such errors using select substitutions obtained from round-trip machine translation. indeed, we show that the approach with select substitutions is capable of generating a wide range of mistakes that non-native speakers make, and is particularly good at producing confusions similar to lexical errors common among non-native speakers. we refer to this approach as selex-rt. we start with a monolingual corpus of language l for which we wish to build a gec model. the sen-tences are translated into another language (pivot), and then back into l. the original sentence is token-aligned with its round-trip translation, and the alignments are used to create confusions resulting from imperfect translations. in contrast to other approaches that employ round-trip translation @xcite , we (1) do not make use of the entire resulting back-translated sentences, but only generate targeted confusion sets of relevant errors that are used to corrupt the data; and (2) generate multiple translation hypotheses in each direction. we demonstrate that both of these innovations are crucial for obtaining a diverse set of high-quality synthetic errors (see section 6.1). we present experiments on two low-resource languages for gec, russian and ukrainian, with additional analyses on russian. we show that only 5%-12% of the lexical errors observed in gold learner data are represented in the data produced with the standard synthetic methods. in contrast, selex-rt is capable of generating synthetic data that represents over 50% of learner lexical errors. experimental results demonstrate substantial performance gains on correcting lexical errors, compared to training with standard synthetic data. the paper makes the following contributions: (1) using two russian benchmarks, we show that the majority of mistakes identified by state-of-theart supervised gec models are in spelling and inflectional morphology; (2) we propose selex-rt, a novel method that uses select substitutions from round-trip translations and generate synthetic errors that involve replacements of unrelated but semantically similar words and complements existing approaches; (3) we use selex-rt to generate synthetic data for two low-resource languages, and substantially improve over standard methods; additional analysis on russian reveals that selex-rt is particularly beneficial for correcting lexical errors. ## background supervised approaches to gec can be broken down into sequence-to-sequence (seq2seq) @xcite @xcite @xcite @xcite @xcite , and sequence-to-editing @xcite @xcite . both achieve state-of-the-art performance on en-glish gec. recent works also achieve strong results with edit ranking (sorokin, 2022) or ensembling @xcite . other works have made additional advancements on english gec, but they rely on large amounts of hand-labeled data that is not available for other languages @xcite @xcite . the edit-based framework (stahlberg and kumar, 2020; @xcite was shown to be competitive on english, however, for other languages proved to be less successful @xcite , due to the fact that the approach requires language-specific knowledge to develop rules @xcite . (it is important to note that almost all non-english gec can be viewed as low-resource, due to scarcity of handlabeled training data). the seq2seq framework, on the other hand, has shown state-of-the-art performance in non-english gec @xcite , and we thus adopt this approach in our work. in the seq2seq approach, gec is cast as a machine translation task with the erroneous sentences treated as the source and corrected sentences treated as the target. sentences with real learner errors and their manually-corrected counterparts or sentences from a monolingual corpus with added synthetic noise can be used for training gec models. pretrained language models (plms) can be used as a starting point @xcite @xcite . we follow rothe et al. (2021) that successfully applied the approach in multilingual settings, making use of mt5. low-resource gec compared to english, russian and ukrainian are low-resource for gec, due to the limited amounts of labeled training data (see ## learner datasets russian and ukrainian gold data we use two datasets of russian learner data manually corrected for errors foot_1 rulec-gec @xcite (henceforth rulec) and @xcite . both datasets were originally annotated with a single gold reference each (a corrected version produced by an expert), however, the annotations have been en for ukrainian, we use the ua-gec dataset @xcite . we report results on the fluency track (that includes both grammar and fluency edits). dataset sizes are in ## experimental setup model architecture drawing on the methods that showed superior performance in multilingual gec @xcite , we chose the seq2seq framework (see section 2): we implement two types of transformer seq2seq models -a smaller one trained from scratch, and a larger model that uses mt5 as a starting point. model 1 our (smaller) model is pre-trained on synthetic data and finetuned on the gold data. it has 275m parameters. model 2 we adopt the approach of rothe et al. (2021), making use of mt5 (xue et al., 2021). rothe et al. (2021) finetune mt5 on gec gold data only, but palma gomez et al. (2023) show the importance of a 2-step finetuning, which we adopt here: in the first step, mt5 is finetuned on the synthetic data, and in the second step we finetune on the gold training data. due to computational constraints, we use mt5-base that has 580m parameters. larger models (large, xl, xxl) lead to stronger results both with 1-step finetuning on gold data only (rothe et al., 2021) and with a 2-step finetuning on russian (palma gomez and rozovskaya, 2024) and ukrainian (palma gomez et al., 2023). appendix a lists the training details and the hyperparameters. baseline synthetic methods we implement four data generation techniques: (1) character-level transformations (char); (2) spell-based transformations (spell); and (3) morphology-based transformations (morph), ( 4 ) and a combination morph+spell. 7 appendix c provides more detail on how we generate errors for the two languages. ## key results baselines we use the smaller model 1 to compare the baseline methods. model 1 is trained on 15m synthetic sentence pairs, while the mt5-models are trained on 2m synthetic sentence pairs (due to computational constraints). gold training data (in as the tables show, the char method is the weakest baseline, whereas morph outperforms spell synth. for russian, while spell outperforms morph for ukrainian. the combination morph+spell produces a stronger model for both languages. we also show the selex-rt performance on its own. the middle part of the tables shows the results of adding selex-rt to the best baselines. selex-rt errors are added to the text after adding errors produced with the baseline methods. 8 the bottom part of the tables compares model 2 types using the best baseline data generation strategy. findings the improvements from adding selex-rt are consistent across all models and both languages. the differences between the best models with the selex-rt component and their respective baselines are all statistically significant at the .03 level on both russian benchmarks. 9 we do not perform statistical significance testing on ua-gec since the gold references are not publicly available. on russian, improvements are larger for model 1 than for model 2. by contrast, improvements in ukrainian are slightly larger for model 2. we believe this may signal that using a pre-trained language model (mt5) is more beneficial for russian, as the mc4 dataset on which mt5 is pre-trained contains 18 times less ukrainian data compared to russian, thus the selex-rt component could have a larger impact on ukrainian. this finding is promising for low-resource gec scenarios. comparing the two model types, model 2 outperforms model 1, with the exception for ru-lang8. we hypothesize this may be due to overfitting (we are finetuning on rulec data, which is really small), and thus a model with more parameters is more likely to overfit, resulting in a less optimal performance on out-of-domain ru-lang8 data. ## conclusion we present a novel approach to creating synthetic data for correcting language learner lexical errors. we hypothesized, drawing on evidence from second language acquisition research, that challenges facing language learners will also manifest in the machine translation output, as many lexical errors result from deviations in the second language from the native usage. our approach creates controlled confusions from token-aligned sentences and their translations. crucially, we do not use full-sentence back-translation and generate multiple translation hypotheses, which allows us to create diverse and high-quality synthetic errors. extensive experiments on two low-resource languages -russian and ukrainian -and three benchmarks, and additional analyses on russian demonstrate the effectiveness of our approach. future work will focus on the application of this method to other languages.