Grammatical Error Correction through Round-Trip Machine Translation (2023)

Abstract: Machine translation (MT) operates on the premise of an interlingua which abstracts away from surface form while preserving meaning. A decade ago the idea of using round-trip MT to guide grammatical error correction was proposed as a way to abstract away from potential errors in surface forms (Madnani et al., 2012). At the time, it did not pan out due to the low quality of MT systems of the day. Today much stronger MT systems are available so we re-evaluate this idea across five languages and models of various sizes. We find that for extra large models input augmentation through round-trip MT has little to no effect. For more ‘workable’ model sizes, however, it yields consistent improvements, sometimes bringing the performance of a base or large model up to that of a large or xl model, respectively. The round-trip translation comes at a computational cost though, so one would have to determine whether to opt for a larger model or for input augmentation on a case-by-case basis.

## introduction grammatical error correction (gec) is the task of detecting and correcting errors in text. it finds application in both assisted writing and second language learning. as training data for the task is scarce, efforts in this space largely focus on transfer learning and data augmentation. in this work, we revisit the use of round-trip machine translation in grammatical error correction, as originally proposed in @xcite . machine translation (mt) aims to preserve the meaning of text while mapping its surface form from one language into another. the ideal mt system would be robust to minor perturbations in the input text like a typo or a grammatical error, producing a well-formed translation true to the intended meaning. if the translated text is then backtranslated into the source language, we can expect to see the original content, now free from errors. this figure 1: our approach. the input to the model is a concatenation of the original text and the round-trip translation. here, english is used as target language and romanian as pivot language for illustrative purposes; in actual experiments english is the pivot language. we explore the impact of this strong mt performance on gec by augmenting the input to a gec system through round-trip translation, such that each input sentence is concatenated with a round-trip translation of itself (see figure 1 ). we evaluate the effect of this procedure on five languages: german, russian, spanish, czech, and romanian (de, ru, es, cs, ro). in our experiments, we fine-tune the multilingual pre-trained language model mt5 @xcite , which is available in a range of sizes. the xxl variant is currently the state-of-the-art on de, ru, and cs @xcite . however, mt5-xxl, with its 13b parameters, is out-of-scope for most academic research and impractical for deployment in many application. therefore, we experiment only with the three smaller variants, base, large and xl. we find that round-trip translation successfully guides the correction of grammatical errors in base models for all languages, with improvements of up to 4.1 points on the f-score (for ru). for large models, it still benefits three out of five languages, leaving scores on the other two unchanged. for xl models, it has a negligible effect in either direction, showing that these models are sufficiently strong by themselves and subsume the knowledge an mt model can provide. for some base and large configurations, the round-trip translation augmentation closes the gap between a model of a given size, e.g. large-ro, and its larger counterpart, e.g. xl-ro. since round-trip translation has an added computational cost itself, one would have to weight the costs and benefits on a per-case base to determine whether a larger model with bare input or a smaller model with augmented input is more suitable for a given application. ## background machine translation makes various appearances across research in grammatical error correction. modeling approaches and training tricks originally developed in the context of mt have been successfully adapted to gec @xcite @xcite . the concept of backtranslation has been used to generate synthetic data for gec @xcite . in all of these works, mt research provides the methods but there is no actual crosslingual translation happening. the 'translation' in this case is from ungrammatical text to grammatical text in the same language. @xcite perform actual translation of chinese text into english using mt systems of varying quality as a way to generate ungrammatical english data, which they then pair with gold standard targets to obtain a synthetic training corpus. in contrast to such works, our work explores the potential of round-trip translation as an intermediate step in the process of gec, active both during fine-tuning and inference. the goal here is to make use of the knowledge one can extract from parallel mt data, generally much more abundant than gec data. most similar to our work is that of @xcite , who perform round-trip translation of an input in eight pivot languages with google translate and use a lattice to combine all hypotheses into a final output. the motivation behind using multiple pivot languages is to ensure meaning preservation on one hand, and to increase the chance of all errors being corrected on the other. the authors observe some successes but also numerous failures in the predictions of their model, attributing the latter to new errors of disfluency and loss of meaning introduced by google translate, which at the time was based on statistical mt. in the decade since that work was published, mt has undergone a paradigm shift from statistical to neural network-based methods, marked by large improvements in performance @xcite . it is therefore time to revisit the potential gains from round-trip mt for gec. as we recognize that gec aims for minimal and necessary revisions of the input whereas round-trip translation can result in valid but unnecessary lexical and syntactic changes, we condition the generation of the final output on both the input sentence and the round-trip translation, in an approach akin to multi-source automatic post-editing @xcite . ## method in general terms, our approach is one of sequenceto-sequence text generation with input augmentation: for a given input sentence, we obtain a roundtrip translation and feed a string concatenation of the original sentence and the round-trip translation, separated by the symbol sequence '=>', to a sequence-to-sequence model. ## results the main results of our work are reported in the blue boxes in the table mark cases where the round-trip translation brings the performance of a smaller model up to or above that of a larger one. in these cases, one has the choice to use a larger model without input augmentation or a smaller one with input augmentation. the factors that would determine this choice are compute availability, access to cloud platforms, and speed requirements, among others. if one has limited gpu memory to work with, but has access to a high-quality translation cloud service, the choice of a smaller model with input augmentation may be more appropriate. ## conclusion the goal of this study was to measure the benefits of round-trip machine translation for the task of grammatical error correction. transferring knowledge from an mt model to a gec model through input augmentation proved effective for smaller models, sometimes bringing their performance up to that of their larger counterparts. in this work, we chose english as a pivot language due the abundance of mt work on this language. future work could explore alternative pivot languages, option-ally ones that are related to the language of the gec data, as this may results in higher lexical and syntactic consistency between inputs and roundtrip translations and thus better guidance for the correction of grammatical errors.