An Extensive Exploration of Back-Translation in 60 Languages (2023)

Abstract: Back-translation is a data augmentation technique that has been shown to improve model quality through the creation of synthetic training bitext. Early studies showed the promise of the technique and follow on studies have produced additional refinements. We have undertaken a broad investigation using back-translation to train models from 60 languages into English; the majority of these languages are considered moderate- or low-resource languages. We observed consistent gains, though compared to prior work we saw conspicuous gains in quite a number of lower-resourced languages. We analyzed differences in translations between baseline and back-translation models, and observed many indications of improved translation quality. Translation of both rare and common terms is improved, and these improvements occur despite the less natural synthetic source-language text used in training.

## introduction back-translation was applied to statistical machine translation at least as far back as 2009 @xcite with modest gains being reported. @xcite applied back-translation in nmt and obtained gains of 2-3 bleu for english/german and about 4 bleu in turkish to english. this renewed interest in back-translation and it became a popular technique used in wmt evaluations, particularly in high-resource settings. research continued in back-translation, with a paper by @xcite who studied iterative back-translation, where the reverse model is itself improved through back-translation. in lowresource scenarios they observed gains of about 1.5 bleu, however, the marginal gain of repeated iterations is small. many studies conducted experiments where a high resource language pair was sampled to artificially create a "low" resource dataset, however, we are concerned that such simulations are not a good proxy due to dissimilar scripts, atypical subject matter, and noisy training data common in low-resource bitext. a few studies have looked look at bona fide low-resource language pairs. one example is @xcite who found 3+ bleu point gains in several languages, and even an 8 point gain in azerbaijani to english. other influential works in back-translation include: @xcite who investigated the optimal amount of monolingual data to use in highresource pairs; @xcite who examined filtering out lower-quality synthetic bitext pairs; @xcite who examined weighting synthetic exemplars differently than humanproduced bitext; and @xcite and @xcite who studied use of sampling. our goal in this study is to reexamine the use of back-translation through extensive experimentation in moderately and low-resourced languages. we believe that this is the largest study to date in terms of the number of languages for which back-translation effectiveness has been analyzed. we describe our experimental setup in section 2. in section 3 we compare back-translation to a baseline model for 60 source languages. an analysis of these results is provided in section 4. in section 5 we examine the amount of synthetic data to use in six languages. and in section 6 we report on experiments using repeated back-translation in 13 languages. ## methods in this section we describe model training and the evaluation datasets we use for evaluation. ## results scores for the baseline models (base) and for models trained using back-translation (bt) are shown in the top tier of languages experience gains of 1-2 bleu points (∼ 4% relative gain); the middle tier sees gains averaging about 5 bleu (23% relative gain); and, the least resourced languages see average gains of about 8 bleu (70% relative gain). languages such as burmese, gujarati, kannada, and khmer attain roughly double the score of their on both tests sets, in every instance, backtranslation conferred gains. there is a strong inverse relationship between the amount of training data used in the baseline model and the improvement in bleu score with back-translation. this is clear in figure 2 where the less-resourced language are plotted towards the left. it was not clear that models in the impoverished languages would improve given the questionable quality of their reverse models, yet large gains are indeed seen. we ran a bootstrap resampling test @xcite comparing bt with base: with the exception of serbian, all bleu improvements in bt are statistically significant (@xmath0.05). this expands the observation of @xcite which measured large bt gains for an earlier version of flores101 consisting of nepali and sinhala. to give context to our baseline models we also report performance using the 1.2 billion parameter m2m100 model released by facebook @xcite , which was trained on 7.5 billion sentence pairs. note that our bilingual models often outperform the multilingual m2m. in figure 3 we show examples of translations. consider the first example, about portuguese explorer vasco da gama. in the kazakh training data, the explorer's name never occurs, neither in kazakh nor english. but in the synthetic bitext, the name appears eight times in the monolingual english, and it is correctly back-translated in kazakh once, along with a couple of partially correct translations and errors. this is apparently enough to learn how to decode the name properly. ## analysis of results we now provide various analyses to better understand the results in section 3. specifically, we are interested to learn why and how back-translation (bt) improves upon the baseline (base). are the improvements in bt consistent across evaluation metrics? yes. the histograms in figure 4 summarize the translation quality in terms of bleu, chrf @xcite , and ter @xcite . the bleu plot corresponds to results in what kinds of words are translated correctly? figure 5 shows the precision/recall of out-ofvocabulary (oov) and high-frequency words, calculated using the compare-mt tool @xcite . we define oov words as words in the testset that do not occur in the training text of base, while frequent words are those with over 1,000 occurrences. for this analysis, mt hypotheses and references in english were processed with the moses tokenizer @xcite . we observed improvements in both precision and recall on both classes of words. figure 3 gave an example of improvement in oov translation, but in figure 5 we see that bt improves word precision and recall across the board. in fact, the high-frequency words note that the scores are generally bimodal. our explanation of the bimodality is based on the fact that 20-25% of the languages we studied have between 1m and 3m lines of training data; below 1m lines the scores tend to be below 20 bleu, and above 3m lines the scores tend to be over 30 bleu. there simply are not as many languages with quantities of training data in the intermediate range. when comparing bt against base, there is a rightward shift of improving bleu and chrf scores and a leftward shift of improving (i.e., decreasing) ter. we also conduct a word accuracy analysis that groups words by part-of-speech tags. the english reference and hypotheses are tagged with corenlp @xcite , then the respective precision and recall values are calculated. we average over the 60 testsets and report the resulting f 1 scores in how accurate are the reverse models? does the reverse model need to be highly accurate for back-translation to perform well? this is a question that is especially pertinent to low-resource conditions. we measure the accuracy of the reverse model that synthesized the 7 million lines of bt data. since the reference is in a non-english lan- the oov rate on the foreign side presents an additional explanation. we use the moses tokenizer and other language-specific tokenizers for this analysis. while the oov rate on the foreign side is higher (10%), there is still considerable reduction by bt (7.5%). the only way for oov rate to reduce on the foreign side is for the reverse model to generate via subword unit combinations new words that were previously not seen in the original bitext. finally, we train language models (4-gram kenlm (heafield, 2011)) on both sides of the bitext for base and bt, and measure the perplexity on flo-res101 validation set. here we use subwords as tokens to ameliorate the presence of oov words, which complicates perplexity calculations. figure 7 (bottom) shows that perplexity of a 4-gram trained base english text is approximately 110 on average, and it drops to 100 for a 4-gram trained on bt english text. surprisingly, the perplexity increases on the foreign side, growing from 75 to 85. for perplexity, these are minor differences, but we make some conjectures: (1) the small change in perplexity is likely due to the bt data being relatively broad domain; if the bt data were selected to be very similar to the test set, the perplexities would drop much more significantly. (2) the upward trend in perplexity for bt on the foreign side suggests that the synthesized foreign text might not be wholly natural (see appendix d). these texts do not improve monolingual perplexity, yet when paired as bitext they do improve mt accuracy. summary: bt improvements over base are measured on multiple metrics, and translation improves across the board on all word types. the reverse model does not need to be highly accurate and the bt bitext (if broad-domain) does not need to be specifically matched to the test domain for bt to work effectively. ## monolingual data size some have studied the effect of the amount of monolingual text used in creating synthetic bitext. a common heuristic is to use a small multiple of the human-produced training bitext, for example two or three times the amount. we wanted to assess this ourselves, and we did this in six languages that varied in the amount of base training data, from 300k lines of bitext up to 11 million. in section 3 we used 7 million sentences per language from the web-crawled english news portion of the leipzig corpus. for these new experiments we expand to 14 million sentences from the years 2005 to 2020, training six additional models per language, each using differing amounts of monolingual text. when back-translation is used, we choose the most recent data up to our desired limit. figure 8 plots bleu scores for six languages: bengali, farsi, hausa, kazakh, marathi, and panjabi. they vary in the amount of base bitext from about 300k lines (panjabi) up to 11 million lines (farsi). at the left is the no back-translation condition, and proceeding left to right, larger amounts of synthetic bitext are used. we make several observations from the plot. first, consistent with we conclude that using even relatively small amounts of data can be effective, and that the risk of using too much data is low. for example, with the use of 14 million lines of synthetic bitext, the panjabi model is using 40x more synthetic data than original human-produced bitext, and this still conveys large gains compared to using less data, and is nearly optimal compared to other choices. ## repeated back-translation earlier work in iterative back-translation @xcite showed small gains when first improving the reverse model, and then using that improved model to generate the final synthetic bitext. it makes sense that an improved synthetic bitext should have fewer errors and lead to an ultimately better model. we decided to investigate this method in thirteen languages, using just one attempt to improve the reverse model. this requires monolingual text in the source language to create synthetic data for the reverse model. for non-english text we used data from the oscar 22.01 corpus @xcite , which was filtered to remove possibly problematic text foot_1 and then performed sentence splitting using ersatz (wicks and post, 2021). this is a somewhat less controlled experiment, as the amount of monolingual text in oscar varies by the language. after the filtering mentioned above we used all of the remaining text. our results are shown in back-translation requires training two separate models, one after the other. however, with the extra step of improving the reverse model, we must train a third model. based on these results, the added expense of improving the reverse model is likely only worthwhile for languages with less than one million lines of human-produced bitext. two recent survey papers on low-resource translation @xcite mention the importance of data augmentation and back-translation in particular, though neither highlights the outsized impact of backtranslation compared to higher resourced settings. bt variants: although we use only the most simple bt technique, there are many advanced variants that may be interesting as future work. in addition to the papers on sampling, filtering, and weighting mentioned in the introduction, bt can be improved with meta-learning @xcite , transliteration @xcite , data selection @xcite , tagging @xcite , lexical/syntactic diversity @xcite . bt for multilingual models: we focus on bilingual models, but bt for multilingual models is an area of growing interest. @xcite observed consistent, yet small gains in multilingual models (seemingly less than 2 bleu, cf. their figs. 4 & 6 ). our experiments were exclusively bilingual and to-english, with larger gains in low-resource conditions, though direct comparison is not possible. in a follow-on study @xcite , meta develop a larger version of the flores data in 200 languages, and built a massively multilingual many-to-many model. as part of that wideranging work, they conducted experiments with back-translation (their sec. 6.4.1). their best results used statistical mt to generate the synthetic bitext. consistent with our results in translation to english, they found gains largest in "very low" resource languages (50.9 vs. 46.1 chrf++), but using multilingual mixture-of-experts models. ## conclusions by revisiting back-translation for an expansive list of 60 mid-and low-resource languages we have come to a better understanding of the landscape. we found that: