Investigating Backtranslation in Neural Machine Translation (2018)

Abstract: A prerequisite for training corpus-based machine translation (MT) systems – either Statistical MT (SMT) or Neural MT (NMT) – is the availability of high-quality parallel data. This is arguably more important today than ever before, as NMT has been shown in many studies to outperform SMT, but mostly when large parallel corpora are available; in cases where data is limited, SMT can still outperform NMT. Recently researchers have shown that back-translating monolingual data can be used to create synthetic parallel corpora, which in turn can be used in combination with authentic parallel data to train a highquality NMT system. Given that large collections of new parallel text become available only quite rarely, backtranslation has become the norm when building state-of-the-art NMT systems, especially in resource-poor scenarios. However, we assert that there are many unknown factors regarding the actual effects of back-translated data on the translation capabilities of an NMT model. Accordingly, in this work we investigate how using back-translated data as a training corpus – both as a separate standalone dataset as well as combined with human-generated parallel data – affects the performance of an NMT model. We use incrementally larger amounts of back-translated data to train a range of NMT systems for German-to-English, and analyse the resulting translation performance.

## introduction neural machine translation (nmt) @xcite @xcite is a relatively new machine translation (mt) paradigm that has quickly become dominant in both academic and industry mt communities, achieving state-of-the-art results @xcite @xcite @xcite on a range of language pairs and domains. as a corpus-based paradigm, the translation quality strongly depends on the quality and quantity of the training data provided. in comparison to statistical machine translation (smt) @xcite , nmt typically requires more data to build a system with good translation performance @xcite . in many use-cases, however, the amount of good-quality parallel data available is insufficient to reach the translation standard required. in such cases, it has become the norm to resort to back-translating freely available monolingual data @xcite @xcite to create an additional synthetic parallel corpus @xcite for training an nmt model. in this paper, we assert that this scenario has become the default in nmt without proper consideration of the merits of the approach. for example, @xcite present an algorithm for filtering noisy content from web-scraped parallel corpora, in order to mitigate the "pollut[ion] [of the web] with increasing amounts of machine-translated content". they note that their algorithm "is capable of identifying machine-translated content in parallel corpora for a variety of language pairs, and that in some cases it can be very effective in improving the quality of an mt system ... thus challenging the conventional wisdom in natural language processing that 'more data is better data'". note too that @xcite demonstrates backtranslation (or 'round trip' translation) to be an untrusted means of mt evaluation. in the same vein, @xcite notes that in order to show that mt is error-prone, "sites like translation party ( http:// www.translationparty.com/ ) have been set up to demonstrate that continuous use of 'back translation' -that is, start with (say) an english sentence, translate it into (say) french, translate that output back into english, ad nauseum -ends up with a string that differs markedly from that which you started out with". surely, then, no-one would argue that building an mt system -whether it be smt or nmt -with solely synthetic data is a good idea; after all, the premise underpinning the paper by @xcite was that adding machine-translated data to high-quality human-translated training data harms performance. nonetheless, nmt developers have been seduced into using back-translated data as a means of necessity; there is simply not enough authentic human-translated parallel data available to obtain high-quality results in all scenarios where we would like to deploy nmt. somewhat surprisingly, despite the inherent problems noted above, adding back-translated data does help improve the quality of nmt output! in this paper we set out to systematically test from the ground up the merits of back-translated data. we investigate three scenarios: (i) nmt systems trained on 'perfect' human-translated (authentic) data; (ii) using only back-translated (synthetic) data for training nmt systems; and (iii) nmt systems trained on a combination of humantranslated and back-translated data. we systematically create multiple training corpora of increasing sizes, using training sets with authentic, synthetic and hybrid (authentic + synthetic) data. for the hybrid case we increment the backtranslated to human-generated data ratio and observe the quality of the resulting nmt systems. we aim to identify to what extent adding synthetic data improves (or harms) the translation capabilities of nmt systems. that is, we investigate whether backtranslation as a core technique in nmt has any limits; given that synthetic data is generated via another imperfect mt system, we hypothesise that nmt trained with 'imperfect' data will -at some point -undo any benefits from the 'perfect' (human-translated) data, and lead the nmt to degrade in performance. 1 in all our experiments, we exploit data that is widely used in the academic community for researching the quality of mt. the datasets that we use in our experiments all come from the translation task of the tenth workshop on machine translation in 2015 (wmt 2015 @xcite ). 2 to build our nmt systems we use opennmt-py (the pytorch port of opennmt @xcite ) with standard settings that allows for easy replicability of our experiments. the remainder of the paper is structured as follows: section 2 presents related work on using back-translated and other synthetic data in mt. section 3 explains how back-translated data affects the training and quality of an nmt system. our data is described in section 4, and our experiments are outlined in section 5. the results are summarised and analysed in section 6. we conclude in section 7 with final remarks and future work plans. ## related work recent studies have shown different approaches to exploiting monolingual data to improve nmt. gülc ¸ehre et al. @xcite present two approaches to integrate a language model trained on monolingual data into the decoder of an nmt system. similarly, @xcite focus on improving the decoder with monolingual data. while these studies show improved overall translation quality, they require changing the underlying neural network architecture. in contrast, backtranslation allows one to generate a parallel corpus that, consecutively, can be used for training in a standard nmt implementation as presented by @xcite . @xcite use 4.4m sentence pairs of authentic human-translated parallel data to train a baseline @xmath0.6m german and 4.2m english target-side sentences. these are then mixed with the initial data to create human + synthetic parallel corpora which are then used to train new models. due to the good results that were obtained, adding synthetic data has become a popular step in the nmt training pipeline @xcite @xcite . @xcite use back-translated data to improve mt for a low-resource language, namely belarusian (be). they transliterate a highresource language (russian, ru) into their lowresource language (be) and train a @xmath1, which is then used to translate monolingual be data into en. finally, an @xmath2. the work of @xcite presents an analysis of models trained only with synthetic data. they train nmt models with parallel corpora composed of: (i) synthetic data in the source-side only; (ii) synthetic data in the target-side only; and (iii) a mixture of parallel sentences of which either the source-side or the target-side is synthetic. note too that in contrast to the efforts of @xcite , backtranslation has been applied successfully in pbsmt. @xcite use back-translated data to optimize the translation model of a pbsmt system and show improvements in the overall translation quality for 8 language pairs. ## issues involved in creating back-translated parallel data intuitively, mt models built using synthetic data should not perform well. a text translated by a machine can contain errors, so a model trained on such data may learn and replicate these mistakes. while @xcite demonstrated that using back-translated data (in combination with human-translated data) during training can have a positive impact on the performance of the model, we hypothesize that the performance of the model will degrade if the synthetic data is overly dominant in the training set, i.e. the benefit of using high-quality authentic parallel data may be outweighed by the synthetic back-translated data. we investigate our hypothesis through a systematic analysis of nmt models trained on different-sized parallel datasets containing increasing amounts of back-translated data. we acknowledge the plethora of factors that may impact such an analysis, e.g. vocabulary size, learning optimizer, learning rate, total amount of training steps/minibatches, etc. however, with this work we aim to provide a solid experimental baseline nmt set-up that would facilitate the analysis of the impacts of adding synthetic data to the training corpus. furthermore, our analysis does not aim to compare the best possible systems, but rather nmt systems trained under the same conditions that would allow a fair comparison. in this regard, we train our systems with word-based dictionaries, rather than with dictionaries based on sub-word units e.g., using byte-pair encoding (bpe) @xcite , although the latter case generally leads to higher mt quality. given two models of the same size (one trained on authentic and one on synthetic data) the same words can be split into sub-words differently. as such, the quality differences could be due to the sub-word units, learned from the specific data rather than the differences in the authentic and synthetic data. our evaluation builds a clearer picture of the progressive effects of adding synthetic data to the training corpus of nmt engines. to the best of our knowledge, such an analysis has not been performed at the time of writing. furthermore, we compare nmt systems built on authentic-only data to systems built on synthetic-only data and put the two extremes to a test. we hypothesise that only synthetic data will not be enough to train an nmt system with good performance due to the errors mediated by the initial mt system used to generate that data. however, our results are more than a little surprising. we present detailed analysis of our empirical results in section 6. ## data for the scope of this work, we use the german-english parallel data of the wmt 2015 translation task @xcite . this corpus is shuffled, tokenized, truecased and cleaned (removing sentences of length over 126 words). in total, it contains 4.48m sentence pairs (225m words). in order to explore the effects of back-translated data, we use human-translated (authentic) and back-translated (synthetic) data in three possible configurations: all the models that we built are evaluated using the same test set. this test set is provided by wmt 2015 news translation task. it consists of 2169 sentences from the news domain. these sentences have also been tokenized and truecased. ## experimental set-up we train sequence-to-sequence nmt models @xcite based on recurrent neural networks with an attention mechanism @xcite . the nmt framework we use is opennmt @xcite and in particular its pytorch foot_2 port. our set-up follows the opennmt guidelines, foot_3 that indicate that the default training configuration is reasonable for training a german-to-english model on wmt 2015 data. we acknowledge the multitude of parameters and values that one can tweak in the set-up of an nmt system, leading to systems with significantly different performance. moreover, the choice of these parameters often depends on the training data. in our experiments, however, we have focused on a static nmt set-up, where the different parameters (e.g. the nmt learning optimizer, number of epochs, etc.) are common for all systems we train. the decision on our set-up is based on two factors: (i) by limiting the variability of parameters, we can more easily investigate the effects of back-translated data by directly comparing the translation quality of the resulting nmt systems; and (ii) while certain new architectures such as transformer @xcite or different settings might obtain even better results, our goal here is not to build the absolutely best possible systems, but rather use configurations that are representative of what is used in the field and allow easy replication. specifically, we use a 2-layer lstm @xcite with 500 hidden units, a vocabulary size of 50,002 for the source language and 50,004 for the target language. a model is trained for 13 epochs, using the stochastic gradient descent learning optimizer and a batch size of 64. any unknown words in the translation are replaced with the word in the source language that has the highest attention. we first trained a baseline @xmath3, 000, 000 parallel sentences of authentic data (base dataset) and a baseline @xmath4. the latter model is used for backtranslation to create synthetic datasets. we found that using 1m sentences to train the model was sufficient for 'good enough' translations. to determine this, we performed preliminary tests that involve human evaluation alongside automatic metrics (on a random sample of the outputs) with models trained on other data sizes. 5 when performing backtranslation, we also replace any unknown words with the word in english (the source language when performing the backtranslation) having the highest attention. we used this engine to then back-translate different portions of our original data set that we then used as parallel training data in two different scenarios: (i) by itself, i.e. synthetic data only, and (ii) in combination with the authentic data used to train the first engine, i.e. the hybrid models, as defined in section 4. to make our comparison fair, we defined two cases of authentic data. the first one starts with the first 1,000,000 sentences and grows incrementally (adding 500,000 parallel sentences each time) until it contains 3,500,000 sentences, i.e. ranging between the 1 st and the 3,500,000 th sentence. we denote these sets as auth 0+ . the hybr data sets are composed of the 1 st 1,000,000 authentic sentences, combined with back-translated data for each following subset of 500,000 sentences. in the second case, the authentic data sets start from the 1,000,000 th sentence. the first one contains 1,000,000 sentences; the next ones increment with 500,000 additional authentic sentences with the last one ranging between the 1,000,000 th to the 4,480,000 th sentence. these sets we refer to as auth 1+ . the synth data sets are simply the backtranslated data sets from the auth 1+ category. in this way we compare engines trained on exactly the same original data -auth 0+ to hybr and auth 1+ to synth -which in one case has been partially or fully back-translated. ## results in figures 2 and 1 we illustrate how the bleu and meteor scores of our models (trained on authentic, synthetic and hybrid data) change with increases in the training data. ## conclusion and future work in this work we studied the performance of nmt german-to-english models when incrementally larger amounts of back-translated (or synthetic) data are used for training. we analysed hybrid nmt models built by adding back-translated data to an initial set of human-translated (or authentic) data, and showed that while translation performance tends to improve when larger amounts of synthetic data are added, performance appears to tail off when the balance is tipped too far in favour of the synthetic data; in our experiments we see a drop in performance of 1.2% for the 3.5m hybrid model compared to the 3m hybrid one. we plan to extend these experiments further in our future work, in order to figure out whether there exists a genuine tipping point, i.e. a ratio between the amount of synthetic and authentic data where the model achieves optimal performance, and beyond which the more synthetic data is added, the worse the nmt quality becomes. we also built models using synthetic data alone. to our surprise, the performance is quite good; the synthetic-only baseline model achieved quality very close to that of the authentic-only engines. astonishingly, the synthetic-only engine trained with 1m sentences performs better as scored by meteor than the authentic-only engine trained on the same amount of data. we believe our findings have important repercussions for resource-poor scenarios, especially where some prior engine -not necessarily an nmt system -exists for the reverse language direction, as this can be used to create arbitrarily large amounts of back-translated data for bootstrapping an nmt engine for the other language direction. we will investigate this further in ongoing work. in other future work, we also want to explore the effect of adding artificial data to different language pairs and domains. we envisage the current research as the first contribution to an ongoing investigation of the true merits and limits of back-translation. it may well turn out that adding incrementally larger amounts of back-translated data is less harmful than we expect, but at least doing this from the ground up will hopefully result in a set of principles for nmt practitioners, rather than the rather haphazard state of affairs we see before us today.