Dynamic Data Selection and Weighting for Iterative Back-Translation (2020)

Abstract: Back-translation has proven to be an effective method to utilize monolingual data in neural machine translation (NMT), and iteratively conducting back-translation can further improve the model performance. Selecting which monolingual data to back-translate is crucial, as we require that the resulting synthetic data are of high quality and reflect the target domain. To achieve these two goals, data selection and weighting strategies have been proposed, with a common practice being to select samples close to the target domain but also dissimilar to the average general-domain text. In this paper, we provide insights into this commonly used approach and generalize it to a dynamic curriculum learning strategy, which is applied to iterative back-translation models. In addition, we propose weighting strategies based on both the current quality of the sentence and its improvement over the previous iteration. We evaluate our models on domain adaptation, low-resource, and high-resource MT settings and on two language pairs. Experimental results demonstrate that our methods achieve improvements of up to 1.8 BLEU points over competitive baselines.

## introduction back-translation @xcite ) is an effective strategy for improving the performance of neural machine translation (nmt) using monolingual data, delivering impressive gains over already competitive nmt models @xcite . the strategy is simple: given monolingual data in the target language, one can use a translation model in the opposite of the desired translation direction to back-translate the monolingual data, effectively synthesizing a parallel dataset, which is in turn used to train the final translation model. further improvements can be obtained by iteratively repeating this process @xcite in both directions. however, not all monolingual data are equally important. an envisioned downstream application is very often characterized by a unique data distribution. in such cases of domain shift, back-translating target domain data can be an effective strategy @xcite for obtaining a better in-domain translation model. one common strategy is to select samples that are both (1) close to the target distribution and (2) dissimilar to the average generaldomain text @xcite . however, as depicted in figure 1 , this method is not ideal because the second objective could bias towards the selection of sentences far from the center of the target distribution, potentially leading to selecting a non-representative set of sentences. even if we could select all in-domain monolin-gual data, the back-translation model has not been trained on in-domain parallel data and thus the back-translated data will be of poor quality. as we demonstrate in the experiments, the quality of the back-translated data can have a large influence on the final model performance. to achieve the two goals of both selecting targetdomain data and back-translating them with high quality, in this paper, we propose a method to combine dynamic data selection with weighting strategies for iterative back-translation. specifically, the dynamic data selection selects subsets of sentences from a monolingual corpus at each training epoch, gradually transitioning from selecting general-domain data to choosing target-domain sentences. the gradual transition ensures that the back-translation model of each iteration can adequately translate the selected sentences, as they are close to the distribution of its current training data. we also assign weights to the back-translated data that reflect their quality, which further reduces the effect of potential noise due to low quality translations. the proposed data selection and weighting strategies are complementary to each other, as the former focuses on domain information while the latter emphasizes the quality of sentences. we investigate the performance of our methods in domain adaptation, low-resource and highresource mt settings and on german-english and lithuanian-english datasets. our strategies demonstrate improvements of up to 1.8 bleu points over a competitive iterative back-translation baseline and up to 1.2 bleu points over the best static data selection strategies. in addition, our analysis reveals that the selected samples can represent the target distribution well and that the weighting strategies are effective in noisy settings. ## background: back-translation back-translation @xcite has proven to be an effective way of utilizing monolingual data for machine translation. given a parallel training corpus d f e , we first train a target-tosource machine translation model m ef . then, we use the pre-trained model m ef to translate a target language monolingual corpus d e to the source language and obtain a synthetic parallel corpus (d f , d e ). last, we concatenate back-translated data (d f , d e ) with the original parallel corpus d f e to train a source-to-target model m f e . algorithm 1 iterative back-translation input: monolingual corpora d f and d e output: translation models m f e and m ef while m f e and m ef have not converged do for all batches (b f train m ef with (b e , b f ) end for end while searchers to investigate and extend the method @xcite . @xcite propose to use iterative back-translation and achieve improvements over previous state-of-theart models. as shown in algorithm 1, at each training step, a batch of monolingual sentences is sampled from one language and back-translated to the other language. the back-translated data is utilized to train the model in the other direction. the process is repeated in both directions. ## methods in our setting, we are given two mt models m f e and m ef pretrained on parallel data d f e , and both source and target monolingual corpora d f and d e . the goal is to select and weight samples from the two monolingual corpora for backtranslation, in order to best improve the performance of the two translation models. ## experiments on domain adaptation we first conduct experiments in the domain adaptation setting, where we adapt models from a general domain to a specific domain. ## experiments on low-resource and high-resource scenarios next, we conduct experiments in both low-and high-resource scenarios over two language pairs: lithuanian-english and german-english. ## related work back-translation @xcite ) has proven to be effective and several extensions of it have been proposed @xcite @xcite , among which iterative back-translation methods @xcite @xcite have demonstrated strong empirical performance. for domain adaptation, moore and lewis (2010) and @xcite use language model cross entropy differences and tf-idf to select data that are similar to in-domain text respectively. @xcite propose dynamic data selection strategies for machine translation models, and @xcite extend the idea to curriculum strategies. as for filtering noisy sentences, junczys-dowmunt (2018) propose to utilize the agreement between forward and backward translation models and @xcite propose uncertainty-based confidence estimation to improve back-translation. @xcite compose dynamic domain-data selection with dynamic clean-data selection. our methods generalize previous data selection strategies and our primary focus is to improve iterative back-translation, but our work could be extended to also include training-time dynamic data selection approaches such as the technique of @xcite . ## conclusion in this paper, we provide a novel insight into a widely-used data selection method @xcite and generalize it to a curriculum strategy for iterative back-translation. we also propose data weighting methods to down-weight examples of poor quality. extensive experiments are performed to evaluate the performance of our methods; analyses reveal the selected samples can represent the target domain well and our weighting strategies benefit noisy settings the most.