On Synthetic Data for Back Translation (2022)

Abstract: Back translation (BT) is one of the most significant technologies in NMT research fields. Existing attempts on BT share a common characteristic: they employ either beam search or random sampling to generate synthetic data with a backward model but seldom work studies the role of synthetic data in the performance of BT. This motivates us to ask a fundamental question: what kind of synthetic data contributes to BT performance?Through both theoretical and empirical studies, we identify two key factors on synthetic data controlling the back-translation NMT performance, which are quality and importance. Furthermore, based on our findings, we propose a simple yet effective method to generate synthetic data to better trade off both factors so as to yield the better performance for BT. We run extensive experiments on WMT14 DE-EN, EN-DE, and RU-EN benchmark tasks. By employing our proposed method to generate synthetic data, our BT model significantly outperforms the standard BT baselines (i.e., beam and sampling based methods for data generation), which proves the effectiveness of our proposed methods.

## introduction since the birth of neural machine translation (nmt) @xcite back translation (bt) @xcite has quickly become one of the most significant technologies in natural language processing (nlp) research field. this is because 1) it provides a simple yet effective approach to advance the supervised nmt by leveraging monolingual data @xcite and it also serves as a key learning objective in unsupervised nmt @xcite ; 2) back translation even plays a significant role in other nlp re-search fields beyond translation such as paraphrasing @xcite and style transfer @xcite . back translation consists of two steps, namely synthetic corpus generation with a backward model and parameter optimization for the forward model. various contributions have been made on improving back translation, for instance, iterative back translation @xcite , tagged back translation @xcite , confidence weighting @xcite , data diversification @xcite . although these efforts differ in some aspects, all of them share a common characteristic: they employ a default way to generate synthetic data in the first step of bt which is either beam search or random sampling with a backward model. seldom work studies the consequences of synthetic corpus to back translation and hence it is unclear how synthetic data influences the final performance of bt. the early study empirically suggests the quality of the synthetic corpus is vital for bt performance @xcite . however, recent studies illustrate better test performance can be achieved by low quality synthetic corpus @xcite . this contradictory observation indicates the quality of synthetic data is not the only element that affects the bt performance. hence, this fact naturally raises a fundamental question: what kind of synthetic data contributes to back translation performance? in this paper, we attempt to take a step forward toward the above fundamental question. to this end, we start from a critical objective in semi-supervised learning, which is defined by the marginal distribution of a target language. then we derive an approximate lower bound of the objective function, which is closely related to the objective of back translation. corresponding to this lower bound, we theoretically find two related elements for maximizing such a lower bound: quality of syn-thetic bilingual data and importance weight of its source. since both elements are mutually exclusive to some extent, it may induce contradictory observation if one judges the bt performance according to a single element. in addition, such a theoretical explanation is supported by our empirical experiments. furthermore, based on our findings, we propose a new heuristic approach to generate synthetic data whose both elements are better balanced so as to yield improvements over both sampling and beam search based methods. extensive experiments on three wmt14 tasks show that our bt consistently outperforms the standard sampling and beam search based baselines by a significant margin. our contributions are three folds: ## revisiting back translation nmt builds a probabilistic model p(y|x; θ) with neural networks parameterized by θ, which is used to translate a sentence x in source language x to a sentence y in target language y. the standard wisdom to train the model is to minimize the following objective function over a given bilingual corpus @xmath0, y i )}: recently @xcite propose a remarkable method called back translation (bt) to improve nmt by using a monolingual corpus m in target language y besides b and back translation becomes one of the most successful techniques in nmt @xcite . at a high level, back translation can be considered as a semi-supervised method because it leverages both labeled and unlabeled data. suppose p(x|y; π) is the backward translation model whose parameter π is optimized over b, the key idea of back translation can be summarized as the following two steps: to make bt more efficient, the standard configuration is widely adopted: each sentence y is required to generate a single source x and both two steps are performed for a single pass. we follow this standard in this paper for generality but our idea in this paper is straightforward to apply to other configurations such as @xcite @xcite . in the first step, there are two main strategies to generate the synthetic corpus, i.e., deterministically decoding and randomly sampling with p(x|y; π). the first strategy aims to search the best candidate as follows, @xmath1; π) the above optimization is achieved by the beam search decoding, which can be regarded as a degenerated shortest path problem with respect to the log p(x|y; π) with limited routing attempts. the alternative strategy is random sampling: it randomly samples a token with respect to the distribution estimated by a back-translation model at each decoding step. such a process can be modelled by, research question prior work points out @xcite ) that the synthetic corpus with high quality is beneficial to the final performance of back translation. however, the recent studies @xcite find that nmt models with unsatisfactory bleu score corpus, for instance, the corpus generated by sampling based strategy, also establish the state-of-the-art (sota) achievement among back-translation nmt models. this contradictory fact indicates that the quality of synthetic corpus is not the sole element for back translation. this motivates us to study a fundamental question for back translation: what kind of synthetic corpus is beneficial to back translation? ## understanding synthetic data by two factors to answer the fundamental question presented in the previous section, we first start from the marginal likelihood objective defined on the target language y, and then we theoretically explain two factors (i.e., quality and importance) that are highly related to the training objective of back translation. finally, we empirically explain why synthetic corpus with low quality may lead to better performance than synthetic corpus with high quality by measuring both factors. ## improving synthetic data for bt as shown in the previous section, both importance and quality of synthetic corpus are beneficial to the overall testing performance of back translation. it is a natural idea to promote both factors when generating synthetic corpus such that running bt on such corpus leads to better testing performance. however, this is difficult because both factors are mutually exclusive as discussed in section 3. in this section, we instead propose two methods (namely data manipulation and gamma score) to trade off both factors in the hope to yield better bt performance. ## related work this section describes prior arts in back-translation for nmt, data augmentation, and semi-supervised machine translation. back-translation @xcite firstly proposed back-translation, then @xcite ; @xcite apply back translation to solve the domain adaptation problems in phrase-based nmt systems. sennrich et al. (2016a) further extend the back translation for training nmt models integrally. for understanding the back-translation synthetic corpus, @xcite use a copy of target as a pseudo source, and find that nmt model performance can still be improved under the low resource settings. caswell et al. (2019 2020 ) diversify the training data by multiple forward and backward models translations and combine them with the original datasets. data augmentation for nmt nmt researchers are the pioneers of data augmentation studies since back-translation is a natural type of data augmentation method. @xcite @xcite . to balance the token frequency in nmt corpus, @xcite create new sentences contain low-frequency words. however, as observed by @xcite , the improvement across different translation tasks is not consistent, and they invent switchout data augmentation policy. recht et al. (2018, 2019); werpachowski et al. (2019) also observe such an inconsistency of variance between training corpus and testing set as well as in the generation tasks. recently, @xcite try to understand data augmentation from input sensitivity and prediction margin, thus obtaining relatively low variance in generation. semi-supervised machine translation however, as high quality bitext is always limited and costly to collect, @xcite study methods for effectively leveraging monolingual data in nmt systems. @xcite develop a duallearning mechanism, under such a learning objective, a nmt system is able to automatically learn from unlabeled data, thus improving nmt performance iteratively. based on iterative learning, @xcite investigates how to learn nmt systems when only large monolingual corpora can be used in each language. for supervision of models, @xcite employ the target language model hidden states into nmt decoder to further improve performance. @xcite show that back-translation improves translation quality of both naturally occurring text and translationese according to professional human translators. for supervision of learning corpus, @xcite study both the source-side and target-side monolingual data for nmt. ## conclusion in this work, we answer a fundamental question about synthetic data for back translation. we theoretically and empirically show two key factors namely quality and importance weight of synthetic data play an important role in back translation, and then we propose a new method to generate synthetic data which better balances both factors so as to boost the back-translation performance. for future work, we think it would be of significance to apply our synthetic data generation method to other bt methods or even to more broad nlp tasks such as paraphrasing and style transfer.