On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation (2021)

Abstract: Pre-training (PT) and back-translation (BT) are two simple and powerful methods to utilize monolingual data for improving the model performance of neural machine translation (NMT). This paper takes the first step to investigate the complementarity between PT and BT. We introduce two probing tasks for PT and BT respectively and find that PT mainly contributes to the encoder module while BT brings more benefits to the decoder. Experimental results show that PT and BT are nicely complementary to each other, establishing state-of-the-art performances on the WMT16 English-Romanian and English-Russian benchmarks. Through extensive analyses on sentence originality and word frequency, we also demonstrate that combining Tagged BT with PT is more helpful to their complementarity, leading to better translation quality. Source code is freely available at https://github.com/SunbowLiu/PTvsBT .

## introduction neural machine translation (nmt; @xcite @xcite models are data-hungry and their performances are highly dependent upon the quantity and quality of labeled data, which are expensive and scarce resources @xcite . this motivates the research line of exploiting unlabeled monolingual data for boosting the model performance of nmt. due to simplicity and effectiveness, pre-training (pt; @xcite and backtranslation (bt; @xcite are two widely-used techniques for nmt, by leveraging a large amount of monolingual data. while empirically successful, the understandings of pt and bt are still limited at best. several attempts have been made to better understand them at the data level, e.g. exploring different kinds of noises for the source data @xcite . however, there are few understandings at the model level that how pt and bt affect the internal module (e.g. encoder and decoder) of nmt models. as recent studies start to combine pt and bt for better model performance @xcite @xcite , there is a pressing need to broaden the understandings of them. to this end, we introduce two probing tasks to investigate the effects of pt and bt on the encoder and decoder modules, respectively. we find that pt mainly contributes to the encoder module while bt brings more benefits to the decoder module. this provides a good explanation for the performance improvement of simply combining pt and bt. motivated by this finding, we explore a better combination method by leveraging tagged bt @xcite . experiments conducted on the wmt16 english-romanian and english-russian benchmarks show that pt can nicely cowork with bt, leading to state-of-the-art model performances. extensive analyses show that the tagging mechanism is helpful for enhancing the complementarity between pt and bt by improving the translation of source-original sentences and low-frequency words. our main contributions are as follows: ## understanding pt and bt in this section, we aim to better understand the similarities and differences between pt and bt on improving model performance. we design two probing tasks to study the research question: which module of nmt do pt and bt respectively play a greater role in enhancing translation quality? ## improving pt and bt the answer of the research question in section 3 is: pt and bt respectively contribute more to the nmt encoder and decoder, demonstrating that they are orthogonal and complementary to each other. this finding motivates us to better combine these two individual techniques together for further improving nmt models. ## conclusion and future works this paper broadens the understandings of pretraining (pt) and back-translation (bt). we propose two probing tasks to investigate the impact of pt and bt on each nmt module and find that pt is more beneficial to the encoder while bt mainly improves the decoder. experimental results on the wmt16 english-romanian and english-russian benchmarks show that pt is nicely complementary to bt. we also demonstrate that tagged bt (i.e., adding tags to bt data) can further improve the complementarity of translating source-original sentences and low-frequency words. in the future, we would like to apply curriculum learning @xcite @xcite to better organize the learning of pt and bt. it is also worthwhile to explore other kinds of methods utilizing monolingual data (e.g., selftraining @xcite @xcite ) and validate the findings on practical nmt systems @xcite and more generation tasks @xcite .