Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation (2022)

Abstract: In this paper, we present a substantial step in better understanding the SOTA sequence-to-sequence (Seq2Seq) pretraining for neural machine translation (NMT). We focus on studying the impact of the jointly pretrained decoder, which is the main difference between Seq2Seq pretraining and previous encoder-based pretraining approaches for NMT. By carefully designing experiments on three language pairs, we find that Seq2Seq pretraining is a double-edged sword: On one hand, it helps NMT models to produce more diverse translations and reduce adequacy-related translation errors. On the other hand, the discrepancies between Seq2Seq pretraining and NMT finetuning limit the translation quality (i.e., domain discrepancy) and induce the over-estimation issue (i.e., objective discrepancy). Based on these observations, we further propose simple and effective strategies, named in-domain pretraining and input adaptation to remedy the domain and objective discrepancies, respectively. Experimental results on several language pairs show that our approach can consistently improve both translation performance and model robustness upon Seq2Seq pretraining.

## introduction there has been a wealth of research over the past several years on self-supervised pre-training for natural language processing tasks @xcite @xcite , which aims at transferring the knowledge of large-scale unlabeled data to downstream tasks with labeled data. despite its success in other understanding and generation tasks, self-supervised pretraining is not a common practice in machine translation (mt). one possible reason is the architecture discrepancy between pretraining model ⇤ work was mainly done when wenxuan wang and yongchang hao were interning at tencent ai lab. to remedy the architecture gap, several researchers propose sequence-to-sequence (seq2seq) pretraining models for machine translation, e.g., mass @xcite and bart @xcite . recently, @xcite extend bart by training on large-scale multilingual language data (i.e., mbart), leading to significant improvement on translation performance across various language pairs. while previous pretraining approaches for nmt generally focus only on transformer encoder @xcite , mbart pretrains a complete autoregressive seq2seq model by recovering the input sentences that are noised by masking phrases. one research question naturally arises: how much does the jointly pretrained decoder matter? in this work, we present a substantial step in better understanding the sota seq2seq pretraining model. we take a fine-grained look at the impact of the jointly pretrained decoder by carefully designing experiments, which are conducted on several wmt and iwslt benchmarks across language pairs and data scales using the released mbart-25 model @xcite . by carefully examining the translation outputs, we find that ( § 2.2): although jointly pretraining decoder consistently improves translation performance, we also identify several side effects due to the discrepancies between pretraining and finetuning ( §2.3): to remedy the above discrepancies, we propose simple and effective strategies, named in-domain pretraining and input adaptation in finetuning ( §3). in in-domain pretraining, we propose to reduce the domain shift by continuing the pretraining of mbart on in-domain monolingual data, which is more similar in data distribution with the downstream translation tasks. for input adaptation, we add noises to the source sentence of bilingual data, and combine the noisy data with the clean bilingual data for finetuning. we expect the perturbed inputs to better transfer the knowledge from pretrained model to the finetuned model. experimental results on the benchmark datasets show that indomain pretraining improves the translation performance significantly and input adaptation enhances the robustness of nmt models. combining the two approaches gives us the final solution to a well-performing nmt system. extensive analyses show that our approach can narrow the domain discrepancy, particularly improving the translation of low-frequency words. besides, our approach can alleviate the over-estimation issue and mitigate the beam search degradation problem of nmt models. ## understanding seq2seq pretraining in this section, we conduct experiments and analyses to gain a better understanding of current seq2seq pretraining for nmt. we first present the translation performance of the pretrained components ( §2.2), and then show the discrepancy between pretraining and finetuning ( §2.3). ## related work pretraining for nmt. previous pretraining approaches for nmt generally focus on how to effectively integrate pretrained bert @xcite or gpt @xcite to nmt models. for example, @xcite propose a concerted training framework, and @xcite propose a dynamic fusion mechanism and a distillation paradigm to acquire knowledge from bert and gpt. in this work, we aim to provide a better understanding of how seq2seq pretraining model works for nmt, and propose a simple and effective approach to improve model performance based on these observations. intermediate pretraining. our in-domain pretraining approach is related to recent successes on intermediate pretraining and intermediate task selection in nlu tasks. for example, @xcite investigate the influence of masking policies in intermediate pretraining. @xcite explore to select tasks for intermediate pretraining. closely related to our work, @xcite propose to continue the pretraining of roberta @xcite on task-specific data. inspired by these findings, we employ in-domain pretraining to narrow the domain gap between general seq2seq pretraining and nmt training. we also show the necessity of target-side monolingual data on in-domain pretraining (see appendix a.3), which has not been studied in previous works of in-domain pretraining. ## conclusion in this paper we provide a better understanding of seq2seq pretraining for nmt by showing both the benefits and side effects. we propose simple and effective approaches to remedy the side effects by bridging the gaps between seq2seq pretraining and nmt finetuning, which further improves translation performance and model robustness. future directions include validating our findings on more seq2seq pretraining models and language pairs.