Multilingual Denoising Pre-training for Neural Machine Translation (2020)

Abstract: This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART—a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective (Lewis et al., 2019). mBART is the first method for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, whereas previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine-tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task- specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show that it enables transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.1

## introduction despite its wide adoption for other nlp tasks @xcite @xcite @xcite , self-supervised pre-training is not yet common practice in machine translation (mt). existing approaches @xcite @xcite have been proposed either to partially pre-train the model or to only focus on english corpora. in this paper, we show that significant performance gains are possible by pre-training a complete autoregressive model with an objective that noises and reconstructs full texts across many languages. in this work, we present mbart-a multilingual sequence-to-sequence (seq2seq) denoising auto-encoder. mbart is trained by applying the bart @xcite to large-scale monolingual corpora across many languages. the input texts are noised by masking phrases and permuting sentences, and a single transformer @xcite model is learned to recover the texts. different from other pre-training approaches for mt @xcite , mbart pre-trains a complete autoregressive seq2seq model. mbart is trained once for all languages, providing a set of parameters that can be fine-tuned for any of the language pairs in both supervised and unsupervised settings, without any task-specific or language-specific modifications or initialization schemes. extensive experiments demonstrate that this simple approach works remarkably well. we first focus on existing mt benchmarks. for supervised sentence-level mt, mbart initialization leads to significant gains (up to 12 bleu points) across low/medium-resource pairs (<10m bi-text pairs), without sacrificing performance in high-resource settings. these results further improve with backtranslation (bt), setting a new state-of-the-art on wmt16 english-romanian and the flores test sets. for document-level mt, our documentlevel pre-training improves results by up to 5.5 bleu points. for the unsupervised case, we see consistent gains and produce the first nondegenerate results for less related language pairs (e.g., 9.5 bleu gain on nepali-english). previous pre-training schemes have only considered subsets of these applications, but we compare performance where possible and demonstrate that mbart consistently performs the best. we also show that mbart enables new types of transfer across language pairs. for example, fine-tuning on bi-text in one language pair (e.g., korean-english) creates a model that can translate from all other languages in the monolingual pretraining set (e.g., italian-english), with no further training. we also show that languages not in the pre-training corpora can benefit from mbart, strongly suggesting that the initialization is at least partially language universal. finally, we present a detailed analysis of which factors contribute the most to effective pre-training, including the number of languages and their overall similarity. ## multilingual denoising pre-training we use the common crawl (cc) corpus ( §2.1) to pre-train bart models ( §2.2). our experiments in the later sections involve fine-tuning a range of models pre-trained on different subsets ( §2.3). ## sentence-level machine translation this section shows that mbart pre-training provides consistent performance gains in low to medium resource sentence-level mt settings, including bi-text only and with back translation, and outperforms other existing pre-training schemes ( §3.2). we also present a detailed analysis to understand better which factors contribute the most to these gains ( §3.3), and show that pre-training can even improve performance for languages not present in the pre-training data ( §3.4). ## document-level machine translation we evaluate mbart on document-level machine translation tasks, where the goal is to translate segments of text that contain more than one sentence (up to an entire document). during pretraining, we use document fragments of up to 512 tokens, allowing the models to learn dependencies between sentences. we show that this pretraining significantly improves document-level translation. ## unsupervised machine translation in addition to supervised machine translation, we also evaluate our model on tasks where no bi-text is available for the target language pair. we define three types of unsupervised translation: ## conclusion we demonstrate that multilingual de-noising pretraining is able to significantly improve both supervised and unsupervised machine translation at both the sentence level and document level. we analyze when and how pre-training is most effective and can be combined with other approaches such as back-translation. our results also show the transfer learning ability of the learned representations from multilingual pre-training. in future work, we will scale-up the current pre-training to more languages, for example, an mbart100 model. the size of our model makes it expensive to deploy in production-future work will explore pre-training more efficient models.