Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders (2021)

Abstract: Previous work mainly focuses on improving cross-lingual transfer for NLU tasks with a multilingual pretrained encoder (MPE), or improving the performance on supervised machine translation with BERT. However, it is under-explored that whether the MPE can help to facilitate the cross-lingual transferability of NMT model. In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In this task, the NMT model is trained with parallel dataset of only one language pair and an off-the-shelf MPE, then it is directly tested on zero-shot language pairs. We propose SixT, a simple yet effective model for this task. SixT leverages the MPE with a two-stage training schedule and gets further improvement with a position disentangled encoder and a capacity-enhanced decoder. Using this method, SixT significantly outperforms mBART, a pretrained multilingual encoder-decoder model explicitly designed for NMT, with an average improvement of 7.1 BLEU on zero-shot any-to-English test sets across 14 source languages. Furthermore, with much less training computation cost and training data, our model achieves better performance on 15 any-to-English test sets than CRISS and m2m-100, two strong multilingual NMT baselines.

## introduction multilingual pretrained encoders (mpe) such as mbert @xcite , xlm (conneau and lample, 2019), and xlm-r @xcite have shown remarkably strong results on zero-shot cross-lingual transfer mainly for natural language understanding (nlu) tasks, including named entity recognition (ner), question answering (qa) and natural language inference (nli). these methods jointly train a transformer @xcite encoder to perform encoder body decoder body encoder embed decoder embed (2) training in the second stage de en multilingual pretrained model zero-shot translate ru en zh en hi en fi en encoder body decoder body encoder embed decoder embed (1) training in the first stage de en multilingual pretrained encoder zero-shot translate masked language modeling task in multiple languages. the pretrained model is then fine-tuned on a downstream nlu task using labeled data in a single language and evaluated on the same task in other languages. with this pretraining and finetuning approach, the mpe is able to generalize to other languages that even do not have labeled data. given that mpe has achieved great success in crosslingual nlu tasks, a question worthy of research is how to perform zero-shot cross-lingual transfer in the nmt task by leveraging the mpe. some work @xcite @xcite explores approaches to improve nmt performance by incorporating monolingual pretrained transformer encoder such as bert @xcite . however, simply replacing the monolingual pretrained encoder in previous studies with mpe does not work well for cross-lingual transfer of nmt (see baselines in in this paper, we focus on a zero-shot crosslingual(x) nmt transfer task (zext, see figure 1 ), which aims at translating multiple unseen languages by leveraging an mpe. different from unsupervised or multilingual nmt, only an mpe and parallel dataset of one language pair such as german-english are available in this task. the trained model is directly tested on many-to-one test sets in a zero-shot manner. we propose a simple cross-lingual(x) transfer nmt model (sixt) which can directly translates languages unseen during supervised training. we initialize the encoder and decoder embeddings of sixt with the xlm-r and propose a two-stage training schedule that trades off between supervised performance and transferability. at the first stage, we only train the decoder layers, while at the second stage, all model parameters are jointly optimized except the encoder embedding. we further improve the model by introducing a position disentangled encoder and a capacity-enhanced decoder. the position disentangled encoder enhances cross-lingual transferability by removing residual connection in one of the encoder layers and making the encoder outputs more language-agnostic. the capacity-enhanced decoder leverages a bigger decoder than vanilla transformer base model to fully utilize the labelled dataset. although trained with only one language pair, the sixt model alleviates the effect of 'catastrophic forgetting' @xcite and can be transferred to unseen languages. sixt significantly outperforms mbart with an average improvement of 7.1 bleu on zeroshot any-to-english translation across 14 source languages. furthermore, with much less training computation cost and training data, the sixt model gets better performance on 15 any-to-english test sets than criss and m2m-100, two strong multilingual nmt baselines. foot_0 ## problem statement the zero-shot cross-lingual nmt transfer task (zext) explores approaches to enhance the crosslingual transferability of nmt model. given an mpe and parallel dataset of a language pair l s -tol t , where l s and l t are supported by the mpe, we aim to train an nmt model that can be transferred to multiple unseen language pairs l i z -to-l t , where l i @xmath0. the learned nmt model is directly tested between the unseen language pairs l i z -to-l t in a zero-shot manner. different from multilingual nmt @xcite , unsupervised nmt @xcite or zero-resource nmt through pivoting @xcite , neither the parallel nor monolingual data in the language l i z is directly accessible in the zext task. the model has to rely on the offthe-shelf mpe to translate from language l i z . the challenge to this task is how to leverage an mpe for machine translation while preserving its crosslingual transferability. in this paper, we utilize xlm-r, which is jointly trained on 100 languages, as the off-the-shelf mpe. the zext task calls for approaches to efficiently build a many-to-one nmt model that can translate from 100 languages supported by xlm-r with parallel dataset of only one language pair. the trained model could be useful for translating resource-poor languages. it can further extend to scenarios where datasets of more language pairs are available. in addition, while currently the cross-lingual transferability of different mpes is mainly evaluated on cross-lingual nlu tasks, the zext task provides a new perspective for the evaluation, which can hopefully facilitate the research on mpes. ## analysis comparison with multilingual nmt in this part, we compare sixt with mbart @xcite , criss @xcite and m2m-100 @xcite on any-to-english test sets. mbart is a strong pretrained multilingual encoderdecoder based transformer explicitly designed for nmt. we follow their setting and directly fine-tune all model parameters on wmt19 de-en training set. criss and m2m-100 are the state-of-the-art unsupervised and supervised multilingual nmt models, respectively. the criss model is initialized with the mbart model and iteratively finetuned on 1.8 billion sentences covering 90 language pairs. m2m-100 is trained with 7.5 billion parallel sentences across 2200 translation directions. the results of criss and m2m-100 are listed as reference, because criss and m2m-100 are manyto-many nmt models whose performance may degrade due to the competitions among different target languages @xcite , while sixt is a many-to-one nmt model. to compare with these models, we train a manyto-one sixt large model with wmt19 german-english training data, which only consists of 41 million sentences pairs. it only requires a pretrained xlm-r large model and do not contain any data in other languages. we remove the residual connection after the self-attention sublayer of the 23-th (penultimate) encoder layer. the dataset and model configuration details are in from the results in language transfer v.s. language distance in this part, we explore the relationship between the cross-lingual transfer performance and the language distance. we train the sixt models on different supervised language pairs including de-en, es-en, fi-en, hi-en and zh-en, and then directly apply them to all test sets, as seen in train set # sents german romance uralic indo-aryan east asian avg. de nl es ro it fi lv et hi ne si gu zh ja ko ## conclusion in this paper, we focus on the zero-shot crosslingual nmt transfer (zext) task which aims at leveraging an mpe for machine translation while preserving its ability of cross-lingual transfer. in this task, only a multilingual pretrained encoder such as xlm-r and one parallel dataset such as german-english are available. we propose sixt for this task, which enables zero-shot cross-lingual transfer for nmt by making full use of the labelled data and enhancing the transferability of xlm-r. extensive experiments demonstrate the effectiveness of sixt. in particular, sixt outperforms mbart, a pretrained encoder-decoder-based model explicitly designed for nmt. it also gets better performance than criss and m2m-100, two strong multilingual nmt models, on 15 any-to-english test sets with less training data and training computation cost. model (418m parameters). the training hyper-parameters of sixt large model are the same with that in section 4.1.