Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation (2022)

Abstract: This paper demonstrates that multilingual pretraining and multilingual fine-tuning are both critical for facilitating cross-lingual transfer in zero-shot translation, where the neural machine translation (NMT) model is tested on source languages unseen during supervised training. Following this idea, we present SixT+, a strong many-to-English NMT model that supports 100 source languages but is trained with a parallel dataset in only six source languages. SixT+ initializes the decoder embedding and the full encoder with XLM-R large and then trains the encoder and decoder layers with a simple two-stage training strategy. SixT+ achieves impressive performance on many-to-English translation. It significantly outperforms CRISS and m2m-100, two strong multilingual NMT systems, with an average gain of 7.2 and 5.0 BLEU respectively. Additionally, SixT+ offers a set of model parameters that can be further fine-tuned to other unsupervised tasks. We demonstrate that adding SixT+ initialization outperforms state-of-the-art explicitly designed unsupervised NMT models on Si<->En and Ne<->En by over 1.2 average BLEU. When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12.3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.

## introduction neural machine translation (nmt) systems @xcite @xcite have demonstrated superior performance with large amounts of parallel data. however, the performance of most existing nmt systems will degrade when the labeled data is limited @xcite . to address this problem, unsupervised nmt, in which no parallel corpora are available, is drawing increasing attention. some prior work @xcite @xcite use pivot-based methods for zero-shot translation between unseen language pairs. in this setting, both source and target languages have parallel data with a pivot language. however, these approaches are infeasible for rare languages where a parallel dataset of any kind is hard to collect. another line of work @xcite @xcite build unsupervised nmt through back-translation and further enhance its performance by cross-lingual transfer from auxiliary languages. these methods are usually complicated with multiple iterations of back-translation and a combination of various training objectives. moreover, their models can only support one or several pre-specified translation directions. recently, @xcite propose sixt, a transferabilityenhanced fine-tuning method that better adapts xlm-r @xcite for translating unseen source languages. sixt is trained once to support all languages involved in the xlm-r pretraining as the source language. however, they focus on exploring a proper fine-tuning approach and build sixt with the parallel dataset from one auxiliary language, which heavily limits the model's zero-shot translation performance. in this paper, we present sixt+, a strong manyto-english nmt model that can support as many as 100 source languages with parallel datasets from only six language pairs. sixt+ is trained by applying sixt to multilingual fine-tuning with largescale data. we first initialize the encoder and embeddings of sixt+ with xlm-r and then train it with a two-stage training method. at the first stage, we only train the decoder layers, while at the second stage, we disentangle the positional informa-tion of the encoder and jointly optimize all parameters except the embeddings. sixt+ improves over sixt by keeping the decoder embeddings frozen during the whole training process, which speeds up the model training while reducing the model size. sixt+ is trained once to support all source languages and can be further extended to many-tomany nmt that can support multiple target languages. it is not only a strong multilingual nmt model but can also be fine-tuned for other unsupervised tasks, including unsupervised nmt, zeroshot cross-lingual transfer for natural language understanding (nlu), and natural language generation (nlg) tasks. extensive experiments demonstrate that sixt+ works remarkably well. for translating to english, sixt+ significantly outperforms all baselines across 17 languages, including criss and m2m-100, two strong unsupervised and supervised multilingual nmt models trained with 1.8b and 7.5b sentence pairs. the many-to-many sixt+ gets better performance than m2m-100 in 6 out of 7 target languages on the flores101 testset. when serving as a pretrained model, sixt+ also performs impressively well. for unsupervised nmt of rare languages, sixt+ initialization achieves better unsupervised performance than various explicitly designed unsupervised nmt models with an average gain over 1.2 bleu. for zero-shot crosslingual transfer for nlu, it significantly outperforms xlm-r on sentence retrieval tasks, while maintaining the performance on most other tasks. on the zero-shot cross-lingual abstractive summarization task, sixt+ improves mbart-ft by 12.3 average rouge-l across 5 zero-shot directions. finally, we conduct detailed analyses to understand the key ingredients of sixt+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder. 1 ## sixt+ sixt+ aims at building a strong many-to-english nmt model, especially for the zero-shot directions. we argue that multilingual pretraining and multilingual fine-tuning are both critical for this goal. therefore, we initialize sixt+ with xlm-r large and fine-tune sixt+ on the multilingual parallel dataset with a simple two-stage training method. 1 the code and pretrained models are available at https: //github.com/ghchen18/acl22-sixtp . ## sixt+ as a pretrained model sixt+ learns language-agnostic encoder representation and performs impressively well on translating various source languages. in this part, we extend sixt+ to two cross-lingual nlg tasks where the direct labeled data is scarce, namely unsupervised nmt for low-resource languages and zero-shot cross-lingual abstractive summarization. ## conclusion in this paper, we introduce sixt+, a strong manyto-english nmt model that supports 100 source languages but is trained once with the parallel dataset from only six source languages. our model makes the most of cross-lingual transfer by initializing with xlm-r and conducting multilingual fine-tuning on the large-scale dataset with a simple yet effective two-stage training method. extensive experiments demonstrate that sixt+ outperforms all baselines on many-to-english translation. when serving as a pretrained model, adding sixt+ initialization achieves new state-of-the-art performance for unsupervised nmt of low-resource and significantly outperforms mbart and xlm-r on zero-shot cross-lingual summarization. type lang source # sent training set de-en europarl v7 1.9m training set de-en wmt19 41m training set es-en ccaligned 20m training set fi-en ccaligned 9.2m training set hi-en ccaligned 7.4m training set ru-en ccaligned 20m training set zh-en wmt18 22.6m valid set de-en newstest 16 2999 valid set es-en newstest 10 2489 valid set fi-en newstest 19 1996 valid set hi-en newsdev 14 520 valid set ru-en newstest 16 2998 valid set zh-en newstest 17 2001