Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020)

Abstract: We investigate the following question for machine translation (MT): can we develop a single universal MT model to serve as the common seed and obtain derivative and improved models on arbitrary language pairs? We propose mRASP, an approach to pre-train a universal multilingual neural machine translation model. Our key idea in mRASP is its novel technique of random aligned substitution, which brings words and phrases with similar meanings across multiple languages closer in the representation space. We pre-train a mRASP model on 32 language pairs jointly with only public datasets. The model is then fine-tuned on downstream language pairs to obtain specialized MT models. We carry out extensive experiments on 42 translation directions across a diverse settings, including low, medium, rich resource, and as well as transferring to exotic language pairs. Experimental results demonstrate that mRASP achieves significant performance improvement compared to directly training on those target pairs. It is the first time to verify that multiple lowresource language pairs can be utilized to improve rich resource MT. Surprisingly, mRASP is even able to improve the translation quality on exotic languages that never occur in the pretraining corpus. Code, data, and pre-trained models are available at https://github.com/linzehui/mRASP .

## introduction pre-trained language models such as bert have been highly effective for nlp tasks @xcite @xcite @xcite . pre-training and fine-tuning has been a successful paradigm. it is intriguing to discover a "bert" equivalent -a pre-trained model -for machine translation. in this paper, we study the following question: can we develop a single universal mt model and derive specialized models by fine-tuning on an arbitrary pair of languages? while pre-training techniques are working very well for nlp task, there are still several limitations for machine translation tasks. first, pre-trained language models such as bert are not easy to directly fine-tune unless using some sophisticated techniques @xcite . second, there is a discrepancy between existing pre-training objective and down-stream ones in mt. existing pre-training approaches such as mass @xcite and mbart @xcite rely on auto-encoding objectives to pre-train the models, which are different from translation. therefore, their fine-tuned mt models still do not achieve adequate improvement. third, existing mt pre-training approaches focus on using multilingual models to improve mt for low resource or medium resource languages. there has not been one pre-trained mt model that can improve for any pairs of languages, even for rich resource settings such as english-french. in this paper, we propose multilingual random aligned substitution pre-training (mrasp), a method to pre-train a mt model for many languages, which can be used as a common initial model to fine-tune on arbitrary language pairs. mrasp will then improve the translation performance, comparing to the mt models directly trained on downstream parallel data. in our method, we ensure that the pre-training on many languages and the down-stream fine-tuning share the same model architecture and training objective. therefore, this approach lead to large translation performance gain. consider that many languages differ lexically but are closely related at the semantic level, we start by training a large-scale multilingual nmt model across different translation directions, then fine-tuning the model in a specific direction. further, to close the representation gap across different languages and make full use of multilingual knowledge, we explicitly introduce additional loss based on random and aligned substitution of the words in the source and target sentences. substituted sentences are trained jointly with the same translation loss as the original multilingual parallel corpus. in this way, the model is able to bridge closer the representation space across different languages. we carry out extensive experiments in different scenarios, including translation tasks with different dataset scales, as well as zero-shot translation tasks. for extremely low resource (<100k), mrasp obtains gains up to +29 bleu points compared to directly trained models on the downstream language pairs. mrasp obtains consistent performance gains as the size of datasets increases. remarkably, even for rich resource (>10m, e.g. english-french), mrasp still achieves big improvements. surprisingly, even when mrasp is fine-tuned on two exotic languages that never occur in the pre-training corpus, the resulting mt model is still much better than the directly trained ones (+3.3 to +14.1 bleu). we finally conduct extensive analytic experiments to examine the contributing factors inside the mrasp method for the performance gains. we highlight our contributions as follows: ## methodology in this section, we introduce our proposed mrasp and the training details. ## experiments this section shows that mrasp obtains consistent performance gains in different scenarios. we also compare our method with existing pre-training methods and outperforms the baselines on @xmath0. the performance further boosts by combining back-translation @xcite technique. otherwise stated, for all experiments, we use the pre-trained model as initialization and fine-tune with the downstream target parallel corpus. ## analysis in this section, we conduct a set of analytical experiments to better understand what contributes to performance gains. three aspects are studied. first, we study whether the main contribution comes from pre-training or fine-tuning by comparing the performance of fine-tuning and no-fine-tuning. the results suggest that the performance mainly comes from pre-training, while fine-tuning further boosts the performance. second, we thoroughly analyze the difference between incorporating ras at the pre-training phase and pre-training without ras. the finding shows that incorporating alignment information helps bridge different languages and obtains additional gains. lastly, we study the effect of data volume in the fine-tuning phase. the effects with fine-tuning . in the pre-training phase, the model jointly learns from different language pairs. to verify whether the gains come from pre-training or fine-tuning, we directly measure the performance without any finetuning, which is, in essence, zero-shot translation task. we select datasets covering different scales. specifically, en-af (41k) from extremely low resource, en-ro (600k) from low resource, en-de (4.5m) from medium resource, and en-fr (40m) from rich resource are selected. as shown in note that we also report pre-trained model without ras (na-mrasp). for comparison, we do not apply fine-tuning on na-mrasp. mrasp consistently obtains better performance that na-mrasp, implying that injecting information at the pre-training phase do improve the performance. ## related works multilingual nmt aims at taking advantage of multilingual data to improve nmt for all languages involved, which has been extensively studied in a number of papers such as @xcite ; johnson et al. (2017); lu et al. (2018); rahimi et al. (2019) ; @xcite . the most related work to mrasp is @xcite , which performs extensive experiments in training massively multilingual nmt models. they show that multilingual many-to-many models are effective in low resource settings. inspired by their work, we believe that the translation quality of low-resource language pairs may improve when trained together with richresource ones. however, we are different in at least two aspects: a) our goal is to find the best practice of a single language pair with multilingual pretraining. multilingual nmt usually achieves inferior accuracy compared with its counterpart, which trains an individual model for each language pair when there are dozens of language pairs. b) different from multilingual nmt, mrasp can obtain improvements with rich-resource language pairs, such as english-frence. unsupervised pretraining has significantly improved the state of the art in natural language understanding from word embedding @xcite , pretrained contextualized representations @xcite @xcite and sequence to sequence pretraining @xcite . it is widely accepted that one of the most important factors for the success of unsupervised pre-training is the scale of the data. the most successful efforts, such as roberta, gpt, and bert, highlight the importance of scaling the amount of data. following their spirit, we show that with massively multilingual pre-training, more than 110 million sentence pairs, mrasp can significantly boost the performance of the downstream nmt tasks. on parallel, there is a bulk of work on unsupervised cross-lingual representation. most traditional studies show that cross-lingual representations can be used to improve the quality of monolingual representations. @xcite first introduces dictionaries to align word representations from different languages. a series of followup studies focus on aligning the word representation across languages @xcite @xcite . inspired by the success of bert, @xcite introduced xlm -masked language models trained on multiple languages, as a way to leverage parallel data and obtain impressive empirical results on the cross-lingual natural language inference (xnli) benchmark and unsupervised nmt @xcite @xcite . @xcite extended xlm with multi-task learning and proposed a universal language encoder. different from these works, a) mrasp is actually a multilingual sequence to sequence model which is more desirable for nmt pre-training; b) mrasp introduces alignment regularization to bridge the sentence representation across languages. ## conclusion in this paper, we propose a multilingual neural machine translation pre-training model (mrasp). to bridge the semantic space between different languages, we incorporate word alignment into the pre-training model. extensive experiments are conducted on different scenarios, including low/medium/rich resource and exotic corpus, demonstrating the efficacy of mrasp. we also conduct a set of analytical experiments to quantify the model, showing that the alignment information does bridge the gap between languages as well as boost the performance. we leave different alignment approaches to be explored in the future. in future work, we will pre-train on larger corpus to further boost the performance. english german (a) en-de w/o ras english german (b) en-de w/ ras figure 8: visualization of word embedding from mrasp w/o ras vs mrasp w/ ras for english-german