Breaking Down Multilingual Machine Translation (2022)

Abstract: While multilingual training is now an essential ingredient in machine translation (MT) systems, recent work has demonstrated that it has different effects in different multilingual settings, such as many-to-one, one-to-many, and many-to-many learning. These training settings expose the encoder and the decoder in a machine translation model with different data distributions. In this paper, we examine how different varieties of multilingual training contribute to learning these two components of the MT model. Specifically, we compare bilingual models with encoders and/or decoders initialized by multilingual training. We show that multilingual training is beneficial to encoders in general, while it only benefits decoders for low-resource languages (LRLs). We further find the important attention heads for each language pair and compare their correlations during inference. Our analysis sheds light on how multilingual translation models work and also enables us to propose methods to improve performance by training with highly related languages. Our many-to-one models for high-resource languages and one-to-many models for LRL outperform the best results reported by Aharoni et al. (2019).

## introduction multilingual training regimens @xcite @xcite are now a key element of natural language processing, especially for low-resource languages (lrls) @xcite . these algorithms are presumed to be helpful because they leverage syntactic or semantic similarities between languages, and transfer processing abilities across language boundaries. in general, english is used as a central language due to its data availability, and three different multilingual training settings are considered: (1) one-tomany: training a model with languages pairs from english to many other languages. (2) many-to-one: training a model with languages pairs from many languages to english (3) many-to-many: training a model with the union of the above two settings' data. ( 1 ) and ( 3 ) can be used for english to other (en-x) translation, while (2) and ( 3 ) can be used for other to english (x-en) translation. however, multilingual training has not proven equally helpful in every setting. @xcite showed that many-to-one training improves performance over bilingual baselines more than one-to-many does. in this paper, we consider this result from the point of view of the components of the mt model. in the many-to-one setting, the model's inputs are from different language distributions so the encoder can be considered a multidomain model, whereas the decoder is trained on a single distribution. in the one-to-many setting, it is the opposite: the encoder shares data, and the decoder is multi-domain. while there are recent studies analyzing multilingual translation models @xcite @xcite , in general, they do not (1) examine the impact of different multilingual training settings such as one-to-many and many-to-one, and (2) they do not examine the different components, such as the encoder and the decoder, separately. this motivates us to ask "how do various types of multilingual training interact with learning of the encoder and decoder?" to answer this question, we set up controlled experiments that decouple the contribution to the encoder and the decoder in various training settings. we first train multilingual models using many-to-one, one-to-many, or many-to-many training paradigms. we then compare training bilingual models with and without initializing the encoder or the decoder with parameters learned by multilingual training. we find that, for lrls, multilingual training is beneficial to both the encoder and the decoder. however, surprisingly, for high-resource languages (hrl), we found multilingual training only beneficial to the encoder but not to the decoder. to further analyze the result, we examine "to what degree are the learned parameters shared across languages?". we use the head importance estimation method proposed by @xcite as a tool to identify the important attention heads in the model, and measure the consistency between the heads sets that are important for different language pairs. the results suggest that the encoder does share parameters across different languages in all settings. on the other hand, the decoder can treat the representation from the encoder in a language-agnostic way for x-en translation, and less parameter sharing is observed for en-x translation. our analyses on parameter sharing also provide a possible explanation to @xcite 's observation that the representation from the encoder is target-language-dependent. our investigation of how multilingual training works leads us to a method for improving mt models. with the comprehensive experiments in multilingual settings, for translations in hrl (ar-en, de-en, he-en, it-en), we discover that fine-tuning multilingual model with target bilingual data outperforms the best results in @xcite by 2.99 to 4.63 bleu score. with the analysis of the parameter sharing in the decoder, we are able to identify related languages. fine-tuning jointly with the identified related languages boosts lowresource translation (en-az, en-be, en-go, en-sk) over the best results in @xcite by 1.66 to 4.44 bleu score. compared to @xcite , our method does not require linguist knowledge, and thus may be more useful for less-studied low-resource languages. in sum, our contributions are three-fold. first, our experiments can be used as a diagnostic tool for multilingual translation to investigate how an encoder and a decoder benefit from multilingual training. second, our results provide insights into how multilingual translation works. third, we improve the translation models based on the findings from our analysis, showing a promising path for future research on multilingual machine translation. ## experimental settings for multilingual training before stepping into our analysis, we first explain our experimental setup. following the setting in @xcite and @xcite , we use the publicly available ted talks dataset @xcite is used to train all our machine translation models. following @xcite , we break words into subwords with bpe jointly learned over all source languages using the sentencepiece toolkit. the vocabulary size is 32,000. we perform experiments with the transformer architecture @xcite using the hyperparameters same as in @xcite foot_0 . all models are implemented and trained using fairseq 0.10.0 @xcite . we trained multilingual translation models with 60 different languages on the ted talks dataset with the three settings described in section 1: one-to-many, manyto-one and many-to-many. for one-to-many and many-to-many settings, we add a special language token to the input of the encoder to indicate the target language. following @xcite , we evaluate our models with bleu score @xcite on the selected 8 languages. they are representative of different language families @xcite . the size of the training is shown in ## how multilingual training benefits each component previous studies have shown that the multilingual training results are generally stronger than the bilingual training @xcite . to understand how multilingual training benefits nmt, we analyze the effect of multilingual training on different components of an nmt model, specifically, the encoder and decoder. ## how multilingual parameters are shared in each component given the previous results, we are interested in exactly how parameters are shared among different language pairs. given that we are using the transformer architecture, for which multi-head attention is a fundamental component, we use the attention heads as a proxy to analyze how multilingual models work differently when translating between different languages. specifically, we analyze our models by identifying the attention heads that are important when translating a language pair. measuring the consistency between the sets of important attention heads for two language pairs gives us hints on the extent of parameter sharing. ## improving translation based on the degree of parameter sharing insights from the previous section provide us with a new way to choose languages for multilingual training. in previous work @xcite , choosing on languages with similar linguistic properties is a popular practice. however, @xcite found the effect is highly language-dependent. sometimes training with similar languages might be worse than training on a set of unrelated languages. here we otherwise propose an entirely model-driven way to find related languages to improve multilingual translation models. we explore choosing languages where parameters can be better shared. ## conclusion in this work, we have the following findings: 1) in section 3, we examine how multilingual training contributes to each of the components in a machine translation model. we discover that, while multilingual training is beneficial to the encoders, it is less beneficial to the decoders. 2) in section 4, our analysis of important attention heads provides insight into the behavior of multilingual components. results suggest that the encoder in the en-all model may generate target-language-specific representation, while the behavior of the decoder of the all-en model may be source-language-agnostic. in addition, in the all-all model, we observe indications of lower parameter sharing between x-en pairs and en-x pairs. 3) in section 5, we explore approaches to improve the model based on our findings. on en-x translation, we outperform the best results in @xcite . with our proposed analysis as diagnostic tools, future work may further improve the multilingual systems. biao zhang, ankur bapna, rico sennrich, and orhan firat. 2021. share or not? learning to schedule language-specific capacity for multilingual translation. in international conference on learning representations.