Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation (2022)

Abstract: Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels corresponding to examples drawn from different language pairs might still reside in distinct sub-spaces. In this paper, we introduce multilingual crossover encoder-decoder (mXEncDec) to fuse language pairs at an instance level. Our approach interpolates instances from different language pairs into joint ‘crossover examples’ in order to encourage sharing input and output spaces across languages. To ensure better fusion of examples in multilingual settings, we propose several techniques to improve example interpolation across dissimilar languages under heavy data imbalance. Experiments on a large-scale WMT multilingual dataset demonstrate that our approach significantly improves quality on English-to-Many, Many-to-English and zero-shot translation tasks (from +0.5 BLEU up to +5.5 BLEU points). Results on code-switching sets demonstrate the capability of our approach to improve model generalization to out-of-distribution multilingual examples. We also conduct qualitative and quantitative representation comparisons to analyze the advantages of our approach at the representation level.

## introduction multilingual modeling has been receiving increasing research attention over the past few years, arising from successful demonstrations of improved quality across a variety of tasks, languages and modalities @xcite @xcite . the success of these models is typically ascribed to vocabulary sharing, parameter tying and implicit pivoting through dominant languages like english @xcite . these conventional techniques are effective, but might not be exploiting the full potential of multilingual models to learn the underlying inductive bias: the learning signal from one language should benefit the quality of other languages @xcite . here we study two related issues that exist in the context of multilingual neural machine translation (nmt) training @xcite @xcite . first, nmt models @xcite are trained with maximum likelihood estimation which has a strong tendency to overfit and even memorize observed training examples, particularly posing challenges for low resource languages @xcite . second, training examples from distinct language pairs are separately fed into multilingual nmt models without any explicit instancelevel sharing (with the exception of multi-source nmt @xcite ); as a consequence, given large enough capacity, the models have the liberty to map representations of different languages into distinct subspaces, limiting the extent of cross-lingual transfer. in this work, we introduce multilingual crossover encoder-decoder (mxencdec) to address these issues following the recent work on xencdec @xcite and mixup @xcite @xcite . inspired by chromosomal crossovers @xcite , mxencdec fuses two multilingual training examples to generate crossover examples inheriting the combinations of traits of different language pairs, which is capable of explicitly capturing cross-lingual signals compared to the standard training which mechanically combines multiple language pairs. mx-encdec has the following advantages: thanks to crossover examples generated by fusing different language pairs, the multilingual nmt is encouraged to learn to transfer explicitly via more languages rather than implicitly via the predominant languages. 2. improving the model generalization and robustness. as vicinity examples around each example in the multilingual corpus (akin to vicinal risk minimization @xcite ), crossover examples produced by mx-encdec can enrich the support of the training distribution and lead to better generalization and robustness respectively on general and noisy inputs @xcite . 3. alleviating overfitting to low-resource languages. mxencdec can increase the diversity of low-resource languages by fusing lowresource examples with others, instead of the simple duplication in the standard training. in mxencdec, we randomly pick up two training examples drawn from the multilingual training corpus and first interpolate their source sentences where we have to prudently deal with language tags. then we leverage a mixture decoder to produce a virtual target sentence. to account for heavy data imbalance of each language pair, we propose a pairwise sampling strategy to adjust interpolation ratios between language pairs. we also propose to simplify the target interpolation to cope with noisy attention and fusions of dissimilar language pairs. different from xencdec fusing two heterogeneous tasks @xcite , we attempt to adapt it to deeply fuse different language pairs. experimental results on a large-scale wmt multilingual dataset show that mxencdec yields improvements of +1.13 and +0.47 bleu points averagely on xx-en and en-xx test sets over a vanilla multilingual transformer model. we also evaluate our approaches on zero-shot translations and obtain up to +5.53 bleu points over the baseline method, which corroborates the better transferabilty of multilingual models with our approaches. the more stable performance on noisy input text demonstrates the capability of our approach to improve the model robustness. to further explain the model behaviors at the representation level, qualitative and quantitative comparisons on representations manifest that our approach learns better multilingual representations, which indirectly explicates the bleu improvements. ## background multilingual neural machine translation. nmt @xcite optimizes the conditional probability p (y|x; θ) of translating a source-language sentence x into a target-language sentence y. the encoder reads the source sentence @xmath0, ..., x i as a sequence of word embeddings e(x). the decoder acts as a conditional language model over embeddings e(y) and the encoder outputs with a cross-attention mechanism @xcite . for clarity, we denote the input and output in the decoder as z and y, i.e., @xmath1, y 1 , • • • , y j-1 as a shifted copy of y, where s is a sentence start token. then the decoder generates y as p (y|x; θ) = j @xmath2, x; θ). the crossattention matrix is denoted as @xmath3. nmt optimizes the parameters θ by maximizing the likelihood of a parallel training set d: multilingual nmt extends nmt from the bilingual to the multilingual setting, in which it learns a one-to-many, many-to-one or many-to-many mapping from a set of languages to another set of languages @xcite . more specifically, the multilingual nmt model is learned over parallel corpora @xmath4: the crossover encoder combines embeddings of the two source sequences into a new sequence of embeddings: on the crossover decoder side, a mixture conditional language model is employed for the generation of the virtual target sentence. the input embedding e(z j ) and output label v(ỹ j ) for the decoder at the j-th position are calculated as: e(z j ) =e(y j-1 )t j-1 + e(y j-1 )(1 -t j-1 ), (4) ## mxencdec in this work, we aim to leverage xencdec to encourage multilingual nmt models to better exploit cross-lingual signals with crossover examples created by explicitly fusing different language pairs. we introduce its variant, called mxencdec as shown in figure 1 , in which the parent examples could belong to either the same or different language pairs. the subsequent subsections discuss how to address new challenges of mxencdec for multilingual nmt. language interpolation. as multilingual nmt involves a large number of language pairs, several techniques have been adopted to distinguish translation directions among them, such as prepending a language tag to source inputs @xcite or both source and target sentences @xcite , training language-specific embeddings for different languages @xcite , and so on @xcite . when following @xcite , it is natural to interpolate language-specific embeddings as we do for token embeddings. however, if we want to adopt a language tag in the first word of a source sentence to indicate the target language @xcite , we need to address how to interpolate them. as figure 1 shows, to make the sentence x still carry language-specific information from x and x , we conduct a soft combination over their language tags, that is: a similar equation can be obtained for using language embeddings. in addition, dispensing with attention can improve the parallel efficiency with 10% speed-up gain. hard target input interpolation. for multilingual nmt with multiple languages on the target side, i.e., one-to-many and many-to-many models, we need to carefully design combinations of target input word embeddings. as representations from the same language are usually close to each other, it can still augment the representation space by linearly interpolating target embeddings in eq. ( 4 ). but for dissimilar languages, in particular distantly related languages, the interpolation points between them are comparatively unreliable. to tackle this issue, we simply quantize t j to 1 if t @xmath5.5, otherwise t @xmath6. ( 4 ). a better solution should consider varying the interpolation ratio based on the language similarity or encourage interpolations of similar languages. we leave this for future exploration. pairwise sampling. the multilingual corpus is usually heavily imbalanced: most of its data distribution concentrates on high-resource language pairs @xcite . when interpolating high-resource and low-resource sentence pairs, we assume the fusion should be encouraged to be in favor of high-resource language pairs because the representation space supported by highresource sentences is relatively reliable and stable @xcite . this indicates a more frequent small p (e.g. @xmath7.5) to weigh highresource sentences over low-resource sentences if (x, y) ∈ d l i is a high-resource sentence and (x , y ) ∈ d l j is a low-resource sentence. to this end, we propose a pairwise sampling method to sample the source shuffle ratio p l i ,l j for interpolating language pair l i and l j : input: corpus m, temperature τ , ratio p. output: batch loss l x (θ). 1 function mxencdec (m, τ , p): 2 (x , y ) ← shuffle (x, y ) ∈ m along batch; to mitigate the data imbalance in the wmt multilingual corpus, we follow @xcite and adopt a temperature-based data sampling strategy to over-sample the low-resource languages where the temperature is set to 5. we apply sentencepiece @xcite to learn a vocabulary of 64k sub-words. we perform experiments in three settings: many-to-one, one-tomany and many-to-many translations. the 15 test language pairs are cast into three groups according to their data size: high (> 10m , 5 languages), low (< 1m , 7) and medium (> 1m & < 10m , 3). we report not only the average detokenized bleu scores for each group as calculated by the sacrebleu script @xcite but also winning ratio (wr) indicating the ratio of all the test sets on which our approach beats the baseline method. models and hyperparamters. following chen et al. (2018), we select the transformer big (6 layer, 1024 model dimension, 8192 hidden dimension) as the backbone model and implement them with the open-source lingvo @xcite . adafactor @xcite is adapted as our training optimizer, in which the learning rate is set to 3.0 and adjusted with 40k warm-up steps. training efficiency. if we adopt the simplified target interpolation, the loss computations for l m (θ) and l x (θ) in eq. ( 13 ) are totally independent. but we have to halve the batch size to load interpolation examples (l x (θ)) into memory. to make the ## related work multilingual nmt has made tremendous progress in recent years @xcite @xcite @xcite . recent research efforts to improve the generalization of multilingual models concentrate on enlarging the model capacity @xcite @xcite , incorporating hundreds of languages @xcite , pretraining multilingual models @xcite , and introducing additional regularization constraints @xcite @xcite . our work is related to the last three ones in that they try to enable models to better transfer across languages by introducing an alignment loss to learn an interlingua @xcite or imposing an agreement loss on translation equivalents @xcite . however, we propose to utilize mxencdec to directly combine language pairs for better exploitation of cross-lingual signals. another related research line is data mixing. since mixup @xcite was proposed in computer vision, we have observed great success in nlp @xcite @xcite . mx-encdec shares the commonality of combining example pairs as inspired by xencdec @xcite . to the best of our knowledge, we are the first to fuse different language pairs to improve cross-lingual generalization and robustness for multilingual nmt. ## conclusion we have presented mxencdec to fuse different language pairs at instance level for multilingual nmt, which enables the model to better exploit cross-lingual signals. experimental results on general, zero-shot and noisy test sets demonstrate that our approach can significantly improve the crosslingual generalization, zero-shot transfer and robustness of multilingual nmt models. representation analyses further confirms that our approach is capable of learning better multilingual representations, which coincides with improvements in bleu. we plan to investigate whether this approach can improve the model generalization in a broader scope like domain generalization. we find that mxencdec can easily achieve notable improvements for xx-en translations because they share an identical target language. however, there still exits huge headroom for en-xx translations. we plan to explore how to interpolate target languages more effectively, for example, possibly considering language similarity.