Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (2020)

Abstract: Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.

## introduction with the great success of neural machine translation (nmt) on bilingual datasets @xcite @xcite , there is renewed interest in multilingual translation where a single nmt model is optimized for the translation of multiple language pairs @xcite @xcite . multilingual nmt eases model deployment and can encourage knowledge transfer among related language pairs @xcite , improve low-resource translation @xcite ), 1 we release our code at https://github. com/bzhanggo/zero . we release the opus-100 dataset at https://github.com/edinburghnlp/ opus-100-corpus . ## multilingual nmt we briefly review the multilingual approach @xcite and the transformer model @xcite , which are used as our baseline. @xcite rely on prepending tokens specifying the target language to each source sentence. in that way a single nmt model can be trained on the modified multilingual dataset and used to perform multilingual translation. given a source sentence @xmath0, x 2 , . . . , x |x| ), its target reference @xmath1, y 2 , . . . , y |y| ) and the target language token t foot_3 , multilingual nmt translates under the encoder-decoder framework @xcite : ## approach despite its success, multilingual nmt still suffers from 1) insufficient modeling capacity, where including more languages results in reduction in translation quality @xcite ; and 2) off-target translation, where models translate into a wrong target language on zero-shot directions @xcite . these drawbacks become severe in massively multilingual settings and we explore approaches to alleviate them. we hypothesize that the vanilla transformer has insufficient capacity and search for model-level strategies such as deepening transformer and devising languagespecific components. by contrast, we regard the lack of parallel data as the reason behind the offtarget issue. we resort to data-level strategy by creating, in online fashion, artificial parallel training data for each zero-shot language pair in order to encourage its translation. deep transformer one natural way to improve the capacity is to increase model depth. deeper neural models are often capable of inducing more generalizable ('abstract') representations and capturing more complex dependencies and have shown encouraging performance on bilingual translation @xcite @xcite . we adopt the depth-scaled initialization method @xcite to train a deep transformer for multilingual translation. language-aware layer normalization regardless of linguistic differences, layer normalization in multilingual nmt simply constrains all languages into one joint gaussian space, which makes learning more difficult. we propose to relax this restriction by conditioning the normalization on the given target language token t (laln for short) as follows: we apply this formula to all normalization layers, and leave the study of conditioning on source language information for the future. language-aware linear transformation different language pairs have different translation correspondences or word alignments @xcite . in addition to laln, we introduce a targetlanguage-aware linear transformation (lalt for short) between the encoder and the decoder to enhance the freedom of multilingual nmt in expressing flexible translation relationships. we adapt eq. random online backtranslation prior studies on backtranslation for zero-shot translation decode the whole training set for each zero-shot language pair @xcite , and scalability to massively multilingual translation is questionable -in our setting, the number of zero-shot translation directions is 9702. we address scalability by performing online backtranslation paired with randomly sampled intermediate languages. algorithm 1 shows the detail of robt, where for each training instance (x k , y k , t k ), we uniformly sample an intermediate language t k (t @xmath2, back-translate y k into ## opus-100 recent work has scaled up multilingual nmt from a handful of languages to tens or hundreds, with many-to-many systems being capable of translation in thousands of directions. following aharoni et al. (2019) , we created an english-centric dataset, meaning that all training pairs include english on either the source or target side. translation for any language pair that does not include english is zero-shot or must be pivoted through english. we created opus-100 by sampling data from the opus collection @xcite . opus-100 is at a similar scale to @xcite 's, with 100 languages (including english) on both sides and up to 1m training pairs for each language pair. we selected the languages based on the volume of parallel data available in opus. to evaluate zero-shot translation, we also sampled 2000 sentence pairs of test data for each of the 15 pairings of arabic, chinese, dutch, french, german, and russian. filtering was used to exclude sentences already in opus-100. ## experiments 6.1 setup we perform one-to-many (english-x) and manyto-many (english-x ∪ x-english) translation on opus-100 (|t | is 100). we apply byte pair encoding (bpe) @xcite to handle multilingual words with a joint vocabulary size of 64k. we randomly 7 for efficiency, we only use 200 sentences per language pair for validation in our multilingual experiments. apart from multilingual nmt, our baselines also involve bilingual nmt and pivot-based translation (only for zero-shot comparison). we select four typologically different target languages (german/de, chinese/zh, breton/br, telugu/te) with varied training data size for comparison to bilingual models as applying bilingual nmt to each language pair is resource-consuming. we report average bleu over these four languages as bleu 4 . we reuse the multilingual bpe vocabulary for bilingual nmt. we train all nmt models with the transformer base settings (512/2048, 8 heads) @xcite . we pair our approaches with the merged attention (matt) @xcite to reduce training time. other details about model settings are in the appendix. ## conclusion and future work this paper explores approaches to improve massively multilingual nmt, especially on zero-shot translation. we show that multilingual nmt suffers from weak capacity, and propose to enhance it by deepening the transformer and devising language-aware neural models. we find that multilingual nmt often generates off-target translations on zero-shot directions, and propose to correct it with a random online backtranslation algorithm. we empirically demonstrate the feasibility of backtranslation in massively multilingual settings to allow for massively zero-shot translation for the first time. we release opus-100, a multilingual dataset from opus including 100 languages with around 55m sentence pairs for future study. our experiments on this dataset show that the proposed approaches substantially increase translation performance, narrowing the performance gap with bilingual nmt models and pivot-based methods. in the future, we will develop lightweight alternatives to lalt to reduce the number of model parameters. we will also exploit novel strategies to break the upper bound of robt and obtain larger zero-shot improvements, such as generative modeling @xcite @xcite .