Complete Multilingual Neural Machine Translation (2020)

Abstract: Multilingual Neural Machine Translation (MNMT) models are commonly trained on a joint set of bilingual corpora which is acutely English-centric (i.e. English either as source or target language). While direct data between two languages that are non-English is explicitly available at times, its use is not common. In this paper, we first take a step back and look at the commonly used bilingual corpora (WMT), and resurface the existence and importance of implicit structure that existed in it: multi-way alignment across examples (the same sentence in more than two languages). We set out to study the use of multi-way aligned examples in order to enrich the original English-centric parallel corpora. We reintroduce this direct parallel data from multi-way aligned corpora between all source and target languages. By doing so, the English-centric graph expands into a complete graph, every language pair being connected. We call MNMT with such connectivity pattern complete Multilingual Neural Machine Translation (cMNMT) and demonstrate its utility and efficacy with a series of experiments and analysis. In combination with a novel training data sampling strategy that is conditioned on the target language only, cMNMT yields competitive translation quality for all language pairs. We further study the size effect of multi-way aligned data, its transfer learning capabilities and how it eases adding a new language in MNMT. Finally, we stress test cMNMT at scale and demonstrate that we can train a cMNMT model with up to 12,432 language pairs that provides competitive translation quality for all language pairs.

## introduction multilingual machine translation @xcite @xcite , which can serve multiple language pairs with a single model, has attracted much attention. in contrast to bilingual mt systems which can only serve one single language pair, multilingual models can serve o(n 2 ) language pairs (n being the number of languages in a multilingual model) @xcite . the amount of available training data can differ a lot across language pairs and the majority of available mt training data is english-centric (tiedemann, 2018; @xcite which in practice means that most non-english language pairs do not see a single training example when training multilingual models (see figure 1a ). as a consequence, the actual performance of language pairs that do not include english on the source or target side lags behind the ones with large amounts of training data. further, when increasing the number of languages, it gets (a) impractical to gather training data for each language pair and (b) challenging to find the right mix during training. which is why models tasked with direct translation between non-english pairs either resort to bridging (pivoting) through a pivot language @xcite , or make use of synthetic parallel data (via back-translation) @xcite or study the problem under zero-shot settings @xcite . in this study, we make use of the potential pre-existing multi-way property in the training corpora and generate as many direct training examples from pre-existing english-centric training data. if we can find training examples for each language pair in a multilingual mix, we call this model complete @xcite . cmnmt is then trained on all bilingual pairs between source and target languages by utilizing multi-way aligned training examples that consist of translations of the same sentence into multiple languages. we resurface multi-way aligned training examples by aligning training examples from different language pairs when either their source or target sides are identical (ie. pivoting through english, for @xmath0. to make use of this data, the model samples a source and target language from the set of multiway aligned corpus during training, which allows the model to see language pairs where originally no training data existed (missing connections in figure 1a ). as our experiments support, this method enables us to get access to training data for all tested language pairs (generating a complete graph (figure 1b )). we will show that it is possible to generate a complete graph for at least a 6-language wmt setup. some of the wmt training data is multi-way parallel by construction. nevertheless, we show that we also find many training examples where the source and target origin from different sources. we further show on our 112 languages internal dataset, that we can find sufficient training data for over 12,000 language pairs by only providing 111 english-centric training corpora. this result indicates that it is possible to generate direct training data for many language pairs without the need for crawling new training examples. our experiments suggest that before falling back to methods like zero-shot translation, you should investigate the structure of your pre-existing training data. to address the problem of finding the right mix of examples from different language pairs during training, we further introduce a hierarchical sampling strategy that is language-specific (as opposed to being language pair specific). in addition to fixing some chronic issues of mnmt (i.e. low quality for out of english translation @xcite @xcite ), the proposed sampling strategy efficiently ensures all source-target pairs are covered. experiments demonstrate that we can train a cm-nmt model on a 30-language-pair wmt setup that outperforms bilingual and multilingual baselines as well as bridging on all non-english language pairs. we further show that the performance of the english language pairs stay stable and do not suffer from the changes in both the training data and the new training data sampling strategy. furthermore, we share experiments at scale by demonstrating that we can train a cmnmt model that can serve 12,432 language pairs. our contribution is three-fold: ## a peek at multi-way aligned examples in bilingual corpora we choose six languages czech (cs), english (en), french (fr), german (de), spanish (es) and russian (ru) from the public wmt datasets. the selection of the languages was driven by the fact that the wmt 2013 evaluation campaign @xcite released a multi-way test set for these six languages. as training data, we used wmt 2013 for spanish, wmt 2014 for german, wmt 2015 for french, and wmt 2018 for czech and russian. we can construct non-english bilingual training examples by pairing the non-english sides of two training examples with identical english translations. some of the extracted non-english training examples are multi-way parallel by construction. the un corpus is a 6-way parallel corpus, and three of the languages (english, french and spanish) are in our 6-language mix. a portion of the europarl corpus is again multi-way aligned. nevertheless, a good amount of the extracted data is coming from different sources. ## complete multilingual nmt we call mnmt models that are trained for all possible source-target pairs as complete mnmt as all languages are connected via training data (also see figure 1 ). before going into details of how the missing pairs' data gathered, we recap mnmt first. multilingual nmt framework mnmt @xcite is an extension of bilingual nmt which uses a single model to translate between multiple languages. the model parameters are trained on a joint set of bilingual corpora from different language pairs. given the data imbalance across the different corpora, it is common to oversample the language pairs with less training data @xcite . for a given language pair p, let d(p) be the size of the available parallel corpus, the sample probability with a temperature t is defined as as a result, @xmath1, and @xmath2. in addition to being able to translate language pairs that the model was trained with, the model can also translate between language pairs never seen explicitly during training which is often referred as zero-shot translation @xcite . using multi-way aligned data in mnmt instead of only relying on bilingual corpora, bilingual examples from different language pairs with identical target sentences can be combined into a single multi-way aligned training example. an example is given in while we can extract direct training data for any source-target pair among the languages considered, the total number of language pairs increases quadratically. the vanilla language pair based sampling strategy in eq. ( 1 ) with adjustable temperature is capable of balancing low-high resource language pairs during training. however, we noticed a critical failure mode, which is further amplified in complete mnmt. the language-pair based sampling strategy (regardless of the temperature being used) over-represents english in english-centric models. notice half of the languages have english on the source side, with the other half on the target side. this over-representation yields a schedule of examples for the encoder (resp. for the decoder) to see english examples half of the time throughout the entire training process. as a result, trained models end up favouring english either on the source and/or target. although the implications on the encoder could be minimal, over-exposing english examples to the decoder curtail the learning signal when the target language is non-english. we hypothesise that this imbalance in the learning signal with respect to the target language is one of the roots of poor translation quality of multilingual models when translating out of english @xcite @xcite . to alleviate the over-representation of english with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule), ii) uniformly sample a source language. formally, for a given target language l, let d(l) be the size of the available training examples with target language l, the sample probability with a temperature t is defined as during training, the scheduler samples a batch of training examples based on the target language only, as opposed to source-target language pair specific sampling. after choosing a target language, for each multi-way aligned example, we randomly (uniformly) pick one of the translations as the source sentence. ## experiments we use a public transformer implementation with the transformer-big model size @xcite for all multilingual setups. all bilingual models use a vocabulary of 32,000 subwords, while all multilingual models use a vocabulary of 64,000 subword units. all multilingual models are trained for 500,000 updates using an average batch size of around 33,000 sentences (∼1 million tokens). all bilingual models are trained for 400,000 steps as they converged earlier using a batch size of around 8,000 sentences (∼260,000 tokens). due to the data imbalance across languages, we use a temperature-based data sampling strategy to oversample low-resource language pairs in standard mnmt models (equation 1 ) and low-resource target languages in cmnmt models (equation 2 ). we use a temperature of @xmath3. all multilingual models add a token at the beginning of the input sentence to specify the required target language. all bleu @xcite scores are calculated with sacrebleu @xcite ). 1 ## analysis and discussion to further understand the impact of multi-way aligned examples on nmt, we run a couple of additional experiments. ## conclusion in this work, we introduced complete multilingual neural machine translation (cmnmt) that exploits the multi-way alignment information in the underlying training data to improve translation quality for language pairs where training data is scared or not available. standard mnmt models are trained on a joint set of different training corpora for a variety of language pairs. cm-nmt combines the different corpora and constructs multi-way aligned training examples that consist of translations of the same sentence into multiple languages. in combination with a novel temperaturebased sampling approach that is conditioned on the target language only, we show that cmnmt is superior to the standard mnmt model and the even better-performing bridging approach. experimental results on a public wmt 30 language pairs dataset and an in-house 12,432 language pairs dataset demonstrated an average bleu increase of more than 10 bleu points for non-english language pairs. this approach leads to a single nmt model that can serve 12,432k language pairs with reasonable quality which also surpasses the translation quality of the bridging approach, which is nowadays used in most modern mt services.