Building Multilingual Machine Translation Systems That Serve Arbitrary XY Translations (2022)

Abstract: Multilingual Neural Machine Translation (MNMT) enables one system to translate sentences from multiple source languages to multiple target languages, greatly reducing deployment costs compared with conventional bilingual systems. The MNMT training benefit, however, is often limited to many-to-one directions. The model suffers from poor performance in one-to-many and many-to-many with zero-shot setup. To address this issue, this paper discusses how to practically build MNMT systems that serve arbitrary X-Y translation directions while leveraging multilinguality with a two-stage training strategy of pretraining and finetuning. Experimenting with the WMT’21 multilingual translation task, we demonstrate that our systems outperform the conventional baselines of direct bilingual models and pivot translation models for most directions, averagely giving +6.0 and +4.1 BLEU, without the need for architecture change or extra data collection. Moreover, we also examine our proposed approach in an extremely large-scale data setting to accommodate practical deployment scenarios.

## introduction multilingual neural machine translation (mnmt), which enables one system to serve translation for multiple directions, has attracted much attention in the machine translation area @xcite . because the multilingual capability hugely reduces the deployment cost at training and inference, mnmt has actively been employed as a machine translation system backbone in recent years @xcite . most mnmt systems are trained with multiple english-centric data for both directions (e.g., @xmath0, chinese} (en-x) and {french, * equal contributions. chinese} → english (x-en)). recent work @xcite @xcite pointed out that such mnmt systems severely face an off-target translation issue, especially in translations from a non-english language x to another non-english language y. meanwhile, freitag and @xcite have extended data resources with multi-way aligned data and reported that one complete many-to-many mnmt can be fully supervised, achieving competitive translation performance for all x-y directions. in our preliminary experiments, we observed that the complete manyto-many training is still as challenging as one-tomany training @xcite , since we have introduced more one-to-many translation tasks into the training. similarly reported in the many-to-many training with zero-shot setup @xcite , the complete mnmt model also suffers from capturing correlations in the data for all the x-y directions as one model training, due to highly imbalanced data. in this paper, we propose a two-stage training for complete mnmt systems that serve arbitrary x-y translations by 1) pretraining a complete multilingual many-to-many model and 2) finetuning the model to effectively transfer knowledge from pretraining to task-specific multilingual systems. considering that mnmt is a multi-task learner of translation tasks with "multiple languages", the complete multilingual model learns more diverse and general multilingual representations. we transfer the representations to a specifically targeted task via many-to-one multilingual finetuning, and eventually build multiple many-to-one mnmt models that cover all x-y directions. the experimental results on the wmt'21 multilingual translation task show that our systems have substantial improvement against conventional bilingual approaches and many-to-one multilingual approaches for most directions. besides, we discuss our proposal in the task 1 (top) and task 2 (bottom), with the respective average improvement of (12e6d, 12e6d+ft, 24e12d, 24e12d+ft) = (+3.6, +4.7, +5.0, +6.0) and (+2.0, +2.9, +3.2, +4.1) against the bilingual baseline ("bi") and the pivot translation baselines ("pivot"). "12e6d/24e12d" denote our two settings, with "+ft" suffix for finetuned systems. ## two-stage training for mnmt models to support all possible translations with |l| languages (including english), we first train a complete mnmt system on all available parallel data for |l| × (|l| -1) directions. we assume that there exist data of (|l| -1) english-centric language pairs and remaining (|l|-1)×(|l|-2) 2 non-english-centric language pairs, which lets the system learn multilingual representations across all |l| languages. usually, the volume of english-centric data is much greater than non-english-centric one. then, we transfer the multilingual representations to one target language l by finetuning the system on a subset of training data for many-to-l directions (i.e., multilingual many-to-one finetuning). this step leads the decoder towards the specifically targeted language l rather than multiple languages. as a result, we obtain |l| multilingual many-toone systems to serve all x-y translation directions. we experiment with our proposed approach in the following two settings: 1) wmt'21 large-scale multilingual translation data with 972m sentence pairs and 2) our in-house production-scale dataset with 4.1b sentence pairs. ## wmt'21 multilingual translation task we experiment with two small tasks of the wmt'21 large-scale multilingual translation task. data size (m) 321 172 141 123 85 63 20 19 18 5 4 - for "12e6d" and (24, 12) for "24e12d" , to learn diverse multilingual dataset. the model parameters are optimized by using radam @xcite with an initial learning rate of 0.025, and warm-up steps of 10k and 30k for the 12e6d and 24e12d model training, respectively. the systems are pretrained on 64 v100 gpus with a mini-batch size of 3072 tokens and gradient accumulation of 16. after the pretraining, the models are finetuned on a subset of x-l training data. we finetune the model parameters gently on 8 v100 gpus with the same mini-batch size, gradient accumulations, and optimizer with different learning rate scheduling of (init_lr, warm-up steps)=({1e-4, 1e-5, 1e-6}, 8k). the best checkpoints are selected based on development loss. the translations are obtained by a beam search decoding with a beam size of 4, unless otherwise stated. baselines for system comparison, we build three different baselines: 1) direct bilingual systems, 2) pivot translation systems via english (only applicable for non-english x-y evaluation) @xcite , and 3) many-to-one multilingual systems with the 12e6d architecture. the bilingual and pivot-based baselines employ the transformer base architecture. the embedding dimension is set to 256 for jv, ms, ta, and tl, because of the training data scarcity. for the x-y pivot translation, a source sentence in language x is translated to english with a beam size of 5 by the x-en model, then the best output is translated to the final target language y by the en-y model. results all results on the test sets are displayed in figure 1 and ## in-house extremely large-scale setting deploying a larger and larger model is not always feasible. we often have limitations in the computational resources at inference time, which leads to a trade-off problem between the performance and the decoding cost caused by the model architecture. in this section, we validate our proposed approach in an extremely large-scale data setting and also discuss how we can build lighter nmt models without the performance loss, while distilling the proposed mnmt systems @xcite . we briefly touch the following three topics of 1) multi-way multilingual data collection, 2) english-centric vs. multi-centric pretraining for x-y translations, and 3) a lighter nmt model that addresses the tradeoff issue between performance and latency. we report the experimental results in the extremely large-scale setting. ## conclusion this paper proposes a simple but effective twostage training strategy for mnmt systems that serve arbitrary x-y translations. to support translations across languages, we first pretrain a complete multilingual many-to-many model, then transfer the representations via finetuning the model in a many-to-one multilingual fashion. in the wmt'21 translation task, we experimentally showed that the proposed approach substantially improve translation accuracy for most x-y directions against the strong conventional baselines of bilingual systems, pivot translation systems, and many-to-one multilingual systems. we also examined the proposed approach in the extremely large-scale setting, while addressing the practical questions such as multiway parallel data collection, the usefulness of multilinguality during the pretraining and finetuning, and how to save the decoding cost, achieving the better x-y quality.