Multi-task Learning for Multilingual Neural Machine Translation (2020)

Abstract: While monolingual data has been shown to be useful in improving bilingual neural machine translation (NMT), effectively and efficiently leveraging monolingual data for Multilingual NMT (MNMT) systems is a less explored area. In this work, we propose a multi-task learning (MTL) framework that jointly trains the model with the translation task on bitext data and two denoising tasks on the monolingual data. We conduct extensive empirical studies on MNMT systems with 10 language pairs from WMT datasets. We show that the proposed approach can effectively improve the translation quality for both high-resource and low-resource languages with large margin, achieving significantly better results than the individual bilingual models. We also demonstrate the efficacy of the proposed approach in the zero-shot setup for language pairs without bitext training data. Furthermore, we show the effectiveness of MTL over pre-training approaches for both NMT and cross-lingual transfer learning NLU tasks; the proposed approach outperforms massive scale models trained on single task.

## introduction multilingual neural machine translation (mnmt), which leverages a single nmt model to handle the translation of multiple languages, has drawn research attention in recent years @xcite @xcite @xcite . mnmt is appealing since it greatly reduces the cost of training and serving separate models for different language pairs @xcite . it has shown great potential in knowledge transfer among languages, improving the translation quality for low-resource and zero-shot language pairs @xcite @xcite . ## background neural machine translation nmt adopts the sequence-to-sequence framework, which consists of an encoder and a decoder network built upon deep neural networks @xcite @xcite . the input source sentence is mapped into context representations in a continuous representation space by the encoder, which are then fed into the decoder to generate the output sentence. given a language pair (x, y), the objective of the nmt model training is to maximize the conditional probability p (y|x; θ) of the target sentence given the source sentence. nmt heavily relies on high-quality and largescale bitext data. various strategies have been proposed to augment the limited bitext by leveraging the monolingual data. back translation @xcite utilizes the target-side monolingual data. self learning @xcite leverages the source-side monolingual data. dual learning paradigms utilize monolingual data in both source and target language @xcite @xcite . while these approaches can effectively improve the nmt performance, they have two limitations. first, they introduce additional cost in model training and translation generation, and therefore are less efficient when scaling to the multilingual setting. second, back translation requires a good baseline model with adequate bitext data to start from, which limits its efficiency on low-resource settings. multilingual nmt mnmt aims to train a single translation model that translates between multiple language pairs @xcite . previous works explored the model architecture design with different parameter sharing strategies, such as partial sharing with shared encoder @xcite , shared attention @xcite , task-specific attention @xcite , and full model sharing with language identifier @xcite @xcite . there are also extensive studies on representation sharing that shares lexical, syntactic, or sentence level representations across different languages @xcite @xcite . the models in these works rely on bitext for training, and the largely available monolingual data has not been effectively leveraged. self-supervised learning this work is motivated by the recent success of self-supervised learning for nlp applications @xcite @xcite @xcite . different denoising objectives have been designed to train the neural networks on large-scale unlabeled text. in contrast to previous work in pretraining with separated self-supervised pre-training and supervised finetuning stages, we focus on a multi-task setting to jointly train the mnmt model on both bitext and monolingual data. multi-task learning multi-task learning (mtl) @xcite , which trains the model on several related tasks to encourage representation sharing and improve generalization performance, has been successfully used in many different machine learning applications @xcite @xcite . in the context of nmt, mtl has been explored mainly to inject linguistic knowledge @xcite @xcite @xcite with tasks such as part-of-speech tagging, dependency parsing, semantic parsing, etc. in this work, we instead focus on auxiliary self-supervised learning tasks to leverage the monolingual data. ## conclusion in this work, we propose a multi-task learning framework that jointly trains the model with the translation task on bitext data, the masked language modeling task on the source-side monolingual data and the denoising auto-encoding task on the targetside monolingual data. we explore data and noising scheduling approaches and demonstrate their efficacy for the proposed approach. we show that the proposed mtl approach can effectively improve the performance of mnmt on both high-resource and low-resource languages with large margin, and can also significantly improve the translation quality for zero-shot language pairs without bitext training data. we showed that the proposed approach is more effective than pre-training followed by finetuning for nmt. furthermore, we showed the effectiveness of multitask learning for cross-lingual downstream tasks outperforming sota larger models trained on single task. for future work, we are interested in investigating the proposed approach in a scaled setting with more languages and a larger amount of monolingual data. scheduling the different tasks and different types of data would be an interesting problem. furthermore, we would also like to explore the most sample efficient strategy to add a new language to a trained mnmt system.