Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation (2020)

Abstract: Over the last few years two promising research directions in low-resource neural machine translation (NMT) have emerged. The first focuses on utilizing high-resource languages to improve the quality of low-resource languages via multilingual NMT. The second direction employs monolingual data with self-supervision to pre-train translation models, followed by fine-tuning on small amounts of supervised data. In this work, we join these two lines of research and demonstrate the efficacy of monolingual data with self-supervision in multilingual NMT. We offer three major results: (i) Using monolingual data significantly boosts the translation quality of low-resource languages in multilingual models. (ii) Self-supervision improves zero-shot translation quality in multilingual models. (iii) Leveraging monolingual data with self-supervision provides a viable path towards adding new languages to multilingual models, getting up to 33 BLEU on ro-en translation without any parallel data or back-translation.

## introduction recent work has demonstrated the efficacy of multilingual neural machine translation (multilingual nmt) on improving the translation quality of low-resource languages @xcite as well as zero-shot translation @xcite @xcite . the success of multilingual nmt on low-resource languages relies heavily on transfer learning from high-resource languages for which copious amounts of parallel data is easily accessible. however, existing multilingual nmt approaches often do not effectively utilize the abundance of monolingual data, especially in lowresource languages. on the other end of the spectrum, self-supervised learning methods, consuming only monolingual data, have achieved great success on transfer learning @xcite and unsupervised nmt @xcite without fully benefiting from the rich learning signals offered by the bilingual data of multiple languages. in this work, we propose to combine the beneficial effects of multilingual nmt with the selfsupervision from monolingual data. compared with multilingual models trained without any monolingual data, our approach shows consistent improvements in the translation quality of all languages, with greater than 10 bleu points improvements on certain low-resource languages. we further demonstrate improvements in zero-shot translation, where our method has almost on-par quality with pivoting-based approaches, without using any alignment or adversarial losses. the most interesting aspect of this work, however, is that we introduce a path towards effectively adding new unseen languages to a multilingual nmt model, showing strong translation quality on several language pairs by leveraging only monolingual data with self-supervised learning, without the need for any parallel data for the new languages. ## method we propose a co-training mechanism that combines supervised multilingual nmt with monolingual data and self-supervised learning. while several pre-training based approaches have been studied in the context of nmt (dai and le, 2015; conneau and lample, 2019; song et al., 2019), we proceed with masked sequence-to-sequence (mass) @xcite given its success on unsupervised and low-resource nmt, and adapt it to the multilingual setting. ## using monolingual data for multilingual nmt we evaluate the performance of the models using sacrebleu @xcite on standard wmt validation and test sets @xcite . the performance of our bilingual baselines for all 30 english-centric language pairs are reported in ## adding new languages to multilingual nmt inspired by the effectiveness of monolingual data in boosting low-resource language translation quality, we continue with a stress-test in which we completely remove the available parallel data from our multilingual model, one language at a time, in order to observe the unsupervised machine translation quality for the missing language. results of this set of experiments are detailed in ## related work our work builds on several recently proposed techniques for multilingual nmt and self-supervised representation learning. while massively multilingual models have obtained impressive quality improvements for low-resource languages as well as zero-shot scenarios @xcite , it has not yet been shown how these massively multilingual models could be extended to unseen languages, beyond the pipelined approaches @xcite . on the other hand, self-supervised learning approaches have excelled at down-stream cross-lingual transfer @xcite @xcite , but their success for unsupervised nmt @xcite currently lacks robustness when languages are distant or monolingual data domains are mismatched @xcite . we observe that these two lines of research can be quite complementary and can compensate for each other's deficiencies. ## conclusion and future directions we present a simple framework to combine multilingual nmt with self-supervised learning, in an effort to jointly exploit the learning signals from multilingual parallel data and monolingual data. we demonstrate that combining multilingual nmt with monolingual data and self-supervision (i) improves the translation quality for both low and highresource languages in a multilingual setting, (ii) leads to on-par zero-shot capability compared with competitive bridging-based approaches and (iii) is an effective way to extend multilingual models to new unseen languages. future work should explore techniques like iterative back-translation @xcite for further improvement and scaling to larger model capacities and more languages @xcite to maximize transfer across languages and across data sources.