A Multilingual View of Unsupervised Machine Translation (2020)

Abstract: We present a probabilistic framework for multilingual neural machine translation that encompasses supervised and unsupervised setups, focusing on unsupervised translation. In addition to studying the vanilla case where there is only monolingual data available, we propose a novel setup where one language in the (source, target) pair is not associated with any parallel data, but there may exist auxiliary parallel data that contains the other. This auxiliary data can naturally be utilized in our probabilistic framework via a novel cross-translation loss term. Empirically, we show that our approach results in higher BLEU scores over state-of-the-art unsupervised models on the WMT’14 English-French, WMT’16 English-German, and WMT’16 English-Romanian datasets in most directions.

## introduction the popularity of neural machine translation systems @xcite @xcite has exploded in recent years. those systems have obtained state-of-the-art results for a wide collection of language pairs, but they often require large amounts of parallel (source, target) sentence pairs to train @xcite , making them impractical for scenarios with resourcepoor languages. as a result, there has been interest in unsupervised machine translation @xcite , and more recently unsupervised neural machine translation (unmt) @xcite , which uses only monolingual source and target corpora for learning. unsupervised nmt systems have achieved rapid progress recently @xcite @xcite , largely thanks to two key ideas: one-the-fly back-translation (i.e., minimizing round-trip translation inconsistency) @xcite done as part of the google ai residency. ## multilingual unsupervised machine translation in this section, we formulate our approach for m-unmt. we restrict ourselves to three languages, but the arguments naturally extend to an arbitrary number of languages. inspired by the recent style transfer literature @xcite and some approaches from multilingual supervised machine translation @xcite , we introduce a generative model of which the available data can be seen as partially-observed samples. we first investigate the strict unsupervised case, where only monolingual data is available. our framework naturally leads to an aggregate back-translation loss that generalizes previous work. we then incorporate the auxiliary corpus, introducing a novel crosstranslation term. to optimize our loss, we leverage the em algorithm, giving a rigorous justification for the stop-gradient operation that is usually applied in the unmt and style transfer literature @xcite @xcite . ## experiments we conduct experiments on the language triplets english-french-romanian with english-french parallel data, english-czech-german with english-czech parallel data and english-spanish-french with english-spanish parallel data, with the unsupervised directions chosen solely for the purposes of comparing with previous recent work @xcite @xcite . ## ablations we perform a series of ablation studies to determine which aspects of our formulation explain the improved performance. impact of the auxiliary data we first examine the value provided by the inclusion of the auxiliary data, focusing on the triplet english-french-romanian. to that end, we study four types of training configurations: (1) our implementation of mass @xcite , with only english and romanian data. the results show that leveraging the auxiliary data induces superior performance, even surpassing the supervised scores of @xcite . these gains can manifest in either pre-training or ## conclusion and future work in this work, we explored a simple multilingual approach to unmt and demonstrated that multilinguality and auxiliary parallel data offer quantifiable gains over strong baselines. we hope to explore massively multilingual unsupervised machine translation in the future.