Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages (2021)

Abstract: Unsupervised translation has reached impressive performance on resource-rich language pairs such as English-French and English-German. However, early studies have shown that in more realistic settings involving low-resource, rare languages, unsupervised translation performs poorly, achieving less than 3.0 BLEU. In this work, we show that multilinguality is critical to making unsupervised systems practical for low-resource settings. In particular, we present a single model for 5 low-resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English directions, which leverages monolingual and auxiliary parallel data from other high-resource language pairs via a three-stage training scheme. We outperform all current state-of-the-art unsupervised baselines for these languages, achieving gains of up to 14.4 BLEU. Additionally, we outperform strong supervised baselines for various language pairs as well as match the performance of the current state-of-the-art supervised model for Nepali-English. We conduct a series of ablation studies to establish the robustness of our model under different degrees of data quality, as well as to analyze the factors which led to the superior performance of the proposed approach over traditional unsupervised models.

## introduction neural machine translation systems @xcite @xcite have demonstrated state-of-the-art results for a diverse set of language pairs when given large amounts of relevant parallel data. however, given the prohibitive nature of such a requirement for low-resource language pairs, there has been a growing interest in unsupervised machine translation @xcite and its neural counterpart, unsupervised neural machine translation (unmt) @xcite , which leverage only monolingual source and target corpora for learning. bilingual unsupervised systems @xcite @xcite have achieved surprisingly strong results on high-resource language pairs such as english-french and english-german. however, these works only evaluate on highresource language pairs with high-quality data, which are not realistic scenarios where unmt would be utilized. rather, the practical potential of unmt is in low-resource, rare languages that may not only lack parallel data but also have a shortage of high-quality monolingual data. for instance, romanian (a typical evaluation language for unsupervised methods) has 21 million lines of high-quality in-domain monolingual data provided by wmt. in contrast, for an actual low-resource language, gujarati, wmt only provides 500 thousand lines of monolingual data (in news domain) and an additional 3.7 million lines of monolingual data from common crawl @xcite . given the comparably sterile setups unmt has been studied in, recent works have questioned the usefulness of unmt when applied to more realistic low-resource settings. @xcite report bleu scores of less than 3.0 on low-resource pairs and @xcite also report dramatic degradation under domain shift. however, the negative results shown by the work above only study bilingual unsupervised systems and do not consider multilinguality, which has been well explored in supervised, zero-resource and zero-shot settings @xcite @xcite @xcite @xcite to improve performance for low-resource languages. the goal of this work is to study if multilinguality can help unmt be more robust in the low-resource, rare language setting. in our setup (figure 1 ), we have a single model for 5 target low-resource unsupervised directions (that are not associated with any parallel data): gujarati, kazakh, nepali, sinhala, and turkish. these languages are chosen to be studied for a variety of reasons (discussed in §3) and have been of particular challenge to unsupervised systems. in our approach, as shown in figure 1 , we also leverage auxiliary data from a set of higher resource languages: russian, chinese, hindi, arabic, tamil, and telugu. these higher resource languages not only possess significant amounts of monolingual data but also auxiliary parallel data with english that we leverage to improve the performance of the target unsupervised directions 1 . existing work on multilingual unsupervised translation @xcite @xcite , which also uses auxiliary parallel data, employs a two-stage training scheme consisting of pre-training with noisy reconstruction objectives and fine-tuning with on-thefly (iterative) back-translation and cross-translation terms ( §4). we show this leads to sub-optimal performance for low-resource pairs and propose an additional intermediate training stage in our approach. our key insight is that pre-training typically results in high xñen (to english) performance but poor enñx (from english) results, which makes finetuning unstable. thus, after pre-training, we propose an intermediate training stage that leverages offline back-translation @xcite to generate synthetic data from the xñen direction to boost enñx accuracy. our final results show that our approach outperforms a variety of supervised and unsupervised baselines, including the current state-of-the-art supervised model for the neñen language pair. additionally, we perform a series of experimental studies to analyze the factors that affect the performance of the proposed approach, as well as the performance in data-starved settings and settings where we only have access to noisy, multi-domain monolingual data. 2019a; @xcite . zero-shot translation concerns the case where direct (source, target) parallel data is lacking but there is parallel data via a common pivot language to both the source and the target. for example, in figure 1 , ruøzh and hiøte would be zero-shot directions. in contrast, a defining characteristic of the multilingual unmt setup is that the source and target are disconnected in the graph and one of the languages is not associated with any parallel data with english or otherwise. enøgu or enøkk are such example pairs as shown in figure 1 . recently @xcite ; liu et al. (2020) showed some initial results on multilingual unsupervised translation in the low-resource setting. they tune language-specific models and employ a standard two-stage training scheme @xcite , or in the case of @xcite directly fine-tuning on a related language pair (e.g. hiñen) and then test on the target xñen pair (e.g. guñen). in contrast our approach trains one model for all the language pairs targetted and employs a three stage training scheme that leverages synthetic parallel data via offline back-translation. offline backtranslation @xcite ) was originally used for unsupervised translation @xcite , especially with phrase-based systems. ## choice of languages the vast majority of works in unmt (multilingual or otherwise) have focused on traditionally highresource languages, such as french and german. while certain works simulate this setting by using only a smaller subset of the available monolingual data, such settings neglect common properties of true low-resource, rare languages: little-to-no lexical overlap with english and noisy data sources coming from multiple domains. given the multifaceted nature of what it means to be a low-resource language, we have chosen a set of languages with many of these characteristics. we give a detailed account of the available data in target unsupervised directions: we select turkish (tr), gujarati @xcite , and kazakh (kk) from wmt . the latter two possess much smaller amounts of data than most language pairs considered for unmt e.g. french or german. in order to vary the domain of our test sets, we additionally include nepali (ne) and sinhala (si) from the recently-introduced flores dataset @xcite , as the test sets for these languages are drawn from wikipedia instead of news. not only do these languages possess monolingual data in amounts comparable to the low-resource languages from wmt, the subset of in-domain monolingual data for both languages make up less than 5% of the available monolingual data of each language. auxiliary languages: to choose our auxiliary languages that contain both monolingual data and parallel data with english, we took into account linguistic diversity, size, and relatedness to the target directions. russian shares the same alphabet with kazakh, and hindi, telugu, and tamil are related to gujarai, nepali and sinhala. chinese, while not specifically related to any of the target language, is high resource and considerably different in structure from the other languages. ## background for a given language pair px, yq of languages x and y, we possess monolingual datasets d x and d y , consisting of unpaired sentences of each language. neural machine translation in supervised neural machine translation, we have access to a parallel dataset d xˆz consisting of translation pairs px, zq. we then train a model by utilizing the crossentropy objective: ## method for the rest of this work, we assume that we want to translate between english (en) and some lowresource languages which we denote by x. in our early experiments, we found that proceeding to the fine-tuning stage immediately after pre-training with mass provided sub-optimal results (see §7.2), so we introduced an intermediate stage which leverages synthetic data to improve performance. this yields a total of three stages, which we describe below. ## main experiment in this section, we describe the details of our main experiment. as indicated in figure 1 , we consider five languages (nepali, sinhala, gujarati, kazakh, turkish) as the target unsupervised language pairs with english. we leverage auxiliary parallel data from six higher-resource languages (chinese, russian, arabic, hindi, telugu, tamil) with english. the domains and counts for the datasets considered can be found in ## further analysis given the substantial quality gains delivered by our proposed method, we set out to investigate what design choices can improve the performance of unsupervised models. to ease the computational burden, we further filter the training data to remove any sample which are longer than 64 sen-tencepiece foot_4 pieces long and cut the batch size in half for the first two stages. additionally, we only do one additional round of training with synthetic data as opposed to the two rounds performed for the benchmark models. while these choices negatively impact performance, the resulting models still provide competitive results with our baselines and hence are more than sufficient for the purposes of experimental studies. ## conclusion in this work, we studied how multilinguality can make unsupervised translation viable for lowresource languages in a realistic setting. our results show that utilizing the auxiliary parallel data in combination with synthetic data through our threestage training procedure not only yields large gains over unsupervised baselines but also outperforms several modern supervised approaches.