When Does Unsupervised Machine Translation Work? (2020)

Abstract: Despite the reported success of unsupervised machine translation (MT), the field has yet to examine the conditions under which the methods succeed and fail. We conduct an extensive empirical evaluation using dissimilar language pairs, dissimilar domains, and diverse datasets. We find that performance rapidly deteriorates when source and target corpora are from different domains, and that stochasticity during embedding training can dramatically affect downstream results. We additionally find that unsupervised MT performance declines when source and target languages use different scripts, and observe very poor performance on authentic low-resource language pairs. We advocate for extensive empirical evaluation of unsupervised MT systems to highlight failure points and encourage continued research on the most promising paradigms. We release our preprocessed dataset to encourage evaluations that stress-test systems under multiple data conditions.

## introduction machine translation (mt) has progressed rapidly since the advent of neural machine translation (nmt) @xcite @xcite and is better than ever for languages for which ample high-quality bitext exists. conversely, mt for lowresource languages remains a great challenge due to a dearth of parallel training corpora and poor quality bitext from esoteric domains. to address this, several authors have proposed unsupervised mt techniques, which rely only on monolingual text for training (e.g., @xcite @xcite @xcite @xcite . recent unsupervised mt results appear promising, but they primarily report results for the highresource languages for which traditional mt already works well. the limits of these methods are so far under-explored. for unsupervised mt to be a viable path for low-resource machine translation, the field must determine (1) if it works outside highly-controlled environments, and (2) how to effectively evaluate newly-proposed training paradigms to pursue those which are promising for real-world low-resource scenarios. unsupervised mt methods must work (1) on different scripts and between dissimilar languages, (2) with imperfect domain alignment between source and target corpora, (3) with a domain mismatch between training data and the test set, and (4) on the low-quality data of real low-resource languages. these factors reflect the real-life challenges of lowresource translation. our main contribution is an extensive analysis of unsupervised mt with regards to factors (1)-(3) above. 1 we find that (a) translation performance rapidly deteriorates when source and target corpora are from different domains, (b) stochasticity during word embedding training can dramatically affect downstream bilingual lexicon induction (bli) and translation performance, and (c) like in the bilingual lexicon induction literature, unsupervised mt performance declines when source and target languages are dissimilar. while (4) is not the focus of this paper, we do observe very low performance on an authentic low-resource language pair, corroborating previous studies @xcite . finally, as there are no standard evaluation protocols to ensure that unsupervised mt systems are robust to the types of data anomalies ubiquitous in low-resource translation settings, we advocate for extensive empirical evaluation of unsupervised mt systems to highlight failure points and encourage continued research on the most promising paradigms. we first discuss related work in section 2, followed by a detailed overview of the unsupervised mt architecture in section 3. in section 4, we discuss our research questions, followed by our evaluation methodology and datasets in sections 5 and 6. section 7 presents our findings, and section 8 discusses the results. we conclude in section 9. ## background: unsupervised mt our experiments employ the models of @xcite as representative of state-ofthe-art for the class of unsupervised mt methods that bootstrap from cross-lingual word embeddings. recent work such as lample et al. (2018b) is based on similar concepts. for our purposes, unsupervised mt follows liu et al. (2020)'s definition (1) from section 2, where no bitext exists. another approach to unsupervised mt involves pretraining a bilingual or multilingual model on monolingual text on a general task before finetuning on translation. such methods include crosslingual language model pretraining @xcite , masked sequence-to-sequence pretraining @xcite , and multilingual denoising pretraining @xcite , and have shown promise. for instance, @xcite record the first good results on the low-resource sinhala-english and nepali-english pairs. while pretraining and multilingual methods are not the subject of this work, they warrant future evaluation. figure 1 depicts the basic training process. it is the publicly-available smt setup of @xcite al. (2018b) training begins with two monolingual corpora which are not necessarily related in any way (i.e. they are not assumed to be parallel nor comparable text). first, word embeddings are trained independently for each corpus, resulting in a source and a target embedding space. specifically, after preprocessing, @xcite train two statistical language models using kenlm @xcite , one for the source language and one for the target. they use phrase2vec foot_3 @xcite , an extension of @xcite 's skip-gram model, foot_4 to generate phrase embeddings for 200,000 unigrams, 400,000 bigrams, and 400,000 trigrams. next, source and target word embeddings are aligned into a common cross-lingual embedding space. they run vecmap foot_5 @xcite which calculates a linear mapping of one space to another based on the intuition that phrases with similar meaning should have similar neighbors regardless of language. given a matrix of source word embeddings x and target word embeddings z which have been length-normalized, meancentered, then length-normalized again, vecmap calculates m @xmath0. each cell m x ij and m z ij is the cosine similarity between words x i and x j , and z i and z j , respectively. m x and m z are symmetric, and if the monolingual vector spaces were fully isometric, m x and m z would be identical besides rows and columns being permuted. each row of m x and m z is a similarity distribution. to exploit this, each row of √ m x and √ m z is sorted (they find that using the square root works better empirically), and length-normalized, mean-centered, and length-normalized again. for each row i in sorted( √ m x ), they find the row j of sorted( √ m z ) that is its nearest neighbor, and assign x @xmath1. a cell d @xmath2, and z j are translations of one another, and 0 otherwise. next, there is an iterative process of calculating the optimal linear mappings and extracting an updated dictionary. for calculating the mapping, the goal is to find the linear transformations w x and w z which maximize the cosine similarity of the words that are translations of one another as defined by the dictionary d, over the entire dictionary: from there, they calculate @xmath3, whereby each cell in m is the cosine similarity of word x i and z j after their transformations with w x and w z . to avoid poor local optima, they stochastically zero-out some cells of m with probability @xmath4.9, decreasing over time. the final score for each potential translation candidate is calculated using cross-domain similarity local scaling (csls) @xcite to mitigate the hubness problem. csls utilizes cosine similarity, which is taken from m . for each pair of words x i and z j , the new dictionary cell d @xmath5, and d @xmath6. the dictionary is created in both directions, and concatenated. readers are directed to @xcite for further details. the next step extracts an initial phrase-table for use in a smt system. they use the softmax over the cosine similarity of the 100 nearest-neighbors of each source phrase embedding as the phrase translation probabilities. this is done in both directions: combining the preliminary phrase table with a distortion penalty and language model produces the initial unsupervised phrase-based smt system @xcite . the smt model weights are tuned using a variant of @xcite designed for unsupervised scenarios, which uses 10,000 parallel sentences generated via backtranslation @xcite . the smt model then undergoes three rounds of iterative backtranslation. @xcite extend their 2018 work by adding a critical "nmt hybridization" final step, which achieves significant gains over smt alone. 7 an nmt system is trained using backtranslated output from smt for one epoch. on the next epoch, a small number of sentences are backtranslated with the newly-trained nmt system and concatenated with a slightly smaller fraction of smt-generated bitext. the procedure continues for 30 epochs, gradually increasing the percentage of synthetic training data created by the nmt system until all of the training data is nmt-generated. the nmt system is trained for an additional 30 epochs of iterative backtranslation using data generated fully by the nmt system of the previous epoch. the test set is translated with beam search using an ensemble of models saved at every tenth epoch (six total), resulting in bleu scores of 33.2 and 26.4 (sacrebleu @xcite ) on newstest2014 for french-english and german-english, respectively. we run @xcite 's implementation for our experiments. specifically, neural models are transformer-big @xcite trained with fairseq @xcite on one nvidia geforce gtx 1080ti gpu. models use shared embeddings, the adam optimizer with β 1 = 0.9, β 2 = 0.98 @xcite , label smoothing, initial learning rate of 1e-07 warming up for 4000 steps to 5e-04 before decaying, and dropout @xcite ) probability of 0.3. we set optimizer delay to 4 to simulate 4 gpus. to elucidate the performance gap due to the unsupervised architecture, we build a standard supervised nmt system using the same neural architecture described above. we train until performance on the development set ceases to improve for 10 epochs. to parallel the unsupervised setup, we translate the test set using an ensemble of 6 models; we perform ensemble selection by performance on a validation set, selecting the best-performing checkpoint along with 5 previous checkpoints. existing unsupervised translation methods work well on languages which are similar to each other, use the same roman script, and have an ample amount of monolingual news data available (which matches the test set domain). questions remain as to whether unsupervised methods will be useful on authentic low-resource settings where few or none of the aforementioned properties hold. namely, does unsupervised mt work with: ## evaluation of unsupervised mt we perform an extensive empirical evaluation of unsupervised mt. our evaluation protocol stresstests an unsupervised mt system under varying conditions to reveal its points of strength and failure. systems should be judged on how well they perform: (1) on dissimilar languages, (2) on increasingly divergent domains between source and target corpora, (3) on diverse datasets, and (4) on authentic low-resource language pairs where data quality is typically low. namely, we: the data conditions above are designed measure how well a system performs in regards to the research questions of section 4. namely, success on a variety of languages with different scripts and linguistic structure indicates robustness to dissimilar languages; success on multiple datasets of different domains indicates that the system is not specifically designed for one domain at the expense of others, and performs well even when training and test data do not match perfectly; step #3 evaluates performance on increasingly divergent domains between source and target data; and step #4 is the true test-whether the system succeeds on authentic low-resource language pairs. 6 datasets training datasets used in our reinvestigation of the unsupervised mt system presented in @xcite are shown in specifics of the datasets used are described in subsequent subsections. is the condition most similar to @xcite . src (m) and trg (m) columns are the token counts, in millions. "supervised" count is in bpe tokens. all others are token counts for smt (pre-bpe). the subsections below are before preprocessing, whereas we will release the preprocessed data splits for others to compare their results with ours. ## reinvestigation of artetxe et. al. first, we replicate @xcite , achieving relatively comparable results ( next, we set up a series of experiments to assess the questions posed in section 4. results are presented in tables 3 and 4 . ## discussion except in the diff. dom. condition, unsupervised mt performance for fr-en is impressive and suggests that sentence alignment may not be required for successful mt under ideal conditions. ru-en results are also impressive, but show that unsupervised mt still struggles when language pairs are dissimilar, especially when data amount is reduced. the gap between disjoint and diff. dom. conditions is perhaps the most striking result in our experiments. it suggests that one cannot naively collect monolingual corpora without considering their relative domain similarity; this may be a challenge in low-resource conditions, where there is less flexibility with data sources. @xcite make a similar claim about unsupervised cles, stating "unsupervised methods are able to yield a good solution only when there is no domain mismatch and for the pair with two most similar languages (english-spanish), again questioning their robustness and portability to truly low-resource and more challenging setups". furthermore, the extremely poor results of ne-en and si-en reflect the reality of low-resource translation; the compound negative effects of language dissimilarity, domain mismatch between monolingual corpora, domain mismatch with the test set, and low amounts of low-quality data. it is the "worst of all worlds"but reflects how current models might perform on the use cases for which they are needed. these challenges highlight the importance of evaluating unsupervised mt under varying realistic data conditions. our evaluation is a step towards this goal, and identifies multiple areas for improvement. a critical step in state-of-the-art unsupervised mt is methods for creating cles. several authors have pointed out that "mapping" methods like vecmap assume that monolingual vector spaces are structurally similar, but that this "approximate isomorphism assumption" is increasingly tenuous as languages and domains diverge @xcite @xcite @xcite . @xcite find this for fr-en and ru-en specifically, the languages examined in this work. @xcite argue that while linearity may hold within local "neighborhoods" of the vector space, the global mapping is non-linear. @xcite use their eigenvector similarity metric to show a strong correlation between vector space similarity and bli performance. analysis of the cles from our experiments demonstrate a relationship between bli performance and downstream bleu on the translation task. coupled with our empirical evidence, the works cited in this section suggest that nonisometric vector spaces lead to poor quality translation. factors observed in our experiments that lead to lower quality translation can be attributable to a "weak isomorphism" between the monolingual vector spaces. dissimilar languages means increasingly different distributional characteristics of words. data from different domains naturally have different word frequencies and distributional characteristics, which become more pronounced as domains diverge. because mapping methods rely on structural similarity of vector spaces, experiments using either un or cc data alone had acceptable downstream performance, where as combining the datasets as source and target resulted in extremely poor translation. we observe the critical effect of word embedding initialization on bli performance and downstream bleu, suggesting that stochasticity during word embedding creation can cause resulting vector spaces to be more or less isomorphic. finally, more data can give a more accurate distribution of words in comparison with the true distribution in the language, leading to a more realistic monolingual vector space. with less data, word embeddings are dependent on the smaller training sample, which may not match the test set or reflect true distributional properties of the language. combining all of these negative factors likely leads to highly nonisomorphic monolingual embedding spaces, as demonstrated by the very poor si-en and ne-en results. ## conclusion & future work progress in unsupervised mt has been impressive, achieving performance near its supervised counterparts under some scenarios. that said, evaluating current approaches under broader settings and datasets reveals that unsupervised mt struggles in realistic low-resource scenarios. as stated by @xcite , "it's an open question whether there are more effective instantiations of these principles [underlying recent successes in fully unsupervised mt] or other principles altogether". in this work, we find that there is room for improvement to become robust to (1) dissimilar languages pairs, (2) dissimilar domains, (3) diverse datasets, and (4) the low-quality data of true low-resource languages-factors ubiquitous in low-resource language pairs for which unsupervised mt is intended. we find that (a) performance rapidly declines when source and target corpora are from different domains, and (b) stochasticity during word embedding training can dramatically affect downstream translation results. the latter is a yet unexplored research area. future work should also evaluate pretraining methods in bilingual and multilingual training contexts. finally, we argue for extensive evaluation of unsupervised mt systems under varying data conditions to assess failure cases and encourage pursuit of promising paradigms. doing so is a step towards solving the real-world problems of low-resource machine translation.