When and Why is Unsupervised Neural Machine Translation Useless? (2020)

Abstract: This paper studies the practicality of the current state-of-the-art unsupervised methods in neural machine translation (NMT). In ten translation tasks with various data settings, we analyze the conditions under which the unsupervised methods fail to produce reasonable translations. We show that their performance is severely affected by linguistic dissimilarity and domain mismatch between source and target monolingual data. Such conditions are common for low-resource language pairs, where unsupervised learning works poorly. In all of our experiments, supervised and semi-supervised baselines with 50k-sentence bilingual data outperform the best unsupervised results. Our analyses pinpoint the limits of the current unsupervised NMT and also suggest immediate research directions.

## introduction statistical methods for machine translation (mt) require a large set of sentence pairs in two languages to build a decent translation system @xcite . such bilingual data is scarce for most language pairs and its quality varies largely over different domains @xcite . neural machine translation (nmt) @xcite , the standard paradigm of mt these days, has been claimed to suffer from the data scarcity more severely than phrase-based mt @xcite . unsupervised nmt, which trains a neural translation model only with monolingual corpora, was proposed for those scenarios which lack bilingual data @xcite . despite its progress in research, the performance of the unsupervised methods has been evaluated mostly on high-resource language pairs, e.g. german↔english or french↔english @xcite @xcite @xcite @xcite @xcite . for these language pairs, huge bilingual corpora are already available, so there is no need for unsupervised learning in practice. empirical results in these tasks do not carry over to low-resource language pairs; they simply fail to produce any meaningful translations @xcite . this paper aims for a more comprehensive and pragmatic study on the performance of unsupervised nmt. our experiments span ten translation tasks in the following five language pairs: here is a summary of our contributions: ## related work the idea of unsupervised mt dates back to wordbased decipherment methods @xcite . they learn only lexicon models at first, but add alignment models @xcite or heuristic features @xcite later. finally, @xcite and @xcite train a fully-fledged phrase-based mt system in an unsupervised way. with neural networks, unsupervised learning of a sequence-to-sequence nmt model has been proposed by @xcite and @xcite . though having slight variations @xcite @xcite , unsupervised nmt approaches commonly 1) learn a shared model for both @xmath0, along with 3) a denoising autoencoder objective. they are initialized with either cross-lingual word embeddings or a cross-lingual language model (lm). to further improve the performance at the cost of efficiency, @xcite , @xcite and @xcite combine unsupervised nmt with unsupervised phrase-based mt. on the other hand, one can also avoid the long iterative training by applying a separate denoiser directly to the word-by-word translations from cross-lingual word embeddings @xcite . unsupervised nmt approaches have been so far evaluated mostly on high-resource language pairs, e.g. @xmath1, for academic purposes. in terms of practicality, they tend to underperform in low-resource language pairs, e.g. @xmath2. to the best of our knowledge, this work is the first to systematically evaluate and analyze unsupervised learning for nmt in various data settings. ## unsupervised nmt this section reviews the core concepts of the recent unsupervised nmt framework and describes to which points they are potentially vulnerable. ## experiments and analysis data our experiments were conducted on wmt 2018 german↔english and russian↔ english, wmt 2019 chinese↔english, kazakh↔ english, and gujarati↔english (