Zero-Shot Neural Machine Translation with Self-Learning Cycle (2021)

Abstract: Neural Machine Translation (NMT) approaches employing monolingual data are showing steady improvements in resource-rich conditions. However, evaluations using real-world lowresource languages still result in unsatisfactory performance. This work proposes a novel zeroshot NMT modeling approach that learns without the now-standard assumption of a pivot language sharing parallel data with the zero-shot source and target languages. Our approach is based on three stages: initialization from any pre-trained NMT model observing at least the target language, augmentation of source sides leveraging target monolingual data, and learning to optimize the initial model to the zero-shot pair, where the latter two constitute a selflearning cycle. Empirical findings involving four diverse (in terms of a language family, script and relatedness) zero-shot pairs show the effectiveness of our approach with up to +5.93 BLEU improvement against a supervised bilingual baseline. Compared to unsupervised NMT, consistent improvements are observed even in a domain-mismatch setting, attesting to the usability of our method.

## introduction since the introduction of nmt @xcite , model learning using unlabeled (monolingual) data is increasingly gaining ground. undoubtedly, the main motivating factor to explore beyond supervised learning is the lack of enough (parallel) examples, a performance bottleneck regardless of the underlying architecture @xcite . a fairly successful approach using monolingual data is the semi-supervised learning with backtranslation @xcite , particularly if the initial supervised model is good enough for augmenting quality pseudo-bitext @xcite @xcite . moreover, back-translation showed to be a core element of new monolingual based approaches. these include zero-shot nmt @xcite @xcite , which relies on a multilingual model @xcite (fig. 1b ) and unsupervised nmt, which initializes from pre-trained embeddings @xcite or cross-lingual language model @xcite (fig. 1d ). at least two observations can be made on the approaches that leverage monolingual data: i) they require high-quality and comparable monolingual examples, and ii) they show poor performance on real-world zero-resource language pairs (zrps) 1 @xcite . to overcome these problems, in this work we propose a zero-shot modeling approach (fig. 1c ) to translate from an unseen source language u to a target language t that has only been observed by a model pre-trained on (s, t ) parallel data (s being a different source language). in literature, zero-shot nmt has been investigated under the assumption that (u, p ) and (p, t ) parallel data involving a pivot language p are available for pre-training (fig. 1b ) @xcite . however, most of the +7, 000 currently spoken languages do not exhibit any parallel data with a common p language. this calls for new techniques to achieve zero-shot nmt by reducing the requirements of the pivoting-based method. to this aim, our approach follows a self-learning cycle, by first translating in the primal zero-shot direction @xmath0, t) parallel data that has never seen u during training. then, the generated translations are used as a pseudo-bitext to learn a model for the dual @xmath1. this inference-learning cycle is iterated, alternating the dual and primal zero-shot directions until convergence of the u ↔ t zero-shot model. through experiments on data covering eight language directions, we demonstrate the effectiveness of our approach in the zrp scenario. in particular, we report significant improvements compared to both a supervised model trained in low-resource conditions (up to +5.93 bleu) and an unsupervised one exploiting large multilingual corpora (up to +5.23 bleu). our contributions can be summarized as follows: ## zero-shot translation from a broad perspective, zst research is moving in three directions, (i) improving translation quality by employing zst specific objectives @xcite @xcite @xcite @xcite , (ii) training favorable large scale multilingual models for the zst languages with lexically and linguistically similar languages @xcite , and (iii) incrementally learning better model for the zst directions with selflearning objectives @xcite @xcite . the common way of employing self-supervised learning in zst modeling is iterative back-translation that generates the source from the monolingual target to construct a new parallel sentence pair. in terms of performance, while (i) and (ii) fall behind, (iii) either approaches or even outperforms the two-step pivot translation approach (@xmath2. despite these progresses, current approaches make identical assumptions, namely: i) the reliance on a multilingual model, and ii) observing both the u and t zero-shot languages paired with the p language(s). however, in a real-world setting these conditions are rarely satisfied, hindering the application of zero-shot nmt to the vast majority of zrp. moreover, conditioning zst on p language(s) creates a performance ceiling that depends on the amount, domain, and quality of parallel data available for the s -p and p -t pairs. to our knowledge, a zero-shot nmt modeling between an unseen-u and t zero-shot pair, and without the p language(s) criterion has not yet been explored, motivating this work. ## experimental settings to build an experimental ground that defines zero-shot translation without the pivot language, we selected a real-world low-resource languages benchmark. in other words, we considered the data to incorporate multiple and diverse languages, including parallel data for building strong baselines and monolingual data to evaluate znmt. moreover, our choice is motivated by the findings of @xcite and @xcite , showing that monolingual-based approaches under-perform when assessed with real-world zero-shot pairs (zsps). ## results and analysis in ## conclusion we presented a new zero-shot nmt modeling variant, specifically targeting languages that have never been observed in a pre-trained nmt. we showed limitations of current approaches with the pivot language premise and zero-shot translation only between observed languages, and proposed a relaxation to zero-shot nmt to incorporate unseen languages. our approach includes initialization, augmentation, and training stages to construct a self-learning cycle to incrementally correct the primal and dual zero-shot translation quality. we empirically demonstrated the effectiveness of the proposed approach using diverse real-world zero-resource languages in in-domain, out-of-domain, domain-mismatch, and mixed domain scenarios. results both from bilingual and multilingual initialization not only revealed the possibility of extending zero-shot nmt for unseen languages but also improved performance over unsupervised, low-resource supervised and semi-supervised nmt.