Cross-lingual Supervision Improves Unsupervised Neural Machine Translation (2021)

Abstract: We propose to improve unsupervised neural machine translation with cross-lingual supervision (), which utilizes supervision signals from high resource language pairs to improve the translation of zero-source languages. Specifically, for training En-Ro system without parallel corpus, we can leverage the corpus from En-Fr and En-De to collectively train the translation from one language into many languages under one model. % is based on multilingual models which require no changes to the standard unsupervised NMT. Simple and effective, significantly improves the translation quality with a big margin in the benchmark unsupervised translation tasks, and even achieves comparable performance to supervised NMT. In particular, on WMT’14 -tasks achieves 37.6 and 35.18 BLEU score, which is very close to the large scale supervised setting and on WMT’16 -tasks achieves 35.09 BLEU score which is even better than the supervised Transformer baseline.

## introduction neural machine translation (nmt) has achieved great success and reached satisfactory translation performance for several language pairs @xcite @xcite . such breakthroughs heavily depend on the availability of colossal amounts of bilingual sentence pairs, such as the some 40 million parallel sentence pairs used in the training of wmt14 english french task. as bilingual sentence pairs are costly to collect, the success of nmt has not been fully duplicated in the vast majority of language pairs, especially for zero-resource languages. recently, @xcite ; ?) tackled this challenge by training unsupervised neural machine translation (unmt) models using only monolingual data, which achieves considerably high accuracy, but still not on par with that of the state of the art supervised models. de en fr (a) pivot nmt de en (b) unsupervised nmt de en fr (c) cunmt w/o para. de en fr (d) cunmt w/ para. most previous works focused on modeling the architecture through parameter sharing or proper initialization to improve unmt. we argue that the drawback of unmt mainly stems from the lack of supervised signals, and it is beneficial to transfer multilingual information across languages. in this paper, we take a step towards practical unsupervised nmt with cross-lingual supervision (cunmt) -making the most of the signal from other language. we investigate two variants of multilingual supervision for unmt. a) cunmt w/o para.: a general setting where unrelated monolingual data can be introduced. for example, using monolingual fr data to help the training of en-de (figure 1(c )). b) cunmt w/ para.: a relatively strict setting where other bilingual language pairs can be introduced. for example, we can naturally leverage parallel en-fr data to facilitate the unsupervised en-de transla-tion (figure 1(d) ). we introduce cross-lingual supervision which aims at modeling explicit translation probabilities across languages. taking three languages as an example, suppose the target unsupervised direction is @xmath0. our target is to model the translation probability p(de|en) with the support of p(fr|en) and p(de|fr). for forward cross-lingual supervision, the system nmt @xmath1, translating the fr part of parallel data (en, fr) to de. the resulted synthetic data (en, fr, de) can be used to improve our target system nmt @xmath2. for backward cross-lingual supervision, we translate the monolingual de to fr with nmt @xmath3, and then translate fr to en with nmt @xmath4. the resulted synthetic bilingual data (de, en) can be used for nmt @xmath5. our contributions can be summarized as follow: a) empirical evaluation of cunmt on six benchmarks verifies that it surpassed individual mt models by a large margin of more than 3.0 bleu points on average, and also bested several strong competitors. particularly, on wmt'16 en-ro tasks, cunmt surpass the supervised baseline by 0.7 bleu, showing the great potential for unmt. b) cunmt is very effective in the use of additional training data. mbart or mass introduces billions of sentences, while cunmt only introduces tens of millions of sentences and achieves super or comparable results. it shows the importance of introducing explicit supervision. ## the proposed cunmt cunmt is based on a multilingual machine translation model involving supervised and unsupervised methods with a triangular training structure. the original unsupervised nmt depends only on monolingual corpus, therefore the performances of these translation directions cannot be guaranteed. formally, given n different languages l i , x i denotes a sentence in language l i . d i denotes a monolingual dataset of l i , and d i,j denotes a parallel dataset of (l i , l j ). we use e to indicate the set of all translation directions with parallel data and w to indicate the set of all unsupervised translation directions respectively. the goal of cunmt is to minimize the log likelihood of both unsuper- ## analyses in this part, we conduct several studies on cunmt to better understand its setting. backward or forward here we have explored the effect of cross-lingual backward supervision and cross-lingual forward supervision, and plot the performance curves along with the training procedure in figure 3. the comparison system is cunmt trained only with monolingual data. to make a fair comparison, we use "cunmt w/ para." as the baseline model and fine-tuning it with only indirect forward supervision or indirect backward supervision. we conduct experiments on wmt16 en-de and en-ro tasks. clearly, the forward supervision outperforms the backward one with big margins, which shows the importance of introducing the forward supervision for multilingual unmt. it is still interesting to find that only introducing the indirect backward translation achieves better results than the unsupervised baseline. we suppose the reasons for the performance gap is that, a) the unmt baseline has included the traditional direct back translation, therefore the information gain from indirect backward translation is limited compared to the forward translation. b) the indirect forward translation provides a more direct way to model the relation across different languages. the results in consist with the previous research that pivot translation can help low resource language translation. ## conclusion in this work, we propose a multilingual machine translation framework cunmt incorporating distant supervision to tackle the challenge of the unsupervised translation task. by mixing different training schemes into one model and utilizing unrelated bilingual corpus, we greatly improve the performance of the unsupervised nmt direction. by joint training, cunmt can serve all translation directions in one model. empirically, cunmt has been proven to deliver substantial improvements over several strong unmt competitors and even achieve comparable performance to supervised nmt. in the future, we plan to build a universal cunmt system that is applicable in a wide span of languages.