Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation (2020)

Abstract: Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs. However, it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. That is, research on multilingual UNMT has been limited. In this paper, we empirically introduce a simple method to translate between thirteen languages using a single encoder and a single decoder, making use of multilingual data to improve UNMT for all language pairs. On the basis of the empirical findings, we propose two knowledge distillation methods to further enhance multilingual UNMT performance. Our experiments on a dataset with English translated to and from twelve other languages (including three language families and six language branches) show remarkable results, surpassing strong unsupervised individual baselines while achieving promising performance between non-English language pairs in zero-shot translation scenarios and alleviating poor performance in low-resource language pairs.

## introduction recently, neural machine translation (nmt) has been adapted to the unsupervised scenario in which nmt is trained without any bilingual data. unsupervised nmt (unmt) @xcite requires only monolingual corpora. unmt achieves remarkable results by using a combination of diverse mechanisms @xcite such as an initialization with bilingual word embeddings, denoising auto-encoder @xcite , back-translation @xcite , and shared latent representation. more recently, @xcite achieves better unmt performance by introducing the pretrained language model. however, conventional unmt can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time @xcite . multilingual unmt (munmt) translating multiple languages at the same time can save substantial training time and resources. moreover, the performance of munmt in similar languages can promote each other. research on munmt has been limited and there are only a few pioneer studies. for example, @xcite and @xcite proposed a multilingual scheme that jointly trains multiple languages with multiple decoders. however, the performance of their munmt is much worse than our re-implemented individual baselines (shown in tables 2 and 3 ) and the scale of their study is modest (i.e., 4-5 languages). in this paper, we empirically introduce an unified framework to translate among thirteen languages (including three language families and six language branches) using a single encoder and single decoder, making use of multilingual data to improve unmt for all languages. on the basis of these empirical findings, we propose two knowledge distillation methods, i.e., self-knowledge distillation and language branch knowledge distillation, to further enhance munmt performance. our experiments on a dataset with english translated to and from twelve other languages show remarkable results, surpassing strong unsupervised individual baselines.this paper primarily makes the following contributions: unmt can be decomposed into four components: cross-lingual language model pretraining, denoising auto-encoder, back-translation, and shared latent representations. for unmt, two monolingual corpora x 1 = {x 1 i } and x 2 = {x 2 i } in two languages l 1 and l 2 are given. |x 1 | and |x 2 | are the number of sentences in monolingual corpora {x 1 i } and {x 2 i } respectively. ## knowledge distillation for munmt to further enhance the performance of our proposed munmt described in section 3, we propose two knowledge distillation methods: selfknowledge distillation (algorithm 1) and language branch knowledge distillation (algorithm 2). figure 2 illustrates the architecture of munmt and the proposed knowledge distillation methods. generally, during unmt training, an objective function l kd is added, to enhance the generalization ability of the munmt model. the general munmt objective function can be reformulated as follows: ## conclusion and future work in this paper, we have introduced a unified framework, using a single encoder and decoder, for munmt training on a large scale of european languages. to further enhance munmt performance, we have proposed two knowledge distillation methods. our extensive experiments and analysis demonstrate the effectiveness of our proposed methods. in the future, we intend to extend the work to include language types such as asian languages. we will also introduce other effective methods to improve zero-shot translation quality.