Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables (2021)

Abstract: Zero-shot translation, directly translating between language pairs unseen in training, is a promising capability of multilingual neural machine translation (NMT). However, it usually suffers from capturing spurious correlations between the output language and language invariant semantics due to the maximum likelihood training objective, leading to poor transfer performance on zero-shot translation. In this paper, we introduce a denoising autoencoder objective based on pivot language into traditional training objective to improve the translation accuracy on zero-shot directions. The theoretical analysis from the perspective of latent variables shows that our approach actually implicitly maximizes the probability distributions for zero-shot directions. On two benchmark machine translation datasets, we demonstrate that the proposed method is able to effectively eliminate the spurious correlations and significantly outperforms state-of-the-art methods with a remarkable performance. Our code is available at https://github.com/Victorwz/zs-nmt-dae .

## introduction multilingual neural machine translation (nmt) system concatenates multiple language pairs into one single neural-based model, enabling translation on multiple language directions @xcite @xcite @xcite . besides, the multilingual nmt system can achieve translation on unseen language pairs in training, and we refer to this setting as zero-shot nmt. this finding is promising that zero-shot translation halves the decoding time of pivot-based method and avoids the problem of error propagation. meanwhile, zero-shot nmt casts off the requirement of parallel data for a potentially quadratic number of language pairs, which is sometimes impractical especially between lowresource languages. despite the potential benefits, achieving high-quality zero-shot translation is a very challenging task. standard multilingual nmt systems are sensitive to hyper-parameter settings and tend to generate poor outputs. one line of research believes that the success of zero-shot translation depends on the ability of the model to learn language invariant features, or an interlingua, for cross-lingual transfer @xcite @xcite . @xcite design auxiliary losses on the nmt encoder that impose representational invariance across languages. @xcite build up a universal encoder for different languages via bridge language model pre-training, while @xcite disentangle positional information in multilingual nmt to obtain language-agnostic representations. besides, @xcite point out that the conventional multilingual nmt model heavily captures spurious correlations between the output language and language invariant semantics due to the maximum likelihood training objective, making it hard to generate a reasonable translation in an unseen language. then they investigate the effectiveness of decoder pre-training and back-translation on this problem. in this paper, we focus on english-centric multilingual nmt and propose to incorporate a simple denoising autoencoder objective based on english language into the traditional training objective of multilingual nmt to achieve better performance on zero-shot directions. this approach is motivated by an observation that: as shown in we evaluate the proposed method on two public multilingual datasets with several english-centric language-pairs, europarl @xcite and mul-tiun @xcite . experimental results demonstrate that our proposed method not only achieves significant improvement over vanilla multilingual nmt on zero-shot directions, but also outperforms previous state-of-the-art methods. ## multilingual nmt the multilingual nmt system @xcite combines different language directions into one single translation model. due to data limitations of non-english languages, multilingual nmt systems are mostly trained on large-scale englishcentric corpus via maximizing the likelihood over all available language pairs s: l m (θ) = (i,j)∈s,(x,y)∈d i,j log p (y|x, j; θ), (1) where (i, j) ∈ s are the sampled source language id and target language id in all available language pairs, d i,j represents for the corresponding parallel data, and θ is the model parameter. the target language id is appended as the initial token of source sentences, to let the model know which language it should translate to. in addition, the multilingual nmt system has proven the capability of translating on unseen pairs in training @xcite , which is a property of zero-shot translation. however, the zero-shot translation quality significantly falls behind that of pivoting methods. the main issue leading to the unsatisfactory performance is that the multilingual nmt model captures spurious correlations between the output language and language invariant semantics due to the maximum likelihood training objective @xcite . ## method in this section, we first introduce the denoising autoencoder task and then analyze the effectiveness of our proposed method from the perspective of latent variables. denoising autoencoder task. given englishcentric parallel data (x/y/...⇔en), we usually optimize the maximum likelihood training objective to build the multilingual nmt model. since the target language id is inserted at the beginning of the source sentence and only treated as a single token, the maximum likelihood training objective easily ignores the functionality of target language id, leading to unreasonable mutual information between language semantic of "x/y/..." and output language of english. to address this problem, we introduce a denoising sequence-to-sequence task, in which we directly replace the original input sentence with a noisy target english sentence in training data. in this way, previous mutual information can be significantly reduced, while enhancing the relationship between language ids and output language. specifically, we simply use all english sentences in parallel data to construct the denoising english corpus d en via text infilling operation @xcite . then we optimize the multilingual nmt model via maximizing the original translation objective l m (θ) and denoising autoencoder objective l d (θ): with above equation, the original objective is transformed into optimizing three sub-tasks p (h|x), p (y|h) and p (h|h). roparl and multiun. the data statistics of two selected datasets are summarized in ## conclusion in this paper, we proposed to introduce denoising autoencoder objective into conventional translation objective to improve the zero-shot performance of multilingual nmt system. we analyze the motivation and effectiveness of proposed method from the perspective of latent variables. the experimental results demonstrate that our proposed method can significantly resolve spurious correlation issue in multilingual nmt and achieves state-of-the-art performance on zero-shot translation. in the future, it is interesting to explore the combination of our method and other language model pre-training methods @xcite .