Reference Language based Unsupervised Neural Machine Translation (2020)

Abstract: Exploiting a common language as an auxiliary for better translation has a long tradition in machine translation and lets supervised learning-based machine translation enjoy the enhancement delivered by the well-used pivot language in the absence of a source language to target language parallel corpus. The rise of unsupervised neural machine translation (UNMT) almost completely relieves the parallel corpus curse, though UNMT is still subject to unsatisfactory performance due to the vagueness of the clues available for its core back-translation training. Further enriching the idea of pivot translation by extending the use of parallel corpora beyond the source-target paradigm, we propose a new reference language-based framework for UNMT, RUNMT, in which the reference language only shares a parallel corpus with the source, but this corpus still indicates a signal clear enough to help the reconstruction training of UNMT through a proposed reference agreement mechanism. Experimental results show that our methods improve the quality of UNMT over that of a strong baseline that uses only one auxiliary language, demonstrating the usefulness of the proposed reference language-based UNMT and establishing a good start for the community.

## introduction recently, the application of neural machine translation (nmt) (sutskever et al., 2014; * corresponding authors. this paper was partially supported by national key research and development program of china (no. 2017yfb0304100), key projects of national natural science foundation of china (u1836222 and 61733011), huawei-sjtu long term ai project, cuttingedge machine reading comprehension and language model. rui wang was partially supported by jsps grant-in-aid for early-career scientists (19k20354): "unsupervised neural machine translation in universal scenarios" and nict tenuretrack researcher startup fund "toward intelligent machine translation". munmt, (c) our proposed runmt, where s stands for source language, t for target language, p for pivot language in pivot translation, and r for the reference language in runmt. @xcite to standard benchmarks has achieved great success @xcite @xcite because of advances in deep learning and the availability of large-scale parallel corpora; however, the applicability of mt systems is limited because of their reliance on large parallel corpora for the majority of language pairs. in real-world situations, the majority of language pairs have very little parallel data, although large volumes of monolingual data are available for each language. unmt removes the dependence on parallel corpora, relying only on monolingual corpora in each language @xcite @xcite . unmt uses translation symmetry for dual learning in each language direction. existing unmt models are mainly built on the encoderdecoder schema. the essence of unmt is to learn unsupervised cross-lingual word alignment and/or sentence alignment. for unsupervised word alignment, the most popular methods are word embedding mapping @xcite @xcite , vocabulary sharing @xcite , and language modeling @xcite . weight sharing can also be adopted in the encoder/decoder, adversarial training, and back-translation (bt) processes for unsupervised sentence alignment. bt aims to train models using iteratively generated pseudo-parallel data, thus overcoming the lack of cross-language signals. specifically, monolingual data in the source language is translated to the target language using a sourceto-target translation model, and then the pseudoparallel data (including both the generated and the original data) is used to train the target-to-source translation model, and vice versa. unfortunately, as the input sentences in the pseudo-parallel data are generated by unsupervised models, random errors and noise are inevitably introduced, resulting in low-quality parallel data for model training and bad translation performance. in addition, when vocabulary sharing unmt models for two distant languages (that is, very little vocabulary overlap between the source language and target language) are trained with bt, the unsupervised model may generate the words in the source language instead of in the target language under source-to-target forward translation. as a result, although the reconstruction loss is small if the forward translation generation is very similar to the input, the model is not sufficiently optimized because the pseudo-parallel corpus contains very little cross-lingual sentence alignment information. multilingualism @xcite is a powerful fact of communication across speech communities. in multilingualism, an important "lingua franca" (or common language) often serves as an aid to cross-group understanding, usually representing the language of a potent and prestigious society with a large number of users. for machine translation, the parallel corpora between languages and some lingua franca are usually more abundant. thus, conventional pivot translation (pt) usually leverages a resource-rich language (mainly english) as the pivot to help the low/zero-resource translation (see appendix a.1 for a detailed analysis). although unmt no longer requires parallel corpora, this feature is still worth exploring and can be used to enhance current unmt systems under low-or zeroresource scenarios. in addition, we can further use the transfer learning capabilities of the model to transfer the translation capabilities of languages and lingua francas to any two languages that need to be learned. in this work, taking the merits of pivot language translation in both supervised nmt and unmt as shown in figure 1 , we propose the reference language-based unmt framework in which the reference language shares a parallel corpus with only the source language (using only the target language follows a similar pattern). in the framework, we use multilingualism and propose a reference agreement mechanism. exploiting the accurate alignment clues between source and reference languages, we can more confidently enhance source-target unmt by taking into account the translation agreement within the source, reference, and target languages. specifically, this previously irrelevant parallel data plays a role in controlling the quality of the pseudosentence pairs through a cross-lingual equivalence (translation agreement). the proposed mechanism is orthogonal to the common multilingual transfer learning methods and different from the general pivot translation method. empirical results on popular benchmarks and distant languages show that the reference agreement mechanism consistently improves the performance of unmt systems. in addition, we explore the impact of multilingual information on the basis of our multilingual unmt baseline and proposed method. ## unmt unmt is a recently proposed mt paradigm that attempts to achieve the co-growth of mt models in two directions while relying solely on monolingual data and for example, would benefit both englishto-french vs. french-to-english. it is a special kind of dual learning @xcite @xcite in both directions of language pairs. currently, state-of-the-art unmt models are based on a sequence-to-sequence encoderdecoder architecture using transfomer @xcite , similar to supervised nmt models. for ease of expression, in the remainder of this paper, we denote the monolingual training data space of the source s and target t languages as φ s and φ t . the parallel training data space between languages s and t is represented as φ s-t . the translation direction symmetry of the unmt model training implies that the translation direction problem @xmath0. in general, the nmt model with parameters θ @xmath1. the model parameters θ @xmath2: as there is a lack of cross-lingual sentence alignment information, the current unmt models, despite their differences in training methods and structure, reach a consensus over the use of the parallel data that was iteratively generated by the bt method. specifically, for a monolingual sentence of target language @xmath3, a source translation s is generated using the primal @xmath4, θ @xmath5, then s and t form a pseudo-parallel pair s, t for @xmath6. similarly, the generated pseudo-parallel pair t, s for a monolingual sentence s in the source language is also used for training the @xmath7. the likelihood of the reconstructions @xmath8: finally, the bt process is optimized by minimizing the following objective function: in this section, we introduce the reference language-based unmt framework and present our three kinds of reference agreement utilization approaches: reference agreement translation (rat), reference agreement back-translation (rabt), and cross-lingual back-translation (xbt). these approaches are illustrated in figure 2 . when applying pivot translation to unmt, any language pair in unmt can be directly trained without any parallel data, which allows translation in both directions due to the nature of unmt. thus, the traditional pivot schema (@xmath9; using a third language (usually a common language) is a more suitable practice for unmt. in order to distinguish from the pivot language in traditional pivot translation, we define the language used to enhance the performance of translation @xmath10, regardless of whether the translation schema is @xmath11. ## related work with the development of the deep neural network @xcite @xcite @xcite , unmt @xcite @xcite has attracted widespread attention in academic research, as only large-scale monolingual corpora are required for training. the performance of unmt has benefited from language model pre-training, denoising autoencoders, and bt techniques between similar languages such as english and french, but still lags behind that of supervised nmt for distant languages such as chinese and english. @xcite extended the generative language model pre-training approach to multiple languages and showed that cross-lingual pre-training could be effective for munmt. aside from the convenience of translation among multiple language pairs, including unseen language pairs, transfer learning should be considered when low-resource languages are trained together with rich-resource ones. as discussed by @xcite , munmt usually performs worse than pivot-based supervised nmt; however, the pivot-based method easily experiences a computationally expensive quadratic growth in the number of source languages and suffers from the error propagation problem. @xcite addressed the zeroshot generalization problem that some translation directions have not been optimized well due to a lack of parallel data. @xcite introduced a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in zero-shot translation, which share similarities with our rat approach. however, in terms of a specific implementation, because of the differences between unmt and nmt, we have provided three new unmt methods, and have alleviated the problem of uncontrollable intermediate bt quality in unmt. @xcite addressed the issue of transfer learning between language pairs with parallel data where there is a lack of parallel corpora in multilingual supervised nmt. as for the agreement in unmt, @xcite investigate the enhancement of unsupervised bilingual word embedding agreement in the unmt training. @xcite propose a multi-hop unmt that automatically selects a good translation path for a distant language pair during unmt. @xcite proposed a cross-lingual pre-training approach that makes use of the source-pivot data to pre-train the language model. as for the multilingualism, liu et al. (2020) proposes a multilingual denoising pre-training technique to improve machine translation tasks. @xcite and @xcite both studied the agreement across language pairs. their method is much the same as one of our proposed approaches, xbt, which relies on the supervision signals from a parallel corpus to build a bridge between language pairs in munmt. compared with these two concurrent works, the other two settings of our proposed approaches, rat and rabt, which use the internal agreement within language pair to improve the translations, can be used not only for munmt, but also for semisupervised nmt to enhance the effect of the only two languages. ## conclusion in this work, we capitalize on the supervised nmt and unmt use of the pivot language in pivot translation. we propose the reference language-based unmt framework, in which a reference agreement mechanism is introduced in several implementations to better leverage the reference agreement in parallel data brought by the reference language to reduce the uncontrollable intermediate quality problem in back-translation. the experimental results show that we achieved an improvement over our strong baseline, and our proposed runmt framework is compatible with and exceeds the traditional pivot translation framework.