Robust Unsupervised Neural Machine Translation with Adversarial Denoising Training (2020)

Abstract: Unsupervised neural machine translation (UNMT) has recently attracted great interest in the machine translation community. The main advantage of the UNMT lies in its easy collection of required large training text sentences while with only a slightly worse performance than supervised neural machine translation which requires expensive annotated translation pairs on some translation tasks. In most studies, the UMNT is trained with clean data without considering its robustness to the noisy data. However, in real-world scenarios, there usually exists noise in the collected input sentences which degrades the performance of the translation system since the UNMT is sensitive to the small perturbations of the input sentences. In this paper, we first time explicitly take the noisy data into consideration to improve the robustness of the UNMT based systems. First of all, we clearly defined two types of noises in training sentences, i.e., word noise and word order noise, and empirically investigate its effect in the UNMT, then we propose adversarial training methods with denoising process in the UNMT. Experimental results on several language pairs show that our proposed methods substantially improved the robustness of the conventional UNMT systems in noisy scenarios.

## introduction recently, unsupervised neural machine translation (unmt) has attracted great interest in the machine translation community @xcite @xcite @xcite . typically, unmt relies solely on monolingual corpora rather than bilingual parallel data in supervised neural machine translation (snmt) to model translations between the source language and target language and has achieved remarkable results on several translation tasks @xcite . however, previous work only focus on how to build stateof-the-art unmt systems on the clean data and ignore the robustness of unmt on the noisy data. in the real-world scenario, there often exists noises or perturbations in the input sentences, for example, word character misspelling, replacement, or word position misordering, etc. the translation model is sensitive to these perturbations, leading to various errors even the perturbations are small. the existing neural translation system, which lacks of robustness, is difficult to be widely applied to the noisy-data scenario (denoted as noisy scenario in the following sections). therefore, the robustness of neural translation system is not only worthy of being studied, but also very essential in the real-world scenarios. the robustness of snmt @xcite @xcite has been well-studied. however, most previous work only focus on the effect of the word substitution for translation performance, and ignore the effect of word order for translation performance. moreover, the noisy robustness of unmt is much more difficult since the noisy input data may be relieved in some degree by the snmt due to its supervised check in training. currently, there is no study considering the noisy robustness of the unmt. in this paper, we first define two types of noises which cover the noise types mentioned above, i.e., word noise and word order noise. then we empirically investigate the performance of unmt in these noisy scenarios. our empirical results show that the performance of unmt model decreased substantially, regardless of any noisy scenario. to improve the robustness, we proposed adversarial training methods to alleviate the poor performance in these noisy scenarios. to the best of our knowledge, this paper is the first work to explore the robustness of unmt. experimental results on several language pairs show that the proposed strategies substantially outperform conventional unmt systems in the noisy scenarios. our main contributions are summarized as follows: there are four primary components of the state-of-the-art unmt (conneau and lample, 2019): crosslingual language model pre-training, denoising auto-encoder, back-translation, and sharing latent representations. consider monolingual data {x i } in language l 1 and {y i } in another language l 2 . |x| and |y | are the number of sentences in monolingual corpora {x i } and {y i } respectively. the encoders and decoders of l 1 , l 2 are trained through denoising and back-translation. the objective function l all of the entire unmt model would be optimized as: cross-lingual language model pre-training: it aims at building a universal cross-lingual encoder that can encode two monolingual sentences into a shared embedding space. the pre-trained cross-lingual encoder is then used to initialize the unmt model. denoising auto-encoder: in contrast with the normal auto-encoder, denoising auto-encoder @xcite could improve the model learning ability by introducing noise in the form of random token deleting and swapping in this input sentence. the denoising auto-encoder, which encodes a noisy version and reconstructs it with the decoder in the same language, acts as a language model during unmt training. it is optimized by minimizing the objective function: ## preliminary experiments toward noisy input to unmt system in this section, we first introduce the two primary types of noises in the corpus of unmt. then we empirically analyze the effect of these noises on unmt. ## proposed methods based on the previous empirical findings and analysis, we propose two adversarial training methods during denoising training to improve the robustness of unmt in the two noisy scenarios as defined in section 3.1. the frameworks are illustrated in figure 2 . the figure 2 (a) is the original denoising training framework in which an encoder-decoder structure is applied. based on this original framework, two adversarial training frameworks are proposed (figure 2 (b) and (c )) in which the word noise and word-order noise blocks are explicitly modeled. although the original structure has the denoising ability to replace some wrong words and adjust word order, the denoising ability is implicitly modeled without considering the embedding representation effect which is important in translation. as an adversarial training, our two proposed frameworks explicitly model the two noise effects on the word embedding and position embedding which is expected to improve the translation performance. the details are explained in the following subsections. ## related work recently, unmt @xcite @xcite @xcite that relies solely on monolingual corpora in each language via bilingual word embedding initialization, denoising auto-encoder, back-translation and sharing latent representations. more recently, conneau and lample (2019) and song et al. (2019) actually, @xcite pointed out that synthetic and natural noise both influenced the translation performance. @xcite , @xcite , and @xcite designed character-level noise, which affects the spelling of a single word, to improve the model robustness. meanwhile, both textual and phonetic embeddings were used to improve the robustness of snmt to homophone noises @xcite . adversarial examples, generated by gradient-based method, attacked the translation model to improve the robustness of snmt @xcite . in contrast with this work, we applied adversarial perturbation to the denoising training of unmt , instead of translation training, to enhance the learning ability of unmt model. as a data driven machine learning method, neural translation model is sensitive to small data perturbations which may result in poor performance in noisy scenarios. in this paper, we focused on the noisy robust problem in unsupervised machine translation. we first explicitly defined two types of noise which are frequently appeared in real applications. then we proposed denoising process on word embedding and word positional embedding based adversarial training in the unmt framework. experimental results confirmed that the proposed adversarial training significantly improved the robustness of the unmt systems in the noisy scenarios. moreover, even for clean scenario, our proposed framework could slightly improve the performance. from the results, we can conclude that our proposed framework could improve the noisy robustness of the unmt without sacrificing the performance for clean condition. in this study, our proposed adversarial training was implemented with the transformer based neural architectures for unmt. in the future, we will examine the proposed method in other natural language processing tasks and integrate other machine learning methods for the robustness of the unmt systems.