A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation (2020)

Abstract: The commonly used framework for unsupervised machine translation builds initial translation models of both translation directions, and then performs iterative back-translation to jointly boost their translation performance. The initialization stage is very important since bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. In this paper, we propose a novel retrieval and rewriting based method to better initialize unsupervised translation models. We first retrieve semantically comparable sentences from monolingual corpora of two languages and then rewrite the target side to minimize the semantic gap between the source and retrieved targets with a designed rewriting model. The rewritten sentence pairs are used to initialize SMT models which are used to generate pseudo data for two NMT models, followed by the iterative back-translation. Experiments show that our method can build better initial unsupervised translation models and improve the final translation performance by over 4 BLEU scores. Our code is released at https://github.com/Imagist-Shuo/RRforUNMT.git .

## introduction recent work has shown successful practices of unsupervised machine translation (umt) @xcite @xcite @xcite @xcite . the common framework is to build two initial translation models (i.e., source to target and target to source) and then do iterative back-translation @xcite with pseudo data generated by each other. the initialization stage is important because bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. previous methods for umt @xcite @xcite usually use the following n-gram embeddings based initialization. they first build phrase translation tables with the help of unsupervised cross-lingual n-gram embeddings @xcite , and then use them to build two initial phrase-based statistical machine translation (pbsmt) @xcite models with two language models. however, there are two problems with their initialization methods. in this paper, we propose a novel retrieve-andrewrite initialization method for umt. specifically, we first retrieve semantically similar sentence pairs from monolingual corpora of two languages with the help of unsupervised cross-lingual sentence embeddings. next, with those retrieved similar sentence pairs, we run giza++ @xcite to get word alignments which are used to delete unaligned words in the target side of the retrieved sentences. the modified target sentences are then rewritten with a designed sequence-to-sequence rewriting model to minimize the semantic gap between the source and target sides. taking the pairs of the source sentences and corresponding rewritten targets as pseudo parallel data, we then build two initial pbsmt models (source-to-target and targetto-source), which are used to generate pseudo parallel data to warm up nmt models, followed by an iterative back-translation training process. our code is released at https://github.com/imagistshuo/rrforunmt.git. our contributions are threefold. (1) we propose a novel method to initialize unsupervised mt models with a retrieve-and-rewrite schema, which can preserve the rich sentence structure and provide high-quality phrases. (2) we design an effective seq-to-seq architecture based on the transformer to rewrite sentences with semantic constraints. (3) our method significantly outperforms the previous non-pre-training based umt results on en-f r and en-de translation tasks, and give the first unsupervised en-zh translation results on wmt17. ## method our method can be divided into three steps as shown in figure 1 . first, we do similar sentences retrieval ( §2.1) from two monolingual corpora with the help of unsupervised cross-lingual sentence embeddings. next, to minimize the semantic gap between the source and retrieved targets, we do target sentences rewriting ( §2.2) by deleting unaligned words in the target side, and generate complete and better-aligned targets via our rewriting model with the help of missing information provided by the source. after that, we treat the rewritten pairs as the pseudo parallel data for translation models initialization and training ( §2.3). ## related work unsupervised machine translation becomes a hot research topic in recent years. the pioneering methods are based on nmt models (transformer) @xcite @xcite trained with denoising auto-encoder @xcite and iterative back-translation. the following work shows that smt methods and the hybrid of nmt and smt can be more effective @xcite @xcite @xcite . they build the initial pbsmt models with language models and phrase tables inferred from unsupervised cross-lingual n-gram embeddings. recently, @xcite propose a pre-training method and achieve state-of-theart performance on unsupervised en-f r and en-de translation tasks. but they use much more monolingual data from wikipedia than previous work and this paper. we must also mention the work of @xcite . they similarly use retrieval and rewriting framework for unsupervised mt. however, ours is different from theirs in two aspects. first, we efficiently calculate the cross-lingual sentence embeddings via a training-free method sif rather than a pre-trained language model. second, our rewriting method is based on the word alignment information which is more explicit than their max pooling, and our rewriting model is more simple but effective so that the rewriting results can be directly used without extra training techniques. ## conclusion in this paper, we propose a novel method for unsupervised machine translation with a retrieve-andrewrite schema. we first retrieve similar sentences from monolingual corpora and then rewrite the targets with a rewriting model. with the pseudo parallel data, we better initialize pbsmt models and significantly improve the final iteration performance as the experiments show.