Filtering Back-Translated Data in Unsupervised Neural Machine Translation (2020)

Abstract: Unsupervised neural machine translation (NMT) utilizes only monolingual data for training. The quality of back-translated data plays an important role in the performance of NMT systems. In back-translation, all generated pseudo parallel sentence pairs are not of the same quality. Taking inspiration from domain adaptation where in-domain sentences are given more weight in training, in this paper we propose an approach to filter back-translated data as part of the training process of unsupervised NMT. Our approach gives more weight to good pseudo parallel sentence pairs in the back-translation phase. We calculate the weight of each pseudo parallel sentence pair using sentence-wise round-trip BLEU score which is normalized batch-wise. We compare our approach with the current state of the art approaches for unsupervised NMT.

## introduction back-translation involves generating a set of pseudo parallel sentence pairs using monolingual data of target language and a target to source machine translation model. back-translation provides the capability to utilize target-side monolingual data for training. unsupervised nmt gained a lot of attention in the last two years. current state of the art approaches for unsupervised nmt even surpasses supervised baseline for english-french language pair @xcite . unsupervised nmt has three main components: cross-lingual embeddings, denoising, and back-translation, where training involves alternating between denoising and back-translation after a good initialization process @xcite . in this paper our focus is on improving finetuning phase, we are introducing a weight component in unsupervised nmt training based on the quality of pseudo parallel sentence pairs generated for training in back-translation phase. these kinds of techniques have been utilized in domain adaptation to give more weight to in-domain sentences @xcite . pretraining is also a key component of unsupervised nmt, we utilize existing pretraining approaches proposed in @xcite and @xcite . ## related work our work majorly involves the exploration of filtering of back-translated data in unsupervised nmt. we briefly describe some related concepts of back-translation, unsupervised nmt and language model pretraining in this section. back-translation utilizes target-side monolingual data to create pseudo parallel sentence pairs using a translation system from target to source which is then utilized to train source to target nmt system @xcite . @xcite show that iteratively generating better synthetic data improves the nmt performance. quality of back-translated data plays an important role in performance of nmt systems @xcite . @xcite show that the target side words which have high prediction loss gets most benefit from the addition of synthetic data. filtered pseudo parallel data selected with a threshold on round-trip bleu score helps in improving the performance of nmt systems for low resource languages @xcite . in reinforcement learning based approaches, rewards for pseudo parallel sentence pairs based on language model score and round trip reconstruction error helps nmt models @xcite . @xcite provides an identification mark for synthetic sentence pairs while training on a mix set of human generated and synthetic sentence pairs . junczys-dowmunt (2018) show that filtering the parallel data based on cross-entropy scores which calculates the agreement between both models of both directions from source to target and target to source helps in selection of good pseudo parallel sentence pairs. @xcite show that for iterative back-translation selecting and weighing sentences based on the quality of sentence pairs improves the performance of nmt systems, they use combination of different scores like round-trip bleu, tf-idf, language model scores etc. to select top sentences, and then on rest of the sentences they use encoder representation similarities and agreement between forward and backward models to provide weights to back-translated data. wang et al. (2019a) in domain adaptation different techniques have been applied to give more weight to in-domain sentences in the training process of nmt, some of these techniques are: providing a weight component in the loss function (junczys-dowmunt, 2018), using curriculum learning @xcite , and dynamically selecting data with iterations @xcite . pretraining in unsupervised nmt is generally focused on language model training of both encoder and decoder to make them understand the language properties and to provide a good initialization to the finetuning phase. finetuning utilizes the approaches proposed in @xcite which involves denoising and back-translation. @xcite proposed a good initialization mechanism using statistical machine translation for training unsupervised nmt. @xcite proposed a new architecture for unsupervised nmt which does not use back-translation but try to find the best possible translations from the target corpus and edit them to make pseudo parallel sentence pairs. @xcite proposed to utilize two independent encoders with sharing some partial weights. @xcite proposed a pretraining mechanism for unsupervised nmt to pretrain encoder and decoder separately using monolingual data. @xcite show that training encoder and decoder simultaneously using monolingual data helps in pretraining of unsupervised nmt. ## approach current state of the art approaches for unsupervised nmt @xcite do not consider the quality of each generated pseudo parallel sentence pair in the process of training, all generated pseudo parallel sentence pairs have the same weight. there exist different methods to filter bad pseudo parallel sentence pairs with a threshold, which is explained in the previous section. in case of iterative back-translation it is difficult to select a threshold for each batch separately as the training progresses. in the initial iterations of back-translation phase the quality of generated pseudo parallel sentence pairs is very poor. to decrease the weights of bad pseudo parallel sentence pairs we propose to modify back-translation training to include weight of each pseudo parallel sentence pair based on sentence wise round-trip bleu score which is normalized batch-wise. we perform batch-wise normalization because it helps in maintaining a steady progress in training and also helps maintaining equal weightage of denoising and back-translation. round-trip bleu score is the bleu score between the source sentence and the translation of source sentence to target then back to source language. as the systems in both directions (source to target and target to source) are trained simultaneously we can calculate the round-trip bleu score based on the current trained systems in both directions. the sentence wise round-trip bleu score is added in the cross-entropy loss function as weight of each pseudo parallel sentence pair. in general, the cross-entropy loss function is given by: ## experiment and results in this section we show the impact of inclusion of filtering of back-translated data using above approach with two state of the art unsupervised nmt benchmarks for three language pairs.