Scrambled Translation Problem: A Problem of Denoising UNMT (2021)

Abstract: In this paper and we identify an interesting kind of error in the output of Unsupervised Neural Machine Translation (UNMT) systems like Undreamt1. We refer to this error type as Scrambled Translation problem. We observe that UNMT models which use word shuffle noise (as in case of Undreamt) can generate correct words and but fail to stitch them together to form phrases. As a result and words of the translated sentence look scrambled and resulting in decreased BLEU. We hypothesise that the reason behind scrambled translation problem is ’shuffling noise’ which is introduced in every input sentence as a denoising strategy. To test our hypothesis and we experiment by retraining UNMT models with a simple retraining strategy. We stop the training of the Denoising UNMT model after a pre-decided number of iterations and resume the training for the remaining iterations- which number is also pre-decided- using original sentence as input without adding any noise. Our proposed solution achieves significant performance improvement UNMT models that train conventionally. We demonstrate these performance gains on four language pairs and viz. and English-French and English-German and English-Spanish and Hindi-Punjabi. Our qualitative and quantitative analysis shows that the retraining strategy helps achieve better alignment as observed by attention heatmap and better phrasal translation and leading to statistically significant improvement in BLEU scores.

## introduction training a machine translation system using only the monolingual corpora of the two languages was successfully demonstrated by @xcite . they train the machine translation system using denoising auto-encoder (dae) and backtranslation (bt) iteratively. recently, pre-training of large language models @xcite @xcite using monolingual corpus is used to initialize the weights of the encoder-decoder models. these encoder-decoder models are later fine-tuned using backtranslated sentences for the task of unsupervised neural machine translation (unmt). while we appreciate language model (lm) pre-training to better initialise the models, it is important to understand the shortcomings of earlier approaches. in this paper, we explore in this direction. we observe that the translation quality of undreamt models @xcite suffers partially due to wrong positioning of the target words in the translated sentence. for many instances, though the reference sentence and its corresponding generated sentence are formed with almost the same set of words, the sequence of words is different resulting in the sentence being ungrammatical and/or loss of meaning. this results in a difference in syntax and semantic rules. we define such generated sentences as scrambled sentences and the problem as scramble translation problem. scrambled sentences can be either disfluent or fluent-but-inadequate. here, if the lm decoder is not learnt well, we observe disfluent translations. if the lm decoder is learnt well, we observe fluent-but-inadequate translations. an example of fluent-but-inadequate translation will be 'leaving better kids for our planet' instead of 'leaving better planet for our kids'. due to this phenomenon, during bleu computation n-gram matching lessens, for @xmath1. however, this error is absent in translation generated from recent state-of-the-art systems @xcite @xcite . we hypothesise, dae introduces uncertainty to the previous unmt @xcite @xcite models, specifically to the encoders. it has been observed that encoders are sensitive to the exact ordering of the input sequence @xcite @xcite . by performing random word-shuffle in all the source sentences, encoder may lose important information about the sentence composition. the dae fails to learn informative representation which affects the decoder resulting in wrong translations generated. if our hypothesis is true, retraining these previous unmt system models with noise-free sentences as input should resolve the problem for previous systems @xcite . moreover, using this retraining strategy will not benefit recent approaches @xcite as they do not shuffle words of input sentence while training with back-translated data. in this paper, we prove our hypothesis by showing that a simple retraining strategy mitigates the 'scrambled translation problem'. we observe consistent improvements in bleu score and word-alignment over the denoising unmt approach by @xcite for four language pairs. we do not wish to beat the state-of-the-art unmt systems with pre-training, instead, we demonstrate a limitation of previous denoising unmt @xcite systems and prove why it happens. ## related work neural machine translation (nmt) @xcite @xcite typically needs lot of parallel data to be trained on. however, parallel data is expensive and rare for many language-pairs. to solve this problem, unsupervised approaches to train machine translation @xcite @xcite was proposed in the literature which uses only monolingual data to train a translation system. @xcite and @xcite introduced denoising-based u-nmt which utilizes cross-lingual embeddings and trains a rnn-based encoder-decoder model @xcite . architecture proposed by @xcite contains a shared encoder and two language-specific decoders while architecture proposed by @xcite contains a shared encoder and a shared decoder. in the approach by @xcite , the training starts with word-by-word translation followed by denoising and backtranslation. here, noise in the input sentences in the form of shuffling of words and deletion of random words from sentences was performed. conneau and lample (2019) (xlm) proposed a two-stage approach for training a unmt system. the pre-training phase involves training of the model on the combined monolingual corpora of the two languages using masked language modelling (mlm) objective @xcite . the pre-trained model is later fine-tuned using denoising auto-encoding objective and backtranslated sentences. @xcite proposed a sequence to sequence pre-training strategy. unlike xlm, the pre-training is performed via masked sequence to sequence (mass) objective. here, random ngrams in the input is masked and the decoder is trained to generate the missing ngrams in the pre-training phase. the pre-trained model is later fine-tuned using backtranslated sentences. @xcite demonstrated that lstm encoders of the nmt system are sensitive to the word-ordering of the source language. they considered the scenario of zero-shot translation from language l 3 to l 2 . they train a nmt system for l 1 → l 2 languages and use l 1 -l 3 languages bilingual embeddings. this enables the trained model to perform zero-shot translation from l 3 → l 2 . however, if the word-order of the languages l 1 and l 3 are different, the translation quality from l 1 -l 3 is hampered. michel and neubig (2018) have also made a similar observation albeit in the monolingual setting. they observe that accuracy of the machine translation system gets adversely affected due to noise in the input sentences. they discuss various sources of noise with one of them being word emission/insertion/repetition or grammatical errors. the lack of robustness to such errors could be attributed to the sequential processing of lstm or transformer encoders. as the encoder processes the input as a sequence and generates encoder representation at each time-step, such errors would lead to bad encoder representations resulting in bad translations generated. similar observations have also been made by @xcite for cross-lingual transfer of dependency parsing. they observe that self-attention encoder with relative position representations is more robust to word-order divergence and enable better cross-lingual transfer for dependency parsing task compared to rnn encoders. here, shuffling is performed by swapping neighboring words l/2 times, where l is the number of words in the sentence. ## baseline approach for completeness, we also experimented with xlm unmt (conneau and lample, 2019) with initialise the model with mlm objective followed by finetuning it with dae and bt iteratively. in this approach, they do not add noise with the input sentence while training with backtranslated data. ## proposed retraining strategy our proposed strategy to train a denoising-based unmt system consists of two phases. in the first phase, we proceed with training using denoised sentences similar to the baseline system @xcite for m number of iterations. adding random shuffling in the input side, however, could introduce uncertainty to the model leading to inconsistent encoder representations. to overcome this, in the second phase, we retrain the model with simple ae and on-the-fly bt using sentences with the correct ordering of words for (n-m) iterations as shown in fig. 2 . here, n is the total number of iterations and @xmath2. more concretely, this training approach consists of 4 more sub-processes other than the 4 subprocesses of the baseline system. these are: (v) ae src : auto-encoding of source sentences in which we train shared-encoder, source-decoder, and attention. (vi) ae trg : auto-encoding of target sentences in which we train shared-encoder, target-decoder, and attention. (vii) bt src : training shared-encoder, target-decoder, and attention with back-translated source sentences as input and actual target sentences as output. (viii) bt trg : training shared-encoder, source-decoder, and attention with back-translated target sentences as input and actual source sentences as output. the second phase ensures that the encoder learns to generate context representation with information about the correct ordering of words. for xlm @xcite , we add these 4 subprocesses only with fine-tuning step. we do not change anything in lm pretraining step. ## experimental setup we test our hypothesis with undreamt as a previous approach and xlm as a sota approach. we applied our retraining strategy on both the approaches and observed the result. for undreamt, we have used monolingual data of six languages, i.e. english (en), french (fr), german (de), spanish (es), hindi (hi), and punjabi (pa). among these languages, hindi and punjabi are of sov word-order where the other four languages are of svo word order. in our experiments, we choose language-pairs such that the word-order of source language matches with that of target language. we have used the newscrawl corpora for en, fr, de of wmt14, and for es of wmt13. for hi-pa, we use wikipedia dumps of the august 2019 snapshot for training. the en-fr and en-de models are tested using wmt14 test-data and en-es models using wmt13 test-data, and hi-pa models using ilci test data @xcite . we have preprocessed the corpus for normalization, tokenization and lowercasing using the scripts available in moses @xcite and indic nlp library @xcite , for bpe segmentation using subword-nmt @xcite with number of merge operations set to 50k. we use the monolingual corpora to independently train the embeddings for each language using skip-gram model of word2vec @xcite . to map embeddings of two languages to a shared space, we use vecmap foot_2 by @xcite . we use undreamt foot_3 tool to train the unmt system proposed by @xcite . we train the baseline model untill convergence and noted the number of steps n required to reach convergence. we now train our proposed system for n/2 steps and re-train the model after removing denoising noise for the remaining n/2 steps. they converge between 500k to 600k steps depending on the language pairs. further details of dataset and network parameters are available in appendix. we also report results on xlm foot_4 approach @xcite . xlm employs two-stage training of unmt model. the pre-training stage trains encoder and decoder with masked language modeling objective. the retraining stage employs denoising along with iterative back-translation. however, xlm uses a different denoising (word shuffle) mechanism compared to @xcite . we replace the denoising mechanism by @xcite with the denoising mechanism used by @xcite . we use the pre-trained models for english-french, english-german, and english-romanian provided by @xcite . we retrain the xlm model until convergence using the denoising approach which makes the baseline system. we later retrain the pre-trained xlm model using our proposed approach where we remove the denoising component after n/2 steps. we report both bleu scores and n-gram bleu scores using multi-bleu.perl of moses. we have tested statistical significance of bleu improvements @xcite . to analyse the systems, we have produced heatmaps of attention generated by the models. ## results and analysis ## conclusion and future work in this paper, we addressed 'scrambled translation problem', a shortcoming of previous denoisingbased unmt approaches like undreamt approach @xcite . we demonstrated that adding shuffling noise to all input sentences is the reason behind it. our simple retraining strategy, i.e. retraining the trained models by removing the denoising component from auto-encoder objective (ae), results in significant improvements in bleu scores for four language pairs. we observe larger improvements in n-gram specific bleu scores for higher value of n indicating better phrasal translations. we also observe robustness of the pre-trained language models to the scrambled translation problem. we would also like to explore applicability of our approach in other ordering-sensitive dae-based tasks.