P hrase O ut: A Code Mixed Data Augmentation Method for M ultilingual N eural Machine Tranlsation (2020)

Abstract: Data Augmentation methods for Neural Machine Translation (NMT) such as back- translation (BT) and self-training (ST) are quite popular. In a multilingual NMT system, simply copying monolingual source sentences to the target (Copying) is an effective data augmentation method. Back-translation aug- ments parallel data by translating monolingual sentences in the target side to source language. In this work we propose to use a partial back- translation method in a multilingual setting. Instead of translating the entire monolingual target sentence back into the source language, we replace selected high confidence phrases only and keep the rest of the words in the target language itself. (We call this method PhraseOut). Our experiments on low resource multilingual translation models show that PhraseOut gives reasonable improvements over the existing data augmentation methods.

## introduction data augmentation methods are popular in the field of computer vision. for example, it is a common practice to obtain extra training data by flipping and cropping images. consistency regularization refers to the idea that a model should output the same label to an augmented example as the original example which in turn encourages distributional smoothness in the model @xcite . consistency regularization has been used to obtain state of the art results in automatic speech recognition (s. @xcite by randomly striking out horizontal and vertical portions of speech spectrogram. but for textual data, which is discrete, consistency regularization techniques are not easily applicable since changing a single word could change the entire meaning of a sentence. this work proposes a data augmentation method for textual data which doesn't change its original meaning and encourage consistency regularization. some of the existing approaches to data augmentation in neural machine translation (nmt) are based on word replacements such as word dropout @xcite . a recent approach, termed switchout @xcite claims that randomly replacing words in the source and target sentences by words uniformly sampled from the respective vocabularies can improve neural machine translation. there are also attempts to inject artificial noise in the clean data according to the distribution of types of actual noise to make nmt systems more robust @xcite . another approach to data augmentation is to make use of monolingual data. the most popular example is back-translation @xcite where a reverse translation model is employed to translate large amounts of target monolingual data back into source language. this (noisy) source and original target pair is added as additional parallel data and is shown to be useful while attempting to obtain state of the art results in machine translation @xcite . multilingual nmt whereby sharing a single model between several languages has become a standard paradigm in nmt including industrial applications such as google translate @xcite . multilingual nmt has several advantages over training individual translation models for each language pair including faster inference, enabling zero shot machine translation and superior performance @xcite . a multilingual nmt model is particularly beneficial when training to translate low resource languages @xcite . the main contribution of this work is to show that a phrase based data augmentation strategy consistently provides improvement especially in lowresource language settings. we show that this is also useful in code-mixed translation settings. ## existing data augmentation methods for nmt let (x , y) denote the available parallel corpus. our task is to find augmented parallel sentences (x, ŷ) from the same distribution where the parallel data is sampled from. after acquiring synthetic parallel data, it is appended to the available parallel data. an encoderdecoder based nmt model is trained to maximize the usual mle objective as follows: below we discuss several existing data augmentation techniques to generate synthetic parallel data. copying monolingual data (copying): the monolingual target sentences are copied as the source sentence as well to create a synthetic parallel corpus to train @xcite . this is shown to improve the target side language fluency as it could learn from large amounts of monolingual sentences. this method is suitable for a multi-lingual setting as the encoder has to deal with sentences in more than one language. note that we can combine word dropout with copying. word dropout: dropping words from the source as well as target sentences (with a probability, say 0.1) has shown to improve the performance of nmt systems @xcite . this could help learn the sentence representation better as in a denoising autoencoder @xcite . ## phraseout we propose to use a partial back-translation technique in a multi-lingual setting. we do the augmentation only in the source side and keep the target sentence as it is similar to back-translation. the augmented sentence ŷ is obtained by replacing a randomly chosen phrase from the target sentence y with a source language phrase. hence we hope that both y and ŷ are close in the semantic space and hence the distributional smoothness is maintained. this augmented sentence is copied as the source sentence. phraseout is described in algorithm 1. using the available parallel data, we learn a phrase mapping table. a phrase alignment tool like mgiza foot_0 can be used for this. next we augment target monolingual sentences if phrases in that sentence is present in the phrase table and replace that phrase with the corresponding source phrase. this essentially creates a code mixed source sentence. we hope that this could help bring the word embedding of similar words in different languages to be aligned. the augmented data generated by phrase-out is concatenated with the original parallel data for further training or finetuning. we experiment on many to one multilingual translation from indian languages to english. we use the wat 2018 dataset @xcite for our experiments. it has english translations of several indian languages. we chose hindi (hi), bengali (bn), malayalam (ml), tamil (ta) and telugu (te). the training data size is shown in 1 we use english sentences from the book corpus, a subset of the iit-b dataset @xcite as our monolingual corpus. we use moses tokenizer 2 to tokenize english and indic nlp library 3 for tokenizing indian languages. ## results we provide evaluations that compare several mono lingual augmentation methods, the effect of monolingual data size and the comparison with backtranslation. ## related works recently a few works have explored the utility of code mixed (also called code switched) augmentations. @xcite proposes word replacements as in phraseout for the purpose of lexicon induction. they perform data augmentation on the parallel data and don't use any monolingual data. recently code switched pretraining @xcite ) @xcite has shown to work favorably against popular cross lingual pretraining methods. but all these methods use unsupervised dictionary induction @xcite to obtain parallel word translations. but unsupervised dictionary induction performs quite poorly for distant language pairs such as english to indian languages @xcite . ## conclusion and future work we propose a simple yet useful data augmentation technique suitable for a multilingual nmt setting, called phraseout. our experiments confirm that phraseout is effective in improving the performance of multilingual nmt systems. the improvement in performance could be attributed to better regularization from code mixing. training using code mixed data could be useful for improving the robustness of nmt systems. social media text usually contains code mixed data. one future direction of research is to see how phraseout can improve robustness in this kind of a setting.