Training Data Augmentation for Code-Mixed Translation (2021)

Abstract: Machine translation of user-generated code-mixed inputs to English is of crucial importance in applications like web search and targeted advertising. We address the scarcity of parallel training data for training such models by designing a strategy of converting existing non-code-mixed parallel data sources to code-mixed parallel data. We present an m-BERT based procedure whose core learnable component is a ternary sequence labeling model, that can be trained with a limited code-mixed corpus alone. We show a 5.8 point increase in BLEU on heavily code-mixed sentences by training a translation model using our data augmentation strategy on an Hindi-English code-mixed translation task.

## introduction code-mixing (cm), the phenomenon of mixing words from two languages in a sentence, is getting increasingly commonplace in several bilingual communities 1 . recently, much research has focused on training language models over codeswitched data for tasks like automatic speech recognition (asr) @xcite . in this paper we focus on the less explored problem of translating code-switched inputs to a high-resource language like english. this task is compelling in applications like web search, targeted advertising, and recommendations, which require matching user-generated code-mixed queries to rich english content. a major challenge in such applications is the lack of parallel data from code-mixed input to english. while years of effort have made available rich parallel datasets for translation, these are mostly over formal sources like news, which tend to be less code-mixed. in this paper we show how to create high-quality parallel data for training a code-mixed translation model by exploiting three 1 https://github.com/gentaiscool/code-switching-papers types of resources: 1) parallel data from non-codemixed sentences to english, 2) code-mixed sentences, and 3) monolingual sentences in english. contributions: (1) we present an mbert @xcite based procedure for converting non-cm parallel data to cm parallel data. the core learnable component of our procedure requires fine-tuning mbert for a three-way sequence labeling task, and can be easily trained using the limited code-switched sentences alone. we apply this model to convert source sentences of the parallel data to code-mixed sentences, while keeping the target english sentences in-tact. we also extend the existing back-translation method of using monolingual target data, with our code-switched augmentation. (2) we experiment on a rich public code-mixed dataset obtained from a literacy promotion project. we show that with our data augmentation strategy the translation bleu improves from 43.9 to 46.4 overall. on sentences that are more heavily code-mixed our accuracy increases by 5.8 bleu points, and on an adversarial test set where the baseline provides poor accuracy we show a 5.4 point bleu increase. (3) we show that our data augmentation strategy improves performance for code-switched test sets while maintaining state of the art performance on non-code-switched inputs. ## related work most prior work on cm has focused on training a language model (lm) in the context of automatic speech recognition. the main challenge addressed in these works is the limited availability of codemixed sentences. @xcite and @xcite propose different methods of training lms for cm sentences without explicitly creating synthetic cm data, but another popular strategy is to first create synthetic cm data and train the lm with such synthetic data. we next summarize existing approaches to generate syn-thetic cm data: @xcite propose to learn switching patterns from code-mixed data using a gan-based adversarial training. @xcite use bert as the generator which is fine-tuned by masking the english words in a cm corpus using gan-based adversarial training. in contrast, ours is a much simpler sequence labeling formulation. @xcite proposes to splice fragments chosen from monolingual corpus of two languages based on statistics of length distribution and phone transition. pratapa et al. (2018) use equivalence constraint theory to define rules on top of parse trees of two sentences to create grammatically valid artificial cm data. samanta et al. (2019) use a variational autoencoder to generate synthetic cs data. winata et al. ( 2019 ) propose a sequence-tosequence model using a copy mechanism that learns when to switch from one language to another. to train their model they depended on highresource commercial translation models for translating the code-mixed input to monolingual sentences in both english and native language. since our goal is to train such a translation model, we did not want to depend on such resources. one key difference is that our goal is translation from code-mixed to english, and not designing a representative lm for code-mixed data. since our final target is english, when designing our data augmentation strategy we give higher priority to preserving the distribution of the target-side (english) than the cm input. ## our approach our strategy is to augment the training data by converting existing non-code-mixed parallel corpus into a parallel corpus with code-mixed source. let l, m, e denote the space of non-code-mixed sentences, code-mixed sentences, and english sentences respectively. we have available a parallel corpus of non-code-mixed and english pairs (l, e) ⊂ (l, e), a code-mixed corpus m ⊂ m, and a monolingual corpus e m ⊂ e. our goal is to train a translation model from code-mixed input to english t : @xmath0. our focus is indic languages such as hindi that are often code-mixed with english. the codemixed corpus in our case contains three kinds of tokens: native tokens in native script e.g. devanagari, english tokens in latin script, and english tokens transliterated to native script. figure 1 shows an dataset en en-trans iitb parallel train 0.021 0.132 code-mixed test (hi) 0.121 0.121 our model is based on mbert, and consists of two phases: the first phase predicts words to switch in a monolingual sentence, and the second phase generates the switched words by harnessing parallel data. we describe these phases next: ## conclusion machine translation of code-mixed inputs to english is an important task for which parallel training data is scarce. we presented a simple mbertbased method of converting existing parallel data into code-mixed parallel data. augmenting existing training data with this synthetic parallel data leads to substantial gains in bleu on heavily codemixed inputs without worsening accuracy on noncode-mixed inputs. however, gains are larger for some language pairs than others. furthermore, code-mixed data from informal sources like twitter presents additional challenges like noisy inputs stemming from non-canonical transliterations, informal language use, and misspellings. our ongoing and future work includes evaluating the model on more languages and handling noisy inputs.