Exploring Text-to-Text Transformers for E nglish to H inglish Machine Translation with Synthetic Code-Mixing (2021)

Abstract: We describe models focused at the understudied problem of translating between monolingual and code-mixed language pairs. More specifically, we offer a wide range of models that convert monolingual English text into Hinglish (code-mixed Hindi and English). Given the recent success of pretrained language models, we also test the utility of two recent Transformer-based encoder-decoder models (i.e., mT5 and mBART) on the task finding both to work well. Given the paucity of training data for code-mixing, we also propose a dependency-free method for generating code-mixed texts from bilingual distributed representations that we exploit for improving language model performance. In particular, armed with this additional data, we adopt a curriculum learning approach where we first finetune the language models on synthetic data then on gold code-mixed data. We find that, although simple, our synthetic code-mixing method is competitive with (and in some cases is even superior to) several standard methods (backtranslation, method based on equivalence constraint theory) under a diverse set of conditions. Our work shows that the mT5 model, finetuned following the curriculum learning procedure, achieves best translation performance (12.67 BLEU). Our models place first in the overall ranking of the English-Hinglish official shared task.

## introduction code-mixing is a phenomenon of mixing two or more languages in speech and text @xcite . code-mixing is prevalent in multilingual societies, where the speakers typically have similar fluency in two or more languages @xcite . for example, hindi, tamil and telugu speakers from india frequently code-mix with english. code-mixing can happen between dialects, for example, modern standard arabic is frequently code-mixed with arabic dialects @xcite . building nlp systems that can handle code-mixing is challenging as the space of valid grammatical and lexical configurations can be large due to presence of syntactic structures from more than one linguistic system @xcite . in this work, we focus on building a machine translation (mt) system that converts a monolingual sequence of words into a code-mixed sequence. more specifically, we focus on translating from english to hindi code-mixed with english (i.e., hinglish). in the literature, work has been done on translating from hinglish into english @xcite . to illustrate both directions, we provide figure 1 . the figure presents sample translation pairs for hinglish to english as well as english to hinglish. the challenges for solving this task include: (i) lack of hindi data in roman script (words highlighted in cyan color), (ii) non-standard spellings (e.g., 'isme' vs 'is me'), (iii) token-level ambiguity across the two languages (e.g., hindi 'main' vs. english 'main'), (iv) non-standard casing (e.g., rotten tomatoes), (v) informal writing style, and (vi) paucity of english-hinglish parallel data. compared with hinglish to english translation, the english to hinglish translation direction is a less studied research problem. english-hinglish translation can have several practical applications. for example, it can be used to create engaging conversational agents that mimic the code-mixing norm of a human user who uses code-mixing. another use of resulting hinglish data would be to create training data for some downstream applications such as token-level language identification. our proposed machine translation system exploits a multilingual text-to-text transformer model along with synthetically generated code-mixed data. more specifically, our system utilizes the state-ofthe-art pre-trained multilingual generative model, mt5 (a multilingual variant of "text-to-text trans-hinglish to english translation @xcite , @xcite ) hinglish: hi there! chat ke liye ready ho? to synthetically generate code-mixed data, we propose a novel lexical substitution method that exploits bilingual word embeddings trained on shuffled context obtained from english-hindi bitext. the method works by replacing select ngrams in english sentences with their hindi counterparts obtained from the bilingual word embedding space. for meaningful comparisons, we also experiment with five different methods to create code-mixed training data: (i) romanization of monolingual hindi from english-hindi parallel data, (ii) paraphrasing of monolingual english from english-hinglish parallel data, (iii) backtranslation of output from the mt5 model trained on english-hinglish parallel data, (iv) adapting social media data containing parallel english-hinglish sentences by removing emoticons, hashtags, mentions, urls @xcite , and (v) code-mixed data generated based on equivalence constraint theory @xcite . we study the impact of different settings (e.g., size of training data, number of paraphrases per input) applicable for most methods on the translation performance. we observe that the mt5 model finetuned on the code-mixed data generated by our proposed method based on bilingual word embeddings followed by finetuning on gold data achieves a bleu score of 12.67 and places us first in the overall ranking for the shared task. overall, our major contributions are as follows: our work involves code-mixed data generation, machine translation involving code-mixed language, and multilingual pretrained generative models. ## shared task the goal of the shared task is to encourage mt involving code-mixing. we focus on translating english to hinglish. a sentence in hinglish may contain english tokens and roman hindi tokens, as shown in figure 1 . the organizers provide 8, 060, 942 and 960 examples for training, validation, and test respectively. ## our approach our approach to the english-hinglish mt task is simple. we first identify the best text-to-text transformer model on the validation set and follow a curriculum learning procedure to finetune the model for the downstream task. the curriculum learning procedure works such that we first finetune the model using synthetic code-mixed data from our generation method, then further finetune on the gold code-mixed data. this training recipe has been explored previously by @xcite and @xcite to build code-mixed language models. curriculum learning itself has been explored previously for different nlp tasks such as parsing @xcite and language modeling @xcite . we now present our proposed method to generate synthetic code-mixed text for a given language pair. for our method, we assume having access to large amounts of bitext from a given pair of languages (lg 1 and lg 2 ) for which we need to generate code-mixed data. let b @xmath0, y i } denote the bitext data, where x i and y i correspond to sentences in lg 1 and lg 2 , respectively. let ngrams(n, x i , y i ) denote the set of unique n-grams in x i and y i . let cumulative-ngrams(n, x i , y i ) = ∪ @xmath1, x i , y i ) denote the cumulative set of unique n-grams in the set of pairs x i and y i . we shuffle the n-grams in the cumulative set and create a "shuffled" code-mixed sentence by concatenating the shuffled set with n-grams separated by a space. for example, let lg 1 denote english and lg 2 denote hindi (assuming roman script for illustration). a sample bitext instance b i can be "i've never seen it" (x i ) and "maine ye kabhi nah dekhi" (y i ). set of unique 1-grams will be {"i've", "never", "seen", "it", "maine", "ye", "kabhi", "nah", "dekhi"} (ngrams(1, x i , y i ), assuming a whitespace tokenizer for simplicity). then, cumulative-ngrams(2, x i , y i ) correspond to {"i've", "never", "seen", "it", "maine", "ye", "kabhi", "nah", "dekhi", "i've never", "never seen", "seen it", "maine ye", "ye kabhi", "kabhi nah", "nah dekhi"}. a shuffled code-mixed sentence can be, "i've ye_kabhi never seen_it seen never_seen it kabhi_nah i've_never maine_ye ye kabhi nah dekhi maine nah_dekhi". we create one shuffled code-mixed sentence per bitext instance, thereby creating a shuffled code-mixed corpus. we train a word2vec model on this shuffled code-mixed corpus to learn embeddings for n-grams in both languages. the resulting word embeddings seem cross-lingually aligned (based on manual inspection), thereby allowing us to do n-gram translation from one language to another language. for example, nearest english neighbor of a hindi 1-gram "nah" can be "never". once the word embeddings are learned, we can create a code-mixed sentence for the given languages: lg 1 and lg 2 . we first find the n-grams in x @xmath2. let num-substitutions denote the number of substitutions performed to convert x i to a code-mixed sentence. we pick one n-gram at a time from the sorted list and replace all occurrences of that n-gram with its top n-gram belonging to language lg 2 based on word embeddings. we continue this substitution process until we exhaust the num-substitutions. for our machine translation task, we assume lg 1 and lg 2 to be english and hindi (native) respectively. 1 we feed the opus corpus foot_1 containing 17.2m english-hindi bitexts (hindi in native script) as input to the algorithm that outputs english-hinglish code-mixed parallel data. ## experiments in this section, we first discuss how we choose a text-to-text transformer model from available models and then introduce our five baseline methods. ## official results in this section, we describe the official test performance obtained by our models. first, we experiment with mt5 model finetuned using promising code-mixing methods identified in our previous experiments (see section 5.3). the best performing baseline method is based on equivalence constraint theory for 100k examples and yields a bleu score of 12.45. for the proposed cmdr method, we experiment not only with the value for num-substitutions, but also the script type. surprisingly, the best combination for our proposed method is based on maximum substitutions of 3, sticking to the original native script, and yields the highest bleu score of 12.67. the variants of our proposed method that romanizes the replacement ngram consistently perform poorly, which confirms our observation that n-gram level romanization is deprived of sentential context and is prone to errors. ## discussion the lessons learned in this shared task can be summarized as follows. similar text-to-text models. off-the-shelf mt5 and mbart models perform similarly, with mt5 being slightly better in our experiments (for english-hinglish mt). a down side of mt5 is that it takes many more epochs than mbart to converge. in the future, it will be interesting to explore recent extensions of mbart 8 , which are already finetuned 8 these are mbart-large-50-many-to-many-mmt, mbart-large-50-one-to-many-mmt, and mbart-large-50-many-to-one-mmt. for multilingual translation. these extensions involve training on english-hindi (native) bitexts, and so can act as an interesting zero-shot translation baseline without further finetuning. they may also serve as a better baseline when finetuned using the curriculum learning approach adopted in our work. code-mixing from distributed representations is useful. our proposed code-mixed data generation method based on bilingual word embeddings can be exploited by mt5 model to achieve the state-of-the-art translation performance, especially when the number of substitutions is high and the script remains in native form. it will be interesting to see the sweet spot for the number of substitutions, as too low value can result in very less code-mixing while too high value can result in more code-mixing along with more noise (possibly grammatically incorrect and unnatural to bilingual speaker). ## conclusion we proposed an mt pipeline for translating between english and hinglish. we test the utility of existing pretrained language models on the task and propose a simple, dependency-free, method for generating synthetic code-mixed text from bilingual distributed representations of words and phrases. comparing our proposed method to five baseline methods, we show that our method achieves competitively. the method results in best translation performance on the shared task blind test data, placing us first in the official competition. in the future, we plan to (i) scale up the size of code-mixed data, (ii) experiment with different domains of english-hindi bitexts such as twitter, (iii) experiment with recent extensions of mbart, and (iv) assess the generalizability of our proposed code-mixing method to other nlp tasks such as question answering and dialogue modeling.