NICT at M ix MT 2022: Synthetic Code-Mixed Pre-training and Multi-way Fine-tuning for H inglish– E nglish Translation (2022)

Abstract: In this paper, we describe our submission to the Code-mixed Machine Translation (MixMT) shared task. In MixMT, the objective is to translate Hinglish to English and vice versa. For our submissions, we focused on code-mixed pre-training and multi-way fine-tuning. Our submissions achieved rank 4 in terms of automatic evaluation score. For Hinglish to English translation, our submission achieved rank 4 as well.

## introduction code-mixed translation is the task of translation involving code-mixed languages. a code-mixed language is one which combines words as well as grammar of two or more languages. code-mixed translation is difficult because of the lack of training data for the same despite its ubiquitous usage. one widely used code-mixed language is hinglish which combines hindi and english. hinglish sentences are typically constructed either by replacing some hindi words or phrases with english ones in a hindi sentence or vice versa. sometimes, a sentence starts off in one language but ends in another. there are also complex cases where the grammatical structures of both languages are melded into one. hinglish is typically written in roman letters, although there are cases when it is written in devanagari. in this paper we describe our submissions to the mixmt task which involves hinglish to english and english/hindi to hinglish translation. the main challenge of this task is that the parallel corpus available for training models is rather scarce. the total amount of clean, non-synthetic data available for mixmt is around 18,000 examples for both directions. therefore, we have no choice but to rely on external sources of data, and use them to pre-train models. in our case, we leverage a large amount of hindi-english parallel data and synthesize pseudo hinglish data. to do this, perform word alignment on the hindi-english data and then replace random english phrases with aligned hindi phrases. we then use the synthetic hinglish-english parallel data for pre-training. the pretrained model is then fine-tuned to train a joint bidirectional hinglish-english translation model. according to the automatic evaluation metrics, we obtain 4th rank and on human evaluation of hinglish to english translation, we also obtain 4th rank. unfortunately, for translation into hinglish our system ends up copying the english inputs as outputs. although automatic evaluation scores for this are reasonably high, their human evaluation scores are lowest since the sentences are not hinglish at all. ## related work work on code-mixed machine translation is relatively new, especially for hinglish. two important works in this regard are hinge @xcite which proposes a dataset for english/hindi to hinglish translation and phinc (srivastava and singh, 2020) which proposes a dataset for hinglish to english translation. the hinge dataset contains natural as well as human rated synthetic examples in both hindi and english as source languages. having two sources is expected to help in hinglish generation, as the model will have the advantage of contexts from both sources. in our case, we did not leverage both sources and focused only on english. on the other hand, phinc is designed for hinglish to english translation and is much larger than hinge. neither of these datasets are perfect and contain some noisy examples, but the lack of other datasets leaves us with no choice. due to lack of code-mixed data, it is natural to consider synthetic code-mixed data creation where @xcite show that leveraging an xlm model (conneau and lample, 2019) and linguistic features can help generate high quality codemixed sentences. however, we opted for a quicker way using word alignment and phrase substitution approach. using pre-trained models, can be very helpful in code-mixed translation as they are able to represent them effectively @xcite . @xcite have shown that pre-trained models @xcite are highly effective, but we focused more on using our own models trained on our synthetic data. apart from machine translation, code-mix hinglish has been reasonably explored for natural language understanding tasks, particularly for sentiment analysis. we refer interested readers to the following works: baroi et al. (2020); singh and lefever (2020); mathur et al. (2018); bhange and kasliwal (2020) . ## methods we describe the synthetic code mixed pre-training and multi-way fine-tuning approaches we used for our submissions. ## experiments we describe our experiments in our submissions. ## conclusion in this paper, we have described our submission to the mixmt shared task at wmt 2022. we have used a combination of synthetic hinglish-english parallel data creation, pre-training and fine-tuning to obtain our submissions which ranked 4th. our analyses reveal that our english to hinglish translation model actually ended up copying the english sentence as target. we will investigate and fix this in the future.