H inglish to E nglish Machine Translation using Multilingual Transformers (2021)

Abstract: Code-Mixed language plays a very important role in communication in multilingual societies and with the recent increase in internet users especially in multilingual societies, the usage of such mixed language has also increased. However, the cross translation be- tween the Hinglish Code-Mixed and English and vice-versa has not been explored very extensively. With the recent success of large pretrained language models, we explore the possibility of using multilingual pretrained transformers like mBART and mT5 for exploring one such task of code-mixed Hinglish to English machine translation. Further, we compare our approach with the only baseline over the PHINC dataset and report a significant jump from 15.3 to 29.5 in BLEU scores, a 92.8% improvement over the same dataset.

## introduction code-mixing is the interleaving of tokens from different languages but in the same conversation. code-mixing is a common phenomenon in multilingual societies like india, china, mexico, and in the last decade itself, there has been a massive surge of internet users and specifically from multilingual societies due to the popularity of various social media and messaging platforms. this has lead to a massive surge in mixed language data in the form of comments, conversations, etc. unfortunately due to the informal nature of the code-mixed, it is hard to set a uniformly defined structure. however, linguists have formulated various hypotheses @xcite @xcite and constraints @xcite @xcite that can define a general rule for code-mixing. recent advances in attention-based mechanisms @xcite and transformers @xcite have again shown significant performance improvement and shifted the communities' approach and interest in training larger neural models with deeper architecture. with the rise in large pretrained language models like @xcite , there's been a lot of improvement in natural language processing problems. prior work done in code-mixing like that of @xcite show the effectiveness of large multilingual pretrained language models like mbert @xcite and xlm @xcite on code-mixed data. while gluecos attempts to set glue benchmark using finetuned mbert, @xcite shows the effectiveness of machine translation from a parallel corpus of english and hindi to code-mixed sentence using xlm and pointer-generator model. srivastava and singh (2020); @xcite proposes new task of hinglish code-mixed to english translation and srivastava and singh (2020) collects a new social media code-mixed dataset called phinc consisting of hinglish code-mixed and parallel english sentences. our work attempts to utilize large seq2seq pretrained multilingual transformer based models like mt5 and mbart for the hinglish to english machine translation task. we propose a dual curriculum learning method where first the models are trained for an english to code-mixed translation task and then finetuned again for a code-mixed to english task. there has been very little prior work involving this task and most of the work dealt with the english to code-mixed translation task for synthetic data generation. our translation from codemixed to english shows a significant improvement over the phinc dataset and even beats the baseline presented by the phinc dataset by a margin of 92.8%. the rest of the paper is as follows -section 2 talks about the prior work done in code-mixed machine translation and large multilingual pretrained transformers. section 3 discusses our model and the dataset used for the hinglish to english translation. section 4 addresses our experiments and our qualitative results for our approach. finally, section 5 addresses the conclusive remarks for this paper and presents a direction for future work. ## related work code-mixing refers to the interleaving of words belonging to different languages. this happens predominately in multilingual societies and is increasing rapidly with the increase in internet users on social media and messaging platforms. this has lead to a rapid increase in research interest in recent years and several tasks have been conducted as part of code-switching workshops @xcite . most of the work in these workshops was single nlp problem-specific which were solved using specifically tailored models like language identification @xcite , named entity recognition @xcite , question answering @xcite , parts-of-speech tagging @xcite , and information retrieval @xcite . this was changed by @xcite which introduced a common benchmark for all the tasks using a single finetuned mbert @xcite model downstreamed for all the benchmark tasks. ## system overview in this section, we propose our mbart and mt5 based machine translation model and the dataset used for finetuning the same. ## experiments & results in this section, we describe our experimental setup and our finetuning results for our models described in section 3. ## conclusion our work proposes using large multilingual transformers (mbart and mt5) and demonstrates how finetuning them in a dual curriculum learning method improves the performance of codemixed hinglish to english machine translation tasks. we show a very significant improvement over the phinc baseline by a margin of 92.8% over the bleu scores. as part of the future work, we would like to further improve our machine translation model by using a large amount of synthetic code-mixed data to improve our english to code-mixed translation model, which would further improve our performance on the code-mixed to english translation task. we would also like to extend this work to other low resource and code-mixed languages.