MUCS @ M ix MT : I ndic T rans-based Machine Translation for H inglish Text (2022)

Abstract: Code-mixing is the phenomena of mixing various linguistic units such as paragraphs, sentences, phrases, words, etc., of one language with that of the other language in any text. This code-mixing is predominantly used by social media users who know more than one language. Processing code-mixed text is challenging because of its characteristics and lack of tools that supports such data. Further, pretrained models can be used for the formal text and not for the informal text such as code-mixed. Developing efficient Machine Translation (MT) systems for code-mixed text is challenging due to lack of code-mixed training data. Further, existing MT systems developed to translate monolingual data are not portable to translate code-mixed text mainly due to its informal nature. To address the MT challenges of code-mixed text, this paper describes the proposed MT models submitted by our team MUCS, to the Code-mixed Machine Translation (MixMT) shared task in the Workshop on Machine Translation (WMT) organized in connection with Empirical models in Natural Language Processing (EMNLP) 2022. This shared has two subtasks: i) subtask 1 - to translate English sentences and their corresponding Hindi translations into Hinglish text and ii) subtask 2 - to translate Hinglish text into English text. The proposed models that translate the code-mixed English text to Hinglish (English-Hindli code-mixed text) and vice-versa, comprises of i) transliterating Hinglish text from Latin to Devanagari script and vice-versa, ii) pseudo translation generation using existing models, and iii) efficient target generation by combining the pseudo translations along with the training data provided by the shared task organizers. The proposed models obtained 5^{th} and 3^{rd} rank with Recall-Oriented Under-study for Gisting Evaluation (ROUGE) scores of 0.35806 and 0.55453 for subtask 1 and subtask 2 respectively.

## introduction in linguistic terms, code-mixing is the practice of switching between two or more languages within or across sentences/words in any text @xcite . due to the widespread use of social media platforms like twitter, facebook, reddit, etc., users are generating more and more code-mixed content. in indian scenario, social media users usually blend english with their mother tongue or local language, for instance, english and hindi, mainly for the technological limitations of computer keyboard or smartphone keypads to enter text in local languages. further, as most of the text processing tasks are developed for handling monolingual and formal text, informal and/or code-mixed text such as hinglish is less explored. as the code-mixed text like hinglish is increasing day by day, many applications such as mt, sentiment analysis, emotion analysis, etc., are also increasing. this has created a great demand for the tools and resources to process code-mixed data. sample hinglish text along with their hindi and english translations are given in in recent years, pre-trained transformer-based language models have become state-of-the-art models for most of the downstream tasks including mt, text classification, text generation, and natural language understanding. to train such models, underlying data is drawn from a sizable monolingual corpus that is available in wikipedia, book corpora, etc. several models like multilingual bidirectional and auto-regressive transformer (mbart) @xcite and multilingual text to text transformer (mt5) @xcite are readily available for many languages. however, due to the scarcity of code-mixed corpus, developing the pretrained language models for code-mixed text is very challenging. mt being one of the important applications of code-mixed texts mainly focuses on translating monolingual text leaving aside the code-mixed data. further, for under-resourced languages with rich morphological features like hindi (sangwan and bhatia, 2021), developing mt models become more challenging in the code-mixed sce- the following is a breakdown of the paper's structure: section 2 contains the related work and the proposed methodology is explained in section 3. section 4 gives the details about experiments and results and the paper concludes in section 5 with future work. ## related work due to the increasing amount of code-mixed text, mt of code-mixed text is gaining attention of the researchers and the description of few of the models developed to translate hinglish text into english text and vice versa are given below: srivastava and singh (2020) manually developed a parallel corpus of 13,738 hinglish sentences and their translations in english with the help of 54 annotators. they proposed a simple tagging approach for tagging each token in a sentence with the language it belongs to and evaluated bing translate (bt) and google translate (gt) models -the two popular mt services using their parallel corpus. among the two models, gt model outperformed with a better bilingual evaluation understudy (bleu) score of 0.153 when compared to that of bt. @xcite manually developed a hinglish-english parallel corpus of 6,096 parallel sentences with the help of 4 human translators. using a language identification technique, they tagged every word in a sentence with the name of the language to which it belongs. they proposed an mt model comprising three steps: i) identifying the matrix language, ii) translation of source text into matrix language, and iii) translation of matrix language into the target language by training bt. considering steps i) and ii) as preprocessing, they obtained considerable translation with bleu score of 25.0. jawahar et al. (2021) created a parallel corpus of 17.8 million english-hinglish sentence pairs by leveraging bilingual word embeddings to translate english text into hinglish text and vice versa. further, they fine-tuned mbart and mt5 -the pretrained text generators using their newly constructed parallel corpus. the mt5 model obtained a better bleu score of 13.95 compared to that of mbart. @xcite proposed an effective fine-tuning of mbart using english-hinglish dataset foot_1 to translate english text to hinglish text and vice versa. they transliterated the hinglish text in latin script to devanagari script to fine-tune the mbart model and obtained bleu scores of 11.86 and 12.22 for hinglish to english and english to hinglish translations respectively. from the literature, it is clear that very few attempts are made to explore english-hinglish codemixed parallel corpus for mt. hence, there is enough space to explore new techniques in this direction. the framework of the proposed model is given in figure 1 and the system descriptions of each subtasks are given below: subtask 1 -in addition to the dataset provided by the organizers 4 for this shared task, monolingual hindi text is collected from the available resources 5 and the further steps used to accomplish the subtask 1 are given below: ## experiments and results the statistics of the dataset provided by the organizers for both the subtasks which are used to build the proposed models are given in en-indic and indic-en models which are trained on samanantar corpus are fine-tuned with the combination of the shared task dataset and pseudo parallel text. exhaustive experiments are carried out to get the best results by tuning the hyperparameters, which control the learning process of en-indic and indic-en models. test set submitted to the organizers of the shared task are evaluated based on rouge score and word error rate (wer). rouge score is calculated based hyperparameters values max-token 1,568 learning rate 0.00003 label smoothing 0.1 optimizer adam dropout 0.2 this paper describes the models submitted by our team -mucs to mixmt 2022 shared task to perform mt from english text and their corresponding hindi translations into hinglish text and from hinglish text to english. the proposed models consist of transliteration and pseudo translation generation followed by fine-tuning the pretrained mt models using the combination of pseudo parallel data and the shared task dataset for target generation. these models obtained rouge scores of 0.35806 and 0.55453 securing 5 th and 3 rd rank for subtask 1 and subtask 2 respectively. the efficient transliteration techniques with effective fine-tuning of the pretrained models for code-mixed hinglish translation will be explored further.