Lost in Translation No More: Fine-tuned transformer-based models for C ode M ix to E nglish Machine Translation (2023)

Abstract: Codemixing, the linguistic phenomenon where a speaker alternates between two or more languages within a conversation or even a single utterance, presents a significant challenge for machine translation systems due to its syntactic complexity and contextual nuances. This paper introduces a set of advanced transformerbased models fine-tuned specifically for translating codemixed text to English, more specifically, Hindi-English (colloquially referred to as Hinglish) codemixed text into English. Unlike standard bilingual corpora, codemixed data requires an understanding of the intricacies of grammatical structures and cultural contexts embedded within the language blend. Existing machine translation efforts in codemixed languages have largely been constrained by the paucity of robust datasets and models that can capture the nuanced semantic and syntactic interplay characteristic of such languages. We present a novel dataset PACMAN trans for Hinglish to English machine translation, based on the PACMAN strategy, meticulously curated to represent natural codemixing patterns. Our generic fine-tuned translation models trained on the novel data outperforms current state-of-theart Large Language Models (LLMs) by 38% in terms of BLEU score. Further, when fine-tuned on custom benchmark datasets, our focused dual fine-tuned models surpass the PHINC dataset BLEU score benchmark by 22%. Our comparative analysis illustrates significant improvements in translation quality, showcasing the potential of fine-tuning transformer models in bridging the linguistic divide in codemixed language translation. The success of our models reflects a promising step forward in the quest to provide seamless translation services for the ever-growing multilingual population and the complex linguistic phenomena they generate.

## introduction codemixing, the grammatical fusion of two or more languages within a single utterance or discourse, is a pervasive linguistic practice among bilingual and multilingual communities @xcite . it is a natural outcome of language contact, often observed in societies where speakers are fluent in both a local and a global language. the hindi-english codemixed language, widely known as "hinglish," is one such instance where the syntactic, morphological, and lexical elements of hindi and english are interwoven, giving rise to a rich tapestry of linguistic expression @xcite . machine translation (mt) systems have traditionally been developed for well-defined language pairs with substantial parallel corpora. however, the translation of codemixed text poses unique challenges due to the absence of consistent grammatical rules and the complexities introduced by the informal and spontaneous nature of codemixing @xcite . the machine translation of such codemixed languages is relatively nascent and has been gaining traction with the rise of neural machine translation (nmt) models that are better at handling linguistic ambiguities @xcite . despite the advances in nmt, the translation quality for codemixed languages, particularly for hinglish to english, remains suboptimal. the existing models struggle with capturing the nuanced interplay of linguistic features from both languages and often fail to maintain the semantic integrity of the source text @xcite . recent efforts in this domain have focused on creating more adept systems through innovations in model architecture and data resources @xcite . in this work, we extend the existing body of research by introducing a suite of fine-tuned transformer-based models tailored for the hinglish to english translation task. our approach bene-fits from a novel dataset specifically curated for hinglish, using the pacman strategy @xcite , reflecting various codemixing patterns that are representative of authentic speech and writing in naturally observed contexts. our proposed fine-tuned models outperforms gpt4 by 38% and the sota benchmark in hinglish to english translation task, defined by the phinc dataset @xcite by 22% in terms of bleu score, to set a new standard for the field. ## related work the translation of codemixed text is an emerging field of study within the domain of natural language processing. initial attempts to address the translation of mixed-language text primarily focused on rule-based systems, which quickly proved to be insufficient due to the unpredictable nature of code-switching and mixing @xcite . with the advent of statistical machine translation (smt), researchers began exploring data-driven approaches, although the scarcity of parallel corpora for codemixed languages remained a hindrance @xcite . the paradigm shift towards neural machine translation (nmt) has opened up new avenues for handling the complexities of codemixed language translation. the flexibility of neural networks, particularly the sequence-to-sequence models, has shown promise in capturing the nuances of mixedlanguage syntax (singh and shrivastava, 2018). transformer-based architectures, introduced by @xcite , have revolutionized nmt by enabling models to consider the entire context of the input sequence, which is particularly beneficial for the disambiguation of codemixed text @xcite . in the context of hinglish to english translation, @xcite introduced the phinc dataset, a benchmark for codemixed machine translation. while several works have utilized this dataset, they have often fallen short in adequately handling the linguistic subtleties of hinglish @xcite . our work builds on these foundations and introduces improvements both in terms of the dataset and the transformer model finetuning, leading to significant advancements over the current state-of-the-art. another line of work that intersects with our research is the exploration of pre-trained language models for codemixed language processing. mod-els such as mbert @xcite and xlm-r @xcite have been fine-tuned for various codemixed nlp tasks with encouraging results @xcite . however, their direct application to machine translation for codemixed languages is still an under-researched area, which our study aims to address. ## dataset as already discussed in the preceding sections, our dataset lies at the core of this work. we created a novel custom dataset generated using the pac-man strategy, as originally proposed by @xcite . unlike the authors of the pac-man dataset, we used the samanantar parallel corpus @xcite , which consists of a substantially larger 94 million data samples, for generating our novel english-hinglish parallel corpus. we named the dataset pacman trans . the pacman strategy uses english-hindi parallel sentences to generate hinglish codemix sentences following the matrix language theory @xcite . we apply the same strategy and pick the english source sentence and the resulting hinglish sentence as parallel sentences for our translation task. to ensure data quality and linguistic fidelity for pacman trans , we performed several pre-processing steps, as outlined below: ## experimental setup in several related works discussed in section 2, specifically with the closest body of research @xcite , the authors have fine-tuned pre-trained models like mbart, mt5, etc. on the phinc dataset. since the pacman trans dataset (5.2m samples) is significantly larger (around 45 times) than the phinc dataset (13k samples), we opted to first train domain-agnostic or generic models for our translation task. this stems from the fact that the samanantar dataset spans several domains @xcite . consequently, in order to compare against the phinc benchmark we followed a dual fine-tuning strategy by fine-tuning our domain-agnostic translation models on the phinc dataset. through this strategy, we were able to generate two sets of transformer-based models, general-purpose and focused. we also observed the role of dataset size for the codemix to english translation task, by compiling separate models for varied dataset sizes. ## results and observations the results of the performances of the models proposed in section 4.1 on the evaluation datasets discussed in section 4.3 are shown in ## conclusion in this work, we have presented a comprehensive approach to the challenge of translating codemixed text from hinglish to english. our contributions and achievements are summarized as follows: in conclusion, the advancements presented in this paper not only push the boundaries of machine translation for codemixed languages but also provide insights that could inform the development of more robust nlp systems capable of handling the fluid dynamics of human language. we believe that our contributions will pave the way for more nuanced and effective translation systems, fostering better communication across language barriers.