Translate and Classify: Improving Sequence Level Classification for E nglish- H indi Code-Mixed Data (2021)

Abstract: Code-mixing is a common phenomenon in multilingual societies around the world and is especially common in social media texts. Traditional NLP systems, usually trained on monolingual corpora, do not perform well on code-mixed texts. Training specialized models for code-switched texts is difficult due to the lack of large-scale datasets. Translating code-mixed data into standard languages like English could improve performance on various code-mixed tasks since we can use transfer learning from state-of-the-art English models for processing the translated data. This paper focuses on two sequence-level classification tasks for English-Hindi code mixed texts, which are part of the GLUECoS benchmark - Natural Language Inference and Sentiment Analysis. We propose using various pre-trained models that have been fine-tuned for similar English-only tasks and have shown state-of-the-art performance. We further fine-tune these models on the translated code-mixed datasets and achieve state-of-the-art performance in both tasks. To translate English-Hindi code-mixed data to English, we use mBART, a pre-trained multilingual sequence-to-sequence model that has shown competitive performance on various low-resource machine translation pairs and has also shown performance gains in languages that were not in its pre-training corpus.

## introduction in the last decade, social media has become a significant part of the lives of a large population in the world. unlike previously popular communication platforms, online messaging is very informal, and in recent years, it has led to an increase in the usage of emojis, slang, and even a hybrid form of language, code-mixed language. code-mixed language is a mixture of multiple languages where words belonging to different languages are interleaved with each other in the same conversation. it is commonly used by multilingual speakers. it does not follow a formally defined structure and often varies from person to person, although some studies @xcite have proposed linguistic constraints on code-switching. code-mixing and code-switching are similar terms that slightly differ technically, but they are often used interchangeably by the research community. we will also be using them interchangeably in our paper. in this paper, we work with english-hindi codemixed data. english-hindi code-mixed language often called hinglish is very common in india because of a large number of bilingual speakers who often use english in their professional lives while using hindi in their personal lives. an example of an english-hindi code-mixed sentence from a dataset released by @xcite is shown below: although there is a large population globally that communicates using code-mixed languages, annotated datasets remain scarce even when the monolingual constituent languages have large-scale datasets. recent work suggests that multilingual models trained on several monolingual datasets perform well with zero-shot cross-lingual transfer in code-switched settings @xcite . however, @xcite conclude that their model had varying performance across tasks and especially struggled with nli and sentiment analysis tasks. another challenge with code-mixed language research is that, unlike monolingual data, there are no formal data sources like news articles or books written in codemixed languages. instead, most research uses informal sources such as social media texts or messages, which are usually challenging to obtain. also, most of the data is written in the roman script, and hindi words are transliterated informally without any standard rules. instead, individuals generally provide a rough phonetic transcription of the intended word, which can vary from individual to individual due to any number of factors, including regional or dialectal differences in pronunciations, differing conventions of transcription, or simple idiosyncrasy @xcite . this makes it challenging to prepare reliable datasets to train robust deep learning models. most of the existing datasets focus on a few language pairs and have been prepared by several shared task organizers. to address these issues, we propose translating the code-mixed data to english (a high-resource language) and applying powerful models trained on english data to perform sequence-level classification tasks on the translated data. to translate the code-mixed data to english, we propose using mbart @xcite , a pre-trained multilingual sequence-to-sequence model. we experiment with our pipeline on two english-hindi code-mixed sequence classification tasks of the gluecos @xcite benchmark -natural language inference and sentiment analysis. we achieve state-of-the-art performance in both tasks. the code for our proposed system is available at https://github.com/ devanshg27/cm_translatify . the main contributions of our work are as follows: the rest of the paper is organized as follows. we discuss prior work related to code-mixed language processing and also discuss work related to machine translation, natural language inference, and sentiment analysis. we describe the translation system we use and show the effect of different training choices. we describe our pipeline for code-mixed sequence level classification tasks on the chosen tasks -natural language inference and sentiment analysis and show its performance against past work. we conclude with a direction for future work and highlight our main findings. ## related work code-mixing occurs when a speaker uses words belonging to different languages interleaved with each other in the same conversation. with the rise of social media and messaging platforms, there has been a significant increase in code-mixed language usage. several shared tasks have been conducted as a part of code-switching workshops @xcite @xcite which were held in notable conferences. these tasks include language identification @xcite , named entity recognition @xcite , information retrieval @xcite @xcite , partof-speech tagging @xcite , sentiment analysis @xcite , and question answering @xcite . although these tasks have helped progress codeswitching language research, most tasks require building specialized systems for the specific task and language pair due to the limited dataset sizes. recently, large pre-trained multilingual models have been used for various code-mixed tasks @xcite . machine translation refers to translating a text from a source language to its counterpart in a target language using machines. it has widespread applications in the real world and has been an active area of research. earlier works in machine translation mostly focused on statistical or rule-based approaches. in contrast, neural machine translation gained popularity in the last decade after kalchbrenner and blunsom (2013) successfully proposed the first dnn model for translation. recent works use transformer-based approaches @xcite . some approaches utilize multilingual pretraining @xcite @xcite ; however, these works focus only on monolingual language pairs. despite the significant usage of english-hindi code-mixing, there has been little work regarding english-hindi code-mixed translation @xcite @xcite , which leads to a massive gap in communication as these texts can only be understood by people who are proficient in both these languages. natural language inference is the task of determining if the given "premise" supports a given "hypothesis" and classifying the hypothesis as true (entailment), false (contradiction), or undetermined (neutral). it is arguably one of the most fundamental tasks in natural language understanding. wang et al. (2018) and yin et al. (2019) suggest that various nlp tasks can be reduced to natural language inference, which makes it an even more valuable task to solve. natural language inference for english texts has been an active area of research. it has been extensively studied under different tasks such as rte (recognizing textual entailment) @xcite , nli (natural language inference) @xcite , fever (fact extraction and verification) @xcite . in recent years, large-scale pre-trained models @xcite @xcite have dominated these tasks and have achieved close-to-human performance. although nli on english data has seen many advances, there has been little work on nli for code-mixed data. @xcite release the first nli dataset for code-mixed languages. it consists of conversations from hindi movies (bollywood) as premises. chakravarthy et al. (2020) compare the effectiveness of various approaches on the dataset. sentiment analysis is the task of understanding the sentiment expressed in the text and classifying the text into positive, negative, or neutral classes. it has several applications such as customer feedback, marketing, and social media monitoring. there has been extensive research on sentiment analysis of english texts with various shared tasks and datasets. sentiment analysis for code-mixed texts is an essential task due to the widespread usage of @xcite al. (2018) srivastava and singh (2020) ## translating code-mixed text in this section, we describe our proposed model, which uses mbart @xcite to translate code-mixed texts to english. ## code-mixed sequence-level classification model architecture dataset(number of samples) #parameters snli multinli fever-nli anli(r1,r2,r3) (570k) (433k) (250k) (170k) (1) (liu et al., 2019) roberta large ∼355m (2) (nie et al., 2020) roberta large ∼355m (3) (nie et al., 2020) xlnet large ∼340m (4) (nie et al., 2020) albert xxlarge ∼223m ## conclusion in this paper, we demonstrate that mbart can be used to translate english-hindi code-mixed sentences to english and show that transfer learning from hindi-english translation improves its performance on code-mixed translation. we evaluate how our translation system can be used to improve performance in code-mixed sequence classification tasks. we develop a pipeline that uses our translation system to translate code-mixed data to english and then uses large-scale pre-trained english models for the downstream tasks. our experiments show that our pipeline achieves state-of-the-art performance on two tasks of the gluecos benchmark -natural language inference and sentiment analysis. the performance of our pipeline shows that improving code-mixed translation can improve the performance of several code-mixed tasks. in future work, we would like to improve our translation system by creating a larger parallel corpus or synthetically generating parallel sentences for data augmentation. we would also like to extend our system to other code-mixing language pairs.