PHINC : A Parallel H inglish Social Media Code-Mixed Corpus for Machine Translation (2020)

Abstract: Code-mixing is the phenomenon of using more than one language in a sentence. In the multilingual communities, it is a very frequently observed pattern of communication on social media platforms. Flexibility to use multiple languages in one text message might help to communicate efficiently with the target audience. But, the noisy user-generated code-mixed text adds to the challenge of processing and understanding natural language to a much larger extent. Machine translation from monolingual source to the target language is a well-studied research problem. Here, we demonstrate that widely popular and sophisticated translation systems such as Google Translate fail at times to translate code-mixed text effectively. To address this challenge, we present a parallel corpus of the 13,738 code-mixed Hindi-English sentences and their corresponding human translation in English. In addition, we also propose a translation pipeline build on top of Google Translate. The evaluation of the proposed pipeline on PHINC demonstrates an increase in the performance of the underlying system. With minimal effort, we can extend the dataset and the proposed approach to other code-mixing language pairs.

## introduction code-mixing is the phenomenon of switching between two or more languages by the speaker in a single sentence of a text or speech. it is a frequently observed pattern of communication in linguistically diverse countries such as india with 23 official languages and 122 major languages. with more than 300 million native speakers each, english and hindi are among the top five most frequently used languages across the world. with the increase in the number of english speakers in hindi speaking communities in india, the popularity *work done during author's stay at iit gandhinagar of hinglish (code-mixing in english-hindi languages) is seeking a boom. @xcite first introduced the word hinglish in 1967. david crystal @xcite projected in 2004 that the number of hinglish speakers may soon outrun the number of native english speakers in the world. other than hinglish, multiple other bilingual code-mixed languages are popular in multilingual communities in india, such as bengali-english, telugu-english, etc. lack of a standard for writing code-mixed text presents several challenges (see section 2 for details) to natural language understanding tasks. due to the source of origin (social media, online gaming, etc.), the code-mixed text is inherently noisy. we frequently observe code-mixing on social media platforms such as twitter, facebook, etc., in contrast to the formal literary sources such as books, poems, and newspapers. we, therefore, use social media platforms like twitter and facebook as the primary data source for our purpose. the recent thrust on user engagement on social media platforms has led to several research directions, particularly in resourceconstraint noisy user-generated content. @xcite discussed the language identification task for the code-mixed data involving bengali-hindi-english. @xcite presented various techniques to identify languages at the token-level for the bengali-english and hindi-english code-mixed corpus. @xcite discussed various techniques to identify the named-entities in the code-mixed hindi-english corpora consisting of 3,638 tweets. @xcite proposed various experiment to identify pos tags of the 1,062 code-mixed hindi-english facebook posts. they collected data from three popular celebrity facebook public pages of mr. amitabh bachchan (well-known actor), mr. shahrukh khan (well-known actor), and mr. narendra modi (current indian prime minister). besides, they leverage the bbc hindi news articles. @xcite presented a rule-based machine translation system to translate the code-mixed hindi-english sentence to monolingual hindi and english forms. @xcite presented an evaluation benchmark for the two code-mixed language pairs (english-hindi and english-spanish). the proposed evaluation benchmark has six nlp tasks, i.e., language identification, pos tagging, named entity recognition, sentiment analysis, question answering, and natural language inference. these tasks have been part of the recently shared tasks co-located with various nlp conferences or the latest research works. even so, it presents two significant challenges and opportunities. first, most of the datasets available for various tasks are significantly less extensive to build robust standalone systems. second, the comparatively less studied task for the code-mixed machine translation presents an opportunity to build datasets and translation systems. @xcite propose a machine translation augmentation pipeline to use on top of the standard machine translation systems. they also create a parallel corpus of 6,096 english-hindi codemixed sentences and their corresponding translation in english. in this paper, we present a good quality largescale parallel corpus 1 for code-mixed english-hindi noisy social media text messages. the main contributions are: hindi-english code-mixed sentences and their corresponding english translations by the human annotators. ## code-mixing and challenges in machine translation code-mixing is the informal style of communication where words from two (in general) or more languages are part of the same utterance of a text or speech. an example code-mixed 1 https://bit.ly/36qkfkn hinglish sentence is, "hamare paas fully autonomous vaahan hai". this style of writing presents several challenges to almost all monolingual natural language processing tasks such as sentiment analysis, pos tagging, dependency parsing, etc. the widely-used machine translation systems, e.g., google translate, bing translate, etc., perform reasonably well on the monolingual translation task, but they fail to perform well on the code-mixed data (see section 4 for details). we identify six potential causes for the failure of the standard machine translations systems on the codemixed text are: similarly, hashtags are often used on social media platforms, and code-mixed hashtags make it challenging to identify the boundaries of code-switching. in translation, the unrecognized code-mixed entity might make the translation semantically incorrect. figure 1 shows three example code-mixed hinglish sentences and the corresponding translations by google translate and the human annotator. in all the examples, we observe various associated challenges (c1 through c6) with an effective translation by google translate. we posit that the above challenges can be addressed to a large extent with the higher availability of a good-quality, manually annotated parallel corpus. however, as discussed in the previous section, the only available codemixed hinglish dataset @xcite is significantly small and less topically diverse. some of the major differences with the previous work @xcite ('pw', hereafter) are: example i sentence: phone ka wallpaper dekhte dekhte zindagi kat rahi hai. google translation: life is cut off while watching the wallpaper of the phone. human translation: i'm spending my life seeing my phones wallpaper associated challenges: c4 and c6 example ii sentence: is shaher ko ye hua kya hai.. kahi rakh hai to kahi dhua dhua.. play interrupted due to bad weather google translation: what has happened to this city .. if there is smoke somewhere, then smoke somewhere .. play interrupted payable then bad weather human translation: what has happened to this city. there is ash and smoke everywhere. play interrupted due to bad weather associated challenges: c1, c4, and c5 example iii sentence: bhai iit wale hai pehle relationship toh bane laundon ki, break up par nacha rahe ho. google translation: brother-in-law is the first relationship to be made of laundries, you are dancing on the brake sub. human translation: brother, you are an iitian. first get in to a relation. then you can worry about break up. associated challenges: c1, c2, c3, and c4 figure 1: comparison of translation of code-mixed sentences by google translate and human annotators. the ineffective translation by google translate has various associated challenges. ## dataset in india, hinglish is a commonly observed pattern of communication on various platforms such as social media, online gaming, product reviews, discussion forums, etc. as outlined in the previous sections, multiple works have explored the various nuances of the code-mixed hinglish text, such as language identification, sentiment analysis, etc. however, curating the code-mixed hinglish dataset for these tasks requires a significant amount of human efforts due to the identification and filtering of noise from the useful content. in this work, we initially curated hinglish sentences from six already existing works ( @xcite , @xcite , @xcite , (barman et al., 2014), (vrishank shete and mittal, 2016), and @xcite ). one major advantage of using these datasets is the availability of high-quality code-mixed sentences without considerable manual filtering. also, it offers diversity in terms of the source of the data collection as the major social networking platforms (twitter and facebook) are present. additionally, the proposed curation process mitigates the topical bias, as we consider multiple topics in social-media discussions. ## evaluation of machine translation systems here, we demonstrate the performance of the widely used machine translation systems on the code-mixed text. we experiment with two popular machine translation systems (google translate and bing translate) and evaluate their performance on our proposed corpus. we use three different metrics to evaluate system performance. bilingual evaluation understudy (bleu) @xcite , word error rate (wer), and translation error rate (ter). the values of these three metrics lie between 0 and 1. to the best of our knowledge, we do not find translation systems build especially for the code-mixed sentences. the majority of the machine translation systems perform well for the monolingual translation tasks. however, these systems demonstrate severe limitations in translating code-mixed text. for the codemixed text, the current machine translation systems assume input text to be in a single source language. next, we describe the two translation systems and our proposed approach. figure 7 shows example translations of codemixed sentences from two machine translation systems namely, bt and gt , and our proposed approach p p gt . in p p gt , we maintain the original order of the chunks as that of the code-mixed sentence while translating. for instance, the order of the chunks in example ii in figure 7 is additionally, we randomly sample 100 codemixed sentences from the corpus. we use human translated sentences as reference. as most of the machine translation systems do not perform well on the code-mixed data, we can build augmentation pipelines, similar to p p gt , on top of these systems that can preprocess and enhance the quality of the input to these systems. we posit that these pipelines can significantly address the challenges to code-mixed machine translation, as outlined in section 2. ## limitations and opportunities the data collection, preprocessing, annotation, and resource expansion of p hin c presents several limitations and opportunities. some of the major insights and the future research prospects of the proposed dataset are: example i code-mixed sentence: @prankoholic tumko matlab kya time hai din ka, kuch samaj nahi aata na type-i chunks: [tumko matlab kya time hai din ka, kuch samaj nahi aata na] type-ii chunks: [@prankoholic] english translation using bt: @prankoholic what time do you mean of the day, some society does not come. english translation using gt: @prankoholic you mean what is the time of day, don't understand anything english translation using ppgt: @prankoholic do you mean what is the time of day, no sense example ii code-mixed sentence: par if its possible and any other guest needs a room , mera room de de kisi ko bhi type-i chunks: [par], [mera room de de kisi ko bhi] type-ii chunks: [if its possible and any other guest needs a room ,] english translation using bt: on if its possible egg any other guest needs coming room , my room day to anyone english translation using gt: par if its possible and any other guest needs a room , mera room de de kisi ko bhi english translation using ppgt: par if its possible and any other guest needs a room , give my room to anyone example iii code-mixed sentence: ab voh bola jisne kisi bhi party ko support karne se mana kardiya tha . . a flop show annaji type-i chunks: [ab voh bola jisne kisi bhi party ko support karne se mana kardiya tha] type-ii chunks: [. . a flop show annaji] english translation using bt: now woh spoke , which was considered to support any party . . come flop show annaji english translation using gt: now say that he had a desire to support any party. . a flop show anna english translation using ppgt: now speak that who had refused to support any party . . a flop show annaji various learning paradigms such as semisupervised learning, active learning, etc. ## conclusion and future work in this paper, we present a parallel corpus for the english-hindi code-mixed machine translation task. we discuss various challenges in understanding and processing code-mixed text for various natural language understanding tasks. we also show limitations of the widely popular machine translation system build for monolingual corpus in dealing with code-mixed corpora. we evaluate the performance of the various translation systems on our parallel corpus. we present a translation pipeline that outperforms the various translation systems on our proposed code-mixed p hin c dataset, demonstrating the opportunities in building efficient translation systems. in the future, we plan to explore other codemixed languages, especially those that are lowresource and endangered. we also plan to extend the corpus for various other code-mixing tasks such as word-embedding, language identification, named-entity recognition, etc. in addition, we can extend the dataset with more annotation using semi-supervised techniques. as the dataset size is significantly small to train a traditional supervised neural machine translation system, we can build the translation systems using few-shots learning techniques.