Challenges and Limitations with the Metrics Measuring the Complexity of Code-Mixed Text (2021)

Abstract: Code-mixing is a frequent communication style among multilingual speakers where they mix words and phrases from two different languages in the same utterance of text or speech. Identifying and filtering code-mixed text is a challenging task due to its co-existence with monolingual and noisy text. Over the years, several code-mixing metrics have been extensively used to identify and validate code-mixed text quality. This paper demonstrates several inherent limitations of code-mixing metrics with examples from the already existing datasets that are popularly used across various experiments.

## introduction code-mixing is the phenomenon of mixing words and phrases from multiple languages in the same utterance of a text or speech @xcite . multilingual societies observe a high frequency of code-mixed communication in the informal setting such as social media, online messaging, discussion forums, and online gaming @xcite . various studies indicate the overwhelming growth in the number of code-mixed speakers in various parts of the world, such as india, spain, and china @xcite . the phenomenal increase of the codemixed data on various platforms such as twitter, facebook, whatsapp, reddit, and quora, has led to several interesting research directions such as token-level language identification @xcite , pos tagging @xcite , machine translation @xcite , and question-answering @xcite . despite such active participation from the computational linguistic community in developing tools and resources for the code-mixed languages, we observe many challenges in processing the codemixed data. one of the most compelling problems with the code-mixed data is the co-existence with the noisy and monolingual data. in contrast to the monolingual languages, we do not find any platform where the code-mixed language is the only medium of communication. the co-existing nature of the code-mixed languages with the noisy and monolingual languages posits the fundamental challenge of filtering and identifying the codemixed text relevant for a given study. over the years, various works have employed human annotators for this task. however, employing humans for identifying and filtering the code-mixed text (in addition to the task-specific annotations) is extremely expensive on both fronts of time and cost. also, since code-mixed languages do not follow specific linguistic rules and standards, it becomes increasingly challenging to evaluate human annotations and proficiency. in order to address some of the above challenges, several code-mixing metrics @xcite @xcite have been proposed to measure the degree of code-mixing in the text. however, we observe several limitations in the metric formulations. this paper outlines several such limitations and supports our claims with examples from multiple already existing datasets for various tasks. for illustrations, we choose hinglish (code-mixing of hindi and english language) due to two major reasons: (i) popularity of hinglish and (ii) active research community. @xcite projected that number of hinglish speakers might soon outrun the number of native english speakers in the world. this strengthens our belief that even though hinglish (and other code-mixed languages) does not enjoy the official status, we need to build robust systems to serve the multilingual societies. with the availability of datasets and tools for the hinglish language, we seek a boom in the active participation from the computational linguistic community to address various challenges. ## hinglish: mixing hindi with english hinglish is a portmanteau of hindi and the english language. figure 1 shows example hinglish sentences. also, we see two example sentences in figure 1 that are non-code-mixed but might appear to contain words from two languages. the presence of named entities from the hindi language does not make the sentence code-mixed. sentence 1: ye ek code mixed sentence ka example hai sentence 2 : kal me movie dekhne ja raha hu. how are the reviews? non-code-mixed sentences sentence 1: tendulkar scored more centuries than kohli in delhi. sentence 2: bhartiya janta party won the 2019 general elections. in this study, we explore 10 hinglish datasets encompassing eight different tasks, namely named entity recognition, sarcasm detection, sentiment analysis, language identification, hate-speech detection, machine translation, irony detection, and natural language inference (see ## code-mixing metrics in this section, we describe several popular codemixing metrics that measure the complexity of the code-mixed text. among the following metrics, code-mixing index (cmi, @xcite ; @xcite ) is the most popular metric. ## limitations of code-mixing metrics this section describes various limitations of the existing metrics that measure the complexity of the code-mixed text. as cmi is most popular among code-mixing metrics, it is reported in five @xcite @xcite @xcite out of the 10 datasets listed in data source spelling variations noisy/monolingual readability/semantic singh et al. (2018c) ab boliye teen talak harram h ya nai aapke khud ki lady's chate h ki aap sai dur hona. shame on u again...#tripletalaq #tripletalaq don't post this @bjp4up @narendramodi @amitshah @bjplive @bjpsamvad @bjp4india #notebandi ke baad ab poori swami et al. (2018) shareef wo hai jisay moqa nae milta! #irony resigned: sri lanka cricket aniyin thodar thoalviyinaal therivuk kuluth thalaivar sanath jayasuriya ullitta athan uruppinarhal raajinaamaa kudakudhinge dhuvasthamee? #maldives #politics joshi et al. (2016) nhi ye log apny lie ayeen change karty he ye konsa mulk k lie sochty he har koi apny lie aur apny families k lie politics me he sary chor he #cricket news 6 saal team ki qeyadat karna mare liye izaz hai ab kisi our ko aagay aana chahiye sabiq captain ab de villiers hiii kam chhe patwa et al. (2020) @divyanshmohit @gulbukhari tum apny indian ki fikkar karo pakistan ko hum khud dykh lyngy. mukti bahini 2 nahi ban @bts_army_fin also stade de france is preparing for the concert. looks so beautiful! see their post on instagram https//t.co/owhp now this i too much ab sare tweet arsal ke support me jab jiya ka man nhi and wo chai nhi bana sakti yasit ke liy barman et al. (2014) @liaqat842 tum sahi keh rhy thy yeh zayda buri timings hain 3 wali match ki subah purany office bhi jna hai kaam hai @saadiaafzaal pagl he ye qaom jo misbah ka cmprezm imraan se kr rhe he. khuda ko maano kaha misbah kaha imran.. shoib akhtar @aashikapokharel haha okay. office time, aba bus ma bore hune wala chhu. also, alumni ko imp kaam chha. viber ma aaye hune. :p bohra et al. (2018) gf khoon peene k liye hoti hai aur apne babu ko thana thilane k liye bas mere marnay ki ya hate deni ki? ke karya karta aise hi baithe hai.kal ye ghatna aap or hum dhar et al. (2018) modi ji aap jesa koi nhi dhanywad aap desh ki kitni sewa karte hai jese ak beta apni ma ko poojta hai girna samal nai lage etni lambi speech sa kuch mi hotta sirf 2 word khna or unka suna sa frk atta h........ sekho i love you sallu?? srivastava and singh (2020) unhone pehle pic ni dkhi ti kya tmhari jo milne k baad hi ignore kia tmhe...? kaun hai ye zaleel insaan? @indiantweeter jain ration gap ho jaega. vijay et al. (2018) 35 sal ma koi hospital esa nai banaya jaha khud ka ilaj hosakai. .. irony and then the irony" sab ko jurisakyo lahana le kahile juraucha ? hi vanitha garu hai andi this is irony , arledy rep icharu ga khanuja et al. (2020) 3 kam padey they kasturi is speaking to his son 31 minutes time hua this formulation makes the metric vulnerable to several limitations, such as the bagof-words model and assigning higher metric scores to meaningless sentences that are difficult to read and comprehend. next, we describe four major limitations that combine one or more than one perspective (see limitation perspective bag of words mf code-mixed lid mf, rl misleading score mf, rl, ha high inference time mf, rl, ha ## conclusion and future work in this paper, we extensively discuss the limitations of code-mixing metrics. we explored 10 hinglish datasets for presenting examples to support our claims. overall, we showcase the need for extensive efforts in addressing these limitations. in the future, we plan to develop a robust code-mixing metric that measures the extent of code-mixing and quantifies the readability and grammatical correctness of the text. also, we aim to create a large-scale hinglish dataset with manual token-level language annotation.