H inglish E val Generation Challenge on Quality Estimation of Synthetic Code-Mixed Text: Overview and Results (2022)

Abstract: We hosted a shared task to investigate the factors influencing the quality of the code- mixed text generation systems. The teams experimented with two systems that gener- ate synthetic code-mixed Hinglish sentences. They also experimented with human ratings that evaluate the generation quality of the two systems. The first-of-its-kind, proposed sub- tasks, (i) quality rating prediction and (ii) an- notators’ disagreement prediction of the syn- thetic Hinglish dataset made the shared task quite popular among the multilingual research community. A total of 46 participants com- prising 23 teams from 18 institutions reg- istered for this shared task. The detailed description of the task and the leaderboard is available at https://codalab.lisn.upsaclay.fr/competitions/1688 .

## introduction code-mixing is the phenomenon of mixing words and phrases from multiple languages in a single utterance of a text or speech. figure 1 shows the example code-mixed hinglish sentences generated from the corresponding parallel hindi and english sentences. code-mixed languages are prevalent in multilingual communities such as spain, india, and china. with the inflation of social-media platforms in these communities, the availability of code-mixed data is seeking a boom. it has lead to several interesting research avenues for problems in computational linguistics such as language identification @xcite , machine translation @xcite , language modeling @xcite , etc. over the years, we have observed various computational linguistic conferences and workshops organizing the shared tasks involving code-mixed languages. diverse set of problems have been hosted such as sentiment analysis (chakravarthi example i hinglish: ye ek code mixed sentence ka example hai english : this is an example code-mixed sentence example ii hinglish : kal me movie dekhne ja raha hu. how are the reviews? english: i am going to watch the movie tomorrow. how are the reviews? figure 1 : example parallel hinglish and english sentences. the code-mixed hinglish sentences contain words from hindi and english languages. @xcite , offensive language identification @xcite , wordlevel language identification @xcite , information retrieval @xcite , etc. despite these overwhelming attempts, the natural language generation (nlg) and evaluation of the code-mixed data remain understudied. the noisy and informal nature of the code-mixed text adds to the complexity of solving and evaluating the various nlg tasks such as summarization and machine translation. these inherent challenges @xcite with the code-mixed data makes the widely popular evaluation metrics like bleu and wer obsolete. various metrics (e.g., cmi @xcite , m-index @xcite , i-index @xcite , burstiness @xcite , memory @xcite , etc.) have been proposed to measure the complexity of code-mixed data, but they fail to capture the linguistic diversity which leads to poorly estimating the quality of code-mixed text @xcite . in this shared task 1 (see section 2) for the de-tailed description), we look forward to the new strategies that cater to the broad requirement of the quality evaluation of the generated code-mixed text. these methods will entail various linguistic features encompassing syntax and semantics and the perspectives of human cognition such as writing style, emotion, sentiment, language, and preference. we also put forward a subtask to understand the factors influencing the human disagreement on the quality rating of the generated code-mixed text. this could help design a more robust quality evaluation system for the code-mixed data. a total of 46 participants comprising 23 teams from 18 institutions registered for this shared task. out of which four teams have submitted their final reports. section 3 presents the overview of the participants and submission methodology. section 4 compares the four submissions and presents discussions around the similarity and differences of the approaches. we conclude and present future directions in section 5. ## the hinglisheval shared task in this shared task, we propose two subtasks to evaluate the quality of the code-mixed hinglish text. first, we propose to predict the quality of hinglish text on a scale of 1-10. we aim to identify the factors influencing the text's quality, which will help build high-quality code-mixed text generation systems. we synthetically generate the hinglish sentences using two different approaches (see section 2.1) leveraging popular english-hindi parallel corpus. besides, we also have at least two human-generated hinglish sentences corresponding to each parallel sentence. the second subtask aims to predict the disagreement on a scale of 0-9 between the two annotators who have annotated the synthetically generated hinglish sentences. various factors influence this human disagreement, and we seek to investigate the reasoning behind this behavior. ## overview of participants and submissions in total, 46 participants grouped in 18 teams have registered for the shared task. this includes teams from top us universities like stanford university and carnegie mellon university, companies like tencent qq, and top indian universities like iisc, iitk, and iitbhu. out of 18 teams, nine and eight teams have submitted at least once during train/validation and test phase, respectively. we requested all teams that submitted the test scores to submit the paper illustrating the methodology. out of eight teams, we received papers from four teams listed below: next, we discuss each of the submissions in detail: ## results and discussion in this section, we compare the four submissions for both sub-tasks. the four teams have leveraged large-scale language models (xlm-r, labse or bert). the 5 https://codalab.lisn.upsaclay.fr/ competitions/1688#results models were either finetuned or used for generating embeddings. the embeddings were passed to a classifier model for final predictions. subtask 2 showcases significant improvements over baseline scores. ## conclusion and future directions in this shared task, the participating teams have created systems to evaluate the quality of the codemixed text. these systems can help develop futuristic nlp tools that filter out noisy poor, quality code-mixed text from the good quality code-mixed text. we also proposed several research questions that need to be answered implicitly with the experiments. however, none of the team has answered these questions. we plan to explore these questions in our future editions. overall, this shared task will help the code-mixing research community build efficient and robust code-mixed text generation and evaluation systems.