Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed H inglish Text (2021)

Abstract: In this shared task, we seek the participating teams to investigate the factors influencing the quality of the code-mixed text generation systems. We synthetically generate code-mixed Hinglish sentences using two distinct approaches and employ human annotators to rate the generation quality. We propose two subtasks, quality rating prediction and annotators’ disagreement prediction of the synthetic Hinglish dataset. The proposed subtasks will put forward the reasoning and explanation of the factors influencing the quality and human perception of the code-mixed text.

## introduction code-mixing is the phenomenon of mixing words and phrases from multiple languages in a single utterance of a text or speech. figure 1 shows the example code-mixed hinglish sentences generated from the corresponding parallel hindi and english sentences. code-mixed languages are prevalent amongst multilingual communities such as spain, india, and china. with the inflation of social-media platforms in these communities, the availability of code-mixed data is seeking a boom. it has lead to several interesting research avenues for problems in computational linguistics such as language identification @xcite , machine translation @xcite , language modeling @xcite , etc. over the years, we observe various computational linguistic conferences and workshops organizing the shared tasks involving the code-mixed languages. diverse set of problems have been hosted such as sentiment analysis @xcite , offensive language identification @xcite , wordlevel language identification @xcite ; molina et al., 2016), information retrieval (banerjee et al., 2016), etc. despite these overwhelming attempts, the natural language generation (nlg) and evaluation of the code-mixed data remain understudied. the noisy and informal nature of the code-mixed text adds to the complexity of solving and evaluating the various nlg tasks such as summarization and machine translation. these inherent challenges @xcite with the code-mixed data makes the widely popular evaluation metrics like bleu and wer obsolete. various metrics (e.g., cmi @xcite , m-index @xcite , i-index @xcite , burstiness @xcite , memory @xcite , etc.) have been proposed to measure the complexity of code-mixed data, but they fail to capture the linguistic diversity which leads to poorly estimating the quality of code-mixed text @xcite . with this shared task 1 (see section 2 and 4 for the detailed description), we look forward to the new strategies that cater to the broad requirement of the quality evaluation of the generated code-mixed text. these methods will entail various linguistic features encompassing syntax and semantics and the perspectives of human cognition such as writing style, emotion, sentiment, language, and preference. we also put forward a subtask to understand the factors influencing the human disagreement on the quality rating of the generated code-mixed text. this could help design a more robust quality evaluation system for the code-mixed data. ## task overview in this shared task, we propose two subtasks evaluating the quality of the code-mixed hinglish text. first, we propose to predict the quality of hinglish text on a scale of 1-10. we aim to identify the factors influencing the text's quality, which will help build high-quality code-mixed text generation systems. we synthetically generate the hinglish sentences using two different approaches (see section 3) leveraging popular english-hindi parallel corpus. besides, we also have at least two human-generated hinglish sentences corresponding to each parallel sentence. the second subtask aims to predict the disagreement on a scale of 0-9 between the two annotators who have annotated the synthetically generated hinglish sentences. various factors influence this human disagreement, and we seek to investigate the reasoning behind this behavior. ## dataset as outlined in section 1, the code-mixed nlg task observes a scarcity of high-quality datasets. consequently, the quality evaluation of the generated code-mixed text remains unexplored. we propose a new dataset with hinglish sentences generated synthetically and rated by human annotators to address this challenge. we create the dataset in two phases. phase 1: human-generated hinglish sentences: we select the english-hindi parallel sentences from the iit-b parallel corpus @xcite to generate the hinglish sentences. the parallel corpus has 1,561,840 sentence pairs. we randomly select 5,000 sentence pairs, in which the number of tokens in both the sentences is more than five. we employ five human annotators and assign each 1,000 sentence pairs. ## evaluation we use the following three evaluation metrics: ## pilot experiment we conducted a simple pilot experiment with a sota multilingual contextual language model m-bert @xcite . we fine-tune the pretrained m-bert model by adding one hidden-layer neural network on the top. we use the relu activation function, adamw optimizer with 0.03 learning rate, cross-entropy loss, and a batch size of 32. we use the contextual word-embedding corresponding to the synthetic hinglish sentences in the dataset as an input to the model. the architecture remains the same for both subtasks. ## conclusion in contrast to the non-code-mixed text, the noisy and informal nature (e.g., spelling variation, missing punctuation, and language switching) of the code-mixed text makes the quality evaluation more loosely defined. consequently, we need to build models that can effectively gauge the human perception of the quality of the code-mixed text. this shared task will help to build efficient and robust code-mixed text generation and evaluation systems.