P re C og IIITH at H inglish E val : Leveraging Code-Mixing Metrics & Language Model Embeddings To Estimate Code-Mix Quality (2022)

Abstract: Code-Mixing is a phenomenon of mixing two or more languages in a speech event and is prevalent in multilingual societies. Given the low-resource nature of Code-Mixing, machine generation of code-mixed text is a prevalent approach for data augmentation. However, evaluating the quality of such machine gen- erated code-mixed text is an open problem. In our submission to HinglishEval, a shared- task collocated with INLG2022, we attempt to build models factors that impact the quality of synthetically generated code-mix text by pre- dicting ratings for code-mix quality. Hingli- shEval Shared Task consists of two sub-tasks - a) Quality rating prediction); b) Disagree- ment prediction. We leverage popular code- mixed metrics and embeddings of multilin- gual large language models (MLLMs) as fea- tures, and train task specific MLP regression models. Our approach could not beat the baseline results. However, for Subtask-A our team ranked a close second on F-1 and Co- hen’s Kappa Score measures and first for Mean Squared Error measure. For Subtask-B our ap- proach ranked third for F1 score, and first for Mean Squared Error measure. Code of our submission can be accessed here.

## introduction code-mixing foot_0 is a phenomenon where linguistic units from two or more languages are interspersed in a single utterance or a speech event and is common in multilingual communities. due to increased penetration of the internet and social media, codemixing has become common and, at the same time, has posed challenges to automatic text processing pipelines @xcite . one such challenge is the dearth of naturally occurring codemixed data. data constraints have been the primary motive for researchers to leverage data augmentation and construct synthetically generated codemixed corpora using monolingual parallel data as input. synthetic code-mixed data generation, using monolingual parallel corpora, is a non-trivial generation task. in the generated code-mixed sentence, one has to be careful about both the adequacy (preserving semantic content of monolingual sentence) and fluency (grammatical correctness). the task is further obscured by the fact that there is no single way of writing a code-mixed sentence. in this work, we describe our approaches implemented in our submission to hinglisheval, a shared task co-located at inlg2022. hinglisheval is based on the hinge dataset @xcite . hinge is created in two phases: a) human-generated hinglish sentences: at least two hinglish sentences corresponding to the 1,976 english-hindi sentence pairs; b) synthetic hinglish sentence generation and quality evaluation: generate hinglish sentences using two rule-based algorithms, with human annotations for quality rating for each synthetically generated sentence. two annotators rated each sentence on a scale of 1(lowquality) to 10 (high-quality). hinglisheval @xcite , consists of two subtasks -a) subtask-1 quality rating prediction: for each synthetically generated sentence, predict the average (rounded-off) quality rating; b) subtask-2 disagreement prediction: predict the disagreement score (absolute difference between the two human ratings) for the synthetically generated sentence. in our approach, we use a combination of codemixed metrics and language model embeddings as features and train an mlp regressor for both the tasks. rest of the paper is organised as follows: section 2 describes the features and models in detail as well as the experimental setup; section 3 covers the results of our experimentation; and we end with section 4 discussing the implications, limitations, and future work. ## system overview figure 1 shows the system architecture of our submission for both the sub-tasks. we describe the pre-processing involved, methodology for feature computation, and model architecture in subsequent sub-sections. ## results the scores for the sub-tasks achieved by our model are given in for sub-task 2, we beat the baseline model and achieved rank three on the leaderboard for f1-score and one for mean squared error (tied with the models achieving ranks 1 and 2). for sub-task 1, our system is closest to the baseline model, as none of the competing models would beat the baseline model's performance. we hypothesize that the low-performance scores can be attributed to the task's hardness and the data's size. as noted earlier, a comprehensive and structured hyper-parameter search will likely improve the results. because of the very low delta between our system and the best performing system, hyperparameter tuning could be crucial to surpassing the baseline models. ## discussion in this work, we propose a system to predict the quality and disagreement scores given code-mixed sentences and their monolingual counterparts. we leverage the combination of code-mixing metrics and mllms embeddings as features and train mlp regressor models. while our approach fails to beat the baseline/best performing system, the performance of our system is a close second or third and ranks first on mse for both sub-tasks. further hyper-parameter tuning can further improve the results. even the best-performing systems/baselines have very low scores across performance measures, which can be attributed to the difficulty of the task at hand, and the subjectivity of annotators while rating a sentence on a scale of 1-10. the size of the dataset could also be a limitation for solving the task at hand. while mlp-based regressors are black boxes, having an explainable/interpretable model could help rank the features that impact the scores. in our system, an ablation study could help prune the feature space and identify the kind of features that are useful in rating prediction, and such features could be augmented. we leave these pursuits as part of our future work.