niksss at H inglish E val: Language-agnostic BERT -based Contextual Embeddings with Catboost for Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed H inglish Text (2022)

Abstract: This paper describes the system description for the HinglishEval challenge at INLG 2022. The goal of this task was to investigate the factors influencing the quality of the code- mixed text generation system. The task was divided into two subtasks, quality rating prediction and annotators’ disagreement prediction of the synthetic Hinglish dataset. We attempted to solve these tasks using sentence-level embeddings, which are obtained from mean pooling the contextualized word embeddings for all input tokens in our text. We experimented with various classifiers on top of the embeddings produced for respective tasks. Our best-performing system ranked 1st on subtask B and 3rd on subtask A. We make our code available here: https://github.com/nikhilbyte/Hinglish-qEval

## introduction with the increase in popularity of social media platforms like blogs, facebook, and twitter in india, the amount of spoken and written hinglish data has been on the rise. hinglish is a blend of english and hindi, involving code-switching between the above-mentioned languages. due to the increasing number of users, the analysis of this new hybrid language using computational techniques has gotten important in a number of natural language processing applications like machine translation (mt) and speech-to-speech translation. @xcite , @xcite . classical nlp problems such as language modeling @xcite , sentiment analysis (singh and lefever, 2020), @xcite ,hate-speech identification @xcite and language identification @xcite are covered for code-mixed textual data. however, the generation and evaluation aspect of cm data hasn't been explored a lot. this shared task aims to further the research of quality evaluation of the generated code-mixed text in a new way, proposing two tasks that will help quantify the quality of the synthetically generated cm text. moreover, the organizers put forward another task that will help estimate the disagreement between the different human annotators, which further strengthens and reduce the noisiness of the ground-truth quality labels of the generated cm text sequence. ## related work there has been an increased interest in code-mixed data for various nlg tasks. @xcite proposed a new pre-training strategy to tackle the complexities in cm text sequences in a non-traditional way. @xcite talks about generating low-resource code-mixed language from a high resource language such as english using various seq2seq models such as mbart @xcite . other than this, various augmentation techniques were also proposed to improve the quality of generated hinglish text sequences @xcite . due to its high linguistic diversity and lack of standardization, the basic natural language generation needs to be tackled and evaluated differently as shown in @xcite where they propose different metrics to evaluate the quality of generated cm data and show why traditional translation metrics such as blue @xcite etc. cannot capture the quality evaluation properly. ## task overview and dataset the task (srivastava and singh, 2021b) was divided into two subtasks. subtask-a comprised of predicting the quality of the generated hinglish sentences text on a scale of 1-10. 1 is low quality and 10 is the highest quality, considering the semantics and meaningfulness of the generated text sequence. however, the code-mixed language is seldom used in a formal setting, leading the popular evaluation techniques such as blue and wer being inappropriate. the organizers tried to tackle this using another way of evaluation to curb the noisiness of labels occurring in subtask-a by proposing another subtask-b. this subtask tests the capacity of the proposed models for estimating the disagreement between individual annotators, which often occurs when trying to evaluate the quality of informal text sequences. the data for this task introduced in (srivastava and singh, 2021a) is called the hinge dataset. its dataset comprises 3,952 instances. where a particular instance i comprises a text sequence triplet in english, hindi, and hinglish language and average rating as the label for subtask-a and annotator disagreement as the label for subtask-b. these instances were shuffled and divided into three parts in a ratio of 70:10:20, leading to 2766, 395, and 791 data instances in train, validation, and test respectively. an instance of the dataset can be found in figure 1 . ## methodology we attempted these tasks as a text triplet classification problem, wherein we have three text sequences side-by-side and a label attached to them. we analyzed the text sequences and found them to be clean and without any redundant information, hence we didn't perform any traditional pre-processing step. the following steps were taken to build the submitend system: three evaluation metrics f1-score (fs),cohen's kappa (ck),mean squared error (mse) were used to measure the performance of the submitted systems. we present the results obtained on test set along with the baselines in ## conclusion we developed a system to evaluate the quality of machine-generated text sequences using a combination of deep learning feature vectors and machine figure 2: system design learning models. the results are nowhere near what would actually be used to evaluate the quality of the generated sequence. however, this is the first installment of the shared task and it sets off the baselines for future research on the same subject.