JU _ NLP at H inglish E val: Quality Evaluation of the Low-Resource Code-Mixed H inglish Text (2022)

Abstract: In this paper we describe a system submitted to the INLG 2022 Generation Challenge (GenChal) on Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text. We implement a Bi-LSTM-based neural network model to predict the Average rating score and Disagreement score of the synthetic Hinglish dataset. In our models, we used word embeddings for English and Hindi data, and one hot encodings for Hinglish data. We achieved a F1 score of 0.11, and mean squared error of 6.0 in the average rating score prediction task. In the task of Disagreement score prediction, we achieve a F1 score of 0.18, and mean squared error of 5.0.

## introduction in india, social media's enduring popularity has resulted in massive amounts of user-generated textual content. during a conversation, multilingual speakers frequently flip between languages. speakers frequently talk in multiple languages, and often transliterate. listeners may not always be able to keep up with the multilingual speakers. that's why we need automated systems for transliterated translations. but we don't have a significant amount of transliterated translation data to train our models. so we might use synthetic data for this purpose. synthetic data has become a common resource for a variety of applications. it may be required because of data unavailability, cost savings, security, or privacy concerns. because synthetic data matches the statistical properties of production data, it can be used to train models, validate models, and evaluate performance. machine learning models have now made it possible to create incredibly fast natural language generating systems by building and training a model. now the next challenge is to evaluate the data which is synthetically generated. in this paper we have introduced an algorithm to check the quality of the generated data. we have proposed a supervised learning model using multiple bi-lstm and dense layers to predict two types of scores (average rating score and disagreement score). in this paper we are using the data from @xcite . this is a transliterated translation verification problem which essentially boils down to a task of document similarity evaluation. document similarity evaluation is a well researched task in nlp. as @xcite suggests, various machine learning techniques, and natural language processing tools can be used for this purpose. linhares @xcite shows us how hybrid models of lstm's can be used for document similarity prediction. some work has also been done in the multilingual senario, as in @xcite . however not much work has been done in transliterated translation verification, and certainly none has been done in the indian domain. @xcite explains the challenges in both generating transliterated translations and evaluating it. ## dataset the phenomena of code-mixing are the mingling of words and phrases from various languages in a single text or spoken utterance. examples of code-mixed hinglish sentences created from parallel hindi and english utterances are shown in fig -1 . in this shared task, there are two subtasks for evaluating the quality of the code-mixed hinglish text in this common task @xcite . in the first sub-task, they proposed using a scale of 110 to determine the quality of hinglish content. they want to figure out what elements influence text quality, so high-quality code-mixed text generating systems can be created. the second sub-task is to predict how much the two annotators who annotated the synthetically generated hinglish sentences differ on a scale of 09. various factors influence human disagreement. the dataset consists of five columns (english, hindi, hinglish, average rating, disagreement). highlish sentences are generated using two rulebased algorithms (i.e., wac and pac). for the two rating columns (average rating & disagreement) each sentence is rated on a scale of 1(low-quality) to 10 (high-quality) by two annotators. the quality of the synthetically generated sentences is calculated by rounding off the average of the two human ratings and using this score (in the range of 1-10) in the average rating column. and the disagreement score is calculated by the absolute difference of the two human ratings as the disagreement score (in the range of 0-9). ## system description we used a sequence of glove embeddings as input for english and hindi sentences. however, for hinglish sentences we used one hot vector as inputs. we fed the english and hindi embeddings to separate bi-lstm's[l-e, l-h], and retrieved sequence output from them. we fed the one hot vector from the hinglish data to a dense layer and received a vector output [d-he]. since one hot vector does not capture the sequential information, we have used a dense layer. we then concatenated these two [l-h-e and d-he] vectors, and passed it through a dense layer to get a final class (score between 1 to 10). we used the same model for both the tasks. please refer to fig2 for complete system architecture. ## training on a total of 2766 training data points, we train the lstm model using the adam optimizer with a batch size of 32. started with loss of 0.1810 & accuracy of 0.9658. in the final epoch loss was 0.0300 & accuracy was 0.9864. in this phase, we validated the input using our developed model. for this phase the total available data was 395. we have validated our model for both average rating as well as disagreement.on 395 data we validated our system to predict average rating for corresponding inputs. please refer to table : 1 for detailed results related to this validation. on 395 data we validated our system to predict disagreement score for corresponding inputs. please refer to table: 2 for detailed results related to this validation. ## test in this phase, our developed model gets tested on test data. for this phase the total available data was 791. model was tested for both average rating as well as disagreement. on 791 data, our system is able to predict average rating for corresponding inputs. please refer to no. of data f1-score cohen's kappa mean squared error 791 0.11582 0.00337 6.00 table: 3 for detailed results related to this validation. on 791 data, our system is able to predict disagreement score for corresponding inputs. please refer to table: 4 for detailed results related to this validation. 6 conclusion for inlg 2022, we created a system to predict the average rating of synthetically generated hinglish no. of data f1-score mean squared error 791 0.18331 5.00