BITS Pilani at H inglish E val: Quality Evaluation for Code-Mixed H inglish Text Using Transformers (2022)

Abstract: Code-Mixed text data consists of sentences having words or phrases from more than one language. Most multi-lingual communities worldwide communicate using multiple languages, with English usually one of them. Hinglish is a Code-Mixed text composed of Hindi and English but written in Roman script. This paper aims to determine the factors influencing the quality of Code-Mixed text data generated by the system. For the HinglishEval task, the proposed model uses multilingual BERT to find the similarity between synthetically generated and human-generated sentences to predict the quality of synthetically generated Hinglish sentences.

## introduction the term "code-mixing" refers to mixing words or phrases from different languages into a single text or speech utterance. it embeds linguistic units from one language, such as phrases, words, and morphemes, into an utterance from another language. an example of the code-mixed data can be seen in the figure 1 @xcite . in countries where bilingualism is a common practice, we often see people naturally switching between the two languages. a significant challenge to research is that there are no formal sources like books or news articles in code-mixed languages, and studies have to rely on sources like twitter or messaging platforms. generating and evaluating the available or produced data without a baseline is primarily reliant on people who are fluent in both languages. furthermore, present language models are ineffective in code-mixed situations, where morphemes, words, and phrases from one language are embedded in the other. as code-mixing has long been a way of communication in a multi-cultural, multi-lingual society, the next generation of ai bots should be able to understand code-mixed text. the inherent challenges with the code-mixed data make the widely popular evaluation metrics like bleu (bilingual evaluation understudy score) and wer (word error rate) less effective. with the given task, the main objective is to propose and develop new strategies that address the overall need for quality evaluation of the generated code-mixed text. this paper aims to assess the quality of the generated hinglish text. the proposed model uses the transformer-based multi-lingual bert @xcite to obtain the embeddings of the hindi, english, and hinglish text. a similarity score is computed between synthetically generated hindi and english text and human-generated hindi and english sentences. this score will serve as the sentence's agreement or disagreement factor. 2 related work developing methodologies and resources for diverse natural language processing applications incorporating multi-lingual and code-mixed languages has recently become popular. few of them are word-embedding (chen and cardie, 2018), question answering system (raghavi et al., 2015; kumari et al., 2022), code-mixed text generation @xcite and code-mixed language modeling @xcite . various techniques were used to generate the code-mixed data, including matrix frame language theory, semi-supervised approach (gupta et al., 2020), using dependency parsing @xcite , equivalent constraint theory @xcite and generative adversarial networks @xcite . a metric independent evaluation pipeline (mipe) @xcite considerably enhances the correlation among evaluation measures and human assessments of the generated code-mixed data. in the candidate's hinglish phrase, mipe minimises spelling differences and language switches for evaluation. based on the significance of the terms missing from the candidate hinglish sentence, deduct credit from the evaluation score. by arranging the candidates as well as the reference sentences into the phrases and using the paraphrasing ability, it also deals with the issue of having a restricted amount of reference sentences. ## dataset hinge @xcite , a recently proposed dataset, is used for the hinglisheval task. the dataset contains 1976 english-hindi sentence pairs and corresponding synthetic and humangenerated hinglish sentences. human generated sentences: each english-hindi sentence pair has at least two hinglish sentences, with a total of 6694 such sentences. synthetically generated sentences: using two different algorithms(wac and pac), synthetic hinglish sentences are generated for each english-hindi sentence pair. each sentence is then given a quality rating by two human annotators. there are 2766 such sentences. for each sentence, the rounded off average of the two quality ratings is provided under the label of 'average rating' and the absolute difference of their scores is provided as 'disagreement'. an instance of the hinge dataset @xcite ) can be seen in ## proposed model the proposed approach used a two-step procedure for both the rating and disagreement prediction tasks. the first step is to fine-tune multi-lingual bert, a language model that has been pre-trained on 104 languages and is used to classify and evaluate disagreements further on. the second step used the deep semantic features obtained from multilingual bert for various phrase categories to train a classifier neural network. ## conclusion and future work this paper used a two-step approach to solve the text classification problem. for each pair of phrase types, deep semantic text features were initially extracted using multilingual bert as cls vectors. these vectors were then properly combined, and processed by a fully-connected classifier neural network. the results suggest the proposed model is useful and that the obtained results can be greatly improved by fine-tuning and training with larger data, which could be a future research direction.