MIPE : A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation (2021)

Abstract: Code-mixing is a phenomenon of mixing words and phrases from two or more languages in a single utterance of speech and text. Due to the high linguistic diversity, code-mixing presents several challenges in evaluating standard natural language generation (NLG) tasks. Various widely popular metrics perform poorly with the code-mixed NLG tasks. To address this challenge, we present a metric in- dependent evaluation pipeline MIPE that significantly improves the correlation between evaluation metrics and human judgments on the generated code-mixed text. As a use case, we demonstrate the performance of MIPE on the machine-generated Hinglish (code-mixing of Hindi and English languages) sentences from the HinGE corpus. We can extend the proposed evaluation strategy to other code-mixed language pairs, NLG tasks, and evaluation metrics with minimal to no effort.

## introduction code-mixing (hereafter 'cm') is a commonly observed communication pattern for a multilingual speaker to mix words and phrases from multiple languages. cm is widespread across various language pairs across the globe, such as spanish-english (spanglish) and hindi-english (hinglish). various studies @xcite have predicted the high growth in the number of cm speakers, which would surpass the number of native speakers in various globally popular languages (e.g., english). with the advent of social-media platforms (e.g., twitter, facebook, etc.), we observe a manifold increase in the cm communication by multilingual speakers. this leads to a large scale availability of cm data for various nlp tasks. recently, we witness magnitude of work to address various cm nlp tasks such as language identification (shekhar et al., 2020; singh et al., 2018a; * equal contribution @xcite @xcite , pos tagging @xcite @xcite , named entity recognition @xcite @xcite , word normalisation @xcite , cm metrics @xcite , sentiment analysis @xcite , stance detection @xcite , natural language inference @xcite , machine translation @xcite , and questionanswering @xcite . we observe a growing interest in the computational linguistic community to study the cm nlg tasks. recently, various resources and systems have been proposed that explore different dimensions of the cm nlg @xcite @xcite @xcite . evaluation of the cm nlg tasks is challenging due to the high linguistic diversity and lack of standardization. to address this challenge, @xcite has proposed hinge corpus for the hinglish cm text generation and evaluation (see section 2 for details). hinge corpus demonstrates the inefficacy of various widely popular metrics on the cm dataset. in this paper, we choose five evaluation metrics (see section 3 for details) as discussed in @xcite to demonstrate the efficacy of mipe. our proposed metric independent pipeline (mipe) augments these metrics and addresses four major linguistic bottlenecks: (i) spelling variations, (ii) language switching, (iii) missing words, and (iv) the limited number of reference sentences associated with the cm nlg systems. the main contributions are: the rest of the paper is organized as follows. in section 2, we discuss the dataset for the cm nlg evaluation task. in section 3, we present the mipe pipeline addressing the four major bottlenecks for effective cm nlg evaluation. we discuss the results in section 4. in section 5, we discuss the current state and future direction. we conclude the discussion in section 6. ## dataset recently, we observe various works to address the underlying challenges with the cm nlg. numerous resources and systems have been proposed recently to advance the field. in our experiments, we use the hinge corpus proposed in @xcite . the hinge corpus contains 1,976 english-hindi parallel sentences from the iit-b parallel corpus @xcite . corresponding to each of the english-hindi parallel sentences, hinge has two variants of cm hinglish sentences: in addition to the machine-generated hinglish sentences, hinge has a human rating corresponding to each generated sentence. the human rating varies between 1-10, indicating low to high generation quality. two human annotators have rated each of the machine-generated cm sentences. figure 1 shows the example cm sentences generated by humans and two rule-based algorithms along with the rating to the machine-generated cm sentences. figure 2 shows the distribution of the human ratings to the machine-generated hinglish sentences. wac-generated sentences receive a relatively high rating (> 6) as compared to pac. in addition, wac showed a low degree of human disagreements than pac. efficacy of nlg evaluation metrics: srivastava and singh (2021b) present a study demonstrating the inefficacy of five widely popular nlg evaluation metric on the hinge corpus. the five metrics are: (i) bilingual evaluation understudy score (bleu, papineni et al. (2002)), (ii) nist (doddington, 2002), (iii) bertscore (bs, zhang et al. (2019)), (iv) word error rate (wer, levenshtein (1966)), and (v) translator error rate (ter, snover et al. (2006)). higher bleu, nist, or bs values and lower wer or ter values represent better generation performance. tables 1 and 2 show the comparison of five metric scores against the human ratings against wac and pac (see scores present in columns with heading 'without mipe'). in addition, @xcite present a correlation study between the human ratings and the metric scores. for this purpose they divide the ratings into three buckets: ## mipe as discussed in the previous section, the widely used evaluation metrics fail to capture the linguistic diversity of the cm data. based on the empirical observation on the 10 datasets used in (srivastava and singh, 2021a), we identify four major reasons for the failure of nlg evaluation metrics on the cm data. we propose a metric independent evaluation pipeline mipe, for effective evaluation. using mipe, we first reduce the spelling variations (see section 3.1) and the language switching (see section 3.2) in the candidate hinglish sentence. next, we introduce a penalty (see section 3.3) on the evaluation score based on the degree of importance of the missing words in the candidate hinglish sentence. finally, we address the challenge of a limited number of reference sentences (see section 3.4) by segmenting the candidate and the reference sentences into phrases and leveraging the paraphrasing capability. figure 3 shows the architecture of the proposed evaluation pipeline. ## results and evaluation we evaluate wac and pac procedures augmented with mipe pipeline against all the five metrics (as discussed in section 2). ## current state and future directions the results discussed in section 4 demonstrate a need to build metrics, theories, and experiments for better cm nlg evaluation. some of the 7 we experiment with pearson correlation coefficient. as discussed, currently there are several limitations with the cm nlg evaluation which need to be addressed in order to build effective cm nlg systems for multilingual societies. some of the lessons learned and the future directions for the cm nlg evaluations are: ## conclusion in this paper, we present a metric independent evaluation pipeline for efficient code-mixed nlg evaluation. the proposed pipeline shows a high correlation between the human scores and the underlying evaluation metrics. besides the four significant challenges to cm nlg evaluation, in the future, we also plan to address other challenges such as code-mixed existence of named-entities, informal writing style, and missing context.