I ndic MT Eval: A Dataset to Meta-Evaluate Machine Translation Metrics for I ndian Languages (2023)

Abstract: The rapid growth of machine translation (MT) systems necessitates meta-evaluations of evaluation metrics to enable selection of those that best reflect MT quality. Unfortunately, most meta-evaluation studies focus on European languages, the observations for which may not always apply to other languages. Indian languages, having over a billion speakers, are linguistically different from them, and to date, there are no such systematic studies focused solely on English to Indian language MT. This paper fills this gap through a Multidimensional Quality Metric (MQM) dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems. We evaluate 16 metrics and show that, pre-trained metrics like COMET have the highest correlations with annotator scores as opposed to n-gram metrics like BLEU. We further leverage our MQM annotations to develop an Indic-COMET metric and show that it outperforms COMET counterparts in both human scores correlations and robustness scores in Indian languages. Additionally, we show that the Indic-COMET can outperform COMET on some unseen Indian languages. We hope that our dataset and analysis will facilitate further research in Indic MT evaluation.

## introduction natural language generation (nlg) has seen rapid progress in the past few years due to advancements in the field of large language models (llms) @xcite @xcite . although initial research had focused on high-resource languages, recently the focus has shifted to middle-resource and lowresource languages. in the context of machine translation (mt), there is increasing interest in building massively multilingual models supporting numerous translation directions. for example, @xcite release a model which supports around 200 languages (40k directions). while this is commendable, to make mt truly inclusive, it is important that various design choices in the mt life-cycle are evaluated for low-resource languages and not simply transferred and adapted from english. one such important choice is of the correct evaluation metric to be used for evaluating mt systems. a recent survey by @xcite has shown that over the last decade, many evaluation metrics have been proposed for mt. in parallel, several works @xcite @xcite @xcite have shown the inadequacy of popular metrics, such as bleu @xcite , rouge @xcite . however, many languages are not represented in these works and most of the focus is on european languages. on the other hand, there is a growing body of work on machine translation focused on language groups such as indian @xcite , indonesian @xcite , and african @xcite . however, these works rely on english centric metrics due to lack of sufficient studies with tried and tested recommendations for their evaluation of the languages. while techniques like mqm (multidimensional quality metric) are being used for collecting better quality human-evaluation data for english and a select few other languages @xcite , such multidimensional evaluations and analyses are not available for several language groups. we narrow our focus to evaluation of one of these language groups, namely indian languages which have more than a billion speakers worldwide. indian languages are morphologically rich, especially dravidian languages, which exhibit agglutination. furthermore, they have relatively freeword order @xcite ) as compared to european languages which means that frequently used metrics such as bleu may not always be reliable. this calls for an independent focused study on the evaluation of metrics for indic languages in order to understand whether these conclusions drawn hold true for the indian languages. in this paper, we aim to bridge this gap by focusing on the evaluation of mt (from english) into 5 indian languages from 2 different families and make significant contributions towards designing mt evaluation metrics for these languages. our main contribution is in the form of the mqm dataset for indian languages created by taking outputs generated by 7 popular mt systems and asking human annotators to judge the quality of the translations using the mqm style guidelines @xcite . with the help of language experts who are experienced in translation, we generate an mqm dataset consisting of 7000 annotated sentences, 1400 per language. we use the aforementioned dataset to establish correlations between the annotator scores and existing automatic metrics scores belonging to the following classes: (i) n-gram and character based such as bleu, meteor, chrf++, (ii) embeddings based such as vector extrema, bertscore, (iii) pre-trained metrics like bleurt-20, comet. we observe that pre-trained metrics show the highest correlations with the annotator scores, with the comet metric performing the best ( §5.1). additionally, we also observe that the metrics are not capable of capturing the fluency-based errors for indian languages ( §5.4). finally, we use our data to train an indic-comet metric which not only shows stronger correlations with human judgement on indian languages, but is also more robust to perturbations ( §6). we hope that our dataset and metric, which are publicly available 1 , will help spur research in this field. ## related work meta-evaluation studies: evaluation metrics have been under intense scrutiny in order to establish their reliability. @xcite discuss that studying evaluation metrics needs to be a meticulous task by showing many potential issues and oversights that could lead to wrong conclusions. other works focus on extending the resources for meta-evaluations (sai et al., 2021; karpinska et al., 1 https://github.com/ai4bharat/indicmt-eval 2022) and different genres @xcite . while most of these works focus on english, there are works that evaluate the efficacy of metrics on other languages such as german, chinese, spanish, etc. @xcite . on the other hand, we focus on indian languages, which have not received much attention. ## indic-mt eval dataset following @xcite , we collect mqm annotations for 5 indian languages, i.e., tamil (ta), gujarati (gu), hindi (hi), marathi (mr), and malayalam (ml). we sample 200 sentences from the flores-101 dataset @xcite and obtain the translation outputs from 7 machine translation systems ( §3.1) for each of the 5 indian languages. ## experimental setup in this section, we discuss the various evaluation metrics under consideration ( §4.1) and evaluating strategies ( §4.2) followed. ## results and discussions in this section, we present the segment-level correlations in §5.1 and system-level correlations in §5.2, followed by analyzing metrics in §5.3, §5.4. ## indic comet having analyzed various metrics, we fine-tune the best performing metric -comet -using our mqm dataset ( §6.1) and show that the new finetuned metric not only outperforms the comet metric on the majority of the languages but also is more robust to perturbations ( §6.2). additionally, we also test the zero-shot evaluation ability of the indic-comet metric in §6.3. ## conclusion we present a large-scale mqm dataset consisting of 7000 fine-grained annotations, spanning 5 indian languages and 7 mt systems, for evaluating machine translation metrics for indian languages. with the help of this dataset, we show that the current pre-trained metrics outperform the overlapbased metrics ( §5.1) in terms of correlations with the human scores. additionally, we also perform an in-depth study ( §5.4) to identify the drawbacks of the current metrics. we then use our dataset to train an indic specific comet metric that outperforms existing metrics in terms of both correlations and robustness scores ( §6.2). we hope that our dataset and analysis will help promote further research in indic mt evaluation.