How Good is Zero-Shot MT Evaluation for Low Resource I ndian Languages? (2024)

Abstract: While machine translation evaluation has been studied primarily for high-resource languages, there has been a recent interest in evaluation for low-resource languages due to the increasing availability of data and models. In this paper, we focus on a zero-shot evaluation setting focusing on low-resource Indian languages, namely Assamese, Kannada, Maithili, and Punjabi. We collect sufficient Multi-Dimensional Quality Metrics (MQM) and Direct Assessment (DA) annotations to create test sets and meta-evaluate a plethora of automatic evaluation metrics. We observe that even for learned metrics, which are known to exhibit zero-shot performance, the Kendall Tau and Pearson correlations with human annotations are only as high as 0.32 and 0.45. Synthetic data approaches show mixed results and overall do not help close the gap by much for these languages. This indicates that there is still a long way to go for low-resource evaluation.

## introduction while there has been a meteoric rise in the amount of data and improvements in architectures for machine translation (mt) models @xcite , in order to scientifically establish whether the translation quality has improved, it is important to have reliable evaluation metrics. however, most of the evaluation metrics were developed with english and a few select other languages in mind. it has been shown that such metrics do not necessarily generalize to other languages and have to be separately meta-evaluated @xcite . the reasons behind this include linguistic aspects that vary across languages, along with factors like the diversity of outputs produced by the models for each language. such qualitative differences will be exacerbated in low-resource languages due to the prominent reliance on extensive data resources by today's models. in this work, we delve deeper into the evaluation of low-resource indian languages, namely assamese, maithili, punjabi, and kannada, belonging to 2 different language families. our goal is to establish the reliability of mt evaluation metrics for low-resource languages. to facilitate this, we collect human scores on the candidate translations using the mqm approach @xcite . we make use of 5 large multilingual models and apis that can output text in these languages to generate candidate translations for evaluation. we then collect 250 annotations per language, amounting to a total of 1000 mqm annotations for low-resource languages. using the data we created, we evaluate multiple existing evaluation metrics of different types, both automatic and learned. in the case of learned metrics, since we do not have training data, we leverage data for related indic languages from @xcite for fine-tuning and performing zeroshot meta-evaluations. we observe that for these learned metrics, despite studies finding decent to good performance in other languages, there is a huge margin for improvement in evaluating lowresource languages. we also explore the influence of the base model and synthetic data generation for low-resource languages. in summary, our contributions are as follows: (i) mqm dataset for 4 low resource languages for evaluation (ii) meta-evaluation of existing metrics on low-resource languages (iii) analysis of potential techniques to improve the metrics, including (a) exposure of metrics to related languages, (b) different base models, and (c) usage of synthetic data. we show that evaluation for low-resource languages is still far behind other languages. ## related work the effectiveness of evaluation metrics has been studied for various languages. most of the existing mt evaluation metrics are typically analyzed for language pairs where english serves as either the source or target language. that has led to several criticism works @xcite @xcite followed by improvements. however, models are getting increasingly multilingual and slowly evaluation metrics are being studied for other languages @xcite @xcite . metrics like chrf @xcite and chrf++ @xcite were proposed for character-based, morphologically-rich languages. while some of these criteria hold for the languages we consider, there is no publicly available open study of such metrics for the specific case of low resource languages. on the other hand, different evaluation metrics are being used to evaluate models in these languages. wmt23 @xcite had a special task track for low resource indic languages for which bleu, chrf, ribes, ter, and comet metrics were used apart from human evaluation. however, to the best of our knowledge, there are no studies analyzing whether these metrics correlate with human judgments or not for these languages. additionally, there is no publicly available data with human scores to study this. @xcite used synthetic data augmentation to build a bleurt-like metric for low resource languages. the only indian language in their set is punjabi (2k size, not publicly released), which initially had a poor pearson correlation of 0.184. this was slightly improved to a value of 0.194 when adding synthetic data to their baseline data, although it is still a poor correlation value. ## methodology we collect mqm annotations as well as direct assessment (da) scores and also create synthetic data for 4 languages, viz., assamese, punjabi, kannada, and maithili. we use the human-curated data as test data to benchmark the performance of various metrics on these low resource languages. the synthetic data is used to investigate the use of such strategies for augmenting resources in these languages for potential improvements in performance. we design experiments to understand the role of other related languages and the base model on the performance. the following subsections provide the details of the data we create and the strategies explored in our experiments. ## results we present the results for the following research questions to find ways to potentially improve performance on these models: (rq0) how do existing metrics fare on lowresource languages? (rq1) does fine-tuning on related languages help? (rq2) does replacing the underlying model of a trained evaluation metric with an alternate backbone model trained on related languages help? (rq3) does synthetic training data help? we report kendall-tau and pearson correlations with human annotations. ## conclusions our work introduced an mqm dataset for four low resource languages consisting of 250 examples per language. using this dataset, we analyzed the zero-shot performance of different types of existing metrics and observed that none of these existing methods showed good results in the case of low resource language. we explored different techniques to improve the performance, which includes finetuning on related language using indic mt eval dataset 4.2, changing the base model to an indicmodel 4.2.1 and using synthetic dataset of these low resource language 4.3. while some of these techniques provide small improvements, we find that there is still a long way to go for low-resource language evaluation.