An Empirical Analysis of Machine Translation for Expanding Multilingual Benchmarks (2025)

Abstract: The rapid advancement of large language models (LLMs) has introduced new challenges in their evaluation, particularly for multilingual settings. The limited evaluation data are more pronounced in low-resource languages due to the scarcity of professional annotators, hindering fair progress across languages. In this work, we systematically investigate the viability of using machine translation (MT) as a proxy for evaluation in scenarios where human-annotated test sets are unavailable. Leveraging a state-of-the-art translation model, we translate datasets from four tasks into 198 languages and employ these translations to assess the quality and robustness of MT-based multilingual evaluation under different setups. We analyze task-specific error patterns, identifying when MT-based evaluation is reliable and when it produces misleading results. Our translated benchmark reveals that current language selections in multilingual datasets tend to overestimate LLM performance on low-resource languages. We conclude that although machine translation is not yet a fully reliable method for evaluating multilingual models, overlooking its potential means missing a valuable opportunity to track progress in non-English languages.

## introduction large-scale evaluation of multilingual language models (mlms) across hundreds of languages has remained a persistent challenge as the existing benchmarks cover a subset, and mostly highresource, of languages @xcite . although human-translated (ht) evaluation sets offer accurate and reliable evaluation of mlms, creating them for hundreds of languages is costly, time-consuming, and sometimes impossible. consequently, tracking the progress of mlms in most languages has lagged behind. moreover, multilingual benchmarks often cover different subsets * equal contribution. recent advances in machine translation (mt) have emerged as a promising solution to these challenges of multilingual evaluation. prior studies have shown that state-of-the-art mt systems achieve human-level translation quality in certain high-resource languages @xcite . while mt has been used to construct evaluation sets @xcite , to our knowledge, there has been no systematic study of the viability of mt-based evaluations across different evaluation setups and diverse downstream tasks in nearly 200 languages. moreover, although previous work has shown the limitations of mt models, such as translation artifacts and stylistic shifts @xcite , there has been no comprehensive analysis of the potential risks of using mtbased datasets for mlm evaluation. in this work, and by considering recent mt advances, we investigate the viability of using mt for large-scale multilingual evaluation where humanannotated data is unavailable. using four popular multilingual tasks, we translate their test sets into 198 languages using nllb ("no language left behind"), which officially supports 200 languages @xcite . we then compare three mlms' performance, i.e., xlm-r (both base and large versions) @xcite , bloomz @xcite , and aya-101 @xcite , on these mt-based test sets against their ht equivalents, using multiple evaluation setups (zero-shot fine-tuning and zeroshot prompting) and both accuracy and rank correlation as metrics. beyond overall performance comparisons, we analyze how translation quality relates to evaluation results, detecting mt-specific error types using an llm-as-a-judge framework in the selected tasks. finally, we assess whether current multilingual benchmarks misrepresent the performance of mlms when limited to small language subsets. more specifically, we try to answer the following research questions: across tasks and models, mt-based evaluations yield results that are highly correlated with ht-based evaluations (average spearman's 0.95), with only small average accuracy differences (<1.5 points), ## related work multilingual large language models (llms) are rapidly expanding their language coverage, reaching to hundreds of languages @xcite . yet, benchmarks have struggled to keep pace, often covering only a fraction of these languages (e.g., global mmlu covers 42 languages @xcite , whereas models like gemma 3 support over 140 @xcite ). this imbalance makes it difficult to measure progress fairly and consistently across languages. on the other hand, creating humanannotated benchmarks for every language is impractical due to high costs and limited expertise. in this regard, machine translation is an attractive potential alternative to expand existing multilingual datasets at scale. although previous studies have shown mt systems may introduce challenges such as hallucinations and translationese artifacts @xcite @xcite , recent improvements, especially for mid and low-resource languages, have considerably enhanced translation quality @xcite . thus, leveraging advanced mt systems is an efficient way to expand the language coverage of benchmarks more quickly. this approach opens the door to more comprehensive and equitable evaluations, better reflecting the multilingual capabilities of llms. while @xcite also investigate whether machine translated benchmarks can reliably assess model performance across languages, they limit the scope of their study to 20 european languages. since european languages are already overrepresented in existing nlp resources and benchmarks @xcite , while many other languages remain severely underrepresented @xcite , we instead scale our evaluation to 198 languages to provide a more balanced assessment. ## methods in this paper, we evaluate multilingual llms on four tasks across 198 languages. to this end, we create mt test data using the nllb model and assess performance under two evaluation paradigms: zero-shot testing and zero-shot prompting. in this section we provide a comprehensive overview of our methods used for large-scale multilingual evaluation. in section 4, we will explain which methods were used to generate the mt test data itself. ## machine translating test data for machine translation, we employ the nllb model covering 202 languages @xcite . through extensive analysis, @xcite have shown that nllb performance surpasses other powerful llms such as chatgpt (openai., 2022) and @xcite when translating out of english (en ⇒ tgt setting). nllb also demonstrates a minimal difference to the closed source system, google translate foot_1 on the flores-101 dataset. to gain a better understanding of the role of the mt system on the translated data quality and, consequently, its perfor-mance on downstream tasks, we experiment with two nllb versions. we choose the distill nllb with 600m parameters and the 3.3b nllb model using greedy sampling. for each example, we translate each sentence (e.g., sentence1 and sen-tence2 in paws-x) separately. in appendix b, we provide some of the lessons learned from our mt experiments. in ## evaluating the reliability of mt data in this section, we study the opportunities and challenges of using mt to extend multilingual benchmarks and assess model performance across a broader range of languages from two perspectives, i.e. the performance gap when measuring on human versus machine translated data and the quality of the mt data itself. thus, in section 5.1, we first study the degree to which machine translation alters the reliability and validity of performance assessments for llms. then, in section 5.2, we evaluate the quality of the machine translated data itself through automatic metrics and llm-based judgments, both to analyze error patterns and to serve as a proxy for filtering low-quality translated data for evaluation. ## large-scale evaluation results having confirmed in section 5 that our translated test sets are of a reliable quality, we now move on to analyze how the mlms perform on them. in figure 1 , we summarize the performance of xlm-r, bloomz, and aya in 196, 168, and 198 languages, categorized by their data scarcity during the pre-training of each model as explained in section 3.3. we find that the average performance for all models is similar for high-and mid-resource languages. yet, while still above the random baseline, there is a notable drop in performance for low-resource and unseen languages. moreover, we find that standard deviations across low-resource languages are larger than high-and mid-resource ones. this shows that performance across lowresource languages varies a lot, making the average score less reliable. still, performance in unseen languages is relatively high; for xnli and paws-x, on average, we obtain +18% and +29% above random performance, suggesting cross-lingual knowledge transfer to the unseen languages. ## conclusions in this paper, we investigate the use of machine translation to create large-scale multilingual evaluation sets in scenarios where human-translated data is unavailable. our experiments show that using sota mt models for evaluation yields results closely aligned with ht ones, with small average accuracy differences and high rank correlations, indicating stable relative performance across languages. translation quality, measured by chrf++, does not strongly predict downstream performance differences, suggesting that minor variations in automatic scores are not critical for many tasks. however, llm-as-a-judge analysis reveals that examples where models succeed on ht but fail on mt often contain more major errors, as such this method could be used to filter low-quality translations to reduce evaluation noise. scaling evaluations to nearly 200 languages further reveals representativeness gaps in existing benchmarks. for high-and mid-resource languages, original language selections approximate broader trends well, but for low-resource languages, they overestimate performance-by up to 8.7 accuracy points in some cases. this demonstrates that mt-based evaluation can both expand coverage and uncover biases in benchmark design. overall, our findings indicate that mt, when coupled with targeted quality control, enables broader, more representative, and more equitable multilingual evaluation, while highlighting the need to reassess how language selections in current benchmarks reflect true model capabilities. mikel artetxe, sebastian ruder, and dani yogatama. 2020. on the cross-lingual transferability of monolingual representations. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4623-4637. akari asai, trina chatterjee, junjie hu, eunsol choi, et al. 2022. beyond counting datasets: a survey of multilingual dataset construction and necessary resources. in 2022 findings of the association for computational linguistics: emnlp 2022. ankur bapna, isaac caswell, julia kreutzer, orhan firat, daan van esch, aditya siddhant, mengmeng niu, pallavi baljekar, xavier garcia, wolfgang macherey, theresa breiner, vera axelrod, jason riesa, yuan cao, mia xu chen, klaus macherey, maxim krikun, pidong wang, alexander gutkin, apurva shah, yanping huang, zhifeng chen, yonghui wu, and macduff hughes. 2022. building machine translation systems for the next thousand languages. eleftheria briakou, colin cherry, and george foster. 2023. searching for needles in a haystack: on the role of incidental bilingualism in palm's translation capability. in proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 9432-9452, toronto, canada. association for computational linguistics. nuo chen, zinan zheng, ning wu, ming gong, dongmei zhang, and jia li. 2024. breaking language barriers in multilingual mathematical reasoning: insights and observations. in findings of the association for computational linguistics: emnlp 2024, pages 7001-7016, miami, florida, usa. association for computational linguistics. alexis conneau, kartikay khandelwal, naman goyal, vishrav chaudhary, guillaume wenzek, francisco guzmán, édouard grave, myle ott, luke zettlemoyer, and veselin stoyanov. 2020. unsupervised cross-lingual representation learning at scale. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 8440-8451. alexis conneau, ruty rinott, guillaume lample, adina williams, samuel bowman, holger schwenk, and veselin stoyanov. 2018. xnli: evaluating crosslingual sentence representations. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2475-2485. markus freitag, george foster, david grangier, viresh ratnakar, qijun tan, and wolfgang macherey. 2021. experts, errors, and context: a large-scale study of human evaluation for machine translation. transactions of the association for computational linguistics, 9:1460-1474. andrew gordon, zornitsa kozareva, and melissa roemmele. 2012. semeval-2012 task 7: choice of plausible alternatives: an evaluation of commonsense causal reasoning. in *sem 2012: the first joint conference on lexical and computational semantics -volume 1: proceedings of the main conference and the shared task, and volume 2: proceedings of the sixth international workshop on semantic evaluation (semeval 2012), pages 394-398, montréal, canada. association for computational linguistics. junjie hu, sebastian ruder, aditya siddhant, graham neubig, orhan firat, and melvin johnson. 2020. xtreme: a massively multilingual multitask benchmark for evaluating cross-lingual generalisation. in proceedings of the 37th international conference on machine learning, volume 119 of proceedings of machine learning research, pages 4411-4421. pmlr. pratik joshi, sebastin santy, amar budhiraja, kalika bali, and monojit choudhury. 2020. the state and fate of linguistic diversity and inclusion in the nlp world. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6282-6293. tom kocmi, eleftherios avramidis, rachel bawden, ondřej bojar, anton dvorkovich, christian federmann, mark fishel, markus freitag, thamme gowda, roman grundkiewicz, barry haddow, marzena karpinska, philipp koehn, benjamin marie, christof monz, kenton murray, masaaki nagata, martin popel, maja popović, mariya shmatova, steinthór steingrímsson, and vilém zouhar. 2024. findings wenhao zhu, hongyi liu, qingxiu dong, jingjing xu, shujian huang, lingpeng kong, jiajun chen, and lei li. 2024. multilingual machine translation with large language models: empirical results and analysis. in findings of the association for computational linguistics: naacl 2024, pages 2765-2781, mexico city, mexico. association for computational linguistics.