WMT 24++: Expanding the Language Coverage of WMT 24 to 55 Languages & Dialects (2025)

Abstract: As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages/dialects in addition to post-edits of the references in 8 out of 9 languages in the original WMT24 dataset. We benchmark a variety of MT providers and LLMs on the collected dataset using automatic metrics and find that LLMs are the best-performing MT systems in all 55 languages. However, we caution against using our results to reach strong conclusions about MT quality without a human-based evaluation due to limitations of automatic evaluation metrics, which we leave for future work.

## introduction benchmark datasets and evaluations are critical for driving research in machine learning and natural language processing. for instance, the penn treebank @xcite , imagenet @xcite , and squad @xcite were all highly influential for advancing state-of-the-art methods. within the field of machine translation (mt), the conference on machine translation (wmt) annually collects and releases new datasets @xcite that are widely used throughout and beyond the mt community @xcite . while these datasets help advance the field, each year has limited language coverage, typically including around ten language pairs. as large language models (llms) become more capable in languages other than english, it wmt24 @xmath0: we collect post-edits of the references for 8 of the original wmt24 @xmath1. along with icelandic from wmt24 (no post-edit), this results in 55 @xmath2. see appendix a for more information about the languages. in this work, we build upon the mt dataset released in wmt24 @xcite : our analysis demonstrates that frontier llms, like openai o1 (openai, 2024), gemini-1.5 pro @xcite , and claude 3.5 @xcite are highly capable mt systems in all 55 languages (according to automatic metrics), outperforming standard mt providers. further, the translations they produce are scored higher by automatic metrics than the human-written references and post-edits. however, we caution against using our results to immediately conclude that llms produce superhuman performance in all languages due to the limitations of automatic metrics, which may be biased against human translations (appendix e) and largely untested in most of the 55 languages. a further human-based evaluation of the translationswhich we intend to perform in future work-is necessary to make claims about mt quality across a large number of languages. ## related work the most closely related works to our own are the wmt machine translation shared tasks @xcite alia) that have been collecting new mt datasets and benchmarking state-of-theart mt systems since 2006. our work directly builds on top of the dataset collected in wmt24 and significantly expands its language coverage. avg. ar_eg ar_sa bg_bg bn_bd bn_in ca_es cs_cz da_dk de_de el_gr es_mx et_ee fa_ir fi_fi fil_ph fr_ca fr_fr gu_in he_il hi_in hr_hr hu_hu id_id is_is it_it ja_jp kn_in ko_kr lt_lt lv_lv ml_in mr_in nl_nl no_no pa_in pl_pl pt_br pt_pt ro_ro ru_ru sk_sk sl_si sr_rs sv_se sw_ke sw_tz ta_in te_in th_th tr_tr uk_ua ur_pk vi_vn zh_cn zh_tw zu_za 3tir%-s 'peyhi +iqmrm4vs +48s +iqmrm*pewl 9rfefip8s[iv 'sqqerh6tpyw 3tir%-sqmrm (iit08verwpexi +sskpi8verwpexi %]e 1mgvswsjx8verwpexsv 200&& =erhi\8verwpexi 7]wxiq6eromrkw&@xmath3: system rankings according to metricx-24. the rankings are significance clusters, so no system with the same rank is statistically better than all other systems within the same cluster, and all systems in one cluster are statistically better than all systems with a worse rank. statistical testing was done with a one-sided permutation test with α = 0.05 @xcite . outside of wmt, there have been several efforts to create mt datasets that cover a large number of languages. flores @xcite is a popular benchmark that covers 200 languages, with an emphasis on lowresource languages, and ntrex @xcite collected new translations from english to 128 languages. the source texts in flores are sentences from wikimedia covering three domains (news, children's books, and travel) and ntrex targets news. our work is complementary to theirs, offering an updated dataset with paragraph-level translations that cover four different domains (news, literary, speech, and social). there have also been several efforts that target languages based on region, including mmtafrica (emezue and dossou, 2021), africomet @xcite , indictrans2 @xcite , and indicmt @xcite . while our work aims for broad language coverage, these regional efforts tend to focus on lower resource languages in the regions of interest. in other nlp tasks, there is increased interest in creating new multilingual benchmarks that focus on languages other than english. several popular examples include global mmlu @xcite , xquad @xcite , xlsum @xcite , mgsm @xcite , in-dicgenbench @xcite , xtreme @xcite , xtreme-up @xcite , tydi qa @xcite . ## conclusion in this work, we create a benchmark dataset called wmt24++ that extends the language coverage of wmt24 to include 55 languages and dialects. through our experiments evaluating different mt systems, we find that (1) mt systems appear to generate translations of higher quality than humans and (2) frontier llms are highly capable mt systems in a large number of languages. due to limitations of automatic metrics, both of the above conclusions should be verified with a human-based evaluation, which we intend to do in future work.