Evaluating WMT 2025 Metrics Shared Task Submissions on the SSA - MTE A frican Challenge Set (2025)

Abstract: This paper presents the evaluation of submissions to the WMT 2025 Metrics Shared Task on the SSA-MTE challenge set, a large-scale benchmark for machine translation evaluation (MTE) in Sub-Saharan African languages. The SSA-MTE test sets contains over 12,768 human-annotated adequacy scores across 11 language pairs sourced from English, French, and Portuguese, spanning 6 commercial and open-source MT systems. Results show that correlations with human judgments remain generally low, with most systems falling below the 0.4 Spearman threshold for medium-level agreement. Performance varies widely across language pairs, with most correlations under 0.4; in some extremely low-resource cases, such as Portuguese–Emakhuwa, correlations drop to around 0.1, underscoring the difficulty of evaluating MT for very low-resource African languages. These findings highlight the urgent need for more research on robust, generalizable MT evaluation methods tailored for African languages.

## introduction in recent years, with the rise of large language models (llms), more and more machine translation (mt) systems have emerged, demonstrating competitive performance. this growth has created an increasingly urgent need for more accurate methods to assess the quality of generated translations. traditional metrics such as bleu @xcite , meteor @xcite , and chrf @xcite , which rely on ngram matching, show only limited correlation with human judgments, indicating their limited ability to capture semantic-level quality @xcite . more recently, neural metrics such as bertscore @xcite have shown improved capability in capturing semantic simi-larity @xcite . comet @xcite further advances this by framing machine translation evaluation (mte) as a regression task using encoder-only language models and training on human-annotated scores. likewise, the metricx @xcite family of metric, based on the mt5 @xcite series multilingual encoder-decoder lm, adopts a regression-based framework similar to comet. these neural, learned metrics have been shown to achieve higher correlations with human assessments across a wide range of languages @xcite . however, before 2024, due to the lack of machine translation evaluation data, the performance of these models was largely untested on sub-saharan african languages. in 2024, afrimte @xcite was created. evaluation results revealed that while existing metrics performed well on some relatively higher-resourced languages, they struggled with translations for very low-resource languages such as twi and luo @xcite . the authors demonstrated that this gap can be partially addressed by further pretraining models on african languages; however, performance remains low for specific language pairs, like eng-luo, underscoring the need for dedicated training data for these languages. although afrimte marked progress in this area, several limitations remained. the dataset was relatively small, with only about 200 cases per language pair. also, afrimte included outputs only from nllb-200 (600m) @xcite and m2m-100 (418m) @xcite . in addition, afrimte did not provide any training data, which meant that neural-based metrics could not be directly optimized for sub-saharan languages. ## ssa-mte we perform our evaluation on the recently released ssa-mte dataset @xcite , a largescale human-annotated benchmark for assessing machine translation quality for african languages. the dataset contains over 73,000 sentence-level annotations across 13 language pairs (lps) in the news domain, of which 10 lps include a dedicated test set. the test set covers 7 english-sourced lps: amharic (eng-amh), hausa (eng-hau), kikuyu (eng-kik), kinyarwanda (eng-kin), luo (eng-luo), twi (eng-twi), and yorùbá (eng-yor); 2 french-sourced lps: ewe (fra-ewe) and wolof (fra-wol); and 1 portuguese-sourced lp: emakhuwa (por-vmw). for this challenge set, the authors additionally introduced another portuguese-sourced lp: nyanja (por-nya). the size of each test set is shown in the selected lps reflect africa's linguistic and regional diversity, covering anglophone, francophone, and lusophone areas. in addition, the languages span three major language families: afro-asiatic (hausa, amharic), niger-congo (kikuyu, kinyarwanda, emakhuwa, nyanja, twi, yorùbá, ewe, wolof), and nilo-saharan (luo), ensuring representation across west, east, central, and southern africa, and thus capturing both geographic and typological diversity. each instance is annotated by one human evaluator with both a continuous adequacy score and span-level (character-based) error labels, enabling fine-grained evaluation of mt outputs. annotators are provided with the source sentence and its translation; instructed to first identify all errors according to the annotation protocol proposed by wang et al., and then assign a final adequacy score. for english-and french-sourced cases, source sentences are drawn from the global voices news website, following harmful-content filtering and topic-diversity-based article selection @xcite . for portuguese-sourced cases, source sentences are extracted from the multilingual open text dataset, which features news articles published by voice of america (voa). all source texts are translated into the target languages by professional translators to serve as reference translations @xcite . for english-and french-sourced sentences, six mt systems are included (but only five for kikuyu, as google translate does not support it): gpt-4o, gemini-1.5, claude-3.5, google translate, and two open-source models: nllb-200-distilled-600m (nllb- @xcite and m2m-100-418m @xcite . each system contributes an equal number of translations. for portuguese-sourced sentences, four mt systems are included: gpt-4omini, gemini-1.5, claude-3.5, and google translate. compared to the african challenge set in the wmt 2024 metrics shared task, this year's set is substantially larger (12,768 vs. 2,815 annotated instances), covers a broader range of mt systems, and introduces portuguese as a new source language. this broader linguistic and system coverage is expected to yield a more accurate and comprehensive evaluation of metric performance for african languages. ## metrics the submissions of wmt 2025 metrics shared task contain baseline metric results provided by the organizers, as well as the results of primary and secondary submissions from participants' metric systems. this section introduces each of these approaches foot_0 . ## analysis ## conclusion we have presented the results of the wmt 2025 metrics shared task for the ssa-mte challenge set, the largest and most diverse benchmark to date for mt evaluation in sub-saharan african languages. the evaluation covered a wide range of metrics, including official baselines, participant submissions, and additional african-focused baselines. overall, correlations with human judgments remain modest, with most submitted systems achieving spearman correlations below 0.4, indicating low-level agreement. reference-based neural metrics generally outperform reference-free approaches, with metricx-25-ref leading among participant systems. ssa-comet-mtl, trained with in-domain ssa-mte data, sets a new reference point for african-language mte, demonstrating clear gains over africomet-1.1-mtl and underscoring the value of domain-matched supervision. per-language analysis shows substantial variation in difficulty, reflecting differences in source-target pairing, linguistic complexity, and resource availability. only en-amh and pt-nya surpass the 0.4 threshold for medium-level agreement, while ptvmw remains particularly challenging with correlations near 0.1. these findings suggest that while progress has been made, significant room remains to improve robustness and accuracy for the most difficult language pairs. future work should explore integrating african-language-enhanced encoders, expanding the diversity of training data, and developing methods that can better generalize across very low-resource languages.