Findings of the WMT 25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help (2025)

Abstract: The WMT25 Shared Task on Automated Translation Evaluation Systems evaluates metrics and quality estimation systems that assess the quality of language translation systems. This task unifies and consolidates the separate WMT shared tasks on Machine Translation Evaluation Metrics and Quality Estimation from previous years. Our primary goal is to encourage the development and assessment of new state-of-the-art translation quality evaluation systems. The shared task this year consisted of three subtasks: (1) segment-level quality score prediction, (2) span-level translation error annotation, and (3) quality-informed segment-level error correction. The evaluation data for the shared task were provided by the General MT shared task and were complemented by “challenge sets” from both the organizers and participants. Task 1 results indicate the strong performance of large LLMs at the system level, whilereference-based baseline metrics outperform LLMs at the segment level. Task 2 results indicate that accurate error detection and balancing precision and recall are persistent challenges. Task 3 results show that minimal editing is challenging even when informed by quality indicators. Robustness across the broad diversity of languages remains a major challenge across all three subtasks.

## introduction the wmt25 shared task on automated translation quality evaluation systems 1 evaluates automated systems for assessing and improving translation quality, including automated quality metrics, * version 2: 7 november 2025 1 www2.statmt.org/wmt25/mteval-subtask.html source non parliamo italiano. translation i don't speak spanish! task 1: score @xmath0: error span @xmath1: post-@xmath2. the shared task consists of three primary subtasks shown in the primary focus of this year's updated task is on robust translation quality evaluation systems that can effectively detect translation errors generated by increasingly accurate llm-based translators as well as handle previously unseen problems related to using llms for translation, such as output verbosity and incorrect output language. newly, the evaluation data, originating from the general mt task, were intentionally chosen to be challenging for mt: they feature longer length, sourcing from some originally non-textual modalities, and translation into a wider variety of languages than in previous years. task 1 this year was designed to evaluate both mt metrics and qe systems. reference-based automatic metrics score mt output by comparing the translation with a reference translation generated by human translators, who are instructed to translate "from scratch" without post-editing from mt. reference-free quality estimation (qe) systems score translations solely based on their adherence to the source language. we collectively refer to all of these systems as "auto-raters" throughout the rest of the paper. all auto-raters were evaluated based on their agreement with human ratings when scoring mt systems and human translations at the system and segment level. in task 2, systems are evaluated on their ability to detect and accurately annotate the spans of translation errors by contrasting them with the human error annotations. for task 3, systems that correct a given translation are evaluated based on the quality of their post-edits, rewarding effective improvements with minimal changes. below are some of the key details and changes implemented for this year's shared task: task level meta-metric 1 system spa 1 segment acc * eq 2 char f1 3 segment ∆-comet our main findings are: the rest of the paper is organized as follows. section 2 introduces our subtasks. section 3 presents the evaluation set and the mt systems whose output was judged. following that, the baselines, participants, meta-evaluation procedure, and main results of each subtask are discussed in section 4 for segment-level quality score prediction, in section 5 for span-level error annotation, and in section 6 for quality-informed post-editing. section 7 presents the submitted challenge sets, and section 8 concludes. the shared task this year consisted of three primary subtasks that address translation quality assessment from three perspectives: (1) segment-level quality score prediction, (2) span-level translation error detection, and (3) quality-informed segment-level error correction. an additional fourth subtask solicited the submission of challenge sets that identify where automated metrics and auto-rater systems fail. all subtasks are introduced below: ## task 1: segment-level quality score prediction this section presents the segment-level quality score prediction subtask in detail. we describe the auto-rater systems participating in the subtask in section 4.1, our meta-evaluation procedure in section 4.2, and our main results in section 4.3. section 4.4 reports some further analysis of the results beyond correlation and accuracy. ## task 2: span-level error detection this section presents the span-level error detection task in more detail. we discuss in more depth the error annotations per language pair (section 5.1), and describe the baselines (section 5.2) and the participant submissions (section 5.3). our metaevaluation is described in section 5.4. we then present the results, mostly focusing on the primary submissions, in section 5.5. ## challenge sets for the third year, our shared task included a subtask involving challenge sets. this subtask is inspired by the build it or break it: the language edition shared task @xcite , which aimed at testing the generalizability of nlp systems beyond the distributions of their training data. whereas the standard evaluation of the shared task is conducted on test sets containing generic text from real-world content, the challenge set evaluation is based on test sets designed with the aim of revealing the abilities or the weaknesses of the metrics or evaluating particular translation phenomena. in order to shed light on different perspectives on evaluation, the subtask takes place in a decentralized manner: contrary to the main metric task, the test sets are not provided by the organizers but by different research teams, who are also responsible for analyzing and presenting the results. ## conclusion this paper documented the results of the wmt25 shared task on automated machine translation evaluation systems, which unified the metrics and qe shared tasks from previous years. the shared task this year consisted of three subtasks: (1) segmentlevel quality score prediction, (2) span-level translation error annotation, and (3) quality-informed segment-level error correction. task 1 results indicate the strong performance of large llm-as-ajudge auto-rater systems at the system level, while reference-based baseline metrics outperform llms at the segment level. task 2 results indicate that accurate error detection and balancing precision and recall are persistent challenges. task 3 results show that minimal editing is challenging even when informed by quality indicators. robustness across the broad diversity of languages remains a major challenge across all three subtasks. as described throughout the paper, this year marked significant changes to multiple dimensions of the evaluation. evaluation data, originating from the general-mt task, was more challenging for mt systems, and covered a diverse set of new language-pairs. the move to long segments and the adoption of esa human annotation for most of the languages were also new. we strongly believe that these changes were all warranted by the changing landscape in the field of mt and that they better align our evaluation with the current landscape. however, these changes are also likely responsible for some of the unexpected results observed this year, particularly for task-1. we encourage further analysis of these results by the mt research community at large.