Findings of the WMT 25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets (2025)

Abstract: This paper presents the results of the General Machine Translation Task organized as part of the 2025 Conference on Machine Translation (WMT). Participants were invited to build systems for any of 30 language pairs. For half of these pairs, we conducted a human evaluation on test sets spanning four to five different domains.We evaluated 60 systems in total: 36 submitted by participants and 24 for which we collected translations from large language models (LLMs) and popular online translation providers.This year, we focused on creating challenging test sets by developing a difficulty sampling technique and using more complex source data. We evaluated system outputs with professional annotators using the Error Span Annotation (ESA) protocol, except for two language pairs, for which we used Multidimensional Quality Metrics (MQM) instead.We continued the trend of increasingly moving towards document-level translation, providing the source texts as whole documents containing multiple paragraphs.

## introduction the tenth conference on machine translation (wmt25) 1 was held in conjunction with the 2025 1 www2.statmt.org/wmt25/ conference on empirical methods in natural language processing @xcite . this 20th iteration of the conference builds on previous editions @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite 2 and hosted ten shared tasks on various aspects of machine translation (mt). this paper describes one of these tasks: the general machine translation shared task. the goal of the general machine translation shared task is to explore the translation capabilities of current systems across a broad range of languages. determining how best to test general mt performance is a research question in itself. numerous phenomena could be evaluated, the most important of which include: for @xmath0, we use the mqm annotation schema instead. finally, as in previous years, this year's shared task included several test suites that focused on a range of translation challenges, described in section 8. all submissions to the general mt task, along with sources, references, and human judgments, are available in the dedicated github repository. 5 this paper is organized as follows. we first describe the data used in the shared task, detailing the collection and preparation of our test sets (section 2) and outlining the permitted training data for the constrained track (section 3). next, we introduce the participating systems, including the large language model baselines added by the organizers (section 4). we then explain our evaluation methodology, covering both automatic metrics (section 5) and human evaluation protocols (section 6). finally, we present the official results (section 7), describe the test suites (section 8), and offer concluding remarks (section 9). findings of the general mt task. we make the following observations: ⋆ the number of participants continues to grow: participation increased again this year, with a total of 36 submissions. the majority of these participants used llms in their systems, most commonly by fine-tuning (section 4). ⋆ automatic scores are biased: although shyhunyuan-mt placed first for all but one language pair in the automatic rankings, human evaluation revealed its performance was considerably lower than that of the top-rated systems (section 7.2). ⋆ human translations are not always in the winning cluster: human references are in the winning cluster for only six out of 15 language pairs (section 7.2). ⋆ constrained models challenge the performance of llms: the top-performing constrained system was shy-hunyuan-mt, which placed in the winning cluster for 11 language pairs within its category followed by algharb placed in winning cluster for 6 language pairs. the best system overall, was gemini 2.5 pro, which was in top cluster in 14 language pairs (section 7.2). ⋆ speech domain was the most challenging: the speech domain texts were most challenging to translate (likely due to asr errors) while literary texts were the easiest (section 7.2). ⋆ sota systems still struggle with robustness: analysis from six specialized test suites reveals that state-of-the-art (sota) systems still struggle with robustness to non-standard input, linguistic complexity, domain-specific terminology and gender choice/agreement in particular language pairs. this is despite notable improvements from advanced llms in areas like inclusivity and performance in certain specialized domains (section 8). ## test data in this section, we describe the test data collection process (section 2.1) and the creation of human reference translations (section 2.2). this year, we introduced a new difficulty-based sub-sampling of source texts procedure (sections 2.1.1 and 6). motivated by the ever-increasing capabilities of modern mt systems, this step is designed to make our test sets more challenging by selecting source documents that are estimated to be more difficult to translate. ## system submissions this year, we received 96 submissions from 36 participating teams. while the number of submissions is slightly lower than last year's 105, the number of participating teams increased by roughly a third. in line with previous years, we included translations from three public mt services, anonymized as online-{b,g,w}. we also added contrastive translations from 20 llms-including commercial products like gpt-4.1 and open weights models like llama3.1-and one encoder-decoder system (nllb). this brought the total to 60 participants. all participating systems are listed in ## automatic evaluation as in the last year, a high number of submissions foot_7 made full comprehensive manual evaluation infeasible. we therefore employed automatic metrics to select systems for human evaluation via a procedure we call autorank. this year's procedure improves upon wmt24 by incorporating a broader set of metrics and a revised aggregation method. ## human evaluation the human evaluation is done primarily using error span annotation (esa; @xcite . for @xmath1; @xcite . the esa protocol. the annotators (professional translators but not experts in mqm/esa-style annotations) were asked to mark each error as well as its severity, "minor" or "major". in addition, the annotators were also asked to assign a score from 0 to 100, similar to direct assessment (da), to the whole annotation segments (usually a paragraph). however, the esa score should be more robust than da alone because the annotators are primed by the highlighted errors at the time of the scoring. the esa interface. the interface is shown in figure 1 with annotator instructions in appendix a. at the start of annotation, each annotator was exposed to an interactive tutorial where they were asked to interact with the system. the source for the speech domain is a video which is shown in a native html video player. the output of the esa annotation is a list of errors and their severity (minor or major) and the final score from 0 to 100 for each segment. task setup. the whole annotation was split into "tasks" where each task had a balanced number of words to make it approximately 1 hour long. each task is done by a single annotator and contains segments from a single domain but contains output from multiple systems. in contrast to previous years, we do not include a quality control check due to their annotation costs and low reliability @xcite . instead, we include "control tasks" for each language, which is the same task that each participating annotator has to fill out exactly once. because these control tasks are fixed, this allows us to model annotator bias and reliability. finally, each segment is annotated exactly twice, which can be used to estimate inter-annotator agreement and is especially useful for the metrics shared task @xcite . see list of changes in contrast to previous version in appendix a. ## official ranking results we now describe how we compute the final ranking and then discuss the final results and potential issues. the ranking is presented in tabular form in section 7.4. ## conclusions the wmt 2025 general machine translation task covered 30 language pairs, with human evaluation conducted on half of them across four to five domains. we prepared more challenging test set by utilizing novel difficulty sampling. we evaluated 60 systems in total: 36 participant submissions and 24 systems collected from llms and popular online providers. participation continued to grow compared to last year, and most teams utilize llms, often via fine-tuning. we adopt esa and mqm as the human evaluation protocols, which show the weak-points of models, especially in egyptian arabic dialect or in extremely low-resource language maasai. automatic rankings did not always match human judgments: systems that topped automated metrics such as shy-hunyuan-mt, did not consistently win under human evaluation, pointing to persistent metric bias in mbr and reinforcing that human evaluation should remain the final arbiter of translation quality. domain analyses showed speech as the most challenging (likely due to asr noise), while literary was the easiest among those tested. targeted test suites revealed remaining weaknesses in robustness to non-standard input, linguistic complexity, domain terminology, and gender choice/agreement, even as advanced llms improved inclusivity and performance in some specialized areas. all source data, system outputs, and human judgments are released to support transparency, reproducibility, and further research on machine translation.