CUNI and Phrase at WMT 25 MT Evaluation Task (2025)

Abstract: This paper describes the joint effort of Phrase a.s. and Charles University’sInstitute of Formal and Applied Linguistics (CUNI/UFAL) on the WMT25Automated Translation Quality Evaluation Systems Shared Task. Both teamsparticipated both in a collaborative and competitive manner, i.e. they eachsubmitted a system of their own as well as a contrastive joint system ensemble.In Task~1, we show that such an ensembling—if chosen in a clever way—canlead to a performance boost. We present the analysis of various kinds ofsystems comprising both “traditional” NN-based approach, as well as differentflavours of LLMs—off-the-shelf commercial models, their fine-tuned versions,but also in-house, custom-trained alternative models. In Tasks~2 and~3 we showPhrase’s approach to tackling the tasks via various GPT models: Error SpanAnnotation via the complete MQM solution using non-reasoning models (includingfine-tuned versions) in Task~2, and using reasoning models in Task~3.

## introduction machine translation (mt) evaluation has evolved rapidly in recent years, driven by the dual rise of large language models (llms) and specialised neural quality estimation (qe) architectures. shared tasks, such as the wmt mt evaluation campaign, provide a rigorous and reproducible framework for assessing advances in automated evaluation. the wmt25 mt evaluation task continues this tradition, offering a benchmark for comparing diverse approaches across multiple subtasks, from systemlevel quality prediction to fine-grained error annotation. in this paper, we present a joint study by phrase a.s. and charles university's institute of formal and applied linguistics (cuni/ufal), combining independent research streams with collaborative experimentation. both teams participated in task 1, each submitting their own primary and secondary systems, as well as a contrastive joint ensemble in task 1. this setting allowed us to examine not only the relative strengths of distinct modelling paradigms but also the potential synergies of crossteam integration. phrase also participated in task 2 and task 3. our contributions are threefold. first, we investigate complementary modelling strategies for task 1, including nn-based regression models trained on large-scale mqm-style data (multidimensional quality metrics, @xcite , and llm-based systems producing full linguistic quality annotations. we analyse how fusing/ensembling these approaches, when carefully configured, can yield measurable performance gains. second, for task 2, we evaluate gpt-based autolqa (automatic language quality assessment) systems-both off-the-shelf and fine-tuned variants-demonstrating their applicability to error span annotation in mqm and esa (error span annotation, @xcite formats. third, for task 3, we explore reasoning-enabled llms with targeted prompting for translation improvement, highlighting trade-offs between fluency-and accuracy-oriented objectives. by documenting both competitive and collaborative results, we aim to provide insight into practical design choices-such as metric selection, data sampling, and system fusion-that influence performance in automated mt evaluation. beyond the leaderboard standings, our analysis identifies patterns that may inform future research on hybrid evaluation architectures, the role of fine-tuning in llm-based evaluators, and the limits of current metrics in capturing subtle quality differences. ## task 1 in task 1, phrase and ufal decided to collaborate and choose the strategy in which each organisation submits its own primary system and one of its own secondary systems. each team dedicated the other secondary slot to a joint system ensemble. the submitted systems are marked with an asterisk* in the respective tables. ## task 3 (phrase) for task 3, our submissions are based on gpt models as well. we experimented with varying prompts, openai models and settings (notably the amount of reasoning effort which can be set to "low", "medium" or "high" in the api). after visual inspection and analysis of the error spans and scores provided for the development set, we deemed them quite noisy and not informative enough. we therefore opted for not using this information in the experiments. our systems only utilise the source sentences and the mt outputs. our secondary submission prod (denoted by a single asterisk in we also experimented with dedicated prompts and a simpler setup targeted towards only improving the general quality of the translations. we simplified the system as well-whereas our production systems typically perform multiple passes over the data, here, we limited the system to just a single pass. in terms of openai models, we find that reasoning models are well suited for this task. we evaluated several combinations of reasoning effort settings and model types (notably o3 and o3-mini). our primary submission, as well as most of the contrastive runs, utilise the o3 foot_4 model with medium reasoning effort. we evaluated four different prompts: foot_5 based on the results on the wmt development set, we selected the prompt fluency_s as our primary submission. this system has the secondhighest improvement in comet-qe score and makes fewer edits than fluency, thereby reaching a better gte-ratio (gain-to-edit ratio, i.e., the difference in comet foot_6 divided by ter, as defined in the task description). it seems noteworthy that comet-qe score grows more when the prompt is focused on improving translation fluency, as opposed to accuracy. ## conclusion we presented our submissions to wmt25 mt evaluation task developed by two teams, cuni and phrase: two primary and four secondary submissions for task 1, one primary and two secondary submissions for task 2 and one primary and one secondary submission for task 3. for system fusion/ensembling used in task 1, the rule "more features are better" generally holds, although not for every individual language-pair or metric. it turns out that the ability of the regressor model "saturates" and gets stuck in the diminishing-reward paradigm soon. it is interesting to see that competitive results can be achieved by combining very few systems if we choose them well, as is the case of "slick" systems. in task 3, we failed to improve on the baseline translation quality in the official rankings. however, our internal post-submission analysis showed a different picture and we currently lack a good explanation for the disagreement. during the submission, we saw few puzzling outliers for certain language pairs in all tasks' leaderboard, and we expect to reveal the issues after the official annotations are released.