M etric X -25 and G em S pan E val: G oogle T ranslate Submissions to the WMT 25 Evaluation Shared Task (2025)

Abstract: In this paper, we present our submissions to the unified WMT25 Translation Evaluation Shared Task. For the Quality Score Prediction subtask, we create a new generation of MetricX with improvements in the input format and the training protocol, while for the Error Span Detection subtask we develop a new model, GemSpanEval, trained to predict error spans along with their severities and categories. Both systems are based on the state-of-the-art multilingual open-weights model Gemma 3, fine-tuned on publicly available WMT data. We demonstrate that MetricX-25, adapting Gemma 3 to an encoder-only architecture with a regression head on top, can be trained to effectively predict both MQM and ESA quality scores, and significantly outperforms its predecessor. Our decoder-only GemSpanEval model, on the other hand, we show to be competitive in error span detection with xCOMET, a strong encoder-only sequence-tagging baseline. With error span detection formulated as a generative task, we instruct the model to also output the context for each predicted error span, thus ensuring that error spans are identified unambiguously.

## introduction large language models (llms) have been evolving at a breakneck speed over the past couple of years, achieving performance in machine translation (mt) that matches or even exceeds that of humans for certain languages and domains @xcite . while human evaluation is still the most reliable way to assess the quality of mt models, in this fast-paced environment of state-of-the-art models being released on a monthly basis, the cost and duration make it infeasible to run human evaluation studies to benchmark models regularly. improving automatic metrics is therefore instrumental to making further progress in mt, especially as the field expands to low-resource languages and more difficult domains. currently, the most successful automatic mt evaluation metrics are trained neural models themselves, following one of two main paradigms: (1) regression models predicting a scalar quality score @xcite , and (2) sequence-tagging or generative models, providing fine-grained quality feedback, including error spans, severities and categories @xcite . in this work, we push the performance of automatic metrics in both categories higher by leveraging a state-of-the-art multilingual open-weights llm, gemma 3 (gemma @xcite , resulting in two separate submissions to the wmt25 evaluation shared task. for the quality score prediction subtask, we develop metricx-25, a successor to metricx-24 @xcite , updated to use an encoderonly architecture and trained on a combination of publicly available direct assessment (da) and multidimensional quality metrics (mqm) scores from wmt shared tasks between 2015 and 2023. for the error span detection subtask, we introduce gemspaneval, a generative model that identifies and categorizes error spans in json format, trained exclusively on mqm error span annotations from wmt20-24. we enable gemspaneval to uniquely identify short, non-unique error spans by training the model to also indicate the error span context where necessary. the key takeaways from our experiments, detailed in this report, include: the systems we developed for both the quality score prediction and the error span detection are trained solely on publicly available data from the wmt metrics shared tasks between 2015 and 2024. quality ratings for system translations across those 10 years were collected using 3 different methods: we use both da and mqm scores for training met-ricx, our score prediction system, and mqm error span annotations for training gemspaneval. in general, we reserved data from wmt24 -both mqm @xcite and esa @xcite ) -for validation of our models, with the exception of one submission to the error span detection task. we provide more details on the training and validation sets in §3 for metricx and in §4 for gemspaneval. ## quality score prediction: metricx-25 our metricx-25 submissions to the quality score prediction subtask are based on the successful metricx-24 @xcite , with several modifications and improvements. the biggest one among them is the switch from mt5 @xcite to gemma 3 as the backbone model. we start this section by providing an overview of the similarities and differences between metricx-24 and metricx-25, then give more details on the training and evaluation data, and finally describe our experiments and the metricx-25 systems we submitted to the shared task. ## error span detection: gemspaneval in this section, we describe our submission to subtask 2: error span detection. we denote our system gemspaneval, as it is a span-level prediction model based on gemma 3. ## related work for decades, right until the recent advent of llms, the most widely adopted automatic evaluation metrics would express the predicted machine translation quality as a scalar score. this is the case for metrics ranging from simple lexical overlap metrics, such as bleu @xcite and chrf @xcite , to learned metrics including bleurt @xcite , comet @xcite and met-ricx @xcite . the feasibility of prompting general-purpose llms to score translations has also been studied @xcite @xcite and this approach has been shown to be competitive with fine-tuned dedicated models, especially in system-level performance. among recent methods, there has been an increasing proportion of metrics providing structured @xcite @xcite or natural language explanations @xcite for the predicted scores, most of which are based on llms. since the era of lexical metrics, which require one or more reference translations to evaluate a machine translation, metrics have evolved to rely increasingly more on the source segment @xcite @xcite . this is enabled by the multilingual pretrained models they are typically built on top, such as xlm-r @xcite or mt5 @xcite . in fact, referencefree (or quality estimation; qe) metrics do not lag far behind their reference-based counterparts anymore, as evidenced by the most recent wmt metrics shared tasks @xcite . that being said, high-quality references do provide added value to automatic metrics in most cases, helping them make even more accurate quality predictions. therefore, metrics nowadays typically employ a unified (or hybrid) input approach, allowing them to make a reference-based prediction whenever a reference is available, and a qe prediction otherwise @xcite @xcite . this is the case with our primary metricx-25 and gemspaneval submissions as well. current approaches to error span annotation are often based on encoder-only models predicting error severities per token. one such approach is cometkiwi @xcite or, more recently, xcomet @xcite . in last year's error span detection shared task, cometkiwi was shown to be a competitive baseline, coming in ahead of the (single) submitted system @xcite . @xcite showed that gpt-4 can simply be prompted to produce mqm error spans, denoting the method gemba-mqm. @xcite showed that fine-tuning for the generative error span prediction can improve performance compared to prompting llms alone. recent translation-oriented llms, such as tower @xcite also include generative error span prediction as part of the supported tasks in the training data. ## conclusion we introduced metricx-25 and gemspaneval, our submissions to the wmt25 evaluation shared task, both built upon the gemma 3 foundation model, but in very different ways. we demonstrated that metricx-25, adapting gemma 3 to an encoder-only architecture, can be trained to effectively predict esa and mqm quality scores and significantly outperforms its predecessor, metricx-24, in segment-level performance. for error span detection, our generative model, gemspaneval, proved to be competitive with a strong sequencetagging baseline. additionally, we showed how error span context can be used to identify unique error spans. our work demonstrates that a strong multilingual foundation model, such as gemma 3, can successfully be used for both regression-based and generative translation evaluation metrics.