Nvidia-Nemo’s WMT 2025 Metrics Shared Task Submission (2025)

Abstract: This paper describes Nvidia-Nemo’s WMT 2025 Metrics Shared Task submission. We investigated two strategies for extending Machine Translation (MT) evaluation to unsegmented documents: 1) first segmenting into sentences and then applying regression-based metrics and 2) directly utilizing the long-context capabilities of LLMs. The base comparison of the segmentation-based and LLM-based metrics on the WMT 2023-24 evaluation sets indicated that the former performs more robustly across language pairs.Thus we sought to improve the LLM-based approach by incorporating relative evaluation - this setting jointly evaluates all candidate translations at once and relative to each other, rather than evaluating each separately. Our experiments using the open-source Qwen3 LLM show that relative evaluation improves score correlations with human judgment, but only if the task is structured as a 2-stage evaluate-then-refine problem.

## introduction for most of its history, machine translation (mt) research has revolved around the sentence as the primary unit of translation and evaluation. standard mt benchmarks and evaluation protocols typically present systems with short, isolated segments, and assess their quality against human-produced references @xcite . this sentence-level paradigm has been driven by practical constraints: statistical and neural mt systems were long limited by both computational costs and modeling power @xcite . and in turn, automatic metrics, whether lexical @xcite or regressionbased @xcite , were also designed to operate on sentence-like segments. however, translation in real-world applications rarely occurs in isolation. professional translators work with continuous documents, where meaning and style are shaped by discourse, context, and document-level coherence @xcite . as neural mt systems have become more capable, particularly with the advent of large language models (llms), research has increasingly shifted toward document-level translation and evaluation @xcite . this shift reflects both a practical need for evaluating translations as they appear in natural, unsegmented contexts and a growing recognition that many important aspects of quality, such as pronoun resolution, lexical consistency, and narrative flow, can only be judged when viewing the text holistically. this recent emergence of long-context mt models has made it technically feasible to translate much larger spans of text at once, raising a new research question in the field of mt evaluation: is it better to apply existing sentence-level metrics to segmented text or apply llm-based metrics directly to document-length text? in the wmt 2025 metrics shared task, we focus specifically on the problem of unsegmented document evaluation: assessing the quality of mt outputs when the entire document is presented as a single sequence. our submission compares two divergent strategies: (1) applying traditional regression-based metrics after segmenting documents into sentences, and (2) leveraging the longcontext capabilities of llms to perform holistic document evaluation directly. to support our investigation, we simulated document-level human evaluations by concatenating the sentence-level mqm annotations from the wmt 2023 and 2024 metrics shared tasks. our experiments showed that the scaled up regression-based metric correlated better with human judgement than the llm-based metric -we thus designed the former as our primary submission and the latter as our secondary submission to the wmt 2025 metrics shared task. while our primary findings favored the segmentation-based regression metric, we also explored ways to strengthen the llm-based approach. in particular, we experimented with relative evaluation -a setup in which all candidate translations for a given source are presented to the model simultaneously and ranked against each other, rather than scored in isolation. we found that relative evaluation yielded only marginal improvements in correlation with human judgments, and only when implemented as a two-stage "evaluate-thenrefine" process. although these gains were not large enough to change our overall ranking of the two strategies, this line of work remains relevant for understanding how llm prompting strategies interact with long-context mt evaluation. we therefore complement our main results with an analysis of why relative evaluation showed limited benefits and where it may still hold promise. ## document-level human evaluation prior to 2025, the wmt shared tasks were limited to the segment-level translation paradigm -all mt systems and all human evaluation operated on segments. to study document-level evaluation, we construct a simulated document-level mqm set by concatenating the source and target segments of each document and summing their mqm error counts. while this offers the best available proxy for document-level score correlations on wmt data, it has two notable limitations: note that we do not simulate document-level evaluation for the wmt 2024 sets because a large portion of segments were not evaluated. we also do not simulate document-level esa data, as its 0-100 scale is not naturally additive, unlike mqm error counts. we therefore focus on the three language pairs from wmt 2023: en-de, he-en, and zh-en. 3 metric descriptions ## experimental setup data: we use evaluation data from the wmt 2023-2024 metrics shared tasks: mqm (both years) and esa (2024). these are inherently segment-level corpora: translations were produced from pre-segmented sources, and human ratings were applied to individual segments. models: for llm-based metrics, we use qwen3-14b @xcite in our experiments. the qwen3 series is a hybrid reasoning/instructionfollowing model, capable of scaling inference-time compute via the generation of reasoning traces; however, we found that the reasoning mode degraded performance, so we disabled it in our experiments. in our attempt to identify the best opensource llm, we also experimented with llama3.1-8b @xcite and gemma-3-12b-it (team, 2025a) -these were less performant than qwen3, as described in the next section. meta-evaluation: for measuring score correlation with human judgment we rely on kendall's tau, as computed by the official wmt repository: https://github.com/google-research/ mt-metrics-eval . we track both system-level and segment/document-level score correlations. ## conclusion we investigated two strategies for unsegmented document-level mt evaluation: scaling traditional regression-based metrics to longer contexts and applying llm-based metrics capable of holistic