U v A - MT at WMT 25 Evaluation Task: LLM Uncertainty as a Proxy for Translation Quality (2025)

Abstract: This year, we focus exclusively on using the uncertainty quantification as a proxy for translation quality. While this has traditionally been regarded as a form of unsupervised quality estimation, such signals have been overlooked in the design of the current metric models—we show their value in the context of LLMs. More specifically, in contrast to conventional unsupervised QE methods, we apply recent calibration technology to adjust translation likelihoods to better align with quality signals, and we use the single resulting model to participate in both the general translation and QE tracks at WMT25.Our offline experiments show some advantages: 1) uncertainty signals extracted from LLMs, like Tower or Gemma-3, provide accurate quality predictions; and 2) calibration technology further improves this QE performance, sometimes even surpassing certain metric models that were trained with human annotations, such as CometKiwi. We therefore argue that uncertainty quantification (confidence), especially from LLMs, can serve as a strong and complementary signal for the metric design, particularly when human-annotated data are lacking. However, we also identify limitations, such as its tendency to assign disproportionately higher scores to hypotheses generated by the model itself.

## introduction in this paper, we describe the details of our submission to the wmt 2025 mt evaluation subtask-1, i.e., segment-level quality estimation (qe), which includes 16 translation directions. this year, we focus exclusively on using the uncertainty quantification as a proxy for translation quality. while this has traditionally been regarded as a form of unsupervised quality estimation @xcite , such signals have been overlooked in recent designs of metric models. in this competition, we aim to examine the strengths and weaknesses of this signal. previous unsupervised quality estimation focused on using the model's internal information to quantify the confidence/certainty of a given translation sentence pair, e.g., using likelihood, entropy, or uncertainty signals under a monte carlo (mc) dropout framework @xcite . notably, they are relying on signals derived from the model itself and are mostly training-free. we apply recent calibration technology @xcite for this year's competition. unlike traditional unsupervised qe, this method aims to calibrate translation likelihood with quality during training time. by extensive experiments, several key advantages of calibrated models can be shown as follows: our offline experiments show that the resulting model's qe ability sometimes even surpasses some accurate metric models, like cometkiwi-22 @xcite foot_0 , without using any human-annotated data. we therefore argue that uncertainty quantification, especially from llms, can serve as a strong and complementary signal for the metric design, particularly when human-annotated data are lacking. however, our preliminary tests on this year's wmt25 test set also reveal limitations in using uncertainty signals for quality estimation-notably, they tend to assign significantly higher scores to hypotheses generated by the model itself-a pattern that is, to some extent, consistent with recent findings that llm-as-a-judge systems tend to favor their own outputs @xcite . in the next sections, we will briefly describe: 1) the framework of calibration, 2) offline experiments for both translation performance and our qe strategy by using the calibration approach, 3) the implementation for our submission at the wmt25-qe track, and 4) a discussion of the strengths and limitations of using model confidence for quality estimation. ## calibration method in this section, we briefly describe the calibration method to keep the paper self-contained; please refer to @xcite for details. formally, given a parameterized auto-regressive language model p θ and a translation instruction x, the log-likelihood of a translation hypothesis y i is denoted as z θ (y i |x) = log p θ (y i |x). meanwhile, the quality of this translation can be defined as q(y i |x) where q represents any external quality evaluation model. when sampling hypotheses y from p θ conditioned on x, both z θ (y|x) and q(y|x) can be viewed as random variables defined over the output space. this method is to calibrate the likelihoods of generated hypotheses with their quality to maximize the correlation between z θ (y|x) and q(y|x). the calibration method uses the statistic, pearson correlation coefficient ρ(a, b), to quantify the correlation. let a, b : @xmath0. the corresponding pearson score between a and b is given by the calibration method calculates and optimizes ρ with respect to the likelihood of hypotheses z θ (y|x) and the quality score q(y|x). practically, given the intractably large decoding space, it employs monte carlo sampling for approximation. for each source sentence x, we generate k hypotheses y i (@xmath1, . . . , k}) by repeatedly prompting a large language model θ with nucleus sampling, and compute the corresponding z θ (y i |x) and q(y i |x), and estimate the corresponding µ z , µ q and σ z , σ q . accordingly, we define the pearson-based loss using estimates under the nucleus-induced distribution p as follows: an off-policy formulation can be obtained by trivially replacing the current model p θ with an external model p θ * for sampling. overall, by minimizing l cal , we encourage the pearson score between z and q to increase. in practice, we use a gradient-based optimizer, adam, to optimize θ for this goal, with gradients propagated through z θ , µ z , and σ z . despite its simplicity, several important characteristics are captured in this formulation: ## offline evaluation in this section, we demonstrate the effectiveness of the calibration method by applying it to the strong llm-based translation system, i.e., tower @xcite . we briefly show this method's capabilities in both (1) translation quality optimization and (2) quality estimation optimization. note that our online submission system is based on gemma-3-12b (see section 4), as it covers more languages for wmt25 than tower. ## implementation of our wmt25 submission in this section, we describe the implementation of our qe system for wmt25. to cover more languages, we use gemma-3-12b as the base model instead of tower, as noted in section 3.1. to construct the calibration dataset, similar to that in section 3.1, we feed the source segments in the wmt25 general translation test set into gpt-4o-mini, using the prompts provided with the official test set 5 , to generate 16 hypotheses per sample. note that we use the wmt25 blind test sets rather than flores. this choice is motivated by (1) better alignment with the domain used in testing and (2) consistency with wmt25's paragraph-level data format. following that in section 3.1, the corresponding hypotheses are decoded using nucleus sampling with a top-p of 0.98 and a temperature of 1.0. each sample is at the paragraph level, where "\n" re5 official prompts can be found here. mains in the original data as a separator. we also use cometkiwi-xxl to score each one-to-many translation pair in our synthetic dataset. we apply the calibration method @xcite , as we mentioned in section 2, as the only post-training method on gemma-3-12b. finally, the resulting model, trained on synthetic data derived from the wmt25 general translation test set, is used as a quality estimation model here. for each sample in the wmt25 qe dataset, we feed the source and target segments into our model (with the corresponding prompt) and directly use the average log-likelihood of the target segments as the quality assessment scores for submission. for the performance of our system on the wmt25-qe track, please refer to this year's findings paper, which has not yet been officially released at the time of this submission. ## on the limitation of using uncertainty as a proxy for translation quality although we demonstrate the effectiveness of using llm uncertainty as a proxy for translation quality in section 3.3, we also identify an important limitation-this method will give significantly higher scores for translations that are from itself-it favors its own output when it uses maximum a posteriori as the decoding rule. this issue was not identified by @xcite , as their qe testing set lacks such special conditions and does not include the model's own translation outputs during qe evaluation. meanwhile, we argue that this issue is likely to be a general limitation of most metrics based on translation uncertainty. more broadly, it to some extent aligns with observations about llm-as-ajudge @xcite , where llms tend to favor their own generations. lastly, we anticipate that the official wmt25 qe test set will be particularly challenging for our metric. as noted above, we use a single calibrated model to participate in both the general translation and quality estimation tasks at wmt25. therefore, if any hypotheses generated by our system appear in this year's qe test set, it is likely to assign them the highest scores. we have found indications of this, although we cannot confirm it because the official description of the test set has not yet been released.