COMET -poly: Machine Translation Metric Grounded in Other Candidates (2025)

Abstract: Automated metrics for machine translation attempt to replicate human judgment. Unlike humans, who often assess a translation in the context of multiple alternatives, these metrics typically consider only the source sentence and a single translation. This discrepancy in the evaluation setup may negatively impact the performance of automated metrics. We propose two automated metrics that incorporate additional information beyond the single translation. COMET-polycand uses alternative translations of the same source sentence to compare and contrast with the translation at hand, thereby providing a more informed assessment of its quality. COMET-polyic, inspired by retrieval-based in-context learning, takes in translations of similar source texts along with their human-labeled quality scores to guide the evaluation. We find that including a single additional translation in COMET-polycand improves the segment-level metric performance (0.079 to 0.118 Kendall’s tau-b correlation), with further gains when more translations are added. Incorporating retrieved examples in COMET-polyic yields similar improvements (0.079 to 0.116 Kendall’s tau-b correlation). We release our models publicly.

## introduction there is a gap between how humans and automated metrics score translations. automated metrics receive the source segment, usually a sentence or a paragraph, a single translation, and optionally a reference translation. they are then tasked with assessing the quality of the translation. in contrast, human evaluation is less episodic. human raters often assess multiple translations in sequence @xcite @xcite , considering them side-by-side. even though annotations are made for each translation ⋆ equal contribution, sorted anti-alphabetically. 1 we release the paper code and pre-trained quality estimation models cometpoly-ic and cometpoly-cand. comet poly-cand score i enjoy apples! ich mag äpfel. i enjoy apples! ich mag äpfel! i enjoy apples! ich mag apples! i enjoy apples! ich freude äpfel. other candidates' translations i enjoy apples! ich mag äpfel! comet poly-ic i enjoy apples! ich mag äpfel! score i love apples. eu amor maçãs. 60 i love pears. ich liebe birne. 100 i love apples 95 i like him. já láska ho. 55 retrieval of relevant examples tôi thích táo. figure 1 : the comet poly-ic model consults a knowledge base of previously human-scored translations before assigning the quality estimation score to the candidate translation. the comet poly-cand considers other possible translations apart from the candidate one. both metrics work better than just providing the source and the translation. we present two conceptual approaches to address these two challenges by incorporating additional context into standard automated metrics, such as comet @xcite . our main motivation is to narrow the gap between human evaluation and automated metrics, enabling automated metrics to score translations in the context of other translations and making them more robust to out-ofdomain data. specifically, we introduce two models trained within this framework, comet poly-cand and comet poly-ic : we publicly release our models under open license, and submit our models to the wmt 2025 metrics shared task. ## methods in this section, we introduce the translation quality estimation task, review comet, and present two extensions for improved quality estimation and domain adaptation. ## experimental setup this section outlines the training and evaluation procedures, as well as the experimental setup. data. we use the direct assessment scorings of wmt up to 2023 (inclusive) for training (600k segments). for testing and evaluation, we use wmt 2024 (105 segments), which has been evaluated with the esa protocol @xcite . this dataset covers eleven language pairs: english to czech, german, spanish, hindi, icelandic, japanese, russian, chinese, czech to ukrainian, and japanese to chinese. from esa, we use the final scores (as opposed to error spans), which have the same scale as direct assessment. for mqm, we convert the error span annotations on a translation to the final score by taking 1 -(5 • major + 1 • minor)/100, where major is the number of annotated major errors, and minor is the number of minor errors annotated in the translation. in this way, the scores are aligned roughly on the same scale compared to da scores. training. we train the baseline comet model, comet poly-cand and comet poly-ic based on pretrained roberta @xcite on wmt human judgment data for five epochs. for comet poly-cand , we retrieve up to five candidate translations, either randomly or based on embedding similarity. for comet poly-ic , we retrieve up to five in-context examples from the training data based on embedding similarity. the metrics are trained in a maximally comparable model setup, which is detailed in appendix a. evaluation. we evaluate the metrics on the segment level in three ways: pearson correlation, @xcite . in contrast to @xcite we do not do perform any group-by-item nor group-by-item. results are macro-averaged across eleven languages. pearson correlation measures the linear relationship between metric scores and human ratings: higher values indicate better alignment, though not necessarily on the same scale. mean absolute error (mae), in contrast, captures the average absolute difference between metric and human scores, with lower values indicating closer agreement in both value and scale. kendall's tau-b focuses on rank correlation, reflecting how well the metric preserves the relative ordering of translations. while pearson and kendall's tau-b range from -1 to 1, mae is unbounded and depends on the scoring scale. experiments. to ensure a controlled evaluation setting, we first train a standard comet ## results and analysis in the following, we discuss and analyse the results of comet poly-cand and comet poly-ic , compare them to the non-parametric k-nn and the large parametric gemba model, and discuss the runtime impact of our method. ## related work this section reviews the broader context of automated metrics and human evaluations that use multiple inputs: either multiple translations or, more commonly, multiple references. automated metrics. early metrics like bleu @xcite and chrf @xcite operate at segment or corpus level and support multiple references but not multiple hypotheses simultaneously. comet @xcite trains an encoder for human-like quality assessment and supports a single reference. adding more references shows limited gains @xcite . closest to our work, dinh et al. ( 2024 ) propose a k-nn quality estimator similar to comet poly-ic , but aggregate train-test similarity of mt models as a quality indicator rather than having a separate qe model that assesses translations based on similarity and contextual relevance. moosa et al. (2024) introduce mt-ranker, which compares translation pairs and outputs a binary preference. with the rise of large language models (llms), an up-to-date approach for quality estimation is to use llm-as-a-judge. simply prompting llms to output the quality score of a translation has become the state-of-the-art approach, with the most prominent example of gemba @xcite . this approach has the potential to improve even further, by applying different strategies such as including in-context examples (few-shot judge), chain-of-thought prompting, pairwise comparison, as recommended by @xcite . human evaluation. human evaluation of machine translation takes many forms. for benchmarking, wmt initially used rankme @xcite , where annotators rank multiple hypotheses simultaneously. due to biases and high cognitive load, this shifted to single-hypothesis assessments such as direct assessment and its variants @xcite , multidimensional quality metrics @xcite , and error span annotation and its variants @xcite . despite judging one hypothesis at a time, annotators gradually see other translations during evaluation, implicitly calibrating their quality judgments. automated metrics, however, lack this contextual grounding and evaluate translations independently. ## discussion and conclusion recommendation. comet poly-cand can be applied in scenarios where multiple translations exist for the same source sentence, such as: (1) enchmarking various competing systems on the same test set (e.g., wmt general shared tasks), (2) comparing outputs from different checkpoints or models during mt development, or (3) cselecting the best translation from a pool of hypotheses during reranking for final output selection. the intended use of comet poly-ic is for quick domain adaptation without retraining the metric (appendix c.3). while different retrieval methods can cause slight variations in performance (see appendix c), it is crucial that the retrieval mechanism is deterministic to ensure reproducible scores. additionally, changing the retrieval mechanism or the set of previously annotated translations that are being retrieved instantiates a new metric with non-comparable scores to the previous evaluations. therefore, when using comet poly-ic , always disclose the retrieval set and retrieval method. training a smaller, specialized module with some tweaks (comet poly-cand ) can be beneficial compared to directly using large, general-purpose language models (gemba). we have shown that comet poly-cand can reach the performance of gemba, while being much more efficient in terms of inference time. submitted models. we submit the following models to the wmt metrics shared task 2025 and make them publicly available under open license (apache license 2.0) on hugging face. the models are trained on wmt data up to 2024 (inclusive).