R anked COMET : Elevating a 2022 Baseline to a Top-5 Finish in the WMT 2025 QE Task (2025)

Abstract: This paper presents rankedCOMET, a lightweight per-language-pair calibration applied to the publicly available Unbabel/wmt22-comet-da model that yields a competitive Quality Estimation (QE) system for the WMT 2025 shared task. This approach transforms raw model outputs into per-language average ranks and min–max normalizes those ranks to [0,1], maintaining intra-language ordering while generating consistent numeric ranges across language pairs. Applied to 742,740 test segments and submitted to Codabench, this unsupervised post-processing enhanced the aggregated Pearson correlation on the preliminary snapshot and led to a 5th-place finish. We provide detailed pseudocode, ablations (including a negative ensemble attempt), and a reproducible analysis pipeline providing Pearson, Spearman, and Kendall correlations with bootstrap confidence intervals.

## introduction machine translation quality estimation (qe) predicts the quality of a translation without reference texts. for many production environments, affordable and reliable qe is more valuable than marginal benefits from retraining large models. we therefore analyze whether a robust, publicly available metric model (unbabel/wmt22-comet-da; @xcite can maintain its competitiveness in 2025 when paired with a simple, computationally inexpensive post-processing step. our per-language rank-based calibration-rankedcomet-is model-agnostic, computationally efficient, and empirically effective: it improved aggregated pearson correlation on the preliminary codabench verification snapshot and was adequate to reach 5th place on that snapshot. to evaluate our approach, we benchmark it against alternative calibration techniques and offer a suite of diagnostics so others can replicate and extend our findings. ## related work neural evaluation metrics (comet and followups) are widely used for mt evaluation @xcite . calibration techniques (platt scaling, isotonic regression) are standard in classification/regression contexts @xcite . in qe, unsupervised and uncertainty-aware approaches have been addressed @xcite . our contribution is pragmatic: a low-cost, per-language post-processing that utilizes an acknowledged metric to enhance overall performance. ## analysis and discussion ranking calibration is consistent: ordinal relationships are maintained and tie behavior is regulated by the ranking method (we use average ranks). pearson changes because the transform replaces arbitrary raw spacing with a uniform rank spacing, often enhances linear alignment to human scores when raw scores are narrowly distributed. the variance analysis ( ## reproducibility all code, scripts, and csv outputs used to generate the figures and tables are provided in the public repository at https://github.com/ sujal390/rankedcomet-wmt25-emnlp . ## conclusion we demonstrate that a simple per-language rankmin-max calibration applied to a robust 2022 comet model yields a competitive qe submission on the preliminary codabench snapshot in the wmt 2025 preliminary evaluation snapshot. the approach is affordable, deterministic, and reproducible; our diagnostics show why it improves aggregated pearson (variance equalization) while preserving ordinal relations.