MSLC 25: Metric Performance on Low-Quality Machine Translation, Empty Strings, and Language Variants (2025)

Abstract: In this challenge set, we examine how automatic metrics for machine translation perform on a wide variety of machine translation output, covering a wider range of quality than the WMT submissions. We also explore metric results on specific types of corner cases, such as empty strings, wrong- or mixed-language text, and more. We primarily focus on Japanese–Chinese data, with some work on English and Czech.

## introduction this paper describes a challenge set submitted to the challenge set subtask of the (unified) machine translation evaluation shared task at the 2025 conference on machine translation (wmt); we focus primarily on segment-level quality score prediction with a brief preliminary note on word-level error detection and span annotation. for this third iteration of the metric score landscape challenge (mslc), 1 we run a smaller set of experiments. we once again focus on @xmath0. we also include analysis of empty strings (as in past iterations), mixed-and wrong-language text, and a preliminary note on english spelling variation; for these tasks, we use a mix of @xmath1, english language data, and @xmath2. our approach, particularly for low-to mid-quality mt and for empty strings, demonstrates a low-cost way to test metrics on the wider quality landscape. we encourage developers of metrics to run such evaluations themselves prior to releasing metrics. for developers of metrics who are unable to run such evaluations themselves, we call on them to explicitly declare that their released metrics have not been 1 mslc data and additional figures can be found at https: //github.com/nrc-cnrc/mslc . ## prior and related work this work describes the third mslc challenge set. in @xcite , the first iteration, our intent was to focus primarily on low-and mid-quality mt output across four language pairs. the process of evaluation that year brought to light two other issues: the scores that metrics assigned to empty strings (as some mt system submission that year included empty strings, providing a natural experimental set) and "universal scores" (scores that were assigned very frequently by certain metrics) similar to the "universal translations" described in @xcite . the second iteration of mslc, @xcite , continued the work on low-and midquality mt, with the addition of experiments on empty strings, mixed-and wrong-language text, and language variants (in that instance, spanish language terminology that differs across language variants). in concurrent work to @xcite proposed comet-specific mitigations to many of the issues observed in both papers: incorporating language id in order to mitigate issues with mixed-or wrong-language text, using signatures in the spirit of sacrebleu @xcite to help explain variations in metric output, and the issue of empty strings. there is a tradition of challenge sets targeting specific linguistic phenomena for mt @xcite @xcite @xcite , while challenge sets for evaluation (i.e., challenge sets targeted at metrics) tend to be relatively newer (see, i.a., the descriptions of the challenge sets at the metrics shared tasks: @xcite @xcite . of these challenge sets, @xcite also explore wrong-language text (among 68 phenomena across a large number of language pairs in the aces challenge set), noting particular issues for reference-free metrics, similar to our observations. our work is situated more broadly in the area of mt evaluation and corner cases for mt evaluation. while there is prior work focusing on specific metrics and corner cases @xcite @xcite , submitting this challenge set to the shared task permits us to examine and compare performance on corner cases and mt quality ranges across metrics in a controlled environment. importantly, readers should note that this paper is a description of a challenge set submitted to a shared task, rather than a complete, in-depth exploration of all the areas on which it touches. we note, in both the limitations section and throughout the work, that there are some components that represent established evaluations (mslc-a and the empty strings work) while other components are presented as initial proof-of-concept experiments to determine whether future in-depth evaluation is indicated (these smaller preliminary experiments should not be used to draw sweeping conclusions). ## data the mslc challenge set is divided into two main components: mslc-a (which covers low-quality and mid-quality mt) and mslc-b (which targets specific corner cases and potential challenges for metrics). ## metrics we focus on analyzing the scores produced by the baseline metrics and the primary submissions. there are 9 baseline metrics (including 2 "sentinel" metrics designed to scrutinize the metric meta-evaluation process) and 5 primary metrics that participated in portions of our challenge set for segment-level evaluation. one baseline and 2 primary metrics participated in the error span detection portion of our challenge set. the segment-level score baselines are bleu @xcite , spbleu (nllb @xcite , chrf @xcite , bertscore @xcite , comet-22 @xcite , cometkiwi @xcite , yisi-1 @xcite , sentinel-cand and sentinel-src @xcite . for the segment-level quality score prediction task, five metrics participated in our experiments. metricx-25 @xcite , an updated version of metricx, is an encoder-only regression model initialized from gemma 3 @xcite 12b and fine-tuned on publicly available da and mqm scores from wmt 2015-23. mr7.2.1 @xcite is based on the gemma 3 27b it model and is prompted with the dspy @xcite framework and its miprov2 optiboth the english and czech sides of the sentence pair). mizer. polycand-2 @xcite is a cometbased metric that incorporates two alternative translations of the same source segment (provided by other translation systems) to better contextualize and assess the quality of the translation being scored. rankedcomet (maharjan and shrestha, 2025) is a comet-based metric post-processed with rank normalization for each language pair. uva-mt (wu and monz, 2025) calibrates quality estimation and likelihood on the gemma 3 12b it model, then directly uses the token average likelihood as a metric for quality estimation. for the error span detection task, three systems participated in our mslc-b experiments: baseline xcomet (guerreiro et al., 2024) and the two primary submissions of aip1 @xcite and gemspaneval @xcite . aip1 uses the openai o3 (openai, 2025) reasoning model and its structured-output mode to detect translation errors at the span level. gemspaneval is based on the gemma 3 27b model and is finetuned to predict mqm error spans. some metrics use reference translations in the process of producing their scores, while others do not. throughout the remainder of the paper we indicate the 3 baseline and 4 primary reference-free (qe) metrics-those that do not use the reference translation in their scoring-with an asterisk before the metric name. there are 3 reference-free primary metrics (*mr7.2.1, *polycand-2, *uva-mt) in the segment-level score prediction task and 1 (*aip1) in the error span prediction task. ## mslc-a results and plots interpretation note of caution: our submitted mslc-a challenge set was scored at the document level by the automatic metrics, while the submitted primary mt systems from the general task were scored at a sub-document segment level by the automatic metrics. in order to be able to compare these, we have averaged the segment-level scores to produce a document-level score for each document. we note that this may not always be identical to the score that the metric would have assigned had it scored the full document directly, and caution should therefore be taken when drawing conclusions. figure 1 shows system average scores for the mslc-a systems (cool colours, left) and the systems submitted the the wmt general mt task, computed only over the news data. these aver-ages are computed from the document-level scores, which in the case of mslc-a data were produced directly by the metrics and which in the case of the submitted systems were produced as an average of segment-level scores. we find several points of interest. three of the metrics, *cometkiwi22, metricx-25, and *uva-mt have some difficulty with the rankings of the lowest-quality mslc-a systems. in particular, *uva-mt ranks the worst system (whose output is almost entirely nonsensical and unreadable) as similar in quality to the high-quality wmt submissions. this indicates that these metrics-for this language pair at leastmay not be trustworthy metrics to use when trying to evaluate low-quality mt. metric users should consider alternative choices of metrics if they have reason to believe that their mt output may be of low quality or mid-range quality. this could also have an impact on mt systems trained using these metrics, particularly in the early stages of training. we also observe that some metrics (*polycand-2, comet22, i.a.) devote a very small portion of their metric's space of possible scores to the high-quality systems, with a wider range of scores for the low-quality systems, while others like chrf distribute the score range more evenly. this may impact how useful a metric is for distinguishing between systems of different levels of quality, and may also play a role in human interpretation of metric score differences; for broader discussion of metric score differences and human interpretations thereof, see @xcite and @xcite , i.a. there is not an inherent right answer to how a metric should use its score space, rather, it is tied to the intended use of the metric. we observe that some metrics show overlap or near overlap between the best of the mslc systems and the lowest-scoring of the submitted systems while others show a large gap between them. since we do not have full human annotations available, we cannot make any claims about whether there would be an overlap or a gap based on human evaluation. figure 2 provides another way to visualize the metric scores, showing histograms of scores assigned to each system along the diagonal, and scatterplots showing correlations between metrics on the off-diagonals. the metrics that give higherthan-expected scores to the low-quality systems once again stand out where they do not correlate with systems that rank the low-quality systems as expected. we can also use the histograms to gain a better understanding of the score distributions assigned to the data, such as the very low scores and comparatively small score range assigned by rankedcomet to the low-quality systems, as com-pared to other metrics, or the somewhat bimodal score distribution from *mr7.2.1. in past iterations of mslc, this type of figure was particularly useful in highlighting unusual properties of some of the metrics, such as discretizing the score space or assigning specific scores very frequently ("universal scores"). we do not observe such results in this year's set of metrics. due to scheduling constraints, we did not receive the human annotation scores in time to display those in these figures; we plan to incorporate them into final additional figures on the mslc website ( https://github.com/nrc-cnrc/mslc ). ## conclusions we observe similar results to past mslc experiments, with some metrics struggling to accurately score extremely low-quality (nonsensical) mt outputs. we continue to encourage discussion around how metrics should score empty strings and encourage additional analysis of how this does or does not correlate with broader metric sensitivity to overgeneration and undergeneration. as we see more metrics shifting to use multilingual embeddings our large language models and more reference-free metrics, we encourage metric builders to consider how to incorporate information about the intended target language into their metrics (see also, @xcite ). while it may be easy for a human-even one who cannot read the languages in questionto tell if an mt system has erroneously generated english when chinese was expected, it may not be so simple for more similar language pairs. we would encourage metric builders to consider how to incorporate intended target language into their systems, and note that this may be an area where ignoring available references may have a real cost when it comes to metric trustworthiness.