Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! (2024)

Abstract: Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics, ranking them according to their correlation with human judgments. Their results guide researchers toward enhancing the next generation of metrics and MT systems. With the recent introduction of neural metrics, the field has witnessed notable advancements. Nevertheless, the inherent opacity of these metrics has posed substantial challenges to the meta-evaluation process. This work highlights two issues with the meta-evaluation framework currently employed in WMT, and assesses their impact on the metrics rankings. To do this, we introduce the concept of sentinel metrics, which are designed explicitly to scrutinize the meta-evaluation process’s accuracy, robustness, and fairness. By employing sentinel metrics, we aim to validate our findings, and shed light on and monitor the potential biases or inconsistencies in the rankings. We discover that the present meta-evaluation framework favors two categories of metrics: i) those explicitly trained to mimic human quality assessments, and ii) continuous metrics. Finally, we raise concerns regarding the evaluation capabilities of state-of-the-art metrics, emphasizing that they might be basing their assessments on spurious correlations found in their training data.

## introduction over the past few years, the machine translation (mt) field has witnessed significant advancements, largely driven by the advent of neural architectures, with the transformer @xcite being the most notable. modern mt systems deliver mostly fluent and accurate translations, posing a challenge for their quality evaluationeven when conducted by human annotators, especially those who lack professional training @xcite . under these circumstances, shallow overlap-based metrics are gradually being replaced by neural-based metrics, which demonstrate a better correlation with human judgments @xcite . however, a significant limitation is that most neural-based metrics are black-box systems trained to predict human judgments in the form of scalar scores, and typically do not provide justifications for their assessments. besides rendering them challenging to interpret, such opacity also complicates their meta-evaluation. in this respect, we found that certain strategies for the assessment of mt metrics' capabilities -which have recently been employed in the context of the metrics shared task at the conference on machine translation (wmt) foot_0 -favor specific metric categories and potentially encourage undesirable metrics behavior. to demonstrate these problems, we introduce the concept of sentinel metrics, i.e., a suite of metrics serving as a probe to identify pitfalls in the meta-evaluation process. sentinel metrics are either trained with incomplete information -which makes them inherently unable to evaluate the quality of machine-translated text properly -or consist of variations of existing metrics -which have been devised to expose specific issues in the meta-evaluation. as an example, in as can be seen, sentinel cand ranks in the upper half. sentinel cand is a sentinel metric designed to assess the quality of a candidate translation based solely on the translation itself, without accessing its source sentence or any reference translation. arguably, such a metric should only be capable of evaluating a translation's fluency, but not its ad- in this work, we: i) illustrate the issues that affect the segment-level meta-evaluation measures used in wmt23, demonstrating their impact experimentally with the help of sentinel metrics; ii) propose solutions for addressing these issues; iii) raise concerns regarding the reliability of state-ofthe-art mt metrics. we publish the code to reproduce our work and the weights of the sentinel metrics at https://github.com/sapienzanlp/ guardians-mt-eval . ## the meta-evaluation of mt metrics yearly, the wmt metrics shared task organizes a competition among metrics, including participants' submissions and baselines, to identify the metric that most closely aligns with human judgments. historically, the organizers have employed correlation with human judgment as a meta-evaluation strategy. recently, significant efforts have been made to refine the meta-evaluation process, encompassing the adoption of new measures, such as those proposed by @xcite and @xcite , and the introduction of the challenge sets sub-task @xcite , among other initiatives. in this section, we provide an overview of wmt's official meta-evaluation setting. first, multiple mt systems are employed to translate source segments found in one or more test datasets. 2 consequently, test datasets contain several translations of the same source segment. second, a manual evaluation campaign is carried out to assess the quality of all translations. finally, metrics' capabilities are assessed based on their alignment with human judgments, which are in the form of scalar scores. such alignment is typically estimated using correlation and accuracy measures. specifically, metrics are evaluated at two granularity levels: at both granularity levels, metrics can be evaluated using several statistical methods, such as the kendall τ and pearson ρ correlation coefficients, which have traditionally been applied at the segment and system levels, respectively. a final metrics ranking is derived by aggregating results from all the chosen statistics. for example, at wmt23, the final ranking was computed from the following three statistics: in this work, we identify two critical issues related to the second and third statistics, and provide the following recommendations to address them: in the following two sections, we provide an overview of some of the aforementioned statistics, illustrate their flaws, and demonstrate their impact by leveraging our sentinel metrics. at early editions of the wmt metrics shared task @xcite @xcite , human assessments were collected in the form of relative rankings (rr). specifically, the annotators were tasked to rank up to 5 translations of the same source sentence, produced by different mt systems. from each ranking, up to 10 pairwise comparisons were extracted. despite metrics assessments being scalar scores -which theoretically enabled the comparison of all pairs of translated segments -correlation was measured only on those pairs of translations for which rr annotations were available. therefore, only translations of the same source sentence were compared. later on, at subsequent editions of wmt, new techniques for human evaluation were adopted: first, direct assessments @xcite -where annotators rate individual translations on a scale from 0 to 100 -then, multidimensional quality metrics @xcite -where annotators tag the spans of a translation that contain errors, specifying their category and severity. with both the new annotation schemas, each translated segment was assigned a scalar quality score independently of the other translations, foot_4 which made it possible to compare all translations, not only those of the same source sentence. this new possibility raised doubts regarding the best way to compute the correlation between metrics and human assessments. indeed, it could be computed using all translations at once -no grouping -or by first grouping translations based on either their source segment -segment grouping -or the system that produced them -system grouping -and then returning the average correlation of these groups. at the wmt21 metrics shared task, @xcite chose the no grouping strategy, arguing that the other options would provide only a partial view of the overall picture. at wmt22, all three grouping strategies were used @xcite , and later at wmt23, @xcite chose no grouping again. although no grouping is the only strategy that assesses the mt metrics' ability to discern between higher-and lower-quality translations in absolute terms, irrespective of the source segment or mt system, we show that both no grouping and system grouping may introduce unfairness and favor trained metrics over the rest. ## the evaluation of ties in this section, we focus on the third statistic among those described in section 2, i.e., the segment-level pairwise ranking accuracy with tie calibration, dubbed acc eq by @xcite . prior to wmt23, the organizers of the metrics shared task used to employ the kendall τ coefficient -which is a statistic used to estimate the rank-based agreement between two sets of measurements @xcite -to measure the correlation between metrics and human judgments at the segment level. @xcite pointed out that the kendall τ coefficient does not account for metrics correctly predicting ties, 8 and introduced acc eq to address this issue. unfortunately, our analysis indicates that acc eq inadvertently compromises evaluation fairness in order to accommodate ties, ultimately biasing the results in favor of continuous metrics foot_9 over discrete ones. ## conclusion in this work, we identified two issues with the current meta-evaluation of machine translation, as conducted at the metrics shared task of the conference on machine translation. we proposed a suite of sentinel metrics designed to highlight these issues and demonstrate their impact on the metrics rankings, revealing that certain metric categories are unfairly advantaged. indeed, the none grouping and system grouping strategies favor trained metrics over overlap-and llm-based ones and the algorithm of tie calibration favors continuous metrics over discrete ones, or vice versa, depending on the percentage of tied assessments in the dataset used for it. specifically, continuous metrics are favored if the tie calibration is conducted on the same test set used for the evaluation. finally, we observed a notably high correlation between sentinel metrics and state-of-the-art metrics, raising concerns about their reliability and suggesting that their assessments might be based on spurious correlations present in the training data.