Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024)

Abstract: Machine Translation (MT) evaluation metrics assess translation quality automatically. Recently, researchers have employed MT metrics for various new use cases, such as data filtering and translation re-ranking. However, most MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. Moreover, MT metrics’ capabilities have historically been evaluated using correlation with human judgment, which, despite its efficacy, falls short of providing intuitive insights into metric performance, especially in terms of new metric use cases. To address these issues, we introduce an interpretable evaluation framework for MT metrics. Within this framework, we evaluate metrics in two scenarios that serve as proxies for the data filtering and translation re-ranking use cases. Furthermore, by measuring the performance of MT metrics using Precision, Recall, and F-score, we offer clearer insights into their capabilities than correlation with human judgments. Finally, we raise concerns regarding the reliability of manually curated data following the Direct Assessments+Scalar Quality Metrics (DA+SQM) guidelines, reporting a notably low agreement with Multidimensional Quality Metrics (MQM) annotations.

## introduction over the past few years, machine translation (mt) evaluation metrics have transitioned from heuristicbased to neural-based, enabling a more nuanced evaluation of translation quality and a greater agreement with human judgments @xcite . additionally, recent metrics shared tasks at the conference on machine translation @xcite have seen the rise of reference-free metrics, which assess translation quality without the need for human-curated * equal contribution. @xcite , metricx-23-qe-xl @xcite , and gemba-mqm @xcite for the provided machine-translated text. in this work, we address these issues by introducing a novel and more interpretable evaluation framework for mt metrics, comprising evaluation setups designed as proxies for new metric use cases. in the following sections, we first illustrate the problem of interpretability, then introduce our framework, and finally present our results. ## the interpretability of mt metrics' assessments in the field of ai, interpretability is defined as "the ability to explain or to provide the meaning in understandable terms to a human" @xcite , and typically refers to the problem of understanding the decision-making process of an ai model. however, our goal here is less ambitious. instead of focusing on the interpretability of mt metrics themselves, we are concerned with the interpretability of their assessments. specifically, most state-of-the-art mt metrics are trained to minimize the mean squared error (mse) with human judgments and return assessments as scalar quality scores, which are difficult to interpret. therefore, we are interested in understanding the meaning of these scores, rather than the internal workings of mt metrics. in light of this, we attribute mt metrics assessments' lack of interpretability to three main factors: foot_1 1. range consistency: it is unclear whether a difference in metric score has the same meaning if it occurs in different regions of the score range. in simpler terms, let us consider the example of figure 1 . due to the lack of error attribution we do not know which translation errors, if any, led comet @xcite to return 0.86. also, the metric comes with no range consistency guarantee, e.g. whether 0.86 is twice as good as 0.43. furthermore, different metrics have different score ranges, making it difficult to compare the assessments from comet with those of the other mt metrics in the figure. finally, lacking a clear understanding of comet's performance beyond human correlation, we cannot be sure whether we can draw conclusions from its assessments safely. for this reason, some efforts have been made to design interpretable metrics. for example, among the primary submissions to the wmt23 metrics shared task @xcite , matese @xcite annotates the spans of a translation that contain errors, specifying their severity, xcomet models @xcite return annotated error spans together with a final regression value, and gemba-mqm @xcite leverages @xcite to produce detailed quality assessments. however, these metrics compromise on other aspects to accommodate the increased interpretability. matese displays a lower correlation with human judgment than several regression-based metrics. 3 trading off performance and interpretability, xcomet models' final assessment is based mainly on the regression component, with annotated spans contributing only 2/9 of the overall score. finally, gemba-mqm is prohibitively expensive to operate and its assessments are not fully-reproducible, due to its dependence on the closed-source gpt-4. in this work, we seek to mitigate the interpretability issue by targeting the problem of performance. specifically, we take inspiration from two popular new mt metrics applications, i.e., data filtering and translation re-ranking, in order to study and measure metrics performance in terms of precision, recall, and f -score. taking advantage of these measures that are more transparent than correlation, we aim to shed light on the meaning and reliability of metrics assessments, especially concerning such new mt metrics use cases. we release our evaluation framework as software at https://github.com/sapienzanlp/ interpretable-mt-metrics-eval . ## an interpretable evaluation framework for mt metrics two popular new mt metrics applications are data filtering and translation re-ranking. in data filtering, mt metrics separate good-quality from poorquality translations. after choosing a threshold value, all translations below the threshold are labeled as poor quality and discarded. in this respect, we are interested in jointly assessing metric performance and studying the meaning of metric scores, finding the thresholds that best separate good-quality ( good ) from poor-quality ( bad ) translations. instead, in translation re-ranking, mt metrics determine the best in a pool of translations of the same source text. for example, in qe reranking and mbr decoding, metrics are tasked to identify the best translation among those sampled from an mt system. with the aim of facilitating practitioners in making design choices for these metrics applications, and with a focus on the interpretability issue, we evaluate mt metrics performance in two settings: i) when metrics are used as binary classifiers, tasked to separate between good and bad translations (acting as a proxy for the data filtering application), and ii) when metrics are used to identify the best translation in a group of translations of the same source (acting as a proxy for translation reranking). ## experimental setup this section outlines the data employed, the metrics evaluated, and our methodology. implementation details regarding the calculation of precision, recall, and f -score are in appendix a. ## results in this section, we report the performance obtained by mt metrics when used as binary classifiers to distinguish between good and bad , as well as perfect and other translations, and in terms of their effectiveness in translation re-ranking, i.e., in selecting the best translations among candidates for the same source text. ## related work previous studies have focused primarily on the problem of error attribution. specifically, the shared task on quality estimation at wmt investigated the ability of mt metrics to predict wordlevel annotations @xcite . 2023 ) proposed metrics that address the lack of error attribution by providing explanations in the form of either span-level annotations or natural language rationales. furthermore, recent studies have introduced dedicated benchmarks to investigate the impact of specific translation errors, such as disambiguation errors @xcite and wrongly translated named entities @xcite . in a different vein, and closer to our work, some studies explored the meaning of raw metrics scores in terms of their alignment with human judgments. mathur et al. (2020a) studied the meaning of system-level score deltas for bleu @xcite , showing that a statistically significant increase of 0-3 bleu points corresponds to significantly better mt systems less than half of the time, in terms of human judgments. similarly, @xcite investigated the relationship between mt metrics' system-level score deltas and human proximation of sampling from a single system. we delve into the differences between mbr decoding and our evaluation setup in the limitations. judgments. finally, in a recent study, @xcite evaluated mt metrics' ability to assess high-quality translations by examining their correlation with human judgments, as well as their precision, recall, and f -score, using a setup similar to ours. however, instead of calculating metrics thresholds from data, they arbitrarily assumed that a metric indicates high-quality only if its normalized assessments fall within the [0.99, 1.00] interval. in contrast, we measure metrics performance in data filtering without making assumptions about the meaning of their assessments, aiming to understand this meaning through the evaluation itself. ## conclusion in this work, we introduce a novel evaluation framework for mt metrics. within this framework, we measure metrics performance in i) binary classification, i.e., distinguishing between good and bad , and perfect and other translations, and ii) in a proxy scenario for translation re-ranking, selecting the best among the translations of the same source text. by measuring performance in terms of precision, recall, and f -score, we fulfill a dual purpose. first, we offer a more intuitive interpretation of metrics' capabilities, as compared to correlation with human judgment, and second, we provide concrete user recommendations concerning novel mt metric use cases. we find that mt metrics perform relatively well in distinguishing between good and bad translations, but struggle with precision, especially when dealing with higher-quality translations like in the perfect vs other scenario. our results show that metricx-23-qe-xl is the best openly available metric for data filtering applications, while metricx-23-xl and comet achieve the highest performance in translation re-ranking. additionally, we demonstrate that reference-based mt metrics, when used as the utility function in an mbr decoding-like scenario, outperform reference-free ones, suggesting that mbr decoding may be superior to qe re-ranking. finally, we report notably poor performance for da+sqm annotations used as a metric within our evaluation framework, raising concerns about its reliability.