Can Automatic Metrics Assess High-Quality Translations? (2024)

Abstract: Automatic metrics for evaluating translation quality are typically validated by measuring how well they correlate with human assessments. However, correlation methods tend to capture only the ability of metrics to differentiate between good and bad source-translation pairs, overlooking their reliability in distinguishing alternative translations for the same source. In this paper, we confirm that this is indeed the case by showing that current metrics are insensitive to nuanced differences in translation quality. This effect is most pronounced when the quality is high and the variance among alternatives is low. Given this finding, we shift towards detecting high-quality correct translations, an important problem in practical decision-making scenarios where a binary check of correctness is prioritized over a nuanced evaluation of quality. Using the MQM framework as the gold standard, we systematically stress-test the ability of current metrics to identify translations with no errors as marked by humans. Our findings reveal that current metrics often over or underestimate translation quality, indicating significant room for improvement in machine translation evaluation.

## introduction the automatic evaluation of machine or humangenerated translations has gained widespread attention over the past few years. evaluation metrics act as proxies for translation quality in the absence of human judgments, offering immediate feedback. they are widely used not only to provide quality indicators to users and translators @xcite @xcite , but also to improve machine translation (mt) systems themselves @xcite @xcite . to ensuring their effectiveness and applicability in diverse scenarios. a recent human evaluation study by the conference on machine translation (wmt) revealed that translations produced by current mt systems often achieve very high-quality scores (ranging from 80 to 90) when judged by humans on a direct assessment (da) scale of 0 to 100 @xcite . @xcite observe that these systems increasingly generate numerous "perfect" translations (translations with zero errors), especially for high-resource language pairs, as shown in most evaluations of automatic metrics primarily assess their ability to distinguish between good and bad source-translation pairs @xcite , often overlooking their capacity to discern subtle differences in quality for a given source. furthermore, in many practical and high-risk applications (e.g., within the medical or legal domains), the main concern is not measuring the accuracy level of a translation but determining whether the translation is accurate and fit for that specific use @xcite @xcite @xcite . while correlations provide valuable insights into the performance of automatic metrics, they do not offer a definitive measure of whether these metrics can reliably confirm translation accuracy. hence, in this work, we systematically investigate how existing mt metrics assess high-quality (hq) correct translations, defined as translations with zero or minor errors only. we find that automatic metrics struggle to distinguish between translations for a given source, especially when comparing hq translations, with reference-free or quality estimation (qe) metrics achieving close correlation scores to reference-based ones. we further show that current metrics severely overestimate (for non-hq translations) or underestimate (for hq translations) translation quality. gemba-mqm @xcite , a gpt-based qe metric, achieves the highest f1 score in detecting the hq translations with no errors (hq-zero). however, it also assigns high scores to erroneous gpt-4 translations, suggesting a preferential bias towards the llm's own outputs. these findings highlight the necessity for more robust evaluation protocols to assess the quality of automatic metrics. the most reliable way to assess translation quality has been through human evaluations, with several frameworks proposed over the years for this purpose. while earlier works consider two dimensions-adequacy and fluency-with a 5point likert scale @xcite , subsequent work on direct assessments (da) considers a single continuous scale of 0 -100 @xcite . however, several studies have questioned the credibility of da-based evaluation @xcite @xcite @xcite . unlike das, which assign a numeric score to a translation, the recent multidimensional quality metrics (burchardt, 2013, mqm) framework relies on explicit error judgments (error types and sever-ities) marked within specific spans of the sourcetranslation pair, providing a more accurate and finegrained evaluation. translations receive a score of 0 if they contain no errors, a penalty of -1 for minor errors, and -5 for major errors that impact the usage or understanding of the content. 1 we present the distribution of gold mqm scores from the wmt23 metrics task @xcite , wmt22 metrics task @xcite , and wmt22 chat translation task @xcite in we define hq translations as those that achieve an mqm @xmath1, i.e., translations without any major errors according to human evaluators. by definition, these translations do not contain errors that impede their comprehension or usability. we consider a subset of qe and reference-based automatic metrics evaluated by the shared tasks (see app. a for more details). ## conclusions and future work this work systematically investigates how automatic metrics assess hq translations. we find that current metrics correlate poorly with human judgments when contrasting translations for a given source, with the correlation being even lower for hq translations. we then study whether metrics can detect hq translations that attain zero mqm scores (hq-zero) and find that many metrics fail to assign them valid scores. while the gpt-4-based gemba-mqm attains the highest f1 for detecting hq-zero, it shows some preference for gpt-4 outputs. therefore, despite its promise, it is essential to complement gemba-mqm with other metrics to ensure robust evaluation.