Results of WMT 23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent (2023)

Abstract: This paper presents the results of the WMT23 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT23 News Translation Task. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). Following last year’s success, we also included a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Furthermore, we improved our meta-evaluation procedure by considering fewer tasks and calculating a global score by weighted averaging across the various tasks. We present an extensive analysis on how well metrics perform on three language pairs: Chinese-English, Hebrew-English on the sentence-level and English-German on the paragraph-level. The results strongly confirm the results reported last year, that neural-based metrics are significantly better than non-neural metrics in their levels of correlation with human judgments. Further, we investigate the impact of bad reference translations on the correlations of metrics with human judgment. We present a novel approach for generating synthetic reference translations based on the collection of MT system outputs and their corresponding MQM ratings, which has the potential to mitigate bad reference issues we observed this year for some language pairs. Finally, we also study the connections between the magnitude of metric differences and their expected significance in human evaluation, which should help the community to better understand and adopt new metrics.

## introduction the metrics shared task 1 has been a key component of wmt since 2008, serving as a way to validate the use of automatic mt evaluation metrics and drive the development of new metrics. we eval-uate reference-based automatic metrics that score mt output by comparing the translations with a reference translation generated by human translators, who are instructed to translate "from scratch" without post-editing from mt. in addition, we also invited submissions of reference-free metrics (quality estimation metrics or qe metrics) that compare mt outputs directly with the source segments. all metrics are evaluated based on their agreement with human rating when scoring mt systems and human translations at the system and sentence level. the final ranking of this year's submitted primary metrics is shown in our main findings are: the rest of the paper is organized as follows: section 2 describes the test data and additional mt systems that we trained. section 3 presents an overview of the conducted expert-based human evaluation. section 4 describes the metrics evaluated this year (baselines and participants). section 5 describes the conducted meta-evaluation. section 6 reports our main results. section 7 interprets and evaluates metrics' scores beyond correlations. section 8 analyses the impact of bad reference translations on the various metrics. section 9 summarizes our results for additional wmt23 translation task language-pairs based on their direct assessment human evaluation. section 10 presents a description of the submitted challenge sets along with their findings. finally, section 11 presents our most relevant conclusions. ## translation systems similar to previous years' editions, the source, reference texts, and mt system outputs for the metrics task are mainly derived from the wmt23 general mt shared task. in addition to the mt system outputs from the wmt evaluation campaign, we included translations from two additional mt systems which we deemed interesting for evaluation. ## mqm human evaluation automatic metrics are usually evaluated by measuring correlations with human ratings. the quality of the underlying human ratings is critical, and recent findings @xcite have shown that crowdsourced human ratings are not reliable for high quality mt output. furthermore, an evaluation schema based on mqm @xcite , which requires explicit error annotation, is preferable to an evaluation schema that only asks raters for a single scalar value per translation. similar to last year, we decided to conduct our own mqmbased human evaluation on a subset of submissions and language pairs that are most interesting for evaluating current metrics. mqm is a general framework that provides a hierarchy of translation errors which can be tailored to specific applications. google and unbabel sponsored the human evaluation for this year's metrics task for a subset of language pairs using either professional translators (@xmath0, @xmath1. the error annotation typology and guidelines used by google's and unbabel's annotators differ slightly and are described in the following two sections. ## baselines and submissions we computed scores for several baseline metrics in order to compare submissions against previous well-studied metrics. we will start by describing those baselines, and then we will describe the submissions from participating teams. an overview of the evaluated metrics can be seen in ## meta evaluation our main goal in evaluating metrics is to establish a ranking that reflects a metric's performance across a range of settings and applications. combining results from different settings is challenging because correlations with human gold scores have different ranges and may be subject to differing degrees of noise. there are also many ways of measuring correlation, with different strengths and weaknesses, and it is often not clear which is best in a given setting. last year, our approach was to define a large number of "tasks" (201 in total) that varied along dimensions such as language pair, domain, granularity, correlation statistic, etc. for each task, we used pairwise significance tests to establish a dense clustered ranking of participating metrics (e.g., 1, 1, 1, 2, 3, 3, ...). motivated by theoretical results pertaining to combining rankings from different knowledge sources @xcite , we established an overall ranking by simply averaging the per-task ranks. this approach has several disadvantages. first, it is difficult to incorporate new metrics into the comparison, since this requires not only computing the score of a new metric on 201 tasks, but also comparing it to all existing metrics on each task using expensive resampling significance tests. adding a new metric also has the undesirable effect of potentially causing other metrics to swap places in the overall rankings. while rank averaging has theoretical underpinnings, as noted above, these apply to settings in which the constituent tasks provide only ranking information themselves. in order to take advantage of richer information available from correlation statistics, we derived dense ranks from pairwise significance tests, but this relies on an ad hoc clustering algorithm, and it is not clear to what extent our average ranks are supported by the original theory. they also lack confidence information, making it difficult to quantify conclusions about the overall superiority of one metric over another. this year we adopted a much simpler approach in order to address these difficulties. we use just 10 main tasks, and compute an overall score by taking a weighted average of results from each task. we perform significance tests on each pair of metrics for each task as before, but also do so for each pair of metrics on the overall average score, allowing us to establish a clearer global ranking. the average score for a new metric can be computed relatively quickly, and it does not affect the scores of other metrics. significance tests still require the expensive step of comparing to all other metrics, but they are no longer necessary for computing a metric's raw overall score. we acknowledge that this approach is not perfect. one problem is that we need to combine correlations and accuracies that may have different dynamic ranges. for example, the mean pearson correlation across all metrics for @xmath2.88 with standard deviation 0.24, while at the segment level it is 0.39 with a standard deviation of 0.17. averaging system-level and segment-level correlations will therefore effectively upweight the system-level contribution. we experimented with different weightings to compensate for this, but found that they did not make a large difference, and decided to use equal weights for simplicity. another problem is that we do not account for dependencies among tasks. although all tasks are at least somewhat complementary, many-such as system-level and segment-level correlations-are based on the same underlying data, and thus violate the assumptions of our hypothesis tests. we leave more sophisticated inference approaches such as proposed by @xcite or hagmann and riezler (2023) for future work. ## main results as we have seen in section 5, the main results are the overall scores by taking a weighted average of the results from the ten main tasks, including system-level and segment-level tasks in different translation directions. similar to last year, since the main use case of automatic metrics is to rank systems, system-level accuracy has a 1/4 weight on the final score with the remaining 3/4 distributed over 9 different settings. it is worth noting that the best-performing baseline, cometkiwi, along with four of the seven top-performing primary submissions, are referencefree. as we will elaborate on in a later section (section 8), there are quality issues with human reference translations. this highlights the challenge of ensuring robustness to poor-quality references for reference-based metrics. in cases where a highquality human reference is not available, referencefree metrics can serve as more robust alternatives. overall, xcomet-ensemble is the best performing metric in terms of average scores over the 10 meta-evaluation settings, with a statistically significant advantage over all other metrics. it consistently correlates best with human mqm scores at segment level for all translation directions, and it is ranked at worst in the 2nd significance cluster for all system-level meta-evaluation tasks. figure 1 shows the correlation scores split by translation direction. there are two key observations: 1) a majority of the metrics have higher correlations for @xmath3, except for mre-score-labse-regular and ebleu, that perform substantially better for @xmath4, and yisi-1 and bertscore, that perform equally in @xmath5; 2) reference-based metrics struggle for @xmath6, except for xcomet- ensemble and sescorex. the reason for the significant drop in correlation for @xmath7. this drop is observed across almost all metrics, whether they are trained or untrained, referencefree or reference-based, and they exhibit varying degrees of degradation. we continue to be interested in metrics' ability to generalise across domains. in figure 2 , 3 and 4 we present the performance of each metric across different domains in each translation direction. most metrics perform well in evaluating translation in the user reviews domain across translation direction, despite lacking annotated data in that domain. further investigation is required to understand whether this is because the translation quality of mt output is more diverse in the user reviews domain, making it easier for metrics to accurately discriminate. figure 5 shows the average correlations of metrics when grouped separately by system-level and segment-level tasks. many metrics fall into the same significance cluster when evaluated on the system-level, as we only have a limited number of mt systems. although most of the metrics compute the system-level score by averaging their segment-level scores, we observe that high correlations between automatic metrics and human judgments at the segment level do not necessarily guarantee high correlations at the system level. for example, prismsrc is in the middle of the pack and has moderate pearson's correlation at segment level for @xmath8. however, it is negatively correlating with human judgements when evaluating the same language pair at system level. ## understanding metrics' scores beyond correlation in the past few years, we demonstrated that new metrics correlate better with human judgments than bleu does. some new baseline metrics even consistently outperform bleu for consecutive years across translation directions and domains. how- ## synthetic reference translations reference-based metrics compare machine translations of source segments to human translations of those same source segments to determine how good they are. the quality of the underlying human translation is crucial and can impact the quality of the predicted scores more than the choice of metric @xcite . motivated by the low human ratings of refa for @xmath9, we investigate a method for generating a synthetic reference translation based on the mt output and the corresponding mqm ratings. ## da+sqm human evaluation in addition to our mqm annotations and as a contrastive evaluation to cover more language pairs, we look into the performance of metrics when compared to the human evaluation campaign conducted by the wmt23 general mt shared task @xcite , who ran human evaluation for all 14 translation directions and all wmt23 submissions. in contrast to previous years, they no longer use mturk neither reference-based evaluation for into-english language pairs. they no longer use z-score normalization because the user interface decision to not track users (i.e., only maintaining hit information) means that the z-scores are likely to be influenced by the distribution of system quality in the hits rather than only annotator variation. they employ the direct assessment scalar quality metrics (da+sqm) technique as presented in @xcite . da+sqm asks bilingual raters to annotate system translations against original sources on a 0-100 labelled scale. the scale is marked with seven points representing expected quality. at the time of writing, the wmt23 general mt shared task had collected data only for 8 translation directions: chinese↔english (zh↔en), german↔english (de↔en), japanese↔english (ja↔en), @xmath10, and @xmath11. we present system-level accuracy results for both mqm and da+sqm in ## challenge sets sub-task for the second year, we included a sub-task on challenge sets. this sub-task is inspired by the build it or break it: the language edition shared task @xcite which aimed at testing the generalizability of nlp systems beyond the distributions of their training data. whereas the standard evaluation of the shared task runs on test sets containing generic text from real-world content, the challenge set evaluation is based on test sets designed with the aim of revealing the abilities or the weaknesses of the metrics on evaluating particular translation phenomena. in order to shed light on different per- there were 3 submissions this year, covering a wide range of phenomena and 146 different translation directions. an overview of the submitted challenge sets can be seen in for each metric, the authors provide a detailed profile of performance across the ten top-level accuracy error categories in aces as well as an overall aces-score for quick comparison. they also measure the incremental performance of the metrics submitted to both wmt 2023 and 2022. they find that: the authors' recommendations are similar to those from wmt 2022. metric developers should focus on: building ensembles of metrics from different design families, developing metrics that pay more attention to the source and rely less on surface-level overlap, and carefully determining the influence of multilingual embeddings on mt evaluation. ## conclusion this paper summarizes the results of the wmt23 shared task on automated machine translation evaluation, the metrics shared task. we presented an extensive analysis on how well metrics perform on our three main translation directions: @xmath15, @xmath16. the results, based on 10 different tasks, confirm the superiority of neural-based learned metrics over overlap-based metrics like bleu, spbleu or chrf. these results are confirmed with da+sqm human judgement. we also found that reference-free metrics were strong contenders this year, partly because they do not rely on the quality of reference translations, an increasingly important issue as mt systems under evaluation become better. in addition, we continued the challenge set subtask, where participants had to create contrastive test suites for evaluating metrics' ability to capture and penalise specific types of translation errors.