Results of WMT 22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022)

Abstract: This paper presents the results of the WMT22 Metrics Shared Task. Participants submitting automatic MT evaluation metrics were asked to score the outputs of the translation systems competing in the WMT22 News Translation Task on four different domains: news, social, ecommerce, and chat. All metrics were evaluated on how well they correlate with human ratings at the system and segment level. Similar to last year, we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). This setup had several advantages, among other things: (i) expert-based evaluation is more reliable, (ii) we extended the pool of translations by 5 additional translations based on MBR decoding or rescoring which are challenging for current metrics. In addition, we initiated a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics’ ability to capture and penalise specific types of translation errors. Finally, we present an extensive analysis on how well metrics perform on three language pairs: English to German, English to Russian and Chinese to English. The results demonstrate the superiority of neural-based learned metrics and demonstrate again that overlap metrics like Bleu, spBleu or chrf correlate poorly with human ratings. The results also reveal that neural-based metrics are remarkably robust across different domains and challenges.

## introduction the metrics shared task foot_0 has been a key component of wmt since 2008, serving as a way to validate the use of automatic mt evaluation metrics and drive the development of new metrics. we evaluate reference-based automatic metrics that score mt output by comparing the translations with a reference translation generated by human translators, who are instructed to translate "from scratch" without post-editing from mt. in addition, we also invited submissions of reference-free metrics (quality estimation metrics or qe metrics) that compare mt outputs directly with the source segments. all metrics are evaluated based on their agreement with human rating when scoring mt systems and human translations at the system or sentence level. the final ranking of this year's submitted primary metrics is shown in our main findings are: the rest of the paper is organized as follows: section 2 describes the additional mt systems. section 3 presents an overview of the conducted expertbased human evaluation. section 4 describes the metrics evaluated this year (baselines and participants). section 5 describes the conducted metaevaluation. section 6 reports our main results. section 7 summarizes our results for additional wmt22 translation task language-pairs based on their direct assessment human evaluation. section 8 presents a description of the submitted challenge sets along with their findings. finally, section 9 presents our most relevant conclusions. ## translation systems similar to the previous years' editions, the source, reference texts, and mt system outputs for the metrics task are mainly derived from the wmt22 general mt task. in addition to the mt system outputs from the wmt evaluation campaign, we added translations from six additional mt systems which we deemed interesting for evaluation. ## mqm human evaluation automatic metrics are usually evaluated by measuring correlations with human ratings. the quality of the underlying human ratings is critical and recent findings @xcite have shown that crowd-sourced human ratings are not reliable for high quality mt output. furthermore, an evaluation schema based on mqm @xcite , which requires explicit error annotation, is preferable to an evaluation schema that only asks raters for a single scalar value per translation. similar to last year, we decided to not use the human ratings from the wmt general mt task, and conducted our own mqm-based human evaluation on a subset of submissions and a subset of language pairs that are most interesting for evaluating current metrics. this not only had the advantage of more reliable ratings for a subset of language pairs, but also gave us the opportunity to add our own translations that might be challenging for current metrics and are not part of an wmt submission. mqm is a general framework that provides a hierarchy of translation errors which can be tailored to specific applications. google and unba-bel sponsored the human evaluation for this year's metrics task for a subset of language pairs using either professional translators (@xmath0, @xmath1. the error annotation typology and guidelines used by google's and unbabel's annotators differ slightly and are described in the following two sections. ## baselines and primary submissions we computed scores for several baseline metrics in order to compare submissions against previous well-studied metrics. we will start by describing those baselines and then we will describe the submissions from participating teams. an overview of the evaluated metrics can be seen in ## meta evaluation our main goal in evaluating metrics is to establish a ranking that reflects a metric's accuracy across a broad range of settings and applications. combining results across different settings is challenging because correlations with human gold scores have different ranges and may be subject to differing degrees of noise. there are also many ways of measuring correlation, with different strengths and weaknesses, and it is often not clear which is best in a given setting. this year, our overall ranking is just each metric's average rank across a large number of "tasks". unlike raw correlation scores, ranks are comparable across tasks. the resulting global ranking approximates the "kemeny consensus" -the ranking with lowest aggregate kendall distance to the per-task rankings -which in turn satisfies several criteria from social choice theory @xcite . our version has the following features: ## main results as we have seen in section 5, the main results are defined across different settings including systemlevel and segment-level tasks. nonetheless, since the main use case of automatic metrics is to rank systems, system-level accuracy has a 1/4 weight on the final score with the remaining 3/4 distributed over 200 different settings. one open question about neural metrics has been their ability to generalise to new domains, since most training and testing data from previous years were based on news data. in figure 2 we present the performance of each metric across four domains: news, social, conversational, and ecommerce. similar to last year, we observe that the neural metrics perform better than lexical overlap metrics across all four domains. figure 3 shows the average rankings when grouped separately by system-level and segmentlevel tasks. many metrics fall into the same significance cluster when evaluated on the system-level as we only have a very limited number of mt systems. nevertheless, we observe that the metric rankings are largely stable across both granularities and that metricx xxl and comet-22 perform best on both the segment-level and system-level tasks. the differences are more prevalent in the segment-level task, though. in figure 4 , we compare the rankings when including human translations as mt systems (with human) or just considering mt submission (without human). overall, the majority of metrics show lower correlation when we include human translations, except comet-22 and matese. ## direct assessment human evaluation in addition to our mqm annotations and as a contrastive evaluation to cover more language pairs, we look into the performance of metrics when compared to the human evaluation campaign conducted by the general mt shared task @xcite , who ran human evaluation for all 21 translation directions and wmt22 submissions. last year, we decided to exclude the human ratings by the wmt main task as they were of lower quality than the best automatic metrics. however, the generalmt task improved their evaluation methodology in particular for all from-english and non-english translation directions and implemented the scalar quality metric (sqm) which has been shown to have high correlation with mqm on at least the system-level @xcite . the generalmt task used two different human evaluation methodologies depending on the language pair: reference-based direct assessment (ref. da) @xcite and sqm style source-based da (da+sqm) @xcite . ref. da has been used for all into-english translation directions and asks human raters to judge each system translation against human reference translation on a 0-100 scale. this technique does not use bilingual speakers and is evaluated by nonprofessional crowd workers. in order to increase quality of assessment, there are several quality control items. out of all collected human annotations, 63% have been removed due to failing quality control. da+sqm asks bilingual raters to annotate system translations against original sources on a 0-100 labeled scale. the scale is marked with seven points representing expected quality. in this setting, @xcite evaluated all from-english and non-english translation directions. they used mainly professional raters. we present system-level accuracy results in on and instead human crowd workers are incentivized to quickly compare the surface forms of translation against reference without understanding. we would advise metric developers and researchers running human evaluations not to use reference-based da, especially when evaluated with non-professional crowd workers. ## challenge sets subtask the challenge sets subtask is inspired by the build it or break it: the language edition shared task @xcite which aimed at testing the generalizability of nlp systems beyond the distributions of their training data. with that said, our goal is to encourage researchers to build a set of test sets that measure metrics' ability to detect different targeted phenomena that might not be well represented in traditional test sets used to evaluate metrics. this subtask is made of three consecutive phases; 1) the breaking round, 2) the scoring round and 3) the analysis round: we had a total of 4 submissions to this shared task, covering a wide range of phenomena and 146 different language pairs. aces the aces (translation accuracy challenge sets; @xcite results from a collaboration between the university of zurich with the university of edinburgh. this challenge set, highly inspired by the mqm framework, consists of 36,499 examples, covering 146 language pairs and 68 phenomena, ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. the data was created artificially for some error types and manually for others. their analysis aimed to reveal the extent to which metrics take into account the source sentence context and the surface-level overlap with the reference, and if they profit by using multilingual embeddings. finally, they recommend that one considers a) combining metrics with different strengths and b) explicitly modelling additional languagespecific information beyond what is available via multilingual embeddings. @xcite , submitted by unbabel and ist evaluates the robustness of mt metrics to 5 different types of translation errors; named entity errors, numerical errors, meaning errors, insertion of content and content missing. these errors are created by perturbing reference translations and then curated by the authors. ## conclusion this paper summarizes the results of the wmt22 shared task on automated machine translation evaluation, the metrics shared task. we presented an extensive analysis on how well metrics perform on our three main language pairs: @xmath4, @xmath5. the results, based on 201 different tasks, demonstrated the superiority of neural-based learned metrics over overlap-based metrics like bleu, sp-bleu or chrf. these results are confirmed with da+sqm human judgement. although this was already the case in the previous years' metric shared tasks, we further strengthened the case for neuralbased fine-tuned metrics by demonstrating their superiority across four different domains. in addition, we initiated a challenge set subtask, where participants had to create contrastive test suites for evaluating metrics' ability to capture and penalise specific types of translation errors.