Results of the WMT 21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021)

Abstract: This paper presents the results of the WMT21 Metrics Shared Task. Participants were asked to score the outputs of the translation systems competing in the WMT21 News Translation Task with automatic metrics on two different domains: news and TED talks. All metrics were evaluated on how well they correlate at the system- and segment-level with human ratings. Contrary to previous years’ editions, this year we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). This setup had several advantages: (i) expert-based evaluation has been shown to be more reliable, (ii) we were able to evaluate all metrics on two different domains using translations of the same MT systems, (iii) we added 5 additional translations coming from the same system during system development. In addition, we designed three challenge sets that evaluate the robustness of all automatic metrics. We present an extensive analysis on how well metrics perform on three language pairs: English to German, English to Russian and Chinese to English. We further show the impact of different reference translations on reference-based metrics and compare our expert-based MQM annotation with the DA scores acquired by WMT.

## introduction the metrics shared task foot_0 has been a key component of wmt since 2008, serving as a way to validate the use of automatic mt evaluation metrics and driving the development of new metrics. we evaluate reference-based automatic metrics that score mt output by comparing the mt with a reference translation generated by human translators, who are instructed to translate "from scratch" without post-editing from mt. in addition, we also invited submissions of reference-free metrics (quality estimation metrics or qe metrics) that compare mt outputs directly with the source segments. all metrics are evaluated based on their agreement with human rating when scoring mt systems and human translations at the system or sentence level. this year, we implemented several changes to the methodology that was followed in previous years editions of the task: our main findings are: nevertheless, once we focus on mt output only, reference-free metrics perform worse compared to reference-based metrics ( ## data similar to the previous years' editions, the source, reference texts, and mt system outputs for the metrics task are mainly derived from the wmt21 news translation task. this year, we expand the domain and evaluate the same mt systems on an additional out-of-domain data -ted talks, for our three primary language pairs: @xmath0, @xmath1. in addition to the mt system outputs from the wmt evaluation campaign, we added translations from five additional mt systems that represent different versions of the same system during system development. ## mqm human evaluation automatic metrics are usually evaluated by measuring correlations with human ratings. the quality of the underlying human ratings is critical and recent findings @xcite have shown that crowd-sourced human ratings are not reliable for high quality mt output. furthermore, an evaluation schema based on mqm @xcite which requires explicit error annotation is preferable to an evaluation schema that only asks raters for a single scalar value per translation. contrarily to the previous versions of the wmt metrics task, for our primary evaluation this year, we decided not to use the crowd-sourced da human ratings from the wmt news translation task, and conducted our own mqm-based human evaluation on a subset of submissions and a subset of language pairs that are most interesting for evaluating current metrics. this not only had the advantage of more reliable ratings for a subset of language pairs, but also gave us the opportunity to run the same human evaluation on a different domain (ted talks) on output generated by the same mt systems, in order to test the generalization capabilities of the metrics. mqm is a general framework that provides a hierarchy of translation errors which can be tailored to specific applications. google and unbabel sponsored the human evaluation for this year's metrics task for a subset of language pairs using either professional translators (@xmath2, @xmath3. the error annotation typology and guidelines used by google's and unbabel's annotators differs slightly and is described in the following two sections. ## metric submissions and baselines bertscore bertscore @xcite leverages contextual embeddings from pre-trained transformers to create soft-alignments between words in candidate and reference sentences using a cosine similarity. based on the alignment matrix, bertscore returns a precision, recall and f1 score. we used f1 without tf-idf weighting. prism prism @xcite ) is an automatic mt metric which uses a sequence-tosequence paraphraser to score mt system outputs conditioned on their respective human references. 7 ter+lang.langpair+tok.tercom-nonorm-punct noasian-uncased+version.1.5.0 8 chrf2+lang.langpair-+numchars.6+space.false-+version.1.5.0. we used the default parameters with version 0.1 and model m39v1. ## main results currently, the main use case of automatic metrics is to rank systems either during system development or by comparing your own output with the one from other research institutes or competitors. consequently, we present system-level correlations as our main metric in this year's wmt21 metrics task. to be in line with the main use case, we present pairwise accuracy numbers for each metric that calculate the accuracy scores on binary comparison of system outputs for each language pair. we refer the reader to section 7 for language pair specific results on both the segment and system level with more traditional correlation metrics. ## challenge sets while the correlation analysis is testing the evaluation metrics on their ability to rank mt systems according to translation quality, we are also interested in understanding metrics' performance on identifying certain types of translation errors. we created three challenge sets containing translation errors that are believed to be challenging for automatic mt evaluation metrics to identify. a good metric should not only rank candidate translations by their quality but also be sufficiently sensitive to these types of errors. each challenge set consists of two mt outputs (and the corresponding source and reference) where one of them contains the type of translation error of interest and the other does not. metrics are expected to give a lower score to the mt output containing the error. we use kendall's tau-like correlation, typically used for darr @xcite @xcite , for evaluating the primary submissions on the challenge sets. kendall's tau-like correlation is defined as follows: ## results per language pair we computed individual correlation results for each focus language pair (@xmath6, @xmath7, @xmath8. the system-level metric scores submitted by the participants pertained to the complete wmt test set, but we collected human mqm scores for only a subset of documents, as shown in since we have multiple reference translations for the focus language pairs, we required participants to submit versions of their (reference-based) metrics for each reference. we used only the scores corresponding to the reference that was judged best by the mqm raters for each language pair. for the news domain, we evaluated metric performance both when using only mt outputs and using mt outputs augmented by human references, adding all remaining references in the latter condition except for @xmath9, where we excluded reference b since it was very similar to the best reference c. we measure correlation using the pearson-r statistic at the system level and the kendall-tau statistic at the segment level. pearson correlation is complementary to the pairwise accuracy used for our global results as discussed in section 5: it tests linear fit with mqm scores, a stringent but reference-free metrics are indicated by italics. all submissions labelled with * participated only in 1 or 2 language pairs. reasonable criterion since we expect these scores to conform to a linear scale (for example, a translation with two minor errors is twice as bad as one with only a single error). pearson has well-known drawbacks @xcite , notably sensitivity to outliers, which we avoided by choosing only relatively high-performing systems. in preliminary tests, pearson also yielded a larger number of pairwise significant differences among metrics than kendall, an important property since our fairly small number of systems makes it difficult to reli-ably distinguish metrics at the system level. segment-level scores-metric or human-are naturally arranged as a system × segment matrix (rows × columns). there are several ways to extract vectors for input to correlation statistics. comparing metric and human row vectors corresponds to a use case of judging the relative quality of different segments output by a given mt system ("where is my system making mistakes on this test set?"); comparing column vectors corresponds to judging the relative quality of outputs for a given source segment across different mt systems ("which systems performed better or worse than mine on this segment?"). to avoid emphasizing either of these scenarios at the expense of the other, we flattened the metric and human score matrices into single vectors (row1, row2, ...) before comparing them. this measures the metrics' ability to assign independent scores to mt segments, abstracting away from system or source segment, and provides a large number of comparisons to boost statistical significance. we used kendall rather than pearson correlation for robustness to segment-level noise. 14 the results for each language pair and granularity are shown in tables 23 to 28 , with corresponding pairwise significance plots derived using the perm-both test in figures 2 to 7 . the tables contain results for all metrics; the significance plots include only primary and baseline metrics. in the tables, primary submissions are in bold, baseline metrics are underlined, and metrics that used only the source have "-src" appended to their name. for each condition (news without human translations, news with human translations, or ted), the scores of primary and baseline metrics in the top cluster are in bold. the top cluster consists of primary or baseline metrics that are not significantly outperformed by other primary or baseline metrics nor outperformed by a primary or baseline metric not in the top cluster. 15 in the significance plots, this corresponds to the leftmost block of columns containing no dark squares. in terms of total "wins", three metrics stand out clearly: c-specpn, bleurt-20, and comet-mqm_2021. these have fairly evenlydistributed performance across languages, granularities, and data conditions, with the exception of bleurt-20, which does relatively poorly on @xmath10. their advantage over other metrics is most pronounced at the segment level and when human translations are included among the systems to be judged (w/ ht)-both of which are more challenging tasks. in contrast, the distribution of metrics that achieve top-level performance is much broader for system-level granularity, the outof-domain ted setting, and to a lesser extent the news w/o ht setting. two metrics that do not use a reference translation-comet-qe-mqm_2021src and openkiwi-mqm-src-do surprisingly well overall, particularly in the w/ ht condition, but perform poorly at the segment level. this could be explained by these metrics benefiting from their ability to distinguish human vs. machine produced text. finally, the surface-level baselines-bleu, ter, and chrf-join the winners exclusively at the system level and almost exclusively in the out-ofdomain ted condition. ## conclusion this paper summarized the results of the wmt21 shared task on automated machine translation evaluation, the metrics shared task. this year, we collected our own human ratings for selected language pairs (@xmath11, @xmath12, and @xmath13. wmt direct assessment (da) scores generally correlate poorly with mqm scores, and exhibit weaker pref-16 ref-a for @xmath14, ref-b for @xmath15, and ref-a for @xmath16. 17 these numbers do not match others in the paper due to the use of a reduced set of mt systems, and, for the w/ ht condition, a reduced set of human outputs. overall, metrics perform very differently based on domain, language pair or setting (with or without human translations among candidate systems) making it hard to declare a clear winner. when counting top performances across all test conditions, three embeddingbased metrics-c-specpn, bleurt-20, and comet-mqm_2021-emerge as distinctly better than the others, especially at the segment level and when rating human translations. nevertheless, it is unclear which test scenario and correlation metric is best to yield reliable results. we would encourage the community to investigate different ways of how to evaluate automatic metrics. we are very open to apply new suggestions in the next round of the metrics shared task. another challenge is to define the overall ground truth (i.e. the human ratings). even though, we are convinced that expert-based ratings via mqm are more reliable, we also found that the two mqm methodologies of unbabel and google disagree for some systems. we would encourage the community to further work on establishing a reliable human evaluation setup. the field would benefit from a reliable human evaluation standard that could be used by everyone. our ted talks test suite is created based on ted transcripts and translations under cc by-nc-nd 4.0 international. additional translations done for this shared task follow the ted talks usage policy. organizers from the national research council canada and unbabel have submitted to this task the frozen stable versions of their metrics (yisi and comet) dated before they joined the organizing committee. newer versions of comet were developed without using any of the test set, test suite or challenge sets.