Results of the WMT 20 Metrics Shared Task (2020)

Abstract: This paper presents the results of the WMT20 Metrics Shared Task. Participants were asked to score the outputs of the translation systems competing in the WMT20 News Translation Task with automatic metrics. Ten research groups submitted 27 metrics, four of which are reference-less “metrics”. In addition, we computed five baseline metrics, including sentBLEU, BLEU, TER and using the SacreBLEU scorer. All metrics were evaluated on how well they correlate at the system-, document- and segment-level with the WMT20 official human scores. We present an extensive analysis on influence of different reference translations on metric reliability, how well automatic metrics score human translations, and we also flag major discrepancies between metric and human scores when evaluating MT systems. Finally, we investigate whether we can use automatic metrics to flag incorrect human ratings.

## introduction the metrics shared task foot_0 has been a key component of wmt since 2008, serving as a way to validate the use of automatic mt evaluation metrics and drive the development of new metrics. we evaluate automatic metrics that score mt output by comparing them with a reference translation generated by human translators, who are instructed to translate "from scratch", without post-editing from mt. in addition, following last year's collaboration with the wmt quality estimation (qe) task, we also invited submissions of reference-free metrics that compare mt outputs directly with the source segment. similar to the last year's editions, the source, reference texts, and mt system outputs for the metric task come from the news translation task (barrault et al., 2020, which we denote as findings 2020). this year, the language pairs were english ↔ chinese, czech, german, inuktitut, japanese, polish, russian and tamil. we further included systems participating in the wmt parallel corpus filtering task @xcite : khmer and pashto to english. 2 all metrics are evaluated based on their agreement with human evaluation. we evaluate metrics at three levels: comparing mt systems on the entire testset, segments (either sentences or short paragraphs), and new this year, documents. we introduce document-level evaluation to incentivize the development of metrics that are take into account broader context of evaluated sentences or paragraphs, following the recent emergence of document-level mt techniques. multiple references this year, we have two independently generated references for english ↔ german, english ↔ russian, and @xmath0. this lets us investigate the influence of references and the utility of multiple references. we instructed participants to score mt systems against the references individually as well as with all available references. in addition, we also supplied a set of references for english to german, that were generated by asking linguists to paraphrase the wmt reference as much as possible @xcite . these references are designed to minimise translationese in the reference which could lead to metrics to be biased against systems that generate more natural text. evaluating human translations given that we have multiple human translations, we asked participants to evaluate each human translation using the other as a reference. for these language-pairs, at least one of these human translations was included in the human evaluation, so we can directly evaluate metrics on how they rank the human translation compared to the mt systems. additional human evaluation finally, we pose the question if some of the discrepancies between metrics and human scores can be explained by bad human ratings. we rerun some of the human evaluations by using the same template, but switching the rater pool from non-experts to professional linguists. in particular, we rerun human evaluation for a subset of translations where all metrics disagree with the wmt human evaluation. this experiment could reveal a new use case of automatic metrics and indicate that automatic metrics can be used to identify bad ratings in human evaluations. we first give an overview of the task (section 2) and summarize the baseline (section 3.1) and submitted (section 3.2) metrics. the results for system-, segment-, and document-level evaluation are provided in sections 4, followed by a joint discussion section 5. section 6 describes our rerunning of human evaluation with linguists before we summarise our findings in section 7. we will release data, code and additional visualisations in the metrics package to be made available at http://www.statmt.org/ wmt20/results.html 2 task setup this year, we provided task participants with one test set for each examined language pair, i.e. a set of source texts (which are commonly ignored by mt metrics), corresponding mt outputs (these are the key inputs to be scored) and one or more reference translations. in the system-level, metrics aim to correlate with a system's score which is an average over many human judgments of segment translation quality produced by the given system. in the segmentlevel, metrics aim to produce scores that correlate best with a human ranking judgment of two output translations for a given source segment. and finally, we also trial document-level evaluation this year. (more on the manual quality assessment in section 2.3). segments are sentences for all language pairs except english ↔ german and czech, and for @xmath1, which do not contain sentence boundaries and are translated and evaluated at the paragraph-level. participants were free to choose which language pairs and tracks (system/segment/document and reference-based/reference-free) they wanted to take part in. ## use automatic metrics to detect incorrect human preference it has been argued that non-expert translators lack knowledge of translation and so might not notice subtle differences that make one translation better than another. @xcite compared the evaluation of mt output of professional translators against crowd workers. results showed that for all language pairs, the crowd workers tend to be more accepting of the mt output by giving higher fluency and adequacy scores. @xcite showed that the ratings acquired by professional translators show a wider gap between human and machine translations compared to judgments by non-experts. they recommend using professional linguists for mt evaluation going forward. @xcite show that non-experts assess parity between human and machine translation where professional translators do not, indicating that the former neglect more subtle differences between different translation outputs. given the previous work and the fact that the wmt human evaluation has been conducted with a mix of researchers and crowd workers, we rerun human evaluation for a subset of the submissions with professional linguists. in particular, we want to investigate if we can use the quality scores obtained by the automatic metrics to detect incorrect human ratings. we filtered out all pairs of systems where the human evaluation results disagree with all automatic metrics. taking the metric scores as a signal, we rerun human evaluation for a subset of submissions for 2 language pairs: @xmath2. we hired 10 professional linguists, who rerun the source-based direct assessment human evaluation with the same documentbased template that has been used for the original wmt ratings. ## conclusion this paper summarizes the results of wmt20 shared task in machine translation evaluation, the metrics shared task. participating metrics were evaluated in terms of their correlation with human judgement at the level of the whole test set (systemlevel evaluation), as well as at a more fine-grained level (document-level evaluation and sentences or paragraphs for segment-level evaluation). we reported scores for standard metrics requiring the reference as well metrics that compare mt output directly with the source text. for system-level, best metrics reach over 0.95 pearson correlation or better across several language pairs. in many cases, this correlation drops considerably when the correlation is recomputed after discarding outlier systems. computing pearson correlation without outliers can change the rankings of metrics, and selecting these outlier systems is not an exact science. we report results both with all systems and after discarding outliers as together, and also include kendall tau correlation, and hope that together, they give a more complete picture than just reporting only one of these numbers. in the end, we believe that it is impossible to adequately describe data with summary numbers, and that it's best to visualise data to understand patterns. the results confirm the trends from previous years, namely metrics based on word or sentencelevel embeddings, achieve the highest performance @xcite . for some language pairs, we had two references available. on these test sets, we found that computing scores with two references rarely helped metrics achieve a higher correlation than using either reference individually. this contradicts earlier research that shows that multiple references improve correlation @xcite , but is in line with more recent papers that show additional independent references might not be helpful @xcite . we believe that the utility of additional independent references is dependent on the mt systems evaluated, that perhaps they are not as helpful when scoring high quality mt systems as with low/mid quality mt. in addition to scoring mt systems, this year, we also requested scores for human reference translations. this highlighted the difference between lexical and embedding-based metrics, as lexical metrics consistently gave low scores to human translations. however, when using the english-german paraphrased references, all metrics scored the other human references above all mt systems, highlighting the advantages of using paraphrased references when scoring human translations. in addition to human references, there are some mt systems where metrics (either the majority of metrics, or only the lexical metrics) make major errors. it remains an open question as to what it is about these systems that metrics struggle with scoring them correctly. compared to last year, the performance of the reference-free metrics has improved, and the correlations this year are competitive with the referencebased metrics, and in many cases, outperform bleu. in particular, comet-qe is good at recognising the high quality of human translations where bleu falls short. in terms of segment-level kendall's τ results, the standard metrics correlations was very low for the to-english language pairs, particularly after discarding translations by outlier systems. the corre-lations of the out-of-english language pair are more in line with recent years, reaching a maximum of above 0.6. it has been shown that context is really important when humans are rating mt outputs @xcite , and the wmt human evaluation is moving towards evaluating segments with the document context @xcite . this creates a mismatch with automatic metrics, all of which, this year, score each segment independently. this year, we introduce document-level evaluation of metrics. when computing document-level scores, some metrics from the comet family include document context when computing segment scores within the document. all other metrics included in this year's evaluation either use the average of the segment scores or compute the document score based on statistics computed independently for each segment. in the future, we hope to see more metrics that consider broader context when evaluating translations at all three levels. for this year, we are unable to draw any meaningful conclusions from the document-level evaluation task, as it is hard to tease apart the influence of noise in the ground truth, inadequate segment-level translations and inadequate translation in context of the document. we believe that the noise in the darr judgements is a big factor in the low correlations in the to-english language pairs. we need further research into understanding the factors that contribute to the kendall tau scores and how much we can trust these results. there are shortcomings in the methods used to evaluate metrics at the system-, document-, and segment-level, and we believe that improving methods for evaluating and analysing automatic metrics is a rich area for future research. finally, we assume that any discrepancies between metrics and wmt manual evaluation is a metric error, and we acknowledge that this might not be true in all cases. there is always scope for improvement in human evaluation methodology, and the best practice recommendations for human evaluation are always evolving. we rerun human evaluation by using the same template as the wmt evaluation, but switching the rater pool from non-experts to professional linguists for a subset of translations where all metrics disagree with the wmt human evaluation. this experiment revealed a new use case of automatic metrics and demonstrated that automatic metrics can be used to identify bad ratings in human evaluations. the new obtained ratings were in line with the scores suggested by the automatic metrics and also confirmed the higher translation quality of human translations when compared to mt output. in this paper, we looked at how outliers influence metric evaluation, and we wonder how the presence of these systems influence da annotations. in a perfect world, annotators score each translation on its own merits without being influenced by previous instances. in this world, given the presence of much worse translations, do annotators assign high scores to the remaining translations that look relatively better? does an mt system receive an unfair advantage if it is consistently scored alongside a low-scoring outlier? and does standardising the scores of individual annotators exacerbate this issue? these and other research questions remain open this year, keeping the wmt tasks increasingly interesting as mt systems are getting closer to human performance.