Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021)

Abstract: Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research on human evaluation, the field still lacks a commonly accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM research study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform human crowd workers. We make our corpus publicly available for further research.

## introduction like many natural language generation tasks, machine translation (mt) is difficult to evaluate because the set of correct answers for each input is large and usually unknown. this limits the accuracy of automatic metrics, and necessitates costly human evaluation to provide a reliable gold standard for measuring mt quality and progress. yet even human evaluation is problematic. for instance, we often wish to decide which of two translations is better, and by how much, but what should this take into account? if one translation sounds somewhat more natural than another, but contains a slight inaccuracy, what is the best way to quantify this? to what extent will different raters agree on their assessments? the complexities of evaluating translationsboth machine and human-have been extensively studied, and there are many recommended best practices. however, due to expedience, human evaluation of mt is frequently carried out on isolated sentences by inexperienced raters with the aim of assigning a single score or ranking. when mt quality is poor, this can provide a useful signal; but as quality improves, there is a risk that the signal will become lost in rater noise or bias. recent papers have argued that poor human evaluation practices have led to misleading results, including erroneous claims that mt has achieved human parity @xcite . our key insight in this paper is that any scoring or ranking of translations is implicitly based on an identification of errors and other imperfections. asking raters for a single score forces them to synthesize this complex information, and can lead to rushed judgments based on partial analyses. furthermore, the implicit weights assigned by raters to different types of errors may not match their importance in the current application. an explicit error listing contains all necessary information for judging translation quality, and can thus be seen as a ''platinum standard'' for other human evaluation methodologies. this insight is not new: it is the conceptual basis for the multidimensional quality metrics (mqm) framework developed in the eu qtlaunchpad and qt21 projects ( www.qt21.eu ), which we endorse and adopt for our experiments. mqm involves explicit error annotation, deriving scores from weights assigned to different errors, and returning an error distribution as additional valuable information. mqm is a generic framework that provides a hierarchy of translation errors that can be tailored to specific applications. we identified a hierarchy appropriate for broad-coverage mt, and annotated outputs from 10 top-performing ''systems'' (including human references) for both the @xmath0, using professional translators with access to full document context. for comparison purposes, we also collected scalar ratings on a 7-point scale from both professionals and crowd workers. we analyze the resulting data along many different dimensions: comparing the system rankings resulting from different rating methods, including the original wmt scores; characterizing the error patterns of modern neural mt systems, including profiles of difficulty across documents, and comparing them to human translation (ht); measuring mqm inter-annotator agreement; and re-evaluating the performance of automatic metrics submitted to the wmt 2020 metrics task. our most striking finding is that mqm ratings sharply revise the original wmt ranking of translations, exhibiting a clear preference for ht over mt, and promoting some low-ranked mt systems to much higher positions. this in turn changes the conclusions about the relative performance of different automatic metrics; interestingly, we find that most metrics correlate better with mqm rankings than wmt human scores do. we hope these results will underscore and help publicize the need for more careful human evaluation, particularly in shared tasks intended to assess mt or metric performance. we release our corpus to encourage further research. 1 we also release mqm viewer, foot_1 an interactive tool to analyze mqm data, compute scores and their breakdowns as described in this paper, and find slices of interesting examples. our main contributions are: the alpac report (1966) defined an evaluation methodology for mt based on ''intelligibility'' (comprehensibility) and ''fidelity'' (adequacy). the arpa mt initiative @xcite defined an overall quality score based on ''adequacy'', ''fluency'', and ''comprehension''. the first wmt evaluation campaign @xcite used adequacy and fluency ratings on a 5-point scale acquired from participants as their main metric. @xcite proposed a ranking-based evaluation approach, which became the official metric at wmt from 2008 until 2016 @xcite . the ratings were still acquired from the participants of the evaluation campaign. @xcite compared human assessor consistency levels for judgments collected on a five-point interval-level scale to those collected on a 1-100 continuous scale, using machine translation fluency as a test case. they claim that the use of a continuous scale eliminates individual judge preferences, resulting in higher levels of inter-annotator consistency. @xcite came to the conclusion that fluency evaluation is highly correlated to adequacy evaluation. as a consequence of the latter two papers, continuous direct assessment focusing on adequacy has been the official wmt metric since 2017 @xcite . due to budget constraints, wmt understandably conducts its human evaluation mostly with researchers and/or crowd workers. @xcite used professional translators to rate mt output on three different tasks: ranking, error classification, and post-editing. @xcite found that crowd workers lack knowledge of translation and, compared w professional translators, tend to be more accepting of (subtle) translation errors. @xcite showed that crowd-worker evaluation has to be filtered to avoid contamination of results through the inclusion of false assessments. the quality of ratings acquired by either researchers or crowd workers has further been questioned by @xcite and @xcite . @xcite re-evaluated a subset of wmt submissions with professional translators and showed that the resulting rankings changed and were better aligned with automatic scores. fischer and läubli (2020) found that the number of segments with wrong terminology, omissions, and typographical problems for mt output is similar to ht. @xcite and @xcite raised the concern that reference-based human evaluation might penalize correct translations that diverge too much from the reference. the literature mostly agrees that source-based rather than reference-based evaluation should be conducted @xcite . the impact of translationese @xcite on human evaluation of mt has recently received attention @xcite @xcite . these papers show that only natural source sentences should be used for human evaluation. as alternatives to adequacy and fluency, @xcite presented reading comprehension for mt quality evaluation. @xcite proposed gap-filling, where certain words are removed from reference translations and readers are asked to fill the gaps left using the machine-translated text as a hint. @xcite proposed to ask annotators to just label problematic parts of the translations instead of assigning a score. the multidimensional quality metrics (mqm) framework was developed in the eu qt-launchpad and qt21 projects (2012-2018) ( www.qt21.eu ) to address the shortcomings of previous quality evaluation methods @xcite . mqm provides a generic methodology for assessing translation quality that can be adapted to a wide range of evaluation needs. @xcite designed an mqm-compliant error taxonomy for slavic languages to run a case study for 3 mt systems for @xmath1. @xcite used mqm labels to fine-tune comet for automatic evaluation. @xcite designed an error annotation schema based on pre-defined error categories for table-to-text tasks. ## human evaluation methodologies we compared three human evaluation techniques: the wmt 2020 baseline; ratings on a 7-point likert-type scale which we refer to as a scalar quality metric (sqm); and evaluations under the mqm framework. we describe these methodologies in the following three sections, deferring concrete experimental details about annotators and data to the subsequent section.