  Human evaluation of modern high-quality machine translation systems is a
difficult problem, and there is increasing evidence that inadequate evaluation
procedures can lead to erroneous conclusions. While there has been considerable
research on human evaluation, the field still lacks a commonly-accepted
standard procedure. As a step toward this goal, we propose an evaluation
methodology grounded in explicit error analysis, based on the Multidimensional
Quality Metrics (MQM) framework. We carry out the largest MQM research study to
date, scoring the outputs of top systems from the WMT 2020 shared task in two
language pairs using annotations provided by professional translators with
access to full document context. We analyze the resulting data extensively,
finding among other results a substantially different ranking of evaluated
systems from the one established by the WMT crowd workers, exhibiting a clear
preference for human over machine output. Surprisingly, we also find that
automatic metrics based on pre-trained embeddings can outperform human crowd
workers. We make our corpus publicly available for further research.
