Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019 (2020)

Abstract: We reassess the claims of human parity and super-human performance made at the news shared task of WMT2019 for three translation directions: English→German, English→Russian and German→English. First we identify three potential issues in the human evaluation of that shared task: (i) the limited amount of intersen- tential context available, (ii) the limited translation proficiency of the evaluators and (iii) the use of a reference transla- tion. We then conduct a modified eval- uation taking these issues into account. Our results indicate that all the claims of human parity and super-human perfor- mance made at WMT2019 should be re- futed, except the claim of human parity for English→German. Based on our findings, we put forward a set of recommendations and open questions for future assessments of human parity in machine translation.

## introduction the quality of the translations produced by machine translation (mt) systems has improved considerably since the adoption of architectures based on neural networks @xcite . to the extent that, in the last two years, there have been claims of mt systems reaching human parity and even super-human performance @xcite @xcite . following @xcite , we consider that human parity is achieved for a given task t if the performance attained by a computer on t is equivalent c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. to that of a human, i.e. there is no significant difference between the performance obtained by human and by machine. super-human performance is achieved for t if the performance achieved by a computer is significantly better than that of a human. two claims of human parity in mt were reported in 2018. one by microsoft, on news translation for @xmath0, and another at the news translation task of wmt for @xmath1, in which mt systems uedin @xcite and cuni-transformer @xcite reached human parity and super-human performance, respectively. in 2019 there were additional claims at the news translation task of wmt @xcite : human parity for @xmath2, by several of the submitted systems, and for @xmath3, by system facebook-fair @xcite , as well as super-human performance for @xmath4, again by facebook-fair. the claims of human parity and super-human performance in mt made in 2018 @xcite have been since refuted given three issues in their evaluation setups @xcite : (i) part of the source text of the test set was not original text but translationese, (ii) the sentences were evaluated in isolation, and (iii) the evaluation was not conducted by translators. however, the evaluation setup of wmt 2019 was modified to address some of these issues: the first issue (translationese) was fully addressed, while the second (sentences evaluated in isolation) was partially addressed, as we will motivate in section 2.1, whereas the third (human evaluation conducted by non-translators) was not acted upon. given that some of the issues that led to refute the claims of human parity in mt made in 2018 have been addressed in the set-up of the experiments leading to the claims made in 2019, but that some of the issues still remain, we reassess these later claims. the remainder of this paper is organised as follows. section 2 discusses the potential issues in the setup of the human evaluation at wmt 2019. next, in section 3 we conduct a modified evaluation of the mt systems that reached human parity or super-human performance at wmt 2019. finally, section 4 presents our conclusions and recommendations. ## conclusions and future work we have conducted a modified evaluation on the mt systems that reached human parity or superhuman performance at the news shared task of wmt 2019. according to our results: (i) for @xmath5, the claim of super-human performance is refuted, but there is human parity; (ii) for @xmath6, the claim of human parity is refuted; (iii) for @xmath7, for which there were two human translations, the claim of human parity is refuted with respect to the best of the human translations, but not with respect to the worst. based on our findings, we put forward a set of recommendations for human evaluation of mt in general and for the assessment of human parity in mt in particular: the first two recommendations were put forward recently @xcite and are corroborated by our findings. we acknowledge that our conclusions and recommendations are somewhat limited since they are based on a small number of sentences (just over 300 for each translation direction) and evaluators (14 in total). claims of human parity are of course not specific to translation. super-human performance has been reported to have been achieved in many other tasks, including board games, e.g. chess @xcite and go @xcite . however, we argue that assessing human parity in translation, and probably in other language-related tasks too, is not as straightforward as in other tasks such as board games, and that the former task poses, at least, two open questions, which we explore briefly in the following to close the paper. 1. against whom should the machine be evaluated? in other words, should one claim human parity if the output of an mt system is perceived to be indistiguishable from that by an average professional translator or should we only compare to a champion professional translator? in other tasks it is the latter case, e.g. chess in which deep blue outperformed world champion gary kasparov. related, we note that in tasks such as chess it is straightforward to define the concept of a player being better than another: whoever wins more games, the rules of which are deterministic. but in the case of translation, it is not so straightforward to define whether a translator is better than another. this question is pertinent since, as we have seen for @xmath8.4), where we had translations by two professional translators, the choice of which one is used to evaluate an mt system against can lead to a claim of human parity or not. in addition, the reason why one claim remains after our evaluation (human parity for @xmath9. therefore, once the three potential issues that we have put forward (see section 2) are solved, we think that an important potential issue that should be studied, and which we have not considered, has to do with the quality of the human translation used. in translation, however, we need a person with knowledge of both languages to assess the translations. we have seen that the outcome is dependent to some extent on the level of translation proficiency of the evaluator: it is more difficult to find human parity if the translations are evaluated by professional translators than if the evaluation is carried out by bilingual speakers without any translation proficiency. taking into account that most of the users of mt systems are not translators, should we in practice consider human parity if those users do not perceive a significant difference between human and machine translations, even if an experienced professional translator does?