What’s the Difference Between Professional Human and Machine Translation? A Blind Multi-language Study on Domain-specific MT (2020)

Abstract: Machine translation (MT) has been shown to produce a number of errors that require human post-editing, but the extent to which professional human translation (HT) contains such errors has not yet been compared to MT. We compile pre-translated documents in which MT and HT are interleaved, and ask professional translators to flag errors and post-edit these documents in a blind evaluation. We find that the post-editing effort for MT segments is only higher in two out of three language pairs, and that the number of segments with wrong terminology, omissions, and typographical problems is similar in HT.

## introduction machine translation (mt) quality has improved substantially over the past years, allegedly to the degree that it is no longer distinguishable from professional human translation (ht). the first claims of human-machine parity were based on mt systems geared to news translation @xcite , and soon refuted due to weaknesses in the evaluation methodology. reproductions with professional translators rather than crowd workers and full documents rather than single sentences likewise concluded that ht was superior to mt in terms of both accuracy and fluency @xcite . human-machine parity claims may not hold with mt systems for broad domains such as news articles, but systems geared to narrower domains c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. ## background how to tell whether a translation is good or bad is one of the most important and one of the most difficult questions asked in connection with translation. best practices for evaluating ht and mt differ, and assessments of human-machine parity have largely ignored the former. ## evaluation design we propose an experimental design for combined evaluation of ht and mt that avoids the weaknesses of previous assessments on humanmachine parity in translation (section 2). ## experimental results we use the evaluation design described in the previous section to compare ht to mt in an experiment with three language pairs and ten professional translators. the study is conducted within the language services department of a multinational insurance company. ## discussion we discuss design decisions in our evaluation and alternative approaches to inference testing, and contextualise our results within the ongoing discussion on human-machine parity in language translation. ## conclusion in a blind evaluation, ten specialised translators post-edited and flagged errors in pre-translated documents in which domain-specific mt was interleaved with professional ht. the evaluation comprised three language pairs: de-en, de-fr, and de-it. mt required more post-editing than ht on average, but surprisingly, the difference is not significant in de-it, where mt produced more segments that needed no post-editing at all, and slightly less segments that needed substantial post-editing. we also analysed if the propor-tion of segments that contain wrong terminology, omissions, or typographical errors varies among ht and mt, and found no significant dependency in any language pair. mt produced considerably more segments with wrong terminology in two out of three language pairs, but slightly less segments with omissions or typographical errors in at least one language pair each. apart from implying that mt can now reach remarkable quality in domain-specific settings, our results show that professional translators may postedit professional ht almost as much as mt, and tend to rate the two similarly in terms of issues with terminology, omission, and typography. the caveat here and an aspect that warrants further investigation is that we made our participants believe that the ht they were evaluating was mt. from a methodological point of view, it would be interesting to test if this experimental manipulation would also work the other way around, and analyse if translators treat ht and mt differently depending on what they believe it is. from a more practical perspective, it might also be worth exploring whether the proposed evaluation design could help demonstrate the potential benefits of mt to people who are still sceptical about the technology.