Detecting Post-Edited References and Their Effect on Human Evaluation (2021)

Abstract: This paper provides a quick overview of possible methods how to detect that reference translations were actually created by post-editing an MT system. Two methods based on automatic metrics are presented: BLEU difference between the suspected MT and some other good MT and BLEU difference using additional references. These two methods revealed a suspicion that the WMT 2020 Czech reference is based on MT. The suspicion was confirmed in a manual analysis by finding concrete proofs of the post-editing procedure in particular sentences. Finally, a typology of post-editing changes is presented where typical errors or changes made by the post-editor or errors adopted from the MT are classified.

## introduction over ten years of wmt (conference on machine translation, barrault et al., 2020) 1 saw a number of manual evaluation methods and established the best strategies for obtaining reference translations for automatic evaluation, see appendix b in wmt 2020 findings (barrault et al., 2020). one of the instructions for preparing the reference translations explicitly prohibits using any machine translation. yet, in 2020, one of the agencies has not followed this instruction. not only was it easy to recognize, but we learned several novel insights into manual evaluation of translation, by examining post-edited and independent reference translations and providing a small contrastive style of manual evaluation. ## dataset we used the english-czech part of wmt2020 (barrault et al., 2020) news test set, which consists of 130 documents (1418 segments) originally written 1 http://www.statmt.org/wmt06 till wmt20 in english -news stories downloaded from web. the test set comes with an official reference translation into czech (ref1) provided by the wmt organizers and done by a professional translation agency. there are also 8 machine translations submitted by the participants of the wmt news translation shared task and 4 translations by online systems anonymized as online-a, online-b, online-g and online-z. we focused on three translations: the official reference, ref1; the best-performing mt system (according to the official wmt manual evaluation), cuni-doctransformer (popel, 2020); and the best-performing online system, online-b. we hired two professional translators (native czech speakers) to translate the whole wmt20 test set, thus creating additional references ref2 and ref3. we also hired 18 annotators to judge the translation quality of ref1, cuni-doctransformer and online-b. @xcite the annotators assessed 90 of the 130 documents, using the rankme evaluation (novikova et al., 2018) following the methodology of popel et al. (2020). in this rankme evaluation, fluency, adequacy and overall quality are evaluated in a source-based sentencelevel document-aware fashion, on a 0-10 scale, where all the evaluated translations are shown on the same screen, allowing thus better reliability in comparisons; see section 5 for details. ## automatic analysis of references the reason for this surprising observation is that most sentences in ref1 are actually post-edited versions of online-b, as we show in sections 3.1 and 4 and as was acknowledged by the agency after our investigation. thus, ref1 and online-b are more similar than if ref1 had been translated from scratch. @xcite it is well-known that bleu (and other automatic metrics based on similarity with reference translations) is biased when evaluating a system which was used as a basis for post-editing the reference translation. it is important to note that the official (manual) evaluation carried out by wmt for the affected english-to-czech translation direction was source-based da. the annotators of the da thus were not affected by the quality of references; instead, they blindly rated ref1 as if it was another competing translation system. the resulting scores of da document that source-level da is sufficiently reliable, robust to invalid references. at the same time, it was a little surprising to us that "mere post-editing" can increase translation quality so substantially that ref1 significantly outperformed all other systems. despite the remaining translation errors in ref1, see below, the translator/post-editor did the job well. for a finer analysis, we process the news test set at the level of individual documents in the following. ## post-editing changes typology in this part of our study, we would like to present a classification of post-editing changes observed in texts which we claim to be post-edited machine translations. these changes signal that the reference translation ref1 has been actually created by post-editing a mt system. for this purpose, we used mt-compareval (klejch et al., 2015) to select 27 sentences which show the highest n-gram overlap with the suspected mt system. we analyzed the edits made by a post-editor in a mt output and compared the source text (english), the mt output (czech) and the output of the manual post-editing process (czech). based on the particular changes found after comparing these three versions of our sentences, we defined the following categories (with examples where src is the source sentence, onlb is the mt online-b and ref1 is the human post-editing). in each category, we would like to present particularly noticeable changes which we assume to be clear evidence of the post-editing procedure. we classify these changes into three categories: furthermore, we also found these errors or conspicuous structures in the post-edited output: for all these four categories, we can state they prove the final output is a result of a post-editing process. the present tense (@xmath0. changes between past and present tense (while keeping the same verb) occurred relatively often (8 cases) in the investigated 27 sentences. category 3: case and noun number changes: src nobel laureate winner liu xiaobo [. . . ] onlb nositel nobelovy ceny za laureát liu xiaobo [. . . ] ref1 nositel nobelovy ceny laureát ů liu xiaobo [. . . ] despite the changes made, the semantic defectiveness of onlb has been preserved. in czech, the nobel prize is nobelova cena and the word laureátů (genitive sg.) is wrong. in this case, laureát nobelovy ceny would be acceptable. the correct comparative of @xmath1. the phrase více vnímavý is considered non-standard and rather rare. foot_3 ## human evaluation -rankme in our blind rankme evaluation by 18 human judges (6 professional translators, 6 students from ma study program translation: czech and english at the institute of translation studies, charles university's faculty of arts, foot_4 and 6 nonprofessionals with excellent knowledge of the english language), 90 documents (887 segments, typically sentences) were evaluated on a sentence level in terms of adequacy, fluency and overall quality (as defined by popel et al. (2020)). every document was scored by two evaluators, and every evaluator scored ten different documents. then we could compare the ratings of the post-edited ref1 and of the suspected onlb. according to the ratings, the translation quality is, in all cases, better in ref1 compared to onlb. the post-editor improved the quality of the onlb in all three categories: in adequacy (increased by 2.23 on average), fluency (2.70) and overall quality (2.55), on a 0-10 scale. as could be expected, the most significant improvement occurred in fluency. apparently, the post-editor was more sensitive to errors in fluency rather than adequacy, as shown also in our analysis of post-editing changes. ## conclusion we suspected that wmt20 reference translations were actually post-edits of one of the participating systems, online-b, and not created independently. we proposed two methods to detect this situation, both confirming our suspicion. in a subsequent manual analysis, we provided numerous examples of translation choices in the reference translation which are extremely unlikely to happen when translating from scratch. @xcite the result of this analysis is a draft typology of post-editing strategies. we see this typology as an interesting basis for further inspection of translations in a world where post-editing becomes the industry standard. by contrasting direct assessment scores with our manual evaluation (rankme), we observed the post-editor improved primarily fluency of the translation and less so its adequacy. it would be useful to confirm this observation on a larger sample.