Statistical Power and Translationese in Machine Translation Evaluation (2020)

Abstract: The term translationese has been used to describe features of translated text, and in this paper, we provide detailed analysis of potential adverse effects of translationese on machine translation evaluation. Our analysis shows differences in conclusions drawn from evaluations that include translationese in test data compared to experiments that tested only with text originally composed in that language. For this reason we recommend that reverse-created test data be omitted from future machine translation test sets. In addition, we provide a re-evaluation of a past machine translation evaluation claiming human-parity of MT. One important issue not previously considered is statistical power of significance tests applied to comparison of human and machine translation. Since the very aim of past evaluations was investigation of ties between human and MT systems, power analysis is of particular importance, to avoid, for example, claims of human parity simply corresponding to Type II error resulting from the application of a low powered test. We provide detailed analysis of tests used in such evaluations to provide an indication of a suitable minimum sample size for future studies.

## introduction human-translated text is thought to display features that deviate to some degree from those of text originally composed in that language. @xcite report that translated text can: be more explicit than the original source, less ambiguous, simplified (lexical, syntactically and stylistically); display a preference for conventional grammaticality; avoid repetition; exaggerate target language features; as well as display features of the source language. the term translationese is often used to describe the presence of such phenomena in translated text. standard evaluation protocol in machine translation (mt) comprises system tests on a sample of human-translated text. since creating this humantranslated text is expensive, re-use of test sets for both directions of translation is commonplace, without regard to whether source or target contain features of translationese. for example, translation shared tasks at the conference on machine translation (wmt) @xcite generally test translation between a given language pair with two portions of data combined to make up the overall test set. @xcite of the test data (accounting for approximately 50% of sentences) is made up of text that originated in chinese that was humantranslated into english, while portion (b) (i.e. the remaining 50%), was translated in the opposite direction, originating in english with manual translation into chinese. the motivation for creating the test data in this way is to create test sets for both directions simultaneously (so at no extra cost). 1 although translationese has been cited as a likely confound in mt evaluation results in the past @xcite @xcite , to the best of our knowledge, no detailed investigation into the impact of translationese on the accuracy of mt evaluation has been reported to date. with this aim, we examine the degree to which translationese phenomena may impact human and automatic evaluation results in mt. we firstly examine past results of wmt shared tasks, a main venue for mt evaluation, and reveal that although system rankings are overall very similar for human evaluation of forward and reverse test data, in a small number of cases system rankings diverge to a more serious degree. for example, for turkish-english translation at wmt-18 forward and reverse system rankings correlate at only @xmath0.703 in one case. apart from human evaluation, much more concerning is the divergence in forward and reverse rankings when bleu is relied upon for evaluation of systems, where the correlation can be as low as 0.106 in the worst case. subsequently, we provide a reassessment of a human evaluation previously criticized for including reverse-created test data that claimed human parity of chinese to english mt. we reveal insights into additional potential sources of inaccuracy of conclusions beyond the presence of translationese with the aim of preventing future inaccuracies. inspired by this work, other authors considered the effect of the 50/50 set-up on evaluation using wmt data. @xcite questioned whether improvements in performance due to backtranslation were just an artifact of the test set construction. they found that, whilst back-translation had a disproportionately large positive effect on bleu for reverse-created test sets, human evaluation showed that back-translation did indeed provide robust improvements to mt for forwardcreated text. related to this, @xcite also showed bleu to be misleading on the reversecreated part of the test sets, when analysing why their automatic post-editing (ape) method produced improved translations according to human evaluation, but not according to bleu. given the concern in the community about using reversecreated test sets, the organisers of the wmt19 news translation task used only forward-created sentences in all their test sets @xcite . in this current paper we provide detailed evidence to justify this decision. we note that @xcite also provide analysis of the effect of reverse-created test sets on wmt evaluation campaigns. however they focus only on the effect of translationese with re-spect to human evaluation, without considering its differing effect on automatic evaluation. also, they do not consider the problem of statistical power in human evaluation, which we raise below. the use of reverse-created test sets was not the only concern raised by @xcite and @xcite . both used more context than the original sentence-level evaluation in @xcite , @xcite now asking human judges to assess entire documents, and @xcite involving assessment of mt output sentences in the order that they appeared in original documents. furthermore, in contrast to the use of direct assessment @xcite by @xcite , both reassessments used relative ranking, a method formerly used in wmt for evaluation @xcite @xcite @xcite @xcite @xcite , but now abandoned, partly due to low inter-annotator agreement. therefore, although both re-evaluations improved the methodology employed in two respects, by eliminating reverse-created test data and including more context, both potentially include other sources of inaccuracy, such as lack of reliability of human judges when human evaluation takes the form of relative ranking. furthermore, @xcite employ trueskill to reach the conclusion that the mt system in question has not achieved human performance, and although trueskill has been used in past wmt evaluations to produce system rankings, its aim is to minimize the number of judgments required to produce those rankings when resources are limited. so results may not be directly comparable with results of standard statistical significance tests, now current practice at wmt evaluations. finally, neither @xcite nor @xcite discuss statistical power of significance tests used to distinguish the performance of system and human, an important aspect of evaluation and one of particular importance with respect to evaluations that aim to investigate claims of human parity, where type ii error could result in false claims. besides criticisms already made of the human evaluation in @xcite , an additional aspect of importance not yet highlighted is the proportion of distinct translations that were included in the original human-parity evaluation of systems, a consideration that also relates strongly to the ques-tion of statistical power. in most mt human evaluations, it is not feasible to evaluate the full test set of sentences for all systems and it is common to instead evaluate a sample of translations, usually drawn at random from the test data. in current wmt evaluations, for example, translations of all test sentences produced by all participating systems are pooled and a random sample is humanevaluated. this method ensures that as great a number as possible of distinct test sentences are examined. alongside system performance estimates, wmt also reports the number of distinct test sentences evaluated, n, and it is this number that they consider the sample size used for statistical significance tests subsequently used to draw conclusions about which competing systems outperform others. for example, all else being equal, a difference in system performance estimates for a pair of systems computed from a larger set of distinct translations is interpreted as more reliable. other mt human evaluations, despite claims of following wmt human evaluation methodology, have diverged from this method of sample size computation, however, including the human-parity evaluation of @xcite and @xcite . for example, although a large sample of human judgments is reported as @xmath1,827 per system in @xcite , firstly this number in fact included quality control check translations, generally removed from data before computing sample sizes. more importantly however, very high numbers of repeat evaluations of the same translations were included in the human-parity evaluation of @xcite . in other words, a very low number of distinct test sentences were in fact human evaluated despite reporting a large sample size. the method of computing sample size therefore diverges from that reported of wmt evaluations in a small but important way. the sample size reported instead corresponds to the total number of human ratings collected as opposed to distinct test sentences (as in wmt evaluations). in this current work, we make this important distinction explicit by referring to the number of distinct test sentences evaluated as n and the number of human judgments collected as n . we also recommend this distinction be made and adopted as common practice in future human evaluations of mt or that the number of distinct translations (as opposed to the number of human evaluations) be reported as the sample size. we therefore rerun the evaluation using the original translation data included in @xcite with entirely up-to-date wmt human evaluation methodology in addition to ensuring that a suf- ## translationese using reverse-created test data is thought to unrealistically decrease the difficulty of mt evaluations @xcite , because in real-world mt scenarios, input text is unlikely to very often comprise text that has already been translated from the target language. we therefore compare results of systems when test data is split according to the creation direction and examine differences in scores for systems in terms of both human and automatic metrics. ## re-evaluation of human parity claims as described in detail in section 2, past reevaluations of human parity claims were hampered by sub-optimal test settings. in our re-evaluation, we firstly carry out statistical power analysis so that, in the case of encountering any ties between systems or indeed human and system, tests used to draw conclusions will have sufficient statistical power to avoid human-parity claims that in fact simply correspond to type ii error. statistical power is of particular importance when considering document-level evaluation due to the fact that gathering ratings of documents as opposed to sentences requires substantially more annotation time and for this reason is highly likely to result in a reduction in the number of assessments collected in any evaluation. for example, @xcite included as few as 55 documents in their re-evaluation of @xcite . our concern about a potential substantial reduction in sample size in future document-level evaluations is well-founded therefore, especially considering standard segment-level mt human evaluations commonly include a sample of 1,500 segments. in the case of @xcite this corresponds to an extreme reduction of approximately 96% to the sample size. since the very nature of the question being investigated involves a potential tie between human and machine, such a small sample size is a serious risk to the reliability of conclusions drawn simply due to its impact in terms of statistical power. as a rough guide to what constitutes sufficient statistical power, we borrow the five-eighty convention from the behavioural sciences that provides a balance between type i versus type ii error, where significance and power levels are set at 0.05 and 0.8 respectively @xcite . ## conclusion we explore issues relating to the reliability of machine translation evaluations. firstly, we provide a detailed analysis of how the presence of translationese phenomena can adversely affect machine translation results. in terms of the legitimacy of machine translation evaluation results, our analysis provides sufficient evidence that translationese is a problem for evaluation of systems, in particular in terms of comparison of system performance with automatic metrics such as bleu. this results in our first recommendation in future mt evaluations to avoid the use of source side test data that was created via human translation from another language. we provided guidance in relation to sample size and statistical power to help planning future human evaluations of mt, particularly relevant to document-level human-parity investigations. this guidance will help to avoid false conclusions due to the application of low powered statistical tests.