Searching for a Higher Power in the Human Evaluation of MT (2022)

Abstract: In MT evaluation, pairwise comparisons are conducted to identify the better system. In conducting the comparison, the experimenter must allocate a budget to collect Direct Assessment (DA) judgments. We provide a cost effective way to spend the budget, but show that typical budget sizes often do not allow for solid comparison. Taking the perspective that the basis of solid comparison is in achieving statistical significance, we study the power (rate of achieving significance) on a large collection of pairwise DA comparisons. Due to the nature of statistical estimation, power is low for differentiating less than 1-2 DA points, and to achieve a notable increase in power requires at least 2-3x more samples. Applying variance reduction alone will not yield these gains, so we must face the reality of undetectable differences and spending increases. In this context, we propose interim testing, an “early stopping” collection procedure that yields more power per judgment collected, which adaptively focuses the budget on pairs that are borderline significant. Interim testing can achieve up to a 27% efficiency gain when spending 3x the current budget, or 18% savings at the current evaluation power.

## introduction in machine translation (mt), pairwise evaluations are conducted to identify the better system over a test domain. mt has long taken intrinsic quality as an object of interest, and assumes it can be determined directly from the output @xcite . most practitioners accept that human judgments reflect such quality, and take human evaluation as the gold standard @xcite . in conducting an evaluation, the experimenter must allocate a budget to collect human judgments, and so evaluation can be an expensive endeavor. no one in the history of mt research has ever been satisfied with the cost or reliability of human evaluation (graham et al., 2017; chaganty et al., 2018; * work done at microsoft. collect 1200 judgments significance: @xmath0.05? interim: collect 600 collect 600 futility: @xmath1.5? ## related work there is a tradition of using test sets to estimate system performance over the general domain in machine learning @xcite . there have been calls for statistically rigorous evaluation in natural language processing using significance testing @xcite , however its adoption in reporting has been mixed. for a classic task such as part-of-speech tagging, evaluation is generally significant/consistent even for small gains (gorman and bedrick, 2019). in mt, even moderate differences in metric gains (e.g. da, mqm) may not be consistent, so there is a stronger need for significance testing. historically, mt evaluation has been heavily based on statistical significance @xcite . mdes have been used to describe the power of experiments in contexts such as education (which program results in increased test scores?) and sociology @xcite . @xcite empirically investigate the conventional wisdom that a certain metric gain corresponds to significance (e.g. 0.5 for bleu). this threshold is exactly an evaluation's mde. they find that a threshold has strong empirical backing, but a few experimental parameters affect this threshold. in our work, we propose taking a sensitivity perspective to evaluation, and reporting the expected mde of an experiment instead of the other experimental parameters. any statistical technique that reduces the cost of human evaluation is, in another view, improving the power offered by some fixed budget. @xcite first proposed applying control variates to human evaluation. control variates increase the sensitivity of an evaluation by leveraging information from a metric. this formulation conveniently allows us to analytically understand its performance based on the experimental conditions. in realistic experimental conditions, they found that the sample efficiency gain is at most 20%, which is in line with results reported in mt @xcite . @xcite propose using online learning to adaptively spend the evaluation budget on determining the best mt systems. however, their technique lacks in statistical rigor for decision making. knowing when to "early stop" an evaluation allows us to adaptively spend the budget on difficult pairs and save on easily distinguished pairs. it is known that peeking at the p-value while data collection is ongoing is problematic. peeking inflates the chance of observing significance and the chance that such significant observation is incorrect (albers, 2019). while always valid p-values can be calculated that adjust for this error and can be reported at any time, they are mathematically difficult to apply @xcite . interim testing has been used in medical trials, where experimenters have an ethical consideration in stopping the experiment early (o' @xcite . by planning the number of peeks in advance, interim testing can offer rigorous statistical inference while potentially saving time and effort, packaged in an easy to understand technique @xcite . our work investigates whether the tradeoff between power and savings is favorable for mt evaluation. ## a primer on inferential statistics we consider pairwise comparisons as the basic unit of evaluation echoing calls from @xcite and @xcite . pairwise comparisons are more interpretable than correlations, and more practical for production deployment scenarios. in a pairwise comparison we test the difference between two systems a and b. if you were just to collect a number of da judgments for each system and declare a winner, a repeat experiment could yield different results due to experimental noise. a statistical test guards against making an incorrect conclusion due to experimental noise. to do this, we assume a null hypothesis (that a is better than b) and examine how likely we could have made observed our data under this assumption. there are two outcomes of conducting a test: in the case of (i), a significance test usually guarantees a false detection rate of at most α, where usually α = 0.05. therefore, the best outcome of statistical testing is the presence of significance, where our inferences enjoy a low false detection rate. the rate at which we can declare significance is called an experiment's power (typically denoted as 1β, where β is the false negative rate). in pairwise comparison, our evaluation should have an accuracy (1α)(1β) against the true, pairwise judgment. 1 intuitively, statistical testing can be loosely thought of as reducing the width of two confidence intervals, spaced by the true system difference of a and b @xcite . the power of an experiment is then a function of these three aspects: the more judgements you can collect the smaller these confidence intervals will be. when the confidence intervals don't overlap, the comparison is likely to achieve significance. these three factors all play a role in whether the intervals will be narrow enough. if we know two of (a), (b), or (c), we can use the appropriate power analysis to deduce the third. typically, we will observe the (b) human judgment variance, and make a guess at what the true difference (a) would be, to compute what (c) the budget we would have to spend is. when providing estimates for the budget, we would provide estimates under a range of guesses at what the true difference is @xcite . alternatively, we may also ask what the minimum detectable effect is for some fixed budget. @xcite conducted power analysis in mt and found that small differences 0 1 2 3 4 5 6 obs. system difference require an infeasible amount of budget. this gives a hint that most of our mt evaluation is underpowered. consistently conducting underpowered experiments run the risk of inflating the error rate in significant observations @xcite . ## dataset mt evaluation has an established tradition of conducting human evaluation and releasing public datasets. at the time of writing, the current annotation method of choice in mt is direct assessment (da; @xcite . direct assessment asks annotators to rate a translation's quality on a sliding point scale from 0-100. we study the shipdata presented in @xcite , which is the largest human evaluation dataset of pairwise comparisons, accumulated over two years from internal evaluation campaigns at microsoft translator. no text is contained i.e. source, references, or outputs, but the raw da scores are sufficient for our purposes. we focus on this dataset because it is large and often contain comparisons between state-of-the-art systems. it contains 4004 pairwise comparisons between two systems, where each system pair contains about 600 human judgments per system (1200 for both systems). ## the sensitivity approach to evaluation the basis of solid comparison is significance. therefore, we need a way to reason about the power of an experiment. in this section, we recommend a sensitivity approach to evaluation, and retrospectively deduce the power of previous evaluations. by looking at the observed effect sizes we can also set a meaningful target power. ## known unknowns now that we have established a way to reason about experimental power, we conduct power analysis to understand how much more gain we need to improve our power to a desired sensitivity. we investigate whether variance reduction techniques are sufficient, and conclude that the only way forward is to increase the annotation budget. ## spending effectively to have a notable gain in sensitivity, variance reduction alone is inadequate. therefore, spending is necessary in the search for higher power. this section describes a simple yet statistically rigorous way of "early stopping" in a human evaluation campaign. interim testing adaptively allocates the budget to borderline significant pairs, and can be seen as an efficient way to spend. ## limitations the most important assumption of our work is in the use of direct assessment (da). while our methods can generalize to any real valued judgment, we analyzed da because of its widely recognized, gold standard status in mt evaluation. da is a particularly noisy judgment, and so the power and variance reduction results are pessimistic. however, we believe that the study of annotation will be the most important direction in mt evaluation. let's take @xcite , where one of the first claims of mt-human parity was made. by their evaluation, which was conducted according to the community standard, no significant difference was found between human and machine translations with a reasonable budget, and so a tie was declared. @xcite reassesses this claim, and essentially presents a series of alternative evaluations and observe significant differences that contradict with @xcite . this is just one of many studies which compels an alternative evaluation with qualitative insight @xcite @xcite . our perspective is that significance is only one pillar of mt evaluation. it is our hope that the analyses in this work will further our understanding of significance and evaluation power. however, the second pillar of mt evaluation is in the annotation method. while power is quantitative, the study of annotation methods will be qualitative. going forward, understanding how we can change the annotation method to increase the power will be crucial. we will need good qualitative understanding to be able to move away from da and establish new gold standards. in addition, we showed that interim testing is only effective for pairwise comparison. future work should look to make savings in the leaderboard styled evaluation of wmt. this may come in the form of generalizing interim sampling for multiple comparisons or formalizing the bandit results from @xcite in terms of statistical inference. ## conclusion our work is motivated by the cost of human evaluation in machine translation. before searching for a higher power from our current budget, we determined how much more power was necessary. in doing so, we recommend taking a sensitivity approach to evaluation. from here we came to the conclusion that to achieve the power/sensitivity necessary, variance reduction alone would be insufficient, and spending is our only option. if we decide to allocate larger budgets, interim testing is a more effective way to spend, which can yield 18% savings at the current evaluation power, or 27% savings at 3x the original budget.