The statistical advantage of automatic NLG metrics at the system level (2021)

Abstract: Estimating the expected output quality of generation systems is central to NLG. This paper qualifies the notion that automatic metrics are not as good as humans in estimating system-level quality. Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimators. We compare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Measuring this error is complicated: predictions are evaluated against noisy, human predicted labels instead of the ground truth, and metric predictions fluctuate based on the test sets they were calculated on. By applying a bias-variance-noise decomposition, we adjust this error to a noise-free, infinite test set setting. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected. In MT, we identify two settings where metrics outperform humans due to a statistical advantage in variance: when the number of human judgments used is small, and when the quality difference between compared systems is small.

## introduction automatic metrics are involved in many developmental settings for natural language generation (nlg) systems. in machine translation (mt), metrics like bleu @xcite enable settings where the amount of human effort required would be infeasible, such as architecture or hyperparameter search @xcite . as objective, reproducible quantities, bleu scores facilitate cross-paper comparisons @xcite . historically, progress in mt has been attributed to its bias δ h s,s (true difference) 0 dist. of estimator metric ( δ m s,s ) human ( δ h s,s ) figure 1 : distribution of estimators for the true difference in system quality δ h s,s between two generation systems (for illustrative purposes). notation is defined in §2.3. an estimate incurs prediction error if its sign is opposite to the true difference. while humans provide an unbiased estimator of the difference, a biased estimator derived from a metric can have a smaller error probability (shaded areas) due to its lower variance. evidence supporting the illustration can be found in §5. in all these subfields, the primary goal when conducting evaluation is typically to compare nlg systems. both human annotators and automatic metrics produce segment-level scores, i.e., scores for individual examples, so comparing systems requires aggregating segment-level scores into an overall system-level score for each system. ideally, we would compare systems by their expected human annotator score (an average over infinite human judgments), which we term the true system quality. in practice, we can only estimate this expectation with a sample mean over a finite number of human judgments. metrics offer a cheaper alternative: we can instead compare systems by their aggregate metric scores on a number of system out-puts. when comparing systems, we care primarily about how well we estimate the difference of their true system qualities, and in particular the sign of this difference (i.e., which system is better), which we term the true pairwise label. there is a gap in our understanding of systemlevel metrics. to recount a perplexing anecdote, in the most recent edition of the wmt metrics shared task @xcite , initial human evaluation disagreed with most metrics on a pairwise prediction of two translation systems. in a manual re-evaluation, the second round results favored the metrics. our paper offers a statistical explanation for how humans could go "wrong": even if human estimation for the difference in system quality is unbiased, it has high variance. on the other hand, while estimators based on metrics are biased, they have low variance. it is therefore possible for metrics to give a more accurate pairwise prediction than humans when the bias is small (see illustration in figure 1 ). our paper explores this distinction in the following three questions: ## decomposing observed metric error two sources of variation distinguish the observed pairwise error (11) from the true error in (10)the noise in the human predicted labels due to finite judgements, and the variance in the metric due to finite test sets. approximating (11) is straightforward with the bootstrap, but disentangling the error from these two sources of variation requires more care. with the bias-variance-noise decomposition, we can adjust our observed error estimates to the noise-free, infinite test set setting of the true error. ## comparing to the human estimator in §4, several mt metrics approach the error of the wmt human evaluation. the wmt human evaluation is expensive, using thousands of judgments per translation system. while each human judgment has associated monetary cost, once a large test set is collected, running metrics only incurs computational cost. this section explores this asymmetry, and seeks to understand how much metric predictions are worth, in terms of human judgments. ## the limits of human evaluation the perfect annotator provides optimistic figures for human annotation, providing the best performance for a fixed number of judgments, and requiring the least judgments for a fixed performance. in §5, we saw that the perfect annotator is weak at low number of judgments, due to its non-zero variance. in this section we identify another consequence of the perfect annotator's variance, where estimating small differences in system quality is hard. ## caveats to the analysis our analysis assumes that the human judgments are unbiased. in wmt16-19, direct assessment @xcite was used to elicit judgments from a combination of crowdworkers and researchers. direct assessment (da) uses an adequacy evaluation prompt ("rate how much you agree that the output translation adequately expresses the meaning of the reference translation") and asks contribu-tors to rate on a 0-100 scale. the unbiased ground truth is not a fixed goalpost. a number of factors are known to change the eventual ranking of translation systems with human scoring. employing a different collection methodology, such as human translation edit rate (hter) of instead of da, can result in divergent system rankings @xcite . in an earlier edition of wmt, da judgments were collected with both a grammaticality prompt and an adequacy prompt, corresponding to different system rankings by the respective attribute @xcite . several studies have shown scoring differences between professional linguists and crowdworkers which are due in part to the fact that linguists are more sensitive to linguistic phenomena @xcite . the goals of an evaluation should be decided by the practitioner. we do not give suggestions on any particular goals, and practitioners should understand what their application is, and which evaluation is the best approximation (refer to @xcite . unfortunately, since the existing data in this domain is limited, our analyses are limited as well. however, the statistical techniques apply to any empirical method. we hope that our analysis inspires others to think about statistical limits in this domain. ## pushing the limits of evaluation to push the limits of what can be evaluated, we need to improve on fundamental aspects of human evaluation. on the human side, we may focus on creating larger effect sizes or reducing noise by adopting new annotation schemes @xcite or employing professional linguists @xcite . to make the human estimator more efficient, we may consider adaptive data collection techniques to stop data collection early when significance is achieved, in a statistically sound manner @xcite . strategies combining human and metric evaluation are also shown to have potential. variance reduction techniques can be applied to the human estimator by taking advantage of strong metrics @xcite . another bottleneck in human evaluation is in the random sampling of the test set. metrics could form the basis of an importance sampling procedure to choose test sets that would best differentiate two systems, as a form of robust evaluation @xcite . on the metric side, if we can reliably estimate metric bias, we can skip human evaluation altogether when the metric is known to be good. probabilistic reinterpretations of current metrics could be a useful technique for confidence estimation @xcite . optimistically, metrics could have provable guarantees, ensuring the correctness of metric decisions @xcite . ## best practices for metrics research we reinterpret problems in evaluating metrics with correlation ( §2.2) as a set of guidelines for metrics research. to next year's organizers of the wmt metrics shared task and the broader metrics community we suggest the following: (1) pairwise accuracy has desirable properties as an evaluation measure for metrics. our bias-variance-noise decomposition shows that the observed pairwise accuracy is very close to the true pairwise accuracy from a noise-free, infinite test set setting ( §4.4). we suggest the use of pairwise accuracy as it reflects metric performance well (which may be verified using this analysis). as a normalized form of pairwise accuracy, kendall's τ is also a suitable measure. (2) since pairwise accuracy is computed against noisy human predictions, on average, it should be impossible for metrics to achieve a perfect accuracy. we suggest providing an upper bound of metric performance ( §4.2) to clarify how much improvement is possible for metrics on the dataset. ## related work the fact that a manual evaluation can be weak, and an automatic one can be better is gaining attention in the metrics community. @xcite studied a disagreement between crowdworkers and metrics, and a reevaluation favored the metrics over the human prediction. recently, @xcite shows that metrics can achieve higher agreement with professional linguists than crowdworkers in judging translation systems. their results fit into our formalization: if we assume professional linguists are unbiased, the bias and variance properties of metrics combined are superior to those of crowdworkers. our analysis assumes that crowdworkers are unbiased, where they assume professional linguists are instead. we wish to highlight several works which inspired the elements of ours: @xcite and @xcite formalize metrics as statistical estimators and provide understanding of their statistical properties and limits. in the replication of imagenet, @xcite found that dataset bias accounted for classifier performance differences between the original and the replicated dataset, and provide a decomposition for the sources of error. in automated essay scoring, scorers are often evaluated against noisy human judgment, and @xcite developed the prmse to calculate the mse between scorer prediction and the true judgment, rather than noisy judgment. finally, in bioinformatics, @xcite derive an upper bound of the r 2 coefficient due to experimental noise when regressing on experiment-derived results. ## conclusion through rigorous comparison between metrics, humans, and the perfect segment-level annotator, we identify the settings where metrics outperform humans due to a statistical advantage in variance. these results challenge the notion that metrics are always secondary to human evaluation. instead, we encourage practitioners to understand when human evaluation is weak, and when metrics are necessary. finally, we hope to provide tools for analysis and future directions for evaluation.