Toward More Effective Human Evaluation for Machine Translation (2022)

Abstract: Improvements in text generation technologies such as machine translation have necessitated more costly and time-consuming human evaluation procedures to ensure an accurate signal. We investigate a simple way to reduce cost by reducing the number of text segments that must be annotated in order to accurately predict a score for a complete test set. Using a sampling approach, we demonstrate that information from document membership and automatic metrics can help improve estimates compared to a pure random sampling baseline. We achieve gains of up to 20% in average absolute error by leveraging stratified sampling and control variates. Our techniques can improve estimates made from a fixed annotation budget, are easy to implement, and can be applied to any problem with structure similar to the one we study.

## introduction as automatic natural language generation systems improve, evaluating them is becoming more challenging for both human and automatic methods @xcite . in machine translation, this has led to increased adoption of techniques such as mqm @xcite , an elaborate error-based methodology for scoring output, typically carried out by skilled human annotators. while mqm is more accurate than traditional crowd-based likert-type scoring, it can also be significantly slower and more expensive, creating a strong incentive to reduce annotation time and cost. in this paper we investigate a simple solution to this problem, namely reducing the number of text segments that a human annotator must rate. we assume a basic scenario in which a single annotator is given a test set to rate, and the task is to predict the average mqm score they would assign to the whole set by having them rate only a selected subset. this is a natural and versatile way to deploy human annotation effort within a framework like mqm; it differs from the tasks considered by recent work with similar motivation, which focus on system ranking @xcite or combining human and metric scores without the express aim of predicting human performance @xcite . although our experiments are carried out with mqm-based scores, our methodology is applicable to any setting in which numerical scores are assigned to items for later averaging. we approach the task of choosing segments as a sampling problem, and investigate classical methods for reducing sample variance and bounding estimation error. to improve accuracy, we employ two sources of supplementary information. first, in keeping with recent practice, we assume segments are grouped into documents. this lets us exploit the tendency of segments within a document to be relatively homogeneous. second, we make use of modern automatic metrics such as comet @xcite and @xcite which correlate better at the segment level with human judgments than traditional surface-based metrics like bleu @xcite . these serve as a rough proxy for human scores. we show that document and metric information can be used to reduce average estimation error by up to 20% over a pure random sampling baseline. due to high sample variance, it is difficult to reliably achieve a similar reduction in annotator effort for a given error tolerance. however, we suggest an alternative perspective in which our technique can be used to improve estimates made on the basis of a fixed rating budget. although there is no guarantee of beating random sampling in any particular case, there is a high probability of improving on average. this improved estimator is easy to implement, and applicable to any human labeling task that produces numerical scores, and for which document membership and automatic metrics are available. our work is most similar to that of @xcite , which we extend in several ways. we adopt their use of control variates, but consider multiple metrics rather than just one, including learned metric combinations; we also employ modern neural metrics rather than metrics based on surface information. we combine control variates with stratified sampling using either proportional or optimal allocation, and additionally evaluate an incremental scenario in which sampling adapts to observed ratings. finally, we investigate two analytical methods for bounding the error in our estimate. ## methods we assume a fixed test set consisting of translated segment pairs, and a single human rater who assigns scores to segments. each segment belongs to a document, and has an associated vector of scores from automatic metrics. our goal is to select an informative subset of segments to be labeled by the rater, and use the subset to predict the average score that would have resulted if we had asked the rater to label the whole set. by exploiting document and metric information, we hope to reduce the number of segments that must be manually labeled. formally, let x 1 , . . . x n be the segment scores, µ = n @xmath0, and σ 2 be the variance of the scores. the following side information is available for each segment i: an index d i that indicates its membership in one of d documents, and a vector of automaticmetric scores y @xmath1. unlike the segment scores, which are only revealed if they are in the selected subset, the side information is always available for the whole test set. we approach this task as the problem of sampling @xmath2, . . . , x n without replacement from the test set and deriving an estimate μ for µ from the sample such that e(μ) = µ (that is, μ is unbiased) and var(μ) is as small as possible. low-variance estimators make it more likely that the estimation error |µ -μ| will be small. a baseline is to draw n segments at random and compute their mean. this gives an estimate that is unbiased, with variance: we investigated two classical unbiased strategies for reducing variance relative to this baseline: stratified sampling and control variates @xcite . ## data our development data consists of mqm ratings made available by @xcite for 10 english-german and 10 chinese-english "systems" (including human translations and mt) from the wmt 2020 news test sets @xcite . each segment was annotated by three expert raters who assigned scores ranging from 0 (perfect) to 25 (worst). there were six annotators per language pair, each of whom rated all system outputs for a set of documents comprising approximately half the complete test set (about 710 segments / rater for german, and 1000 segments / rater for chinese). we created simulations for each rater and system combination, excluding the human-a "system", as it was the reference for the mt metrics we used as features. this resulted in 54 simulations for each language pair. for each simulation, the task is to predict the average score over the complete subset of segments annotated by a single rater for a single system. no knowledge of other segments, system outputs, or rater decisions is permitted to leak across simulations. as features, we used the 10 metrics submitted to the wmt 2020 metrics task @xcite ) that had highest average segment-level pearson correlation with the mqm scores in our dev data. 2 these correlations are generally poor: from 0.279-0.410 for english-german, and 0.425-0.465 for chinese-english. 3 to eliminate the effects of hyper-parameter tuning on the development data, we carried out additional evaluation on a test set consisting of newstest data from the wmt 2021 metrics shared task @xcite for english-german (17 systems), chinese-english (15 systems), and english-russian (16 systems). this is similar to the dev data, except that only one mqm rating is available per segment. the number of rated segments was 527 for german and russian, and 650 for chinese. english-russian ratings were annotated using a different mqm methodology (from unbabel rather than google), resulting in scores on a 0-100 scale, with 100 being best. as before, we created separate simulations for each system, omitting the human "system" used as a reference for the metrics. to avoid bias, rather than selecting metrics according to correlation, we chose the wmt 2021 primary submissions of two top-performing metrics from the dev data: bleurt and comet. 4 appendix a contains further details about scores and rater assignments for the dev and test sets. ## results we tested the sampling and estimation strategies described in section 2 by comparing them to the baseline of simple random sampling with a mean estimator. for each simulation we considered sample sizes ranging from 5-50% of the available data, at 5% intervals. 5 for each sample size and technique for establishing μ, we drew 100 random samples, computed the average and std deviation of the error |µ -μ| across the samples, then averaged the results across simulations to summarize performance at that sample size. we also measured the number of "wins"-simulations in which a technique had a lower average error than the baseline. finally, we aggregated these results across sample sizes to summarize performance in a single number. we begin by evaluating the stratified sampling methods described in section 2.1, comparing stratification over documents and over bins defined by metric scores. the latter were formed by scoring each segment with an average of the standardized metric scores assigned to it, then sorting and partitioning so each bin contained approximately 80 segments (8x larger than the average document). more elaborate clustering and metric-selection techniques did not improve over this method. performance was also quite flat as a function of bin size, though it worsened as bin size approached average document size. we tested both stratification methods with proportional and optimal allocation using averaged metric scores as proxies for human scores when estimating the variance in each bin. figure 2 shows absolute error for these methods as a function of sample size, and although stratification clearly reduces the error on average, the usefulness of this result is tempered by the large variances shown in we now turn to experiments with the controlvariate estimators described in section 2.2. figure 3 and ## discussion how should we interpret these results? if we had a more reliable way of binning segments with similar human ratings, or metrics that correlated better at the segment level, it would be possible to reduce variance to levels that would permit realistic error bounds. that would enable a scenario in which we could determine the number of segments n that need to be rated in order to estimate the complete test-set score to within a given tolerance. as it is, however, our error bounds are very large-and we do not manage to reduce them significantly with improved sampling and estimation methods. this is unlikely to change soon for complex annotation tasks like mt because because humans are noisy raters; as shown in in the absence of more reliable signals for reducing variance, a way to make practical use of the techniques we study is to flip the scenario around and aim to improve the quality of an estimate made from a fixed budget of n human ratings. it is common practice to obtain human annotations for only a portion of a larger test set due to time or cost constraints @xcite . in this setting, our techniques can lead to improved estimates compared to just taking the mean of randomly-selected segments (although there is no guarantee that they will do so for any given sample). the risks in applying this strategy are low. stratified sampling with proportional allocation provides an unbiased estimate of the test-set mean, with variance that @xmath5, and equality only in the case that the bins have identical statistics. the situation is trickier for control variates. in theory, the control-variate estimator is also unbiased, with lower variance than the sample mean, but this assumes that the test-set covariance cov(x, z) between scores x and the auxiliary variable z is known. since we only know the scores in the sample, we must rely on an estimate for cov(x, z), creating the possibility for errors if this is significantly larger than the true covariance. however, as @xcite point out, the error in the sample estimate for cov(x, z) diminishes as 1/n, much faster than the 1/ √ n rate for the error |µ -μ| in the estimated score. in our data, we found no appreciable degradation of performance on small samples, even ones containing as few as 30 items. based on these observations, we can make the following recommendations for improving the estimated mean score of a test set containing n items given a fixed number @xmath6: recent work has investigated incremental labeling tasks and/or combining human scores with automatic metrics. @xcite apply online learning algorithms to an mt system-ranking task in which different segments are selected for human evaluation on each iteration, using comet to fill in missing human scores in wmt 2019 data. their algorithm converges to correct results after several hundred iterations, but this condition is not detected automatically. thorleiksdóttir et al. (2021) use hoeffding's inequality to measure confidence in pairwise ranking decisions of varying difficulties for controlled text generation output; they consider human scores only. singla et al. (2021) sample foreign-language test responses for human grading, with the aim of improving over purely automatic scoring; a reverse problem to ours. hashimoto et al. (2019) propose a synergistic combination of human and automatic scoring for evaluating text generation. finally, there has been considerable work on measuring and rectifying inaccuracies in human annotation @xcite @xcite . we sidestep this issue by aiming to predict the performance of a single human rater, assuming that if this can be done accurately, conflicts among raters can be resolved in a post-processing step. ## conclusion we investigate two classical variance-reduction techniques for improving the accuracy of sampled human ratings of mt output, measured against the mean of all ratings for a given test set. we find that stratified sampling and control variates are complementary, contributing about equally to gains of up to 20% in average absolute error reduction com-pared to random sampling. exploiting this result to dynamically reduce annotator effort given a target error tolerance is not feasible due to the high variance in our data, but we propose that our techniques could instead be used to improve estimates made from a fixed annotation budget. concrete recommendations for this scenario are provided in section 5. our method is easy to implement, and can be applied to any setting involving averaged numerical item-wise scores where document (or other prior grouping) and automatic metric side information is available. in future work we look forward to delving into questions raised by our results: why doesn't optimal allocation work better, particularly in the incremental setting; is there a better way to estimate variance from metrics; why aren't metric combinations more helpful; and can error bounds be improved, perhaps with bootstrapping methods?