Finding Replicable Human Evaluations via Stable Ranking Probability (2024)

Abstract: Reliable human evaluation is critical to the development of successful natural language generation models, but achieving it is notoriously difficult. Stability is a crucial requirement when ranking systems by quality: consistent ranking of systems across repeated evaluations is not just desirable, but essential. Without it, there is no reliable foundation for hill-climbing or product launch decisions. In this paper, we use machine translation and its state-of-the-art human evaluation framework, MQM, as a case study to understand how to set up reliable human evaluations that yield stable conclusions. We investigate the optimal configurations for item allocation to raters, number of ratings per item, and score normalization. Our study on two language pairs provides concrete recommendations for designing replicable human evaluation studies. We also collect and release the largest publicly available dataset of multi-segment translations rated by multiple professional translators, consisting of nearly 140,000 segment annotations across two language pairs.

## introduction when conducting an evaluation to rank natural language generation (nlg) systems, including modern "generative ai" systems, the goal is to reliably recover the "true" ranking of the systems by quality on the tasks/domains of interest. to date, human evaluation remains the best way to do this. however, human raters can exhibit different behaviors when rating nlg outputs. for example, raters can have different stylistic preferences, and some may grade more leniently or harshly than others. rigorous rater training procedures and precise annotation guidelines are helpful in addressing this but are not guaranteed to completely eliminate rater differences. coupled with the fact that any given study will evaluate only a subset of potential nlg outputs of interest using a finite set of raters, decisions around which raters rate which items can alter the final system ranking. this inherent instability can make it difficult to confidently attribute ranking changes between evaluations to changes in the underlying systems. our goal in this work is to offer recommendations on various aspects of designing multi-system nlg human evaluation studies. we propose evaluating results through the lens of stability: the degree to which a specific evaluation methodology produces the same system ranking when repeated. specifically, we seek to answer these questions: to do this, we use machine translation (mt) evaluation as a case study. for mt, multidimensional quality metrics (mqm) is the state-of-the-art human evaluation framework @xcite . in mqm, expert raters identify error spans within translations, which are automatically converted to numeric scores. section 4.1 contains additional details. using mt and mqm as a specific use-case, we make the following contributions: ## terminology a few key terms used throughout this work are defined as follows: ## research questions and findings here we pose the research questions considered in this work. each is expressed in a way that seeks a recommendation on how to implement some facet of an nlg ranking study. we provide these recommendations below, and justify them experimentally in section 7 based on their positive effect on stability, after describing our meta-evaluation methodology in the intervening sections. how should rateable items be grouped? some types of nlg evaluation use a side-by-side methodology, where raters are shown an input and two outputs from different systems, and then assess their relative quality. in contrast, raters only see one output at a time in the mqm framework that we use for our analysis. thus, the closest analogue is what we term a pseudo-side-by-side (psxs) methodology, which means that all system outputs from a given input are assigned to the same rater. based on our results, we strongly recommend use of the psxs methodology. how should workload be distributed? when sending a collection of rating items to a pool of raters, one consideration is whether to limit how many items any single rater can annotate. doing so can slow down progress as fast raters hit their limits, but in exchange for preventing results from being skewed toward any particular rater. we recommend that all raters be given an equal share of the total rating workload. however, we observe that excessive rating noise from differences in both rater behaviors and input documents can affect this recommendation, so we couple it with a recommendation to leverage clear annotation guidelines and rater training to limit noise of this type. section 7.2 further discusses the effect of this noise. how should scores be normalized? one potential technique for controlling noise from differences in rater behavior is to apply normalization to collected ratings. this is especially attractive as an option to address noise introduced in the setup of an evaluation after it has already been run. we weakly recommend applying rater-wise zscore normalization. we make this recommendation weakly because the most pronounced gains of z-score normalization occur in settings that run counter to other recommendations provided here. how many items should be annotated? for nlg it is generally infeasible to evaluate system performance on all possible inputs of interest, and practitioners must select a comparatively small subset to be evaluated. when allocating budgets, it is useful to understand how much reliability is gained by annotating additional items. we do not make a concrete recommendation here, in part because judging the trade-off between reliability and cost is highly setting-dependent. however, we show all of our results over a wide range of item counts to assist practitioners in making this judgment for themselves. how many ratings per item are needed? collecting more ratings for each item can limit noise from differences in rater behavior, but in a practical context with fixed budgets, this generally requires reducing the number of distinct items proportionally, introducing additional noise from item-level differences. the conclusion therefore depends on which source of noise is greater. we recommend that each item be annotated by a single rater to allow for the maximum number of distinct items given a fixed budget. note that evaluation settings other than mqm for mt may have different noise profiles that affect this recommendation (see section 9). ## quantifying replicability in human evaluation research, it is often difficult to define a "ground truth" to compare to, as we are in some sense attempting to determine what to use as the ground truth in the first place. however, an important feature of an evaluation methodology is replicability, which in the context of system ranking studies we examine through the lens of stability. we imprecisely define stability as the tendency of an evaluation methodology to produce the same system rankings when repeated. stability is a critical property of a sound evaluation methodology, because unstable methodologies make it difficult to attribute changes in system rankings to real changes in quality. we propose to estimate the stability of a methodology using a metric calculated on pairs of studies designed with that methodology. we experimented with approximately a dozen different metrics to quantify stability, including well-known correlation measures like average pearson correlation and average kendall's tau. all of them pointed toward the same answers to our research questions, so we present only one metric in the main body of this work: stable ranking probability (srp). given a set of studies with a common methodology, the stable ranking probability of that set is the proportion of ordered study pairs for which all significant system pair differences in the first study are reflected in the rankings of the second study, ignoring significance. that is, if a study finds that system a is significantly better than system b and that system c is significantly better than system d, srp measures the probability that repeating the study with the same methodology will agree that a is better than b and that c is better than d. this can be expressed with the following equations: srp is fairly strict in that it requires directional agreement over all significant system pair differences in a given study, with no partial credit for agreement on only some system pairs foot_2 . we find this to be important because we observe that many system pairs in our datasets have such large quality differences that almost all simulated studies detect a significant difference, and we want to focus on the more difficult case of reliably detecting differences between systems with similar quality. we found that other metrics such as average kendall's tau tend to be dominated by these easilydistinguishable system pairs, yielding a narrow range of values. another benefit of srp is that it is interpretable because it is a probability. to determine statistical significance between mean mqm scores for two systems, we use a random permutation test on segment scores, with the caveat that system labels for segments in the same document are always permuted as a group. we always use 500 permutations, and conclude that a difference is significant if the p-value is less than or equal to α = 0.05. ## simulation analysis our approach to answering the questions listed in section 3 is to simulate a collection of studies sharing a common evaluation methodology by repeatedly sampling a subset of ratings from our datasets. we then evaluate the replicability of that methodology using our srp stability metric. ## results as with sections 3 and 6.1, we organize this section by research question, with experimental results justifying the recommendations made in section 3. however, for answering "how many items should be annotated?" we examine the effect of the number of documents feature in all experiments in this section (the x-axis in our figures). figure 7 examines the effect of normalization on studies that do not use item grouping, with psxsgrouped studies shown for reference. again, z-score normalization provides a significant stability boost. mean normalization provides a slightly smaller boost, with error normalization providing little benefit. in english-chinese, normalization can close the gap below psxs-grouped studies. these latter two experiments show that normalization is especially important when our other recommendations are not followed. we make a few modifications to the experimental settings for the analysis of the ratings per item feature presented in figure 8 . first, because we consider a fixed annotation budget, each data point on a double-rated line corresponds to a set of studies each with half as many documents as the single-rated data point at the same position along the x-axis. second, we modify the document resampling feature to use resampled documents and calculate srp over all study pairs, not just those with the 4914 21 35 70 91 112 147 161 181 total item ratings per system 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 stable ranking probability stable ranking probability, english-german 18 36 72 92 110 146 164 181 total item ratings per system 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 stable ranking probability common feature values: psxs grouping single-rated 50 simulations / document set 250 simulations stable ranking probability, english-chinese fully balanced, unnormalized entropy-balanced (0.38), unnormalized entropy-balanced (0.38), mean-normalized entropy-balanced (0.38), error-normalized entropy-balanced (0.38), z-score-normalized figure 6: evaluation of the normalization feature in balanced vs. highly imbalanced studies. 21 35 70 91 112 147 161 181 total item ratings per system 0.20 0.30 0.40 0.50 0.60 0.70 0.80 stable ranking probability stable ranking probability, english-german 18 36 72 92 110 146 164 181 total item ratings per system 0.20 0.30 0.40 0.50 0.60 0.70 0.80 stable ranking probability common feature values: fully balanced single-rated 50 simulations / document set 250 simulations stable ranking probability, english-chinese unnormalized, psxs grouping unnormalized, no grouping mean-normalized, no grouping error-normalized, no grouping z-score-normalized, no grouping figure 7: evaluation of the normalization feature in psxs-grouped vs. ungrouped studies. 21 35 70 91 112 147 161 181 total item ratings per system 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 stable ranking probability stable ranking probability, english-german 18 36 72 92 110 146 164 181 total item ratings per system 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 stable ranking probability common feature values: fully balanced unnormalized psxs grouping resampled documents 100 simulations stable ranking probability, english-chinese single-rated double-rated same set of documents. this is because, when all documents are shared, a pair of simulated doublerated studies can be expected to share twice as many exact ratings as a pair of single-rated studies. this is especially important given that we only have 3 ratings to draw from for each item. by resampling documents, the doubled chance of sharing a rating on a shared document is exactly offset by the halved chance of sharing the document in the first place. therefore, the expected number of shared ratings between study pairs due to chance is the same in the single-and double-rated case. this results in a fair comparison that essentially measures whether the noise from differences in rater behavior is greater than the noise from differences in relative translation quality for different documents. third, we calculate each data point from 100 simulated studies instead of 250. this is because relaxing the restriction of comparing studies with the same documents increases the total number of comparisons, which is approximately offset by the reduction in the number of simulations. with those considerations covered, we can now conclude from figure 8 that single-rated studies on n documents are consistently more stable than double-rated studies on n 2 documents, indicating that annotation budget is best spent on increasing the number of items rather than ratings per item. ## related work the work of saldías fuentes et al. ( 2022 ) is broadly similar to ours in that it aims to improve mqm. however, they focus on predicting individual system scores rather than rankings, and investigate ways to reduce the number of items annotated by a single rater while minimizing error relative to scores derived from the full test set. @xcite construct a statistical model of an mqmlike annotation process, and run simulations with it to estimate confidence intervals for various sample sizes. in contrast, we characterize the effects of different rater/item configurations and sample sizes, and focus on stability rather than accuracy compared to an assumed ground truth provided by a larger sample. complementary to our work, popović (2021) presents a fine-grained study on how rater agreement varies with error type, and popović and belz (2022) investigate how agreement depends on different methods for normalizing error counts, finding that raw error counts (which we use) provide the most reliable estimates. other work has investigated properties of direct likert-style human scoring for mt with non-expert crowd annotators. building on earlier proposals for score collection and normalization @xcite , @xcite measure the number of ratings per item required to achieve good correlation with ground-truth scores, concluding that at least 15 ratings are required. more recently, @xcite examined the effect on ranking stability of assigning raters to different systems and documents, making recommendations that are consistent with our psxs proposal. @xcite characterize how the number of samples required for adequately-powered statistical tests depends on the magnitude of the difference between system scores, and suggest strategies to reduce this number in practice. finally, @xcite propose a new adequacy-focused rating scheme with better inter-annotator agreement than generic ratings, and a method for calibrating ratings so they are comparable across different languages. beyond mt, human assessment of automaticallygenerated text is an area of increasing research focus. the surveys by @xcite and @xcite provide an excellent guide to work in this area. as for mt evaluation datasets, zouhar et al. (2024) released a dataset of 25,000 mqm ratings in the biomedical domain. the most similar dataset to ours is that of @xcite , consisting of approximately 100,000 mqm ratings on wmt data. in contrast, our dataset contains approximately 140,000 ratings. ## conclusion in this work we have proposed a framework for evaluating reliability of nlg evaluations and used it to provide recommendations to practitioners. our recommendations are supported by analysis of mt data in two language pairs. additionally, we release these mqm annotations to the public to allow for follow-up research by the community.