Follow the leader(board) with confidence: Estimating p-values from a single test set with item and response variance (2023)

Abstract: Among the problems with leaderboard culture in NLP has been the widespread lack of confidence estimation in reported results. In this work, we present a framework and simulator for estimating p-values for comparisons between the results of two systems, in order to understand the confidence that one is actually better (i.e. ranked higher) than the other. What has made this difficult in the past is that each system must itself be evaluated by comparison to a gold standard. We define a null hypothesis that each system’s metric scores are drawn from the same distribution, using variance found naturally (though rarely reported) in test set items and individual labels on an item (responses) to produce the metric distributions. We create a test set that evenly mixes the responses of the two systems under the assumption the null hypothesis is true. Exploring how to best estimate the true p-value from a single test set under different metrics, tests, and sampling methods, we find that the presence of response variance (from multiple raters or multiple model versions) has a profound impact on p-value estimates for model comparison, and that choice of metric and sampling method is critical to providing statistical guarantees on model comparisons.

## introduction ai and nlp evaluation is facing a scientific reproducibility crisis that, despite increasing awareness, continues to worsen @xcite . published results may often show only epsilon improvements to state-of-the-art results, with no effort to estimate whether or not the results are statistically significant. the reasons for this crisis are complex, and it is easy to implicate the culture created by leaderboards (e.g., @xcite ). our work is motivated by the need to provide statistical testing alongside nlp results in order to reliably demonstrate model improvement, * work completed while interning at google. as opposed to solely depending on leaderboards. our work naturally ensues from studies of rater response disagreement (see e.g., (r artstein; @xcite @xcite , among others). further, the issue of insufficient statistical analysis in nlp work is well-documented, with many acl papers not reporting statistical significance @xcite . considering the reliance on system comparison for benchmarking and leaderboards, statistical guarantees that consider the performance of both systems are critical, yet understudied. statistical tests for paired data (e.g. @xcite ) are not appropriate for this setting because of their reliance on strong assumptions about the data @xcite ; even extensions of mc-nemar's test such as the cochran-mantel-haenszel test @xcite only apply when the metric can be applied independently to each item or responder (human or machine), and is then aggregated. therefore, these metrics are not applicable for this use case, in large part due to three potential challenges: (1) three sets of data are involved in this comparison, (2) there is variance in all three of those sets, and (3) many different metrics are used in nlp evaluation. moreover, variance can come at the item or response level, due to stochastic inference or training, changes in training data such as cross-validation, or annotator disagreement in gold labels. we investigate the use of null hypothesis significance tests (nhst) to add a dimension of confidence to nlp evaluations. the purpose of nhst is to determine whether differences between multiple sets of observations are significant, after accounting for sampling variance in the observations. when comparing two nlp systems, each is first compared to a gold standard, resulting in some metric score (e.g. bleu @xcite ), and then those metric scores for the two models are compared to each other. while all p-values are esti-mates, there are many ways to sample and measure the results from a single test set, each producing a different p-value estimate. we explore how to determine which method (of sampling, aggregating, and measuring responses) produces the most accurate p-value estimate from a single test set in comparison to the true/ground truth p-value. in this work, we present a framework for effective p-value estimation when comparing two systems against gold judgments, with the aim of identifying with statistical rigor if one system outperforms the other. our findings indicate that the amount of response variance has an impact on pvalue estimates, item-wise mean absolute error is consistently a reliable metric, and-while most metrics and sampling methods perform well when machine output is dissimilar-metric choice and sampling method is especially critical when the performance of the two machines is similar. our primary contributions include: ## related work our approach to generating a statistical guarantee associated with the comparison of two nlp systems (a p-value) is rooted in the statistical inference method of nhst. our formulation also incorporates variance in rater and system responses. ## results of simulation study we examine which of the metrics and sampling methods on a single test set best estimate the true p-value, by calculating the error between the estimated p-value and true p-value across five response distribution perturbations (ε @xmath0,ε @xmath1.0,0.05,0.1,0.3,0.7} , q.v. alg. 3). we expect that as the amount of perturbation applied to system b increases, it should be clearer that the data is drawn from two separate distributions. a metric that is more sensitive to the effect of perturbation/distance should have a smaller dif-ference between the estimated p-value and true p-value when the perturbation is increased. consequently, the metric should have a harder time producing the estimated p-value when the systems are closer together-meaning a larger difference between the estimated p-value and true p-value when the perturbation is less. multiple metrics (cosine similarity, wins mae , and spearman rho) show lower minimum differences at each increasing interval of perturbation. this suggests that these metrics, when operating under unknown conditions / distances between system a and system b, may behave most predictably. wins mae has the lowest difference in true and estimated p-value for ε > 0, making this the preferred metric. the least consistent metric is aggregated emd vectorized, which increased, decreased, and increased again in minimum difference between estimated and true p-values at increasing levels of perturbation ( it is important to note that @xmath3.05 is a critical value when considering statistical guarantees, so differences in estimated and true p-values close to or exceeding 0.05 are not acceptable; if the difference in estimated and true p-value is close to 0.05, there is sufficient room for error for it to seem like there is evidence of model difference, when in fact there is not. ## application to real-world data to apply our method on actual data, we need the item and response data for the ground truth and the two machines (g ref , a ref , and b ref , respectively). for our example, we chose @xcite , a dataset of 107,620 social media comments that are labeled by five annotators each on the toxicity of each comment, using a 5-level likert scale from 0-4. we randomly sampled 1000 items from it for g ref , normalizing the annotations into [0,1], yielding possible responses {0,0.2,0.4,0.6,0.8}. to use, and then choose the hyperparameters that minimizes the mean absolute error between the observed data distributions and those predicted by the models. this process is described in appendix b. we also assume that, after sampling from a normal distribution the results in the range [0,0.2) are converted to 0.2, those in the range [0.2,0.4) are converted to 0.4 etc. to simulate the discrete nature of likert responses. with these parameters set, we can run the framework described in §3.2, with the toxicity dataset sample as our reference test set and simulated system responses to choose the best metric and sampling method to use on g ref . we expect to see results similar to those from the pure simulation study, although the fact that responses are now discrete, rather than continuous, there will be sharper differences in performance between different values of ε b . the results on the toxicity dataset ( however, it should be noted that for ε @xmath4,0.05,0.1} the maximum amount of perturbation is relatively small compared to the 0.2 interval between successive elements in the response do-main. there is not much observable difference in the performance between a and b until ε @xmath5.3. at this point, many of the metrics do well. another interesting pattern in the metric results is that spearman rho is among the better performers in most cases, particularly for ε @xmath6.3,0.7}. ## discussion these experiments suggest answers to our four research questions: rq1. can response-level variance be used to estimate p-values? yes. in