With Little Power Comes Great Responsibility (2020)

Abstract: Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community. Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings. By meta-analyzing a set of existing NLP papers and datasets, we characterize typical power for a variety of settings and conclude that underpowered experiments are common in the NLP literature. In particular, for several tasks in the popular GLUE benchmark, small test sets mean that most attempted comparisons to state of the art models will not be adequately powered. Similarly, based on reasonable assumptions, we find that the most typical experimental design for human rating studies will be underpowered to detect small model differences, of the sort that are frequently studied. For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point. To improve the situation going forward, we give an overview of best practices for power analysis in NLP and release a series of notebooks to assist with future power analyses.

## introduction despite its importance to empirical evaluation, relatively little attention has been paid to statistical power in nlp. in particular, if it is the case that typical experiments in nlp are underpowered, not only would we expect many meaningful improvements to go undetected, we would also expect many apparently significant differences to be exaggerated @xcite . in this paper, we build on past work calling for greater rigor 1 https://github.com/dallascard/nlp-power-analysis figure 1 : cartoon example of statistical power in comparing two models: 65% of all people in the population always prefer system b (left). a comparison using a sample of 100 people would be well-powered (middle): over 80% of such samples will show a significant difference (plotted in red) from the null hypothesis that the models are equally good (dashed line). in samples of 25 people (right), far fewer tests will be significant (power ⇡ 30%). note that the observed mean of significant findings (dotted line) slightly overestimates the true proportion that prefer system b when @xmath0. in evaluation @xcite , including the need for careful hypothesis testing @xcite @xcite , and show why and how power matters to nlp, addressing challenges unique to this domain. roughly speaking, power is the probability that a statistical test will successfully detect a true effect. as an illustrative example, imagine comparing two dialog systems (see figure 1 ). we want to know if people tend to prefer one system over the other. to test this, we will need multiple people to evaluate the systems. but how many? once we have collected data, a statistical test will tell us if we can reject the null hypothesis the systems are equally good. assuming the systems are not identical, statistical power is the probability that the experiment will return a significant result (or equivalently, it is one minus the probability of failing to detect the difference as significant). although we don't know the magnitude of this difference, power analysis helps to estimate how much power an experiment will have under various assumptions. power depends on multiple factors, including the statistical test used, the significance threshold, true effect size, variance, and sample size. all else being equal, experiments with larger samples will have greater power than smaller samples, as shown in figure 1 . similarly, larger effects and those with less variance are easier to detect, and therefore require fewer samples for equivalent power. importantly, note that if we do find a significant difference, this does not imply that the experiment had high power. 2 proceeding with a test that is underpowered (i.e., too few subjects or items; often taken to mean less than 80% power; @xcite means that one is less likely to be able to draw any useful statistical conclusion from the experiment, and has contributed, in part, to the replication crisis in other fields @xcite @xcite . routinely running experiments with low statistical power undermines the scientific enterprise. not only will true effects go undetected; when significant effects are found, they are likely to be noisier and have lower positive predictive value @xcite . moreover, significant findings from underpowered experiments are more likely to exaggerate or reverse the true effect -so-called type-m (magnitude) and type-s (sign) errors, respectively @xcite . this problem can lead to systematic distortions in the literature if only significant findings are published, especially if these results are based on underpowered experiments @xcite . the effect of type-m error can be seen in figure 1 ; significant differences are less likely to be found in smaller samples (right), but among those tests that are significant, the observed difference will tend to exaggerate the true difference (left) by more than a larger sample (middle). for further discussion of type-m and type-s errors, please refer to appendix b. here, we investigate how these issues affect nlp. although retrospective analysis of power involves challenges, we present evidence that underpowered experiments are widespread in nlp research. among human evaluations, we find most experimental designs involve too few items and/or raters 2 using the observed outcome from a single experiment to compute power falls into the trap of post-hoc power analysis and is not recommended. for additional background on statistical power, power analysis, null-hypothesis significance testing, and post-hoc analysis, please refer to appendix a. to detect small effects ( §5). for comparing models in terms of accuracy, we find that some widely used benchmark datasets, including mrpc and sst-2, are now too small to be able to properly measure future progress against top performing models ( §3). we also introduce a novel approach to power analysis for machine translation and characterize power in experiments testing for differences in bleu ( §4). finally, a survey of recent papers reveals a general lack of statistical evaluation and a dearth of detailed reporting ( §5.1). to improve future practice, we suggest broader adoption of power analyses prior to evaluation, provide guidance on running power analyses in nlp, and release a series of notebooks for this purpose. ## power analysis for nlp because most nlp tasks do not take the form of standard experiments in other sciences @xcite , it is nontrivial to run power analyses for many tasks of interest. while we cannot cover every scenario, we present here a generalizable, simulation-based approach to power analysis, along with three sample applications, which can be extended as necessary. such an approach is modular, reusable, and transparent, and encourages planning of analyses in advance of data collection. every power analysis requires assumptions, and there is not likely to be a single correct approach. rather, the point is to make one's assumptions explicit, and include enough detail so as to account for whatever is likely to be observed. by using reasonable assumptions, one can help to ensure that one's experiment is sufficiently well-powered, in the case of nlp, this means that one recruits enough subjects, collects enough ratings, or uses a large enough test set. the general procedure we suggest for power analysis is described in detail in figure 2 . at a high level, the idea is to estimate power by running simulations. recall that power is the probability of detecting a true effect, conditional on the experimental setting (effect size, variance, etc.) and significance threshold. thus, if one can translate these assumptions into a process for generating simulated data, we can estimate power by generating many simulated datasets using assumed or estimated parameter values, running each sample through a significance test, and reporting the proportion that are found to be significant. define a generative process g(n, e ⇤ , h) parameterized by number of items, n, hypothesized effect e ⇤ for the statistic of interest e, and other relevant parameters h (e.g., variance). also choose a statistical test t (d), which returns a p-value p when performed on data d sampled from g(n, e ⇤ , h). finally, choose the size of the dataset to be sampled, n, significance threshold, ↵, and number of repetitions, r. ## comparing models on accuracy it is common in nlp research to look for models which improve over state of the art (sota) on various benchmarks. however, an important but rarely asked question is, can these benchmarks support the kinds of comparisons we want to make? many have emphasized the need for proper significance testing to avoid spurious findings, but if an experiment's test set is small, the minimum detectable effect (mde) size may be large: only large improvements will yield sufficiently powered comparisons (i.e., 80% power). if an experiment is badly underpowered, it cannot provide useful evidence that one model achieves slightly better performance than another for the underlying data distribution. reliance on such evidence risks leading to over-confidence about the relative ranking of various models. as we show in §3.3, there is legitimate reason to be concerned about this in the case of certain widely used benchmarks. ## machine translation to show how our approach to power analysis can be applied to a more difficult setting, we consider automated evaluation of machine translation using bleu scores @xcite . as with accuracy, we would like to know what scale of improvements can be detected with reasonable power on typical test sets. this setting is more complicated because (1) bleu is a corpus-level metric, rather than being averaged across instances, and (2) typical models are trained on vast amounts of parallel data, with little data available that has not been used in training, making it difficult to estimate variation in performance. significance testing for bleu: to test for a significant difference between two mt models we use the randomization test, as recommended in @xcite : given the paired output translations from both models, swap the outputs for a random subset of test examples and compute the resulting difference in bleu. repeating this thousands of times gives us a null distribution, which can be used to test the observed difference between models. generative process for simulations: if large amounts of untouched evaluation data were available, we could approach power analysis by simply evaluating bleu score on many random subsets of n sentences, and computing the mean and variance of each system. unfortunately, because mt depends on parallel text (most of which is used in training), evaluation data tends to be scarce. in-stead, we introduce a generative process that can produce the necessary inputs for power analysis. for intuition, note that if we swap the i th pair of model outputs (as is done in the randomization test), leaving rest as they are, we change the difference in bleu between models by a specific amount, i , which we call the effect of making that swap. while these individual effects are not independent of each other due to the corpus-level nature of the metric, in practice, the sum of individual effects closely approximates the net effect of swapping entire subsets (see figure 15 in appendix g). based on analyzing several models and datasets, we find the typical distribution of these individual effects can be approximated using a mixture of a delta distribution at zero, and a laplace distribution (see appendix g for details). concretely, if we assume b is the expected difference in bleu between two models on a dataset of n examples, and p 0 is the expected proportion of examples for which @xmath1, we can simulate a dataset { i } n @xmath2: with probability p 0 , @xmath3. with probability and b 0 is a user-specified parameter that controls the variance, independent of the sample size. by construction, e[ p n @xmath4. foot_4 given this generative process, we can then estimate power using the algorithm in figure 2 . on each iteration, draw a simulated dataset from the generative process, compute the observed difference between models as ˆ @xmath5, and test if this is significantly different from zero using a modified randomization test, in which we assume that the net effect of swapping a subset of instances is simply the sum of the i 's in the subset. (please see online materials for an interactive example). empirical estimates: in order to estimate reasonable values for the required parameters, we use several pretrained models from the fairseq library @xcite for the wmt english-german translation task. we evaluate these models on the shared task test sets from 2016-2019 and compute bleu scores using sacrebleu @xcite . fitting a delta-laplace mixture to the effects of swapping individual output pairs, we estimate values for p0 and b0 , reported in using these estimates, we can now characterize how much power test sets of different test set sizes (n) would have for a range of possible differences in bleu ( b ). figure 4 shows this for p 0 and b 0 set to the average of the observed values. 10 based on this estimate, we conclude that for typical mt test sets of around 2,000 examples, an improvement of 1 bleu point can likely be detected with approximately 75% power. as shown in figure 4 this power level increases dramatically with sample size and effect size. this analysis has served, in part, to show how a simulation-based approach to power analysis can be adapted to virtually any task. additional work is required to test how well these specific parameter estimates will generalize, but the same process can easily be adapted to new language pairs. more generally, there would be great value in the mt community curating larger held-out test sets, both to validate this analysis, and for better powered future comparison. ## likert-scale human evaluations tasks such as natural language generation are difficult to evaluate using automated methods; as such, human evaluations are central to nlp. past work has reported great variation in how human evaluations are done @xcite . therefore, we begin with a meta-analysis of a subset of human evaluation experiments from emnlp 2019, which we then use as the basis for claims about the power of human evaluations in nlp more generally. ## conclusion recent progress in nlp has been extraordinarily rapid, sometimes at the cost of experimental rigor. in this paper, we have presented evidence that underpowered experiments are widespread in nlp. for comparisons based on small samples, there is little reason to think that such an evaluation could reliably provide evidence of a significant improvement, and good reason to believe that improvements found to be significant will exaggerate or reverse the true effect. going forward, a combination of larger test sets, simple power analyses, and wider sharing of code, data, and experimental details will help to build the foundation for a higher standard of experimental methodology in nlp.