What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021)

Abstract: Previous work has shown that human evaluations in NLP are notoriously under-powered. Here, we argue that there are two common factors which make this problem even worse: NLP studies usually (a) treat ordinal data as interval data and (b) operate under high variance settings while the differences they are hoping to detect are often subtle. We demonstrate through simulation that ordinal mixed effects models are better able to detect small differences between models, especially in high variance settings common in evaluations of generated texts. We release tools for researchers to conduct their own power analysis and test their assumptions. We also make recommendations for improving statistical power.

## introduction human evaluation remains the gold standard for many natural language generation tasks, including machine translation, data-to-text, summarisation, and dialogue & interactive systems. one common way to elicit text quality ratings from study participants is to use a rating scale, e.g. a likert scale which measures agreement with a statement, or other visual or verbal analogue scales, as in figure 1a . unfortunately, typically chosen statistical analyses of these scores often rely on the flawed assumption that the rating scales are interval, i.e. that the distance between any two adjacent points on the scale is the same across the full range of values, so that, for example, the difference between 'very disfluent' & 'disfluent' is the same as the distance between 'slightly disfluent' & 'slightly fluent' on a 6-point scale (see figure 1 ). the distributions in figure 1 illustrate the different underlying assumptions of interval and ordinal models of rating scale data: the rating scale in (1a) is used to collect human judgements of text quality * work completed while at heriot-watt university. (e.g. fluency), which results in a distribution of ordinal data as in (1b). in (1c) we follow the interval assumption, that each point on the six-point rating scale corresponds directly to a real-valued integer and that we can model the relative probability of any pair of ratings based on a gaussian probability density function. in contrast, (1d) assumes that there is a latent variable for text quality and that the ordinal scores from our surveys correspond to different ranges of values on this latent scale. 1 different model assumptions influence the choice of statistical significance test: when the data distri-bution is known, it is often possible to choose a parametric test to achieve greater statistical power at lower computational cost @xcite . while the debate about whether and when ordinal scales can be treated as interval has been fought for several decades @xcite @xcite @xcite , we argue that ordinal data needs to be analysed as ordinal in nlp: in this paper we demonstrate that this misinterpretation of rating scales does in fact limit the statistical soundness of our studies by simulating the effects. previous research has shown that human evaluations are notoriously under-powered @xcite . we show that these effects will be exaggerated if we treat ordinal data as interval. we compare the linear mixed effects models proposed by @xcite , which treats rating scale data as interval, and compare it to a corrected version, which uses ordered probit models and appropriately treats the data as ordinal. we show that ordinal models are more likely to detect a real effect, especially when the effect size is small, the variance is high, or the sample size is small, all of which are common in human evaluations. we release all of our code so that other researchers can adjust the assumptions of our models to match the reality of their evaluation settings and easily estimate appropriate sample sizes using the same simulation methods: https://www.github.com/ dmhowcroft/ordinal-models 2 current reporting practices significance testing provides an assessment of how extreme the observed values are according to a random noise model. for example, if an observed difference in performance between two systems is not distinguishable from noise centered at zero, then we would not want to rank one system above the other, with implications for leaderboards and the replicability of results (van der @xcite @xcite . 2 however, not many studies include significance tests: regardless of whether using automated metrics or human evaluations, only about a third of studies reported significance tests according to recent surveys @xcite . and even when researchers do include significance tests, they often apply the tests incorrectly @xcite , with @xcite reporting that the majority of recent papers incorrectly interpret rating and likert scales as interval data (up to 84% for likert scales; figure 1 illustrates why this is a problem). in the common lme4 @xcite notation, a model comparing ratings for several systems with random effects for participants and items is: rating ~system + (system|participant) + (system|item) this specifies a model for ratings with a fixed effect of system and random effects (system|participant) and (system|item). as a 'maximal model' @xcite , it includes random intercepts to represent the general bias of individual participants and items (e.g. some participants give higher or lower ratings on average) and random slopes to represent the interactions between random and fixed effects (e.g. some users may systematically prefer system a or system b). these random effects are designed to control for the fact that our participants and items are samples from larger populations: we are not interested in the behaviors of these individuals, but rather in more general assessments of text quality that should generalise to a larger population instead of using lme models, we argue that researchers should use ordinal mixed-effects models to analyse ordinal data. unlike lme models, ordinal regression models do not assume that the data is metric. here we focus on ordered probit models as implemented in ordinal (christensen, 2019) for ease of explication, but researchers are free to use alternative linking functions (e.g. logit) or tools as needed. 4 the key difference from the lme model is that we no longer assume that our observed ratings y are on a continuous scale we can model directly. instead, we assume that there is an underlying latent variable y l which is continuous. we represent this latent variable with a standard gaussian distribution with a mean of 0 and a standard deviation of 1. the link between the observed variables y with k possible categories and y l is then based on fitting a series of k -1 thresholds τ such that: when comparing systems, then, the goal of the model is to fit these thresholds as well as a fixed effect representing the differences between the systems while controlling for noise. these differences can be thought of as shifting the thresholds along the latent variable axis or, equivalently, as shifting the mean of the underlying latent variable. ## simulation experiments our experiments take the form of a power simulation, i.e. an analysis of the statistical power of a given test in typical or expected experimental conditions. in a power simulation we generate a set of data with known parameters (e.g. a known 'effect size' difference between conditions) and measure how often a statistical test correctly identifies that effect in the simulated data. in order to limit the complexity of a power simulation, the researcher estimates 'typical' values of as many model parameters as possible and then systematically explores possible values for the other parameters. in our case, we begin by fitting ordinal models on several nlg evaluation datasets. the resulting models then allow us to simulate nlg datasets with different numbers of raters, different amounts of variance, and different effect sizes in order to understand how many participants are needed to detect effects of different sizes. ## discussion in contrast to the (common) assumption followed by @xcite that ordinal data can be analysed as interval data, we show that treating ordinal data as interval makes human ratings even more under-powered. this is a problem because, in practice, nlp evaluations often aim to detect small differences (i.e. effect sizes) in high variance settings while operating under a limited budget or with limited access to human raters. since our proposed framework is independent from the concrete instantiation of the scale and generalises well, our hope is that other researchers can adapt our code to gain a better understanding of what kind of scale and statistical model to use for their next experiment. we also recommend setting simulation parameters based on e.g. their own past experiments if similar. one open question is how to best choose the best scale and model. in general, each researcher needs to choose appropriate tools based on their knowledge of the data. on the one hand, they may prefer to start with 5+ points on their scale, use ordinal regression to measure variance, and only later conclude that the differences seem large enough for their task & survey instruments that they can switch to simpler scales and/or models. on the other hand, they may reason that 'yes-no questions are easy/cheap to ask, so let's see if those are in-formative enough for our needs'. if the differences between systems are large enough, they may even be able to use an even simpler model than an lme model (for example, a simple chi-squared test on 'the proportion of positive responses'). however, if the differences are not in fact large enough for such a simple scale & analysis to capture, then they have wasted time and resources to collect data they cannot use. both approaches are reasonable, but researchers should be aware of the power problems highlighted in our paper when they start planning and choosing an approach. ## conclusion we see three core ways to improve the power of human evaluations: first, reduce noise in human ratings. the renem dataset's clear definitions, guidance, and training reduced noise in the resulting human ratings, which reduces between-participants variance and increases the ability of a statistical model to distinguish between similar systems. similar studies have been conducted for machine translation @xcite . in addition to providing clear instructions, we can also design experiments to include more items and more participants, using power analyses like the ones presented in this paper to estimate how large a sample we need before collecting any data. most importantly, however, we recommend researchers use ordinal models to analyse ordinal data to have the greatest statistical power when testing hypotheses. this is especially important for setups with high variance and small data samples, as often the case for human evaluations in nlp. ratings/text and ratings/participant report how many ratings were associated with each text or participant in the most frequent case (*except in two cases where the median is more representative of the distribution). for the nem 1-2 and renem datasets, the number of unique raters is not known.