The Authenticity Gap in Human Evaluation (2022)

Abstract: Human ratings are the gold standard in NLG evaluation. The standard protocol is to collect ratings of generated text, average across annotators, and rank NLG systems by their average scores. However, little consideration has been given as to whether this approach faithfully captures human preferences. Analyzing this standard protocol through the lens of utility theory in economics, we identify the implicit assumptions it makes about annotators. These assumptions are often violated in practice, in which case annotator ratings cease to reflect their preferences. The most egregious violations come from using Likert scales, which provably reverse the direction of the true preference in certain cases. We suggest improvements to the standard protocol to make it more theoretically sound, but even in its improved form, it cannot be used to evaluate open-ended tasks like story generation. For the latter, we propose a new human evaluation protocol called system-level probabilistic assessment (SPA). When human evaluation of stories is done with SPA, we can recover the ordering of GPT-3 models by size, with statistically significant results. However, when human evaluation is done with the standard protocol, less than half of the expected preferences can be recovered (e.g., there is no significant difference between curie and davinci, despite using a highly powered test).

## introduction human ratings are treated as the gold standard in nlg evaluation @xcite . for example, say one wants to claim that that their nlg model x is better than the current state-of-the-art y and z for story generation. the standard protocol is outcome-level absolute assessment (oaa): hire crowdworkers as annotators, collect individual ratings of a sample of stories generated by each model, and then claim that x is the best because its average rating is the highest @xcite . there is inconsistency in how this is im-plemented in the literature: terms such as 'text quality' are often left undefined when instructing annotators @xcite and different papers use different rating scales @xcite . however, such criticism has been restricted to the implementations-little to no consideration has been given as to whether oaa can faithfully capture human preferences to begin with. we start by analyzing the standard evaluation protocol through the lens of utility theory from economics ( §2). we find that oaa can only capture an annotator's preferences under certain assumptions, which are unstated in the nlg literature and often violated in practice. in such cases, annotator ratings become an unfaithful reflection of their preferences ( §3). for example, by framing ratings as utility estimates, we extend a result from @xcite to prove that using the same scale is insufficient for aggregating ratings across annotators-they must agree on the maximum-and minimum-utility outcomes as well. this precludes annotator ratings from being averaged unless they are given both maximally "correct" and "incorrect" references, which are available for some nlg tasks (e.g., machine translation) but not for open-ended ones (e.g, story generation), since the space of high-quality outputs is too diverse. we provide concrete suggestions on how to improve the standard protocol to a point where it can faithfully capture human preferences in some nlg tasks and settings ( §4); however, for open-ended generation, an entirely new evaluation protocol is needed. though uncommon nowadays, a historic alternative to oaa was outcome-level relative assessment (ora): create random pairs containing an output from x and y , ask annotators to pick the one they prefer in each, infer a score for x and y that explains the results-based on a comparison model such as @xcite -and argue that x is better because its estimated score is higher @xcite . however, this also makes untenable assumptions about annotators; for example, even if x's outputs are preferred to y 's over 50% of the time, x may be less preferred to y if it has a tendency to fail catastrophically. we observe that the main limitation of both oaa and ora is their reliance on outcome-level judgments. to this end, we propose system-level probabilistic assessment (spa), which can be used for both open-and close-ended nlg tasks ( §5). spa's key insight is that while an annotator cannot categorically determine whether they prefer system x or y -because the output space is too large for them to observe-they can estimate the probability with which they prefer x or y based on some fixed level of exposure to both. spa obviates assumptions about annotator preferences by delegating the responsibility of aggregating preferences over individual outputs into a preference over the underlying systems to the annotator themself, acknowledging that there is no canonical way to do so. because we are working with probabilities, aggregating across annotators is also straightforward. we then ask annotators to use both the standard protocol (with a 5-point likert scale) and spa to express their preferences about the different gpt-3 variants w.r.t. their story-generation ability. given that larger gpt-3 variants generate more coherent, grammatical, and creative text, annotators foot_0 should prefer each gpt-3 variant to the next smallest one, giving us 3 ground-truth preferences @xcite . past work also suggests that annotators can distinguish human-written text from ada (the smallest variant) but not from davinci (the largest) @xcite , which gives us two additional ground-truth preferences, for a total of 5. when human evaluation is mediated by spa, we can recover all 5 out of 5 expected preferences, with statistically significant results. however, when human evaluation is done with the standard protocol, we can only recover 2 of the 5 preferences, despite all tests having statistical power ≈ 1 for α = 0.001. the standard protocol also yields the surprisingand likely incorrect-result that human-written text is significantly less preferred to davinci. the failures of the standard protocol suggest that its theoretical limitations have practical consequences, and the flexibility of spa makes it the better option in most intrinsic evaluation settings. ## reframing human evaluation to understand what causes human preferences to be misrepresented, we will analyze nlg evaluation through the lens of economic theory on preference modeling. in doing so, we find that comparing nlg systems is an instance of a common problem in utility theory. to begin, let x, y denote the nlg systems to be compared and annotator a i be the agent making the comparisons. as we are drawing from the economics literature, we will primarily use economic terms such as lottery and utility in our framing, which we will define as we go along. ## causes of misrepresentation by framing human evaluation in terms of utility theory, we found that the standard protocol in nlg evaluation serves to estimate the cardinal utility of a system via outcome-level absolute assessment ( §2.3). we then listed the assumptions that agent preferences need to satisfy in order to make this estimation valid ( §2.4). in this section, we discuss how these assumptions are often violated in nlg evaluation, and how this begets misrepresentation of an annotator's true preferences. we limit our criticism to oaa in this section, since ora has already been criticized in prior work @xcite and has, over the past several years, become far less common than oaa (van der @xcite . we begin by noting that rating generated text is done one of two ways @xcite : our first critiques apply only to likert scales, but our last two apply to the standard protocol at-large. ## improving the standard protocol by making some minor changes, the oaa-based standard evaluation protocol can be improved to a point where it can adequately capture human preferences in some nlg tasks and settings: the wmt competition for machine translationwhich has experimented with many evaluation schemes-has had, since 2017, a protocol that follows many of these suggestions @xcite . it uses continuous scales, provides maximum-utility references, and hires translators, meaning lottery independence is safe to assume. still, this improved protocol cannot be applied to open-ended tasks where there is no singular notion of correctness, tasks where maximal-utility outcomes can be diverse (e.g., story generation), or when lottery independence is likely to be violated in the real-world (e.g., offensive chatbots). such tasks and settings demand an entirely new evaluation protocol ( §5). ## system-level probabilistic assessment the limitations of both ora and oaa stem from trying to aggregate preferences over outcomes into a preference over systems, despite there being no canonical way to do so. for example, one annotator may prefer x to y only if the former wins head-tohead comparisons of outputs over 50% of the time, but another annotator may choose by comparing the worst-case output from each system. therefore we propose directly asking annotators to estimate the probability p [x ≻ i y ] that a preference holds across two systems, a protocol we call system-level probabilistic assessment (spa). ## related work soliciting humans to directly evaluate the quality of generated text is known as intrinsic evaluation. the text can be judged for its overall quality or along a specific dimension such as coherence, though these terms are not consistently defined @xcite . this is most often done in the nlg literature by having annotators assign a likert rating from 1-to-k, where k is usually 5 @xcite . given the ubiquity of this standard protocol, little justification is given when it is used and implicit assumptions, such as equal intervals for likert scales, are entirely omitted @xcite . the earliest shared tasks in nlg, such as the tuna @xcite and grec @xcite challenges for expression generation, used a continuous scale for scoring, explicitly noting that annotators may not perceive the intervals on a likert scale to be equal. in contrast, early modeling work-such as the storybook system for narrative prose generation (callaway and lester, 2002)-used discrete ratings. this difference in evaluation protocol between shared challenges and individual modeling papers continued over the years. for example, the e2e nlg challenge @xcite used continuous scores based on relative magnitude estimation @xcite . however, these challenges have not served as a bulwark against the popularity of likert-based oaa. even recent attempts to standardize human evaluation in nlgusing evaluation platforms-collect likert ratings @xcite . compared to oaa, outcome-level relative assessment (ora) is far less common nowadays, in large part because the cost of pairwise output comparisons grows combinatorially as you evaluate more systems @xcite . recall that given binary outcome-level preferences (e.g., x i ≻ y i ) as labels, ora uses a preference model such as bradley-terry to estimate the scores of the systems, analogous to how elo scores are calculated for chess players @xcite . in explicitly stating its assumptions about annotator preferences using a preference model, ora was easier to criticize than oaa, which contributed to the former's decline @xcite . the one area in which comparison-based evaluation still prevails is when conducting a turing test-seeing whether annotators do better than chance when guessing whether a text is human-or machine-generated @xcite @xcite . this is acceptable, since what is being measured is not annotator preference but rather discriminability. over the years, machine translation (mt) has had spirited debate about evaluation. @xcite found that compared to ranking outputs, annotators took more time and agreed less when providing likert scores. citing this, @xcite use the trueskill algorithm @xcite to estimate scores for nlg systems based on pairwise preferences of their output. this approach, called relative ranking (rr) was used in the wmt competition until 2016, when direct assessment (da) on a 0-to-100 continuous scale were trialled and found to produce systems rankings that strongly correlated with rr @xcite . da also had the advantage of providing an absolute measure of quality, so it was adopted as the standard for wmt in 2017 and used thereafter @xcite . ## conclusion we analyzed the standard evaluation protocol in nlg through the lens of utility theory, finding that it makes untenable assumptions about annotator preferences. when these assumptions are violated, annotator ratings become an unfaithful reflection of their preferences, both in theory and in practice. we proposed a new evaluation protocol called spa that makes minimal assumptions-not only is it more theoretically sound than the standard protocol, but it performs better in practice as well, consistently recovering the expected preference with statistically significant results. an important direction of future work will be re-evaluating conclusions in the nlg literature with spa and seeing which ones stand up to scrutiny.