All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021)

Abstract: Human evaluations are typically considered the gold standard in natural language generation, but as models’ fluency improves, how well can evaluators detect and judge machine-generated text? We run a study assessing non-experts’ ability to distinguish between human- and machine-authored text (GPT2 and GPT3) in three domains (stories, news articles, and recipes). We find that, without training, evaluators distinguished between GPT3- and human-authored text at random chance level. We explore three approaches for quickly training evaluators to better identify GPT3-authored text (detailed instructions, annotated examples, and paired examples) and find that while evaluators’ accuracy improved up to 55%, it did not significantly improve across the three domains. Given the inconsistent results across text domains and the often contradictory reasons evaluators gave for their judgments, we examine the role untrained human evaluations play in NLG evaluation and provide recommendations to NLG researchers for improving human evaluations of text generated from state-of-the-art models.

## introduction human-quality text has long been a holy grail for the output of natural language generation (nlg) systems, serving as an upper bound on their performance. since we lack a good way of encoding many aspects of what constitutes human-quality output in an automated method, we often must rely on human evaluation for our models. though evaluations with end-users in an applied setting are encouraged @xcite , in practice, most human evaluations instead ask people to rate generated text's intrinsic quality (van der @xcite . sometimes the generated text is explicitly compared to human-authored text (e.g., @xcite @xcite figure 1 : excerpts from human evaluators' explanations for why they believe a gpt3-generated story (also excerpted) was written by a human (left) or a machine (right). the evaluators point to a wide range of text attributes to make their decisions, sometimes using the same aspect of the text to come to opposite conclusions. @xcite , but even when no human-authored text is evaluated, evaluators implicitly compare the generated text to their knowledge of language and norms within specific domains. evaluators are often asked to assess a text holistically, e.g., based on its overall quality, naturalness, or humanlikeness (van der @xcite , where the exact evaluation criteria is left to the discretion of the evaluator. though other evaluations are broken down along specific dimensions of text quality (e.g., grammaticality, coherence, etc.), @xcite and @xcite found that these dimensions are often correlated and may be conflated in some evaluation settings. this is con-cerning because, as nlg models improve, evaluators are asked to read longer passages of text conditioned on large amounts of context. in these cases, fluency-related aspects of quality (i.e., the ones that don't require careful reading of the context and meaning of the passage) are the easiest to assess, particularly in small-batch evaluations with non-expert evaluators where speed is incentivized. this poses a challenge when collecting human evaluations for state-of-the-art language models, as errors are often content-based (e.g., factual inaccuracies or inconsistencies with the context) rather than fluency-based @xcite , so a superficial read may not be sufficient to catch model errors. for accurate assessments of generated text, we need human evaluations that are designed to encourage a sufficiently careful reading of the text to examine these subtler aspects of text quality. we asked non-expert evaluators to assess the humanlikeness (operationalized as how believably human an evaluator finds a text) of text generated by current nlg models (gpt2 and gpt3) to test what current human evaluation practices can reveal about the models' quality ( §2). we found that evaluators were unable to distinguish between gpt3and human-authored text across story, news, and recipe domains. however, when we categorized the aspects of text the evaluators used to make their judgments, we found they primarily focused on the grammar, spelling, and style of the text. the evaluators' responses also indicated that they underestimated the quality of text current models are capable of generating (as seen in figure 1 ). to our knowledge, this paper is the first to evaluate human evaluations of gpt3-generated text across multiple domains. we then looked at three different evaluator training methods-providing detailed instructions, annotated examples, and human-machine paired examples-to test whether we could improve evaluators' accuracy ( §3). while we found including examples in the task increased the set of texts evaluators thought could be machine-generated and increased their focus on textual content, no training method significantly increased evaluators' performance consistently across domains. based on our results (discussed in §4), we recommend moving away from small-batch evaluations with little training when collecting human evaluations of nlg models ( §5). we also encourage practitioners to consider alternative evaluation frame-works that capture the usefulness of generated text in downstream settings rather than its humanlikeness. in our first study, we ask how well untrained evaluators can distinguish between human-and machinegenerated text. this task format, inspired by the @xcite test, is used to compare the quality of machine-generated text to human-authored text and, as models' fluency improves, to analyze nlg models' ability to "fool" readers @xcite . by asking evaluators to assess the humanlikeness of the text with only minimal instructions (see figure 2 ), we observe how well untrained evaluators can detect state-of-the-art machine-generated text and which attributes evaluators focus on and think are important for detecting machine-generated text. ## discussion overall, none of our three training methods significantly improved evaluators' ability to detect machine-generated text reliably across text domains while still maintaining the small-batch nature of amazon mechanical turk. this speaks to the improving quality of nlg models, but we also found that untrained evaluators mainly focused on the format of the text, deciding if it was human or machine-generated based on whether ## related work a subfield of nlg analyzes the role of human evaluations, including discussions of the tradeoffs of human and automatic evaluations @xcite . there are critiques and recommendations for different aspects of human evaluations, like the evaluation design @xcite , question framing @xcite , and evaluation measures like agreement @xcite , as well as analyses of past nlg papers' human evaluations (van der @xcite . additionally, crowdsourcing literature has work on effectively using platforms like amazon mechanical turk (e.g., @xcite @xcite . in this work, we focus on the role evaluator training can play for producing better accuracy at distinguishing human-and machinegenerated text, though other quality control methods are worth exploring. previous work has asked evaluators to distinguish between human-and machine-authored text; for example, @xcite found that trained evaluators were able to detect open-ended gpt2-l-generated text 71.4% of the time and @xcite found evaluators could guess gpt3-davinci-generated news articles' source with 52% accuracy, though these results are not directly comparable to ours due to differences in the evaluation setup, data, and participants. finally, our findings that untrained evaluators are not well equipped to detect machine-generated text point to the importance of researching the safe de-ployment of nlg systems. gehrmann et al. (2019 ) proposed visualization techniques to help readers detect generated text, and work like zellers et al. (2019b), ippolito et al. (2020), and uchendu et al. (2020) investigated large language models' ability to detect generated text. ## conclusion we found that untrained evaluators were unable to distinguish between human-and gpt3-generated text from three domains. however, we also found that the evaluators focused on surface-level text qualities to make these decisions and underestimated current nlg models' capabilities. we experimented with three methods for training evaluators, and while example-based trainings led to increases in recall and the amount of content-based evaluations, they did not lead to significant improvements in accuracy across all domains. given that evaluators struggled to distinguish between human-and machine-generated text in this setting, we should shift how we think about collecting human evaluations for current nlg models.