GENIE : Toward Reproducible and Standardized Human Evaluation for Text Generation (2022)

Abstract: While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.We revisit this problem with a focus on producing consistent evaluations that are reproducible—over time and across different populations. We study this goal in different stages of the human evaluation pipeline. In particular, we consider design choices for the annotation interface used to elicit human judgments and their impact on reproducibility. Furthermore, we develop an automated mechanism for maintaining annotator quality via a probabilistic model that detects and excludes noisy annotators. Putting these lessons together, we introduce GENIE: a system for running standardized human evaluations across different generation tasks.We instantiate GENIE with datasets representing four core challenges in text generation: machine translation, summarization, commonsense reasoning, and machine comprehension.For each task, GENIE offers a leaderboard that automatically crowdsources annotations for submissions, evaluating them along axes such as correctness, conciseness, and fluency.We have made the GENIE leaderboards publicly available, and have already ranked 50 submissions from 10 different research groups. We hope GENIE encourages further progress toward effective, standardized evaluations for text generation.

## introduction while the emergence of powerful language models @xcite @xcite has made text generation omnipresent, effective evaluation of the resulting systems' performance on open-ended generation tasks remains a challenge. this has motivated adoption of human evaluation in recent works @xcite , even though it poses seahawks' defense comes up big late against patriots news article predicted summary bleu, bertscore rouge, ... automatic eval. human eval. this work presents an investigation toward reliably repeatable and standardized human evaluation. first and foremost, we study the reproducibility of human annotations, in two stages of the annotation pipeline. we study this goal empirically as a func-tion of various design choices ( §4) such as the way the judgments are aggregated. we then propose a probabilistic framework for detecting malicious annotators ( §5) and isolating their annotations from the resulting performance estimates. 2 guided by the earlier studies, we present ge-nie (figure 1 )-a framework for human evaluation of text generation, which scales to a variety of tasks and datasets ( §6). genie posts model predictions to a crowdsourcing platform, 3 where human annotators evaluate them according to predefined, dataset-specific guidelines. we describe mechanisms introduced into genie to quantify annotator variance and spread the annotations across various days, showing that genie achieves reliable scores on the studied tasks. to show its applicability, we instantiate genie with leaderboards for several popular text generation datasets in english from four diverse tasks-machine translation, question answering, summarization, and commonsense reasoning-and invite developers to extend it with more datasets. since its deployment, genie has analyzed and ranked about 50 submissions from 10 different groups across all of our tasks, indicating the interest in standardized human evaluation. the genie infrastructure opens the door for three avenues of research: (1) genie provides developers of text-generation models with the ease of the "leaderboard experience," alleviating the evaluation burden while ensuring high-quality, standardized comparison against previous models. (2) genie facilitates the study of human evaluation interfaces @xcite @xcite , addressing challenges such as annotator training, inter-annotator agreement, and reproducibility, all of which can be integrated into genie to compare against other evaluation metrics on past and future model submissions. (3) genie helps developers of automatic evaluation metrics @xcite , by serving as a hub of model submissions and associated human scores. ## related work we survey relevant work on automatic and humanin-loop evaluation of text generation. see @xcite ; van der @xcite ; @xcite for further in-depth discussion. ## genie principles for human evaluation of generative models there are many ways to run human evaluations. reflecting on what's needed to compare text generation models across time, we formulated the following principles to guide our design choices. application-motivated ultimately, the evaluation's purpose is to identify useful models and techniques. thus, it should measure something informative about the their usefulness in applications (such as a generated text's correctness or fluency). reproducible to compare different models over time, the evaluation must be reproducible. if repeated, it should give largely the same results. for example, results should hold across different groups of annotators, and remain stable across appropriate lengths of time. interpretable the evaluation should help a researcher understand how the system behaves, and thus must measure an aspect of the system that is easy to understand. an evaluation which ranks models but isn't interpretable has limited usefulness, since different applications might prioritize different things and researchers must navigate costbenefit trade offs between more expensive, higher performing models and cheaper ones. ## design decisions for consistent human evaluations when designing an evaluation, some questions can be answered with principles, while others must be answered empirically. we investigate several questions around the prompt design that commonly occur across various tasks and impact evaluations' reproducibility and confidence. one label for three times foot_2 as many annotation examples (unilabeling). ## automatically managing human evaluation leaderboards this section reviews the genie system, which automates much of the management of text generation leaderboards with human evaluations. while we note that some human management, such as providing support and handling disputes, should never be fully automated, genie alleviates much of the overall burden. the next section ( §7) describes its instantiation into the genie leaderboards for four diverse text generation tasks. at a high level, the genie system coordinates a leaderboard ui, data processing backend, and crowdsourcing campaigns on amazon mechanical turk. after retrieving newly uploaded submissions, the backend computes automatic metrics. upon success, the backend then creates annotation tasks on amazon mechanical turk (amt) using amti foot_10 (a mechanical turk inferface), an opensource python package for working with amt. each leaderboard is a separate instance of the system, with its own crowdsourcing templates, including instructions, examples, and prompts (see §7). following our observations in §4, all templates use likert scales which are then mapped to real-valued scores (cf. footnote 6) and averaged. the system also maintains a history of past annotations (per-instance and per-worker), updating statistics after each evaluation. this has several immediate and future benefits: worker statistics enable spam detection ( §5), while the annotations can be used for future studies on human evaluation. these components enable the following features: extensibility new tasks can be modularly added to the genie system, creating new leaderboards. each task requires a crowdsourcing template and a code object specifying how to push model predictions into and pull workers' annotations from the crowdsourcing templates. we release an extensible open-source annotation template library, foot_11 seeded with the four task templates used in this work. uncertainty quantification to better inform model comparisons, we report scores with uncertainty estimates. bootstrap resampling (samples with replacement from the observed annotations) provides the 95% confidence intervals for the estimated submission quality scores, as commonly done in machine translation @xcite . ## limitations as with other works which deal with human annotation, the results generated via our evaluation framework will have inherent variability. while we tried to mitigate sources of variation in various ways (see §5,6), some are bound to remain and are hard to account for. these include, for example, selection bias in the pool of annotators that choose to work on our tasks, who may come from specific countries and social status and select for certain tasks and their templates. we welcome future evolution of all parts of the genie architecture, including its evaluation metrics. ## conclusion and future work we introduce genie, a unified approach to humanin-the-loop evaluation of text generation over a wide set of text generation tasks. genie is open for use and will be adapted based on future adoption. we encourage submissions from all researchers interested in text generation models.