Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020)

Abstract: Human assessment remains the most trusted form of evaluation in NLG, but highly diverse approaches and a proliferation of different quality criteria used by researchers make it difficult to compare results and draw conclusions across papers, with adverse implications for meta-evaluation and reproducibility. In this paper, we present (i) our dataset of 165 NLG papers with human evaluations, (ii) the annotation scheme we developed to label the papers for different aspects of evaluations, (iii) quantitative analyses of the annotations, and (iv) a set of recommendations for improving standards in evaluation reporting. We use the annotations as a basis for examining information included in evaluation reports, and levels of consistency in approaches, experimental design and terminology, focusing in particular on the 200+ different terms that have been used for evaluated aspects of quality. We conclude that due to a pervasive lack of clarity in reports and extreme diversity in approaches, human evaluation in NLG presents as extremely confused in 2020, and that the field is in urgent need of standard methods and terminology.

## introduction evaluating natural language generation (nlg) systems is notoriously complex: the same input can be expressed in a variety of output texts, each valid in its own context, making evaluation with automatic metrics far more challenging than in other nlp contexts @xcite . human evaluations are commonly viewed as a more reliable way to evaluate nlg systems @xcite , but come with their own issues, such as cost and time involved, the need for domain expertise @xcite , and the fact that the experimental setup has a substantial impact on the reliability of human quality judgements @xcite . moreover, there is little consensus about how human evaluations should be designed and reported. methods employed and details reported vary widely, issues including missing details (e.g. number of evaluators, outputs evaluated, and ratings collected), lack of proper analysis of results obtained (e.g. effect size and statistical significance), and much variation in names and definitions of evaluated aspects of output quality @xcite . however, we currently lack a complete picture of the prevailing consensus, or lack thereof, regarding approaches to human evaluation, experimental design and terminology. our goal in this work, therefore, is to investigate the extent of the above issues and provide a clear picture of the human evaluations nlg currently employs, how they are reported, and in what respects they are in need of improvement. to this end, we examined 20 years of nlg papers that reported some form of human evaluation, capturing key information about the systems, the quality criteria employed, and how these criteria were operationalised in specific experimental designs. the primary contributions of this paper are (1) an annotation scheme and guidelines for identifying characteristics of human evaluations reported in nlg papers; (2) a dataset containing all 165 inlg/enlg papers with some form of human evaluation published in 2000-2019, annotated with the scheme, and intended to facilitate future research on this topic; (3) analyses of our dataset and annotations, including analysis of quality criteria used in evaluations, and the similarities and differences between them; and (4) a set of recommendations to help improve clarity in reporting evaluation details. ## paper selection we selected papers for inclusion in this study following the prisma methodology @xcite recently introduced to nlp by @xcite in his structured review of the validity of bleu. as summarised in while many papers on nlg are published in other venues, including the *acl conferences, emnlp, aaai, ijcai, etc., focusing on inlg and enlg provides a simple selection criterion which at the same time ensures a set of papers representative of what researchers specialising in nlg were doing across this time period. we screened the 578 papers looking for mention of a human evaluation, first by skimming for relevant section headings and then by searching in the pdfs for 'human', 'subject', and 'eval'. this left 217 papers. during annotation (section 3), we retained only papers that reported a human evaluation in the following sense: an experiment involving assessment of system outputs in terms of an explicitly or implicitly given quality criterion, either via (1) conscious assessment of outputs in terms of the criterion by evaluators (e.g. (dis)agreement with quality statement, direct and relative assessment, qualitative feedback); or (2) counts and other measurements of outputs and user interactions with them (e.g. user-text and user-system interaction measurements, task performance measurements). we decided to allow evaluations matching the above conditions even if they did not evaluate system generated texts. this allowed the inclusion of papers which, e.g., assess wizard-of-oz or corpus texts to inform the design of an nlg system. figure 1 shows the distribution of the 165 papers meeting these conditions across publication years. the general increase of papers with human evaluations since 2012 aligns with the evaluation trends found by @xcite , who also reported an increase in the proportion of papers with intrinsic human evaluations between 2012-2015 compared to 2005-2008. however, only 28.54% of the papers in our sample contained a human evaluation compared to 45.4% reported by @xcite . ## paper annotation in order to quantitatively study the evaluations in our dataset, we needed a systematic way of collecting information about different aspects of evaluations. therefore, we developed an annotation scheme to capture different characteristics of evaluations, allowing us to investigate how human evaluations have been designed and reported in nlg over the past two decades, in particular what conventions, similarities and differences have emerged. below, we summarise our approach to studying aspects of quality assessed in evaluations (section 3.1), present the final annotation scheme (section 3.2), describe how we developed it (section 3.3), and assessed inter-annotator agreement (iaa) (section 3.4). foot_0 ## analysis and results in this section, we present results from analyses performed on the annotations of the 165 papers in our dataset. the dataset and code for analysis are available in the project repository. the 165 papers in the dataset correspond to 478 individual evaluations assessing single quality criteria, i.e. 2.8 per paper. for the quality criterion attributes (section 3.2.2) and the operationalisation attributes (section 3.2.3) it makes most sense to compute occurrence counts on the 478 individual evaluations, even if that slightly inflates counts in some cases. for example, if multiple criteria are evaluated in the same experiment, should we really count multiple occurrences for every operationalisation attribute? but the alternatives are to either count per paper, leaving the question of what to do about multiple experiments in the same paper, or to count per experiment, leaving the problem of variation within the same experiment and also that it is not always clear whether separate experiments were carried out. for these reasons we opted to compute statistics at the individual-evaluation level for the quality-criterion and operationalisation attributes. for the system attributes (section 3.2.1), we report paper-level statistics. we do sometimes find more than one system type (with different language, input, output or task) being evaluated in a paper, but for those cases we add all attribute values found for the paper. below we first report paper-level statistics for the system attributes (section 4.1), followed by evaluation-level statistics for quality-criterion and operationalisation attributes (section 4.2). ## discussion & recommendations perhaps the most compelling evidence we found in our analyses in this paper is that (i) there is very little shared practice in human evaluation in nlg, in particular with respect to what to name the aspects of quality we wish to evaluate, and how to define them; and (ii) the information presented in nlg papers about human evaluations is very rarely complete. the latter can be addressed through better reporting in future work (see below). the former is far less straightforward to address. one key observation from our data is that the same quality criterion names are often used by different authors to refer to very different aspects of quality, and that different names often refer to the same aspect of quality. we further found that more than half of the papers failed to define the criteria they evaluated, and about a quarter omitted to name the criteria being evaluated. our analysis has emphasised the need for better reporting of details of evaluations in order to help readers understand what aspect of quality is being evaluated and how. it took the first nine authors of the paper 25-30 minutes on average even in the final round of annotations to annotate a single paper, a measure of how hard it currently is to locate information about evaluations in papers. based on this experience we have put together a list of what we see as reporting recommendations for human evaluations presented in ## conclusion we have presented our new dataset of 165 papers each annotated with 16 attribute values that encode different aspects of the human evaluations reported in them. we described the carefully developed and validated annotation scheme we created for this