The Human Evaluation Datasheet: A Template for Recording Details of Human Evaluation Experiments in NLP (2022)

Abstract: This paper presents the Human Evaluation Datasheet (HEDS), a template for recording the details of individual human evaluation experiments in Natural Language Processing (NLP), and reports on first experience of researchers using HEDS sheets in practice. Originally taking inspiration from seminal papers by Bender and Friedman (2018), Mitchell et al. (2019), and Gebru et al. (2020), HEDS facilitates the recording of properties of human evaluations in sufficient detail, and with sufficient standardisation, to support comparability, meta-evaluation,and reproducibility assessments for human evaluations. These are crucial for scientifically principled evaluation, but the overhead of completing a detailed datasheet is substantial, and we discuss possible ways of addressing this and other issues observed in practice.

## introduction human evaluation plays a special role in nlp and nlg in particular as it is generally seen as the most reliable form of evaluation @xcite @xcite . comparability and reproducibility of evaluations (both human and automatic) are gaining in recognition and importance across nlp, as the field develops towards scientific maturity. for both reasons, it is of concern that there continues to be little consensus or standard practice across all aspects of human evaluation in nlp. other efforts are aimed at standardisation of evaluation methods and quality criteria @xcite . with the human evaluation datasheet (heds) we wish to provide simply a way of recording properties of human evaluations in a standard hence comparable form (regardless of the level of standardisation of the deployed methods themselves). heds is a template for recording properties of single human evaluation experiments. it uses multiple-choice questions where possible, for in-creased standardisation and automatic comparability. heds is designed to be generally applicable to human evaluations across nlp. it consists of 29 experiment-level questions plus 17 for each quality criterion, available as an online form which also contains explanations and guidance for completing it. the idea is that researchers use completed forms for preregistration of experiments and for archiving their details, to aid in comparability of evaluations across experiments, meta-evaluation of evaluation methods, and reproducibility of results. our intention is that heds should be suitable for all human evaluation experiments in nlp. human evaluations in nlp typically get participants to assess system outputs or to interact with systems, but the heds sheet also accommodates what we call 'human-authored stand-ins' below, i.e. manually created 'system outputs' (e.g. in a wizard-of-oz scenario or when reference outputs are included in an evaluation) evaluated in a way that can at least in principle be used to evaluate actual system outputs. the sheet is completed for a single human evaluation experiment by which we mean an experiment that evaluates a single set of directly comparable systems in a single experimental design, but may assess multiple quality criteria. this is the intended meaning when we refer to 'the evaluation experiment' in questions below. ## relationship to existing work a first version of the datasheet (heds 1.0) was reported via a non-archival pre-print @xcite . in this paper, we present heds in its next revision (v2.0) alongside a summary of what we learnt from observing the datasheet being used in practice since its first publication. we focus discussion in this paper on questions 4.1.1-4.2.3 (sections 5 and 6) relating to quality criteria and their operationalisation: these caused some difficulty to users in practice, and were revised more substantively than other questions. the full datasheet is provided in the appendix for reference. heds directly benefited from several papers and resources. questions 2.1-2.5 relating to evaluated system, and 4.3.1-4.3.8 relating to response elicitation, are based on @xcite , with some significant changes. questions 4.1.1-4.2.3 relating to quality criteria, and some of the questions about system outputs, evaluators, and experimental design (3.1.1-3.2.3, 4.3.5, 4.3.6, 4.3.9-4.3.11) are based on @xcite . heds was also informed by @xcite and by @xcite 's foot_0 data card guide. more generally, the original inspiration for creating a 'datasheet' for describing human evaluation experiments of course comes from seminal papers by @xcite , @xcite and @xcite . heds is related to other efforts in the nlp community related to reproducibility and ethics. different nlp checklists have been introduced in recent years, e.g. the reproducibility checklist @xcite adopted by many conferences, and the acl rolling review's responsible nlp checklist. 2 these checklists mainly deal with the recreatability of computational experiments, details of used datasets and models, and risks and limitations of research studies and applications. the focus of heds is recording properties of human evaluation experiments, which are not covered by the above checklists. ## heds structure and resources the human evaluation datasheet package consists of the following three resources: a collection of completed heds datasheets is also available at the heds github repository. the full heds sheet can be found in appendix a. in its template form as well as in this paper, heds is divided into five sections, addressing topics and containing questions as follows: section a.1 records bibliographic information: link to the paper reporting the evaluation experiment, shared evaluation resources (e.g., a webpage, repository), contact author details. section a.2 describes information about outputs that are evaluated in the evaluation experiment and how they were produced. for example, it records the task performed by the system, types of system input and output, input and output language. section a.3 collects information about the evaluated sample (size, selection process, statistical power), the evaluators participating in the experiment, and experimental design (collection method, quality assurance, conditions for evaluators). section a.4 captures information about quality criteria assessed in the human evaluation experiment. we discuss this section in the main body of the paper in section 6. section a.5 relates to ethical aspects of the evaluation: approval by ethics committees, and collection of personal and other sensitive data. ## insights from use of heds 1.0 in practice heds 1.0 was used in the 2021 reprogen shared task on reproducibility of evaluations in nlg foot_2 @xcite . all shared task participants, as well as the authors of the original papers up for reproduction in track a, completed the heds 1.0 form. moreover, the heds 1.0 sheet was completed another three times outside of the shared task context. general feedback from users was that the heds 1.0 sheet was for the most part straightforward to complete, but that completion still represented a considerable overhead. this is the general conundrum of initiatives such as heds: what is the incentive for researchers to complete the sheet when (i) it is merely good scientific hygiene rather than a component of the work without which it could not be carried out, and (ii) it mainly benefits follow on research rather than the paper it is completed for? unless it is a requirement for submission to a particular event, or it is generally expected practice, the tendency may always be to avoid the overhead. to address this, we are in the process of preparing a shorter version of the sheet, with the aim of cutting the effort involved in half, for use in contexts where less detail is acceptable. we also observed that there were a number of questions in heds 1.0 that users found difficult to complete for different reasons. question 3.1.3 (what is the statistical power of the sample size?) posed difficulties partly because power calculations are a relatively new tool in nlp. we address this in this paper with additional information, and in the future by providing a new resource to support calculation. a more fundamental issue was caused by how heds 1.0 captured information about quality criteria, and the use of duplicate subsections for multiple quality criteria. we believe that this was due largely to insufficient context, motivation and explanation being provided in the documentation and form about quality criteria and their properties, and we seek to provide the latter in version 2.0 (in this paper as well as in the form). using heds as part of the reprogen shared task demonstrated the utility of having information about original studies and reproduction studies available in the same standard format: it meant it was straightforward for organisers to capture and analyse the similarities and differences between original and reproduction studies, e.g. to identify sources of variation in results for the results report @xcite . it also gave participants (authors of reproduction papers) a tool with which to verify whether their reproduction study of human evaluation was the same as the original study in all important respects, at a fine-grained level of detail. ## concepts underlying quality criterion questions 4.1.1-4.3.11 in heds the overall aim of human evaluations in nlp is generally to assess some aspect of the quality of a system or component. researchers use terms such as fluency and informativeness to refer to different aspects of quality. however, as discussed in detail by @xcite and @xcite , just because two studies used the same term (e.g. fluency) it does not mean they evaluated the same aspect of quality. in order to establish what was evaluated, we need to know the term and definition used, but also how it was 'operationalised,' i.e. what was presented to evaluators and how their assessments were recorded. this is why heds, picking up from the two publications above, records properties relating to evaluation criteria and their operationalisation separately for each quality criterion. because within the same experiment different quality criteria are often assessed in similar ways (e.g. using the same rating instrument), this can result in some repetition when completing a heds sheet, albeit unfortunately not in predictable ways. following @xcite , properties relating to quality criteria and their operationalisation in heds fall into three groups: quality criteria, evaluation mode, and experimental design. a quality criterion is a criterion in terms of which the quality of system outputs is assessed, and is in itself entirely agnostic about how it is evaluated. evaluation modes are properties that need to be specified to turn a quality criterion into an evaluation measure that can be implemented, and are orthogonal to quality criteria, i.e. any given quality criterion can be combined with any mode. experimental design is the full specification of how to obtain a quantitative or qualitative response value for a given evaluation measure, yielding a fully specified evaluation method. in sum: each of the above concepts is covered by one or more questions in heds. three heds questions capture properties of quality criteria in terms of (i) what type of quality is being assessed (question 4.1.1); (ii) what aspect of the system output is being assessed (question 4.1.2); and (iii) whether system outputs are assessed in their own right or with reference to some system-internal or systemexternal frame of reference (question 4.1.3). three questions capture evaluation mode: (i) subjective vs. objective (question 4.2.1), (ii) absolute vs. relative (question 4.2.2), and (iii) intrinsic vs. extrinsic (question 4.2.3). experimental design is covered by questions 4.3.1-11 (operationalisation of quality criteria assessment), and questions 3.* (other aspects). we present, explain and discuss each of the above questions in the following section. ## heds questions about properties of quality criteria and their operationalisation in this section, we present, verbatim, the questions referred to in the last section relating to properties of quality criteria and their operationalisation. all questions in this section need to be completed once for each quality criterion assessed in the single evaluation experiment that a heds sheet is being completed for. e.g. if an evaluation assesses fluency and grammaticality, then the questions below need to be filled in, separately, for each. we refer below to 'output' as shorthand for that which is being assessed by evaluators. the latter is most often some form of language output assessed at different lengths (phrase, sentence, document), but it can also be a more complete form of system behaviour (e.g. language displayed along with audio and/or visual elements, on interfaces, etc.). it is in this more general sense that we intend the term 'output' to be understood in the present context. note that if nlp had a generally accepted standard set of quality criteria with common names, definitions and operationalisations, then most of the questions in this section could be replaced by a much smaller set capturing criterion name and operationalising techniques. the reason there are so many questions is precisely because we do not have such a standard nomenclature. question 4.1.1: what type of quality is assessed by the quality criterion? multiple-choice options (select one): question 4.1.2: which aspect of system outputs is assessed by the quality criterion? multiple-choice options (select one): question 4.1.3: is each output assessed for quality in its own right, or with reference to a system-internal or external frame of reference? multiple-choice options (select one): ## conclusion in this paper we have presented the human evaluation datasheet (heds), intended as a way of recording properties of human evaluations in nlp in a sufficiently standardised way to support comparability between evaluation experiments, metaevaluation of evaluation methods, and reproducibility assessments of evaluation results. we have reported insights from observing heds 1.0 being used in practice, and have described improvements we have made in response to these insights. in particular, we have provided additional context, motivation and explanation to the heds questions relating to evaluation criteria assessed in evaluation experiments and their operationalisation. moreover, we are currently developing a shorter version of heds, a version with reduced effort for use in certain contexts. we view heds as continuing to develop in response to feedback received and insights gathered through use in practice. we continue to welcome feedback on any aspect of heds, and hope the growing repository of completed sheets will prove useful for future comparisons, meta-evaluation and reproducibility assessments, as demonstrated in the reprogen shared task.