HEDS 3.0: The Human Evaluation Data Sheet Version 3.0 (2025)

Abstract: This paper presents a new version of the Human Evaluation Datasheet (HEDS), numbered 3.0 This update is the result of our experience using HEDS in the context of numerous recent human evaluation experiments, including reproduction studies, and of feedback collected from other researchers. Our main overall goal was to improve clarity, and to enable users to complete the datasheet more consistently and comparably. The HEDS 3.0 package consists of the digital data sheet, documentation, and code for exporting completed data sheets as latex files, all available from the HEDS 3.0 GitHub.

## introduction the human evaluation datasheet (heds), first introduced in 2021 @xcite , is conceived as a template for recording and reporting the details of human evaluation experiments in a standardised and comparable way with nlp-wide scope. it has been extensively used in practice, in particular in the context of the re-progen/repronlp shared task series @xcite @xcite @xcite , where organisers and participants have been completing heds sheets for original studies and reproduction studies, respectively. 2 this in turn has provided new insights into what information heds needs to capture, what functionality is needed in an interactive tool for its completion, and what guidance needs to be provided to users to enable them to complete heds sheets quickly and consistently. we have channelled these insights into a new version update of heds, numbered 3.0 which has (i) major updates to questions and answers, (ii) new resources provided as part of the heds 3.0 package, and (iii) improved detail and clarity in the user guidance. re i, we have added two new questions, and replaced seven questions with two or more specific ones each. re ii, we have replaced the original google form with the tailored interactive heds 3.0 tool which supports browsing, revision, prefilling of some questions, and exporting to latex and json. re iii, we have revised, extended and improved the clarity of completion instructions and incorporated them into the heds 3.0 tool. the paper is structured as follows. we summarise contributions to previous versions of heds on which heds 3.0 is based (section 2). we present an overview of heds 3.0 in terms of the components that make up the heds 3.0 package in section 3.1, followed by a description of question types and presentational conventions (section 3.2). section 3.3 presents the parts of the instructions from the heds 3.0 tool that relate to the content of the form (omitting those relating to technical aspects of the tool only). a summary of differences between questions in heds 3.0 vs. heds 2.0 can be found in section 3.4. section 4 gives an overview of the heds 3.0 tool, and section 5 describes envisaged uses of heds. in section 6 we provide additional explanations for some aspects of heds 3.0 that we know from experience users may find more difficult. we end with some discussion and conclusions in section 7. the complete heds 3.0 sheet is included in the appendix, as a printout of questions and possible answers automatically generated from the version of the sheet used in the heds 3.0 tool (appendix a). the questions newly added in heds 3.0 (see section 3.4) were created by the authors of this paper to address documentation needs that arose primarily in the context of the reprohum project and related repronlp shared task series @xcite . 3 for example, whereas q3.2.2 previously asked a single broad question about the type of evaluators used; there are now separate questions for domain expertise (q3.2.2.1), payment (q3.2.2.2), and whether the participants were authors (q3.2.2.4), or previously known to the 4. definition and operationalisation of quality criteria (questions 4.1.1-4.3.12.2); 5. ethics (questions 5.1-5.4). in appendix a we present the heds 3.0 form in its entirety, in a similar look/feel to the online version that users complete (in fact, the whole section is generated automatically from the form). questions come in the following types and presentation formats: for example, question 4.2.2 asks "are outputs assessed in absolute or relative terms?", with response options of "absolute" or "relative". ## the heds 3.0 tool a web-based version of heds 3.0 has been implemented in html and javascript. it can be accessed for online completion, 4 or alternatively, users can download the code 5 and run it on their own computer. figure 1 shows a screenshot of the heds 3.0 tool homepage. the sidebar to the left contains: the main body of the form has seven expandable headers. first there is the introduction, which explains what heds is and how to use the form. then are five numbered sections that correspond to the numbered heds sections as shown in section 3.2 and as can also be seen in appendix a. when expanded, these sections contain further expandable headers and ultimately, questions. for example, in figure 2 , the section 1 header and then the 1.3 and 1.3.1 headings have been expanded, revealing the three q1. finally, there is the all form errors section (bottom left of figure 1 ) which when expanded will show the numbers of all questions that have errors. ## envisaged uses we envisage the main uses of heds to be as follows. ## discussion and conclusion it is the norm @xcite in nlp to publish very little detail about human evaluations, with complete sharing of details practically unheard of @xcite . this is true even in cases where major conclusions in a paper depend on the results. for example, it is quite common to mention just the number of evaluators used, and the quality criteria assessed, before presenting tables of mean ratings. clearly, in this situation it's not possible to assess whether the evaluation is sound, the methods of analysis applied are appropriate, or conclusions supported. moreover, without publishing details of human evaluations, it can't be established whether two evaluations assess the same thing, thus whether they agree with each other or not in their assessment of different types of systems. without that, our ability to build on results, to progress collectively as a field of science, is greatly reduced @xcite . diligent reporting always represents an overhead in terms of effort, one that in the fast moving field of nlp it is tempting to avoid. however, the more impactful nlp (and ai more generally) becomes, the more important it is that it adopts scientific practices, and reporting full details of evaluations is an important part of that. with heds, our aim is to contribute to this change, reducing the load on researchers somewhat by making it possible to report full details about a human evaluation by completing an interactive form, then exporting a fully formatted pdf that can simply be attached as an appendix or supplementary material of the paper reporting the work. it can also be exported to json format for use in automatic comparison between multiple evaluations for use in e.g. comparability and reproducibility assessments. the resources used in the experiment can be found here (enter url(s)). no resources shared. 1.3 contact details 1.3.1 details of the person completing this sheet. question 1.3.1.1: name of the person completing this sheet. question 1.3.1.2: affiliation of the person completing this sheet. question 1.3.1.3: email address of the person completing this sheet. 1.3.2 details of the contact author question 1.3.2.1: name of the contact author. question 1.3.2.2: affiliation of the contact author. question 1.3.2.3: email address of the contact author.