Non-Repeatable Experiments and Non-Reproducible Results: The Reproducibility Crisis in Human Evaluation in NLP (2023)

Abstract: Human evaluation is widely regarded as the litmus test of quality in NLP. A basic requirementof all evaluations, but in particular where they are used for meta-evaluation, is that they should support the same conclusions if repeated. However, the reproducibility of human evaluations is virtually never queried, let alone formally tested, in NLP which means that their repeatability and the reproducibility of their results is currently an open question. This focused contribution reports our review of human evaluation experiments reported in NLP papers over the past five years which we assessed in terms oftheir ability to be rerun. Overall, we estimatethat just 5% of human evaluations are repeatable in the sense that (i) there are no prohibitivebarriers to repetition, and (ii) sufficient information about experimental design is publicly available for rerunning them. Our estimate goesup to about 20% when author help is sought. We complement this investigation with a survey of results concerning the reproducibilityof human evaluations where those are repeatable in the first place. Here we find worryinglylow degrees of reproducibility, both in terms ofsimilarity of scores and of findings supportedby them. We summarise what insights can begleaned so far regarding how to make humanevaluations in NLP more repeatable and morereproducible.

## introduction human evaluation is widely seen as the most reliable form of evaluation in nlp. the traditional view in the field, here expressed for mt, is that "automatic measures are an imperfect substitute for human assessment of translation quality" @xcite . numerous papers have reported meta-evaluations of metrics in terms of correlation with human judgments @xcite @xcite @xcite . however, recently several papers have highlighted issues aris-ing from lack of standardisation @xcite , incomplete details reported for evaluation design @xcite , and poor experimental standards (van der @xcite . in this paper, we address issues that intersect with all of these. our starting premise is that in order to act as litmus test of quality, human evaluations need to be able to be relied upon to produce the same results, at least in the sense of supporting the same conclusions, when run multiple times. this ought to be a low-threshold requirement, but is in fact very rarely assessed at all, let alone routinely established for new evaluation methods. inter-evaluator agreement is more commonly assessed, but falls far short of establishing whether an experiment when repeated produces similar results and/or supports similar findings. our aim in the work reported here foot_0 is establishing the reproducibility, or otherwise, of current human evaluation practices, in order to provide evidence-based indications regarding how they can be improved, thereby going beyond recent opinionbased recommendations regarding better practice. this paper makes five main contributions: (i) an annotation scheme capturing experimental properties playing a role in repeatability and reproducibility (section 2 and we use the terms repeatability and reproducibility as follows. repeatable is a property of experiments meaning able to be repeated with identical experimental design. reproducible is a property of property definition dataset which dataset was used? quality criterion which quality criterion is being assessed? language(s) which language(s) were evaluated? task which nlp task best describes the experiment? participant type what type of participants were recruited? (crowd-source, student, etc) intrinsic or extrinsic is the evaluation intrinsic or extrinsic? absolute or relative is the evaluation absolute or relative? objective or subjective is the evaluation objective or subjective? total participants how many total participants are there? total items how many items in total? participants per item how many participants per item? items per participant how many items per participant? training session do participants take part in a training session? participant instructions are participants given instructions or shown worked examples? (beyond basic instructions on using the interface or basic criterion definitions) quality criterion definitions are participants shown definitions for any criterion they are asked to evaluate? participant practice task do participants complete practice tasks prior to the main experimental tasks? participant custom qualification are participants required to pass a custom qualification exercise? participant expertise (controlled?) does the experiment control for participant expertise? participant expertise (self report?) is any reported expertise self-reported (with no interaction with researchers, i.e. on ## repeatability of human evaluation experiments in this section, we describe (section 2.1) our 4stage process for assessing human evaluations in terms of repeatability as a precondition for inclusion in a coordinated set of reproductions. as part of this process, we annotated papers and then experiments with evaluation properties, and we examine what these reveal. because the final stage of this selection process introduced non-systematic selection (to meet the needs of the coordinated studies design), we also verify our findings on a separate, randomly selected subset of papers (section 2.2). for an overview of the selection/filtering process, see the flow diagram in appendix a. ## reproducibility of results from human evaluations to complement the assessment of the repeatability of human evaluations in nlp above, here we look at the reproducibility of results, as collated from recent reproduction studies. we examine similarity in system-level scores between original and reproduction studies (section 3.1), and assess whether scores support the same conclusions which can be the case even for dissimilar scores (section 3.2). ## discussion when corresponding with authors to find missing information (section 2.1), and when trying to find information from publicly available sources (section 2.2), properties were often not obtainable for similar reasons. high-level properties such as the dataset, task, and language, could usually be found in the paper. the total number of items was usually available, but the relationship between participants and items was not. information regarding recruitment of participants, as well as what they saw and did during the experiment almost always required additional information from the author. if authors were to make the files they used for the experiment, and a record of how these were processed (including the way they were presented to participants), then it would go a long way towards making the recreation of more experiments possible. repeatability, in the sense of being able to be repeated, is a basic requirement of all scientific experiments, perhaps most importantly as a prerequisite to independent verification through reproduction: "an experimental result is not fully established unless it can be independently reproduced" (acm, 2020). it is therefore of concern in and of itself that the large majority (95%) of human evaluations in nlp is not repeatable from publicly available information (section 2.2). this is further compounded by our finding (section 2.1) that even with considerable effort (up to three emails to first and if necessary other authors) to obtain missing information to enable repetition, 80% of experiments remain non-repeatable. finally, where we were able to obtain and review all information needed for a repetition, we found multiple reporting mistakes, errors in scripts, and ad-hoc manual interference in live experiments that call into question for scientific and/or ethical reasons whether experiments should be repeated. our analysis of reproduction results ( we would argue that we urgently need to (i) improve the repeatability of human evaluation experiments by making available publicly, as standard, full information about how the experiment was conducted, in sufficient detail to enable others to re-run it; (ii) test the results reproducibility ## conclusion nlp needs human evaluation as a litmus test of quality, including as a reliable reference for metaevaluating other types of evaluation. in order to play this role, human evaluation needs to be verifi-ably reliable, and that includes being reproducible; in order to assess the reproducibility of results, we need to be able to repeat an experiment. however, our results showed that current human evaluations have very poor repeatability (we estimated that just 5% do not have prohibitive barriers to being repeated, and can be re-run without recourse to non-public information), and where we are able to repeat human evaluations, the growing number of results from human evaluation reproduction studies show that they have low degrees of reproducibility of both scores and conclusions. we derived recommendations for making human evaluations in nlp more repeatable and more reproducible, something that we surely need to do if we are to continue treating them as our most trusted assessment of system quality. thibault sellam, dipanjan das, and ankur parikh. 2020. bleurt: learning robust metrics for text generation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 7881-7892, online. association for computational linguistics. anastasia shimorina and anya belz. 2022. the human evaluation datasheet: a template for recording details of human evaluation experiments in nlp. in proceedings of the 2nd workshop on human evaluation of nlp systems (humeval), pages 54-75, dublin, ireland. association for computational linguistics. chris van der lee, albert gatt, emiel van miltenburg, sander wubben, and emiel krahmer. 2019. best practices for the human evaluation of automatically generated text. in proceedings of the 12th international conference on natural language generation, pages 355-368, tokyo, japan. association for computational linguistics. chris van der lee, emiel krahmer, and sander wubben. 2017. pass: a dutch data-to-text system for soccer, targeted towards specific audiences. in proceedings of the 10th international conference on natural language generation, pages 95-104, santiago de compostela, spain. association for computational linguistics.