Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP (2023)

Abstract: We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP.

## introduction there is increasing awareness in natural language processing (nlp) that reproducibility of results, most particularly of results from system evaluations, matters greatly, and that currently the field does not assess reproducibility of results rigorously enough, and lacks a common approach to it. recent work has made progress particularly with respect to automatic evaluation @xcite , but reproducibility of human evaluation, widely considered the litmus test of quality in nlp, has received less attention. it could be argued that if it is not known how reproducible human evaluations are, it is not known how reliable they are; and if it is not known how reliable they are, then it is not known how reliable automatic evaluations meta-evaluated against them are either. the work reported in this paper forms part of the reprohum project foot_0 in which our aim is to build on existing work on recording properties of human evaluations datasheet-style @xcite , and assessing how close results from a reproduction study are to the original study @xcite , to investigate systematically what factors make a human evaluation more-or lessreproducible. in this paper, we present the findings from our work on the project so far which necessitated a rethink of our entire approach to designing such an investigation. section 2 outlines our motivation for carrying out a multi-lab multi-test (mlmt) study of factors affecting reproduciblity in nlp, and our original design for the study. section 3 describes our paper selection, annotation and filtering process which yielded a surprisingly small number of candidate papers for reproduction. in section 4 we describe the numerous further issues with original evaluation studies we encountered in the process of setting up reproductions of them with partner labs. section 6 summarises our negative findings regarding the infeasibilty of assessing the reproducibility of previously conducted human evaluations in nlp as they are, and outlines the changes to our multilab multi-test study necessitated by the findings. ## motivation and overall study design individual studies can tell us how close a reproduction study's results are to those in the original study. a large number of such studies can show general tendencies regarding what kinds of evaluations have better reproducibility. however, we do not currently have a large number of reproduction studies in nlp and because of their cost and lack of appeal, this is unlikely to change. moreover, accumulations of individual studies do not provide the conditions in which the effect size and significance of specific factors on reproducibility, and interactions between them, can be measured. to create such conditions, a controlled study of equal numbers of reproductions with and without factors of interest is needed. moreover, we know from existing work @xcite that different reproductions of the same original work can produce very different results. finally, while it is instructive to test for reproducibility under identical conditions, it is also of interest to test how far good reproducibility can stretche.g. is reproducibility affected by replacing, say, a 7-point quality scale with a 5-point one. a study of factors that increase/decrease reproducibility therefore needs to (i) conduct more than one reproduction of each original study, (ii) carried out by a good mix of different teams, and to (iii) incorporate multiple rounds with decreasing similarity of conditions. the steps in setting up such a study would be as follows: ## selection and assessment of candidate evaluation experiments figure 1 shows the selection and annotation process in the form of a flow diagram showing the decreasing number of remaining papers/experiments. the first step was to conduct a search on the acl anthology for papers published in acl (main conference) or tacl in the 2018-2022 period foot_1 which included the phrases "human evaluation" and "participants;" we found 177 such papers. ## setting up reproductions beginning the process of reproduction of the six experiments finally selected for reproduction (for common agreed approach to reproduction see appendix a.5) necessarily involved delving into full implementational details for each of them. one particularly troubling finding has been the number of experimental flaws, errors and bugs we unearthed in the process. the more we dug into the properties of evaluation experiments that we needed in order to repeat an evaluation experiment, the more we uncovered flaws which made us question whether it made sense to repeat the experiment at all, in some cases because any conclusions drawn on the basis of the flawed experiments would be unsafe. six specific issues are listed in section a.6. that only one of our six selected experiments had none of these issues. we are still discovering more. the structure we designed for our original study is shown in the appendix section a.1, figure 2 . ## discussion the reasons why we decided to abandon our original study design were as follows. one, we struggled to find enough papers that did not have (i) prohibitive barriers to reproduction, and/or (ii) unavailable information that would be needed for repeating experiments, and/or (iii) experimental flaws and errors. two, no matter how much effort we put into obtaining full experimental details from authors, there still remained questions, albeit increasingly fine-grained, that we did not have the answer to, such as if the presentation order of evaluated items was randomised, or what instructions/training participants were given. in some cases, information about additional things that had been done, but could not be guessed from previously provided information, transpired coincidentally, necessitating further changes to experimental design. a potential solution to not having enough papers at the end is selecting more papers at the start (more years, more events). however, given the inordinate amount of work we put into obtaining enough information from authors, simply tripling or quadrupling our initial pool of papers was not a viable solution. similarly, there was little we were able to do about the reproduction barriers of excessive cost and highly specialised evaluators. on the other hand, accepting to work from less than complete experimental information would have been problematic because information for different papers is incomplete in different ways, and we would not have been comparing like with like. correcting flaws and errors would similarly have introduced differences between original and reproduction studies, moreover different ones in different cases. in this case we would strictly speaking no longer have been conducting reproductions. we considered designing new evaluations from scratch with the properties we wanted for our mlmt study. however, it would have been very difficult to ensure that newly created studies were somehow representative of the kind of studies that are actually being conducted in nlp. we have now opted for a solution incorporating elements from most of the above, where we select a somewhat larger set of existing studies in a process similar to before, reduce the number of different values of factors we control for, and then standardise and where necessary correct studies before reproduction. reproducibility is then measured between two new studies, rather than between them and the original study. ## conclusion the track record of nlp as a field in recording information about human evaluation experiments is currently dire @xcite . we saw in the paper-level annotations (appendix our overall conclusion is that, on the basis of the unobtainability of information about experiments, barriers to reproduction and/or experimental flaws in our sample of 177 papers, only a small fraction of previous human evaluations in nlp can be repeated under the same conditions, hence that their reproducibility cannot be tested by repeating them. the way forward would appear to be to accept the overhead of detailed recording of experimental details, e.g. with heds @xcite , in combination with substantially increased standardisation in all aspects of experimental design. our initial email to authors asked if they would be able to provide the following information: