(Mostly) Automatic Experiment Execution for Human Evaluations of NLP Systems (2024)

Abstract: Human evaluation is widely considered the most reliable form of evaluation in NLP, but recent research has shown it to be riddled with mistakes, often as a result of manual execution of tasks. This paper argues that such mistakes could be avoided if we were to automate, as much as is practical, the process of performing experiments for human evaluation of NLP systems. We provide a simple methodology that can improve both the transparency and reproducibility of experiments. We show how the sequence of component processes of a human evaluation can be defined in advance, facilitating full or partial automation, detailed preregistration of the process, and research transparency and repeatability.

## introduction the traditional method for recording the steps performed in a scientific experiment is the pen and paper logbook. @xcite argues that in the event of a fire in the lab, it is the only thing that one should grab, leaving computers, physical samples, and expensive equipment behind. in fields such as chemistry, students are taught systematic approaches for completing such records, which commonly include the date of the experiment, the hypothesis, the steps carried out, and the results. 1,2 these days, researchers may feel less compelled to grab their paper records (or even their computer) in case of fire, since they can record their notebooks digitally and have them immediately backed up to the cloud. however, at least in natural language processing (nlp), it appears that this has not helped to ensure survival of records of experimental procedures which are rarely available after the fact, in any form @xcite . even basic records and other data files such as the set of system outputs that were evaluated or the question that participants were asked are seldom made publicly available @xcite . when contacted, around two thirds of corresponding authors do not respond @xcite , and only around half of those who do can provide this basic information. mistakes by researchers whilst running experiments are depressingly common @xcite and reproduction attempts often struggle to find and follow the original procedure, even with the help of the authors @xcite @xcite . automated experimentation techniques @xcite , where the experimental process is defined in advance and researcher intervention kept to a minimum during experiment execution, can remove reliance upon error-prone manual data entry. such techniques also benefit from having a clear experimental procedure which must be defined in advance, making it impossible for researchers to change the configuration part way through a run (accidentally or nefariously). automating processes is essential for large scale experiments where massive volumes of data are collected and processed in real time, e.g., in particle physics @xcite . for the field of economics, @xcite propose that researchers should automate everything they can, ideally with a single code script, such that repeatability is ensured. the state of human evaluation in nlp research more generally is dire @xcite . most work reporting on the state of human evaluation in nlp research has focused on aspects of design such as participant guidelines @xcite , quality criterion names and definitions assessed @xcite , or the comparability of experiments @xcite . such aspects of the experimental design are vitally important, but separate to the question of how the experiment procedure is recorded and executed. we argue that many of the above issues would be at least ameliorated by automating experimental execution as much as possible. some experiments, such as those that use crowd platforms like amazon mechanical turk or prolific, can be fully automated using the available apis. at a minimum, it is straightforward to see from figure 1 that everything prior to present participants with evaluation items can be automated as one pipeline, as can everything from responses onwards. in both cases, we would simply be pipelining a series of operations on data. automation can also be applied to the process of collecting responses and checking/excluding them. 3 in the rest of this paper, we start by investigating whether the individual files and component processes that make up a human evaluation tend to be reported (section 2), before proposing a methodology for achieving automation (section 3). we describe an example application of the methodology (section 4) and end with some conclusions and a look to future work (section 5). all steps except for presenting the evaluation items to participants are simple to automate before the experiment. 3 an example of how to do this is included in our example experiment on github: https://github.com/nlgcat/ mostly_automated . ## availability of experiment components we performed a systematic analysis of papers made available in the repronlp 2024 shared task on reproducibility of evaluations in nlp @xcite , with the aim of establishing which evaluation experiment components were (not) made available by researchers. the shared tasks organisers made available resources that were obtained from the authors, including the evaluation items and interface. with only 5% of authors making such details publicly available @xcite and only 17% of authors being able to do so after being contacted @xcite , re-pronlp provides a good sample of 20 papers where authors have made the effort to share resources. we broke down the experimental process into the data files and component processes shown in figure 1 . rather than use a more complex process with exhaustive options that cover all types of human evaluation, we use the simplest overall process that includes exclusion of responses. we argue that most human evaluations of nlp system quality will require these component processes, even if they also include other ones or the control flow logic differs (for a more generally applicable breakdown into component processes see @xcite ). it therefore is a good vanilla design that is useful for both designing experiments, and for checking that published papers include at least minimal data files and component process definitions. note that response exclusion needs to be handled with care and should always be fully specified in advance. we then annotated each paper, first checking to ensure that the overall process shown in figure 1 was applicable to the experiment being carried out in the paper (it was in all cases). we then checked files and component process definitions were available. when doing so, we looked only for evidence of the resources being available; we did not check their validity. anonymised results of our annotation process are shown in we propose a simple and flexible high-level methodology for creating mostly-automated experiments that evaluate the performance of nlp systems. for the sake of brevity, below we refer to these simply as human evaluations, with the caveat that the vanilla experiment structure introduced above is likely not applicable to all experiments. the overall procedure for human evaluation experiments can be broken down into component processes, where each component process takes one or more data files as input, performs some operation on them, and then outputs one or more data files. figure 1 shows an example of a minimal human evaluation experiment modelled in this way. an example component process is to input the system inputs and system outputs to the subset selection component process that outputs the evaluation items. the flow control of the overall process can then be modelled with simple loops and other conditional logic, using basic computing sci-ence concepts. processes may be fully automated, partially automated, or entirely manual. the crucial thing is that they are fully defined in advance of the experiment and then automated as much as possible. most experiments will require additional steps in practice, although, with the exception of response exclusion, those shown in figure 1 are core steps that most human evaluations would require in order to function at all. for examples of similar but more fine-grained methods designed for the similar task of dataset annotation, please see @xcite and @xcite . each of the files and processes in figure 1 can be mapped to a question in the human evaluation datasheet (heds), which includes comprehensive details of possible components in a human evaluation @xcite . note that we do not consider here the design of the interface or questions that participants are asked. these are important considerations but separate from issues of process. ## example experiment in this section we describe an example experiment where data-to-text system outputs are evaluated. for this, we use data and system outputs from the webnlg 2017 challenge @xcite , where systems convert structured input (triples) to text. the entire experiment is encoded in a jupyter notebook which is included as supplementary material. for system texts we use the constant string "example text" since we are not showing any texts to participants during our implementation. ## conclusion and future work many of the suggestions we make in this paper may seem obvious to most computing science researchers. nevertheless such a structured approach to human evaluation experiments is rarely followed in research. that the methodology proposed here is so simple means it should be straightforward to implement for most experiments. doing so comes with the benefits of reduced manual data entry errors, improved repeatability, ease of preregistration, and assurance to readers that the experiment has not undergone ad hoc and biased changes as the researcher made observations during the process.