How reproducible is best-worst scaling for human evaluation? A reproduction of ‘Data-to-text Generation with Macro Planning’ (2023)

Abstract: This paper is part of the larger ReproHum project, where different teams of researchers aim to reproduce published experiments from the NLP literature. Specifically, ReproHum focuses on the reproducibility of human evaluation studies, where participants indicate the quality of different outputs of Natural Language Generation (NLG) systems. This is necessary because without reproduction studies, we do not know how reliable earlier results are. This paper aims to reproduce the second human evaluation study of Puduppully & Lapata (2021), while another lab is attempting to do the same. This experiment uses best-worst scaling to determine the relative performance of different NLG systems. We found that the worst performing system in the original study is now in fact the best performing system across the board. This means that we cannot fully reproduce the original results. We also carry out alternative analyses of the data, and discuss how our results may be combined with the other reproduction study that is carried out in parallel with this paper.

## introduction although human evaluation remains the gold standard for determining the quality of a text, little is known about the reproducibility of evaluation methods that are used to determine the quality of texts generated by natural language generation (nlg) systems (belz et al., 2020). to be sure, there are many different ways to assess nlg output. as van der lee et al. (2019) and howcroft et al. (2020) showed: different authors tend to focus on different quality dimensions (e.g. grammaticality, coherence, conciseness) and they also differ in the way they elicit quality judgments (e.g. using likert scales or ranking tasks). if we want to understand how reliable these methods are, we need to carry out multiple reproduction studies to establish the amount of variance we may expect between different studies (belz, 2022). ## method next to the original paper and materials, 3 we also have the support of the original authors. because multiple labs are all reproducing individual experiments from each paper-to-be-reproduced, we contacted the authors through the coordinator of the reprohum project, who collated all answers in a shared online document. for the current paper under investigation, this meant that four labs (and the reprohum coordinator) critically read the paper and asked questions about the methodology. although this resulted in useful additional documentation, some details about the original study were still missing (as documented below). design. we tried to match the original experiment as closely as possible. the original authors used a classical crowdsourcing design, where each ranking decision (indicating which summary is better in terms of a given quality dimension) was distributed as a separate human intelligence task (hit) on the mechanical turk platform. figure 1 provides an example hit (without the information letter or informed consent form). ## results we first provide some descriptive statistics ( §3.1) to contextualise the results, before moving on to the inter-rater reliability ( §3.2) and the system comparison ( §3.3). ## additional/alternative analyses 4.1 annotator quality next to krippendorff's alpha, we can also compute other agreement metrics. for example, we can compute proportions for how often each participant agrees with the majority (i.e., at least 2 out of 3 ratings, for any given item). the resulting weighted agreement score is higher (between 0.76 and 0.78). ## conclusion we carried out a reproduction of experiment 2 from puduppully and lapata (2021), with support from the original authors. we were not able to reproduce the exact results, instead finding opposite trends. for example, the template-based approach seems to achieve the best performance across the board, where it was actually the worst performing system in the original paper. (see next to the reproduction of the original study, we also provide an extensive selection of descriptive statistics, as well as a set of alternative analyses of the results. with these alternative approaches, we hope to have shown the possibilities and limitations of the experimental design. one key takeaway here is that it is important to have a sufficient amount of ratings per annotator (and ideally the same amount for each annotator). this enables us to dive deeper into the variation within and between ratings from different annotators. understanding this variation also brings us closer to understanding the replicability of different research results. rolf a zwaan, alexander etz, richard e lucas, and m brent donnellan. 2017. making replication mainstream. behavioral and brain sciences, 41:1-61.