Unveiling NLG Human-Evaluation Reproducibility: Lessons Learned and Key Insights from Participating in the R epro NLP Challenge (2023)

Abstract: Human evaluation is crucial for NLG systems as it provides a reliable assessment of the quality, effectiveness, and utility of generated language outputs. However, concerns about the reproducibility of such evaluations have emerged, casting doubt on the reliability and generalisability of reported results. In this paper, we present the findings of a reproducibility study on a data-to-text system, conducted under two conditions: (1) replicating the original setup as closely as possible with evaluators from AMT, and (2) replicating the original human evaluation but this time, utilising evaluators with a background in academia. Our experiments show that there is a loss of statistical significance between the original and reproduction studies, i.e. the human evaluation results are not reproducible. In addition, we found that employing local participants led to more robust results. We finally discuss lessons learned, addressing the challenges and best practices for ensuring reproducibility in NLG human evaluations.

## introduction human evaluations have long been considered the appropriate method for reliably evaluating nlg systems, due to the shortcomings of automatic evaluation metrics @xcite . notwithstanding the popularity and acceptance of human evaluations in the nlg community, the reproducibility of human evaluations has not been thoroughly documented. it is widely accepted that replicating human evaluation results can be really hard due to insufficient documentation @xcite , variability in the generated text that leads to varying assessment results due to evaluator preferences, background, or other characteristics @xcite , confusion and diversity in defining evaluation criteria @xcite , high costs @xcite or even difficulty in identifying factual errors @xcite . the reprogen/repronlp challenges led by @xcite aim to address the lack of understanding surrounding the reproducibility of human evaluations in nlg research by facilitating and encouraging research on the reproducibility of current evaluation methods, and the factors leading to irreproducibility. it involves a global multi-lab shared task study aimed at assessing the reproducibility of human evaluations conducted in selected published nlg research papers. the challenge organisers initially selected studies to be reproduced and allocated them to each participating lab. each lab is then tasked with reproducing the results of the allocated human evaluation study, allowing for an assessment of the reproducibility of human evaluations across different methods, tasks, and participant characteristics. labs were also allowed to explore additional research questions or perform further analyses of the results. our lab was tasked with reproducing one of the human evaluations reported in @xcite , which aims to identify supporting and contradictory facts in text grounded on tabular data foot_0 , namely basketball and baseball reports generated from game statistics tables. here, we explore two research questions: (1) whether we can reproduce a human evaluation study on amazon mechanical turk (amt) following as closely as possible the original design as presented by the authors of the study; and (2) explore the impact of using local evaluators instead of amt evaluators for this task. our contributions are as follows: the rest of the paper is shaped as follows: section 2 presents the original human evaluation study, section 3 presents our effort to reproduce the original study, section 4 discusses our results in comparison to the original study, and section 5 discusses an additional study we performed with local evaluators, and finally, section 6 discusses the results and implications for reproducibility studies in nlg. ## original study the original study selected for our reproduction is "data-to-text generation with macro planning" by @xcite which proposed a new macro planning phase for data-to-text generation. this new phase aims to enhance the structure and accuracy of the generated content by emphasising higher-level content organisation, including entities, events, and their interactions. these highlevel features, termed macro plans, are learned from the provided data and are then used as inputs to guide text generation. the authors employed both automatic and human evaluations to obtain accurate assessments of their model's performance. the human evaluations conducted in the original study compared the model's performance on two datasets: mlb (mlb dataset consisting of baseball games' box line-score tables, and play-by-play tables) @xcite and rotowire (rotowire dataset consisting of nba basketball games' box and line-score boxes) @xcite . the model's performance on both datasets was compared to four different nlg systems: gold, template (template-based generators from wiseman et al. (2017) ), ed+cc (encoder-decoder with attention and copy mechanism), and ent (entitybased model) for the mlb dataset, and rbf-2020 for the rotowire dataset. the original paper reports two human evaluation studies: a fact-counting study and the quality of generated summaries. here, we reproduce the fact-counting study. in this study, human evaluators were asked to count the number of supporting and contradicting facts in the outputs of the nlg systems by comparing them with the input data. for the mlb dataset, the input consisted of a baseball game box, line-score, and play-by-play tables, while for the rotowire dataset, participants were provided with an nba basketball game box-and line-score tables. the fact-counting study involved a total of 600 evaluations or human intelligence tasks (hits) that required human evaluators. to facilitate these evaluations, the authors utilised amazon mechanical turk (mturk) to crowdsource the completion of the hits. specific qualifications were set for workers to be eligible to participate, including having an mturk approval rating greater than 98%, a minimum of 1000 previously completed hits on mturk, and being based in one of the following english-speaking countries: us, uk, canada, ireland, australia, or new zealand. in the original study, the authors reported that human evaluators were not required to have prior knowledge of basketball or baseball, as they were provided with a cheatsheet explaining the semantics of the box score tables. each summary was evaluated by three different workers, and there were a total of 131 distinct mturk workers involved in the evaluations. the 600 hits were divided into eight mini-batches (four per dataset), and attention checks were employed to ensure the quality of the responses. if a worker reported more than 20 total facts, their response was rejected and rerun. the agreement among the three responses for each distinct hit was calculated using krippendorff's alpha, resulting in 0.44 for supported facts and 0.42 for contradicting facts. ## reproduced study in the reproduced study, we followed the design and methodology of the original study as closely as possible, however, we only reproduced the tasks for the rotowire dataset due to the less complex task and cheat-sheets provided, which allowed for better control over the cognitive complexity of the hits. this decision aligned with the recommendations by @xcite in controlling factors such as the number of evaluators, the cognitive complexity of the task, and the level of training/expertise of the evaluators. by narrowing down these factors, our goal was to improve the accuracy and effectiveness of reproducing the results. we obtained the necessary model outputs and human evaluation datasheet (heds) from the original authors and filled out our own heds for reproduction. then using the amazon mechanical turk (mturk) platform, we set up hits for the factcounting evaluation. workers were asked to count the supporting and contradicting facts in summaries generated by different nlg systems, using the exact same ui and cheatsheet as the original study. we conducted multiple mini-batches of hits with attention checks in between. a total of 167 distinct workers participated in the study, and we ran 300 hits divided into 4 batches (agreement using krippendorff's α was -0.12 for supported and 0.12 for contradicting facts, as opposed to 0.44 and 0.42 respectively in the original study). we reran several batches where workers had failed the attention checks outlined in the original paper, resulting in 121 failed attention check tasks. including paying for reruns, the total cost for the study came to $665.18. workers received compensation at uk's living wage foot_1 level. to analyse the results, we mirrored the original study by using the exact same code for statistical analysis using a one-way anova test with posthoc tukey hsd test. this analysis helped us identify significant differences in the performance of the nlg systems for comparison with the proposed macro system. ## results in this section, we compare the results from the fact-counting hit on the rotowire dataset for both the original and reproduced studies (see in the original study, the template-based generator (template) showed a statistically significant higher number of reported supporting facts (7.57) when compared to the macro system (4.00). simi-larly, the rbf-2020 system showed a statistically significant increase in supporting facts (5.08). in terms of contradicting facts, ed+cc and rbf-2020 reported higher numbers, with statistical significance underlined at 0.91 and 0.67 respectively. the reproduced study results demonstrated a different pattern. the template system once again recorded a statistically significant higher number of supporting facts (when compared to the macro system) at 6.3167, but the differences in the contradicting facts were less pronounced across the systems, without statistical significance against macro. ## additional study with local evaluators in addition to the above reproduction study, we conducted a supplementary study on a smaller scale with a selected pool of academic evaluators. this study aimed to provide further insights into the reproducibility of human evaluations as well as the impact of sampling participants with different characteristics. to carry out this additional study, we adapted the html task interface to work locally without relying on the mturk platform. the interface was modified to allow participants to save their answers to a json file, which they would then email back to us. for each of the five nlg systems, two tasks were randomly selected from both the rotowire and mlb datasets (total of 20 hits). unlike the large-scale reproduction, each task in the additional study was completed by two distinct participants instead of three. to gather respondents for the study, we allocated the individual hit html files to participants and asked them to submit the json file once they completed the task(s). the participants were instructed to count the number of supporting and contradicting facts in the summary, following the same approach as the original and large-scale reproduction studies. a total of 17 distinct evaluators, including 4 non-native english speakers (who however live and work/study in the uk), participated in the additional study (agreement using krippendorff's α was 0.65 for supported and 0.56 for contradicting facts). we ran 40 hits in total, with each task having two participants. there were no failed attention checks in this study. the collected responses from the additional study were analysed using the same statistical analysis python script used for the original and large-scale reproduction studies. this allowed us to compare the performance of the different nlg systems in the additional study as well. ## discussion & conclusions the attempt to reproduce the results of the original study yielded mixed outcomes, with substantial differences observed in the reproduction studies. while the original study showcased certain statistical significance in the reported performance of the systems against the macro system, this significance was often lost in reproduction studies, particularly concerning the number of contradicting facts. the full, large-scale reproduction exhibited a noticeable increase in the number of contradicting facts across various systems, and the alignment between the original and reproduced studies was limited. strikingly, the local study displayed more consistency with the original study but also brought forth its unique variations. as expected the local study resulted in higher annotator agreement than the amt study. across different nlg systems, there was a clear fluctuation in the number of reported supporting and contradicting facts. this variation, although intriguing, added to the complexity of drawing definitive conclusions regarding the reproducibility of human evaluations in these contexts.