Human Evaluation Reproduction Report for Data-to-text Generation with Macro Planning (2023)

Abstract: This paper presents a partial reproduction study of Data-to-text Generation with Macro Planning by Puduppully et al. (2021). This work was conducted as part of the ReproHum project, a multi-lab effort to reproduce the results of NLP papers incorporating human evaluations. We follow the same instructions provided by the authors and the ReproHum team to the best of our abilities. We collect preference ratings for the following evaluation criteria in order: conciseness, coherence, and grammaticality. Our results are highly correlated with the original experiment. Nonetheless, we believe the presented results are insufficent to conclude that the Macro system proposed and developed by the original paper is superior compared to other systems. We suspect combining our results with the three other reproductions of this paper through the ReproHum project will paint a clearer picture. Overall, we hope that our work is a step towards a more transparent and reproducible research landscape.

## introduction recent efforts have advanced the quality of automatic evaluation metrics, but these metrics still suffer from many shortcomings and flaws (e.g., a lack of correlation between scores and human judgments, such as that reported by @xcite , @xcite , @xcite , @xcite , @xcite , and van der @xcite , among others) that render reliance on them less than ideal. human evaluation eliminates most of these concerns, making it central to evaluating many machine learning, and in particular natural language processing (nlp), approaches. nevertheless, evaluating the quality of algorithms and models using human raters still raises several unique challenges that can discourage researchers from doing so. for example, one prohibiting factor is cost: while automated metrics can be used repeatedly, essentially free of charge, human evaluations require the recruitment of paid raters with appropriate background knowledge or skillsets. the costs associated with this often force researchers only to evaluate a limited number of samples when conducting human evaluations, using crowd-sourcing platforms such as amazon mechanical turk (amt). 1 the use of crowd-sourcing platforms as a primary vehicle for subject recruitment can raise its own issues, as has been extensively documented by others even outside of the nlp research community @xcite @xcite . there have been many efforts to understand and mitigate the risks associated with human evaluation. common practices include measuring interannotator agreement, calculating the power laws to select an appropriate sample size, and using statistical tests to measure the significance of the results @xcite @xcite @xcite . undoubtedly, these practices further boost confidence in the results of human evaluation. however, they focus on pre-and post-analysis without providing insight into the human evaluation process. the lack of a systematic process for human evaluation has become a major concern in the last few years @xcite . therefore, one may suggest that efforts to document and evaluate the human evaluation process are the next logical step to further improve the quality of human evaluation results without introducing any additional cost. this increased transparency and scrutiny is aligned with the goals of open science, will improve reproducibility, and will help the community to conduct higher-quality research. from a broader perspective, concerns regarding scientific reproducibility are not new. in fact, the term reproducibility crisis has been used to describe the widespread barriers and inattention to reproducibility in many scientific fields @xcite @xcite @xcite . with the increasing prominence of supervised machine learning methods that rely on empirical evidence in contemporary research, the importance of having reproducible results has become more important than ever. a global movement to promote increased reproducibility standards is gaining momentum (un-esco, 2021), with the united nations educational, scientific and cultural organization (unesco) taking a prominent role by underlining the value of open science with increased scrutiny and reproducibility as one of its main pillars. ultimately, we can address reproducibility concerns by actively and systematically analyzing the current state of affairs, finding flaws, and proposing solutions @xcite @xcite @xcite . over the last few years, many researchers have attempted to address the reproducibility crisis in nlp, often through meta-analyses and reproducibility studies of papers using automated metrics @xcite @xcite . much less attention has been given to reproducibility studies of papers using human evaluations, mainly due to the additional complications of doing so @xcite . the reprohum project aims to address this by conducting a large-scale, multi-lab reproducibility study of 50+ nlp papers incorporating human evaluations. as a participating lab in the reprohum project, we were assigned a human evaluation experiment from data-to-text generation with macro planning by @xcite . in this paper, we present our attempt to reproduce the results of that experiment. thanks to the efforts of @xcite and the organizers of reprohum, we were able to access most of the information required to reproduce our assigned experiment. ## background reproduction approaches were standardized across the reprohum project, as summarized in this section ( §2.1). we also present relevant evaluation details from the paper itself ( §2.2), and we provide additional information from the paper's authors that was not included in the original paper itself but was necessary for reproducing the results ( §2.3). ## methods our methods for reproducing the paper were as follows. we followed the same instructions provided by the reprohum team to the best of our abilities, even following the exact same order of evaluation criteria as the other team. specifically, we collected preference ratings for our evaluation criteria in the following order: the exclusion process based on ratings of coherence was simpler than that used for ratings of conciseness. for coherence, if a crowd worker selected the template system output as superior to the gold output they were excluded from the rest of the tasks. since we conducted our experiment after the other team assigned to this paper had finished their reproduction, workers excluded from the first team's study were also excluded from ours. workers were paid for all tasks that they completed regardless of whether they were excluded. we paid workers @xmath00.15 in the original paper. this difference was due to adjustments for inflation and local minimum wage. ## results our results are summarized in while the original study reported krippendorff's α = 0.47, ours was much worse (α = -0.011). note that the original authors calculated this coefficient using the results on both datasets; however, we computed our results using half the number of responses they used. the feedback we received from the crowd workers was positive. we can observe from the results that the magnitude of difference reported between conditions in the original study's results is much higher than ours. for example, when evaluating grammaticality, the original study reports a best-worst scaling (bws) score of -61.67 for the template system (the lowest score reported among all conditions), while ours is -23.33 (the lowest score reported among all conditions in our reproduction). similarly, for coherence, our bws score of 12.50 is much smaller than the reported @xmath1.25. we utilized the same statistical significance test as the original study (a one-way anova with post-hoc tukey hsd tests). the results of this test suggest that only two conditions (the template system's scores for grammar and coherence) yield results with statistically significant differences from the macro system. this is a different finding from the original study, which reported statistically significant different results for four measures. these measures were templ for grammar, coherence, and conciseness, and gold for coherence. in our analyses of the observed errors, we found a high level of similarity between the original experiment and our reproduction. we used pearson's r and spearman's ρ to measure the correlation between the two experiments. with pearson's @xmath2.90 and spearman's ρ = 0.83, we can conclude that the outcomes from the two experiments are highly correlated. in other words, in spite of the differences explained and observed between the two studies, our results do not invalidate the original study's findings. ## discussion to discuss the implications of our findings, we first reiterate the contributions of the original study and the scope of our reproduction. @xcite presented a novel technique with the goal of improving the quality of data-to-text generation. they used a combination of automatic and human evaluation methods to show that their approach was superior to existing state-of-the-art models on two datasets, rotowire and mlb. the scope of our reproduction was limited to the second human evaluation task reported in their paper, examining the quality of generated text based on coherence, grammaticality, and conciseness. furthermore, we only reproduced the results on the rotowire dataset. to provide a better perspective, mlb dataset, is larger (nearly ten times as many tokens) than the rotowire dataset. hence, we cannot form conclusive judgments based on a full reproduction of this experiment; rather, we focus on a subset of it. thus, our outcomes are currently inconclusive but promising, with evidence of a high level of similarity between our findings and those originally reported. through our focus on the results that are available, we do not believe that there is enough evidence to claim that the macro system proposed and developed by the original paper is superior compared to other systems. however, we believe that combining our results with the three other reproductions of this paper through the reprohum project will paint a clearer picture. therefore, we leave the final judgment to the reprohum team. regarding the reproduction process itself, we found that many details required to successfully reproduce the original work were missing from the paper. we believe that this is likely due to many factors associated with the current nlp research climate, including an overemphasis on novelty, formatting, and paper length, that are all beyond the original authors' control. thanks to the cooperation of the authors, we were able to find answers to the most important questions. we underscore that this level of communication is hard to find. unfortunately, there are still little to no guidelines regarding the long-term support of research artifacts and files once studies have been published. it is hard to imagine the contemporary machine learning and natural language processing research landscapes without empirical studies driving them forward. at the same time, perhaps conferences and journals should consider potential avenues for collecting technical details beyond what has been made available in the paper itself. another option is to further encourage the publication of reproduction studies in primary publication venues. ## conclusion in this work, we have presented our attempt to reproduce the human evaluation of one experiment from data-to-text generation with macro planning by @xcite . overall, with pearson's @xmath3.90 and spearman's ρ = 0.83 when comparing outcomes of the original study and our reproduction, we can conclude that when reproducing the experiment as described in the paper we observe highly correlated results. nonetheless, we believe that without the help and cooperation of the original authors, we might have observed a different outcome. we note that the reproduced results in this work are only a portion of the results presented in the original paper. therefore, concluding that the claims made by the original study are valid at this point would be premature. we leave the final judgment to the reprohum team.