The R epro G en Shared Task on Reproducibility of Human Evaluations in NLG : Overview and Results (2021)

Abstract: The NLP field has recently seen a substantial increase in work related to reproducibility of results, and more generally in recognition of the importance of having shared definitions and practices relating to evaluation. Much of the work on reproducibility has so far focused on metric scores, with reproducibility of human evaluation results receiving far less attention. As part of a research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the first shared task on reproducibility of human evaluations, ReproGen 2021. This paper describes the shared task in detail, summarises results from each of the reproduction studies submitted, and provides further comparative analysis of the results. Out of nine initial team registrations, we received submissions from four teams. Meta-analysis of the four reproduction studies revealed varying degrees of reproducibility, and allowed very tentative first conclusions about what types of evaluation tend to have better reproducibility.

## introduction there has been growing interest in reproducibility across natural language processing (nlp) over recent years. 1 however, work has mostly focused on determining what information and resources need to be shared to enable others to obtain the same metric results. the reproducibility of human evaluation has received far less attention and currently very little is known about how reproducible, hence trustworthy, the human evaluations we routinely apply in nlp really are. this is of particular concern in natural language generation (nlg) where human evaluations have always played a central role @xcite . the last few years have seen a growth in publications, projects, workshops, shared tasks and other initiatives on the topic of reproducibility. for example, neurips'19 introduced the ml reproducibility checklist for submitted papers @xcite which was also adopted by emnlp'20 and aaai'21. the reproducibility challenge has been running since 2018, initially in conjunction with iclr then neurips @xcite . the challenge is focused on ml results and metric scores, and is organised as a 'live' challenge, where participants pick one of the accepted papers, and try to reproduce its ml results @xcite . the reprolang'20 shared task @xcite asked participants to reproduce results from 11 papers in different areas of nlp. while in the case of ten of the papers, the results up for reproduction were automatic scores, in one case @xcite they included human evaluation scores. 2 in their reproduction study of this work, cooper and shardlow (2020) reannotated original system outputs using their own annotators, in order to be able to compare annotation results. their results suggested a drop in both quality metrics of close to 15%. apart from the above reproduction study involving text simplification carried out within re-prolang, there appears to have been just one other paper reporting reproduction studies of human evaluation results in nlg @xcite which re-ran two evaluation experiments with the same evaluator cohorts, one in data-to-text generation, the other in visual referring expression generation. here, strong correlations between annotator scores were found for two quality criteria for each task, 0.87 pearson's in one case, >0.94 in the other three. with the reprogen shared task, our aim was to add to this currently small body of literature, in order to shed more light on how reproducible current human evaluation methods are, and what we may need to change in how we design and carry out human evaluations in order to improve reproducibility. in section 2 we start by describing the organisation and structure of the shared task. next we provide an overview of the participating teams (section 3) and look at the properties of submitted systems (section 4). we compare and analyse the results from the submitted systems in detail (section 5), before we conclude with some discussion (section 6) and tentative conclusions (section 7). ## organisation of shared task reprogen'21 foot_3 had two tracks, one a shared task in which teams try to reproduce the same prior human evaluation results, the other an 'unshared task' in which teams attempt to reproduce their own prior human evaluation results: a main reproducibility track: for a shared set of selected human evaluation studies, participants repeat one or more studies, and attempt to reproduce the results, using published information plus additional information and resources provided by the authors, and making common-sense assumptions where information is still incomplete. b ryo track: reproduce your own previous human evaluation results, and report what happened. unshared task. for the main track (a above), we issued a call for proposals of papers, asking people to propose papers via an online form. 4 this yielded seven proposed papers, from which we selected four on the grounds of suitability for reproduction studies, diversity of languages and cost of reproduction. the selected papers and studies, with many thanks to the authors for supporting reprogen, are: we made available broad guidelines 6 to participating teams about how to report reproduction results, and provided light-touch review with comments and feedback on papers. ## overview of participants and submissions four submissions were received by the deadline on august 15, 2021. two of the submissions were from germany, one from ireland, and one was a collaboration between groups in spain, brazil and ireland. two of the teams participated in track a track team original paper reproduction paper a technical university of darmstadt (tuda) qader et al. (2018) richter et al. (2021) upf barcelona, uf minas gerais, adapt dublin van der lee et al. (2017) mille et al. (2021) b trivago gmbh, düsseldorf mahamood et al. ( 2007) mahamood (2021) adapt dublin popović (2020) popović and belz (2021) ## comparison of properties of original vs. reproduction studies overall, all teams tried to follow the original studies as closely as possible. all of the reproduction studies evaluated the same texts as reported in the original experiments, with the same criteria and measurement methods. three of the four submissions used the same number of evaluators. cohorts of human evaluators involved were different across all pairs of original and reproduction studies. below we summarise differences in each pair of studies and highlight the possible factors that might have affected reproduction results. in the case of track a contributions, our notes are based on the heds datasheets completed by both the original study authors and the shared task participants. for track b, we describe differences as reported by the authors themselves in their original and reproduction reports, also consulting the heds sheets completed by them. see also ## comparing reproducibility in the reprogen studies looking at the above cv * ranges for each reproduction study, a first indication of a ranking emerges for the four study pairs in terms of degree of reproducibility, with ( 1 in in lee et al./mille et al., clarity and fluency are compound measures each derived from two separately assessed quality criteria, which map to the normalised quality criterion names shown in rows 4-7 in looking at ## discussion there were considerable differences in evaluator cohorts between original and reproduction study in all four reprogen study pairs. in mahamood et al./mahamood, the texts being evaluated were about progress towards getting a postgraduate degree (e.g.: you haven't qualified for a postgraduate diploma. you have been awarded a postgraduate certificate instead. average cas results were achieved in cs5052, cs5038, cs5540, cs5548.) @xcite asked postgraduate students to evaluate these texts, whereas @xcite asked work colleagues to evaluate the texts. it is possible that students and non-students reacted differently to statements about degree progress, and that the students were much more familiar with terms such as 'postgraduate certificate' and 'cas'. there were also important differences in evaluator cohorts in lee et al. @xcite in both cases, the reproduction cohort included people known to the authors personally who may have had more of an incentive to perform the task conscientiously and perhaps also to select higher scores. in the case of lee et al./mille et al., reproducibility was nevertheless good, whereas for qader et al./richter et al., it was less good. across all of our reproduction studies, there were differences in evaluators: age, recruitment, professional status, domain knowledge, background, etc. such differences have the potential to impact reproducibility, but the picture from the four reprogen studies was mixed, and further research is needed to understand which characteristics were most important from this perspective. knowing this would be very helpful in designing and interpreting experiments, as well as replicating them. both track a reproduction studies contacted the original authors for additional information, highlighting the importance of original authors being willing to support reproduction studies of their work. it is clear from reprogen'21 as well as other research @xcite @xcite that we need a lot of information about evaluators and other aspects of evaluations in order to conduct reproduction studies, so it's essential that experimenters fill out a datasheet such as heds which conveys information in a standardised, comparable way. a rarely mentioned aspect that should not be underestimated is that being willing to support a reproduction study of your work means being willing to take what some perceive as a substantial risk associated with having others publish assessments of the reproducibility of your work. some authors are very uncomfortable with a reproduction study showing low reproducibility. in fact, one of the authors of a paper which had been the subject of a reproduction study that we wanted to include in our survey of reproduction studies @xcite worried that the considerable gap in results would be interpreted as academic misconduct. 11 clearly there is a need for reproduction studies to be carried out in nlp. we need to know how reproducible different types of evaluation measures are, because measures with low reproducibility will result in unreliable results and unreliable conclusions based on them. reproduction studies are the only way to know if/where we're going wrong in this sense. however, given prevailing sensitivities, it seems the right thing to do to conduct reproduction studies with the original authors' consent. reproduction studies are expensive and a lot work, and we were told by the five teams that registered for reprogen but did not submit that these were the main reasons why they were ultimately not able to participate. publication only provides so much of an incentive/motivation. significant numbers of reproduction studies may only be feasible in the context of a funded project such as reprohum, 13 where uniformity of approach can moreover be ensured and the number and type of re11 we therefore did not include the study in question in the published survey. ## conclusions we first proposed the reprogen shared task at generation challenges 2020 foot_10 @xcite and, taking into account feedback received, developed it into the shared task presented here, with the main track offering four original studies (sets of human evaluation results) for reproduction, and an open track inviting reproduction studies of own results. bearing in mind we had just one reproduction study for each original study available to us, and that as discussed we have to be cautious drawing conclusions based on sample sizes of 2, there are very few tentative first conclusions concerning reproducibility of human evaluation in nlg we have been willing to draw from reprogen. we pointed out that the study with the highest degree of reproducibility obtained a comparatively large number of scores for each evaluated item, while also assessing a medium number of items per system. in contrast, the study with the lowest degree of reproducibility obtained a different number of scores for each evaluated item in the original and reproduction studies, while assessing a very small number (2) of items per system. we also observed that there was some evidence that the grammaticality evaluation criterion has a comparatively good and stable degree of reproducibility. when we read human evaluation results in nlg papers, unless there is an obvious red flag such as a very small number of evaluators, or evaluation items, we tend to trust those results more than metric results. yet as we delve deeper into the reproducibility of our human evaluation results, it is beginning to become clear that, as a general assumption, this trust may be misplaced. more generally, that we need to do much more as a field to ensure that our human evaluation methods are fit for purpose, including in the sense that a rerun of an experiment will produce at least broadly similar results. with the reprogen shared task, and the re-prohum project 13 which it is part of, we are aiming to make a contribution to this important goal.