R epro G en: Proposal for a Shared Task on Reproducibility of Human Evaluations in NLG (2020)

Abstract: Across NLP, a growing body of work is looking at the issue of reproducibility. However, replicability of human evaluation experiments and reproducibility of their results is currently under-addressed, and this is of particular concern for NLG where human evaluations are the norm. This paper outlines our ideas for a shared task on reproducibility of human evaluations in NLG which aims (i) to shed light on the extent to which past NLG evaluations are replicable and reproducible, and (ii) to draw conclusions regarding how evaluations can be designed and reported to increase replicability and reproducibility. If the task is run over several years, we hope to be able to document an overall increase in levels of replicability and reproducibility over time.

## introduction human evaluations play a central role in natural language generation (nlg) @xcite , so it is of concern that we do not currently know to what extent their results are reproducible, hence whether they are reliable or not. reproducibility is on many nlp researchers' minds at present. there have been workshops on replicability and reproducibility in nlp most years since 2015. 1 the reproducibility challenge has been running since 2018, initially in conjunction with iclr'18 and iclr'19 @xcite , then at neurips'19 @xcite and neurips'20 (to appear). coling'18 had a reproduction paper special category, for which it reported 35 submissions. neurips'19 had a reproducibility programme comprising a code submission policy, a reproducibility challenge for machine learning (ml) results, and the ml reproducibility checklist for submitted papers @xcite which has also been adopted by emnlp'20 and aaai'21. lrec'20 ran a reproducibility track @xcite . other conferences have foregrounded reproducibility via calls, chairs' blogs, special themes and social media posts. all this is against the wider background of what has been called a 'reproducibility crisis' @xcite in science, where 70% of scientists report failing to reproduce someone else's results on at least one occasion, and over half report failing to reproduce own results. in nlp, 24.9% of attempts to reproduce own results, and 56.7% of attempts to reproduce another team's results, fail to reach the same conclusions @xcite . progress is being made in nlp regarding reproducibility, as can be seen from the long list of events and initiatives above. the habit of sharing code, data and supplementary material providing details about data, systems and training regimes is firmly established in the field, virtually all main events now encouraging and making space for it. moreover, reproducibility is beginning to be addressed formally in the reviewing process, e.g. emnlp'20 followed neurips'19 in adding the ml reproducibility checklist @xcite to submission forms, foot_1 where authors had to indicate compliance with reproducibility criteria (although this was not used in selection decisions). while progress is being made on many fronts, there is one big gap in efforts to achieve greater reproducibility, and that concerns human evaluation. if a paper complies with all of the neurips'19/emnlp'20 reproducibility criteria, it should be possible to reproduce metric results reported in it closely. however, any human evaluation results reported for the same system(s) in the same paper may or may not be reproducible, because the criteria say nothing at all about those. this shared task proposal is part of a wider effort to address reproducibility of human evaluations in nlg, a field in which they play a central role and which has always been wary of automatic evaluation metrics and their limitations @xcite @xcite . below we start by briefly diagnosing the reproducibility problem in nlg (section 2), and looking at what the conditions are for reproducibility testing of results from human evaluation of nlg systems (section 3). we then outline our ideas for a shared task that could help to understand and potentially address the problem (section 4). we describe related research that has provided inspiration (section 5), and conclude with next steps (section 6). ## issues with human evaluation in nlg in an era dominated by metrics and leaderboards, human evaluation can be an afterthought, often carried out and reported in a slapdash way with tiny numbers of evaluators and included to add a veneer of credibility. data collected for a recent survey of 20 years of human evaluation in nlg @xcite indicates that 33% of evaluations use fewer than 10 evaluators, and 22% use between 1 and 4, numbers so small that experiments are unlikely to yield meaningful results, or to be reproducible, in many contexts. the survey also revealed that the roughly 170 papers reviewed often provide woefully inadequate information about human evaluations: numbers of evaluators, experimental design, the quality criterion assessed, even system language, are often unclear. researchers moreover use a wide variety of different quality criteria, with a startling 200-odd different quality criterion names found in the survey. even when researchers do use the same criterion name they often don't use it with the same meaning, and vice versa (see also @xcite . inter-rater and intra-rater agreement, two indicators of human evaluation reliability, are rarely reported. @xcite surveyed 135 nlg papers from 2008-2018 and found that just 18% reported any annotator agreement, and where agreement scores were reported they were low, casting doubt on the reliability of results. we're aware of one published attempt to reproduce someone else's human evaluation results in nlg: @xcite , as part of re-prolang, successfully reproduced system rankings in text simplification (however, with lower means). another paper tested the ability of evaluators to reproduce their own evaluations in exactly the same experimental set-up which found that for some tasks and some evaluation instruments, evaluators struggled to reproduce their own evaluations @xcite . nlg has valued and trusted human evaluation perhaps more than any other nlp subfield, so it's concerning that we currently don't know if any given set of human evaluation results are reproducible, meaning we don't know whether we can, in fact, trust them. ## testing human evaluations for reproducibility for results from human evaluations to be deemed reproducible, the first condition is that the system outputs assessed in the human evaluation must be reproducible, and the neurips'19/emnlp'20 criteria provide a pretty comprehensive set of conditions for that to be achievable. we take this as our starting point, i.e. we assume we have been successful in reproducing system outputs. in the shared task we will not try to reproduce system outputs, but start from existing sets of outputs (and inputs, where applicable), and try to reproduce the results of human evaluations performed on them. ## organisation of shared task we envisage reprogen to have two tracks, one an 'unshared task' in which teams attempt to reproduce their own prior human evaluation results, the other a shared task in which teams try to reproduce the same prior human evaluation results. we envisage a fairly simple challenge first, where we nominate about five papers as replication targets for participants to choose from. attendees can then either (a) replicate one of these experiments, or (b) replicate one of their own previous experiments: a main reproducibility track: for a shared set of selected human evaluations, participants attempt to reproduce their results, using published information plus extra detail provided by the authors (discussion with and support from the original authors e.g. played a big role in the reproducibility @neurips'19 challenge for ml results @xcite )), and making common-sense assumptions where information is still incomplete. b ryo track: reproduce your own previous human evaluation results, and report what happened. unshared task. for the main track (a above), the plan is to ask authors to volunteer their papers for inclusion via an open call for expressions of interest. authors will be invited to provide additional details and/or software to help teams reproduce the results. it's not clear how much this will help: while some studies indicate large increases from author help @xcite , other large-scale studies show no improvement at all @xcite . neither of the tracks would have winners or leaderboards in the normal shared-task sense. however, 'winners' in the main track would provide toplines of reproducibility for the included papers, and taken together, reprogen challenge contributions would help shed light on how to improve the reproducibility of human evaluations in nlg. we expect teams to participate for a variety of reasons, ranging from researchers new to human evaluation (the reproducibility challenge @neurips for ml results encouraged computer science courses to get their students to participate), to researchers experienced in human evaluation specifically interested in reproducibility. participation will have financial implications in the case of some papers. we will endeavour to keep such cost as low as possible by selecting mostly papers that were not expert or crowd-evaluated. we are also applying for funding to support crowdbased evaluations. ## related work reproducibility investigations are commonly conducted in a closed project. for example, the open science collaboration (2015) conducted 100 attempts to reproduce studies in psychology, mainly evaluating reproducibility using significance, pvalues, effect sizes, and meta-analysis of effect sizes. they found substantial decreases from original to replication study for all three indicators, while just 39% of effects were subjectively rated to have reproduced the original result. in contrast, the shared task framework ensures openness to all members of a research community. the reproducibility challenge @neurips'19, focusing on ml results and metric scores, was organised as a 'live' challenge, where participants pick one of the neurips accepted papers, and try to reproduce its ml results @xcite . neurips'19 authors were strongly encouraged to submit code and data, which 73% did, resulting in a 'codebase' the reproducibility challenge participants could choose from to participate in one of three tracks: (i) a baselines track (rigorous analysis of baseline results, re-implementing them if necessary), (ii) an ablation track (rigorous ablation experiments, modifying model and hyperparameters using the authors' code), and (iii) a replications track (replication of experiments in paper from scratch without using code from codebase). the challenge was a big success, attracting 83 submissions after participants initially claimed 173 neurips papers. the submissions were peerreviewed as part of the neurips reviewing process which relied heavily on the openreview platform, and 10 papers were selected for publication in re-science c, an open access journal intended as a forum for replication work in computing science. the one paper on reproducing human evaluation results in nlg mentioned above @xcite ) was part of the reprolang'20 initiative which followed on from two earlier, smaller-scale foot_2 lrec workshops on reproducibility and citation, and offered a shared task @xcite which asked participants to reproduce results from one of 11 papers from different areas of nlp. while in the case of ten papers, the results up for reproduction were automatic scores, in one case they included human evaluation scores. 4 ## next steps with this shared task proposal we hope to engage the nlg community in a discussion about how best to design and organise the reprogen challenge. following feedback and input, we will finalise the task specification and organisational aspects, expecting to be able to launch the task in 2021 for a pilot run with around five sets of human evaluation results up for reproduction. we would hope that the reprogen challenge will both shed light on the reproducibility of current human evaluations in nlg, and allow conclusions about how evaluations can be designed and reported to increase reproducibility. over repeated instances of the shared task, we hope to be able to document an overall increase in the reproducibility of new human evaluations in nlg.