The 2022 R epro G en Shared Task on Reproducibility of Evaluations in NLG : Overview and Results (2022)

Abstract: Against a background of growing interest in reproducibility in NLP and ML, and as part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the second shared task on reproducibility of evaluations in NLG, ReproGen 2022. This paper describes the shared task, summarises results from the reproduction studies submitted, and provides further comparative analysis of the results. Out of six initial team registrations, we received submissions from five teams. Meta-analysis of the five reproduction studies revealed varying degrees of reproducibility, and allowed further tentative conclusions about what types of evaluation tend to have better reproducibility.

## introduction interest in reproducibility continues to grow across natural language processing (nlp). 1 however, we still do not understand well enough what makes evaluations easier or harder to reproduce, and reproduction studies often reveal alarmingly low degrees of reproducibility not only for human evaluations but also for automatically computed metrics. with the reprogen shared task on reproducibility of evaluations in nlg, our aim is to add to the body of reproduction studies in order to increase the data points available for investigating reproducibility, and to begin to identify properties of evaluations that are associated with better reproducibility. we start in section 2 by describing the organisation and structure of the shared task, followed 1 see our systematic review of reproducibility research in nlp carried out in part as background research for reprogen @xcite . by an overview of the participating teams (section 3). next, we present high-level degree-ofreproducibility results for each reproduction study, and in the case of the more complex studies, also for subsets of results (section 4). we look at the properties of the reprogen evaluation studies in standardised terms as facilitated by the heds sheets completed by participants, and explore if any properties appear to have an effect on degree of reproducibility (section 5). we conclude with some discussion (section 6) and a look to future work (section 7). ## reprogen 2022 like its predecessor, reprogen 2022 foot_0 had two tracks, one a shared task in which teams try to reproduce the same previous evaluation results, the other an 'unshared task' in which teams attempt to reproduce their own previous evaluation results: a main reproducibility track: for a shared set of selected evaluation studies, participants repeat one or more studies, and attempt to reproduce the results, using published information plus additional information and resources provided by the authors, and making commonsense assumptions where information is still incomplete. b ryo track: reproduce your own previous evaluation results, and report what happened. unshared task. for the main track (a above), we used the same papers as in reprogen 2021, with the addition of one paper @xcite previously used in the reprolang 2020 shared task @xcite , all with consent and confirmation of willingness to support from the authors: ## participants and submissions five submissions were received by the deadline on june 6, 2022. one submission was from the netherlands, one from the uk, one from the us, one from ireland, and one was a collaboration between groups in czechia, brazil and ireland. three of the teams participated in track a @xcite @xcite ; the other two in track b @xcite . two of the submissions reported a reproduction study of @xcite , one of @xcite , and two reproduced own earlier work. all of the evaluated systems produced outputs in english. note that all evaluation scales need to be shifted to start at zero, to ensure fair comparison across evaluations, because both percentage change and cv in general underestimate variation for scales with a lower end greater than 0. rather than standard cv, qra uses cv * , a de-biased version of cv @xcite , because sample size (number of repeat measures) tends to be very small in nlp. 5 for the simpler reproductions in we have ordered the studies by study-level mean cv * (lowest, i.e. best, first). study-level mean cv * ranges from the perfectly reproduced metric scores in @xcite 's first reproduction, to the particularly high cv * of thomson and reiter (2022)'s second reproduction of an error annotation. in the case of the former, the authors managed to obtain the exact same sari and bleu scores, by running the scripts for these metrics provided by the original authors on the system outputs also provided by the original authors. @xcite 's reproductions involve error-type labelling of system outputs which appear to be a particularly difficult to reproduce form of evaluation: this was the reproduction target in the four studies in the lower half of under these circumstances, cv * expresses how reproducible (stable) the inter-rater consistency/agreement is from one experiment to a repetition of it, in other words whether inter-rater consistency/agreement is similarly high, or similarly low, across multiple repeats of the same evaluation. because @xcite repeated the evaluations for four different rating instruments, the mean cv * figures can tell us whether this differs for different rating instruments (as well as for different evaluation criteria and inter-rater consistency/agreement measures). the answer is that it does differ substantially for different rating scales, is equally low for both evaluation criteria, and does differ for the two inter-rater measures. taking a slightly closer look, the inter-rater measures (iccs) for the likert scale have remarkably higher (worse) mean cv * than the other three scales, while nevertheless achieving strong pearson's and spearman's between individual icc scores in the original and reproduction studies. while the iccs for the other three scales have similarly good cv * , only the bws scale also has strong pearson's and spearman's, with bme having no correlation and rme having medium-strength negative correlation. this shows that cv * and pearson's and spearman's correlation coefficients provide complementary information in assessments of the similarity of original vs. reproduction scores. looking at these in combination, it would seem that the bws scale (best-to-worst ranking) achieves the most similar levels of inter-rater agreement and consistency across repeat studies. from the results for huidrom et al. ( 2022 )'s reproductions, we can see that the error annotations produced for outputs for webnlg data have worse cv * figures than for e2e (the difference is not just the data but also a subset of the error categories which are tailored to the data). here, better cv * is aligned with better correlations. finally, the results for thomson and reiter (2022)'s reproductions show that the hierarchical encoder based data-to-text system produced outputs for which both mean cv * and correlations were worse on average than for the other two systems. however, this latter observation ought to be read with the proviso that each reproduction used a different sample from the three systems. ## comparison of properties of original vs. reproduction studies overall, all teams tried to follow the original studies as closely as possible (see also discussion section below), but cohorts of human evaluators involved were different across all pairs of original and reproduction studies, except for the two reproductions by @xcite , and one of the two by @xcite . in this section, we summarise differences in each pair of studies and highlight the possible factors that might have led to different results in reproduction results. in the case of track a contributions, our notes are based on the heds datasheets completed by both the original study authors and the shared task participants. for track b, we describe differences as reported by the authors themselves in their original and reproduction reports, also consulting the heds sheets completed by them. see also ## discussion in metrological terms, a repeatability assessment keeps all conditions under which a measurement was taken the same, whereas a reproducibility assessment varies some of them. strictly speaking, only the first reproduction by @xcite can be considered a repeatability assessment, as it keeps all conditions exactly the same. all other re-progen'22 reproductions were human evaluations, and for these, conditions can only be the same if the same evaluators are used again. one of the studies (the first reproduction by @xcite ) did use the same evaluators, but instructions were written down and used for the first time instead of evaluators conferring. nevertheless, all studies tried to keep things as much as possible the same. one study which looked at automatic metrics (only) @xcite ) went beyond reusing system outputs provided by original authors, (a) regenerating outputs with unchanged author-provided code, and (b) regenerating outputs with a retrained system, including with a substantial correction to the code. interestingly, evaluation results were very similar in all versions where outputs were regenerated, including switching word2vec embeddings on/off. for the studies looking at human evaluations, new cohorts of evaluators were rarely able to achieve low cv * scores, generally only in very simple assessments. pearson and spearman correlations were generally better, with some exceptions where comparison was between inter-rater similarity measures, rather than evaluation scores @xcite . we saw that correlation coefficients and mean cv * often but not always give the same indication of similarity between a set of original and reproduction scores. for example, the results in this year we had a few (new) firsts at reprogen, in addition to automatic metrics being reproduced for the first time: e.g. @xcite investigated the effect of swapping out the data sample (from the same superset), while keeping all other conditions the same including annotators. as the sample size is fairly small, and differed in size between original study and the two reproductions, it's perhaps not surprising that error label counts varied substantially between studies. some of the reprogen'22 participants' reports mention less than ideal support from original authors during reproductions, despite the fact that all original authors had agreed to support and help with reprogen'22 reproductions. of course, such help is essential to testing reproducibility, and in future shared tasks, we will consider the option of obtaining more of the resources and information prior to the start of the shared task. ## conclusions we first proposed the reprogen shared task at generation challenges 2020 foot_3 @xcite and, taking into account feedback received, developed it in a research culture that prizes leaderboard success, it was always going to be difficult to incentivise people to carry out tasks that are basically just good scientific hygiene, but we hope we have made a contribution to raising awareness of the importance of having reproducible evaluations, and of testing our methods for reproducibility. after all, how else are we going to know for sure that one approach is better than another.