The 2023 R epro NLP Shared Task on Reproducibility of Evaluations in NLP : Overview and Results (2023)

Abstract: This paper presents an overview of, and the results from, the 2023 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP’23), following on from two previous shared tasks on reproducibility of evaluations in NLG, ReproGen’21 and ReproGen’22. This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, all against a background of an interest in reproducibility that con- tinues to grow in the two fields. This paper describes the ReproNLP’23 shared task, summarises results from the reproduction studies submitted, and provides comparative analysis of the results.

## introduction reproducibility continues to be a topic dividing and troubling the natural language processing (nlp) community @xcite . despite a growing body of work on the topic, we still do not understand well enough what makes evaluations easier or harder to reproduce, and reproduction studies often reveal alarmingly low degrees of reproducibility not only for human evaluations but also for automatically computed metrics @xcite . with this fourth reproduction-focused shared task in nlp, following reprolang'20 @xcite , reprogen'21 @xcite and reprogen'22 @xcite , our aim is to continue to add to the body of reproduction studies in nlp and machine learning (ml) in order to increase the data points available for investigating reproducibility, and to begin to identify properties of evaluations that are associated with better reproducibility. we start in section 2 with a description of the organisation and structure of the shared task, fol-lowed by details of track c and the participating teams (section 3). next, we present per-experiment results for each experiment in track c, in terms of the reproduction task, degree of reproducibility assessments, and confirmation of findings (section 4). we next look at the quality criteria assessed by evaluations and the properties of the repronlp evaluation studies in standardised terms as facilitated by heds datasheets, and explore if any properties appear to have an effect on degree of reproducibility (section 5). we conclude with some discussion (section 6) and a look to future work (section 7). ## repronlp 2023 repronlp 2023 foot_0 consisted of three tracks. tracks a and b were identical to the tracks in predecessor event reprogen 2022: track a a shared task in which teams try to reproduce the same previous evaluation results, track b an 'unshared task' in which teams attempt to reproduce their own previous evaluation results. track c forms part of the reprohum project foot_1 and the studies reproduced in it were selected according to criteria of suitability and balance to form part of a larger coordinated multi-lab multi-test reproduction study, as described in detail elsewhere @xcite . the three tracks in overview were as follows: a main reproducibility track: for a shared set of selected evaluation studies, participants repeat one or more studies, and attempt to reproduce the results, using published information plus additional information and resources provided by the authors, and making commonsense assumptions where information is still incomplete. b ryo track: reproduce your own previous evaluation results, and report what happened. unshared task. c reprohum track: for one or more of the set of papers selected for reprohum round 0, and for the specific experiments selected only, repeat one or more studies, and attempt to reproduce the results, using information provided by the repronlp organisers only. there were no submissions for tracks a and b this year. for the reprohum track (c), the specific experiments that are listed and described below were the subject of two reproduction studies each in the reprohum project, and were also open to repronlp'23 participants. the original authors agreed to us using their experiments in the repro-hum project as well as in repronlp, and provided very detailed information about the experiments. the experiments, with many thanks to the authors for supporting reprohum and repronlp, are: for track c, the reprohum project team gathered all code and other resources needed for repeating the study, and acted as a go-between in those cases were there were additional questions from the reproducing teams; this was to avoid using more of the original authors' time than was absolutely necessary. authors of reproduction papers were also asked to complete a heds datasheet. foot_2 @xcite . we issued a call for participation in one or more tracks, and made available broad guidelines 4 to participating teams about how to report reproduction results, and provided light-touch review with comments and feedback on papers. in addition, for track c, the reprohum team and partners agreed a common approach to reproduction which repro-hum participants were expected to follow. ## per-experiment results by design, each of the six experiments in track c was repeated by two reprohum partner labs, and in this section we take a look at how results achieved in the two repeat experiments compare to each other and to results from the original experiment, for each of the six experiments. ## results by quality criterion the evaluation from chakrabarty et al. (2022) has the next best degrees of reproducibility, mean cv* for plausibility after idiom and plausibility after simile both being medium (in the 15-20 range). the assessments of grammaticality, coherence and conciseness for the experiment from puduppully and lapata (2021) (a) have slightly worse reproducibility at just above 20 for all three criteria. finally, the second experiment from puduppully and lapata (2021) (b) has good reproducibility for the mean number of facts supported by the input (#supp), but the worst reproducibility by far for the mean number of facts contradicted by the input (#cont). for comparison, in the reprogen'22 studies, annotation-based evaluation (4.3.8=anno) was clearly associated with lower reproducibility. evaluations which involved assessment of content alone (4.1.2=cont) also tended to have worse reproducibility. assessing evaluation items relative to a system input (4.1.3=rti) was also associated with lower reproducibility for three of the studies (where comparison of outputs to inputs was far more complex than a straightforward is-it-simpler decision as in e.g. @xcite . finally, correctness assessment (4.1.1=corr) was also associated with lower reproducibility. for those of these properties that were present in reprogen'21, the tendencies were the same. ## discussion in terms of general tendencies found in repronlp reproductions, there were quite a few issues (see notable issues sections above) that made carrying out a repeat experiment difficult. these were discussed in detail in a previous paper @xcite . in some cases, there were striking differences between the two paired reproduction studies: for example, repro 2 for @xcite achieved much closer results to the original study than repro 1 in terms of both pairwise cv* and pearson's, and while repro 1 for (puduppully and lapata, 2021) (a) achieved very similar results to the original study, repro 2 results had very little in common with either the original study or repro 1. this very clearly highlights the importance of carrying out more than one reproduction study to get a rounded picture of an evaluation's degree of reproducibility. none of the reproductions produced the same system ranks for all quality criteria evaluated, although in some cases it was close. given that sys-tem ranks are the single most important result from the above types of evaluations, this is concerning. in terms of patterns emerging about what properties make an evaluation more or less reproducible, we can glean two tendencies from the properties examined in ## conclusions our intention in track c had been to create a situation where we would have more than one reproduction of the same original study to analyse, in order to obtain truer estimates of the original study's reproducibility. moreover, all three studies were supposed to be identical for as close as possible to ideal comparability. two main problems arose: (a) the flaws, errors and bugs reported previously @xcite were in some cases fixed differently by reproducing authors, leading to different raw results; (b) reproducing authors in some cases chose different results to reproduce and compare, resulting in non-comparability; and (c) reproducing authors did not always manage to stick as closely as we had intended to the original experimental details, e.g. using different interfaces, revealing that the experiment was a reproduction, and most significantly, using very different numbers of evaluators. the latter is particularly significant, because it appears to be associated with worse reproducibility (see preceding section). our next step will be to fully standardise analysis and other scripts, and ask reproducing authors to both provide the same fully standardised set of results (something we did not have time for within the repronlp schedule). this will then provide the basis for more detailed analysis to be carried out and reported in future work. we will also run another round of paired reproductions in the reprohum project, using a differ-ent set of experiments for which we have corrected any issues prior to sharing them with the reproducing partners and where we are relaxing the strictrepetition requirement somewhat. we will again open up reproductions to any additional reproducing teams in repronlp 2024.