The 2025 R epro NLP Shared Task on Reproducibility of Evaluations in NLP : Overview and Results (2025)

Abstract: This paper presents an overview of, and the results from, the 2025 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP’25) which followed on from four previous shared tasks on reproducibility of evaluations, ReproNLP’24, ReproNLP’23, ReproGen’22 and ReproGen’21. This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of the topic across the two fields. We describe the ReproNLP’25 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results, including for the first time additional, ‘sanity-check’ evaluations by LLMs.

## introduction natural language processing (nlp) and machine learning (ml) are still far from solving the reproducibility crisis that has been well documented over recent years @xcite . authors still don't make enough resources and information available about published work to enable repetitions of it despite reproducibility checklists being introduced by conferences. 1 when reproducibility is tested, results often fail to confirm original findings @xcite @xcite . the core aim of this sixth reproduction-focused shared task in nlp, following reprolang'20 @xcite , reprogen'21 @xcite , reprogen'22 @xcite , re-pronlp'23 @xcite , and re-pronlp'24 @xcite , is to continue to add to the body of reproduction studies 1 for an example see the aaai'26 one at https://aaai.org/conference/aaai/aaai-26/ reproducibility-checklist/ . in nlp and ml, but also to produce and analyse multiple reproductions of shared original evaluations, to shed more light on how best to assess reproducibility in nlp/ml and ultimately how to improve the degree to which our findings in the field are reproducible. the eight new reproduction studies (for an overview see we start in section 2 with a description of the organisation and structure of the shared task, along with track details. next, we summarise results at the level of individual experiments, in terms of the reproduction task, and different degree-ofreproducibility assessments (section 3). we report results from llm sanity checks carried out in those cases where at least one reproduction disagreed with the original study (section 4). in section 5, we look at the quality criteria assessed in evaluations and other properties of the repronlp evaluation studies in standardised terms as facilitated by heds datasheets, and explore if any of these show signs of affecting degree of reproducibility. we conclude with some discussion (section 6) and a look to future work (section 7). ## repronlp 2025 like its predecessor, repronlp 2025 foot_0 consisted of two tracks, one an 'unshared task' in which teams repeat their own or any other previous work @xcite , the other a standard shared task in which teams re-run one of a set of experiments for which the shared-task organisers make available all necessary information and resources (track b): a open track: repeat any previously reported work developing and evaluating systems, and report the approach and outcomes. unshared task. ## track b results in this section, we report results for the eight submissions (listed in as noted above, we report type i, ii, and iv qra++ results only. this is because in most cases there are no type iii results, and in some cases where there are type iii results we do not have access to all of the raw annotations from the original studies (which would be needed in order to calculate type iii qra). ## llm sanity check results in section 3 we saw three sets of evaluations where at least two evaluations produced contradicting results: august et al., @xcite for these three we report additional llm evaluations following the general approach outlined in section 2.3, and using the specific method described in appendix c. ## reproducibility by quality criterion and other properties in this section, we look at some additional properties of our five sets of studies, to see if any pattern emerges as to which properties may be associated with better, and which with worse, reproducibility. what we are looking for in this table is any indication that one of the heds properties affects experiment-level mean cv * (last three columns). one such property is number of evaluators (heds question 3.2.1): the pattern is for larger number of evaluators @xcite to be associated with better reproducibility, a pattern also observed in previous repronlp shared tasks (see another trend that was previously observed and is also observable here is that evaluations that are more cognitively complex tend to have poorer reproducibility than cognitively simpler evaluations. an example is the evaluation of factual truth in august et al. which had the highest study-level, mean cv * of all studies reported. it also had the smallest number of evaluators. another example is meaning preservation in reif et al. which had some of the worst qra++ values, and was also the most inconclusive of our sets of studies. the two standout studies in terms of reproducibility on all measures were @xcite ing & lapata which share very similar properties as captured in ## discussion as in previous editions of repronlp, we saw that degree of reproducibility can look very different depending on which qra++ measure is applied. for example, for yao et al., the type ii measures applied (pearson's and spearman's correlations) showed excellent reproducibility, as did type iv (p , the proportion of identical pairwise ranks), but cv * was only medium (study-level mean cv * was 25.85). while we've seen this happen a few times in repronlp, the inverse, excellent study-level, mean cv * , and then terrible correlations and p , we have never seen (as one would expect). in seven of the eight studies with a cv * under 11 have a very small number of possible response values (3 or fewer). both of the two studies with the worst cv * values by a very large margin asked evaluators to count items directly. relative quality estimation (rqe) seems to have the edge over direct quality estimation (dqe): the former has an average of 16.35 cv * , the latter 23.06. in terms of combinations of properties, using a larger number of evaluators together with a small number of response values in rqe of goodness has in all seven cases resulted in a cv * of under 22, in four cases, under 15. we have three studies assessing meaning preservation: two use rqe and achieve excellent cv * ; the other one uses dqe and has poor cv * . we applied llm sanity checks for the first time in repronlp 2025 in order to shed light on which of two disagreeing studies is likely to be closer to the true picture. of the three cases where there were disagreeing studies, the llm sanity check was able to answer the question, but in the remaining case @xcite , the llm results correlated better with the original study than with r1, but cv * was worse and p was very close for both o and r1. we will return to this analysis once the missing reproduction for reif et al. is complete. ## conclusion a shared task results report is almost invariably written under pressure of time and to a deadline. there are other aspects than are reported here which we would like to have investigated, but will have to leave for future work. repronlp 2025 is the fifth and likely the last edition of this shared task series. it has contributed new data and insights into reproducibility and the factors that impact it, and we plan to release our resources and results so that further analyses can be conducted and insights gleaned. cessing, funded by epsrc (uk) under grant number ep/v05645x/1 which ended in may 2024. in particular, we thank our numerous collaborators from nlp labs across the world who carried out many of the reproductions reported in this paper as part of the second batch of coordinated reproductions resulting from the reprohum project. the repronlp work has also benefitted from the work being carried out in association with the adapt sfi centre for digital media technology which is funded by science foundation ireland through the sfi research centres programme and is co-funded under the european regional development fund (erdf) through grant 13/rc/2106. craig thomson's work was funded in part by reprohum, and in part by adapt.