Evolving Stances on Reproducibility: A Longitudinal Study of NLP and ML Researchers’ Views and Experience of Reproducibility (2025)

Abstract: Over the past 10 years in NLP/ML, as in other fields of science, there has been growing interest in, and work on, reproducibility and methods for improving it. Identical experiments producing different results can be due to variation between samples of evaluation items or evaluators, but it can also be due to poor experimental practice. Both can be mitigated by bringing multiple comparable studies together in systematic reviews that can draw conclusions beyond the level of the individual studies, but such systematic reviews barely exist in NLP/ML. The alternative is to focus on improving experimental practice and study-level reproducibility, and the first step in this direction is awareness of the importance of reproducibility and knowledge of how to improve it. Here we aim to assess (i) what NLP/ML practitioners’ current views and experience of reproducibility are, and (ii) to what extent they have changed over the past two years, a period of rapidly growing interest in reproducibility. We report for the first time, results from two identical surveys, the first carried out in 2022 and the second in 2024, each time surveying 149 NLP and ML researchers. The results from the 2024 survey assess i above. We then compare the results of the two surveys in order to address ii above. We find that views and experience overall are moving towards better practice and appreciation of reproducibility.

## introduction poor reproducibility of experimental results is a point of concern not just in nlp/ml @xcite @xcite , but also in science more generally @xcite . in nlp/ml, reproducibility concerns have led to a flurry of activity aimed at diagnosing the problem @xcite @xcite @xcite @xcite @xcite , encouraging researchers to share full details and resources of their work @xcite @xcite , and understanding how differences between evaluation methods impact reproducibility @xcite @xcite . some conferences have introduced checklists and/or guidelines for reproducibility, e.g. acl (an nlp venue) and aaai (a more general ml venue) use reproducibility checklists. foot_0 , foot_1 iclr encourages authors to include a reproducibility statement, wheras icml encourages authors to "submit code to foster reproducibility." see appendix b for more details. some work has looked at whether previous interventions have had any effect on poor levels of reproducibility. @xcite describe existing interventions as a 'siren song,' pointing out that whilst many conferences have appointed reproducibility chairs, they rely mainly on self-reporting of code inclusion, clearly not the same thing as reproducibility. @xcite perform an analysis of the *cl reproducibility checklist data, albeit without answering the question of whether it is effective at improving reproducibility. for interventions like checklists to be effective, they need to result in changed understanding and practices relating to reproducibility, beyond a small increase in reviewer scores for reproducibility. what we currently do not know is whether the checked boxes and reviewer scores actually lead to improved reproducibility and understanding of related issues. the work presented here, in tandem with other research on reproducibility of nlp evaluations @xcite ; @xcite is intended to address this question. we conducted two identical surveys of researchers' views and experience of reproducibility two years apart, in 2022 and 2024, to determine (i) what current stances on reproducibility are, and (ii) how they have changed over the past two years, a period of increasing research focus on reproduciblity. we start below with a summary of related research (section 2). we describe our study design (section 3), and present the results of the 2024 survey (section 4) and an analysis of the differences between the 2024 survey and an identical survey conducted in 2022 (sections 5 and 6). we finish with some discussion and conclusions in section 7. ## related research sporadic surveys have investigated reproducibility for science in general @xcite , and nlp @xcite in particular. @xcite asked whether respondents thought there was a reproducibility crisis, as well as how severe they felt it was, both generally and within their field. they also asked about attempts to repeat experiments and reproduce results, respondents' experience publishing such work, and contributing factors and barriers that lead to poor reproduction practices. @xcite surveyed scientists in all disciplines, finding that 70% of researchers had attempted to reproduce the work of others, and failed. biologists made up 45% of responses, with only around 1.5% working broadly in computer science, including one participant in ai (none in nlp). it is therefore not representative of nlp and the issues it faces. @xcite asked about respondents' experiences when trying to repeat their own work, and that of others. questions on where respondents accessed code, data, and parameters were also included, as was a question on the outcome of attempts at contacting original authors. this survey focuses on whether similar results were obtained, where researchers were able to access resources (the authors website, @xcite , and what respondents' experiences were when contacting authors. respondents thought the issues were important, although the majority obtained different results when reproducing the work of others. the survey conducted by @xcite looked at the barriers encountered when attempting to perform an error analysis of previously published results. whilst not directly investigating reproducibility, it similarly found that whilst researchers felt error analysis to be important, there are barriers such as time, money, publication page limits, and peer-reviewer interest. ## overview of study design we designed a survey of nlp/ml researchers' views and experience of reproducibility and conducted it once in 2022, near the start of the repro-hum project, foot_2 and again in 2024, after the project concluded. for the 2024 survey, we collected responses from the same number of respondents with the same characteristics (see below) as in 2022. following survey question design principles (described in appendix c), we created our survey of 25 multiple-choice/select-one questions, and 6 multiple-choice/select-all-that-apply questions, to take about 10 minutes on average. the survey flow structure is shown in appendix figure 9 . respondents were first shown a landing page and introduction (appendix, figure 10 ). the survey itself was structured into seven sections (as reflected both in the flow chart and in ssections 4.1-4.4 which present the 2024 results): we ensured that the proportions of high-level participant characteristics in the 2024 survey were the same as in the 2022 instance: 125 nlp researchers (45 students, 80 non-students); and 24 ml researchers (8 students, 16 non-students). we first present results from the 2024 survey (section 4), then analyse the change from 2022 to ## 202survey results this section reports the results of the 2024 survey, reflecting current views in nlp/ml. we present results for multiple-choice/select-one questions as pie charts unless the possible answers form an ordinal range in which case they are presented in hor-izontal bar charts. if the responses are in the form (yes/no/unsure) they are presented in vertical bar charts. results from multiple-choice/select-all-thatapply questions are presented as vertical bar charts. all figures include the verbatim question(s) in the caption, and the set(s) of possible answers in the chart legend. to make charts accessible, we use the light colour scheme by paul tol. foot_3 ## change analysis 2022 vs. 2024 in this section, we look at the proportional differences (up or down) between the 2022 and 2024 response counts. we can't just describe the change in terms of how many more/fewer people selected different responses, because in some cases, a question was answered by a different number of people in 2022 and 2024 (due to the conditional logic of the question flow, see figure 9 , appendix). instead, we discuss change in terms of the difference in the proportion of people who selected a given answer a i,j to a question in two different years y 1 and y 2 : in year y k . in our comparison, y 1 = 2022 and y 2 = 2024. we discuss some of the bigger differences in the remainder of this section, grouped into subsections as in section 4. see also figures 11-39 in the appendix. ## high-level recoded analysis whilst there were no significant differences in individual questions (other than q3 in the participant characteristics section), we were also interested in whether there was a higher-level improvement in reproducibility work and awareness of reproducibility issues. we therefore recoded responses, to address three higher-level questions: hq1 (reproduction activity): have researchers increased in engaging in reproduction efforts? hq2 (enabling reproducibility practices): has there been an increase in researchers actively taking steps to make their work reproducible? hq3 (awareness): is there an increase in researchers valuing reproducibility and understanding its relevance to the field? for recoding responses, we used the following questions for each construct: hq1-are researchers doing, or attempting, recreation/reproduction studies?; hq2-are researchers doing things to make recreation/reproduction of research easier?; hq3-are researchers aware of the importance of issues related to recreatability/reproducibility? in order to prevent bias, the recoding was performed by two non-author annotators from the same group. for each construct, we mapped each question/response pair, excluding demographic questions (q1-7), to one of the below options: supports yes: the question-response pair supports an answer of "yes" to the construct question. ## discussion and conclusion among the results that stand out in the 2024 survey are the large proportion of respondents that have tried to recreate other people's work (120/149), yet only a small proportion (3/120) was always able to obtain the same result. the most common response to this question of whether they had obtained the same results (q10) was sometimes (56/149). it is encouraging that more researchers are engaging with reproducibility work (hq1, from the 6 3 participants had not performed a human evaluation in the past 5 years, but do so when they report metrics. unsurprisingly, lack of time and funding were the two top issues preventing people from doing reproduction work, with 60% of respondents saying they would perform more reproduction work if funding bodies earmarked money for it. just behind time and money for reasons respondents did not perform human evaluations, was that they did not know how (13 of 41). six participants also suggested a toolkit would be useful. that significant numbers of respondents are not doing more things to make reproducibility easier for others (hq2, from the high-level analysis) is perhaps less because they are unwilling, and more that individual researchers are limited in what they can do. giving reproduction work the status it deserves as a cornerstone of science is something that funders and conference organisers can address, and our survey indicates that researchers would do more work on reproduction if this was done. given the lack of confidence in carrying out reproduction studies, developing guidance, methodologies and frameworks for performing evaluations would enable nlp researchers to learn how to perform repeatable experiments, and reduce their workload by not having to reinvent the wheel. overall, we have seen that virtually all 2024 respondents agree that nlp/ml work should be easy to recreate (up from 2022). at the same time, falling numbers are happy with how the field addresses reproducibility, and fewer researchers are sharing data, and are confident in their own work being reproducible. these results may in fact reflect growing understanding not only of the importance of reproducibility, but also of the difficulties in doing it properly. our recoded results show that researchers are now (i) engaging in reproduction efforts more, and (ii) valuing reproducibility and understanding its relevance to the field more. taken together, these surely are encouraging signs.