Reproducibility in NLP : What Have We Learned from the Checklist? (2023)

Abstract: Scientific progress in NLP rests on the reproducibility of researchers’ claims. The *CL conferences created the NLP Reproducibility Checklist in 2020 to be completed by authors at submission to remind them of key information to include. We provide the first analysis of the Checklist by examining 10,405 anonymous responses to it. First, we find evidence of an increase in reporting of information on efficiency, validation performance, summary statistics, and hyperparameters after the Checklist’s introduction. Further, we show acceptance rate grows for submissions with more Yes responses. We find that the 44% of submissions that gather new data are 5% less likely to be accepted than those that did not; the average reviewer-rated reproducibility of these submissions is also 2% lower relative to the rest. We find that only 46% of submissions claim to open-source their code, though submissions that do have 8% higher reproducibility score relative to those that do not, the most for any item. We discuss what can be inferred about the state of reproducibility in NLP, and provide a set of recommendations for future conferences, including: a) allowing submitting code and appendices one week after the deadline, and b) measuring dataset reproducibility by a checklist of data collection practices.

## introduction reproducibility is a foundational component of scientific progress. nlp systems are complex, and even when their behavior is carefully measured, incentives to publish quickly and limitations in the publishing process can lead to underreporting of information necessary for reproducible science. the ramifications of this extend beyond the research community; the audience of nlp papers published years ago was largely other nlp researchers, but today the world is watching developments in the field, looking for advances that will lead to broadlyadopted applications. as the impact of nlp grows, papers with more yes responses are more likely to be accepted, except those that mark yes to all checklist items, which we hypothesize contain responses which do not accurately represent the associated paper. so too do the consequences of reproducibility in our field. of course, nlp is not the first field to evaluate reproducibility; some have even described a "reproducibility crisis" in science @xcite . one tool designed to improve reproducibility is a checklist filled out at paper submission time. such a checklist can descriptively remind authors of relevant information to report, while preserving the freedom for authors to do so however they see fit. for example, the journal nature requires authors fill out a reporting checklist for life sciences articles @xcite . in 2019 neurips started to require that submissions fill out the ml reproducibility checklist @xcite , partly inspired by the nature checklist, and in 2021 aaai required their own checklist. foot_0 cvpr, iccv, eccv, and ijcai provide a checklist but do not collect responses. in this work, we provide the first analysis of the nlp reproducibility checklist @xcite . we have gathered 10,405 anonymized responses from emnlp 2020 and 2021, naacl 2021, and acl 2021. for the latter two we are also able to obtain reviewer scores, reproducibility judgements, and feedback on the checklist. our findings include: (1) most checklist items are frequently reported, and submissions reporting them are more often accepted and perceived as reproducible. (2) submissions that collect new data are accepted less and viewed as less reproducible, and these gaps are not explained by non-reporting of any current checklist items. (3) only about half of submissions report open sourcing code and many that do not also lack reporting on efficiency measures and even evaluation metrics. (4) a majority of reviewers describe the checklist as useful, and by contrasting responses to observed rates prior to the checklist we evidence a possible increase in reporting. we conclude with a discussion of what can be inferred from these findings about the state of reproducibility in nlp and offer recommendations to address the gaps we have measured. ## the nlp reproducibility checklist the nlp reproducibility checklist was originally introduced by @xcite . each item on the checklist is phrased as a statement, like "the number of parameters in each model," and authors can mark yes if they include that information in their paper, no if they do not include it in their paper, or n/a if that information does not make sense for their submission (e.g., they do not use any models to report parameter counts for). the checklist items were a part of the submission form, and it was required that authors fill it out to submit their paper. 2 thus, the checklist responses act as a (self-reported) overview of the contents of papers submitted to nlp conferences. importantly, authors were not required to include any information in their papers, they were only required to indicate whether or not they did include information. answers were made available to reviewers, who were expressly asked to assess the reproducibility of the work. the filled checklists were not released with the published papers. ## data and methodology in ## what can we learn about how reproducibility already works? we begin by measuring current practice, according to the (self-reported) checklist data. across all items and conferences, 62.7% of responses were yes. figure 4 shows that most items are reported in most submissions. moreover, we can measure reviewers' perception of reproducibility as well as differences in rates of reporting for items among papers that do and do get accepted by *cl review. more yes responses to checklist items associate with higher acceptance. in figures 1 and 2 we show positive associations between answering more items as yes and accept rate. each point in these figures represents the accept rate among all the submissions with the same number of yes responses among items. we regress the accept rate on a single variable counting checklist items answered yes for a submission. when pooling responses across all shared questions on all conferences r 2 = 0.53. 6 notably, submissions with yes responses to all items are consistently below the trend. we hypothesize in appendix a.2 that these submissions include responses which do not accurately represent the associated paper; recall that authors were required to fill out the checklist in order to submit, so marking the same response to all items is, in some sense, as close as they can get to not filling it out. the lower acceptance rate suggests that reviewers are not scoring papers based on the responses to the checklist itself, instead evaluating the contents of the paper, as intended. ## the data collection gap natural language processing has long been a field driven by data. a body of work has proposed best practices for documenting the characteristics and creation of datasets @xcite @xcite @xcite . among other concerns, such documentation is critical for the difficult task of dataset reproduction @xcite . from the checklist item newdatadescription, which asks that collection is described if new data is presented, we find that 38.2% and 6.3% of submissions mark yes and no, respectively. this implies that 44.5% of submissions to our nlp conferences collect new data; if almost half of submissions collect new data, we argue that data collection and dissemination practices deserve further attention. this also highlights clear room for improvement in the community: 14.1% of submissions that collect new data do not describe how it was collected, totalling 650 papers. submissions with new data have lower than average accept rate and avgreprod scores. alarmingly, submissions that collect new data (i.e., submissions that mark yes or no to new-datadescription) have a 5.1% lower acceptance rate than those that do not (i.e., mark n/a or blank). a low acceptance rate for answering no to newdatadescription would, by itself, be encouraging, perhaps indicating that reviewers expect data collection to be well documented. however, figure 5b shows that, even when answering yes m o d e l d e s c r ip t io n l in k t o c o d e i n f r a r u n t im e p a r a m e t e r s v a l id a t io n p e r f m e t r ic s n o t r a in in g e v a l r u n s h y p e r b o u n d h y p e r b e s t c o n f ig h y p e r s e a r c h h y p e r m e t h o d e x p e c t e d p e r f d a t a s t a t s d a t a s p l it d a t a p r o c e s s in g d a t a d o w n l o a d n e w d a t a d e s c r ip t io n d a t a l a n g u a g e s emnlp 2020 emnlp 2021 naacl 2021 acl 2021 overall 98.2% 47.0% 70.0% 47.5% 60.0% 50.4% 80.8% n/a 49.2% 71.1% 37.6% 59.8% 23.7% 94.7% 89.6% 85.9% 63.3% 38.4% n/a 97.2% 46.7% 70.2% 47.3% 58.5% 52.4% 81.3% 61.2% 52.0% 71.0% 39.9% 57.8% 51.0% 93.5% 88.1% 83.7% 56.7% 38.3% n/a 88.6% 43.8% 60.1% 41.4% 54.1% 47.7% 86.0% 52.8% 46.0% 62.6% n/a 56.7% 45.0% 82.1% 77.1% 73.7% 55.4% 34.9% 71.3% 96.4% 44.7% 68.4% 43.9% 57.8% 52.9% 79.2% n/a 52.1% 69.9% 40.1% 57.8% 32.0% 93.7% 88.4% 85.3% 59.8% 39.4% n/a 96.2% 45.9% 68.4% 45.7% 58.2% 51.4% 81.2% 58.9% 50.5% 69.7% 39.2% 58.2% 37.1% 92.6% 87.3% 83.6% 59.3% 38.2% 71.3% any item 0.20 0.25 0.30 0.35 0.40 0.45 0.50 acceptance rate conditioned on a p (accept | a) 113,627 30,243 34,717 (a) validationperf expectedperf newdatadescription acceptance rate conditioned on x, a p (accept | x, a) 5,347 3,859 3,977 3,065 2,767 650 1,847 3,580 5,611 (b) p (accept) @xmath0: accept rates over all conferences for submissions with a given response. (a) shows rate conditioned on response regardless of item. (b) shows the only items where yes accept rates are below average. total count of each response is shown on the bar. items with higher no acceptance than yes could indicate the community has not fully embraced these practices as norms. high acceptance rate for newdatadescrip-tion n/a indicates that papers that do not collect data are more likely to be published. to describing the data collection process, accept rate is 1.6% lower than average. meanwhile, there is a similar gap in avgreprod. scores for submissions that collect data are lower by 2.4% relative to those that do not. again, this is not limited to submissions that fail to describe the data collection process. figure 6b reveals that the mean score over submissions that do describe their data collection is 0.04 below the mean of all submissions. when considering only accepted papers, however, the gap in avgreprod disappears. the review process ends up with accepted dataset papers with similar avgreprod to non-dataset papers, but along the way many more dataset than non-dataset submissions are rejected and those have lower avgre-any item 3.0 3.2 3.4 3.6 3.8 4.0 mean percieved reproducibility score conditioned on a μavgreprod | a 42,785 10,806 11,825 (a) linktocode newdatadescription mean percieved reproducibility score conditioned on x,a μavgreprod | x,a 1,779 1,500 1,540 226 457 2,051 prod. we hypothesize that these phenomena arise both because dataset papers may indeed be more challenging to (re)produce and also because of the persistent (and problematic) tendency to value modeling over data collection @xcite . high compliance among the dataset checklist items does not reveal the source of the accept rate and avgreprod gap. datastats, datas-plit, datadownload, and datalanguages receive the highest rates of reporting other than modeldescription and metrics. this only grows when looking just at submissions presenting new datasets, reaching 97.3%, 91.7%, 91.4%, and 86.6% respectively, and newdatadescription is also reported in 86.0% of these submissions. un- datadownload newdatadescription 62.4% 22.6% 8.9% 6.1% proportion of filtered submissions with pattern y y n y n n y n 0.0 0.2 0.4 0.6 0.8 1.0 acceptance rate of filtered submissions with pattern p (accept | @xmath1, no}) p (accept) figure 7: proportion (row labels) and accept rates (horizontal purple bars) for all response patterns on dataset availability and creation (excluding instances where either item is n/a or blank). nearly 1 in 11 of these neither share the data nor describe its collection, yet 28.9% of those are accepted. ## code (un)availability we see that on average 45.9% of submissions report linking to code (47.5% for accepted papers). we see in figure 6 that whether submissions answer linktocode as yes or no has the largest difference in avgreprod scores, with a gap of 0.30. yet accept rates for submissions with or without linktocode are nearly the same. we find similar rates of links to code as at ml conferences. @xcite reveal a 38.8% self-reported rate of code availability at submission time for neurips 2019. they find this number drops to 27.7% when checked by at least one reviewer. extrapolating from this false reporting rate, the true code availability rate among accepted papers in our data might be 32.8%. meanwhile, a study on icml 2019 by @xcite finds 36% of submitted and 43% of accepted papers have code at submission time, though it unclear if these are self-reported. previous efforts to measure camera-ready code availability have found widely different rates than our reported linktocode at submission time. unfortunately our data does not cover code availability at camera-ready, except insofar as some authors may interpret this checklist item to permit promises to later release code. 24.3% of papers at naacl 2022 opted in to submitting a code link to the reproducibility track and received an open source code badge. 7 we recognize this was optional for authors, and thus it is likely the case that the true number of camera-ready papers that included a link to code was higher. the studies mentioned before found that 74.4% and 64% of camera-ready papers had links to code at neurips 2019 and icml 2019. narrowing the range of these measurements should be a worthwhile effort, as these studies found code being available during review was useful in 1,315 reviews in neurips 2019, and 18.3% of icml 2019 reviewers surveyed were able to look at code and found it useful. items on compute efficiency are completely reported in only 29.8% of submissions without code. figure 8 shows patterns for these efficiency items that occurred more than 100 times. while accept rates are somewhat lower when items are not reported, 21.2% of these without-code submissions report none of the efficiency measures. there may be unavoidable impediments to making code available, such as intellectual property. but in this case even greater emphasis should be placed on reporting efficiency measures, as estimating these without code is quite difficult. similarly 19.6% of submissions with no code report no to explaining metrics which may render evaluations irrecoverably ambiguous if there are varying implementations of a metric. compared against manually checked data from before the checklist introduction, our data shows increases in 8 of 10 items. figure 9 shows rates of a subset of items that were manually checked by @xcite in 50 papers sampled from emnlp 2018. the self-reported rates available in our data are not ideal comparisons as they likely overestimate. however, the emnlp 2018 sample may also overestimate, as only "experimental results" are included for which we would expect fewer n/as, given the checklist's focus on empirical work. there is little variation in response proportions between conferences. excluding two types of outliers likely caused by changes in the checklist (see appendix a.3), the maximum difference between conferences for an item is 6.6% and the maximum difference averaged over all items is 2.2%. this does demonstrate that measured response patterns are robust across conferences. however it also indicates that reproducibility reporting has stagnated over this one year period. when asked, a majority of reviewers found the checklist to be somewhat or very useful. in naacl 2021 and acl 2021 reviewers gave feedback on the checklist. 59.9% found the checklist "somewhat useful," 17.0% found it "very useful," and 23.2% found it "not useful." while this is higher than the 34% of reviewers who answered "yes" that the similar neurips 2019 checklist was "useful for evaluating the submission" @xcite , it is worth noting that respondents to the question for neurips could answer that they did not read the checklist results. author comments from submissions where the majority of reviewers found the checklist "not useful," show possible gaps in checklist coverage. some comment on not training mod-els or using hyperparameters from previous work. many such submissions are represented among the 22.0% that answer n/a to all hyperparameter questions. others comment on referring readers to citations for details of standard models, data, or metrics. re-elaboration is pedagogically important but, comments argue, especially onerous for survey papers. finally a comment notes that the checklist is less relevant to psycholinguistics and cognitive modeling, and indeed the n/a rate of "linguistic theories, cognitive modeling and psycholinguistics" track submissions is 33.7%, an increase of 14.6% above the n/a rate over all tracks. ## discussion our findings from the nlp reproducibility checklist can both help inform new interventions and guide improvements to future checklists that will measure the outcomes of those interventions. these findings suggest that, after an initial increase, rates of reporting have stagnated in the period examined and will need new approaches to improve further. the conference system should better support papers that collect new data. as discussed in section 5, papers that collect new data have 5.1% lower acceptance rate than those that do not. whether or not this gap is a cause or effect of the lack of prestige given to data work that rogers (2021) describes, increasing awareness and resources for this work can help more high quality data reach publication. checklists should also increase coverage of this topic. in our data a single item, newdatade-scription, covers all reporting regarding data collection. we find that papers with new data are perceived as less reproducible both when answering no or yes to describing how they collected data. likely a combination of several factors lead reviewers to score the reproducibility of papers with new data lower by 2.4% relative to papers without. to discover which are lacking, best practices in data reproducibility documentation @xcite should be tracked individually with checklists. incentivize authors releasing code. we find that releasing code is the single most influential checklist item on perceived reproducibility. this aligns with work across diverse fields that argues open source code is key for transparent and reproducible science @xcite @xcite . these works also sug-gest that beyond reproducibility, open source code enables more impactful research by allowing other researchers to build on introduced methods and better understand findings through reading code. however, we find that less than half of papers in our study report releasing code at submission. we encourage conferences to incentivize code release at submission and especially camera-ready, and authors should be made aware of the significant benefit that code submission can have for the review process. 8 initiatives like the naacl 2022 reproducibility track are a step in the right direction, as they publicly recognize open source code and verify code availability rather than only relying on self-reporting. however, in our data we see no evidence that code availability is increasing over time, so more direct incentives from publication venues are needed. make checklist responses public. self-reported data is notoriously unreliable, but making the checklist responses public will add accountability. 9 in addition, the checklist responses can reference specific sections and act as an index of the paper, so a reader knows where to look for what information. this will be implemented at acl 2023, and we recommend other conferences follow. conferences should allow submission of checklists, unlimited appendices, and code a week after the main deadline. doing so can help establish a norm of code submission as part of the review process. likewise, additional time could improve completeness and accuracy of the checklist. many pieces of information important for reproducibility are appropriate to include in the appendix of a paper without counting towards the page limit (e.g., a full list of hyperparameter values). this need not increase the burden on reviewers, as they can consult checklists rather than the appendix to assess reporting.