A Systematic Review of Reproducibility Research in Natural Language Processing (2021)

Abstract: Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP,

## introduction reproducibility is one of the cornerstones of scientific research: inability to reproduce results is, with few exceptions, seen as casting doubt on their validity. yet it is surprisingly hard to achieve, 70% of scientists reporting failure to reproduce someone else's results, and more than half reporting failure to reproduce their own, a state of affairs that has been termed the 'reproducibility crisis' in science @xcite . following a history of troubling evidence regarding difficulties in reproducing results @xcite , where 24.9% of attempts to reproduce own results, and 56.7% of attempts to reproduce another team's results, are reported to fail to reach the same conclusions @xcite , the machine learning (ml) and natural language processing (nlp) fields have recently made great strides towards recognising the importance of, and addressing the challenges posed by, reproducibility: there have been several workshops on reproducibility in ml/nlp including the reproducibility in ml workshop at icml'17, icml'18 and iclr'19; the reproducibility challenge at iclr'18, iclr'19, neurips'19, and neurips'20; lrec'20 had a reproducibility track and shared task @xcite ; and neurips'19 had a reproducibility programme comprising a code submission policy, a reproducibility challenge for ml results, and the ml reproducibility checklist @xcite , later also adopted by emnlp'20 and aaai'21. other conferences have foregrounded reproducibility via calls, chairs' blogs, foot_0 special themes and social media posts. sharing code, data and supplementary material providing details about data, systems and training regimes 2 is firmly established in the ml/nlp community, virtually all main events now encouraging and making space for it. reproducibility even seems set to become a standard part of reviewing processes via checklists. far from beginning to converge in terms of standards, terminology and underlying definitions, however, this flurry of work is currently characterised by growing diversity in all these respects. we start below by surveying concepts and definitions in reproducibility research, areas of particular disagreement, and identify categories of work in current nlp reproducibility research. we then use the latter to structure the remainder of the paper. ## terminology and frameworks reproducibility research in nlp and beyond uses a bewildering range of closely related terms, often with conflicting meaning, including reproducibility, repeatability, replicability, recreation, re-run, robustness, repetition and generalisability. the fact that no formal definition of any of these terms singly, let alone in relation to each other, is generally accepted as standard, or even predominant, in nlp at present, is clearly a problem for a survey paper. in this section, we review usage before identifying common-ground terminology that will enable us to talk about the research we survey. the two most frequently used 'r-terms', reproducibility and replicability, are also the most problematic. for the acm (association for computing machinery, 2020), results have been reproduced if "obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts 3 grep -e 'title * =. * (r|r)\} * (eproduc|epl ica)' anthology.bib the different definitions of reproducibility and replicability above, put forward in six different contexts, are not compatible with each other. grappling with degrees of similarity between properties of experiments such as the team, data and software involved, and between results obtained, each draws the lines between terms differently, and moreover variously treats reproducibility and replicability as properties of either systems or results. all are patchy, not accounting for some circumstances, e.g. a team reproducing its own results, not defining some concepts, e.g. sameness, or not specifying what can serve as a 'result,' e.g. leaving the status of human evaluations and dataset recreations unclear. the extreme precision of the definitions of the international vocabulary of metrology (vim) (jcgm, 2012) (which the acm definitions are supposed to be based on but aren't quite) offers a common terminological denominator. the vim definitions of reproducibility and repeatability (no other r-terms are defined) are entirely general, made possible by two key differences compared to the nlp/ml definitions above. firstly, in a key conceptual shift, reproducibility and repeatability are properties of measurements (not of systems or abstract findings). the important difference is that the concept of reproducibility now references a specified way of obtaining a measured quantity value (which can be an evaluation metric, statistical measure, human evaluation method, etc. in nlp). secondly, reproducibility and repeatability are defined as the precision of a measurement under specified conditions, i.e. the distribution of the quantity values obtained in repeat (or replicate) measurements. in vim, repeatability is the precision of measurements of the same or similar object obtained under the same conditions, as captured by a specified set of repeatability conditions, whereas reproducibility is the precision of measurements of the same or similar object obtained under different conditions, as captured by a specified set of reproducibility conditions. see appendix c for a full set of vim definitions of the bold terms above. to make the vim terms more recognisable in an nlp context, we also call repeatability reproducibility under same conditions, and (vim) reproducibility reproducibilty under varied conditions. finally, we refer to experiments carrying out repeat measurements regardless of same/varied conditions as 'reproduction studies.' categories of reproducibility research: using the definitions above, the work we review in the remainder of the paper falls into three categories (corresponding to sections 3-5): reproduction under same conditions: as near as possible exact recreation or reuse of an existing system and evaluation set-up, and comparison of results. 7 reproduction under varied conditions: reproduction studies with deliberate variation of one or more aspects of system and/or measurement, and comparison of results. multi-test studies: multiple reproduction studies connected e.g. because of an overall multi-test design, and/or use of same methodology. ## reproduction under same conditions papers reporting reproductions under same conditions account for the bulk of nlp reproducibility research to date. the difficulty of achieving 'sameness of system' has taken up a lot of the discussion space. as stressed by many papers @xcite , recreation attempts have to have access to code, data, full details/assumptions of algorithms, parameter settings, software and dataset versions, initialisation details, random seeds, run-time environment, hardware specifications, etc. a related and striking finding, confirmed by multiple repeatability studies, is that results often depend in surprising ways and to surprising degrees on seemingly small differences in model parameters and settings, such as rare-word thresholds, treatment of ties, or case normalisation @xcite @xcite . effects are often discovered during system recreation from incomplete information, when a range of values is tested for missing details. the concern is that the ease with which such nlp results are perturbed casts doubt on their generalisability and robustness. the difficulties in recreating, or even just rerunning, systems with same results have led to growing numbers of reproducibility checklists @xcite , and tips for making system recreation easier, e.g. the pytorch @xcite recommended settings. 8 we analysed reproduction studies under same conditions from 34 pairs of papers, and identified 549 individual score pairs where reproduction object, method and outcome were clear enough to include in comparisons (for a small number of papers this meant excluding some scores). we also examined the size of the differences between original and reproduction scores. for this purpose we computed percentage change (increase or decrease) for each score pair, and looked at the distribution of size and direction of change. for this analysis, we excluded score pairs where one or both scores were 0, as well as 4 very large outliers (all increases). results are shown in the form of a histogram with bin width 1 (and clipped to range -20..20) in figure 2 . the plot clearly shows the imbalance between worse (60% of non-same scores) and better (40%) reproduction scores. the figure also shows that a large number of differences fall in the -1..1 range. however, the majority of differences, or 3/5, are greater than +/-1%, and about 1/4 are greater than +/-5%. ## reproduction under varied conditions reproduction studies under varied conditions deliberately vary one or more aspects of system, data or evaluation in order to explore if similar results can be obtained. there are far fewer papers of this type (see horsmann and zesch (2017) pick up strong results by @xcite showing lstm tag-ging to outperform crf and hmm taggers, and test whether they can be confirmed for datasets with finer-grained tag sets. using 27 corpora (21 languages) with finer-grained tag sets, they systematically compare results for the 3 models, and show that lstms do perform better, and that their superiority grows in proportion to tag set size. htut et al. (2018a) recreate the grammar induction model prpn @xcite , testing different versions with different data. prpn is confirmed to be "strikingly effective" at latent tree learning. in a subsequent repeat study under same conditions, htut et al. (2018b) test prpn using the authors' own code, obtaining the same headline result. millour et al. (2020) attempt to get the pos tagger for alsatian from @xcite to work with the same accuracy for a different dataset. collaborating with, and using resources provided by, the original authors and recreating some where necessary, the best result obtained was 0.87 compared to the original 0.91. abdellatif and elgammal (2020) varied conditions of reproduction for classification results by @xcite , and were able to confirm outcomes for three new non-english datasets, in all three respects (value, finding, conclusion) identified by @xcite . pluciński et al. (2020) and garneau et al. (2020) both find that the cross-lingual word embedding mappings proposed by @xcite yield worse results on more distant language pairs. vajjala and rama (2018)'s automatic essay scoring classification system was tested on different datasets and/or languages in three studies @xcite ; caines and buttery, 2020; huber and c ¸öltekin, 2020) all of which found performance to drop on the new data. ## multi-test and multi-lab studies work in this category is multi-test, in the sense of involving multiple reproduction studies, in a uniform framework using uniform methods. some of it is also multi-lab in that reproduction studies are carried out by more than one research team. for example, in one multi-test repeatability study, @xcite randomly select five papers each from acl'11 and acl'16 for which code/data was available. in a uniform design, original authors were contacted for help if needed, a maximum time limit of 8h was imposed, and all work was done by the same masters student. it's not clear how scores were selected (not all are attempted), and reasons for failure are not always clear even from linked material. of the 120 score pairs obtained, 60 were the same, 12 reproduction scores were better, 22 were worse, and 26 runs failed (including exceeding the time limit). see rodrigues et al. ( 2020 ) recreated six semeval'18 systems from the argument reasoning comprehension task, following system descriptions and/or reusing code, with no time limit. scores were the same for one system, and within +/-0.036 points for the other five; the semeval ranking was exactly the same. systems were also run on a corrected version of the shared-task data (which contained unwitting clues). this resulted in much lower scores casting doubt on the validity of the original shared task results. reprolang @xcite is so far the only multi-lab (as well as multi-test) study of reproducibility in nlp. it was run as a selective shared task, and required participants to conform to uniform rules. 11 papers were selected for reproduction via an open call and direct selection. participants had to 'reproduce the paper,' using information contained/linked in it. participants submitted (a) a report on the reproduction, and (b) the software used to obtain the results as a docker container (controlling variation from dependencies and run-time environments) on gitlab. submissions were reviewed in great detail, submitted code was test-run and checked for hard-coding of results. 11 out of 18 submissions were judged to conform with requirements. one original paper @xcite attracted four reproductions (bestgen, 2020; huber and c ¸öltekin, 2020; caines and buttery, 2020; @xcite in what must be a groundbreaking first in nlp. see ## conclusions it seemed so simple: share all data, code and parameter settings, and other researchers will be able to obtain the same results. yet the systems we create remain stubbornly resistant to this goal: a tiny 14.03% of the 513 original/reproduction score pairs we looked at were the same. at the same time, worryingly small differences in code have been found to result in big differences in performance. another striking finding is that reproduction under same conditions far more frequently yields results that are worse than results that are better: we found 258 out of 436 non-same reproduction results (59.2%) to be worse, echoing findings from psychology @xcite . why this should be the case for reproduction under same conditions is unclear. it is probably to be expected for reproduction under different conditions, as larger parameter spaces, more datasets and languages etc., are tested subsequently, and the original work may have selected better results. there is a lot of work going on in nlp on reproducibility right now; it is to be hoped that we can solve the vexing and scientifically uninteresting problem of how to rerun code and get the same results soon, and move on to addressing far more interesting questions of how reliable, stable and generalisable promising nlp results really are. the measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. for computational experiments, this means that a researcher can reliably repeat her own computation. reproducibility (different team, same experimental setup)* the measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. for computational experiments, this means that an independent group can obtain the same result using the author's own artifacts. replicability (different team, different experimental setup)* the measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. for computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently. results validated: this badge is applied to papers in which the main results of the paper have been successfully obtained by a person or team other than the author. two levels are distinguished: results reproduced v1.1 the main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author. results replicated v1.1 the main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts. in each case, exact replication or reproduction of results is not required, or even expected. instead, the results must be in agreement to within a tolerance deemed acceptable for experiments of the given type. in particular, differences in the results should not change the main claims made in the paper.