Reproducibility in Computational Linguistics: Is Source Code Enough? (2022)

Abstract: The availability of source code has been put forward as one of the most critical factors for improving the reproducibility of scientific research. This work studies trends in source code availability at major computational linguistics conferences, namely, ACL, EMNLP, LREC, NAACL, and COLING. We observe positive trends, especially in conferences that actively promote reproducibility. We follow this by conducting a reproducibility study of eight papers published in EMNLP 2021, finding that source code releases leave much to be desired. Moving forward, we suggest all conferences require self-contained artifacts and provide a venue to evaluate such artifacts at the time of publication. Authors can include small-scale experiments and explicit scripts to generate each result to improve the reproducibility of their work.

## introduction modern natural language processing and computational linguistics research progresses at a rapid pace, making it expedient to build upon the groundwork of earlier research rather than reinventing solutions from scratch. this is complicated by the field's reliance in recent years on deep learning models that are difficult to interpret and sensitive to small changes in architecture and environment. these distinct characteristics hinder their reproducibility, as do the substantial computing resources often required to replicate them. thus, many strong nlp models fall short on two crucial parameters: accessibility and reproducibility. although deep learning models today allow for effective processing in highly complex search spaces and in most cases outperform solutions from the past, researchers must consider the risks and potential ethical implications associated with their use alongside the performance benefits. there have been numerous calls and attempts @xcite to push the community into taking more responsibility for the reproducibility of their research. several venues have introduced and adapted reproducibility checklists and standards @xcite into their submission process @xcite @xcite @xcite , and others have organized reproducibility challenges @xcite to encourage the community to reproduce published research. although this is undeniably a step forward, it is unclear how large of a step it has been. there is also some concern that asking reviewers to evaluate reproducibility burdens them with another time-consuming task that might extend beyond their expertise. in this work, we analyze and report on the state of reproducibility in nlp. we investigate the extent to which content necessary for reproducing research is currently available, and study the influence of reproducibility checklists on this availability over the last seven years. we also conduct an eight-paper reproducibility case study to develop a deeper understanding of current strengths and weaknesses in research reproducibility in nlp. our key contributions include the following: we make the artifacts from our own repro-ductions publicly available as self-contained docker containers. our recommendations include incorporating small-scale experiments in papers to increase accessibility, creating well-documented scripts to reproduce reported results, publishing executable selfcontained artifacts, and embedding artifact evaluation in the publication pipeline. we elaborate on our study process and findings in the remainder of this paper. our hope is that this paper serves as a call to action for systemic improvements to research reproducibility in nlp. ## background numerous studies have assessed the reproducibility of scientific publications. this task often involves attempting to achieve results close enough to the ones reported in the paper with little to no reliance on the released software artifacts, if available. @xcite attempts to quantify the reproducibility ratio of 255 papers published from 1984 to 2017. he selects different thresholds for a minimal acceptable error for algorithmic and empirical claims, ultimately reporting a 63% reproducibility ratio. in a similar study, @xcite survey 395 papers presented at the acl 2011 and 2016 conferences and identify whether links to data and code were provided. then, they attempt to reproduce the results of ten papers using provided code and data. they ultimately find results close to those reported for six papers. @xcite attempt to independently investigate the claims of six studies on text mining for citation screening. in the authors' words, 27% of machine learning papers lack the necessary information required for achieving reproducible results; hence, they introduce a checklist to help mitigate this issue. the challenge of dealing with missing information has also been brought up by @xcite . utilizing checklists and guiding authors towards better standards during the paper submission process has become a common practice amongst several venues (nature, 2022; aaai, 2022; acm, 2022; acl, 2022b; @xcite . aside from guidelines, communities have organized reproducibility challenges @xcite that attempt to promote improved reproducibility across the field. while this increased attention towards openness and availability is a welcome change, the lack of consensus on terms and definitions has diminished progress. there are a variety of definitions of and perspectives on reproducibility. @xcite define reproducing as running the same software on the same input data and obtaining the same results. replicating then is limited to running new software and achieving results judged as similar enough by an expert in the field. the association for computing machinery (acm) (acm, 2022) considers the team and the experimental setup as contributing factors; furthermore, they add another term, repeatability, to the glossary of definitions. @xcite and schloss (2018) introduced two additional concepts known as robustness and generalisability to cover other missing dimensions. these definitions attempt to cover an open-ended number of dimensions. therefore, they often do not age well and become obsolete upon the arrival of a more comprehensive definition @xcite . @xcite suggest the community should adapt the definitions provided by the international vocabulary of metrology (jcgm, 2012, vim). vim defines reproducibility as a measurement precision under reproducibility conditions of measurement. these conditions must be known and recorded and include but are not limited to the source code, hyperparameters, dependencies, and runtime environment. as a result, this framework enables the use of precision as a measure of statistical variability to quantify how close results are to one another. doing so will provide far more information than just binary assessments of whether research is reproducible or irreproducible. a virtual machine (e.g., vmware or virtual-box) or within a container (e.g., docker) so that they run "out-of-the-box" on any machine. thus, research artifacts are much more than simply releasing the source code for the prototype on a public website (i.e., github). conferences in other fields of computer science (e.g., the 2021 acm sigplan international conference on object-oriented programming systems, languages, and applications (wadler and drossopoulou, 2021, oopsla), the 2022 acm sigplan conference on programming language design and implementation (jhala and dillig, 2022, pldi), the 2022 european conference on object-oriented programming @xcite , and the 2021 acm international conference on architectural support for programming languages and operating systems @xcite ) allow authors to submit artifacts, typically after the paper is accepted. the quality of the submitted artifacts is then assessed by the artifact evaluation committee (aec), who run the artifacts following the authors' instructions to judge whether the artifact reproduces all the results reported in the paper, and whether the artifact can be reused for future research on the same subject. at the recommendation of the aec, published papers then get a set of badges on the first page. as a result, reproducing and building on research papers with an aec badge should be straightforward: users should simply need to download the artifact and follow the instructions. conferences in the broad areas of nlp and machine learning have recently adopted reproducibility checklists (aaai 2022 (aaai, 2022), neurips 2022 (neurips, 2022)) which require authors to provide an appendix with the source code used in the experiments. additionally, they recommend the inclusion of dependency specifications, training and evaluation code, pre-trained models, and a readme file containing the information required to achieve the results in the paper. unfortunately, as we show in the following sections, reproducing publications that follow these good practices is not easy. it requires a considerable amount of engineering involving fixing compilation errors, "guessing" configurations not documented, figuring out which code to execute for which experiment, manually processing experimental data, and dealing with obsolete tools or libraries. given the numerous challenges one faces to re-produce a research paper's results, there has been a recent surge of venues solely focused on reproducibility (e.g., the ml reproducibility challenge 2021 @xcite and reprogen 2021 @xcite ), in which participants attempt to reproduce results of a selected published paper. unlike the research artifacts described above, these reproducibility efforts do not lead to a self-contained and self-executable artifact. self-contained and self-executable artifacts should be capable of reproducing the results in short order while avoiding vulnerability to "bit rotting" as time passes and widely available tools and libraries grow obsolete. ## definitions and methods we adapt the metrology-based reproducibility assessment originally proposed by @xcite to achieve our reported results. in this framework, reproducibility is a measurement of precision and is directly connected to the conditions in which the measurement is being recorded. in the context of machine learning and natural language processing, these conditions include but are not limited to source code, trained models, evaluation methods, and datasets. furthermore, repeatability is defined as a special case of reproducibility where the conditions across different measurements are the same. this formulation enables the use of common terms used in reporting precision. we report and focus on coefficient of variation (cv * ), defined as the unbiased sample standard deviation over the mean. ultimately, the gold standard for scientific artifacts is to have the highest level of reproducibility precision (or low variability). however, this standard is not practical. an alternative approach is to gradually push the community towards improving existing reproducibility standards. existing reproducibility tracks are a great example of this effort. one of the primary objectives of the reproducibility movement is to increase availability of data and source code. source code is an integral part of the reproducibility process since it allows other researchers to implement or execute conditions and details that may have been omitted in the publication itself. here, we investigate trends in source code availability for scientific papers published in major conferences of the association for computational linguistics (acl) in recent years, using information from the acl anthology. observing these trends is especially interesting since not all acl conferences have introduced a reproducibil-ity track in their submission process. we expected to observe a higher code availability ratio among venues that actively promote reproducibility. following this, we switch our focus and investigate whether the released source code is enough to reproduce the reported results. empirical results have become harder to reproduce over time for known and unknown reasons (several of these reasons are discussed in section 5.2). although we quantify reproducibility using cv * , we consider a reproducibility attempt successful if we are able to achieve any results. we understand that this process is not comprehensive-we cannot determine absolute reproducibility, since one may be able to spend days debugging a specific source code to ultimately achieve the reported results. however, absolute reproducibility and its time cost is not feasible at scale or for many researchers. in our case study, we clearly explain all steps taken by following instructions provided by the authors. we raise any questions or issues we face along the way, opening direct lines of communication with authors to overcome challenges and improve the reproducibility of the released source code as part of our process. we hypothesize that despite the increased availability of source code for papers at large nlp conferences, achieving reproducible results may still be an extremely challenging task. for our case study, we randomly selected eight papers from the 2021 conference on empirical methods in natural language processing @xcite . we tried to reproduce their results using the code and instructions provided. out of over 1300 accepted papers at emnlp 2021, 723 had urls to a repository containing the code required to run their experiments. our selected papers are provided below. to save space, we refer to each paper by its associated number in this list. ## results in this section we describe the results obtained, and how they answer the following research questions (rqs): rq1 has code availability for published nlp literature improved in recent years? rq2 is code availability enough for reproducibility? rq3 what new guidelines can be used to support reproducibility? ## discussion it is encouraging to observe the recent upward trend in releasing source code across the broad nlp community, thanks to the recent focus on reproducibility. unsurprisingly, conferences not promoting reproducibility standards have fewer submissions with included code. therefore, we encourage all conferences to include such standards in their submission process. unfortunately, our results suggest that submitting the code alone does not seem to be enough, as the released code does not meet a minimum requirement for reproducibility, defined as achieving the results reported in the paper using the provided source code. we believe it is time to improve reproducibility standards to address the concerns we raise in regards to quality of the released source code. in particular, we strongly suggest shifting the focus from source code to research artifacts which include (1) a self-contained runtime environment (e.g., a docker container or a vm image) with scripts for achieving every single result reported in the paper, (2) smaller experiments to quickly validate the integrity of the artifact, and (3) extensive documentation explaining how to run the code. we took a first step in this direction by packaging all the results in the paper in their own artifact, and releasing it with this paper. these artifacts are available through zenodo @xcite . reproducibility is a desired attribute for solutions in natural language processing, but it comes with a cost. there may exist cases in which the required conditions for reproducing the results are not practical. defining what is practical, of course, depends on the problem at hand. for example, reproducing a named entity classifier that requires using the same hardware may not be considered practical. this phenomenon is quite similar to the bias-variance tradeoff. bias-variance, a property of statistical and machine learning models, suggests that the variance of the parameter estimated across samples can be reduced by increasing the bias. a dilemma exists when trying to minimize these two sources of error simultaneously. we have a dilemma when it comes to assessing the reproducibility of results. many attempts have focused on controlling all the variables. yet, while they have their use cases, their complexity makes them less viable. perhaps a better alternative is to reduce the emphasis on the top-performing results and utilize techniques that attempt to aggregate and report the results of a set of experiments. ## conclusions in this paper, we examined the trends of source code availability at major nlp conferences in recent years. we also performed a reproducibility study on eight randomly selected papers. after achieving a 25% success rate, we draw attention to the primary causes of irreproducibility of the selected papers. based on the findings from our case study, we close by providing recommendations for improving the state of reproducibility in nlp. researchers concerned about the added workload should come to terms with the fact that accessibility and reproducibility are fundamental, rather than auxiliary, cornerstones of strong research. it is also advantageous to researchers: following the recommendations proposed in this paper is anticipated to prolong and future-proof work, minimizing the required support in the long term.