Experimental Standards for Deep Learning in Natural Language Processing Research (2022)

Abstract: The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, compared to more established disciplines, a lack of common experimental standards remains an open challenge to the field at large. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in NLP into a single, widely-applicable methodology. Following these best practices is crucial to strengthen experimental evidence, improve reproducibility and enable scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.

## introduction spurred by the advances in machine learning (ml) and deep learning (dl), the field of natural language processing (nlp) has seen immense growth over the span of the last ten years, as illustrated by the number of publications in figure 2 . while such progress is remarkable, rapid growth comes at a cost: akin to concerns in other disciplines @xcite , several authors have noted major obstacles with reproducibility @xcite and a lack of significance testing @xcite or published results not carrying over to different experimental setups, for instance in text generation @xcite and with respect to new model architectures @xcite . others have questioned commonly-accepted procedures @xcite @xcite as well as the (negative) impacts of research on society @xcite @xcite and environment @xcite @xcite . these problems have not gone unnoticed-many of the mentioned works have proposed a cornucopia of solutions. in a quickly-moving environment however, keeping track and implementing these proposals becomes challenging. in this work, we weave these open issues together into a cohesive methodology for gathering stronger experimental evidence, that can be implemented with reasonable effort. based on the scientific method (section 2), we divide the empirical research process-obtaining evidence from data via modeling-into four steps, which are depicted in figure 1 : data (section 3), including dataset creation and usage, codebase & models (section 4), experiments & analysis (section 5) and publication (section 6). for each step, we survey contemporary findings and summarize them into actionable practices for empirical research. using insights from adjacent sub-fields of ml / dl, we extract useful insights to help overcome current challenges with replicability in nlp. ## preliminaries our proposed methodology must be built on the scientific principles for generating strong evidence for the general advancement of knowledge, as defined by the following terms: the scientific method knowledge can be obtained through several ways including theory building, qualitative methods, and empirical research @xcite . here, we focus on the latter aspect, in which (exploratory) analyses lead to falsifiable hypotheses that can be tested and iterated upon @xcite . 2 this process requires that anyone must be able to back or dispute these hypotheses in the light of new evidence. in the following, we focus on the evidence-based evaluation of hypotheses and how to ensure the scientific soundness of the experiments which gave rise to the original empirical evidence, with a focus on replicability and reproducibility. in computational literature, one term requires access to the original code and data in order to re-run experiments exactly, while the other requires sufficient information in order to reproduce the original findings even in the absence of code and original data (see also figure 1 ). 3 replicability within dl, we take replicability to mean the exact replication of prior reported evidence. in a computational environment, access to the same data, code and tooling should be sufficient to generate prior results. however, many factors, such as hardware differences, make exact replication difficult to achieve. nonetheless, we regard experiments to be replicable if a practitioner is able to re-run them to produce the same evidence within a small margin of error dependent on the environment, without the need to approximate or guess experimental details. ## data frequently, it is claimed that a model solves a particular cognitive task, however in reality it merely scores higher than others on some specific dataset according to some predefined metric @xcite . of course, the broader goal is to improve systems more generally by using individual datasets as proxies. admitting that our experiments cover only a small slice of the real-world sample space will help more transparently measure progress towards this goal. in light of these limitations and as there will always be private or otherwise unavailable datasets which violate replicability, a practitioner must ask themselves: which key information about the data must be known in order to reproduce an experiment's findings? in this section we define requirements for putting this question into practice during dataset creation and usage such that anyone can draw the appropriate conclusions from a published experiment. ## codebase & models the nlp community has historically taken pride in promoting open access to papers, data, code, and documentation, but some have also noted room for improvement @xcite . one practice has been to open-source all components of the experimental procedure in a repository, consisting of all code, necessary scripts, and detailed documentation. the benefit of such a repository is in its ability to enable direct replication. in particular, a comprehensive code base directly enables replicability. in practice, such documentation is often communicated through a readme file, in which user-oriented information is described. 4 in dl, full datasets can be large and impractical to share. due to their importance however, it is essential to carefully consider how one can share the data with researchers in the future. therefore, repositories for long-term data storage backed by public institutions should be preferred (e.g., lindat/clarin by @xcite , more examples in appendix c). nevertheless, practitioners often can not distribute data due to privacy, legal, or storage reasons. in such cases, practitioners must instead carefully consider how to distribute data and tools to allow future research to produce accurate replications of the original data @xcite . hyperparameter search hyperparameter tuning strategies remain an open area of research (e.g., @xcite , but are central to the replication of contemporary models. the following rules of thumb exist: grid search or bayesian optimization can be applied if few parameters can be searched exhaustively under the computation budget. otherwise, random search is preferred, as it explores the search space more efficiently @xcite . advanced methods like bayesian optimization @xcite and bandit search-based approaches @xcite can be used as well if applicable @xcite . to avoid unnecessary guesswork, the following information is expected: hyperparameters that were searched per model (including options and ranges), the final hyperparameter settings used, number of trials, and settings of the search procedure if applicable. as tuning of hyperparameters is typically performed using specific parts of the dataset, it is essential to note that any modeling decisions based on them automatically invalidate their use as test data. models contemporary models (e.g., @xcite @xcite have very large computational and memory footprints. to avoid retraining models, and more importantly, to allow for replicability, it is recommended to save and share model weights. this may face similar challenges as those of datasets (namely, large file sizes), but it remains an impactful consideration. in most cases, simply sharing the best or most interesting model could suffice. it should be emphasized that distributing model weights should always complement a welldocumented repository as libraries and hosting sites might not be supported in the future. ## experiments & analysis experiments and their analyses constitute the core of most scientific works, and empirical evidence is valued especially highly in ml research @xcite . therefore, we discuss the most common issues and counter-strategies at different stages of an experiment. ## publication in this section, we discuss some additional trends in the dl field that researchers should consider when publishing their work, even though they might not directly be related to reproducibility & replicability. citation control while frequently, researchers cite non-archival versions of papers, the published version of a paper is peer-reviewed, increasing the probability that any mistakes or ambiguities have been resolved. in appendix c, we suggest tools to verify the version of any cited papers. hardware requirements the paper should report the computing infrastructure used. at minimum, the specifics about the cpu and gpu. this is for indicating the amount of compute necessary for the project, but also for the sake of replicability issues due to the non-deterministic nature of the gpu @xcite . moreover, @xcite demonstrate that test performance scores alone are insufficient for claiming the dominance of a model over another, and argue for reporting additional performance details on validation data as a function of computation budget, which can also estimate the amount of computation required to obtain a given accuracy. environmental impact the growth of computational resources required for dl over the last decade has led to financial and carbon footprint discussions in the ai community. schwartz et al. (2020) introduce the distinction between red ai-ai research that seek to obtain state-of-the-art results through the use of massive computational power-and green ai-ai research that yields novel results without increasing computational cost. in the paper the authors propose to add efficiency as an evaluation criterion alongside accuracy measures. hershcovich et al. (2022) advocate for the usage of a climate performance model card, in which energy and emission statistics are being detailed. strubell et al. (2019) approximate financial and environmental costs of training a variety of models (e.g., bert, gpt-2). in conclusion, to reduce costs and improve equity, they propose (1) reporting training time and sensitivity to hyperparameters, (2) equitable access to computation resources, and ## discussion since previous sections have emphasized the need to overhaul some experimental standards, we dedicate this last section to discuss some structural issues that might pose obstacles to this. compute requirements specifically with regard to statistical significance in section 5, there is a stark tension between the hardware requirements of modern methods @xcite and the computational budget of the average researcher. only the best-funded research labs can afford the increasing computational costs to account for the statistical uncertainty of results and to reproduce prior works @xcite . under these circumstances, it becomes difficult to judge whether the results obtained via larger models and datasets actually constitute substantial progress or just statistical flukes. at the same time, such experiments can create environmental concerns @xcite . 8 the community must decide collectively whether these factors, including impeded reproducibility and weakened empirical evidence, constitute a worthy price for the knowledge obtained from training large neural networks. incentives in publishing as demonstrated by figure 2 , nlp has gained traction as an empirical field of research. at such a point, more rigorous standards are necessary to maintain high levels of scholarship. unfortunately, we see this process lagging behind, illustrated by repeated calls for improvement @xcite . why is that so? we speculate that the reason for many of these problems are caused by adverse incentives set by the current publishing environment: as the career of researchers hinges on their publications and more rigorous experimental standards are often not required to get published, reproducing and reproducible works are not rewarded. instead, actors are tempted to "rig the benchmark lottery" @xcite , since achieving state-of-the-art results remains important for publishing @xcite . as of now, better experimental standards often do not increase the acceptance probability: the more details are provided for replicability purposes, the more potential points of criticism are exposed to reviewers. this state of affairs might still seem like progress to some, but @xcite show how an increased amount of papers actually leads to slowed progress in a field, making it harder for new, promising ideas to break through. furthermore, raff (2022) shows that reproducible work can actually have a positive impact on a paper's citation rates, and thus should be more embraced. culture change how can we change this trend? as researchers, we can start implementing the recommendations in this work in order to drive bottomup change and reach a critical mass @xcite . as reviewers, we can shift focus from results to more rigorous methodologies @xcite , and allow more critiques and reproductions of past works and meta-reviews to be published @xcite . as a community, we can change the incentives around research and experiment with new initiatives. @xcite and @xcite give recommendations on how to improve the peer-review process by better paper-reviewer matching and paper scoring. other attempts are currently undertaken to encourage reproduction of past works. 9 other ideas change the publishing process more fundamentally, for instance by splitting it into two steps: the first part, where authors are judged solely on the merit of their research question and methodology; and the second one, during which the analysis of their results is evaluated @xcite . in a similar vein, @xcite recommend a procedure similar to clinical studies, where whole research projects are pre-registered, i.e., specifying the parameters of research before carrying out any experiments @xcite . the implications of these ideas are not only positive, however, as a slowing rate of publishing might disadvantage junior researchers @xcite . ## conclusion being able to (re-)produce empirical findings is critical for scientific progress, particularly in fastgrowing fields like nlp @xcite . to reduce the risks of a reproducibility crisis and unreliable research findings @xcite , experimental rigor is imperative. being aware of possible harmful implications and to avoid them is therefore important. every step carries possible biases @xcite . this paper aims at providing a toolbox of actionable recommendations for each research step, and a reflection and summary of the ongoing broader discussion. with concrete best practices to raise awareness and call for uptake, we hope to aid researchers in their empirical endeavors.