Preregistering NLP research (2021)

Abstract: Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research.

## introduction scientific results are only as reliable as the methods that we use to obtain those results. recent years have seen growing concerns about the reproducibility of scientific research, leading some to speak of a 'reproducibility crisis' (see @xcite for an overview of the debate). although the main focus of the debate has been on psychology (e.g. through open science collaboration 2015) and medicine @xcite , there are worries about the reproducibility of natural language processing (nlp) research as well @xcite @xcite . the reproducibility debate has led to @xcite manifesto for reproducible science, where the authors discuss the different threats to reproducible science, and different ways to address these threats. we will first highlight some of their proposals, and discuss their adoption rate in nlp. our main observation is that preregistration is rarely used. we believe this is an undesirable situation, and devote the rest of this paper to argue for preregistration of nlp research. munafò et al. recommend more methodological training, so that e.g. statistical methods are applied correctly. in nlp, we see different researchers picking up the gauntlet to teach others about statistics @xcite , achieving language-independence @xcite , or best practices in human evaluation @xcite . moreover, every *acl conference offers tutorials on a wide range of different topics. while efforts to improve methodology could be more systematic (e.g. by actively encouraging methodology tutorials, and working towards community standards), foot_0 the infrastructure is in place. munafò et al. also recommend to diversify peer review. instead of only having journals, that are responsible for both the evaluation and dissemination of research, we can now also solicit peer feedback after publishing our work on a platform like arxiv or openreview. the nlp community is clearly ahead of the curve in terms of the adoption of preprints, and actively discussing ways to improve peer review (acl reviewing committee 2020a,b; rogers and augenstein 2020). to improve the quality of the reviews themselves, acl2020 featured a tutorial on peer reviewing @xcite . another advice from @xcite to adopt reporting guidelines, so that papers include all relevant details for others to reproduce the results. the nlp community is rapidly adopting such guidelines, in the form of @xcite reproducibility checklist that authors for emnlp2020 need to fill in. beyond reproducibility, we are also seeing more and more researchers adopting data statements @xcite , model cards @xcite , and datasheets @xcite for ethical reasons. munafò et al.'s final recommendation, preregistration, means that authors should specify what they are going to do, and what they expect to find, before carrying out their studies @xcite ## preregistration in nlp research to determine what a preregistration for nlp research should look like, we need to consider the different kinds of research contributions in nlp. for this, we use the paper types proposed for coling 2018. 5 these are: computationally-aided linguistic analysis; nlp engineering experiment paper; reproduction/resource/position/survey paper. of these, position papers are less suitable for preregistration, since these are more opinion/experiencedriven, and the process of writing them cannot be formalised. we treat the others below. analysis, experiments, and reproduction papers typically have one or more hypotheses, even though they may not always be marked as such. 6 this means we can ask many of the same questions for these studies as for experimental research. if your study contains an error analysis, then you could ask the more qualitatively oriented questions in the main benefit of asking these questions beforehand is that they force researchers to carefully consider their methodology, and they make researchers' expectations explicit. this also helps to identify unexpected findings, or changes that were made to the research design during the study. resource papers are on the qualitative side of the spectrum, and as such the questions from @xcite , presented at the bottom of survey papers should follow the prisma guidelines for structured reviews @xcite . according to these guidelines, researchers should state exactly where they searched for existing literature, what search terms they used, and what criteria they used to select relevant papers. this increases reproducibility, allows readers to find any gaps in the survey, and avoids a biased presentation of the literature (i.e. only citing researchers you know, or work that fits your preferred narrative). a recent nlp example of a structured review is provided by @xcite . ## registered reports registered reports "[split] conventional peer review in half" @xcite . first, authors submit a well-motivated research plan for review, before carrying out the study (similar to a preregistration). this plan may go back-and-forth between the authors and the reviewers, but once the plan is accepted, the authors receive the guarantee that, if they carry out the study according to plan, their work will be published. as with preregistration, deviations from the original plan are allowed, but these should be indentified in the final report. the main advantage of registered reports is that they provide a means to avoid publication bias. because studies aren't judged on the basis of their results, positive results are equally likely to be published as negative results. as long as the study is deemed valuable a priori, it should get published. an additional benefit of registered reports is that reviews may actually correct flaws in the research design, meaning that we reduce the chance of running an expensive study all for nothing. in the case of nlp research, this may save a lot of energy (cf. @xcite . we are not aware of any nlp journals that offer registered reports, but strongly encourage the nlp community to take steps in this direction. 7 5 feasibility @xcite touch upon the feasibility of preregistration, noting that: "[f]or most of our own research projects this strategy hardly seems possible: in our many applied research projects, we have learned so much by looking at the data. our most important hypotheses could never have been formulated ahead of time." this certainly rings true for nlp as well. however, we should be careful about conclusions that are drawn on the basis of pre-existing data. @xcite note that in such cases, if it is feasible to collect more data, it is good to follow up positive results with a pre-registered replication to confirm your initial findings. one way to do this is to collect and evaluate your model on a new test set (cf. @xcite . this tells us to what extent trained models generalise to unseen data. another idea could be to preregister the human evaluation (or error analysis) of the model output. we believe that preregistration, and especially registered reports, could ease the pressure to publish as soon as possible. if your analysis plan is accepted for publication, you can take as long as you want to actually carry out the study, without having to worry about being scooped. this provides new opportunities for slow science in nlp (also see min-yen kan's keynote at coling 2018). ## questions about preregistration below we address some common questions about preregistration. we thank our anonymous reviewers for raising some of these questions. is preregistration more work? in our experience, preregistration adds little overhead to a research project. especially if a project requires approval by an institutional review board (irb), you need to write a description along similar lines anyway. for projects not requiring irb approval, it is good practice to provide a model card @xcite , data sheet @xcite or data statement @xcite with your model or resource. given the ethical aspects of nlp research, it is advisable to consider all dimensions of your study before you carry it out. moreover, preregistration is a good way to start writing the paper before carrying out the research, a practice advocated by @xcite to maximise the impact of your work. finally, it may be more work to prepare a registered report, but this comes with the benefit of having a pre-approved methodology. once the project is completed, reviewers will not reject your paper based on methodological choices. ## limitations although preregistration is offered as a solution to improve our work, it does not solve all of our problems. @xcite mention three limitations: 1. flexibility. it may be difficult or infeasible for authors to foresee all possible outcomes, and as such there may be gaps in the preregistration, which still allow for flexibility in the analysis. 2. fraud. there is no way to prevent fraudulent researchers from, e.g., creating multiple preregistrations, or falsely 'preregistering' studies that were already run. at some point we just have to trust each other to do the right thing, but increased transparency does make it harder to commit fraud. 3. applicability. preregistration may not be possible for all kinds of studies. as discussed above, it has mainly been developed for quantitative studies (particularly experiments), and there are proposals for the preregistration of qualitative research (haven and grootel, 2019), although we have yet to see whether this idea will catch on. finally, @xcite argue that, although preregistration might offer greater transparency, it does not by itself improve scientific reasoning and theory development. since large parts of nlp are pre-theoretical (we have observed effects but do not have any theoretical explanations for why these effects occur), one might reasonably argue that we should focus on theory development first, before we can carry out any meaningful experiments. ## conclusion we have discussed how preregistration could benefit nlp research, and how different kinds of contributions could be preregistered. we have also proposed an initial list of questions to ask before carrying out nlp research (and see appendix a for example preregistration forms). with this paper, we hope to encourage other nlp researchers to consider preregistering their work, so that they will no longer get lost in the garden of forking paths. still, there is no silver bullet to cure sloppy science. although preregistration is certainly helpful, it does not guarantee high-quality research, and we do need to stay critical about preregistered studies, and the way they are carried out.