A Two-Sided Discussion of Preregistration of NLP Research (2023)

Abstract: Van Miltenburg et al. (2021) suggest NLP research should adopt preregistration to prevent fishing expeditions and to promote publication of negative results. At face value, this is a very reasonable suggestion, seemingly solving many methodological problems with NLP research. We discuss pros and cons - some old, some new: a) Preregistration is challenged by the practice of retrieving hypotheses after the results are known; b) preregistration may bias NLP toward confirmatory research; c) preregistration must allow for reclassification of research as exploratory; d) preregistration may increase publication bias; e) preregistration may increase flag-planting; f) preregistration may increase p-hacking; and finally, g) preregistration may make us less risk tolerant. We cast our discussion as a dialogue, presenting both sides of the debate.

## encouraging confirmatory research we let socart and zeny discuss whether preregistration will succeed in distinguishing between confirmatory and exploratory research. a decade ago, when preregistration was being implemented and discussed in epidemiology, the worry that preregistration would introduce a bias against "the end of the research spectrum that constitutes the quirky, brilliant work that is not enterprise-driven" (sørensen and rothman, 2010), was the main concern among its opponents. socart and zeny discuss the consequences of insisting on a distinction that is not trivial to uphold in practice. socart: it seems to me, dear zeny, that many nlp projects are driven not by an explicit hypothesis, but by a desire to understand the behavior of a model, to be able to characterize its strengths and weaknesses, or by a simple gut feeling that at the locus of interacting variables, interesting dynamics can be observed. zeny: can you provide me with an example? socart: certainly. pires et al. (2019), e.g., showed that knowledge encoded in multilingual bert (devlin et al., 2019), could be transferred across languages-even across scripts, that such transfer worked best between typologically similar languages, that it could process code-switching and find translation pairs. they also showed systematic deficiencies affecting some language pairs. how would they have foreseen these findings? or even the dimensions that turned out to be of interest? even if they had foreseen how they wanted to explore transfer across scripts and typological classes, what if genealogy or demography turned out to be more interesting than typology? foot_4 zeny: i am unconvinced that preregistration would be a serious obstacle to such work. pires et al. (2019) could have defined the search space in advanceor maybe this is exploratory work that would not have to register in the first place? remember also, socart, that the preregistered plan can be updated and refined in the course of a research project. plans can be revised, but this does not cancel out the benefits of planning. socart: preregistration may accommodate deviation from the plan, but would risk losing its benefit if researchers were allowed to preregister too many hypotheses or update their plans too frequently. let us illustrate this with another example. zhao and bethard (2020) study how bert models' learned self-attention functions change during fine-tuning to reflect the target task. they find this to be the case only in smaller models; with more parameters, the change disappears. imagine now that their hypothesis was confirmed only for select combinations of positional encodings, regularizers and optimizers. zeny: this sounds suspiciously like a case of forking paths. dror et al. (2017) warned us about this risk, encouraging us to at least validate our hypotheses on multiple datasets to reduce the chance of phacking. 11 again, preregistration would not be required for all research. socart: so if authors submitted exploratory work for peer review, would reviewers then decide if bypassing preregistration was appropriate? zeny: yes. preregistration clarifies the distinction between exploratory and confirmatory research. socart: but what if pires et al. (2019) had pointed to earlier work already hypothesizing that transfer works best between typologically similar languages? would this not have made their research confirmatory in the eyes of their readers and therefore in need of preregistration? 12 zeny: it very well might have. if they consider it exploratory, they should also point to alternative hypotheses that would explain different results. socart: moreover, if preregistration becomes a badge of honor or increases your chances of getting your work accepted, because the findings have a different air of trustworthiness, 13 would this not be a reason to encourage your students to perform confirmatory rather than exploratory research? preregistration would, in other words, inject a bias toward confirmatory research into nlp. zeny: i, for one, would welcome this kind of bias. socart: but would you agree that granting her the freedom to hark most likely reduces her temptation to cherry-pick? zeny: this would simply reclassify ann's work as exploratory rather than confirmatory. i see no reason why preregistration should not allow this. socart: the two researchers both departed from their original plans, but ann's willingness to depart from her original hypothesis serves us better than bob's cherry-picking. in this way, harking can prevent a researcher from taking on a fishing expedition. ✓ preregistration should allow for re-classification of confirmatory research as exploratory research. foot_7 5 solving publication bias? sørensen and rothman (2010) argue against preregistration solving publication bias, because researchers can still selectively register studies after preliminary data explorations. imagine hippocrates, the greek physician, was asked to preregister his vivisection experiments. if hippocrates was studying 10 soldiers with brain lesions, what would prevent him from using one soldier to generate hypotheses, preregister those and conduct the final experiments on the remaining nine? or worse, peek at all, preregister and go back to the data? socart: say hippocrates has two hypotheses about the soldiers, such as that the heart is the seat of compassion and that the brain is the seat of rational thought. upon preliminary exploration, he sees many soldiers have turned cold-hearted by the atrocities of war, but few complain of heartaches. nearly all soldiers who are delusional or suffer from memory loss, also suffered blows to their heads. hippocrates pursues and preregisters only the hypothesis that the brain is the seat of rational thought. he has now preregistered, not a prediction, but a post-diction, ignoring the negative result. zeny: but socart, did you not, a moment ago, argue that preregistration would dampen the creativity of research by preventing fishing expeditions and harking? @xcite in theory, yes. preregistration will dampen creativity if properly sanctioned, but i am skeptical that this would be practically possible, rendering preregistration ineffective, an unnecessary administrative burden for all-and a bottleneck for the honest few. zeny: if preregistration prior to data collection is encouraged, this would solve the problem, no? 16 socart: surely, but this would mean only one preregistered study per dataset. since few nlp papers introduce new datasets, this would render preregistration ineffective for the vast majority of nlp research. zeny: this is a good point, socart, but communitywide overfitting to benchmarks is a vice, not a virtue. if preregistration encourages the introduction of new test datasets, that's a good thing, no? some even argue that all papers should ideally introduce new test data. to the contrary? . . . in which socart and zeny continue to discuss whether preregistration could actually make publication bias worse. so-cart suggests that preregistration could amplify publication bias, if positive results are still preferred over negative ones and preregistration forces researchers to focus on predictably positive results, arguably a small subset of the positive results. if papers are accepted on the basis of preregistration, this could increase an arguably already existing bias toward incremental improvements. @xcite socart: you say preregistration will make it easier to publish negative results, because studies are evaluated prior to obtaining results? zeny: that is correct, socart. socart: . . . but do we really know why there are so few nlp papers about negative results? see, in nlp, negative results are much harder to establish than positives. if i want to show that selfattention or weight averaging does not lead to improvements for some problem, i need to show that holds across all implementations, all architectures and all available datasets. the value of @xcite this is implicit in preregistration in scientific fields where data is not re-used, e.g., in psychology (wiseman et al., 2019). @xcite lash and vandenbroucke (2012), for example, argue that: "prespecified hypotheses often take little risk, invoke little imagination and stray only a short distance from what is already well understood." a report stating that for one such combination, self-attention didn't do much, would be next to nothing. isn't that the real explanation for the skew in the nlp literature? zeny: negative results are key to scientific progress (barwich, 2019), but are hard to establish if they are very general. published positive results often overclaim their generality. both should be confirmed only by accumulated evidence in diverse settings. foot_8 socart: no, no, you fail to see there's a qualitative difference, zeny! imagine if ann was evaluating self-attention for sentiment analysis. to answer the hypothesis that self-attention works in the positive, she just needs a significant result in a single setting. in contrast, in order to establish a negative result, she has to explore all possible settings. @xcite how would preregistration make establishing a negative result less formidable a challenge? zeny: i agree that it would not. my only claim is that evaluating studies prior to obtaining results would prevent any bias on behalf of peer reviewers to evaluate negative results more harshly. socart: . . . but in reality, we do not know if such a bias exists, or whether it is only fair that such a bias exists, because the bar by definition should be higher for negative results? flag-planting is one of the motivations for preregistration for van miltenburg et al. ( 2021 ), but exclusive preregistration may also, conceivably, have the opposite effect. socart: say ann and bob get the same great ideae.g., to evaluate the sensitivity of textual entailment models to presupposition projection-and worry that they will be scooped before getting around to publishing it. ann and bob now follow two different strategies: bob rushes to preregister a study hypothesizing that state-of-the-art models are sensitive to such phenomena, while ann rushes to run the experiments and publish the paper. zeny, which strategy is better for science? zeny: i would say it's bob's, since rushed experiments are more likely to be flawed. socart: but bob plants his flag faster than ann, essentially scooping her. by doing so, bob discourages ann from pursuing this idea by planting his flag first. what if bob fails to conduct proper experiments altogether? had it not been for preregistration, both researchers would have pursued their idea, providing mutual replication and increasing the likelihood of the idea materializing into an actual result. zeny: some would say this is one of the advantages of preregistration: ann pursuing the same idea would have been a waste of time. socart: this assumes ann and bob would have conducted their research in exactly the same way and that none of them were prone to error. in other words, that researchers are machines that simply execute their unambiguous experimental protocols. i think preregistration just moves flagplanting to earlier in the research process, lowering the bar for researchers to plant their flags, since less work is required to plant a flag. and when a bar is lowered, more researchers are likely to plant more flags. zeny: van miltenburg et al. (2021) explicitly encourage concurrent work. socart: yes, i did read that passage, but they do not discuss how preregistration would impact concurrent work. do they envisage a review system in which ann is allowed to follow up on the idea that bob preregistered? zeny: i'd do that, in the spirit of open science. socart: such inclusive preregistration would clearly discourage protectionist researchers from preregistering their studies. if a preregistered study is up for grabs for other research labs, labs with more resources could likely wrap up the experiments faster than the researchers who registered it. zeny: . . . unless we envisage a review system allowing ann to preregister the same study, giving ann and bob equal chances to pursue the study. socart: this would be equivalent to telling reviewers of a paper to consider as "concurrent" any other work published within the last 1-2 years (assuming this is the approximate life span of a research project), including preregistered studies. today, reviewers are told to disregard work published within the last three months, but already, reviewers seem to ignore this guideline in practice, presumably because they do not want to compromise the fast turn-around in nlp research. zeny: but shouldn't we incentivize slow science? many nlp papers neglect related work and keep reinventing the wheel. we need deeper analysis to enable disruptive scholarship and novel ideas. @xcite socart: slow science also has disadvantages. fast turn-around has had many positive effects on nlp, including rapid replication. projects can become "too big to fail," causing confirmation bias. @xcite lowering false positive rates is important, but so is healthy distrust in published results. inflation bias, also known as p-hacking, refers to selective reporting to produce statistically significant results. søgaard et al. (2014) lists several p-hacking techniques used, perhaps inadvertently, in nlp papers. if a statistically significant result is seen as the key to getting your paper accepted, researchers are presumably willing to go far to squeeze out a small p-value. but if preregistration facilitates the publication of negative results, it seems it would also reduce the incentive to engage in so-called p-hacking, e.g., obsessive fiddling with data and models until reaching the magical @xmath0.01. it has been noted, however, that preregistration leaves plenty of room for p-hacking (bakker et al., 2020). generally, eliminating p-hacking entirely is unlikely when career advancement is assessed by publication output, and positive results are favored by scientific peers (head et al., 2015). socart and zeny discuss whether preregistration will reduce or amplify the incentive to engage in p-hacking: socart: imagine ann again, who is now evaluating if self-attention is helpful for sentiment analysis. say she preregisters the hypothesis that selfattention is helpful, only to find that her first results are negative. we would now like ann to go ahead and acknowledge the negative results on print, right? however, as we just saw, when your first results are negative, more results are typically needed to draw a firm, negative conclusion that self-attention does not help. sometimes more data collection is needed and more human evaluations may be needed. pursuing the negative result will, in other words, be a lot of work. zeny: but very important! socart: preregistration increases the amount of work that goes into moving your focus to establishing a negative result: you will need to augment your preregistration with information about the experiment, your new hypothesis and the new experiment you plan to perform. zeny: documentation has to be light-weight. socart: . . . but preregistration would get people more invested in their ideas and bias them in how results are interpreted. when people go on record with a study description, they will defend why it's reasonable and likely leading to a positive result. researchers are always prone to confirmation biases, but now social expectations and reputation will amplify their existing biases. this would lead to the opposite of the effect intended. ## risk tolerance attempts to reduce false positives tend to also lead to reductions in true positives. many applications require near-zero false positive rates, but most nlp experiments show low risk of direct negative impact on society or individuals therein, foot_10 as indicated by the relatively few papers receiving ethical reviews. hence, we can afford to take risks and explore hypotheses that end up wrong. parascandola (2010) reminds us how this is a key ingredient in increasing knowledge and reducing uncertainty, getting us off the beaten track. nlp benefits from being frequently wrong and implementing preregistration to prevent false positives has a drawback. in §9, we will argue that what is important is to balance preregistration with our risk tolerance. socart: imagine ann works on hate speech detection for a social media company. bob works on topic classification of social media posts at the same company. they both validate and evaluate the models in the wild on beta users. they both can use logistic regression and svms. svm is sometimes superior, but exhibits more variance across hyper-parameters. if i were to advise ann or bob to use logistic regression, who then? zeny: probably ann, since we can tolerate less risk in her situation. but how would preregistration affect the risk tolerance of researchers? socart: imagine if you were asked today to carry a solid bottle of olive oil over to plato's house and tomorrow to bring him a fragile, beautiful vase decorated with gold. on which of the two days would you be more inclined to run there? zeny: today, but how is this relevant? what are the tasks where we can afford to 'run'? socart: for tasks in which false positives are associated with high risk, we should hedge our bets by preregistering conservative hypotheses; for other tasks, this sort of inhibition of is unfortunate. zeny: this is exactly why exploratory research still has a place in a world with preregistrationnamely, for tasks where we can tolerate risk. one may argue that risk mitigation is not what preregistration is for. the purpose of institutional review boards (irbs) and ethics reviewing is to flag and prevent too risky studies (irbs focus on risk to human participants, while ethics reviewing also addresses potential applications). we have three reasons to think preregistration requirements should depend on expected risk: (a) it is impossible to review the implications of a study before you have a solid study plan. if preregistration includes risk assessment, this could provide input for irbs and ethics reviewing (or, in a more distant future, be part of the same process). (b) a partial roll-out of preregistration may help us balance type 1 and type 2 errors. expected risk affects the cost of false positives and hence the optimal balance between type 1 and type 2 errors. since bureaucracy, by the end of the day, also incurs a cost on society, this reinforces our belief that mechanisms should be implemented only for where there is direct impact on society at large. (c) finally, ethics reviewing is typically part of the standard review process, i.e. after the fact and can therefore not respond to malpractice in the experimental design or prevent publication of preprints. overly cautious preregistration practice may, in sum, decrease our true positive rate and add bureaucratic overhead to research practices without proper motivation. an all-over-the-map roll-out of preregistration would change the risk tolerance in research and society, just like registration and documentation has increased risk sensitivity in the past. simultaneously, evaluating risk early on has clear advantages over the current review process. ✓ preregistration may lower our risk tolerance. ## a proposal we have tried to present pros and cons of preregistration. if we have focused a bit more on the cons, this is only because van miltenburg et al. (2021) did a great job highlighting its advantages. we will, if anything, argue for only a partial roll-out of preregistration of nlp research. preregistration is a way to minimize harms of nlp research, but only when risk is high. to motivate this, consider, as first noted by lash and vandenbroucke (2012), two seemingly opposed arguments for preregistration: a) preregistration counters the suppression of (negative) results. b) preregistration identifies false positives. lash and vandenbroucke (2012) argue that while (b) is a valid argument for preregistration of clinical trials, it is not a valid argument for preregistration in the context of mere "accumulation of evidence" (lash and vandenbroucke, 2012). here, concerns about balancing type 1 and type 2 errors disappear. preregistration mitigates risks associated with research, reducing potential harms, but at the cost of scientific progress. this calls for a cost-benefit analysis: how much risk can be tolerated for what potential gains? one way to frame this discussion in nlp is to ask how afraid we should be of being wrong. in clinical trials, there is a significant cost to being wrong. in biomedical studies, false positive rates have been found to be around 14% (jager and leek, 2014). whatever the number is for nlp, lowering it by adding more checks, will lead to a drop in the true positive rate. if a false positive could result in human tragedy, the price of a lower true positive rate is worth paying, but in nlp, the cost of a false positive is often paid for in compute and human hours. while both can be scarce resources, the open access nature of nlp makes being wrong less dangerous, since mistakes are quickly corrected. @xcite zeny would object that pretraining of language models is not easily reproducible (bender et al., 2021). pretraining very large language models should maybe be required to preregister and this would possibly require revising the questionnaires provided by van miltenburg et al. (2021). another concern is how nlp contributes to social and cultural inequality (hershcovich et al., 2022). if nlp research is likely to help some more than others, this may be reason to require preregistration. here, the questionnaires provided by van miltenburg et al. (2021) would also be insufficient. @xcite so what we propose here reflects a middle-ofthe-road position on preregistration. the idea is to limit preregistration to research for which our risk tolerance is low. this prevents most of the adverse effects of preregistration, e.g., publication bias, flag-planting and p-hacking. nlp research is subject to irb and ethics reviewing, but we believe this should be merged with preregistration ( §8). reports should be reviewed and reviewers follow the submission ( §2). @xcite currently, we accept and reject papers through blind peer-reviewing, but some papers are accepted conditional on a positive ethics review. we propose a reviewing procedure in which some work is only accepted conditional on the work having already been registered with positive reviews. for researchers, this would mean you need to get your preregistered reports accepted, before you initiate the research project. once completed you will send the final submission in for a new set of reviews, hopefully by the same reviewers. this procedure is somewhat cumbersome and has all the disadvantages we discussed above. therefore, it should only be used when it is deemed necessary, i.e., when the expected risk of the nlp research is sufficiently high. a paper which a) is confirmatory and b) concerns an application for which risk tolerance is low, can be rejected for not having preregistered. we noted the necessity of allowing preregistered research to re-classify as exploratory, i.e., conditional acceptance for non-preregistered, flagged research, if it explicitly labels itself as exploratory. this would lead to three different categories of accepted papers: (i) non-preregistered (confirmatory or exploratory) research, (ii) preregistered, confirmatory research and (iii) non-preregistered research marked explicitly as exploratory. this would, we argue, give us the advantages of preregistration where they are most needed, e.g., where false positives are associated with very high risk. we left one important thing in the open for now: how do we fairly decide if a research subject and protocol warrants low risk-tolerance? ethics review board members are already asked to flag work that 'exhibits an increased risk of harm outside the current norms of nlp or cl research'. @xcite this can be hard to determine, but board members already have to make this difficult decision. ethics reviews could learn from established risk assessment frameworks (schwerdtner et al., 2020). ## conclusion our two-sided dialogue has discussed pros and cons of preregistration in nlp, building on similar discussions in epidemiology. what opponents elsewhere have proposed as alternatives to preregistration is already found in nlp research: open access, common repositories and data sheets (lash and vandenbroucke, 2012). preregistration, we argue, is less urgently needed in fields that already facilitate replication and where risk of false positives is low. the fast turn-around of nlp research means the advantages of transparency and early feedback are smaller. nevertheless, society's risk-tolerance varies across nlp applications. legal or medical decision support systems are high-risk application areas. here, we need to consider all safety measures on the table, including preregistration.