On “Scientific Debt” in NLP : A Case for More Rigour in Language Model Pre-Training Research (2023)

Abstract: This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: “How exactly does each factor contribute to the progress that we have today?” We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how — under comparable conditions where the baselines are tuned to a similar extent — these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights. We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.

## introduction in recent years, language models that are pretrained on large amounts of data have become the foundation models @xcite for achieving state-of-the-art results on many nlp tasks @xcite @xcite , and performed well at novel tasks through demonstrations @xcite . hence, given the vast potential of pre-trained language models (plms), substantial effort has since been dedicated into develop-ing the next state-of-the-art plm, whether through new objective functions @xcite @xcite @xcite , increasing model size @xcite @xcite , better data filtering @xcite , or making a better use of the compute budget by training longer @xcite . to a large extent, this rapid progress is made possible by a strong emphasis on designing better foundation models and achieving new state-ofthe-art results. in pursuit of this goal, each new plm often leverages all possible means for improving performance: indeed, it is often the case that a new state-of-the-art plm paper not only proposed a new pre-training loss function as its key novelty, but was also trained on more data, used a larger model size, and benefited from the latest proven hyper-parameter settings and tricks of the trade compared to earlier plms. we argue, however, that progress through these practices also comes at a cost: as each new plm differs from earlier one in multiple dimensions at once, it has become increasingly harder to (i) disentangle how these different components contribute to the model performance and progress that we observe today; (ii) understand which approaches work well under what circumstances; (iii) distill generalizable patterns and understand how well each result would transfer to new tasks, datasets, and settings (e.g., in low-resource data and compute scenarios); and (iv) replicate prior results and conduct the appropriate credit assignment for the techniques and prior work that are most responsible for our progress today. much like how technical debt often arises when developing new software at breakneck speed, 1 we propose the term "scientific debt" to refer to the issues above that arise due to these plm research practices. in this evidence-based position paper, we argue that scientific progress in nlp should strike a delicate balance between achieving the best performance on various benchmarks and leaderboards -an area where great progress has been made in recent years -and also on understanding: how exactly does each different component affect the plm performance that we observe today? while this question is difficult to answer in light of the current plm research practices that conflate different sources of model performance, we encourage the community to dedicate more effort into disentangling the performance gains from these interacting factors. doing so would pave the way for achieving more progress in the future, in a way that is more scientifically rigorous, generalizable, reproducible, and well-grounded in a better understanding of how well each approach works under different settings. we begin by motivating the importance of disentangling multiple possible factors of model improvement through an analogy with medicine, and discuss their parallels for plm research ( §2). we then provide empirical evidence by revisiting the success of bert @xcite , and demonstrate that prior plms like elmo @xcite and gpt-1 @xcite can, in fact, achieve nearly the same performance (∼ 1% difference in aggregate glue) as bert under comparable experimental conditions ( §3). these findings serve to (i) further our understanding of the effectiveness of the masked language modelling loss compared to prior approaches, under comparable experimental conditions; and (ii) provide an example for how such work can yield valuable new insights regarding which approaches should be used under what conditions. we then conclude with several key recommendations, lessons learnt, and calls for change that would encourage and facilitate this line of work in the future, and accelerate our progress towards resolving the scientific debt that arises due to current plm research practices ( §4). reasons for the scientific debt. while answering the questions above would drive betterinformed progress in the field, in practice there are multiple barriers to doing so; to some extent, these barriers account for why this scientific debt arises in the first place. these include (i) variations in the choice of hyper-parameters and experimental settings, which can result in a large variance in model performance @xcite ; (ii) the proprietary and opaque nature of many plms and their training datasets -especially large-scale ones -rendering standardization difficult; (iii) the increasing computational costs of large-scale plms, which increases the costs of running multiple pre-training experiments (e.g., with different model sizes or training data); and (iv) a strong emphasis in the field for achieving state-ofthe-art results -indirectly creating an incentive to spend all of one's compute, time, and effort to achieve the best results, albeit sometimes at the expense of scientific rigour, performing rigorous experiments under comparable conditions, tuning the baselines, and disentangling different possible sources of model improvements. we revisit these barriers, and outline our recommendations in §4. the costs and benefits of scientific debt, and why we should address it. in practice, the community goes into scientific debt because there are certain benefits of doing so. indeed, by rapidly sharing and publishing new progress and state-ofthe-art models -even when we do not yet fully understand how each factor contributes to the final performance of the model -the community is able to share, use, and build on exciting findings (e.g., more accurate and faster models, etc.) much more quickly at a time of rapid progress. yet on the other hand, accumulating too much scientific debt -without a good plan to address it and eventually pay it off -also carries an important risk: the lack of fair comparisons and proper ablations can lead the community down the wrong path, make sub-optimal choices, and waste precious community time, effort, and computational resources in the wrong direction. paying off this scientific debt would crucially (i) enable the community to direct our collective efforts and resources into the research directions that matter the most for improving model performance, (ii) understand what factors enable plms' remarkable success today, and (iii) better comprehend the trade-offs between different approaches under various types of settings. large variability in current plms. to illustrate the extent of this issue, we summarize several key design choices behind some well-known plms in at the time of its release, bert @xcite attracted a lot of attention by virtue of its strong performance on many tasks, outperforming earlier plms like elmo @xcite and gpt-1 @xcite by substantial margins. at its core, bert combines the following: based on the different components of bert above, it combines previous known techniques with a key novelty: the masked language modelling objective, which -unlike prior approaches like elmo and gpt-1 -enables it to leverage and fuse bidirectional context at pre-training. nevertheless, bert differs from its prior approaches on many other factors (e.g., the pre-training data corpus and size, model size, using transformers vs lstms, length of the training cycle, tokenizer, etc.). this renders a principled comparison difficult, and makes it hard to isolate the importance of the masked language modelling objective from other factors that also affect model performance. we therefore here ask: ## paying off the scientific debt: recommendations and lessons learnt we proceed to outline several key recommendations and lessons learnt for encouraging, incentiviz-9 ensembling independently pre-trained and fine-tuned leftto-right and right-to-left models is a late-fusion approach, without the need for elmo's joint fine-tuning stage. establish standard, publicly available pretraining corpora at multiple data scales. as seen in §3, the size and quality of the pre-training data is an important driver behind model performance @xcite , which makes a rigorous comparison between different plms difficult. hence, our first recommendation is to establish standard pre-training corpora that are publicly available. 10 we further recommend releasing the pre-training corpora under multiple data scales, as approaches that work best under strict compute or data resource requirements may be different from the case where there is a large amount of compute and data available ( §3, @xcite . note that this does not mean that we are discouraging the use of non-standard or even-larger corpora than those that are publicly available. on the contrary, researchers should continue to push the boundaries of what is possible by training on more, better quality, and more recent data. in such cases, we recommend researchers to also release versions of their models that are trained on the standard pre-training corpora -above and beyond the version trained on proprietary & large-scale data that would presumably be necessary to achieve a new state-of-the-art -in order to facilitate a fair and principled comparison with prior work. we encourage the community to continually release new standardized pre-training datasets as time passes to avoid the effect of pretraining data staleness @xcite . explicitly delineate the different types of contributions behind each work, including both the key novelty and engineering contributions. we recommend that plm research explicitly state the key novelty behind each work (e.g., the bidirectional masked lm loss for bert), delineate and explicitly state other contributions (including engineering ones) and design choices that can impact performance, and outline how these differ from prior work (e.g., better model partitioning for train invest comparable effort into tuning both the baselines and the newly-proposed models. in practice, many of the contributions (e.g., better hyper-parameters, better data, etc.) would also be applicable to the baselines. we recommend each plm work to discern which of their design choices can also be applied to the baselines, and apply those techniques in order to create stronger baselines that may nevertheless rival the performance of more recent models ( §3, @xcite . more extensive ablation studies. when proposing multiple contributions at once (as many plm papers do), we recommend conducting as many ablation studies as is feasible to isolate the impact of each component under comparable conditions. in light of recent trends where models -including open-sourced ones -are publicly released without technical reports or papers that outline technical details regarding model evaluation and benchmarking @xcite , we argue that our recommendation for conducting more thorough evaluations is even more critical. better credit assignment is needed. as shown in reward and encourage a line of work that focuses on understanding (not just those that chase a new state-of-the-art), even when they are imperfect. the current, rapid pace of the field provides an incentive to spend one's (finite) computational resources and effort for building the next state-of-the-art, albeit at the expense of scientific rigour and principled comparisons. given a finite amount of compute, there is arguably more incentive in tuning one's proposed approach through all possible means (e.g., using larger datasets and larger models, training for longer, etc.), topping the leaderboards, and publishing the paper, even if this leaves no computational resources to tune the baselines and conduct rigorous ablations. furthermore, the rapidly increasing cost of training ever-larger plms means that any principled comparisons are most likely imperfect ( §7) -e.g., how do our findings in §3 change with models that are trained for longer, like roberta? or with encoderdecoder models like t5? or in other languages? indeed, our experiments in §3 are fairly narrow in scope, involving only three non-recent models (bert, elmo, gpt-1) and a training dataset that is small by today's standards. yet due to the rigorous hyper-parameter tuning of all three models, conducting these principled comparisons required an enormous amount of compute resources -equivalent to training 10 berts from scratch. this cost would have been even higher with the inclusion more models, languages, and larger datasets. on this point, we remark that doing such principled comparisons -even when they are limited in scope and done on smaller models -still contributes towards paying off the scientific debt, better understanding where our current progress is coming from, and deriving valuable insights that can contribute to the development of next generation plms. we additionally call on those in our community who serve as reviewers to recognize and reward these types of research contributions, which are complementary (if perhaps equally important) to a parallel line of work that pushes the state-ofthe-art in plm research through all possible means. we need more comprehensive plm scaling laws. our experiments and recommendations still leave a major open question: how can we scale these kinds of investigations to much larger plms, which are much more computationally expensive? to that end, scaling laws @xcite provide an account of how plm performance changes with respect to different factors, allowing us to accurately extrapolate that a plm with x parameters and y training steps should achieve a perplexity of z. however, we argue that current scaling laws are still overly narrow in scope: concretely, existing scaling laws often only apply to decoder-only / unidirectional plms, and only provide an account of how their performance changes with respect to (i) model size and (ii) the number of training tokens. we call on the community to develop more comprehensive scaling laws that take into account and characterize how other factors impact lm performance and downstream behavior, including how model performance and behavior change with respect to the choice of the objective function and model hyper-parameters, and the quality of the pre-training data. the existence of such scaling laws -which can happen by pooling community data on various plm pre-training runs and their corresponding perplexity and downstream performance -would allow other researchers to accurately extrapolate how their findings would generalize to other plm model sizes, objective functions, etc. most importantly, comprehensive scaling laws can disentangle and quantify how these different factors contribute to determine the final model performance under various experimental conditions. how conducting rigorous experiments and ablation studies can lead to new state-of-the-art results. lastly, we argue that conducting rigorous experiments and ablation studies for paying off the scientific debt should not necessarily come at the expense of achieving a new state-of-theart. in contrast, doing so can be a key ingredient for building the next state-of-the-art plms. in 2020, kaplan et al. (2020) proposed a seminal scaling law that showed how larger plms are more sample-efficient, and that one should always increase model size when increasing the pre-training compute budget, leading the community to develop ever-larger plms in response (rae et al., 2021; smith et al., 2022, inter alia). nevertheless, subsequent rigorous experiments from hoffmann et al. @xcite demonstrated that the optimal pre-training compute allocation should, in fact, also be scaled in another dimension: the amount of pre-training data that the model is trained on. this insight was then used to build smaller, more efficient, and cheaper-to-run plms that, at the time of its release, achieved new state-of-the-art results that outperformed much larger plms that were under-trained in comparison. going forward, we conjecture that rigorous experiments and ablation studies that look at factors above and beyond model size and data quantity, such as the quality of the pre-training data, the exact hyper-parameters, the pre-training objective, etc., will not only be useful to understand how these factors improve performance and thus pay off the scientific debt, but also form a key ingredient for building the next generation of better plms. ## related work a number of prior work has made progress in disentangling the impact of different language modelling pre-training objectives by conducting principled ablation studies under comparable experimental conditions @xcite @xcite . however, some of the recently released models do not provide any technical details on how they are trained, such as chatgpt foot_8 and gpt-4 @xcite . we discuss these in an extended related work section (appendix c), but briefly remark on how our findings complement theirs. first, we revisit and augment elmo -which incorporates a degree of bidirectionality at fine-tuning (albeit not at pre-training) -with transformers and whole model fine-tuning, facilitating a fair comparison with bert. we show that the resulting elmo achieves competitive performance with bert on glue; to our knowledge, no such elmo baseline -or an even simpler ensemble of a left-to-right and right-to-left plm -was explored in prior work. while our work shares several similarities with @xcite , our work differs by virtue of being a position paper that focuses on an important issue in the field (i.e., the lack of fair comparisons between past plms), and chart the way forward for mitigating this issue. our experiments in this work mostly aim to provide an example of this issue in action, and form the basis for some of the lessons learnt and recommendations that we outline in §4. unlike @xcite , our experiments are not designed to achieve new stateof-the-art results. moreover, above and beyond our empirical contributions, we outline key recommendations that would encourage and incentivize this line of work; we hope that these recommendations would be adopted by the broader community, with the aim of accelerating progress towards resolving the scientific debt in foundation model research. ## conclusion recent rapid progress within the plm literature has led to tremendous advances within nlp. despite this progress, current plm research practices that change multiple different things at once -often without proper ablation studies and conducting principled comparisons that disentangle the impact of different components -have introduced certain issues that we call "scientific debt". through experiments that disentangle the contribution of bert's bidirectional masked lm objective through principled comparison with prior work, we demonstrate how asking "which factors contribute the most to the model performance that we observe today?" can lead to valuable new insights, including the existence of simple yet stronger-thanexpected and more efficient baselines. we outlined several recommendations that would encourage and incentivize this line of work that aims to better understand how each factor contributes to the rapid progress of our plms today, and better address the ongoing issue of accumulating scientific debt within our current plm research literature.