How to Train BERT with an Academic Budget (2021)

Abstract: While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford. How can one train such models with a more modest budget? We present a recipe for pretraining a masked language model in 24 hours using a single low-end deep learning server. We demonstrate that through a combination of software optimizations, design choices, and hyperparameter tuning, it is possible to produce models that are competitive with BERT-base on GLUE tasks at a fraction of the original pretraining cost.

## introduction large language models, such as bert @xcite , roberta @xcite , and gpt3 @xcite , have become the de facto models used in many nlp tasks. however, their pretraining phase can be prohibitively expensive for startups and academic research groups, limiting the research and development of model pretraining to only a few well-funded industry labs. how can one train a large language model with commonlyavailable hardware in reasonable time? we present a recipe for training a bert-like masked language model (mlm) in 24 hours in a limited computation environment. our approach combines multiple elements from recent work: faster implementation @xcite , faster convergence through over-parameterization @xcite , best practices for scaling language models @xcite , single-sequence training @xcite , and more. moreover, we conduct an extensive hyperparameter search tailored to our resource budget, and find that synchronizing learning rate warmup and decay schedules with our 24 hour budget greatly improves model performance. when evaluating on glue @xcite , our recipe produces models that are competitive with bert base -a model that was trained on 16 tpus for 4 days. this recipe can also be applied to other corpora, as we demonstrate by training a french-language model on par with camembert base @xcite on the xnli french benchmark @xcite . overall, our findings demonstrate that, with the right recipe and an understanding of the available computational resources, large language models can indeed be trained in an academic setting. ## problem setup we investigate the task of pretraining a large language model under computational constraints. to simulate an academic computation budget, we limit the training time to 24 hours and the hardware to a single low-end deep learning server. 2 using current cloud-compute prices, we estimate the dollar-cost of each training run at around @xmath0100. under these constraints, our goal is to pretrain a model that can benefit classification tasks, such as in glue @xcite . therefore, we follow the standard practice and focus on bert-style transformer encoders trained on the mlm objective @xcite . we retain the standard pretraining corpus of english wikipedia and the toronto bookcorpus @xcite , containing 16gb of text, tokenized into subwords using bert's uncased tokenizer. ## combining efficient training methods to speed up our training process, we combine a variety of recent techniques for optimizing a masked language model. to the best of our knowledge, this is the first time that such techniques are combined and evaluated as a unified framework for training large models with limited computational resources. ## hyperparameter search calibrating hyperparameters is key to increasing model performance in deep learning and nlp @xcite . we re-tune core optimization hyperparameters to fit our lowresource setting, rather than the massive computation frameworks for which they are currently tuned. our hyperparameter search yields substantial improvements in mlm loss after 24 hours of training. ## downstream evaluation we test the performance of our optimized, calibrated 24hbert model on the glue benchmark @xcite . 4 for finetuning, we follow the practice of @xcite , and run a grid 4 see appendix d for a full description of tasks. ## discussion comparison with electra while @xcite show impressive pretraining speedups with electra, we argue that having a generative model (mlm or lm) is important nowadays, given the recent rise of few-shot learning and prompting approaches @xcite . to emphasize this point, we run 24hbert on the sst-2 @xcite task both with and without prompts in the few-shot setting. figure 3 shows that there is a significant advantage in the ability to prompt the model, which is perhaps not trivial for non-generative electra-style models. flops as a measure of efficiency while measuring floating point operations is commonly used to compare efficiency in a hardware-agnostic manner, it is not an accurate tool for comparing the actual time (and therefore budget) associated with training a model. specifically, measuring flops ignores the fact that many operations run in parallel (e.g. via batching), and are thus much less costly in practice @xcite . limitations our investigation is limited to classification tasks. while it is true that it is not fully comparable with bert base in that using short sequences does not allow for reading comprehension tasks (without resorting to sliding windows), it might be possible to continue training the model for a few more hours with sequences longer than 128 tokens, as done by @xcite . we leave such experiments for future work. ## conclusions we present a recipe for pretraining a masked language model in 24 hours using a low-end deep learning server. we show that by combining multiple efficient training methods presented in recent work and carefully calibrating the hyperparameters it is possible to pretrain a model that is competitive to bert base on glue tasks. in contrast to other works in this area, which often focus a single method for improving efficiency, our recipe consists of many different components that together amount to very large speedups: