  While large language models \`a la BERT are used ubiquitously in NLP,
pretraining them is considered a luxury that only a few well-funded industry
labs can afford. How can one train such models with a more modest budget? We
present a recipe for pretraining a masked language model in 24 hours, using
only 8 low-range 12GB GPUs. We demonstrate that through a combination of
software optimizations, design choices, and hyperparameter tuning, it is
possible to produce models that are competitive with BERT-base on GLUE tasks at
a fraction of the original pretraining cost.
