Energy and Carbon Considerations of Fine-Tuning BERT (2023)

Abstract: Despite the popularity of the pre-train then fine-tune paradigm in the NLP community, existing work quantifying energy costs and associated carbon emissions has largely focused on language model pre-training. Although a single pre-training run draws substantially more energy than fine-tuning, fine-tuning is performed more frequently by many more individual actors, and thus must be accounted for when considering the energy and carbon footprint of NLP. In order to better characterize the role of fine-tuning in the landscape of energy and carbon emissions in NLP, we perform a careful empirical study of the computational costs of fine-tuning across tasks, datasets, hardware infrastructure and measurement modalities. Our experimental results allow us to place fine-tuning energy and carbon costs into perspective with respect to pre-training and inference, and outline recommendations to NLP researchers and practitioners who wish to improve their fine-tuning energy efficiency.

## introduction fine-tuning pre-trained language models is a frequent occurrence in natural language processing (nlp) research and practice, yet the vast majority of work quantifying the energy and carbon footprints of nlp workloads has focused on pretraining @xcite @xcite or inference @xcite . the typical lifecycle of an nlp model includes data ingestion, pre-training, fine-tuning and inference, all of which contribute non-trivially to energy use and corresponding emissions @xcite . better understanding of the role that each phase plays in overall energy and carbon footprint is vital to inform policy decisions, yet we still lack basic data quantifying the rela- * denotes equal contribution. in this work we perform an empirical study to quantify the energy requirements of language model fine-tuning, including in the context of pretraining and inference energy requirements. while this may seem like it should be a straightforward calculation, there are several variables that can influence compute time and energy consumption, ranging from: (1) the type of hardware used for both pre-training and fine-tuning (since this usually differs between the two), (2) the type of task and the type of computation required to carry it out, and (3) intrinsic characteristics of the dataset, such as average sequence length, its similarity with the pre-training dataset, etc. in order to isolate the factors that have the most influence on fine-tuning dynamics, we compare fine-tuning energy use across a suite of common supervised nlp datasets including the tasks of entailment, sentiment analysis, question answering, and named entity recognition, and with training data sizes ranging from 6k to 400k examples. we also measure energy use across two different sets of hardware, using the codecarbon @xcite software package and a physical energy measurement device at the wall, to quantify variance due to physical factors. to enable carefully controlled comparison of the roles of pre-training and fine-tuning in nlp model lifecycle energy use, we additionally pre-train bert variants from scratch on the same hardware. we find that pre-training bert is equivalent to anywhere from 400 (mnli) to 45,000 (rte) fine-tuning runs depending on the dataset size, and that number of training tokens 1 is a reasonable heuristic for estimating fine-tuning energy use. further comparison of fine-tuning in-ference energy intensity across tasks confirms that example sequence length holds a much stronger influence on energy intensity in the fine-tuning phase than in the inference phase, in alignment with expectations from previous work @xcite . together, our observations contextualize the energy and carbon requirements of fine-tuning in the broader model lifecycle and highlight the need to study fine-tuning energy efficiency separately from pre-training and inference workloads in nlp models. we hope that our careful measurement of the relative costs of different nlp workloads will serve as a valuable datapoint informing decision-making both within and beyond the nlp community. ## related work measurement of energy consumption and carbon emissions of nlp models has become an active area of research since it was first identified that modern state-of-the-art models based on deep learning can produce substantial greenhouse gas emissions due to the energy required to train them @xcite . these measurements have mostly focused on two research directions. first, there has been a series of empirical studies on different model architectures, focused on estimating the carbon emissions generated by their training process and the relative efficiency of different methods @xcite @xcite . recent work by luccioni et al. has built upon this, aiming to encompass the embodied emissions of manufacturing computing hardware as well as those produced via the inference process (2023). there has also been complementary work measuring the energy used by transformer models during inference and ways of predicting those costs for different architectures and models @xcite . closest to our work, the hulk benchmark @xcite was proposed to measure the relative efficiency-accuracy trade-offs of different pre-trained models, measuring the wall-clock time for different pre-trained models to reach a target accuracy on one of three fine-tuning tasks. different from @xcite , our work explicitly focuses on the energy and carbon required for fine-tuning (theirs uses time and financial cost as proxies), evaluates a wider variety of fine-tuning tasks and hardware settings in order to elucidate the factors that predict fine-tuning energy requirements, and further contextualizes fine-tuning energy re-quirements in the bigger picture of ml model lifecycle emissions. another related direction of research examines the dynamics of pre-training and fine-tuning of language models and the influence of factors like random seeds and early stopping @xcite , scaling @xcite and learning dynamics @xcite . while all of these studies have shed important light on these processes, in practice most of the decisions made remain empirical, with practitioners either referring to previous work (when hyperparameters and other training details are reported), or using techniques such as grid or random search @xcite to converge on optimal parameter values. our own work builds upon both of these research directions. we study both the pre-training and fine-tuning process, and our experiments for studying their energy intensity are based on the works cited above. ## methodology full training details can be found in appendix a.1. we release code for replicating our measurements. 2 we encourage others to run our code on their hardware to add to a repository of measurements across different hardware platforms. ## results and discussion ## conclusion we share a procedure for rigorous measurement of energy consumption from causal lm fine-tuning given multiple concrete hardware settings. we hope our work is useful to researchers and practitioners who are interested in obtaining measurements for their own specific hardware, gaining intuitions about factors affecting relative energy costs of different types of fine-tuning tasks, or understanding these energy costs in context of the model lifecycle.