Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models (2022)

Abstract: The energy requirements of current natural language processing models continue to grow at a rapid, unsustainable pace. Recent works highlighting this problem conclude there is an urgent need for methods that reduce the energy needs of NLP and machine learning more broadly. In this article, we investigate techniques that can be used to reduce the energy consumption of common NLP applications. In particular, we focus on techniques to measure energy usage and different hardware and datacenter-oriented settings that can be tuned to reduce energy consumption for training and inference for language models. We characterize the impact of these settings on metrics such as computational performance and energy consumption through experiments conducted on a high performance computing system as well as popular cloud computing platforms. These techniques can lead to significant reduction in energy consumption when training language models or their use for inference. For example, power-capping, which limits the maximum power a GPU can consume, can enable a 15% decrease in energy usage with marginal increase in overall computation time when training a transformer-based language model.

## introduction artificial intelligence and machine learning (ml) are increasingly used in diverse areas ranging from nlp to autonomous driving. broadly, larger and 1 this material is based upon work supported by the assistant secretary of defense for research and engineering under air force contract no. fa8702-15-d-0001, and united states air force research laboratory cooperative agreement number fa8750-19-2-1000. any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the assistant secretary of defense for research and engineering, or the united states air force. the u.s. government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation herein. however, estimating energy usage for a particular ai application depends on a number of parameters such as model architecture, hardware details, environmental parameters, and implementation details. for this reason, important works such as @xcite and @xcite rely on estimates extrapolated from industry averages for some language models in some of their analysis of large neural net training, including total floating point operations per second (flops) and hardware estimates, rather than values that otherwise might have been measured. further, these articles call for new research that identifies mitigation techniques that can reduce the energy usage of nlp applications. this paper proposes and characterizes potential ways to reduce the energy impact of nlp applications. to our knowledge this is the first presentation of power-capping as a useful tool for reducing gpu energy consumption. particularly in the context of deep learning and nlp, this work provides an approach alongside estimates for possible energy savings for training large, energy-intensive language models. moreover this method does not affect the predictions of trained models or consequently their performance accuracy on tasks. that is, if two networks with the same structure, initial values and batched data are trained for the same number of batches under different power-caps, their resulting parameters will be identical and only the energy required to produce them may differ. section 2 presents related work on tracking energy usage of nlp applications and their environmental impact. section 3 introduces different techniques, including power-capping and energy-aware scheduling, that can be used to reduce the energy usage, including experiments and other relevant data to characterize their effectiveness. in section 4, we discuss these approaches with broader recommendations before concluding with future avenues of research. ## prior work energy efficiency considerations for deep learning have trailed model developments targeted at improving accuracy among other metrics with new, often growing architectures. highlighting this focus, the growth of neural network architecture sizes is considered in @xcite . that study offers a comparison of state-of-the-art image recognition models where their computational performance is analyzed including inference time and power utilization. techniques for model compression have been widely studied including knowledge distillation and pruning @xcite . in nlp, distillation has been used to reduce the size of large language models @xcite , and other methods of compression have been effective at shrinking model parameters such as embedding layers @xcite . recent attention has focused on the size of nlp models alongside their extensive training times and environmental impact @xcite @xcite . these works illustrate efforts to place greater consideration on the efficiency or inefficiencies of large neural network architectures. for instance, @xcite weighs the advantages of different metrics to evaluate efficiency while advocating for the use of floating point operations as a way to objectively compare models. another area of focus has been on the dependence of model test accuracies on the amount of computation expended on hyperparameter tuning @xcite . some of these works propose considering efficiency alongside accuracy as a metric for evaluating ml models, and at the very least to require reporting energy consumption and carbon impact used in research for conference and journal submissions. while calls to prioritize more efficient methods of training nlp models are made in the previously cited papers among other works, to the best of our knowledge this is not reflected in publicly available academic or industry research. in fact, in @xcite a random sampling of 100 neurips papers showed that few papers tracked and reported these statistics -and none reported carbon impact. this and the previous works point out that tracking energy usage is not yet a standard practice, in part because of the difficulty in implementing a framework for collecting these statistics from hardware. an implementation for accurately capturing this data on common hardware (specifically intel and nvidia hardware) is presented in @xcite which relies on querying device software tools. we describe another, similar approach for gathering power expenditure and energy usage in this work, in order to present a straightforward process for obtaining accurate measurements of energy consumption. compared to these works, this paper presents steps that can be taken to reduce the energy required for training and inference with language models. there is limited prior research investigating power-capping as a method for reducing energy consumption @xcite , and it has focused on cpus for scientific computing applications. our focus is specifically on widely-used ai/ml frameworks used with available commodity hardware. this approach is described with experiments showcasing its effectiveness for a range of settings. additionally our findings for gpu energy reduction when training neural networks are comparable and consistent with the outcomes for cpu consumption presented in @xcite . similar recent work investigates distributed dnn training fitting power law models that describe how training time scales with available compute resources and energy constraints @xcite . additionally we address other approaches towards reducing energy footprints by considering shifting habits in training. utilizing datacenters and climate-aware workload scheduling can provide considerable savings, and we share statistics from our institutional datacenter to support this @xcite . ## reducing the energy impact of nlp this section outlines various approaches that can be used to reduce the energy consumption of nlp workloads. we focus primarily on a simple yet effective method -power-capping -that yields significant benefit with minimal cost and translates across different computing platforms. experiments measuring the effect of power-caps on energy consumption are presented. for completeness, we discuss other potential avenues for reducing the carbon impact of nlp applications. data is presented for the monthly and daily variation in energy efficiency of our institution's datacenter. this illustrates in detail how much energy usage can be reduced by simple approaches like timing workloads to certain hours or seasons if possible. while factors like efficiency and daily variation depend heavily on characteristics unique to each organization's datacenter, we share general insights that will hold true for most cases. measuring energy usage: currently, there are two vendor-provided utilities to monitor resource consumption on nvidia gpus. the nvidia data center gpu manager (dcgm) is a suite of tools for managing and monitoring nvidia gpus in cluster environments (nvidia, 2021a) and the nvidia system management interface (nvidia, 2021b) (or nvidia-smi) utility, which can also perform similar monitoring. broadly, these tools enable monitoring of gpu usage on a node and the collection of metrics on streaming multi-processor (sm) utilization, gpu memory footprint, power draw, gpu temperatures, pci express (pcie) bandwidth, and several other hardware settings. on our system, this data is collected on every node and every gpu assigned to a job. the data is collected every 100ms and data collection is started and stopped automatically using the scheduler that manages resources on the system. ## discussion and recommendations we believe the approaches proposed here offer easy-to-implement solutions for reducing the carbon footprint of nlp applications without significant algorithmic or software changes. though they do not involve new algorithmic methods which are outside the scope of this article, these represent early steps towards more efficient nlp. coupling them with algorithmic changes would further improve energy consumption. the goal of this article is to initiate a conversation between nlp researchers and those in the hardware and datacenter domains. below, we list additional recommendations that may help shape such a conversation. understanding your computational environment's characteristics: previous works highlighted the carbon footprint of computationally expensive nlp applications, and their recommendations of tracking and reporting energy usage was intended to encourage researchers to be aware of their individual impact. similarly we highlight the importance of datacenter characteristics and pue variation to promote a deeper understanding of research energy requirements and the factors that constitute them. we hope this work leads nlp researchers to question assumptions about the datacenters where their workloads are running and what the relative efficiency of those datacenters are. for example, researchers should opt for energy-efficient datacenters and encourage their organizations to deploy or leverage energy-efficient datacenters. if possible, it would also be helpful for researchers to learn these operating characteristics of their datacenters or computing providers. further we encourage the nlp community to work with their computing facility or datacenter to implement frameworks for tracking energy consumption like that outlined in section 3 and other works (e.g. henderson et al. (2020 ## conclusions this article presents techniques that can improve the energy efficiency of training and inference for nlp applications. importantly, the methods discussed can be used jointly with each other to achieve a compounding effect of energy savings. future work relevant to these topics would include a wider survey of ai hardware and power-capping capabilities. while we focused on nvidia gpus, evaluation of ai hardware from other vendors and cloud providers could have a potentially large impact for cloud computing as well as large shared high performance computing centers.