Towards Accurate and Reliable Energy Measurement of NLP Models (2020)

Abstract: Accurate and reliable measurement of energy consumption is critical for making well-informed design choices when choosing and training large scale NLP models. In this work, we show that existing software-based energy estimations are not accurate because they do not take into account hardware differences and how resource utilization affects energy consumption. We conduct energy measurement experiments with four different models for a question answering task. We quantify the error of existing software-based energy estimations by using a hardware power meter that provides highly accurate energy measurements. Our key takeaway is the need for a more accurate energy estimation model that takes into account hardware variabilities and the non-linear relationship between resource utilization and energy consumption. We release the code and data at https://github.com/csarron/sustainlp2020-energy .

## introduction state-of-the-art nlp models of today @xcite @xcite consume large amounts of energy. such high-levels of energy consumption adds to the worsening global warming and can cause significant social health and safety impacts (glo; @xcite . recent studies have raised awareness of the carbon footprints and potential energy impacts and suggest ways to estimate and reduce consumption @xcite . the success of these and future efforts depend on our ability to accurately and reliably estimate the energy consumption of nlp models. a common technique to predict the energy consumption is to measure the utilization of hardware components involved in the computation-the cpu, the gpu, and memory. each of these components is associated with a single power counter value that is provided by the underlying hardware; this power counter represents the power drawn of a given component. the total energy consumption is computed as the sum of the (utilization × power counter) of the cpu, gpu, and memory, which is then adjusted by a compensation constant @xcite . we call this technique software-based power measurement. however there are two potential sources of inaccuracies in the software-based power measurement techniques. first, the software tools are known to be inaccurate because they only consider the energy consumed by three specific hardware components, which may not reflect the energy consumption of the entire system. second, accurately mapping hardware utilization to the energy consumption is a difficult problem. the mapping depends on the underlying hardware make and type, energy is not always linearly related to the utilization @xcite , and energy consumption often continues even after the nlp model has finished running @xcite . in this work, we use a hardware power meter to measure ground truth energy consumption, which is more accurate. our goal is to quantify how far software-based measurements are from the hardware energy measurements. we compare the energy estimates obtained using prior software based models for four transformer-based nlp models fine-tuned for a question answering (qa) task. in the experiments, we find that (1) software energy estimates can differ from the hardware power measurements by 20% on average. further, the standard deviations are 2× larger than hardware power meters. (2) power-models need to take into account the underlying hardware, make, and configuration. hardware-agnostic energy measurements results in large errors, for example, when applied to machines with different configurations (e.g. dif-ferent gpu models, # of gpus used). finally, we show the importance of accurate power-models to make the right accuracy/energy trade-off. ground-truth energy measurements using a hardware meter show that roberta-base incurs 13% more energy on average. but robertabase can answer 2.2% more questions correctly over bert-base. however, existing power-models estimate the additional power consumption of roberta-base to be 25%. such inaccuracies can lead to wrong conclusions and poor optimizations for model practitioners. the results in this paper suggests that we need better estimation models that are calibrated to account for hardware variabilities and the non-linear relationship between power consumption and resource utilization. ## experiments methodology in this section, we describe our setup and methodology for energy measurements. we focus on energy consumption of inference for a qa task using a hardware power meter. for comparison purposes, we track software reported energy values as well. ## energy results of nlp models in this section, we discuss the energy results of the four transformer-based nlp models on a question answering task. ## related work and discussion energy estimation is an important research topic in both the machine learning and system community. we discuss two threads of research related to energy estimation for nlp models: energy estimation in machine learning and nlp. @xcite use a software framework called experiment-impact-tracker to report the aggregated energy of benchmark programs. the experiment-impact-tracker collects hardware resources statistics, and uses a simple linear model @xcite to estimate the total energy where the coefficients are fixed to a constant without considering the actual hardware device configurations. we have shown in the experiments that such software based energy estimation methods are neither accurate nor robust. we recommend using hardware power meters to measure the energy consumption, and then possibly calibrate the software energy values. @xcite presents an energy efficient benchmark for nlp models. however, they only report the time (hours) and cost (dollars) for training and testing nlp models, the actual energy numbers remain unknown. the green ai @xcite work suggests using metrics like floating point operations (fpo) to measure energy efficiency. however, @xcite argues such metrics alone cannot accurately reflect energy consumption. @xcite provide a comprehensive survey of energy estimation methods in machine learning, but no energy measurements for nlp models were reported. energy modeling for systems and applications. energy estimation for battery powered de-vices such as mobile phones is critical since mobile applications utility can be limited by the battery life. previous work @xcite @xcite study various fine-grained system-level power modeling and profiling techniques to help understand energy drain of applications. nlp models essentially power many emerging applications such as personal assistants with mobile intelligence. however, the energy implications for these nlp models are not studied. it is unclear how to apply the existing energy estimation methods for mobile applications to nlp models. we believe it is important to understand the computational semantics in nlp models before leveraging these existing power modeling methods. using fine-grained power estimation models and profiling techniques could further improve our understanding of how nlp models consume energy and is an interesting future work. limitations. in this work, we collect the energy values every 1 second, which can reflect the total amount of energy consumed for batched inferences that often last over 10 seconds. however, if one needs to understand the energy spent inside the model for a single inference, the energy values are still coarse-grained, we will explore fine-grained energy measurement solutions to study this issue in the near future. more complex energy issues like tail power, energy distortion @xcite also affect hardware power meters if analyzing the energy spent inside the model. further, it is not clear yet which machine component (cpu, gpu or memory reads/writes) takes how much energy for the nlp models. we leave this to future work. ## conclusions as nlp models keep getting larger, reducing the energy impact of deploying these models is critical. recent work has enabled estimating and tracking the energy of these nlp models. these works design a software-based technique to estimate energy consumption by tracking resource utilization. however, we show that currently used software-based measurements method is not accurate. we use a hardware power meter to accurately measure energy and find that this measurement method has an average error of 20% and can lead to making inaccurate design choices. going forward, we hope this paper encourages the nlp community to build on current systems research to design more accurate energy models that take into account the underlying power dynamics and device variabilities.