I r E ne: Interpretable Energy Prediction for Transformers (2021)

Abstract: Existing software-based energy measurements of NLP models are not accurate because they do not consider the complex interactions between energy consumption and model execution. We present IrEne, an interpretable and extensible energy prediction system that accurately predicts the inference energy consumption of a wide range of Transformer-based NLP models. IrEne constructs a model tree graph that breaks down the NLP model into modules that are further broken down into low-level machine learning (ML) primitives. IrEne predicts the inference energy consumption of the ML primitives as a function of generalizable features and fine-grained runtime resource usage. IrEne then aggregates these low-level predictions recursively to predict the energy of each module and finally of the entire model. Experiments across multiple Transformer models show IrEne predicts inference energy consumption of transformer models with an error of under 7% compared to the ground truth. In contrast, existing energy models see an error of over 50%. We also show how IrEne can be used to conduct energy bottleneck analysis and to easily evaluate the energy impact of different architectural choices. We release the code and data at https://github.com/StonyBrookNLP/irene .

## introduction accurately measuring the energy consumption of nlp models is becoming ever more important. models are growing exponentially, with billions, even approaching trillions, of parameters with correspondingly large resource consumption (e.g. gpt-3 @xcite has 175 billion parameters and switch transformers can have 1.6 trillion parameters @xcite ). recent works have sought to estimate energy consumption and suggest ways to reduce the resulting costs and car-bon impacts @xcite @xcite unfortunately, there are no easy-to-use and accurate solutions for measuring or predicting the energy consumption. on the one hand, measuring energy consumption directly through hardware power monitors is not feasible as it requires exclusive access to the hardware and detailed instrumentation. on the other hand, there are software models that predict energy as a function of resource utilization @xcite but these energy prediction models are inaccurate @xcite . the inaccuracy stems from the prediction models not accounting for the complex interactions between energy consumption and resource utilization. in this work, we focus on inference energy which can incur substantial costs especially for models that support high-volume web services. we ask how we can build an energy prediction method that is accurate, interpretable, and extensible. we make three contributions in answering this question. first, we frame the problem of interpretable energy prediction over a model tree abstraction. this abstraction represents the model as the root node that is composed from model-specific modules, which themselves are recursively composed from lower-level machine learning (ml) primitives, ones that are not model-specific. given a model, the energy prediction problem is framed as the task of predicting the energy of all the nodes in its model tree abstraction. the result is that irene can predict not only the inference energy consumption of the entire model, but also of its components, making the energy prediction highly interpretable. second, we develop irene, that includes a multilevel prediction method that predicts energy in all nodes of the abstraction tree in a bottom-up fashion using resource utilization and model description features. for each of the leaf-nodes that are re-used in different models, the ml primitives, irene uses a separate regressor trained on ground-truth energy measurements. one simple way to get energy for all other higher-level nodes is to recursively sumup the values. while this works reasonably well (even better than a prior prediction model), direct summing of the raw predictions is sub-optimal because the error can propagate through the model tree thus making upper-level nodes estimation more erroneous. instead, we learn a single regressor for all intermediate nodes, one that essentially adjusts the sum of children's predicted energy values based on features of the children. since irene is built on top of energy predictions of ml primitives that are not model specific, it is generalizable and can be used to predict the energy for previously unseen (transformer-based) models. third, to evaluate irene, we create an evaluation dataset with ground-truth energy measurements for multiple transformer-based models at all levels in the model tree abstraction. evaluations show that irene is more accurate -with an average modellevel energy error of 5 ∼ 7% compared against the ground-truth, while existing software-based method @xcite has over 55% error. the module-level energy errors are also substantially small showing that irene is both accurate and interpretable. last, we also conduct multiple analyses that show the utility of irene for interpretable energy predictions. ## related work over the last couple of years, there has been increased interest in the energy consumption of nlp models, starting with the work by strubell et al. @xcite . this work, and a follow up software framework called experimentimpact-tracker @xcite tracks the resource (i.e., cpu, gpu, memory) utilization of an nlp model and predicts energy consumption as a function of resources. however, our previous study shows that this type of resource utilization only modeling can be highly inaccurate @xcite . this is in part due to the complex relationship between resource utilization and energy consumption. further, there are other activities that are not accounted via resource utilization such as data movement in gpu memory which can also cause significant energy footprint @xcite . other works @xcite report the energy numbers through alternate metrics including dollar cost or in terms of floating point operations. however, these do not directly map to the energy consumption. energy prediction of applications on mobile devices is a well-studied topic in the systems community @xcite @xcite but these work require fine-grained understanding of the application. none of the existing systems predict energy for nlp applications. ## interpretable energy prediction in this section we first state our design goals, motivate the abstraction, and problem formulation for interpretable energy prediction. ## interpretable prediction with irene resource features model features model specs bertmodel bertembeddings bertencoder bertpooler embedding:word layernorm bertlayer:0 bertattention bertintermediate bertoutput bertselfattention bertselfoutput linear:query matmul softmax linear:dense layernorm linear:dense linear:dense layernorm linear:dense tanh jit tracing profile estimation model predicted energy for each node irene energy estimation (e.g. size of inputs) enabling accurate multi-level energy prediction. irene represents higher-level modules via generalizable features and the ml primitives. even if the intermediate modules are model-specific (e.g. bert-selfattention), the features are general, allowing irene to predict energy of unseen models. the irene model is trained using ground-truth energy measurements of ml primitives and a handful of nlp models; we use a highly accurate hardware power monitor to measure ground truth energy ( §a). of course, one can use the power monitor to measure energy directly at runtime. however, this is cumbersome and requires physical access to the device which is not always feasible with cloudbased deployments. further, the hardware meter only measures the total energy, which is not interpretable in terms of its components. ## irene evaluation our evaluation is aimed at measuring the accuracy of irene relative to ground truth and the state-ofthe-art. we show the irene only causes 5-7% error for the model energy prediction. we also show that for a given transformer model, irene can be used to find the energy bottlenecks and analyze the energy versus task performance trade-offs. ## conclusions energy consumption of nlp models is an important consideration from a cost perspective and increasingly, from an environmental impact perspective as well. designing energy efficient and costeffective models requires both accurate and interpretable energy modeling. in this work, we showed that by carefully combining resource utilization with model description based features, we can develop a multi-level energy prediction model that is not only highly accurate but is also able to provide a break-down of how its different components contribute to its overall energy.