I r E ne-viz: Visualizing Energy Consumption of Transformer Models (2021)

Abstract: IrEne is an energy prediction system that accurately predicts the interpretable inference energy consumption of a wide range of Transformer-based NLP models. We present the IrEne-viz tool, an online platform for visualizing and exploring energy consumption of various Transformer-based models easily. Additionally, we release a public API that can be used to access granular information about energy consumption of transformer models and their components. The live demo is available at http://stonybrooknlp.github.io/irene/demo/ .

## introduction pretrained transformers have shown strong results on downstream nlp tasks, resulting in wide-spread adoption. with their deployment in large-scale public-facing systems serving hundreds of millions of requests per day, it has become important to study their energy footprint at inference time. inference energy can incur substantial costs especially for models that are critical for high-volume web services. designing energy efficient and cost-effective models requires both accurate and interpretable energy modeling. current approaches to energy modeling treat the model as a monolithic entity. in our previous work @xcite , we introduced a tree-like abstraction to decompose a model into its components. we designed a multi-level prediction method that predicts energy in all the components of the abstraction tree in a bottom-up fashion using resource utilization and model description features. this system called irene is used as the base of this work. irene provides more accurate energy prediction than other methods and is designed to be interpretable. however, it is nontrivial to retrieve data from that system, making it difficult to perform analysis or visualization for the same. in this work, we present irene-viz, a userfriendly dashboard that allows visualization of inference energy consumption of a transformer-based model and its various components. users will be able to interact with the different operations present in a model. our interface allows people to easily understand the energy bottlenecks during inference. additionally, we make our pipeline public by exposing it as an api endpoint. having such data readily available will further research in the area and allow the community to use it for their own purposes, such as analyzing accuracy or latency trade-offs against energy. for instance, @xcite compared accuracy of bert on a specific task while varying the number of layers and made observations about the energy-accuracy tradeoff. we design irene-viz to be: ## related work there has been increased interest in the energy consumption of nlp models in recent years. despite some progress in modeling, there is a lack of visualisation and analysis tools for the same. ## irene -prediction engine we briefly review the irene system which we use as the energy prediction engine. please refer to @xcite for more details. irene is an interpretable energy prediction system. it represents transformer models in a tree-based abstraction, and generates energy prediction for each node of the tree, thus directly supporting interpretability. irene also comes with data it was trained on -for each tree node, it has associated resource utilization and model-related features, and ground-truth energy measured with a hardware power monitor. ## user interface and functionality the goal of irene-viz is to provide an easy way for users to analyze the energy of a given transformer model (for a specified input size). to do so, we design a browser-based user interface (ui) batch_size : batch size seq_len : # of input tokens flops : floating point operations (unit: million) mem_bytes : memory read and write (unit: mib) cpu_util : cpu utilization (unit: %) mem_usg : memory usage (unit: %) gpu_util : gpu processor utilization (unit: %) gm_usg : gpu memory usage (unit: %) g_clk : gpu processor clock speed (unit: mhz) gm_clk : gpu memory clock speed (unit: mhz) latency : inference latency (unit: s) gpu_energy : gpu driver energy (unit: joule) in irene-viz that controls the input size and selects the model, as shown in figure 3 . we then estimate the energy consumption of the model and visualize the energy for each part in the transformer model. specifically, an user selects a predefined transformer model foot_2 via the dropdown menu and enters the batch size and input sequence length. after pressing the visualize button, irene-viz backend server will run the energy estimation and send the energy result back to the browser for visualization. in irene-viz, we support two core functionalities: functionality 1 -explore the energy consumption of the model. besides the entire model energy, users can interactively explore the energy consumed by any block inside the model, as shown in figure 4 . additionally, we support inspecting the resource and model features used to estimate the energy, as described in figure 5 . functionality 2 -find energy bottlenecks. at each level of the model, users can easily identify operations that can be improved (or pruned) in terms of their relative energy usage. the visualization dashboard also displays a list of model operations along with their predicted energy usage, as presented in figure 6 . ## system implementation to make irene-viz modular and extensible, we design an energy analysis pipeline consisting of three components: a visualization panel that accepts user requests and presents energy results, a prediction engine (irene) that predicts energy consumption and a backend server that encapsulates irene and serves information through an api endpoint. the api and the prediction engine can be used as individual entities as well. they are also designed to be extensible, so adding new features is easy. the visualization panel is intuitive and informative, allowing easy exploration of data. figure 7 shows the full pipeline used for this application. the visualization panel queries the api with the user-desired model name, input sequence length and batch size. this information is passed on to the prediction engine. the engine performs resource collection for the corresponding model specifications and predicts the energy usage of each component. the api sends the visualization panel a full tree representation of the model containing all the model information. ## conclusion and roadmap irene-viz provides an integrated ui and components for visualizing and exploring the energy consumption of various transformer models. it is under active development and is being constantly refined for release. we are adding support for live models immediately. for new models, users will be sent an email with a custom link to their requested visualization. as the community uses it, we will cache resource usage and predictions for more intermediate nodes found in various transformer-based models. this optimization will gradually result in lower times for newer models. our end-to-end pipeline, served as an api, can be used to build an energy leaderboard. this platform can be extended to compare the energy of architectural modifications (e.g. activation or normalization function) of different models for given input. by extending this work to other harware, we aim to provide energy optimization suggestions based on energy profiles of a model on the given hardware. in our previous work, @xcite we also studied accuracy vs energy trade-offs, which will be integrated into the dashboard.