EnergyNet : Energy-Efﬁcient Dynamic Inference
Abstract
The prohibitive energy cost of running high-performance Convolutional Neural
Networks (CNNs) has been limiting their deployment on resource-constrained platforms including mobile and wearable devices.
We propose a CNN for energy-aware dynamic routing, called EnergyNet , that achieves adaptive-complexity inference based on the inputs, leading to an overall reduction of run time energy cost while actually improving accuracy.
This is achieved by proposing an energy loss that captures both computational and data movement costs.
We combine it with the accuracy-oriented loss, and learn a dynamic routing policy for skipping certain layers in the networks that optimizes the hybrid loss.
Our empirical results demonstrate that, compared to the baseline CNNs, EnergyNet can trim down the energy cost by up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining the same testing accuracy.
It is further encouraging to observe that the energy awareness might serve as a training regularization that can improve the prediction accuracy: our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Introduction
While deep learning-powered Internet of Things (IoT) devices promise to dramatically revolutionize the way we live and work by enhancing our ability to recognize, analyze, and classify the world around us, this revolution has yet to be unleashed due to many fundamental challenges.
Edge devices, such as smart phones, smart sensors, drones and robots, have limited energy and computation resources since they are battery-powered and have a small form factor.
On the other hand, high-performance
Convolutional Neural Networks (CNNs) come at a cost of prohibitive energy consumption [1].
The
CNNs with the highest accuracy have hundreds of layers and tens of millions of parameters.
When deployed in practice, such networks drain the battery very quickly [2].
Recently, there have been a number of methods proposed to reduce energy cost in CNNs, while not hampering their predictive power.
Most of them aim to reduce the model size or the number of computations [3, 4, 5, 6, 7, 8, 9, 10, 11, 12].
However, [2] shows that a smaller model size and fewer operations might not necessarily lead to a lower energy cost.
[2] uses energy cost to guide the pruning process, where the layer with the highest energy cost is pruned ﬁrst.
[13] formulates the CNN training process as an optimization problem under a certain energy budget constraint.
While both methods [2, 13] show promising results towards pursuing more energy-efﬁcient CNN models, they do not incorporate energy costs into the training loss function to explicitly learn a more energy-efﬁcient model.
Furthermore, once their model structures are learned from training, it can only be ﬁxed during the inference time, and there is no room for input-dependent adaptivity.
This paper proposes a new CNN model that combines energy cost with a dynamic routing strategy to enable adaptive energy-efﬁcient inference.
Our proposed model, termed as EnergyNet , is a gated
CNN architecture which employs conditional computing to route the input data through the network in an efﬁcient path.
Built on a base network (such as ResNet-34 or ResNet-50 [14]), EnergyNet uses an additional gating network [11] to decide whether the current input should skip certain layers in the network or not.
It optimizes a weighted combination of an accuracy loss and an energy loss which captures both the computational and memory data movement costs, under which EnergyNet is trained to ﬁnd the optimal routing policy to reduce the energy cost of the model without degrading the prediction accuracy.
Our empirical results demonstrate that, compared to the base network without gating nor dynamic inference, EnergyNet can trim down the energy cost up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining almost the same testing accuracy.
Interestingly enough, we ﬁnd the energy-aware EnergyNet can even achieve win-win, by simultaneously improving the prediction accuracy and saving energy, potentially due to its equivalent effect as a training regularization to avoid overﬁtting.
For example, our models achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Proposed EnergyNet Model
Overview: EnergyNet implements an effective dynamic routing algorithm using a set of gating networks, which shares similar ideas with [11], as depicted in Figure 1.
Each gating network associates with a block of layers in the EnergyNet .
Given an input image, the gating networks decide if the corresponding block should be skipped or not.
The input to each block is ﬁrst sent to the gating network G, whose output is either 0 or 1.
If it is 0, the block will be skipped; otherwise, it will process the input normally as in the base model.
If the input and output of the block have different dimensions, then we can perform a linear projection using a shortcut connection to match the dimensions as in [14]. The core innovation in EnergyNet is the adoption of a new energy-aware loss function for learning the gating (skipping) policy, whose details we defer to the next subsection. In our implementation, we adopt the recurrent gates (RNNGates) as in [11] (see Figure 2). SkipNet
[Figure]
[Figure]
I
[Equation]
[Figure]
[Equation] [Figure] shared [Equation]
Fig.
3: Gate designs.
(a) FFGate-I contains two 3x3 convolutional layers and has roughly 19% of the computation of the residual block.
(b) FFGate-II is composed of one convolutional layer with stride of 2 and has about 12 .
5% of the computation of residual blocks.
(c) RNNGate contains a one-layer LSTM with both input and hidden unit size of 10.
The cost of the RNNGate is negligible at 0 .
04% of the cost of the residual blocks.
map of the gates and then linearly project the feature to the input size of 10.
We adopt a single layer Long Short Term Memory [14] (LSTM) with hidden unit size of 10.
At each gate, we project the LSTM output to a one-dimensional vector to compute the ﬁnal gate decision.
Compared to the cost of computing residual blocks, the cost of this recurrent gate design is negligible (roughly 0 .
04% of the computation of residual blocks).
In our later experiments, we ﬁnd that the recurrent gate dominates the feed-forward gates in both prediction accuracy and computation cost.
We also evaluated simpler feed-forward gate designs without convolution layers and while these matched the computation cost of the recurrent gates the prediction accuracy suffered.
We conjecture that the recurrent gate design better captures the cross-layer dependencies.
3.2 Skipping Policy Learning with Hybrid RL During inference the most likely action is taken from the probability distribution encoded by each gate: the layer is skipped or executed .
This inherently discrete and therefore non-differentiable decision process creates unique challenges for how we train SkipNets.
A natural approximation, similar to that used in Highway Networks [27], would be to use differentiable soft-max decisions during training and then revert to hard decisions during inference.
While this approach enables gradient based training, it results in poor prediction accuracy (Sec. 4.3) as the network parameters are not optimized for the subsequent hard-gating during inference.
We therefore explore the use of reinforcement learning to learn the model parameters for the non-differentiable decision process.
Because SkipNets make a sequence of discrete decisions, one at each gated layer, we frame the task of estimating the gating function in the context of policy optimization through reinforcement learning.
We deﬁne the skipping policy:
[Equation]
[Equation] ) as a function from the input x i to the probability distribution over the gate action g i to execute ( g i = 1 ) or skip ( g i = 0 ) layer i .
We deﬁne a sample sequence of gating decisions drawn from the skipping policy starting with input x as:
[Equation]
It is composed of a global average pooling followed by a linear projection that reduces the features to a 10-dimensional vector.
A
Long Short Term Memory (LSTM) [15] network that contains a single layer of dimension 10 is applied to generate a binary scalar.
As mentioned in [11], this RNNGate design incurs a negligible overhead compared to its feed-forward counterpart (0.04% vs. 12.5% of the computation of the residual blocks when the baseline architecture is a ResNet).
In order to further reduce the energy cost due to loading parameters into the memory, all RNNGates in the EnergyNet share the same weights.
Energy-aware Learning for Dynamic Routing:
The dynamic routing in EnergyNet is learned by minimizing an energy cost together with the accuracy loss.
In particular, the learning goal in the
EnergyNet is deﬁned as:
[Equation]
Here, α is a weighting coefﬁcient of the energy loss, and W and G denote the parameters of the base model and the gating network, respectively.
Also, L ( W, G ) denotes the prediction loss, and
E ( W, G ) denotes the energy cost of the CNN model associated with W and G , which is calculated by accumulating the energy cost of the layers that are not skipped.
In order to compute the energy cost of each layer, we adopt the following energy model:
[Equation]
where e i and e MAC denote the energy costs of accessing the i -th memory hierarchy and one multiplyand-accumulate (MAC) operation [16], respectively, while # MAC and # acc i denote the total number of MAC operations and accesses to the i -th memory hierarchy, respectively.
Note that state-of-the-art
CNN accelerators commonly employ such a hierarchical memory architecture for minimizing the dominant memory access and data movement costs.
In this work, we consider the most commonly used design of three memory hierarchies including the main memory, the cache memory, and local register ﬁles [16], and employ a state-of-the-art simulation tool called "SCALE-Sim" [17] to calculate the number of memory accesses # acc i and the total number of MACs # MAC .
Experiments
Summary: We show that EnergyNet saves more energy than the baseline ResNet after training on CIFAR10 and Tiny ImageNet [18].
In particular, compared to the baseline ResNet, EnergyNet saves up to 40% and 65% energy cost without degrading the prediction accuracy, when processing
CIFAR10 and TinyImageNet images, respectively.
More encouragingly, our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Architectures and Training Details: We use the ResNet-38 and ResNet-50 in [14] as the baseline models for constructing and evaluating EnergyNet models on CIFAR-10 and Tiny ImageNet, with the resulting models denoted as EnergyNet-38 and EnergyNet-50 , respectively.
The training process contains three steps.
In step I, we set the weighting coefﬁcient α to a small value (e.g., 0.1), which helps the model converge to the baseline accuracy ﬁrst.
In step II, we increase α to a larger value (e.g., 0.9) and retrain the model obtained from step I. For step III , it is only triggered if the model sees an accuracy loss larger than a threshold (default 0.1%) from step II: we then set α to a small value (e.g., 0.1) again to retrain the resulting model from step II for restoring the accuracy.
Such a three-step strategy proves to help stabilize training and gain better performance.
Discussion: We use energy savings as a metric to quantify the resulting energy efﬁciency improvement of EnergyNet .
The energy savings is deﬁned as E s /E
total , where E total and E s are the energy cost of the baseline model and the skipped layers due to EnergyNet .
From Figure 3, we can conclude that EnergyNet achieves the goal of reducing energy cost while preserving or even improving the prediction accuracy.
In particular, the accuracy of EnergyNet-38 and EnergyNet-50 will not drop when the energy savings is as high as 40% and 65%, respectively.
To conﬁrm that these experimental results are not just a coincidence, we performed 20 trials of experiments using EnergyNet-38 and observed that the conﬁdence interval with a 95% conﬁdence level for the mean of the prediction accuracy and the energy savings are [92.47%, 92.58%] and [39.55%, 40.52%], respectively, verifying the reproducibility of EnergyNet ’s prediction accuracy and resulting energy savings.
We observe that EnergyNet can achieve a higher accuracy than the original ResNet model.
We conjecture that this is because EnergyNet can overcome overﬁtting when performing the dynamic routing for energy savings.
Further EnergyNet can aggressively reduce energy cost by about 4 × , over both the ResNet-38 and ResNet-50 baselines, at the cost of 3% and 4% testing accuracy losses , respectively.
Acknowledgements
We would like to acknowledge Dr. Edwin Park and Qualcomm Technologies, Inc for providing funding support in this research.