EnergyNet : Energy-Efﬁcient Dynamic Inference
Anonymous Author s)
AfﬁliationAddress email
Abstract
The prohibitive energy cost of running high-performance Convolutional Neural
Networks (CNNs) has been limiting their deployment on resource-constrainedplatforms including mobile and wearable devices.
We propose a CNN for energyaware dynamic routing, called the EnergyNet , that achieves adaptive-complexityinference based on the inputs, leading to an overall reduction of run time energycost while actually improving accuracy.
This is achieved by proposing an energyloss that captures both computational and data movement costs.
We combine itwith the accuracy-oriented loss, and learn a dynamic routing policy for skippingcertain layers in the networks, that optimizes the hybrid loss.
Our empirical resultsdemonstrate that, compared to the baseline CNNs, EnergyNet can trim down theenergy cost by up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining the same testing accuracies.
It is further encouraging to observe that the energy awareness might serve as atraining regularization and can even improve prediction accuracy: our models canachieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 whensaving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNetwhen saving up to 50% energy, respectively.
Introduction
While deep learning-powered IoT devices promise to dramatically revolutionize the way we live andwork by enhancing our ability to recognize, analyze and classify the world around us, this revolution has yet to be unleashed due to many fundamental challenges.
Edge devices, such as smart phones,smart sensors, drones and robots, have limited energy and computation resources since they arebattery-powered and have a small form factor.
On the other hand, high-performance Convolutional
Neural Networks (CNNs) come at a cost of prohibitive energy cost [14].
The CNNs with the highestaccuracy have hundreds of layers and tens of millions of parameters.
When deployed in practice,such networks drain the battery very quickly [19].
Recently, there have been a number of methods proposed to reduce energy cost in CNNs, whilenot hampering their predictive power.
Most of them aim to reduce the model size or the numberof computations [2, 3, 7, 9, 11, 12, 13, 16, 17].
However, [19] shows that a smaller model size andfewer operations might not necessarily lead to a lower energy cost.
[19] uses energy cost to guidethe pruning process, where the layer with the highest energy cost is pruned ﬁrst.
[18] formulatesthe CNN training process as an optimization problem under a certain energy budget constraint.
While both methods [19, 18] show promising results towards pursuing more energy-efﬁcient CNNmodels, they do not incorporate energy costs into the training loss function to explicitly learn a moreenergy-efﬁcient model.
Furthermore, once their architecture is learned from training, it can only beﬁxed during the inference time, i.e., no input-dependent adaptivity.
This paper proposes a new CNN model that combines energy cost with a dynamic routing strategyto enable adaptive energy-efﬁcient inference.
Our proposed model, termed as EnergyNet , is a gatedarchitecture which employs conditional computing to route the input data through the network inan efﬁcient path.
Built on a base network (such as ResNet-34 or ResNet-50), EnergyNet uses an additional gating network [16] to decide whether the current input should skip certain layers in thenetwork or not.
It optimizes a weighted combination of an accuracy loss and an energy loss whichcaptures both the computational and memory data movement costs, under which EnergyNet is trainedto ﬁnd the optimal routing policy to reduce the energy cost of the model without degrading theprediction accuracy.
Our empirical results demonstrate that, compared to the base network withoutgating nor dynamic inference, EnergyNet can trim down the energy cost up to 40% and 65%, duringinference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining almost thesame testing accuracies.
Interestingly enough, we ﬁnd the energy-aware textitEnergyNet can evenimprove the prediction accuracy, potentially due to its equivalent effect as a training regularization toavoid overﬁtting: our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNetwhen saving up to 50% energy, respectively.
Proposed EnergyNet Model
Overview: EnergyNet implements an effective dynamic routing algorithm using a set of gatingnetworks, which shares similar ideas with [16], as depicted in Figure 1.
Each gating networkassociates with a block of layers in the EnergyNet .
Given an input image, the gating networks decideif the corresponding block should be skipped or not.
The input to each block is ﬁrst sent to the gatingnetwork G.
The output of the gating network is either 0 or 1.
If 0, the block will be skipped; otherwise,it will process the input normally.
If the input and output of the block have different dimensions, then
SkipNet
[Figure]
shared
[Figure]
Fig.
3: Gate designs.
(a) FFGate-I contains two 3x3 convolutional layers and has roughly 19% of the computation of the residual block.
(b) FFGate-II is composed of one convolutional layer with stride of 2 and has about 12 .
5% of the computation of residual blocks.
(c) RNNGate contains a one-layer LSTM with both input and hidden unit size of 10.
The cost of the RNNGate is negligible at 0 .
04% of the cost of the residual blocks.
map of the gates and then linearly project the feature to the input size of 10.
We adopt a single layer Long Short Term Memory [14] (LSTM) with hidden unit size of 10.
At each gate, we project the LSTM output to a one-dimensional vector to compute the ﬁnal gate decision.
Compared to the cost of computing residual blocks, the cost of this recurrent gate design is negligible (roughly 0 .
04% of the computation of residual blocks).
In our later experiments, we ﬁnd that the recurrent gate dominates the feed-forward gates in both prediction accuracy and computation cost.
We also evaluated simpler feed-forward gate designs without convolution layers and while these matched the computation cost of the recurrent gates the prediction accuracy suffered.
We conjecture that the recurrent gate design better captures the cross-layer dependencies.
3.2 Skipping Policy Learning with Hybrid RL During inference the most likely action is taken from the probability distribution encoded by each gate: the layer is skipped or executed .
This inherently discrete and therefore non-differentiable decision process creates unique challenges for how we train SkipNets.
A natural approximation, similar to that used in Highway Networks [27], would be to use differentiable soft-max decisions during training and then revert to hard decisions during inference.
While this approach enables gradient based training, it results in poor prediction accuracy (Sec. 4.3) as the network parameters are not optimized for the subsequent hard-gating during inference.
We therefore explore the use of reinforcement learning to learn the model parameters for the non-differentiable decision process.
Because SkipNets make a sequence of discrete decisions, one at each gated layer, we frame the task of estimating the gating function in the context of policy optimization through reinforcement learning.
We deﬁne the skipping policy:  ( x i , i ) =
P ( G i ( x i ) =
g i ) (3) as a function from the input x
i to the probability distribution over the gate action g i to execute ( g i = 1 ) or skip ( g i = 0 ) layer i .
We deﬁne a sample sequence of gating decisions drawn from the skipping policy starting with input x as: g =
[ g 1 , . . .
, g N ]   F  , (4) reduces a negligible overhead compared toits feed forward counterpart
parameters into memory
RNNGates share weights
Energy-aware Learning for Dynamic Routing:
The dynamic routing in the EnergyNet is learnedby optimizing an energy cost together with the accuracy loss.
In particular, the learning goal in the
EnergyNet is deﬁned as:
[Equation] [Equation]
86 where e i and e MAC denote the energy costs of accessing the i -th memory hierarchy and one multiplyand-accumulate (MAC) operation [4], respectively, while # acc i and # MAC denote the total numberof accesses to the i -th memory hierarchy and MAC operations, respectively.
Note that state-of-the-art
CNN accelerators commonly employ such a hierarchical memory architecture for minimizing thedominant memory access and data movement costs.
In this work, We consider the most commonlyused design of three memory hierarchies including the main memory, the cache memory, and localregister ﬁles [4], and employ a state-of-the-art simulation tool called "SCALE-Sim" [1] to calculatethe number of memory accesses # acc
i and the total number of MACs # MAC .
Experiments
We show that the EnergyNet saves more energy than the baseline ResNet after training on CIFAR10and Tiny ImageNet.
In particular, compared to the baseline ResNet, the EnergyNet saves up to 40%and 65% energy cost without degrading the prediction accuracy, when processing CIFAR10 and
TinyImageNet images, respectively.
More encouragingly, our models can achieve 0.7% higher top-1testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% highertop-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Architectures and Training Details: We use the ResNet-38 and ResNet-50 in [8] as the baselinemodels for evaluating EnergyNet on CIFAR-10 and Tiny ImageNet, respectively.
The training processcontains three steps.
In step I, we set the weighting coefﬁcient α to a small value (e.g., 0.1), whichhelps the model converge to the baseline accuracy ﬁrst.
In step II, we increase α to a larger value (e.g.,0.9) and retrain the model obtained from step I. Note that step III is only needed when the accuracyloss of the model resulted from step II is larger than a speciﬁed value (e.g., 1%).
Speciﬁcally, if theaccuracy loss due to step II is larger than the expected value, then α is set to a small value (e.g., 0.1)again to retrain the model for restoring accuracy.
Discussion: We use energy savings as a metric to quantify the resulting energy efﬁciency improvement of EnergyNet .
The energy savings is deﬁned as E s /E
total , where E total and E s are the energycost of the baseline model and the skipped layers due to EnergyNet .
From Figure 3, we can concludethat EnergyNet achieves the goal of reducing energy cost while preserving or even improving theprediction accuracy.
In particular, the accuracy of EnergyNet-38 and EnergyNet-50 will not dropwhen the energy savings is as high as 40% and 65%, respectively.
In addition, we have anothertwo observations.
First, we can observe that EnergyNet can achieve a higher accuracy than theoriginal ResNet model.
We conjecture that this is because EnergyNet can overcome overﬁtting whenperforming the dynamic routing for energy savings.
Second, EnergyNet leads to about 4 × (i.e., 80%energy savings) over both the ResNet-38 and ResNet-50 baselines with a testing accuracy loss of 3%and 4%, respectively.
Future Works
It is interesting to further explore the energy reduction and accuracy improvement of the EnergyNetover different CNN baselines including DenseNet [10], GoogLeNet [15], and MobileNet [9] onpopular benchmark datasets.
Furthermore, EnergyNet can be extended to employ reinforcementlearning models for gating decisions.
Finally, Expectation-Maximization [5] can be employed toiteratively update the gating decisions for learning better dynamic routing policies in EnergyNet .