EnergyNet : Energy-Efﬁcient Dynamic Inference
Anonymous Author s)
AfﬁliationAddress email
Abstract
The prohibitive energy cost of running high-performance Convolutional Neural
Networks (CNNs) has been limiting their deployment on resource-constrained platforms including mobile and wearable devices.
We propose a CNN for energy-awaredynamic routing, called EnergyNet , that achieves adaptive-complexity inferencebased on the inputs, leading to an overall reduction of run time energy cost whileactually improving accuracy.
This is achieved by proposing an energy loss thatcaptures both computational and data movement costs.
We combine it with theaccuracy-oriented loss, and learn a dynamic routing policy for skipping certainlayers in the networks that optimizes the hybrid loss.
Our empirical results demonstrate that, compared to the baseline CNNs, EnergyNet can trim down the energycost by up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNettesting sets, respectively, while maintaining the same testing accuracy.
It is further encouraging to observe that the energy awareness might serve as a trainingregularization that can improve the prediction accuracy: our models can achieve0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving upto 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet whensaving up to 50% energy, respectively.
Introduction
While deep learning-powered Internet of Things (IoT) devices promise to dramatically revolutionizethe way we live and work by enhancing our ability to recognize, analyze and classify the world aroundus, this revolution has yet to be unleashed due to many fundamental challenges.
Edge devices, suchas smart phones, smart sensors, drones and robots, have limited energy and computation resourcessince they are battery-powered and have a small form factor.
On the other hand, high-performance
Convolutional Neural Networks (CNNs) come at a cost of prohibitive energy cost [1].
The CNNswith the highest accuracy have hundreds of layers and tens of millions of parameters.
When deployedin practice, such networks drain the battery very quickly [2].
Recently, there have been a number of methods proposed to reduce energy cost in CNNs, whilenot hampering their predictive power.
Most of them aim to reduce the model size or the number ofcomputations [3, 4, 5, 6, 7, 8, 9, 10, 11, 12].
However, [2] shows that a smaller model size and feweroperations might not necessarily lead to a lower energy cost.
[2] uses energy cost to guide the pruningprocess, where the layer with the highest energy cost is pruned ﬁrst.
[13] formulates the CNN trainingprocess as an optimization problem under a certain energy budget constraint.
While both methods[2, 13] show promising results towards pursuing more energy-efﬁcient CNN models, they do notincorporate energy costs into the training loss function to explicitly learn a more energy-efﬁcientmodel.
Furthermore, once their model structures are learned from training, it can only be ﬁxed duringthe inference time, and there is no room for input-dependent adaptivity.
This paper proposes a new CNN model that combines energy cost with a dynamic routing strategyto enable adaptive energy-efﬁcient inference.
Our proposed model, termed as EnergyNet , is a gated CNN architecture which employs conditional computing to route the input data through the networkin an efﬁcient path.
Built on a base network (such as ResNet-34 or ResNet-50), EnergyNet uses an additional gating network [11] to decide whether the current input should skip certain layers inthe network or not.
It optimizes a weighted combination of an accuracy loss and an energy losswhich captures both the computational and memory data movement costs, under which EnergyNet istrained to ﬁnd the optimal routing policy to reduce the energy cost of the model without degrading theprediction accuracy.
Our empirical results demonstrate that, compared to the base network withoutgating nor dynamic inference, EnergyNet can trim down the energy cost up to 40% and 65%, duringinference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining almost thesame testing accuracy.
Interestingly enough, we ﬁnd the energy-aware EnergyNet can even achievewin-win, by simultaneously improving the prediction accuracy and saving energy, potentially due toits equivalent effect as a training regularization to avoid overﬁtting.
For example, our models achieve0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy,and 1.0% higher top-5 accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Proposed EnergyNet Model
Overview: EnergyNet implements an effective dynamic routing algorithm using a set of gatingnetworks, which shares similar ideas with [11], as depicted in Figure 1.
Each gating networkassociates with a block of layers in the EnergyNet .
Given an input image, the gating networks decideif the corresponding block should be skipped or not.
The input to each block is ﬁrst sent to the gatingnetwork G, whose output is either 0 or 1.
If it is 0, the block will be skipped; otherwise, it will processthe input normally as in the base model.
If the
SkipNet (a) FFGate-I (b) FFGate-II shared (c) RNNGate
Fig.
3: Gate designs.
(a) FFGate-I contains two 3x3 convolutional layers and has roughly 19% of the computation of the residual block.
(b) FFGate-II is composed of one convolutional layer with stride of 2 and has about 12 .
5% of the computation of residual blocks.
(c) RNNGate contains a one-layer LSTM with both input and hidden unit size of 10.
The cost of the RNNGate is negligible at 0 .
04% of the cost of the residual blocks.
map of the gates and then linearly project the feature to the input size of 10.
We adopt a single layer Long Short Term Memory [14] (LSTM) with hidden unit size of 10.
At each gate, we project the LSTM output to a one-dimensional vector to compute the ﬁnal gate decision.
Compared to the cost of computing residual blocks, the cost of this recurrent gate design is negligible (roughly 0 .
04% of the computation of residual blocks).
In our later experiments, we ﬁnd that the recurrent gate dominates the feed-forward gates in both prediction accuracy and computation cost.
We also evaluated simpler feed-forward gate designs without convolution layers and while these matched the computation cost of the recurrent gates the prediction accuracy suffered.
We conjecture that the recurrent gate design better captures the cross-layer dependencies.
3.2 Skipping Policy Learning with Hybrid RL During inference the most likely action is taken from the probability distribution encoded by each gate: the layer is skipped or executed .
This inherently discrete and therefore non-differentiable decision process creates unique challenges for how we train SkipNets.
A natural approximation, similar to that used in Highway Networks [27], would be to use differentiable soft-max decisions during training and then revert to hard decisions during inference.
While this approach enables gradient based training, it results in poor prediction accuracy (Sec. 4.3) as the network parameters are not optimized for the subsequent hard-gating during inference.
We therefore explore the use of reinforcement learning to learn the model parameters for the non-differentiable decision process.
Because SkipNets make a sequence of discrete decisions, one at each gated layer, we frame the task of estimating the gating function in the context of policy optimization through reinforcement learning.
We deﬁne the skipping policy:  ( x
[Equation]
= g i ) (3) as a function from the input x
i to the probability distribution over the gate action g i to execute ( g i = 1 ) or skip ( g i = 0 ) layer i .
We deﬁne a sample sequence of gating decisions drawn from the skipping policy starting with input x as: [Equation]
[Equation]
It is composed of a global average poolingfollowed by a linear projection that reducesthe features to a 10-dimensional vector.
A Long Short Term Memory (LSTM) [15] network that contains a single layer of dimension 10 is applied to generate a binary scalar. As mentioned in [11], this RNNGate designincurs a negligible overhead compared to itsfeed-forward counterpart (0.04% vs. 12.5%of the computation of the residual blockswhen the baseline architecture is a ResNet). In order to further reduce the energy cost dueto loading parameters into the memory, all RNNGates in the EnergyNet share the same weights.
Energy-aware Learning for Dynamic Routing:
The dynamic routing in EnergyNet is learned byminimizing an energy cost together with the accuracy loss.
In particular, the learning goal in the
EnergyNet is deﬁned as:
[Equation] [Equation]
86 where e i and e MAC denote the energy costs of accessing the i -th memory hierarchy and one multiplyand-accumulate (MAC) operation [16], respectively, while # MAC and # acc i denote the total numberof MAC operations and accesses to the i -th memory hierarchy, respectively.
Note that state-of-the-art
CNN accelerators commonly employ such a hierarchical memory architecture for minimizing thedominant memory access and data movement costs.
In this work, we consider the most commonlyused design of three memory hierarchies including the main memory, the cache memory, and localregister ﬁles [16], and employ a state-of-the-art simulation tool called "SCALE-Sim" [17] to calculatethe number of memory accesses # acc i and the total number of MACs # MAC .
Experiments
Summary: We show that EnergyNet saves more energy than the baseline ResNet after trainingon CIFAR10 and Tiny ImageNet [18].
In particular, compared to the baseline ResNet, EnergyNetsaves up to 40% and 65% energy cost without degrading the prediction accuracy, when processing
CIFAR10 and TinyImageNet images, respectively.
More encouragingly, our models can achieve 0.7%higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively.
Architectures and Training Details: We use the ResNet-38 and ResNet-50 in [14] as the baselinemodels for constructing and evaluating EnergyNet models on CIFAR-10 and Tiny ImageNet, withthe resulting models denoted as EnergyNet-38 and EnergyNet-50 , respectively.
The training processcontains three steps.
In step I, we set the weighting coefﬁcient α to a small value (e.g., 0.1), whichhelps the model converge to the baseline accuracy ﬁrst.
In step II, we increase α to a larger value(e.g., 0.9) and retrain the model obtained from step I. For step III , it is only triggered if the modelsees am accuracy loss larger than a threshold (default 0.1%) from step II: we then set α to a smallvalue (e.g., 0.1) again to retrain the resulting model from step II for restoring the accuracy. Such athree-step strategy proves to help stabilize training and gain better performance.
Discussion: We use energy savings as a metric to quantify the resulting energy efﬁciency improvement of EnergyNet .
The energy savings is deﬁned as E s /E
total , where E total and E s are the energycost of the baseline model and the skipped layers due to EnergyNet .
From Figure 3, we can concludethat EnergyNet achieves the goal of reducing energy cost while preserving or even improving theprediction accuracy.
In particular, the accuracy of EnergyNet-38 and EnergyNet-50 will not drop whenthe energy savings is as high as 40% and 65%, respectively.
We observe that EnergyNet can achievea higher accuracy than the original ResNet model.
We conjecture that this is because EnergyNet canovercome overﬁtting when performing the dynamic routing for energy savings.
Further EnergyNetcan aggressively reduce energy cost by about 4 × , over both the ResNet-38 and ResNet-50 baselines,at the cost of 3% and 4% testing accuracy losses , respectively.
Future Works
It is interesting to further explore the energy reduction and accuracy improvement of the EnergyNetover different CNN baselines including DenseNet [19], GoogLeNet [20], and MobileNet [7] onpopular benchmark datasets.
Furthermore, EnergyNet can be extended to employ reinforcementlearning models for gating decisions.
Finally, Expectation-Maximization [21] can be employed toiteratively update the gating decisions for learning better dynamic routing policies in EnergyNet .