From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/AnonReviewer2
title: This is my official review
review: My actual review was my first review and mistakenly an "official comment." Please read the official comment for my review, and I post here to complete this outstanding task. 
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/Authors
comment: We have some updates for question 3. We did multiple trials experiments of EnergyNet-38 on the point which has similar accuracy as that of the baseline ResNet-38 (92.50%). The average energy savings is 40.0475% and average accuracy is 92.535%. The variance of energy savings and accuracy is 0.64059 and 0.09233 respectively. Thanks for your valuable suggestion.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/Authors
comment: Thank you for bringing up this excellent point. We were not able to elaborate this in our manuscript due to the 3-page limit.

1)
Yes, the energy of a memory access (e_i) changes depending on the corresponding memory hierarchy. In our energy model, we consider the most commonly used design of three memory hierarchies including the main memory, the cache memory, and local register files. Specifically, e_i(main memory) = 200, e_i(cache memory) = 6, and e_i(local register file) = 1, according to the chip implementation data in [16].

2)
Yes, in our energy model we do attempt to figure out the optimal ordering of values stored in different memories to ensure that “more commonly accessed memory values should be stored in a location that takes less energy” for minimizing the total energy cost. In particular, we employ state-of-the-art dataflow optimization tools in [16],  [17] and [22] to obtain such an optimal ordering, and assume a state-of-the-art hardware architecture which consists of three memory hierarchies (including a main memory, a cache memory, and the local register files) and a processing unit array. Next, the resulting number of accesses corresponding to different hierarchical memories are then plugged into our energy model (see Eq. (2)) to obtain the associated energy cost.

[22] Hyoukjun Kwon, Michael Pellauer, and Tushar Krishna, An analytic model for cost-benefit analysis of dataflows in DNN accelerators, arXiv:1805.02566v2 [cs.DC], 2018

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/Authors
comment: Yes, it is the small variance of 40.0475%+-0.64059%

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/AnonReviewer2
comment: Just to clarify, this variance is in percent, correct? That s, the average is 40.0475%+-0.64059% (a small variance) and not plus or minus 64% (clearly a big variance).

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/AnonReviewer2
comment: Thank you for these responses! I am even more confident that this paper passes the acceptance threshold owing to these comments, so I definitely suggest that this paper be suggested, as it is a solid contribution and the responses have been very informative. I am still interested of course if pruning techniques can be combined with the skip policy technique.

I ask some more questions for clarification below:

As you say, the value of e_i and e_MAC denote energy costs takes into consideration the wiring length of these accesses as derived from chip implementation data, does:
1. the energy of a memory access change depending on where the value in memory is stored?
2. If so, does the optimization procedure figure out (or attempt to figure out), an optimal ordering of values stored in memory? The idea being: more commonly accessed memory values should be stored in a location that takes less energy. 
Perhaps for the particular implementation the value of each memory access is the same (which is reasonable I suppose for certain types of memory). if so, could you let us know? Thanks!

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/Authors
comment: Thanks for your accurate description of our work and the valuable suggestions. Below is our one-on-one response.

0) To the question of "How does the model used contrast to, say, the model used in this paper (On Learning Wire-Length Efficient Neural Networks) also submitted in parallel to this workshop?"
Our model doesn't contrast to the model in the paper "On Learning Wire-Length Efficient Neural Networks". In fact, our energy model also captures the influence of the wiring length as in the unit energy cost, e_i and e_MAC, which denote the energy costs of accessing the i-th memory hierarchy and one MAC operation, respectively. We obtain the value of e_i and e_MAC from the chip implementation data in [16].

1)
Thanks for your insightful suggestion. EnergyNet is designed to reduce inference costs adaptively for general deep learning networks, while pruning leaves no room for inference adaptivity. In addition, EnergyNet makes an orthogonal effort to trim down the energy cost as compared to the pruning technique. As such, we would expect a pruned backbone to not hurt the effectiveness of EnergyNet, and applying EnergyNet on top of a pruned network would lead to further energy savings while maintaining or even improving the accuracy. We are working on the suggested experiments and will update you once there are any new results. 

2)
The left and right plots in Figure 3 show our results for EnergyNet-38 trained on CIFAR10 and EnergyNet-50 trained on Tiny ImageNet, respectively. Taken EnergyNet-38 as an example (the left plot), points on the blue line in the plot represent EnergyNet-38 models with different testing accuracy versus energy savings ratios. In other words, these points correspond to trainings of EnergyNet-38 with different values of the coefficient alpha in Eqn. (1) in our paper. We control the testing accuracy versus energy savings ratio by setting different values for this coefficient. alpha determines how much weight the model puts on energy savings. The larger the alpha, the more energy is saved, but the model will also become less accurate. The blue line in the plot demonstrates the trade-off between accuracy and energy savings.

3)
Thank you for your valuable suggestion. We are conducting the experiments and will try to report the suggested statistics in our final version (it might take a longer time than the rebuttal timeframe). Also, we observe the results to be stable and reproducible in all our experiments so far. 

Thank you very much for your careful reading. We will correct the typos in our next version.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/Authors
comment: We appreciate your reviewing efforts and are providing our one-to-one responses to your comments below.

1)
Our dynamic routing algorithm follows the idea sketch of [11]. However, while [11] uses the number of operations to optimize accuracy and reduce the number of operations, we optimize the energy cost in addition to accuracy. By recognizing that energy cost and the number of operations are often not aligned [2], our main highlight is to adopt a realistic energy estimation model (2), which also takes into account the memory access and data movement costs. 
Furthermore, using the energy model to derive the policy learning, we show that nontrivial energy reduction could be achieved with little performance sacrifice. More encouragingly, we observe that the energy awareness might serve as a training regularization and can even improve the prediction accuracy: our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively. Up to our best knowledge, similar observations have not been reported in [11] or related works. In addition, while [11] suggests that reinforcement learning outperforms the standard supervised training, we did not find the necessity of using reinforcement learning, which potentially due to the change of the training objective function, and hence stay with the standard SGD to learn the proposed EnergyNet. We will also clarify the above differences in our final version.

2)
The reason that our energy model neglects the cost of nonlinear operations is that the energy cost of the nonlinear operations in most CNNs is commonly recognized to be negligible (< 1%). First, the number of nonlinear operations is much smaller than that of the MAC operations. Taking a convolutional layer as an example, the number of MAC operations is (R*S*C*M) times of that of the nonlinear operations, where R/S is the weight filter height/width, C is the number of the weight filter channels, and M is the number of the weight filters. Second, the unit energy cost of the nonlinear operation is also smaller than that of a MAC operation. Third, it is well recognized that the energy cost due to the memory accesses and data movements dominates in CNNs [16]. 
Although the energy model itself might suggest going for narrow-deep networks, please be reminded that it says nothing about the resulting accuracy. That is exactly why we want to optimize a hybrid loss combining the accuracy and energy cost: while narrow-deep networks can favor the latter, they will be resisted by the former. In experiments, we observe that optimizing this hybrid loss leads to highly favorable trade-offs between accuracy and energy costs, on state-of-the-art ResNets.

3)
We previously tried ignoring the first step as you suggested. It turns out that the model needs much more iterations (about 50% more) to get to the same accuracy level. Using pre-trained models can trim down the required training time and therefore may be practically preferred, if one has off-the-shelf base models to choose from. 
Besides, we in this paper focus on reducing the energy consumption during inference time (a more demanded feature in IoT), and didn't take into account the energy cost of training. It is one of our future works to extend EnergyNet to energy-efficient on-the-fly training. 

4)
As pointed out by [11] etc., even in original ResNets, an image input may likely only activate a subset of layers while (almost) skipping the others. That is due to the selective/discriminative processing of different class information in different Residual Blocks. Utilizing this natural observation, EnergyNet further promotes such input-specific, intentional "skipping" on unnecessary layers that have minor impacts on final accuracy. Such "skipping" is different from random perturbation of preceding layers that will inevitably degrade accuracy. 

5)
The suggested architecture search/lifelong learning is a very neat and worthy idea to explore. Compared to EnergyNet and other "pruning"/"skipping" ideas in CNN compression, it is like two complementary views to save: bottom-up versus top-down. However, we note that the goal of EnergyNet is to trim down costs during inference, without the need of retraining, after the routing is learned (we call it "adaptivity for free"): when deployed at edge devices, the skipping and routing will happen adaptively and automatically to save energy. In contrast, to achieve the same "adaptivity" using the architecture search, it has to research the architectures at the edge, whose corresponding costs can become unacceptable.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/AnonReviewer2
comment: The authors introduce EnergyNet: a deep convolutional neural network model which has the purpose of classifying images with lower energy. The model uses standard techniques with a key innovation: adding energy to the loss function in training to learn a policy for when layers can be "skipped" in the computation. Their experiments show that the technique can reach state of the art accuracy and reduce energy consumption. Overall I think the paper is solid and worthy of acceptance, although since pruning algorithms tend to have energy savings far more than this paper, a comparison to a pruned network as a baseline would make it much stronger. [3/5].

Strengths:

1. The idea is clever, seems to be well motivated, the implementation is solid, the paper is well-written and clear, and the results comparable to state-of-the-art. The authors consider an energy model that to me seems reasonable that posits that energy consumption is related to the number of memory accesses and number of computations. The idea is novel: since "skipping" sections of the convolutional model can save energy (by saving on memory accesses and computations), the authors propose to learn a skipping policy that optimizes for accuracy and their (very reasonable) model of energy. 

2. The authors use a more sophisticated energy model than the commonly used number of paramaters metric. This seems realistic. Though there are space constraints, if the authors submit a longer version to a conference/journal, illuminating this and contrasting the work to previous work would help make the paper more distinguished.  How does the model used contrast to, say, the model used in this paper also submitted in parallel to this workshop?

https://openreview.net/forum?id=rygpbR2Pi7

Areas for Improvement:
1. The energy savings are decent, but compared to, say, pruning papers, the energy savings are small at best, when the pruning papers can save something like 50 times the number of parameters. A better baseline would be to compare this technique to pruning techniques (and perhaps use it in conjunction with pruning techniques) to show that it provides an additional energy savings. A key doubt I still have (that could be verified by experiment) is that a pruned network would consume less energy than this non-pruned skip connection network. I suspect that the algorithm can be adapted to learn the skip policy along with pruning, but possibly in that case the skip policy wouldn't work. We don't know and I think this requires experimental verification. 

2. It's not clear to me what is present in the figures. From what I see described there are two different architectures designed and tested: so why do I see multiple points connected by lines? Shouldn't there be just one energy value and one accuracy value for each model? Now, I am supposing then that something was varied and this is a result of multiple trials, but what exactly was varied? Does it make sense to connect these lines? I think this should be clarified very briefly (given space constraints) in the paper.

3. The result of one trial is shown. It would give us more confidence if the average and variance from multiple trials is shown that the particular outcome is not just a coincidence. 


Typos:
line 108: "the model sees am accuracy loss"-> should be an.
line 24: the authors awkwardly use "cost" twice in this sentence.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper41/AnonReviewer1
title: I think the paper is interesting. However, some issues need to be addressed. Maybe due to lack of space, some details are missing.
review: The paper is well-written and is suitable for this workshop. The suggested method could potentially be used in industrial applications. There are however some comments that need to be addressed:

1- About the originality, it is not clear how this paper is different from reference 11. Could the authors elaborate on that?  

2- In equation 2, where energy costs are defined, it seems that only linear operations have been taken into account. Why did you neglect the nonlinear operations? Also, does it imply that in your algorithm, a very narrow network is preferred over a wide network? This can have severe effects on the accuracy.

3- The network is initially trained like a base network, and only after that, alpha is increased and the network is retrained. Why is it not possible to skip the first step? The retraining phase is not efficient and can consume lots of energy.

4- It is not clear how (based on what criteria) one layer is skipped, and how the gates take a value of zero or one for further processing. In lines 55-59 this has not been explained. Also, it is not clear why when one layer is skipped, the next layer will not undergo degradation, and therefore does not change the accuracy of the output. It has been shown in the literature that perturbing initial layers of a ResNet can have drastic effects on the accuracy. If these layers are removed, the whole retraining will take approximately as long as training the original network. Did the authors consider that? 

5- What is the advantage of this algorithm over a very simple network that is initially designed as narrow and shallow as possible and then later extra depth or width is added to the network?
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct