From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/AnonReviewer3
title: Official review for comment from 21st of Oct.
review: My comment from 21st of Oct was intended as a review. Back then, the review functionality was either not available, or I somehow missed it. Instead of repeating the whole text, I will just update the scores and refer to the original comment for the detailed review.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/Authors
comment: We thank the reviewer to find the idea interesting, simple and effective. 

We agree with the reviewer that the proposed method is related to the early prediction method in [17]. However as the reviewer noticed here different objective function is utilized and it is one of the contributions of this paper. To show the effectiveness of this new objective function, we conducted the second experiment mentioned in section 3 and we reported the result in Figure 2.
As seen "with the same number of FLOPs in the network, the network where the decision gates were trained based on the proposed hinge loss provides much higher modelling accuracy compared to that trained via cross-entropy loss."

As discussed with other reviewers as well, we agree with the reviewer a more comprehensive experiment is needed for the full version of the paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/AnonReviewer2
title: Need more clarity in writing
review: This paper proposed an interesting idea to inserting decision gates to layers of neural networks and train with hinge loss. The inference time could be greatly saved because prediction could be made from these d-gates without running the whole network.

Pros: simple but effective ideas.
cons: 
(1)  The idea is not novel. I think this paper mainly follows the idea of early prediction method [17] . Although the authors claim that  "not only are these techniques quite difficult to set up, they also require the network to be trained from scratch with these techniques integrated", in this paper they didn't clearly emphasize the difference of the proposed method compared with previous methods. I noticed that the authors used differnt objective function compared with [17] but there are no theoretical and experimental comparison to validate the choices of the proposed method. In addition, from my point of view, previous prediction methods such as [17] can also be simply modifed  not to train from scratch. In summary, the authors need to both theoretically and experimentally compare the proposed method with existing methods to validate the novelty of their paper.


rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/Authors
comment: We thank the reviewer for finding the new experiment to be interesting and illustrative of the contribution of the paper. 
We have now moved the new experiment from the appendix to the main text of the updated paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/Authors
comment: We thank the reviewer for finding the updated paper to be worthy of discussion with the other colleagues in the workshop.
We also agree with the reviewer that the method must be thoroughly compared against other methods both theoretically and experimentally for the full version of the paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/AnonReviewer3
comment: Thanks for addressing some of the issues raised by the reviewers and showing some additional results.
For me personally, this pushes the paper above the threshold for presentation at a workshop in order to discuss and further refine the work with colleagues from the field *before* preparing a full paper. 
I want to strongly encourage the authors to take into account some of the more substantial comments from the reviewers when working on a full-paper version. In particular, the method must be thoroughly compared against other methods both in theory but also empirically (on a fair basis that allows for comparison).

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Thank you for the clarification and the updated version.

I appreciate that you show an interesting comparison between hinge loss and cross entropy loss and this itself is a good contribution. However, I find it not convincing to compare with prior art by just viewing the numbers:
1. To really justify your claim that [10] and [16] (reinforcement learning learned controller) is worse than the proposed hinge loss learned classifier, the placement of the gate and the network structure has to be the same.
2. The comparison between this work and [15] is not an apple to apple comparison either.
3. The authors claim "The early prediction function learning proposed in [18] is quite different than that of the proposed d-gate method, which is one of the key contributions", which is vague. Please be specific to justify your contribution.

I'm willing to raise my score for the interesting experiments demonstrate in the supplementary material (hinge loss vs cross-entropy loss) and I appreciate that discussion to be in the main text since I think that is the main contribution.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/Authors
comment: The authors thank the reviewer for finding the idea to be interesting and useful for industrial applications. 

To better show the effectiveness of the proposed method compared to past approaches, we conducted a new experiment, which has been included in the revised paper.
Specifically, to demonstrate the effectiveness of the proposed hinge loss in the training of decision gates, a quantitative performance comparison was made against when cross-entropy loss, which is what was used in previous work, is leveraged in training the decision gates instead.  Results clearly demonstrate that leveraging the proposed hinge loss for training decision gates yields significant performance benefits over the use of a cross-entropy loss.  The paper has been updated and the new experimental results has been included in the revised paper.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper13/Authors
comment: The authors thank reviewer for finding the paper to be clear and easy to follow and understand, as well as appreciate that the proposed method does not require re-training for model acceleration, which is one of the key contributions of the paper alongside better accuracy.  Here are the responses to the reviewer's comments:

- With regards to the comment that methods like [16] and [10] does not need to be retrained, while it is technically true that it is possible to use these methods without retraining the network, in practice there is very significant drops in accuracy if retraining is not performed.  It is clearly mentioned in [16] (Impact of Joint finetuning Section) that, without the joint training of the controller and the network, there is huge 18.2% drop in accuracy for CIFAR10 using ResNet-110 (Results in Table 1).  Similarly, it is clearly mentioned in [10] ("Joint Training of All Nodes" section) that all nodes including the controller nodes and regular nodes are trained jointly such that authors explained how the backpropagation is applied in that complex network.  Therefore, in both these cases retraining is needed to greatly improving the accuracy of the network for it to be useful in practical situations. 

With regards to [15], as we mentioned in the introduction, the proposed method is related to the method proposed in [15].  We agree with the reviewer that one of the novelties of our work is to use of the hinge loss, which addresses a very important limitation of [15] leveraging cross entropy loss that makes it very difficult to scale and achieve proper accuracy.  It has been argued in studies such as [20] that cross entropy loss provides a small margin between the decision boundaries and the training data. This drawback is one of the main issues that makes it very difficult to trust the confidence values of the Softmax layer to decide about the sample. By leveraging hinge loss, we are able to mitigate this drawback and achieve much better decisions about the sample.  It is also worth mentioning that the accuracy for CIFAR10 using ResNet-110 reported in [15] is 80.70%, which is much lower that what can be achieved in past literature (>93%).  In comparison, the results achieved using the proposed d-gate method is much higher than that reported in [15], which illustrates the effectiveness of the proposed method.  We have now added this discussion to the revised paper as requested by the reviewer for clarity.

- Thank you for highlighting [18], which we have added to the revised paper.  The early prediction function learning proposed in [18] is quite different than that of the proposed d-gate method, which is one of the key contributions, and the performance drop demonstrated in [18] is quite large in comparison to that demonstrated in the proposed method.  We have now added this discussion in the revised paper for clarity.

Furthermore, we have updated the revised paper and included more experiments to demonstrate the effectiveness of hinge loss in training of the decision gates and compared it with when cross-entropy loss is leveraged in training the decision gates instead.  Results clearly demonstrate that leveraging the proposed hinge loss for decision gates yields significant performance benefits over the use of a conventional cross-entropy loss.


[20] Liang, Xuezhi and Wang, Xiaobo and Lei, Zhen and Liao, Shengcai and Li, Stan Z, "Soft-margin softmax for deep classification", NIPS 2017

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: To reduce computational burden during forward-pass, the authors propose using “d-gates” for conditional execution of deeper layers of feed-forward neural networks. The main idea is to attach linear classifiers at various stages of the network and determine whether an input can already be classified with high confidence. If so, further processing of the input through the network can be skipped, thus saving computation. The authors demonstrate their method on ResNet-101 and DenseNet-201 using the CIFAR-10 data-set.

The idea of conditional feed-forward execution is interesting and of high relevance for industrial applications, where tight cost- and compute-budgets must often be met. As such, the idea has been previously proposed and explored in the scientific literature. As the other reviewer also points out, the current paper is of limited novelty. While the paper has a somewhat lengthy Introduction and Method section, it fails to compare in-depth against competitor methods, both on a principled level but also empirically. Given the results presented in the paper, it is currently not clear whether there is a significant benefit to using the proposed method. I argue for postponing the presentation of this work until it has been fleshed out more thoroughly and compared against other methods in a set of experiments.

Rating: 6 of 10  (above acceptance threshold) for the revised manuscript
Confidence: 3 of 5  (somewhat familiar with related literature, understood the method)

Major issues:
1) The method is strikingly simple and straightforward - which is a plus. However, the paper lacks a detailed comparison against competitor methods, both conceptually (there is currently one sentence in the paper) but also empirically (no comparison of results). Since the novelty of the method is limited, better empirical performance could be a convincing reason for considering the method - but this needs to be shown.

Minor issues:
3) While the writing is good in principle, it could need some more focus. Given the very restricted length of the workshop-paper, I would suggest to significantly shorten the Intro (almost a third of the whole paper) and write a crisp method section. This would allow to extend the comparison against other methods and have a larger results-section.

From: NIPS.cc/2018/Workshop/CDNNRIA/Reviewers
comment: Rating: 3 of 5 [revised]

Review:

The authors propose d-gate, which is used to determine whether or not the deep neural network should proceed to deeper layer or conduct an early stopping. With the proposed d-gate, empirical results demonstrate computational saving without accuracy loss on two modern deepnets. However, the idea is not new and it lacks comparison with similar approaches.

Pros
- Clear motivation and easy to follow and understand.
- No re-training needed for model acceleration.
- [revised] Authors provide an interesting experiment showing the benefit of hinge loss classifier over cross-entropy loss classifier in the context of early exit.

Cons
- The idea itself is not novel and is similar to the cited papers [10,15,16] and the argument made in this paper that those methods require the network to be trained from scratch does not necessarily hold. For example, [16] does not train the network but the controller (skip or not) with reinforcement learning and [10] also discusses the case of training only the controller with reinforcement learning. Moreover, the benefit of this work naturally exists in [15], where the network could be trained by freezing the parameters other than the controller with commonly adopted SGD. As a result, the value of this work boils down to whether hinge loss trained classifier is better than cross entropy loss trained classifier [15] or reinforcement learning trained controllers [10,16], which are not discussed.
- Missing one ICML 2017 reference that is really relevant [18]. In Section 3 of [18], the author defined formally of the problem this paper is addressing with various kinds of controller function families. As a result, [18] also enjoys the fact that no re-training is needed for the pre-trained network and when the function family is simple (e.g., linear classifiers as in this paper), the optimization process is easy to solve. Therefore, it again reveals that the idea itself is not novel and requires more comparisons to justify the technical contributions.

Minor suggestions
- A function doesn't have to be convex to be optimized with gradient descent.
- The equations (1) and (2) are for binary SVM classifier but the context in this paper is multi-class SVM.

[18] Bolukbasi, Tolga, et al. "Adaptive neural networks for efficient inference." In Proceedings of the 34th International Conference on Machine Learning, 2017.

Confidence: 3 of 3