From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper28/Authors
comment: Thank you for your feedback. We were indeed unaware of MorphNets, which also use a relaxation of the FLOPs objective as part of regularization. After reviewing their paper, we believe that these are the most important differences:

1) MorphNets (Gordon et al., 2018) are more closely related to neural architectural search. The authors train multiple models to find reasonable configurations, and then expand the layer widths of the best architecture to satisfy a FLOPs constraint. In our work, we learn both the weights and architecture that satisfy a given FLOPs requirement in a single training run.

2) To approximate the FLOPs objective, MorphNets use an L_1-norm penalty, which promotes sparsity by shrinkage. In our case, we use a similar framework in Louizos et al. (2018), where we follow an approach motivated by variational inference.

We again thank the reviewer for informing us about prior art. We will note the similarities and differences in the revision.

References
Gordon, A., Eban, E., Nachum, O., Chen, B., Wu, H., Yang, T. J., & Choi, E. (2018). Morphnet: Fast & simple resource-constrained structure learning of deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Louizos, C., Welling, M., & Kingma, D. P. (2018). Learning Sparse Neural Networks through $ L_0 $ Regularization. In International Conference on Learning Representations (ICLR).

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper28/AnonReviewer1
title: One important reference is missing and required to compare
review: This paper proposes to use FLOPs as  regularization terms in objective function for training sparse neural networks, and demonstrates its effectiveness through a set of experiments on MNIST and CIFAR-10 /100. 

The idea itself is interesting, but not very novel. There is a prior art in this year CVPR by Google named "MorphNet", which the authors are not aware and cited. I suggest the authors making a thorough comparison to this paper both on idea and experiments. 

rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper28/Authors
comment: We thank the reviewer for their comments and concerns. We provide the following responses to the questions:

1) Here, the pseudo-reward can be considered the clipped FLOPs objective itself under the Bernoulli r.v.'s (term within second expectation of Equation 4). Since we are doing gradient descent using REINFORCE/SFE as a black-box estimator, the FLOPs penalty is minimized. At each training step, the number of FLOPs is computed for the architecture associated with every random sample from the Bernoulli r.v.'s; for example, in LeNet5-Caffe, a possible random draw could be 18-45-720-430 -- the FLOPs for each layer according to this architecture are computed using the formulas in Section 2.1. Then, to compute the objective, the FLOPs for each layer are summed up and shifted and clipped appropriately. Finally, we use REINFORCE/SFE to optimize the objective.

2) T can be interpreted by the practitioner to be the desired number of FLOPs incurred by the final sparse network, and λf controls the strength of the penalty and hence rate of "convergence" to the desired FLOPs target. A good rule of thumb would be to first fix T to (a range of) desired number of FLOPs, and then set λf to something reasonable, i.e., a λf that results in neither immediate nor slow convergence -- in that sense, it is analogous to the process of choosing a good learning rate.

3) On the CIFAR-10 runs, actual FLOPs had mostly converged to the target by epoch 40, while the expected FLOPs continued to fluctuate very slightly due to the stochasticity of the gates. Due to space constraints, we were not able to include these graphs.

4) We agree and think large-scale ImageNet experiments will be necessary in a future paper submission.

We again thank the reviewer for their feedback. We will clarify several points in the revision.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper28/AnonReviewer2
title: An extension of Louizos et al.'s ICLR 2018 work
review: This paper proposes to use FLOPs as an additional regularization term for training structured sparse neural networks. Validation experiments are conducted on popular datasets like MNIST, Cifar10/100.

Introducing FLOPs into the optimization objective for compressing neural networks is really interesting. The proposed formulation extending Louizos et al.’s ICLR 2018 work also seems sound.

Here are some questions I would like the authors to address.

1 As the FLOPs term in the objective is not differentiable, REINFORCE algorithm is used to address the issue. Here, what is the pseudo reward? How it is updated during the training? Besides, how FLOPs is actually computed at training phase?

2 Is there any principled way to choose hyperparameter T? Also, in the experiments, hyperparameter λf  has different values w.r.t. tested neural networks. How about the performance when changing its values.

3 How do the expected and actual FLOPs change during neural network training?    

4 The performance of the proposed method on large scale datasets like ImageNet shall be really helpful to better understand its potentials. 

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct