From: NIPS.cc/2018/Workshop/CDNNRIA
title: Acceptance Decision
decision: accept

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper32/Authors
comment: Thanks to the Reviewer for the constructive comments.

1. We compared our method with AFP[7] on VGG-16 on ImageNet, and the result is as follows (increased top-5 error on ImageNet validation dataset, in the order of "ours, AFP").
2x: 0.0%, 0.1%
4x: 0.8%, not converged
5x: 1.5%, not converged

Implementation background: according to the paper of AFP, their pruning process is basically three steps: (1) train the network with their proposed auto-balanced regularization, (2) prune each layer based on L1-norm by a pre-specified ratio (i.e. the pruning ratio in our paper), (3) finetune to regain accuracy. Among them, step (1) and (2) can be repeated many times for iterative pruning, for which, AFP proposed an 'Abreast Advancing' scheme to schedule the whole iterative process. We didn't use 'Abreast Advancing' in our experiment, since it's not about the pruning itself, and our method can also be applied in this way. Pruning of AFP was conducted based on the rule proposed in AFP: regularization factor (i.e., alpha in their paper) is set to one order larger of the original weight decay.

In our experiment, AFP[7] doesn't converge for the 4x and 5x speedup setting, after the step (2). We repeated but got the same results. After checking the regularization factors and losses during pruning, we think a probable explanation is that, the regularization factors of AFP are still quite small (10 * original weight decay), which cannot drive weights into two distinctively different groups (to-be-pruned and to-be-saved), i.e., it cannot cut off the dependence between the important and unimportant weights. Then when *a large ratio* (note that under 2x speedup, AFP converged and performed just slightly worse than ours) of parameters are removed from the network suddenly, the network suffers from an irreversible expressiveness damage (an evidence is that, the losses jumped up significantly just after step (2)) and cannot converge.

In contrast, our method increases the regularization factors incrementally. In this way, we can finally increase the regularization factor to an unprecedentedly large level (10000 * original weight decay). And so large regularizations manage to drive weights into two distinctively different groups. There is hardly dependence between the important and unimportant weights. Then at this time, we remove these unimportant weights, which hurts little to the network.

To further compare with AFP[7], we also evaluate them with ResNet-56 on CIFAR-10:
AFP   speedup:2.6x   increased error:1.0
Ours  speedup:3.1x   increased error:0.4
AFP   speedup:3.4x   increased error:3.4
Ours  speedup:4.0x   increased error:0.9
It can be seen that our method consistently achieves less increased error under larger speedup. And especially, like the experiment of ConvNet, when the speedup ratio is larger, the advantage of our method is more significant. In fact, the 'Abreast Advancing' scheme proposed in AFP is similar to our incremental regularization in spirit, but our scheme is clearly more "gentle" and flexible in dealing with hard situations (compact networks, larger pruning ratios).

2. It would be interesting to dig deeper to see how our method works with more regularization-based methods [29-30], but given the limited time of this open review, we will leave that for future work. Many thanks to the Reviewer for this idea.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper32/Authors
comment: Thanks to the Reviewer for the constructive comments.

1. The layer sparsity (both row and column sparsity considered) for pruned VGG-16 is listed as below, in the order of the 13 conv layers.
2x: 0.625-0.773-0.586-0.771-0.395-0.770-0.160-0.655-0.145-0.655-0.027-0.541-0
4x: 0.625-0.858-0.896-0.817-0.848-0.855-0.789-0.743-0.689-0.669-0.536-0.541-0
5x: 0.625-0.892-0.923-0.844-0.880-0.890-0.837-0.804-0.765-0.722-0.658-0.627-0

As for ResNet-50, since it has too many conv layers (53), we don't list the exact numbers here. But it's easy to estimate the average layer sparsity. We mentioned in our paper that the pruning ratio of ResNet-50 is uniformly set for all conv layers and given the 2x GFLOPs reduction, so averagly the final layer sparsity is 50%. Some layers are a little higher and some are a little lower, but overall 50% is fairly close.

2. Truly sorry for the missed citation, which we have added in the revised version.

3. Unimportant weight groups are selected according to a proposed importance criterion (Line 62), which is based on the commonly-used L1-norm, but we added an averaging over time for smoothing.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper32/Authors
comment: Dear Reviewer,
Thanks for your constructive suggestions.

1. The experiment of comparison with AFP on VGG-16 on ImageNet is going on. We will release the result soon. (update: the result  and analysis of AFP on VGG-16 is releasd, please see the comment reply  in the part of Reviewer 1 below)

2. We are not sure whether our understanding about the "comparisons with naive ways of incremental regularizations such as linear or exponential increment" is right. Our method increases the regularization in a linear way (Fig.1) , meanwhile assigning less penalty to the more important weight groups. Do you mean, we should compare with using weight decay as regularization form, while increasing the regularization factor in a linear or exponential way *for all weight groups*?

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper32/AnonReviewer2
title: Review comments
review: This paper proposes a regularization-based pruning method to incrementally assign different regularization factors to different weight groups based on their relative importance. Evaluations are performed on Cifar-10 comparing with state-of-the-art methods.

Pros: 
Easy to read and implement/reproduce
Effective

Cons:
Lack of comparisons with AFP 
Lack of comparisons with naive ways of incremental regularizations such as linear or exponential increment. 
rating: 4: Top 50% of accepted papers, clear accept
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper32/AnonReviewer3
title: Incremental regularization based pruning of NNs: A sound and novel approach achieving good results.
review: The authors propose an incremental regularization based pruning method. The method incrementally assigns different regularization factors to different groups based on a relative importance.

Pros:
The method is sound, novel and achieves good results on speedup-accuracy trade-off.

Cons:
Although there is a speedup analysis, I would also expect a sparsity percentage analysis after the method is applied.

Comments:
1) In first page, line 31, please add a citation to the claim: "few works have noticed that the effectiveness of CNNs is so fragile ..." The provided reference at the end of this sentence refers to compact networks, not the "few works" you have mentioned in the sentence.

2) Please clarify how to selection of the weight groups were made.



rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper32/AnonReviewer1
title: No Title
review: Review:
This paper is mainly based on AFP [7] and propose to add the balanced regularization incrementally. The proposed method provides better results compared to the AFP [7] baselines and some prior art. Overall, the idea of incremental regularization is interesting and leads to good results on CIFAR-10 compared to the baseline AFP [7], which worth digging deeper for a more comprehensive analysis, for example, use it in more regularization-based methods [29,30] and compare.

Pros
- Easy to follow
- Interesting idea that the regularization term should be applied incrementally
- Good results

Cons
- Table 2 is not comparing with AFP [7]. Comparing this work with SPP is not apple to apple since SPP does not use balanced regularization in the first place. To support the claim that incremental regularization is effective, it would be better to compare AFP with this approach on ImageNet.


[29] Liu, Zhuang, et al. "Learning efficient convolutional networks through network slimming." Computer Vision (ICCV), 2017 IEEE International Conference on. IEEE, 2017.
[30] Gordon, Ariel, et al. "Morphnet: Fast & simple resource-constrained structure learning of deep networks." IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct