Paper: Pruning via Ranking (PvR): A unified structured pruning approach (ICLR-2024)

Reviewer comment (Experiments):
Lacks validation on the proposed grouping module.

Author response:
Based on the feedback received, we have added an ablation study to observe the change in accuracy of pruned networks with increasing group size as well as the time required for pruning under varying group sizes. Specifically, we look at the $98$% pruned version of the VGG16 and ResNet34 networks for group sizes varying from $2-32$ and $4-32$, respectively. We perform this ablation study for only the maximum level of pruning as it is the worst-case scenario in terms of both accuracy and time. According to Fig. 2a and 3a in Section A.2 of the Appendix, PvR is generally robust to group size as the top-1 accuracy drop from group size of $2$ to $32$ is about $1$% and the top-5 accuracy drop from group size of $4$ to $32$ is about $2$%. At the same time, with a group size of $32$, PvR is able to prune VGG16 and ResNet34 under $20$ seconds and $12$ minutes, respectively, as demonstrated in Fig. 2b and 3b.

Reviewer comment (Experiments):
How can different grouping settings impact the performance of pruned model?

Author response:
Based on the feedback received, we have added an ablation study to observe the change in accuracy of pruned networks with increasing group size as well as the time required for pruning under varying group sizes. Specifically, we look at the $98$% pruned version of the VGG16 and ResNet34 networks for group sizes varying from $2-32$ and $4-32$, respectively. We perform this ablation study for only the maximum level of pruning as it is the worst-case scenario in terms of both accuracy and time. According to Fig. 2a and 3a in Section A.2 of the Appendix, PvR is generally robust to group size as the top-1 accuracy drop from group size of $2$ to $32$ is about $1$% and the top-5 accuracy drop from group size of $4$ to $32$ is about $2$%. At the same time, with a group size of $32$, PvR is able to prune VGG16 and ResNet34 under $20$ seconds and $12$ minutes, respectively, as demonstrated in Fig. 2b and 3b.

Reviewer comment (Evaluation):
How much it can speed up the overall process?

Author response:
Based on the feedback received, we have added an ablation study to observe the change in accuracy of pruned networks with increasing group size as well as the time required for pruning under varying group sizes. Specifically, we look at the $98$% pruned version of the VGG16 and ResNet34 networks for group sizes varying from $2-32$ and $4-32$, respectively. We perform this ablation study for only the maximum level of pruning as it is the worst-case scenario in terms of both accuracy and time. According to Fig. 2a and 3a in Section A.2 of the Appendix, PvR is generally robust to group size as the top-1 accuracy drop from group size of $2$ to $32$ is about $1$% and the top-5 accuracy drop from group size of $4$ to $32$ is about $2$%. At the same time, with a group size of $32$, PvR is able to prune VGG16 and ResNet34 under $20$ seconds and $12$ minutes, respectively, as demonstrated in Fig. 2b and 3b.

Reviewer comment (Evaluation):
What makes PvR perform well on both vision and NLP models? Why couldn't other pruning methods achieve this? Would like to hear more insights on this claim.

Author response:
We are delighted by this insightful question posed by the reviewer. The main idea of pruning is to generate a smaller model that performs at par with the unpruned network. The brute-force solution to this task is to mask each neuron/filter/weight at a time and look at the change in the final output. If the neuron/filter/weight causes a large change in the output then it can marked as important. Most of the works in literature avoid solving the pruning problem in this way as it is extremely inefficient and requires multiple forward passes over the entire data to compute the importance for a single neuron/filter/weight. PvR takes this exact brute-force route but avoids the large computational overhead by requiring a very small subset of the data to determine the importance of neurons/filters/weights. It also uses a carefully designed grouping module to mask multiple neurons/filters at the same time. CURL$^{[1]}$ is another method which solves the problem in a similar way but we have shown in our work that the ranking function proposed by us is superior as it takes all possible scenarios into account. We urge the reviewer to look at the ablation study that has been added to the manuscript wherein we demonstrate the robustness of PvR to group size as well as the small computation time required to prune large networks.

Reviewer comment (Evaluation):
How long does Grouping and Ranking processes each take? How does the total time PvR takes compared to other pruning methods?

Author response:
We urge the reviewer to look at the ablation study that has been added to the manuscript wherein we demonstrate the wall-clock time required to prune large networks using PvR.

Reviewer comment (Reproducibility):
Is there any strategy to initialize and retrain the pruned model (does it have any relationship, like some proportional to the original model)?

Author response:
The pruned networks are initialized using the same process as the unpruned models where the weight initialization strategy for each unpruned model is reported in their respective papers. We use the latest weight initialization techniques recommended by PyTorch. The recipe for training both pruned and unpruned models is exactly the same. These details are provided in the Configuration sub-section under Section 5.1.

Reviewer comment (Experiments):
Lack of essential ablation studies. The authors introduce two very important hyperparameters: the constant error "d" and the group size. I think they will have a great influence on the final accuracy. The authors claim their algorithm applys to both vision and NLP models. However, it would be meaningless if we have to carefully adjust these two hyper-parameters for the best accuracy on different models and tasks. This is my main concern.

Author response:
We thank the reviewer for bringing this to light. As explained in Section 3.2 after Eqn. $2$, the value of $d$ needs to be set to $1$ irrespective of the model and dataset. In fact, this is the value that we chose for all our experiments. The reasoning for this exact value is also present in the manuscript in Section 3.2. We agree that calling $d$ as a hyper-parameter is misleading and have updated the manuscript by replacing $d$ with a constant value of $1$ in Eqn. $2$.  
As explained in the configuration subsection of section 5.2, ideally you would want to set the group size close to 1 while higher values produce faster rankings at the cost of more coarse-grained results. This is a hyper-parameter that needs to be set according to the user’s time and hardware budget. We have updated the manuscript with an ablation study to observe the change in accuracy of pruned networks with increasing group size as well as the time required for pruning under varying group sizes. Specifically, we look at the $98$% pruned version of the VGG16 and ResNet34 networks for group sizes varying from $2-32$ and $4-32$, respectively. We perform this ablation study for only the maximum level of pruning as it is the worst-case scenario in terms of both accuracy and time. According to Fig. 2a and 3a in Section A.2 of the Appendix, PvR is generally robust to group size as the top-1 accuracy drop from group size of $2$ to $32$ is about $1$% and the top-5 accuracy drop from group size of $4$ to $32$ is about $2$%. At the same time, with a group size of $32$, PvR is able to prune VGG16 and ResNet34 under $20$ seconds and $12$ minutes, respectively, as demonstrated in Fig. 2b and 3b.

Reviewer comment (Novelty):
The authors claim the proposed algorithms consider the paramete budget. While as I know, all existing pruning methods can achieve this because it is a basic requirement. Take L1 norm based pruing as an example, you can consider the parameter budget by adjusting the pruning threshold (This process is very fast with ignorable cost).

Author response:
We respectfully disagree with the reviewer. A number of algorithms such as L1 norm, as you rightly pointed out, have a parameter that needs to be tuned by performing multiple forward passes in order to achieve the precise parameter budget before beginning the pruning process. For techniques like L1 norm this process is fast while for methods such as HRank which finds the rank of the activations, it is slow. Other algorithms such as LayerDrop are unable to achieve all possible parameter budgets while methods like OTO$^{[1]}$ provide no such parameter. Our algorithm, on the other hand, explicitly considers a given user-specified parameter budget to prune the larger network.

Reviewer comment (Novelty):
Limited creativity. GC based initialization method is proposed by another paper, this paper only uses it.

Author response:
We have not claimed that GC is a measure proposed by us. The paper titled "Rethinking the value of network pruning"$^{[2]}$ empirically demonstrated that re-initializing a structurally pruned network and training from scratch is a better heuristic than fine-tuning the preserved weights. We used GC, a recently proposed model complexity measure, to shed light on this. Based on the fact that lower GC is a proxy for better performance in deep learning, we made the informed decision to re-initialize and train from scratch instead of simply relying on the performance of a model.

Reviewer comment (Reproducibility):
I am not sure but the authors may not describe how to compute group importance according to neuron/channel/.. importance.

Author response:
The process of computing the importance of a single neuron is explained in Section 3.1. The process of computing groups is explained in Section 3.2. Then, the first line in Section 3.3 explains how the ranking of each group is computed. Just like we mask a single neuron to compute its importance, in the same way, we mask all neurons in a group (at the same time) and then compute the importance of that group. There is no "averaging" involved.

Reviewer comment (Reproducibility):
Lack of some details. As the paper says, the proposed method achieves lower latency because of layer pruning. Could you please describe in which case the proposed method will prune an entire layer?

Author response:
The first few lines in Section 3.3 explain how the groups are removed based on the ranking, i.e., once the groups are formed and ranked, they are globally sorted and the least important groups are removed. In the last paragraph, of the same section we then explain that removing groups after a global sort can lead to the removal of all neurons/filters from a layer rendering that layer “collapsed” since we do not set any minimum layer pruning threshold. The removal of a layer entirely depends on the pruning budget, the model, and the dataset. Our algorithm PvR automatically decides and removes specific layers as and when required.

Reviewer comment (Novelty):
Lack of reference papers such as [1] (correlation based pruning) and [2] (reconstruction error based pruning).  
[1] COP: Customized Deep Model Compression via Regularized Correlation-Based Filter-Level Pruning. IJCAI 2019  
[2] ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Author response:
We would like to bring to the reviewer's notice that the paper ThiNet has already been cited in the manuscript. We already have 20+ citations in our work but we can add three more if the reviewer feels that citing these papers can improve the quality of our manuscript.

Reviewer comment (Experiments):
The baseline models compared in the experiment are not quite state-of-the-art. More recent models need to be covered. Some of the existing methods are relevant to this work, such as Kuang J, Shao M, Wang R, et al. Network pruning via probing the importance of filters[J]. International Journal of Machine Learning and Cybernetics, 2022, 13(9): 2403-2414.

Author response:
We compare our method against $11$ different pruning techniques from 2016 to 2022 with Random Channel Pruning (RCP) being the latest (Li, Yawei, Kamil Adamczewski, Wen Li, Shuhang Gu, Radu Timofte, and Luc Van Gool. "Revisiting random channel pruning for neural network compression." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 191-201. 2022.). We believe that RCP is a stronger and better-known baseline in comparison to the one provided by the reviewer.