Paper: A Fast Framework for Post-training Structured Pruning Without Retraining (ICLR-2024)

Reviewer comment (Experiments):
In section 3.2, the authors claim that the direct application of layer-wise reconstruction to minimize the discrepancy between original and pruned model to structured pruning does not produces satisfying results. However, there are no experiments or citations to justify this claim - weakening the justification of the proposed methodology.

Author response:
* We extended Adaprune and gAP to directly apply layer-wise reconstruction on structured pruning, and experiments on ImageNet have shown that these methods cannot effectively alleviate the accuracy decrease caused by pruning (See Table 2). We will explicitly reference the results in the modified version.

Reviewer comment (Evaluation):
The authors do not report the FLOPs and wall-clock time required by the post training structured pruning framework. Also, a comparision of compute with finetuning methods is missing since this is a motivation provided behind the work.

Author response:
* We previously thought that the report on actual running time was not very important, so it was included in appendix A.1. We will consider presenting this result in the main text in the revised version. In addition, we have reported the reduction in FLOPs, so no specific FLOPs were displayed.

Reviewer comment (Experiments):
The retraining based pruning baselines selected are weak. Also, information is not provided as to how are those pruned models obtained. For example, it is not clear how the L1-norm pruned model has been generated in Table 2. Moreover, [a], [b], [c] show much superior performance with respect to the L1-norm pruned baseline chosen upon retraining.

Author response:
* Our paper focuses on pruning without retraining, so we only compare the $L_1$-norm based method to show the gap with retraining-based methods. We reproduced the pruning algorithm in [1] as $L_1$-norm pruned baseline and we retrained 100 epochs after pruning. 
    * We have added the results of [2] and [3] in the revision. By the way, DFPC ([4]) is proposed as a data-free method retrained, it does not generalize to higher sparsity. Moreover, it retrained the model when reporting results on ImageNet. This method does not match our settings and therefore was not selected for the baseline.

Reviewer comment (Experiments):
How were the callibration samples selected for plots in Fugure 4? How sensitive are these plots do different set of callibration samples? Would the accuracy of the models change if the callibration samples change?

Author response:
* We randomly sample indexes to obtain random samples. And the selection of samples only causes minor changes in the results. The results of ten calibrations after pruning ResNet-50 on ImageNet are \{72.51, 72.57, 72.64, 72.56, 72.65, 72.70, 72.58, 72.60, 72.56, 72.54\}

Reviewer comment (Reproducibility):
In Algorithm 1, how are the layer-wise pruning ratios selected? (Please note the typo in the last weight update of Algorithm 1 (third last line). Do you intend to minimize the output layer's reconstruction loss here?)

Author response:
* We use global pruning and do not set hierarchical pruning rates. And the third last line in Algorithm 1 minimizes the reconstruction loss.

Reviewer comment (Theory):
Connected to equation (2), the authors mention "naive optimizing...induces imbalanced parameter updates...earlier layers being over-optimized". I'm not sure I understand what imbalanced updates and over-optimization actually mean. Empirically the authors attempt to support this statement using Figure 6 to illustrate that earlier layers are updated more aggressively than later layers so the proposed update attempts to smooth out these updates. When earlier layers are optimized too much does this negatively impact the reconstruction loss in equation (2) because the pruned output activations are not and should not be closely correlated with the unpruned activations?

Author response:
**Explanation about imbalanced updates and over-optimization.**  
    * The pruned model lacks the activation values of some channels, and we can only reconstruct the activation values of the remaining channels, which introduces error. We consider holistic layer reconstruction loss to reduce error accumulation. However, holistic reconstruction loss causes gradients of different layers to be unbalanced and causes the model to update incorrectly. 
    For example, the first layer is optimized by the reconstruction error of itself and all subsequent layers, but the last layer is optimized by the reconstruction error of itself only. So, we need to balance the updates of different layers.

Reviewer comment (Experiments):
PSP yields similar results to AdaPrune in terms of RP and RF factors in Table 2 but with much better accuracy. I think a natural question would be the effectiveness of the structural pruning compared to the application of the calibration updating steps. It's possible the increased accuracy is due to the post pruning calibration steps, do any of the other non-training methods perform any post pruning calibration steps?

Author response:
**Explanation about settings and results in Table2**  
    * The results of AdaPrune and gAP are produced by our extended version of structured pruning. We use the same framework and importance evaluation, prune the same structure and yield RP and RF factors. The difference lies in the calibration process, our method is more efficient in recovering the accuracy. The other two non-retraining methods are designed for structured pruning, so we do not need to modify. Neuron Merging merge parameters of neurons and does not require data for calibration. However, it yields the worst accuracy. Reborn Filters concentrates information on newly generated filters before pruning. This process requires calibration data. It can fine-tune or do nothing after pruning.

Reviewer comment (Evaluation):
In Figure 5 the results for ResNet18 on CIFAR100 seem to be particularly bad for "w/o REC" compared to the other graphs even for a small number of pruned parameters. Is there a reason for this sharp drop-off?

Author response:
**Explanation of the results for ResNet18 on CIFAR100 in Figure 5**  
    * CIFAR-100 is more difficult than CIFAR-10. ResNet-18 trained on CIFAR-100 contains less redundancy, and small amounts of pruning can result in significant accuracy reduction.

Reviewer comment (Writing):
The writing and presentation of the formulas during the main text could use more polishing before publication.

Author response:
Besides, we have revised unclear expressions in an updated version.

Reviewer comment (Experiments):
Motivation. I'm not sure about the point of using unlabeled data setting. Since the unlabeled data are still from the original dataset (which violates the data privacy regulations) and a small amount of labeled calibration input data is already a realistic setting.

Author response:
**Motivation of our paper (W2).**
    * **Privacy or commercial concerns:** While full business data is difficult to obtain, companies may agree to provide small samples. Moreover, our focus is on the amount of data, not whether it is labeled
    * **Computational cost of retraining:** The high cost of retraining is also one of the obstacles to the practical application of pruning technology. This is also the motivation for our retraining-free framework.

Reviewer comment (Experiments):
It's unclear why the group importance and two phases layer reconstruction strategy work so well in this paper. I suggest the author provide a more comprehensive experiment report to help us understand that.

Author response:
**Could authors provide a more comprehensive experiment report (W3)?**
    * Figure 3 provides intuition for why sparsity penalty works. It reduces the information loss caused by pruning. Moreover, Figure 5 provides ablation experiments to demonstrate the role of reconstruction and sparsity penalty.  Besides, Figure 6 shows that balanced updates are key to holistic reconstruction loss success. We think we have provided a more comprehensive experiment. We sincerely hope to receive your specific experimental suggestions. By the way, we provided more experiments in the appendix and code in the supplementary material, which may help you understand our approach.

Reviewer comment (Evaluation):
The performance is mediocre compared to the SOTA data-free algorithms, such as RED++.

Author response:
Our paper focuses on pruning without retraining, so the pruning methods with the same settings is the focus of our comparison. We have compared rich and representative baselines. As for red++ ([4]), since it does not provide code and we failed to reproduce their results, this method has not been selected as a baseline.  In addition, in order to show the gap with the retraining method, we selected the simple $L_1$-norm based baseline in Table 1 and Table 2, which we retrained 100 epochs after pruning.
    We have added the results of [1] and [2] in the revision.

Reviewer comment (Novelty):
Pruning coupled structures is proposed by DepGraph. The remaining contribution is not significant, especially considering the limited performance improvement.

Author response:
Our main contribution is the post-training structured pruning framework and two-stage reconstruction method. Compared with methods that do not require retraining, our method has significant improvements, especially on the large-scale dataset ImageNet. Other reviewers have appreciated our performance improvements.

Reviewer comment (Experiments):
Retraining-based techniques for performance comparison should be chosen among more recent ones, not the naïve L1-norm.

Author response:
Our paper focuses on pruning without retraining, so the pruning methods with the same settings is the focus of our comparison. We have compared rich and representative baselines. As for red++ ([4]), since it does not provide code and we failed to reproduce their results, this method has not been selected as a baseline.  In addition, in order to show the gap with the retraining method, we selected the simple $L_1$-norm based baseline in Table 1 and Table 2, which we retrained 100 epochs after pruning.
    We have added the results of [1] and [2] in the revision.

Reviewer comment (Experiments):
The experiments are limited in the diversity of datasets and models.

Author response:
**Dataset**: CIFAR-10/100 and ImageNet datasets are the common settings for pruning papers. 
    * **Dataset**: We use the same settings as other papers for retraining-free pruning. And we use VGG-19 rather than VGG-16 for consistent settings. Besides, we provided results for more models in Appendix A.3.

Reviewer comment (Novelty):
Claiming this study as "no retraining" seems exaggerated, especially compared to genuine data-free methods since this research employs some training data.

Author response:
No retraining means avoiding costly training processes after pruning rather than no data.  Our framework replaces tedious retraining with a simple and fast calibration process. Moreover, a small amount of calibration data is a realistic setting.

Reviewer comment (Experiments):
Is it possible to precisely control FLOPs and parameters of the pruned model?

Author response:
Our implementation using global pruning rate does not allow precise control of model parameters and FLOPs. We have used the same group-wise pruning wherever possible. We extended AdaPrune and gAP to use group-wise pruning. However, Neuron Merging and Reborn Filters are difficult to be compatible with group pruning, so we used the original method.

Reviewer comment (Theory):
In Section 3.2.1, what causes Equation 2 to over-optimize the earlier layers?

Author response:
Holistic reconstruction loss causes gradients of different layers to be unbalanced. For example, the first layer is optimized by the reconstruction error of itself and all subsequent layers, but the last layer is optimized by the reconstruction error of itself only. So, we need to balance the updates of different layers.

Reviewer comment (Experiments):
Why VGG-19 is used instead of VGG-16?

Author response:
**Dataset**: CIFAR-10/100 and ImageNet datasets are the common settings for pruning papers. 
    * **Dataset**: We use the same settings as other papers for retraining-free pruning. And we use VGG-19 rather than VGG-16 for consistent settings. Besides, we provided results for more models in Appendix A.3.