Paper: LeanFlex-GKP: Advancing Hassle-Free Structured Pruning with Simple Flexible Group Count (ICLR-2024)

Reviewer comment (Novelty):
The contribution of this paper is limited. This paper seems an incremental work of TMI-GKP, and the main difference between this work and TMI-GKP is the group count evaluation.

Author response:
### **[W1 - Incremental to TMI-GKP with only different being group count evaluation]: Our method's technical procedure differs from TMI-GKP in every possible stage, and can be applied in scenarios where TMI-GKP can't.**

We respectively disagree with this assessment, though we understand this potential "misunderstanding" might be due to the lack of community recognition of GKP [1] for being a recently proposed pruning granularity awaiting future improvements. We disagree with this assessment as LeanFlex-GKP has a vastly different technical procedure to TMI-GKP; yet, it can be applied in scenarios where TMI-GKP can't (or is not ideal) with many handy characteristics and advantages.

From a technical perspective — as illustrated in [Figure 2](https://openreview.net/pdf?id=YhNXGWVH1N#page=3) of our paper — every Grouped Kernel Pruning (GKP) method has the following stages: *1) Filter grouping, 2) Grouped kernel pruning, and 3) Reconstruction of the pruned model*. **LeanFlex-GKP utilizes a totally different filter grouping and grouped kernel pruning approaches than TMI-GKP.** It considers aspects TMI-GKP does not consider (group count) and proposes an integral optimization to make the connection between stages that TMI-GKP solves individually. **Such designs lead to many performance and efficiency gains**, as demonstrated in [Table 3, 4](https://openreview.net/pdf?id=YhNXGWVH1N#page=17), and [Table 7](https://openreview.net/pdf?id=YhNXGWVH1N#page=18). **Other reviewers also tend to recognize and appreciate the technical novelty of our proposed method** ([R `SmAM`](https://openreview.net/forum?id=YhNXGWVH1N&noteId=uyaRnx6ga1): *"the authors ... exploits GKP with self-designed clustering and pruning methods"*; [R `HKS2`](https://openreview.net/forum?id=YhNXGWVH1N&noteId=ymbBiH0Ih6): The *"idea of making Conv2d flexible under an integral optimization is interesting."*)

The only similarity between the two methods is they both prune grouped kernels, and their pruned model is reconstructed to a grouped convolution format — which are common characteristics shared by almost all proposed methods following the GKP granularity. **We respectfully argue that improving upon a proposed pruning granularity/framework is not incremental in nature.** There are thousands of methods published under the filter pruning granularity/framework [2, 3], bringing impact to the framework and providing massive contributions to the efficiency community. We argue that GKP, being a promising but newly proposed granularity/framework with only a few established methods under its wing, could also use some healthy advancements, as we provided here. 

On the practical application aspect, unlike TMI-GKP, LeanFlex-GKP does not require access to model training checkpoints, **making it applicable when such access is restricted (e.g., pruning a pretrained model, which is a pretty common pipeline)**. Also, because we get to "lean down" on the grouping and pruning stages (and especially to avoid the expensive dimensionality reduction operations like k-PCA, as analyzed in [Section 2.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=4)), the efficiency advantage makes LeanFlex-GKP applicable to wide models under a practical context. E.g., with 64-core of EPYC 7742 and an 80G A100 allocated, it takes more than 50 hours to prune just half of the ResNet101 w.r.t. TMI-GKP. In contrast, we can prune full ResNet101 w.r.t. LeanFlex-GKP in just under 2 hours. 

**The reviewer has already recognized the adaptability/usability advantage of our method in [*Strength 3*](https://openreview.net/forum?id=YhNXGWVH1N&noteId=jaYFnjLNfn); we hope the above comparative discussion can put such advantage into the context concerning TMI-GKP.**


We emphasize that our work adopts different approaches to almost all stages of GKP to TMI-GKP (though still towards the same goals to leverage the GKP granularity) with significant efficiency, adaptability, and usability improvement by solving a common pain point of exiting GKP implementations (dynamic operations). We consider this a worthy contribution and hope the reviewer would appreciate it too.

Reviewer comment (Reproducibility):
It would be better if the authors can provide the experimental settings, such as hyperparameters they used for different models.

Author response:
### **[W2 - Add experiment setting (e.g., hyperparameters) clarification]: Sure!**

We have now added such a table in [Appendix C.2](https://openreview.net/pdf?id=YhNXGWVH1N#page=17). Given that this information is mostly used for result replications, we will also open source the experiment setting files and model checkpoints should our manuscript be accepted.

Reviewer comment (Experiments):
From my understanding, LeanFlex-GKP also includes dynamic choices of clustering schemes in each of its convolutional layers as TMI-GKP. I would appreciate it if the author can give results of the performance of LeanFlex-GKP and TMI-GKP. I can only get limited information from Table 1 to Table 5.

Author response:
### **[W3.1 - LeanFlex-GKP also includes dynamic choices of clustering schemes in each of its convolutional layers as TMI-GKP]: Not really, its clustering operation is deterministic.**

TMI-GKP applies dynamic clustering schemes, as each of its convolutional layers will go through various combinations of dimensionality reduction and clustering techniques (then one clustering result is chosen according to its *tickets magnitude increase (TMI)* score; see [Section 2.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=4) of our paper or [Section 3.2.1 & 3.2.2](https://openreview.net/pdf?id=LdEhiMG9WLO#page=4) of TMI-GKP [1]). In LeanFlex-GKP, we apply the same *KPP-aware filter grouping procedure* illustrated in [Figure 5](https://openreview.net/pdf?id=YhNXGWVH1N#page=7) for every conv layer, so there is no *"dynamic choice of clustering schemes."*

To provide a more faithful rebuttal, we suspect the reviewer perceives LeanFlex-GKP to be dynamic in terms of filter grouping because we'd consider different `Conv2d(groups)` settings, which obviously influence the clustering/grouping results. While we do consider such settings, `Conv2d(groups)` is just one (and the only method-related) hyperparameter to the same grouping operation, instead of multiple other unique filter grouping approaches, as in TMI-GKP. 

This should now be crystal clear with our above [*response to [W2]*](https://openreview.net/forum?id=YhNXGWVH1N&noteId=2EmNfWye86); we thank the reviewer for suggesting this clarification. We have further provided an early definition & example of dynamic operation in [Section 1](https://openreview.net/pdf?id=YhNXGWVH1N), as well as a pseudocode walkthrough of our method in [Appendix B.3](https://openreview.net/pdf?id=YhNXGWVH1N#page=16) to facilitate the digestion of our research motivation and proposed procedure.

### **[W3.2 - Give more performance comparision between LeanFlex-GKP and TMI-GKP]: We have already compared them under 8 (now 11) different setups.**

There were more comparisons done on the two methods in other tables placed in [Appendix D](https://openreview.net/pdf?id=YhNXGWVH1N#page=19). Here, we provide a gap table for your viewing convenience; **we also added three new results** since our initial submission to make a more comprehensive comparison:


> $\Delta$Acc gap between TMI-GKP and LeanFlex-GKP in %, `+` indicates LeanFlex-GKP is better.
> 
| Model-Dataset            | $\Delta$Acc Gap |  Source | 
| - | :-: | :-: |
| ResNet50-ImageNet        | +0.11                                      | [Table 9](https://openreview.net/pdf?id=YhNXGWVH1N#page=20) |
| ResNet56-TinyImageNet    | +3.65                                      | [Table 10](https://openreview.net/pdf?id=YhNXGWVH1N#page=20) |
| ResNet101-TinyImageNet   | +1.57                                      | Table 10 |
| ResNet20-CIFAR10         | +0.31                                      |  [Table 16](https://openreview.net/pdf?id=YhNXGWVH1N#page=23) |
| ResNet32-CIFAR10         | +0.02                                      | Table 16 |
| ResNet56-CIFAR10         | +0.05                                      | Table 16 |
| ResNet110-CIFAR10        | +0.02                                      | Table 16 |
| VGG16-CIFAR10            | +0.11                                      |  [Table 12](https://openreview.net/pdf?id=YhNXGWVH1N#page=21) |
| ResNet56-CIFAR100 (new)  | +0.32                                      |  [Table 14](https://openreview.net/pdf?id=YhNXGWVH1N#page=21) |
| ResNet110-CIFAR100 (new)  | +0.63                                      | Table 14 |
| DenseNet40-CIFAR10 (new) | +0.35                                      | [Table 13](https://openreview.net/pdf?id=YhNXGWVH1N#page=21) |

**It is clear that LeanFlex-GKP has a performance advantage to TMI-GKP across 11 different experiment settings.** We'd say the performance between the two methods is indeed close under some particular setups (e.g., ResNet32/56/110-CIFAR10). But, as we noted in [*Section 5 - Discussion and Conclusion*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8), these BasicBlock CifarResNets experiments are getting saturated, as methods with significant performance gaps on more difficult model-dataset combinations tend to show little difference upon them. Our gap table above also supports this observation with much more meaningful gaps observed under harder settings. 

At the risk of being redundant (as the reviewer already recognizes it in [*Strength 3*](https://openreview.net/forum?id=YhNXGWVH1N&noteId=jaYFnjLNfn)), we again emphasize that performance improvement is just one aspect of our contribution. LeanFlex-GKP has significant efficiency, adaptability, and usability advantages over TMI-GKP, as detailed in our [*[W1] response*](https://openreview.net/forum?id=YhNXGWVH1N&noteId=2EmNfWye86) above; yet, our dynamic group count design may inspire all future GKP implementations.

Reviewer comment (Experiments):
Could the benefits of dynamic group counts extend to other structured pruning granularities like filter or channel pruning?

Author response:
### **[Q1 - Can dynamic group counts benefits filter/channel pruning?]: Depending on how you want them to work together.**

Since group count `Conv2D(groups)` is a hyperparameter specific to the grouped convolution format, it is incompatible with the standard filter/channel pruning method where the pruned model is not in a grouped convolution format. However, we can do it the other way around, e.g., we may first prune filters to obtain a smaller layer tensor, then apply GKP (with or without dynamic group counts) to the reduced layer tensor. 

While this will require a much different pipeline beyond the scope of our paper, we are confident that dynamic group count will provide benefits, as a filter-pruned model is similar to a smaller (unpruned baseline) model, and our dynamic group count design has showcased improvements across different setups, as demonstrated in [Appendix C](https://openreview.net/pdf?id=YhNXGWVH1N#page=16).

Reviewer comment (Evaluation):
The evaluation lacks comprehensiveness.

Author response:
### **[W1 & Q1 - Evaluation lacks comprehensiveness]: We have already compared with 16 popular structured pruning methods. But sure, we can add 16 more.**


We kindly direct the reviewer's attention to [Appendix D.2](https://openreview.net/pdf?id=YhNXGWVH1N#page=19), where we have compared 16 structured pruning methods by the time of initial submission (listed as the first part of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). To our understanding, this is already a pretty comprehensive coverage that is probably more extensive than most (if not all) pruning arts we compared and suggested by Reviewer `HKS2`.

**As the reviewer is interested, we now add comparisons to 16 *more* methods.** 10 of them are from the reviewer's suggestions, yet 6 additional methods are also included during the expansion exploration (respectively listed as the second and the third portion of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). We also kindly note that some of the suggested methods by the reviewer are incomparable due to reasons like being under a different paradigm (e.g., LAMP [4], for being unstructured) and not having comparable experiment setups (e.g., EMOFP [12], which is MNIST-focused); yet, methods like 3D [7] and OTOv2 [11] are already compared in our initial submission. **We also added MobileNetV2 results in [Table 10](https://openreview.net/pdf?id=YhNXGWVH1N#page=20)**, as requested by the reviewer.

**Now, we are comparing LeanFlex-GKP against 32 methods and evaluating it under 20 different settings.** To the best of our knowledge — and with many of our experiments running upon an identical baseline under a controlled pipeline — we believe our evaluation is considered the most comprehensive in the structured pruning field. 

We guess the reviewer was left with the impression that our evaluation lacks comprehensiveness because of the limited results posted in the main text of our initial submission. We now provide an abbreviated table in the main text ([Table 1](https://openreview.net/pdf?id=YhNXGWVH1N#page=9)), as well as the above-mentioned Method Overview table ([Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)) to illustrate all compared methods. As the reviewer suggested, we also expanded the discussion and provided more insight in [*Section 5 - Discussion and Conclusion*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) regarding the results posted in [Section 4.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=8). We hope the reviewer may find the new additions helpful.

Reviewer comment (Experiments):
Baselines are restricted. A lot of related works about iterative structured pruning [1-10] and one-shot pruning [11-15] are not compared.

Author response:
### **[W1 & Q1 - Evaluation lacks comprehensiveness]: We have already compared with 16 popular structured pruning methods. But sure, we can add 16 more.**


We kindly direct the reviewer's attention to [Appendix D.2](https://openreview.net/pdf?id=YhNXGWVH1N#page=19), where we have compared 16 structured pruning methods by the time of initial submission (listed as the first part of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). To our understanding, this is already a pretty comprehensive coverage that is probably more extensive than most (if not all) pruning arts we compared and suggested by Reviewer `HKS2`.

**As the reviewer is interested, we now add comparisons to 16 *more* methods.** 10 of them are from the reviewer's suggestions, yet 6 additional methods are also included during the expansion exploration (respectively listed as the second and the third portion of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). We also kindly note that some of the suggested methods by the reviewer are incomparable due to reasons like being under a different paradigm (e.g., LAMP [4], for being unstructured) and not having comparable experiment setups (e.g., EMOFP [12], which is MNIST-focused); yet, methods like 3D [7] and OTOv2 [11] are already compared in our initial submission. **We also added MobileNetV2 results in [Table 10](https://openreview.net/pdf?id=YhNXGWVH1N#page=20)**, as requested by the reviewer.

**Now, we are comparing LeanFlex-GKP against 32 methods and evaluating it under 20 different settings.** To the best of our knowledge — and with many of our experiments running upon an identical baseline under a controlled pipeline — we believe our evaluation is considered the most comprehensive in the structured pruning field. 

We guess the reviewer was left with the impression that our evaluation lacks comprehensiveness because of the limited results posted in the main text of our initial submission. We now provide an abbreviated table in the main text ([Table 1](https://openreview.net/pdf?id=YhNXGWVH1N#page=9)), as well as the above-mentioned Method Overview table ([Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)) to illustrate all compared methods. As the reviewer suggested, we also expanded the discussion and provided more insight in [*Section 5 - Discussion and Conclusion*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) regarding the results posted in [Section 4.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=8). We hope the reviewer may find the new additions helpful.

Reviewer comment (Evaluation):
In Section 4, it will be better if some insights can be provided, not only listing numbers.

Author response:
As the reviewer suggested, we also expanded the discussion and provided more insight in [*Section 5 - Discussion and Conclusion*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) regarding the results posted in [Section 4.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=8). We hope the reviewer may find the new additions helpful.

Reviewer comment (Experiments):
The Ablation Study is missing.

Author response:
### **[W2 & Q2 - Ablation study is missing]: It is in Appendix C. We now expand its coverage with better description.**

We kindly direct the reviewer's attention to [Appendix C](https://openreview.net/pdf?id=YhNXGWVH1N#page=16), where we provided a full-scale ablation study at the time of initial submission. For better readability, we have expanded the ablation study coverage, added more descriptions, and placed an anchor in bold text in [*Section 4 - Experiments*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) to help navigate among related sections.

Reviewer comment (Presentation):
The structure can be improved. The Introduction occupies too much space. The first 3.5 pages are all about the Introduction.

Author response:
### **[W3 - Paper structured can be improved with shorter Intro]: Sure, we changed it up a little!**

We agree that our paper has a (possibly overly) prolonged intro & motivation section ([Section 1 and 2](https://openreview.net/pdf?id=YhNXGWVH1N)). However, given our method is built upon specific observations of the recently proposed GKP granularity [16] with only limited exposure, we feel like we owe it to our readers to provide a clear and unified introduction of the GKP framework and its adaptations without asking them to jump among different papers. Should we remove too much of it, our readers might get confused about GKP and none of our contributions matter. **All reviewers, including you, happen to appreciate the clarity of our background introduction**; so we don't want to drastically modify these sections, if possible.

That being said, we have cut off some repeated material in [Section 1 and 2](https://openreview.net/pdf?id=YhNXGWVH1N), added more connections between our methodology and contribution claims in [Section 3](https://openreview.net/pdf?id=YhNXGWVH1N#page=6) (suggested by [Reviewer `SmAm`](https://openreview.net/forum?id=YhNXGWVH1N&noteId=uyaRnx6ga1)), and added more result discussions in [Section 5](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) (suggested by you).

Reviewer comment (Novelty):
The discussion about the related works is insufficient, such as iterative structured pruning [1-10], one-shot pruning [11-14], Grouped Kernel Pruning [15], and automatic pruning (with little-to-no hyper-parameter tuning) [2, 3].

Author response:
### **[W4 - Discussion of related work is insufficient]: Agreed, we have now added them.**

As noted in our response to [*[W1 & Q1]*](https://openreview.net/forum?id=YhNXGWVH1N&noteId=mrbcwq5SxY), we have already provided comprehensive coverage of different pruning methods, but we haven't properly discussed them. We have referenced and added such proper discussion in [Appendix A](https://openreview.net/pdf?id=YhNXGWVH1N#page=14) for all compared methods, as well as some landmark work and important aspects of structured pruning.

We were probably too focused on getting the GKP intro right (and from the feedback, we did) and, therefore, lost the big picture a little. We thank the reviewer for pointing this out and directing us to provide a more zoom-out view of the structured pruning field to our readers.

---
(here, we follow the citation index initiated by the reviewer for better consistency)     
[4] Lee et al., Layer-adaptive sparsity for the Magnitude-based Pruning. ICLR 2021    
[7] Wang et al., Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework. ICML 2021    
[11] Chen et al., OTOv2: Automatic, Generic, User-Friendly. ICLR 2023 — this is `v2` of the reviewer suggested method [11].    
[12] Wu et al., Evolutionary Multi-Objective One-Shot Filter Pruning for Designing Lightweight Convolutional Neural Network. Sensors 2021     
[16] Zhong et al., Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions. ICLR 2022

Reviewer comment (Experiments):
Can authors compare the proposed method with more related pruning works [1-14]?

Author response:
### **[W1 & Q1 - Evaluation lacks comprehensiveness]: We have already compared with 16 popular structured pruning methods. But sure, we can add 16 more.**


We kindly direct the reviewer's attention to [Appendix D.2](https://openreview.net/pdf?id=YhNXGWVH1N#page=19), where we have compared 16 structured pruning methods by the time of initial submission (listed as the first part of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). To our understanding, this is already a pretty comprehensive coverage that is probably more extensive than most (if not all) pruning arts we compared and suggested by Reviewer `HKS2`.

**As the reviewer is interested, we now add comparisons to 16 *more* methods.** 10 of them are from the reviewer's suggestions, yet 6 additional methods are also included during the expansion exploration (respectively listed as the second and the third portion of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). We also kindly note that some of the suggested methods by the reviewer are incomparable due to reasons like being under a different paradigm (e.g., LAMP [4], for being unstructured) and not having comparable experiment setups (e.g., EMOFP [12], which is MNIST-focused); yet, methods like 3D [7] and OTOv2 [11] are already compared in our initial submission. **We also added MobileNetV2 results in [Table 10](https://openreview.net/pdf?id=YhNXGWVH1N#page=20)**, as requested by the reviewer.

**Now, we are comparing LeanFlex-GKP against 32 methods and evaluating it under 20 different settings.** To the best of our knowledge — and with many of our experiments running upon an identical baseline under a controlled pipeline — we believe our evaluation is considered the most comprehensive in the structured pruning field. 

We guess the reviewer was left with the impression that our evaluation lacks comprehensiveness because of the limited results posted in the main text of our initial submission. We now provide an abbreviated table in the main text ([Table 1](https://openreview.net/pdf?id=YhNXGWVH1N#page=9)), as well as the above-mentioned Method Overview table ([Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)) to illustrate all compared methods. As the reviewer suggested, we also expanded the discussion and provided more insight in [*Section 5 - Discussion and Conclusion*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) regarding the results posted in [Section 4.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=8). We hope the reviewer may find the new additions helpful.

Reviewer comment (Experiments):
Can some ablation study be provided?

Author response:
### **[W2 & Q2 - Ablation study is missing]: It is in Appendix C. We now expand its coverage with better description.**

We kindly direct the reviewer's attention to [Appendix C](https://openreview.net/pdf?id=YhNXGWVH1N#page=16), where we provided a full-scale ablation study at the time of initial submission. For better readability, we have expanded the ablation study coverage, added more descriptions, and placed an anchor in bold text in [*Section 4 - Experiments*](https://openreview.net/pdf?id=YhNXGWVH1N#page=8) to help navigate among related sections.

Reviewer comment (Presentation):
The structure of the paper can be improved. Sections 1 and 2 are too detailed that Section 3 corresponds to only a small part, which is not enough to explain the major contribution.

Author response:
### **[W1 - Paper structured can be improved (Sec 1&2 are too long and Sec 3 should be expanded)]: Sure, we changed it up a little.**

We agree that our paper has a (possibly overly) prolonged intro & motivation section ([Section 1 and 2](https://openreview.net/pdf?id=YhNXGWVH1N)). However, given our method is built upon specific observations of the recently proposed GKP granularity [1] with only limited exposure, we feel like we owe it to our readers to provide a clear and unified introduction of the GKP framework and its adaptations without asking them to jump among different papers. Should we remove too much of it, our readers might get confused about GKP and none of our contributions matter. **All reviewers, including you, happen to appreciate the clarity of our background introduction**; so we don't want to drastically modify these sections, if possible.

That said, we agree with the reviewer that *[Section 3 - Proposed Method](https://openreview.net/pdf?id=YhNXGWVH1N#page=6)* should be expanded to better connect our claimed contribution and actual methodology. **We added such connections in [Section 3](https://openreview.net/pdf?id=YhNXGWVH1N#page=6) and cut off some repetitive material in [Section 1 and 2](https://openreview.net/pdf?id=YhNXGWVH1N)**; we thank the reviewer for suggesting this.

Reviewer comment (Experiments):
5 baseline approaches are compared while some lossless approaches or other baselines can be added as baseline approaches.

Author response:
### **[W2.1 - More than 5 baselines should be compared]: We have in fact compared 16 (now 32) pruning methods. We now provide such results with better presentation for improved readability.**


We kindly direct the reviewer's attention to [Appendix D.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=19), where we have compared 16 structured pruning methods by the time of submission (first part of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). **Now**, after including the recommended work by [reviewer `HKS2`](https://openreview.net/forum?id=YhNXGWVH1N&noteId=ymbBiH0Ih6), **we are comparing LeanFlex-GKP against 32 pruning methods** (see [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). To the best of our knowledge — and with many of our experiments conducted with the identical baseline under a controlled pipeline — we believe our evaluation is considered the most comprehensive in the structured pruning field.


We guess the reviewer was left with the impression that we only compared with 5 baselines because of the limited results posted in the main text of our initial submission. We now provide an abbreviated table in the main text ([Table 1](https://openreview.net/pdf?id=YhNXGWVH1N#page=9)), as well as a Method Overview table ([Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)) to illustrate all compared methods. We hope the reviewer may find them helpful.

Reviewer comment (Theory):
It is not clear whether the proposed method can achieve lossless pruning. Some theoretical analysis may be beneficial to the paper.

Author response:
### **[W2.2 & W3.1 & Q2.1 & Q3 - Is the proposed method "lossless"?]: It depends on your definition of "lossless." Good accuracy retention? — Yes; Exact output? — No.**

Almost all pruning methods will alter the output of the original model (unless under some very special and often trivial setups). Thus, if by "lossless," the reviewer means "exact," like FlashAttention [2], then our method is not "lossless."

This conclusion is rather obvious, given we reported $\Delta$Acc in all our experiments, indicating the difference between the unpruned and prune models. To make the rebuttal more faithful, we suspect the reviewer meant to ask if our method has good accuracy retention (maybe even an improvement after prune). **The answer to this question is model-dataset-setting dependent, but we'd say our method can generally provide good accuracy retention under a reasonable setup**. This is evidenced by 17 out of 20 reported results of LeanFlex-GKP showing improvements after pruning, where no other method can provide a $\uparrow$ $\Delta$Acc under the three exception setups.

Reviewer comment (Writing):
Many grammar errors, e.g., an higher ***, they can be run, with in, is determine by, with lowest etc.

Author response:
### **[W4 - Typos]: Fixed, and thank you for the close read!**

Reviewer comment (Writing):
Dependable experience may be independable experience.

Author response:
### **[W5 & Q1 - Should "dependable experience" be "independable experience"?]: Will change to "predictable" for disambiguation.]**

At the end of [Section 2](https://openreview.net/pdf?id=YhNXGWVH1N#page=6), we note that LeanFlex-GKP's pruned size/compute is directly estimable by multiplying the pruning rate by the readings of the original model, thus *"making the whole pruning procedure a standardized and [dependable] experience."* Here, we use the term "dependable" to indicate our method is reliable and predictable in terms of pruned model size/compute estimation (e.g., in contrast to methods like GAL [6], where one needs to trial-and-error different hyperparameter combinations to reach the desired reduction). So, we indeed meant "dependable." 

However, we realize this might cause confusion, as "dependable" can imply hard coupling, which is often not a favorable character. Thus, we will change the term to "predictable" for disambiguation. We thank the reviewer for catching this small but vital detail.

Reviewer comment (Writing):
I wonder if "dependable experience" should be independable experience.

Author response:
### **[W5 & Q1 - Should "dependable experience" be "independable experience"?]: Will change to "predictable" for disambiguation.]**

At the end of [Section 2](https://openreview.net/pdf?id=YhNXGWVH1N#page=6), we note that LeanFlex-GKP's pruned size/compute is directly estimable by multiplying the pruning rate by the readings of the original model, thus *"making the whole pruning procedure a standardized and [dependable] experience."* Here, we use the term "dependable" to indicate our method is reliable and predictable in terms of pruned model size/compute estimation (e.g., in contrast to methods like GAL [6], where one needs to trial-and-error different hyperparameter combinations to reach the desired reduction). So, we indeed meant "dependable." 

However, we realize this might cause confusion, as "dependable" can imply hard coupling, which is often not a favorable character. Thus, we will change the term to "predictable" for disambiguation. We thank the reviewer for catching this small but vital detail.

Reviewer comment (Theory):
I wonder if the proposed approach is lossless. In addition, I wonder if the proposed approach can be applied to other structures.

Author response:
### **[W2.2 & W3.1 & Q2.1 & Q3 - Is the proposed method "lossless"?]: It depends on your definition of "lossless." Good accuracy retention? — Yes; Exact output? — No.**

Almost all pruning methods will alter the output of the original model (unless under some very special and often trivial setups). Thus, if by "lossless," the reviewer means "exact," like FlashAttention [2], then our method is not "lossless."

This conclusion is rather obvious, given we reported $\Delta$Acc in all our experiments, indicating the difference between the unpruned and prune models. To make the rebuttal more faithful, we suspect the reviewer meant to ask if our method has good accuracy retention (maybe even an improvement after prune). **The answer to this question is model-dataset-setting dependent, but we'd say our method can generally provide good accuracy retention under a reasonable setup**. This is evidenced by 17 out of 20 reported results of LeanFlex-GKP showing improvements after pruning, where no other method can provide a $\uparrow$ $\Delta$Acc under the three exception setups.

### **[Q2.2 - Can the proposed approach applied to other structures?]: Again, it depends on your definition of "structure." With filter/channel pruning? — Yes; Outsides CNNs? — No.**

Suppose the reviewer is referring to "structures" as model architectures; given that the convolutional kernel is a concept specific to CNN-like architectures, our method cannot apply to other architectures (e.g., transformers), and neither do most filter/channel pruning methods. However, if the reviewer refers to "structures" as pruning structures, then yes: LeanFlex-GKP can work in conjunction with a filter/channel pruning method — e.g., first remove some filters deemed unimportant, then apply GKP to the pruned model.

Reviewer comment (Experiments):
I wonder if the authors can compare the proposed approach with lossless pruning methods.

Author response:
### **[W2.1 - More than 5 baselines should be compared]: We have in fact compared 16 (now 32) pruning methods. We now provide such results with better presentation for improved readability.**


We kindly direct the reviewer's attention to [Appendix D.1](https://openreview.net/pdf?id=YhNXGWVH1N#page=19), where we have compared 16 structured pruning methods by the time of submission (first part of [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). **Now**, after including the recommended work by [reviewer `HKS2`](https://openreview.net/forum?id=YhNXGWVH1N&noteId=ymbBiH0Ih6), **we are comparing LeanFlex-GKP against 32 pruning methods** (see [Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)). To the best of our knowledge — and with many of our experiments conducted with the identical baseline under a controlled pipeline — we believe our evaluation is considered the most comprehensive in the structured pruning field.


We guess the reviewer was left with the impression that we only compared with 5 baselines because of the limited results posted in the main text of our initial submission. We now provide an abbreviated table in the main text ([Table 1](https://openreview.net/pdf?id=YhNXGWVH1N#page=9)), as well as a Method Overview table ([Table 8](https://openreview.net/pdf?id=YhNXGWVH1N#page=19)) to illustrate all compared methods. We hope the reviewer may find them helpful.

### **[W2.2 & W3.1 & Q2.1 & Q3 - Is the proposed method "lossless"?]: It depends on your definition of "lossless." Good accuracy retention? — Yes; Exact output? — No.**

Almost all pruning methods will alter the output of the original model (unless under some very special and often trivial setups). Thus, if by "lossless," the reviewer means "exact," like FlashAttention [2], then our method is not "lossless."

This conclusion is rather obvious, given we reported $\Delta$Acc in all our experiments, indicating the difference between the unpruned and prune models. To make the rebuttal more faithful, we suspect the reviewer meant to ask if our method has good accuracy retention (maybe even an improvement after prune). **The answer to this question is model-dataset-setting dependent, but we'd say our method can generally provide good accuracy retention under a reasonable setup**. This is evidenced by 17 out of 20 reported results of LeanFlex-GKP showing improvements after pruning, where no other method can provide a $\uparrow$ $\Delta$Acc under the three exception setups.