Paper: ($\texttt{PASS}$) Visual Prompt Locates Good Structure Sparisty through a Recurent HyperNetwork (ICLR-2024)

Reviewer comment (Experiments):
Prompt learning relies on a strong foundation of pre-trained models. To demonstrate its effectiveness in network pruning, favorable experiments and analyses are essential.

Author response:
**reply**: Thank you for your insightful feedback. We totally agree with the significance of a strong foundation of pre-trained models for prompt learning. Due to the time limitation, we could not extend to evaluating the effectiveness of VP on our proposed PASS using a very strong foundation of pre-trained vision models such as CLIP[1]. However, we provide a similar analysis to study the effect of model size on the pruning performance caused by visual prompts, by varying the pre-trained model size from ResNet-18 to ResNet-34, and to ResNet-50. The results are shown in **Table 2**.

- The results show an increase in the efficacy of VP as we progress from ResNet-18 to more advanced models like ResNet-34 and ResNet-50. This observation suggests that the impact of VP on the PASS framework might be more pronounced when implemented on stronger foundational vision models.

    **Table 2: Performance of subnetworks (Channel Sparsity: 30%) found by PASS w/ VP and w/o VP with varying different scales of the pre-trained models. The  task is CIFAR-100.**
    |                | ResNet-18 | ResNet-34 | ResNet-50 |
    |----------------|-----------|-----------|-----------|
    | LSTM+Weights   | 81.13     | 82.68     | 82.84     |
    | LSTM+Weights+VP(PASS) | 81.72 | 83.69 | 83.91 |
    | Δ              | 0.59      | 1.01      | 1.07      |

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

Reviewer comment (Experiments):
In cases where pruning a model without fine-tuning, the visual prompt is unnecessary, and in such a scenario, it seems that the paper may not work.

Author response:
**Reply:** Thank you for your comments. We would like to clarify that our proposed PASS still works in cases where pruning a model without fine-tuning. It is because our proposed PASS concurrently optimizes both the visual prompt and the hypernetwork for finding the channel topology.

- To substantiate our claim, we conducted additional experiments comparing the performance of subnetworks identified by PASS with and without the visual prompt (VP). Our results in **Table 3** demonstrate that subnetworks pruned using PASS with the visual prompt outperform those pruned without it, underscoring the visual prompt's value in structural pruning.

    **Table 3: Performance of subnetworks before fine-tuning. The experiments are based on CIFAR-100 and ResNet-18.**

    |Channel Sparsity | 10%   | 30%   | 50%   | 70%   |
    |---------------|-------|-------|-------|-------|
    | LSTM+Weights  | 33.22 | 32.80 | 21.19 | 8.42  |
    | LSTM+Weights+VP(PASS) | 35.64 | 33.79 | 22.38 | 9.15  |

Reviewer comment (Experiments):
The experiments conducted on small datasets, such as CIFAR and Tiny-ImageNet, with very low resolution and data scale are not entirely convincing. The reviewer suggests including experiments on at least ImageNet-1k or ImageNet. In the era of big data, ImageNet is considered a relatively small dataset.

Author response:
**Reply:** Thank you for your comments. In response, we expanded our empirical investigation to include the ImageNet-1k dataset. Following [5], we adopt the Vision Transformer (ViT-B/16) and ResNeXt-50 architectures. We utilized the pre-trained models from PyTorch Vision ([1]). 

- The results are shown in **Table 4**. The results show that our method PASS achieves superior performance over baselines with significant speed-ups with only minimal impacts on accuracy, affirming the effectiveness of our proposed PASS on large datasets.

    **Table 4: Pruning results  on ImageNet with Advanced models.**

    | Arch.      | Method       | Base  | Pruned | Δ Acc. | FLOPs |
    |------------|--------------|-------|--------|--------|-------|
    | ResNeXt-50 | ResNeXt-50   | 77.62 | -      | -      | 4.27  |
    |            | SSS [2]      | 77.57 | 74.98  | -2.59  | 2.43  |
    |            | GFP [3]      | 77.97 | 77.53  | -0.44  | 2.11  |
    |            | DepGraph [4] | 77.62 | 76.48  | -1.14  | 2.09  |
    |            | Ours (PASS)  | 77.62 | 77.21  | -0.41  | 2.01  |
    | ViT-B/16   | ViT-B/16     | 81.07 | -      | -      | 17.6  |
    |            | DepGraph [4] | 81.07 | 79.17  | -1.90  | 10.4  |
    |            | Ours (PASS)  | 81.07 | 79.77  | -1.30  | 10.7  |

[1] [TorchVision](https://pytorch.org/vision/stable/index.html)

[2] Huang, Zehao, and Naiyan Wang. "Data-driven sparse structure selection for deep neural networks." Proceedings of the European conference on computer vision (ECCV). 2018.

[3] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.

[4] Fang, Gongfan, et al. "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Theory):
Could the author explain why the visual prompts improve channel pruning? Since the visual prompts are static across a task or a dataset, why does the author state their pruning method as “from a data-centric perspective” while it is not even input dependent?

Author response:
**Reply:** Thank you for your insightful questions. We answer your questions step by step.

**Why visual prompts help channel pruning.** 
- Several studies demonstrated that model sparsification is influenced by the nature of tasks or datasets[1][2]. Visual prompts act as a form of meta-information, encapsulating crucial insights about the task or dataset. It provides the task-dependent information for the pruning process, leading to the identification of more relevant and efficient channel structures. The robust performance of our proposed PASS, demonstrated across six diverse datasets and multiple architectures, further substantiates this claim. Additionally, our ablation studies provide concrete evidence of the efficacy of visual prompts in enhancing channel pruning.

- Furthermore, the increasing volume of research indicates that incorporating information from inputs is crucial for enhancing the performance of pruning algorithms [3][4][5][6]. For example, the recent study [6] highlights that leveraging prompts post-pruning can significantly improve the efficiency of sparse LLMs. Inspired by these insights, we explored how to integrate visual prompts into the pruning process of the vision models and proposed our PASS, showing its superiority over other baselines.

[1] Liu, Shiwei, et al. "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!." ICLR (2023).

[2] Yin, Lu, et al. "Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity." arXiv preprint arXiv:2310.02277 (2023).

[3] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models." arXiv preprint arXiv:2306.11695 (2023).

[4] Frantar, Elias, and Dan Alistarh. "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot." ICML (2023).

[5] Yin, Lu, et al. "Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity." arXiv preprint arXiv:2310.05175 (2023).

[6] PROMPT, TRANSFERABLE. "Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt." arXiv preprint arXiv:2305.11186 (2023).

**Our proposed PASS is from a data-centric perspective.** 

- Firstly, let us elucidate the concept of a data-centric technique. A data-centric technique **does not necessarily hinge on being input sample-dependent**. Instead, it can also be task-specific or dataset-dependent[1]. This kind of technique primarily focuses on engineering and managing data to boost model performance. A notable manifestation of this in the data-centric paradigm is the utilization of prompts, including visual prompts, which are specific to a task or dataset[1]. In our proposed PASS, the reliance on visual prompts categorically aligns it with a data-centric methodology, as these prompts are dataset-specific. Therefore, our method's designation as data-centric is aptly justified.

[1] Zha, Daochen, et al. "Data-centric artificial intelligence: A survey." arXiv preprint arXiv:2303.10158 (2023).

Reviewer comment (Evaluation):
Model Complexity: While the channel pruning reduces size, the added LSTM network introduces new parameters. An analysis of its impact on model parameters considering the LSTM and training/testing time would be beneficial.

Author response:
**Q1:** *Model Complexity: While the channel pruning reduces size, the added LSTM network introduces new parameters. An analysis of its impact on model parameters considering the LSTM and training/testing time would be beneficial?*

**Reply:** Thank you for your insightful suggestion regarding the analysis of model complexity in relation to the LSTM network used in our approach. We appreciate the emphasis on understanding the overall impact on model parameters and time efficiency during training and testing. In response, we have conducted a comprehensive analysis, the details of which we will incorporate into our revised manuscript (in Appendix E).

- **Regarding the impact on time complexity:** Our recurrent hyper-network is designed for efficiency. The channel masks are pre-calculated, eliminating the need for real-time generation during both the inference and subnetwork fine-tuning phases. Therefore, the recurrent hyper-network does not introduce any extra time complexity during the inference and the fine-tuning phase. The additional computing time is limited to the phase of channel mask identification.

- **Moreover, the hyper-network itself is designed to be lightweight:** The number of parameters it contributes to the overall model is minimal, thus ensuring that any additional complexity during the mask-finding phase is negligible. This claim is substantiated by empirical observations: the hyper-network accounts for only about **0.2% to 6%** of the total model parameters across various architectures such as ResNet-18/34 and VGG-16, as illustrated in **Table 1**.

    **Table 1: The number of parameters for our Hypernetworks.**

    |                           | ResNet-18 (11M) | ResNet-34 (21M) | ResNet-50 (25M) | VGG-16 (138M) |
    |---------------------------|-----------------|-----------------|-----------------|---------------|
    | #Parameters-HyperNetwork  | 0.31M (2.8%)    | 0.56M (2.6%)    | 1.5M (6%)       | 0.34M (0.2%)  |

- Additionally, we assessed the training time per epoch with and without the hyper-network during the channel mask identification phase. Our findings in **Table 2** indicate that the inclusion of the LSTM network has a marginal effect on these durations, further affirming the efficiency of our approach.

    **Table 2: Training Time (s) per Epoch w/ and w/o Hypernetworks during Channel Mask Identification Phase with single A100 GPU.**

    |                  | ResNet-18 (11M) | ResNet-34 (21M) | ResNet-50 (25M) |
    |------------------|-----------------|-----------------|-----------------|
    | w/o HyperNetwork | 70.05           | 73.95           | 95.65           |
    | w/ HyperNetwork  | 72.2            | 76.95           | 111.6           |

Reviewer comment (Experiments):
Backbone Networks: this paper uses ResNet and VGG as the backbone networks. I recommend the authors also explore more contemporary and potentially powerful architectures, such as ResNeXT and ViT used in DepGraph.

Author response:
**Q2:** *Backbone Networks: this paper uses ResNet and VGG as the backbone networks. I recommend the authors also explore more contemporary and potentially powerful architectures, such as ResNeXT and ViT used in DepGraph.*  **Q3:** *Benchmarks: The paper's benchmarks are limited in size. Testing on larger datasets like ImageNet, as used in GrowReg, DepGraph, and other baselines, is recommended.*

**Reply:** We appreciate your suggestion to explore more contemporary architectures and larger datasets. 

- In response, we have extended our experiments to include the Vision Transformer (ViT-B/16) and ResNeXt-50 on ImageNet. We utilized pre-trained models from PyTorch Vision ([1]), ensuring that our experiments are replicable and grounded in standard benchmarks. The results are reported in **Table 3**.

- Notably, our method PASS demonstrates a significant speed-up with minimal accuracy loss, as indicated by the $\Delta$Acc., which are superior to existing methods like SSS, GFP, and DepGraph. the resulting empirical evidence robustly affirms the effectiveness of PASS across both advanced neural network architectures and large-scale datasets.

    **Table 3: Pruning results based on ImageNet and Advanced models.**

    | Arch.      | Method      | Base  | Pruned | Δ Acc. | FLOPs |
    |------------|-------------|-------|--------|--------|-------|
    | ResNeXt-50 | ResNeXt-50  | 77.62 | -      | -      | 4.27  |
    |            | SSS [2]     | 77.57 | 74.98  | -2.59  | 2.43  |
    |            | GFP [3]     | 77.97 | 77.53  | -0.44  | 2.11  |
    |            | DepGraph [4]| 77.62 | 76.48  | -1.14  | 2.09  |
    |            | Ours (PASS) | 77.62 | 77.21  | -0.41  | 2.01  |
    | ViT-B/16   | ViT-B/16    | 81.07 | -      | -      | 17.6  |
    |            | DepGraph [4]| 81.07 | 79.17  | -1.90  | 10.4  |
    |            | Ours(PASS)  | 81.07 | 79.77  | -1.30  | 10.7  |

[1] [TorchVision - Stable](https://pytorch.org/vision/stable/index.html)

[2] Huang, Zehao, and Naiyan Wang. "Data-driven sparse structure selection for deep neural networks." Proceedings of the European conference on computer vision (ECCV). 2018.

[3] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.

[4] Fang, Gongfan, et al. "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
Benchmarks: The paper's benchmarks are limited in size. Testing on larger datasets like ImageNet, as used in GrowReg, DepGraph, and other baselines, is recommended.

Author response:
**Q2:** *Backbone Networks: this paper uses ResNet and VGG as the backbone networks. I recommend the authors also explore more contemporary and potentially powerful architectures, such as ResNeXT and ViT used in DepGraph.*  **Q3:** *Benchmarks: The paper's benchmarks are limited in size. Testing on larger datasets like ImageNet, as used in GrowReg, DepGraph, and other baselines, is recommended.*

**Reply:** We appreciate your suggestion to explore more contemporary architectures and larger datasets. 

- In response, we have extended our experiments to include the Vision Transformer (ViT-B/16) and ResNeXt-50 on ImageNet. We utilized pre-trained models from PyTorch Vision ([1]), ensuring that our experiments are replicable and grounded in standard benchmarks. The results are reported in **Table 3**.

- Notably, our method PASS demonstrates a significant speed-up with minimal accuracy loss, as indicated by the $\Delta$Acc., which are superior to existing methods like SSS, GFP, and DepGraph. the resulting empirical evidence robustly affirms the effectiveness of PASS across both advanced neural network architectures and large-scale datasets.

    **Table 3: Pruning results based on ImageNet and Advanced models.**

    | Arch.      | Method      | Base  | Pruned | Δ Acc. | FLOPs |
    |------------|-------------|-------|--------|--------|-------|
    | ResNeXt-50 | ResNeXt-50  | 77.62 | -      | -      | 4.27  |
    |            | SSS [2]     | 77.57 | 74.98  | -2.59  | 2.43  |
    |            | GFP [3]     | 77.97 | 77.53  | -0.44  | 2.11  |
    |            | DepGraph [4]| 77.62 | 76.48  | -1.14  | 2.09  |
    |            | Ours (PASS) | 77.62 | 77.21  | -0.41  | 2.01  |
    | ViT-B/16   | ViT-B/16    | 81.07 | -      | -      | 17.6  |
    |            | DepGraph [4]| 81.07 | 79.17  | -1.90  | 10.4  |
    |            | Ours(PASS)  | 81.07 | 79.77  | -1.30  | 10.7  |

[1] [TorchVision - Stable](https://pytorch.org/vision/stable/index.html)

[2] Huang, Zehao, and Naiyan Wang. "Data-driven sparse structure selection for deep neural networks." Proceedings of the European conference on computer vision (ECCV). 2018.

[3] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.

[4] Fang, Gongfan, et al. "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Writing):
Please correct the typos in the title, "sparsity" and "recurrent".

Author response:
**Q4:** *Please correct the typos in the title, "sparsity" and "recurrent". It's preferable to place figures and tables at the top of a page.*

**Reply:** Thank you very much for highlighting these important details. 

- We have carefully reviewed the manuscript and corrected the typos in the title.

- Additionally, in line with your suggestion for improved readability and standard formatting practices, we have adjusted the layout to position figures and tables at the top of the pages in our revision.

Reviewer comment (Presentation):
It's preferable to place figures and tables at the top of a page.

Author response:
**Q4:** *Please correct the typos in the title, "sparsity" and "recurrent". It's preferable to place figures and tables at the top of a page.*

**Reply:** Thank you very much for highlighting these important details. 

- We have carefully reviewed the manuscript and corrected the typos in the title.

- Additionally, in line with your suggestion for improved readability and standard formatting practices, we have adjusted the layout to position figures and tables at the top of the pages in our revision.

Reviewer comment (Presentation):
The authors may consider switching the sequence of Figure 4 and Figure 5 to align with their respective mentions in the text.

Author response:
**Q5:** *The authors may consider switching the sequence of Figure 4 and Figure 5 to align with their respective mentions in the text.*

**Reply:** Thank you so much for pointing it out. Following your helpful suggestion, we have rearranged these figures in the revised version.

Reviewer comment (Experiments):
Is the proposed method effective to vision transformer based models?

Author response:
**Q1:** *Is the proposed method effective for vision transformer-based models?*

**Reply:** Thanks for your concern in transformer-based architecture. We report the performance of PASS on the ViT-B-16 and Swin-T models on ImageNet where the pre-trained ViT-B-16 and Swin-T are from torchvision [1]. The results in **Table 1** show that our method PASS achieves higher accuracy than baselines with similar FLOPs, validating PASS is a general method that can be applied to various architectures.

**Table 1: Pruning results Based on vision transformers on ImageNet.**

| Arch.    | Method      | Base  | Pruned | Δ Acc. | FLOPs |
|----------|-------------|-------|--------|--------|-------|
| ViT-B/16 | ViT-B/16    | 81.07 | -      | -      | 17.6  |
|          | DepGraph [2]| 81.07 | 79.17  | -1.90  | 10.4  |
|          | Ours (PASS) | 81.07 | 79.77  | -1.30  | 10.7  |
| Swin-T   | Swin-T      | 81.4  | -      | -      | 4.49  |
|          | X-Pruner [3]| 81.4  | 80.7   | -0.7   | 3.2   |
|          | STEP [4]    | 81.4  | 77.2   | -4.2   | 3.5   |
|          | Ours (PASS) | 81.4  | 80.9   | -0.5   | 3.4   |

[1] [TorchVision - Stable](https://pytorch.org/vision/stable/index.html)

[2] Fang, Gongfan, et al. "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Yu, Lu, and Wei Xiang. "X-Pruner: eXplainable Pruning for Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Li, Jiaoda, Ryan Cotterell, and Mrinmaya Sachan. "Differentiable subset pruning of transformer heads." Transactions of the Association for Computational Linguistics 9 (2021): 1442-1459.

Reviewer comment (Evaluation):
The recurrent hyper network approach might introduce complexity, especially in the LSTM network. Does the FLOPs computation involve the hyper-network? This requires more clear explanation in the paper.

Author response:
**Reply:** Thank you for your helpful feedback. We give detailed explanations here and will incorporate this detailed explanation into the revised version of our paper (in Appendix E).

- We acknowledge the concern about potential complexities arising from the incorporation of an LSTM network. It's crucial to emphasize that our paper primarily **focuses on achieving inference speedup**. The role of the hyper-network is solely dedicated to generating sparse masks. After that, **it will not be used during both the inference and fine-tuning phases**. Therefore, the FLOPs computation does not involve the hyper-network.

- Moreover, the hyper-network itself is designed to be lightweight. The number of parameters it contributes to the overall model is minimal, thus ensuring that any additional complexity during the mask finding phase is negligible. This claim is substantiated by empirical observations: the hyper-network accounts for only about **0.2% to 6%** of the total model parameters across various architectures such as ResNet-18/34/50 and VGG-16, as illustrated in **Table 1**.

    **Table 1:** The number of parameters for our Hypernetworks.

    |                           | ResNet-18 (11M) | ResNet-34 (21M) | ResNet-50 (25M) | VGG-16 (138M)  |
    |---------------------------|-----------------|-----------------|-----------------|----------------|
    | #Parameters-HyperNetwork  | 0.31M (2.8%)    | 0.56M (2.6%)    | 1.5M (6%)       | 0.34M (0.2%)   |

- Additionally, we assessed the training time per epoch with and without the hyper-network during the channel mask identification phase. Our findings in Table 2 indicate that the inclusion of the LSTM network has a marginal effect on these durations, further affirming the efficiency of our approach.

    **Table 2:** Training Time (s) per Epoch w/ and w/o Hypernetworks during Channel Mask Identification Phase with single A100 GPU.

    |                  | ResNet-18 (11M) | ResNet-34 (21M) | ResNet-50 (25M) |
    |------------------|-----------------|-----------------|-----------------|
    | w/o HyperNetwork | 70.05           | 73.95           | 95.65           |
    | w/ HyperNetwork  | 72.2            | 76.95           | 111.6           |

Reviewer comment (Experiments):
This paper only experiments with the convolution-based method. While the transformer-based approach, such as vision transformers and swin-transformers, has no investigations. To validate the generalization of the proposed approach, the authors need to provide more experiments on transformer-based networks.

Author response:
**Reply:** Thank you for your valuable suggestions emphasizing the need for broader experimentation to validate the generalizability of our proposed PASS methodology.

- In line with your recommendations, we have explored our approach to transformer-based networks and larger dataset, i.e., Vision Transformers (ViT-Base-16) and Swin Transformers (Swin-Tiny) on ImageNet-1K. The pre-trained weights are obtained from torchvision [1].

- The results from these extended experiments, as shown in **Table 3**, illustrate that PASS achieves higher accuracies than baselines with similar FLOPs on ImageNet with various advanced architectures, affirming that our proposed PASS can be generalized to advanced architectures and large datasets.

**Table 3: Pruning results based on ImageNet and Advanced models.**

| Arch.      | Method       | Base  | Pruned | Δ Acc. | FLOPs |
|------------|--------------|-------|--------|--------|-------|
| ResNeXt-50 | ResNeXt-50   | 77.62 | -      | -      | 4.27  |
|            | SSS [2]      | 77.57 | 74.98  | -2.59  | 2.43  |
|            | GFP [3]      | 77.97 | 77.53  | -0.44  | 2.11  |
|            | DepGraph [4] | 77.62 | 76.48  | -1.14  | 2.09  |
|            | Ours (PASS)  | 77.62 | 77.21  | -0.41  | 2.01  |
| ViT-B/16   | ViT-B/16     | 81.07 | -      | -      | 17.6  |
|            | DepGraph [4] | 81.07 | 79.17  | -1.90  | 10.4  |
|            | Ours (PASS)  | 81.07 | 79.77  | -1.30  | 10.7  |
| Swin-T     | Swin-T       | 81.4  | -      | -      | 4.49  |
|            | X-Pruner [5] | 81.4  | 80.7   | -0.7   | 3.2   |
|            | STEP [6]     | 81.4  | 77.2   | -4.2   | 3.5   |
|            | Ours (PASS)  | 81.4  | 80.9   | -0.5   | 3.4   |



[1] [TorchVision - Stable](https://pytorch.org/vision/stable/index.html)

[2] Huang, Zehao, and Naiyan Wang. "Data-driven sparse structure selection for deep neural networks." Proceedings of the European conference on computer vision (ECCV). 2018.

[3] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.

[4] Fang, Gongfan, et al. "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Yu, Lu, and Wei Xiang. "X-Pruner: eXplainable Pruning for Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[6] Li, Jiaoda, Ryan Cotterell, and Mrinmaya Sachan. "Differentiable subset pruning of transformer heads." Transactions of the Association for Computational Linguistics 9 (2021): 1442-1459.

Reviewer comment (Experiments):
The experiments performed in small-scale datasets, such as cifar10, cifar100. It is worth reporting results on large datasets such as imagenet.

Author response:
**Reply:** Thank you for your valuable suggestions emphasizing the need for broader experimentation to validate the generalizability of our proposed PASS methodology.

- In line with your recommendations, we have explored our approach to transformer-based networks and larger dataset, i.e., Vision Transformers (ViT-Base-16) and Swin Transformers (Swin-Tiny) on ImageNet-1K. The pre-trained weights are obtained from torchvision [1].

- The results from these extended experiments, as shown in **Table 3**, illustrate that PASS achieves higher accuracies than baselines with similar FLOPs on ImageNet with various advanced architectures, affirming that our proposed PASS can be generalized to advanced architectures and large datasets.

**Table 3: Pruning results based on ImageNet and Advanced models.**

| Arch.      | Method       | Base  | Pruned | Δ Acc. | FLOPs |
|------------|--------------|-------|--------|--------|-------|
| ResNeXt-50 | ResNeXt-50   | 77.62 | -      | -      | 4.27  |
|            | SSS [2]      | 77.57 | 74.98  | -2.59  | 2.43  |
|            | GFP [3]      | 77.97 | 77.53  | -0.44  | 2.11  |
|            | DepGraph [4] | 77.62 | 76.48  | -1.14  | 2.09  |
|            | Ours (PASS)  | 77.62 | 77.21  | -0.41  | 2.01  |
| ViT-B/16   | ViT-B/16     | 81.07 | -      | -      | 17.6  |
|            | DepGraph [4] | 81.07 | 79.17  | -1.90  | 10.4  |
|            | Ours (PASS)  | 81.07 | 79.77  | -1.30  | 10.7  |
| Swin-T     | Swin-T       | 81.4  | -      | -      | 4.49  |
|            | X-Pruner [5] | 81.4  | 80.7   | -0.7   | 3.2   |
|            | STEP [6]     | 81.4  | 77.2   | -4.2   | 3.5   |
|            | Ours (PASS)  | 81.4  | 80.9   | -0.5   | 3.4   |



[1] [TorchVision - Stable](https://pytorch.org/vision/stable/index.html)

[2] Huang, Zehao, and Naiyan Wang. "Data-driven sparse structure selection for deep neural networks." Proceedings of the European conference on computer vision (ECCV). 2018.

[3] Liu, Liyang, et al. "Group fisher pruning for practical network compression." International Conference on Machine Learning. PMLR, 2021.

[4] Fang, Gongfan, et al. "Depgraph: Towards any structural pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Yu, Lu, and Wei Xiang. "X-Pruner: eXplainable Pruning for Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[6] Li, Jiaoda, Ryan Cotterell, and Mrinmaya Sachan. "Differentiable subset pruning of transformer heads." Transactions of the Association for Computational Linguistics 9 (2021): 1442-1459.