Paper: Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective (ICLR-2024)

Reviewer comment (Experiments):
**Inadequate experiments.** This is a primary concern for the reviewer. The paper only presents experiments on CIFAR-10 and CIFAR-100, which, in the era of big data, are considered insufficient. These experiments do not adequately demonstrate the performance of the proposed method. Conducting experiments on larger datasets, such as ImageNet-1k, is essential.

Author response:
**[Cons 1. Only presenting experiments on CIFAR-10 and CIFAR-100 and conducting experiments on large datasets is essential.]** 

This is a possible misunderstanding of our experiment setting. As we state in the Abstract, Line 15; Section 4.1, Paragraph 1; Section 4.2, Paragraph 1; [Figure 4](https://imgur.com/YF7ODNx), we conduct our experiments on eight datasets which are Tiny-ImageNet, Food101, CIFAR100, CIFAR10, DTD, Flowers102, StanfordCars, and OxfordPets. We also display the results of pruning on ImageNet and fine-tuning on Tiny-ImageNet and CIFAR100 in [Figure 6](https://imgur.com/pOVS2DX).

**Additional experiments on ImageNet-1K.** To further alleviate reviewer hiB3’s concern, we conducted additional experiments on ImageNet-1K using our method and some of the best baselines such as HYDRA and OMP on ImageNet-1K pre-trained ResNet-18 network. The empirical results are shown in Table R10, **our method achieves the best accuracy** at 50% and 90% sparsity levels, which illustrates the effectiveness of our method on ImageNet. We also report the results in the revision of our paper in Table A5. Due to the time limitation of rebuttal, we provide outcomes only at two sparsity levels, more sparsity levels are promised in our final version.

Table R10. Performance comparison of our method, HYDRA, and OMP on ImageNet pre-trained ResNet-18 and ImageNet.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA | 68.91 | 66.62 |
| OMP | 69.31 | 64.27 |
| VPNs (ours) | 69.47 | 67.14 |

Reviewer comment (Experiments):
The method utilizes ImageNet-1K pre-trained weights, which were trained on a resolution of 224, but is tested on CIFAR data with a resolution of 32. Padding the data to a resolution of 224 can significantly boost performance. Therefore, experiments specifically conducted on ImageNet with a resolution of 224, and direct performance comparisons with fine-tuning on this resolution, are crucial.

Author response:
**[Cons 2. Our method uses a resolution of 32.]** 

Thanks for pointing out the question. We are aware of the role of resolution in the experiments and our responses to this concern are as follows.

1. [We use a resolution of 224]  We kindly indicate that we unify the resolution of 224 in our method and the baselines in all of our results in Figure [4](https://imgur.com/YF7ODNx) \ [5](https://imgur.com/7WL9Q9v) \ [6](https://imgur.com/pOVS2DX) \ [7](https://imgur.com/zqmXSrW) \ [8](https://imgur.com/VkoqArm). As displayed in Section 4.1, Paragraph 3, Line 5, we use an input size of 224 and pad size of 16, which corresponds to [Figure 3](https://imgur.com/y1fRg7v) that i = 224 and p = 16. This is also illustrated in our code, the main.py file, line 30, we use an input size of 224. For reviewer hiB3’s concern, we significantly polished our paper in Section 4.1, Paragraph 3 in our revision by specifying both our method and baselines using a resolution of 224 to avoid misunderstanding.

2. [We have results on datasets that have a resolution of 224] We gently state that we present results on Tiny-ImageNet which has an original resolution of 224 in Figure [4](https://imgur.com/YF7ODNx) \ [5](https://imgur.com/7WL9Q9v) \ [6](https://imgur.com/pOVS2DX). The findings of these experiments also demonstrate our method is superior to all the baselines at an original resolution of 224 circumstances. 

3. [Additional results on ImageNet] The results on ImageNet in Table R10 also illustrate that our method achieves superior performance on ImageNet.

Reviewer comment (Experiments):
The reviewer suggests providing performance comparisons with smaller models, as performance metrics on sparser models, such as using MobileNet, would be more indicative and informative.

Author response:
**[Cons 3. Providing performance comparisons with smaller models such as MobileNet.]**

Thank you for the great suggestion. We conducted additional experiments on ImageNet pre-trained MobileNet and CIFAR100 using our method, HYDRA, and OMP. **We observe that our method achieves {2.24%, 1.27%} higher accuracy** than {HYDRA, OMP} at 50% sparsity level as shown in Table R11. The results are also included in the revision of our paper in Table A12. Due to the time limitation, we provide outcomes only at two sparsity levels, more sparsity levels are promised in our final version.

Table R11. Performance comparison of our method, HYDRA, and OMP on ImageNet pre-trained MobileNet and CIFAR100.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA | 79.12 | 70.02 |
| OMP | 80.09 | 41.82 |
| VPNs (ours) | 81.36 | 75.5 |

Reviewer comment (Experiments):
The method exhibits significant limitations, as it necessitates the use of pre-trained weights from a larger dataset. The current version seems to require transforming the ImageNet model to CIFAR. It would be insightful to explore the performance without pre-trained weights. Additionally, conducting comparisons on a larger pre-trained dataset, such as ImageNet, appears necessary for the current version.

Author response:
**[Cons 4. It would be insightful to explore the performance without pre-trained weights]**

Thank you for the advice. We state that the pruning paradigm of our method which requires first finding masks in a trained model and then tuning the subnetwork to recover the accuracy can’t be used on models without pre-training. This is because a model without pre-training has random weights making the mask-finding stage of our method useless. 

**Additional experiments of training from scratch.** To further alleviate reviewer hiB3’s concern, we conducted additional experiments by applying the VPNs pruning paradigm on SynFlow named VPNs w. SynFlow and pruning from scratch on ResNet-18 and CIFAR100. We observe that **VPNs w. SynFlow(with VP) achieves {8.85%, 5.91%} higher accuracy** than the original SynFlow (without VP) at {50%, 90%} sparsity levels as shown in Table R12, which indicates visual prompting significantly enhances models’ sparsification in the setting of training from scratch. We also report the results in the revision of our paper in Table A13. Due to the time limitation, we provide outcomes only at two sparsity levels, more sparsity levels are promised in our final version.

Table R12. Performance comparison of VPNs w. SynFlow and SynFlow pruning from scratch on ResNet-18 and CIFAR100.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| SynFlow (without VP) | 66.77% | 64.46% |
| VPNs w. SynFlow (with VP) | 75.62% | 70.37% |

Reviewer comment (Presentation):
The plotting style, with "ours" appearing as a straight line, has the potential to cause confusion; upon initial review, it might be perplexing why this is the case.

Author response:
**[Cons 5. It might be perplexing why the curve representing "ours" appears as a straight line.]**

Thank you for the great point. We use a similar plot format as in [15]. Regarding reviewer hiB3’s concern, we deleted the “Our Best” dashed lines in all plots in the revision of our paper.

[15] Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. Advances in Neural Information Processing Systems, 35:18309–18326, 2022a.

Reviewer comment (Experiments):
I think the authors should focus more on the structured sparse case, unstructured pruning is well known that will not contribute to any acceleration in practice.

Author response:
**[Cons 1. Focusing more on the structured sparse case.]** 

Thank you for the suggestion. We kindly point out that our main contributions focus on introducing the new data-model co-design pruning paradigm and we respectfully argue that focusing on unstructured pruning is highly meaningful. Unstructured pruning is extremely useful as both a mathematical prototype and an empirical testbed for new SNN algorithms and it is also receiving increasingly better support in practice. The evidence is as follows:

1. [Better performance than structured pruning] As the finest-grained and most flexible sparsity level, unstructured sparsity is superior to other more structured forms of sparsity [7].

2. [Widely used on nonGPU hardware] Unstructured sparsity has widely proven its practical relevance on nonGPU hardware, such as CPUs or customized accelerators. For instance, in the range of 70-90% high unstructured sparsity, XNNPACK [8] has already shown significant speedups over dense baselines on smartphone processors. 

3. [Receive increasing support] The hardware support of unstructured sparsity may be relatively limited on “off-the-shelf” commodity GPUs/TPUs, but it keeps improving quickly over the years. For example, advanced GPU kernels such as NVIDIA cuSPARSE [9] and Sputnik [10] have built the momentum to better support finer-grained sparsity.

4. [Our method achieves time reduction in mask finding] We demonstrate that our method (unstructured pruning) achieves significant time reduction in terms of mask finding compared to other pruning methods like HYDRA and BiP as shown in [Figure 9](https://imgur.com/t2ysbTc).

Reviewer comment (Evaluation):
For structured pruning, the speedup ratio should use latency, not theoritical FLOPs. And more recent methods should be compared.

Author response:
**[Cons 2. The speedup ratio should use latency.]** 

Thank you for the advice. We use FLOPs following [11, 12] and the FLOPs are also widely used in calculating the speedup ratio. To alleviate reviewer eRqj’s concern, we provide the latency speedup ratio of our method on the Quadro RTX 6000 GPU in Table R6. **Our method achieves latency speedup ratios of 1.1× and 1.2× at 10% and 20% channel-wise sparsity** respectively without compromising the performance relative to the dense network, as indicated in [Figure 8](https://imgur.com/VkoqArm). We kindly point out that our main contributions focus on introducing the new data-model co-design pruning paradigm and unstructured pruning, we provide the results of structured pruning only to indicate a potential research direction. We also include the latency results in the revised paper in Table A9.

Table R6. The latency and FLOPs of VPNs structured pruning on ImageNet pre-trained ResNet-18 and CIFAR100.
| Sparsity | Latency(ms) | FLOPs(G) |
| :----------: | :----------: | :----------: |
| Dense | 2.23 &plusmn; 0.03 | 1.82 |
| 10% sparsity | 2.02 &plusmn; 0.02 | 1.68 |
| 20% sparsity | 1.86 &plusmn; 0.02 | 1.45 |

Reviewer comment (Experiments):
The visual prompting method essentially uses lower resolution for the input images. It is necessary to compare a baseline that using a lower resolution image as input, and then prune less parameters to maintain the same FLOPs as the proposed VPN.

Author response:
**[Cons 4. Our method essentially uses lower resolution input and needs to compare the baselines with lower resolution.]**

This might be a misunderstanding of our setting and we respond to the concern from the following aspects:

1. [The resolution of our method and baselines is always 224] In our design, the resolution of our method is always 224 which equals the resolution of the input images in all baselines.  Although the visual prompt as learnable parameters are added to the margin of the image, the full image pixel information is preserved. Evidence can be found in Section 4.1, Paragraph 3, Line 5 that we use an input size of 224 and a pad size of 16, and in [Figure 3](https://imgur.com/y1fRg7v)  that i = 224 and p = 16.

2. [**Additional experiments of baselines using lower resolution and pruning fewer parameters**] To further alleviate reviewer eRqj's concern, we conducted additional experiments on some of the best baselines such as HYDRA and OMP with lower resolution and pruning fewer parameters on ImageNet pre-trained ResNet-18 and CIFAR100. HYDRA and OMP use an input resolution of 192 and prune 13k (the number of parameters in the visual prompt) fewer parameters than our method. We find that **our method achieves {3.94%, 3.69%} higher accuracy** than {HYDRA, OMP} at 90% sparsity level from the results displayed in Table R8, which indicates our method is superior to baselines using lower resolution and pruning fewer parameters. We also add the results in the revision of our paper in Table A11. Due to the time limitation, we provide outcomes only at two sparsity levels, more sparsity levels are promised in our final version.

Table R8. Performance comparison of our method, HYDRA, and OMP on ImageNet pre-trained ResNet-18 and CIFAR100. HYDRA and OMP use 192 as the input resolution and prune 13k fewer parameters than our method. 
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA | 80.9 | 76.64 |
| OMP | 80.89 | 76.89 |
| VPNs (ours) | 83.18 | 80.58 |

Reviewer comment (Experiments):
Following the previous point, I also wonder whether this method will deteriorate some applications that are senstive to resolution, such as object detection or etc.

Author response:
**[Cons 5. Whether our method will deteriorate in applications that are sensitive to resolution such as object detection.]**

This is an interesting direction to explore. We provide detailed responses below.

1. [Our method is insensitive to resolution] As illustrated above, our method and the baselines all use a resolution of 224 in the original setting and our method surpasses all baselines as shown in [Figure 4](https://imgur.com/YF7ODNx). In the additional experiments displayed in Table R8, our method is still better than the baselines using a resolution of 192. No matter when the baselines use the resolution of 224 or 192, our method is consistently better than the baselines, demonstrating our method is insensitive to resolution.

2. [**Superior performance on object detection**] To further alleviate reviewer eRqj's concern, we conducted supplemental experiments on Pascal VOC 2007, which is a well-known object detection task. We compare our method to HYDRA and OMP on YOLOv4 with ImageNet pre-trained ResNet-18 backbone. **Our method achieves {3.78%, 2.67%} higher AP** than {HYDRA, OMP} at 90% sparsity level as shown in Table R9, which indicates the superiority of our method in resolution-sensitive tasks. The results are also included in the revision of our paper in Table A6. Due to the time limitation, we provide outcomes only at two sparsity levels, more sparsity levels are promised in our final version.

Table R9. AP comparison of our method, HYDRA, and OMP on YOLOv4 with ImageNet pre-trained ResNet-18 backbone and Pascal VOC 2007.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA | 35.25 | 32.74 |
| OMP | 35.01 | 33.85 |
| VPNs (ours) | 38.37 | 36.52 |

Reviewer comment (Writing):
The paper is not easy to read. There's a particular emphasis on "data model co-design", but it takes quite a while to understand what this refers to concretely.

Author response:
**[Cons 1. The statement of “data-model co-design” is not concrete.]** 

Thanks for putting forward the point. Specifically, data-model co-design indicates both the input data and the model are optimized in pruning. This paradigm of our method is significantly different from the baselines which are model-centric. 
1. [Model-centric design] Focusing on searching and preserving crucial weights by analyzing network topologies (Abstract, Line 6) with fixed inputs as illustrated in [3].
2. [Data-model co-design] Optimizing weight masks and visual prompts jointly by combining data-centric design that constructs well-designed prompts (Section 1, Paragraph 3, Line 3) and model-centric design.
More explanation of data-model co-design can be found in Section 1, Paragraph 5, Point 2; [Figure 2](https://imgur.com/tcdNFkO); and Section 3, Paragraph 1.

To further alleviate reviewer sGzb’s concern, we significantly polished our paper in Section 1, Paragraph 5, and Section 3, Paragraph 1 by interpreting the data-model co-design more concretely to avoid misunderstanding in our revision.

Reviewer comment (Experiments):
Why is the visual prompts essential? How about learning additional parameters without using the visual prompt?

Author response:
**[Cons 2. Why is the visual prompt essential?]** 

Thank you for the question. The significance of the visual prompt is detailed as follows:

1. [Motivation] 

- Xu et al. [4] demonstrate that prompts can recover compressed LLMs which illuminates the efficacy of post-pruning prompts in enhancing the performance of compressed LLMs. Observing that post-pruning prompts bolster the efficiency and performance of LLMs, It is only natural to inquire about the impact of VPs on vision models.

- We observe from [Figure 1](https://imgur.com/37Np8Dh) that post-pruning prompts escalate the performance of the subnetworks before fine-tuning. However, neither of these settings consistently surpasses the standard no-prompting approach (pruning + fine-tuning). We postulate visual prompts are capable of enhancing the sparsification of vision models by utilizing them in a different way.

2. [Empirical evidence] 
- As shown in [Figure 4](https://imgur.com/YF7ODNx) and [Figure 5](https://imgur.com/7WL9Q9v), we demonstrate the essential role of VP by comparing our method (with VP) to HYDRA (without VP) on eight downstream datasets and three architectures.

- As shown in [Figure 7](https://imgur.com/zqmXSrW), we observe that VP combined with existing prunings consistently surpasses their original counterpart (Section 4.2, Paragraph 4), which further illustrates the essential of VP.

3. [Previous literature] 

- [5, 6] showcases visual prompting as a parameter-efficient method that substantially improves the generalization and accuracy of vision models over multiple tasks. We believe these advantages brought by visual prompts also benefit pruning.

Reviewer comment (Novelty):
How would visual prompts be different from data augmentation?

Author response:
**[Cons 4. How would visual prompt be different from data augmentation?]**

Visual prompt is essentially different from data augmentation, and the reasons are as follows:

1. [Purpose] Visual prompts are usually designed to improve model adaption. While data augmentations are used to avoid overfitting.

2. [Methodology] Visual prompts can be optimized in a data-driven way. However, most data augmentations are data-agnostic.In our method, visual prompting is the key to enabling the data-model exploration of crucial sparsity topologies.

3. [**Additional experiments of more data augmentations**] To further address reviewer sGzb’s concern, we compare our method without mix-ups to HYDRA and OMP with mix-ups on ImageNet pre-trained ResNet-18 and CIFAR10. **Our method achieves {5.46%, 3.50%} higher accuracy** than {HYDRA, OMP} at 90% sparsity level as shown in Table R5, which indicates that visual prompting is better than using more data augmentation. We add the results to the revised paper in Table A8. Due to the time limitation, we provide outcomes only at 50% and 90% sparsity levels, more sparsity levels are promised in our final version.

Table R5. Performance comparison of our method (without mix-up), HYDRA (with mix-up), and OMP (with mix-up) on ImageNet pre-trained ResNet-18 and CIFAR10.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA (with mix-up) | 92.72 | 90.83 |
| OMP (with mix-up) | 94.63 | 92.79 |
| VPNs (without mix-up) | 96.47 | 96.29 |

Reviewer comment (Experiments):
Introducing visual prompts into vision models seems to boost their performance. However, comparing models with visual prompts (the proposed method) to those without (other baseline methods) might not be entirely fair. What if we apply both the proposed and baseline methods to a model that has already been fine-tuned with visual prompts?

Author response:
**[Cons 1. Comparing models with visual prompts to those without might not be entirely fair.]** 

We respectfully disagree. We have tried our best to guarantee fairness in the empirical results shown in [Figure 7](https://imgur.com/zqmXSrW). Specifically, in the setting of [Figure 7](https://imgur.com/zqmXSrW): 
1. [For our method (VPNs)] We leverage VP in mask finding and subnetwork tuning. 
2. [For baselines (VPNs w. Random, VPNs w. OMP, VPNs w. LTH)] We apply the same VP as our method to Random, OMP, and LTH in mask finding and subnetwork tuning. 

Both our method and baselines use exactly the same fine-tuning of subnetwork weights and VP. But our method apparently surpasses all baselines with visual prompting at all sparsity levels as shown in [Figure 7](https://imgur.com/zqmXSrW), which indicates the superiority of our method in the “entirely fair” setting.

**Additional experiments of baselines with VP.** To further address reviewer 8b5d’s concern, we conduct supplemental experiments using our method and some of the best baselines such as LTH and OMP with VP on ImageNet pre-trained ResNet-18 network and Tiny-imageNet. We observe that **our method is superior to baselines with VP** as displayed in Table R1. We also include the results in the revision of our paper in Table A4. Due to the time limitation of rebuttal, we provide outcomes at 50% and 90% sparsity levels, and more sparsity levels are promised in our final version. 

Table R1. Performance comparison of our method, VPNs w. LTH, and VPNs w. OMP on ImageNet pre-trained ResNet-18 and Tiny-ImageNet.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| VPNs w. LTH | 73.71 | 67.03 |
| VPNs w. OMP | 73.56 | 64.78 |
| VPNs (ours) | 73.82 | 67.89 |

Reviewer comment (Experiments):
I have some doubts regarding the generality and performance of this paper. Why must we conduct experiments on downstream tasks of ImageNet? Why not directly on ImageNet itself, since most pruning work actually focuses more on performance on ImageNet?

Author response:
**[Cons 2. Why not directly conduct experiments on ImageNet itself?]** 

Thank you for the great advice. We provided results of pruning on ImageNet and fine-tuning on downstream datasets for transfer study in [Figure 6](https://imgur.com/pOVS2DX). 

**Additional experiments on ImageNet.** To further alleviate reviewer 8b5d’s concern, we conducted additional experiments on ImageNet using our method and some of the best baselines such as HYDRA and OMP, on ImageNet pre-trained ResNet-18 network. **Our method has {0.52%, 2.87%} higher accuracy** than {HYDRA, OMP} at 50% sparsity level as shown in Table R2, which illustrates the effectiveness of our method on ImageNet.  We also report the results in the revision of our paper in Table A5. Due to the time limitation of rebuttal, we provide outcomes at 50% and 90% sparsity levels, more sparsity levels are promised in our final version.

Table R2. Performance comparison of our method, HYDRA, and OMP on ImageNet pre-trained ResNet-18 and ImageNet.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA | 68.91 | 66.62 |
| OMP | 69.31 | 64.27 |
| VPNs (ours) | 69.47 | 67.14 |

Reviewer comment (Experiments):
The method proposed in this paper seems to be limited to scenarios where visual prompts can be applied, with their main application currently being in classification tasks. How can the proposed approach be used for other tasks, such as detection, segmentation, etc.?

Author response:
**[Cons 3. How can the proposed approach be used for other tasks, such as detection?]**

Thank you for the great point. Following the reviewer’s suggestion, we carried out additional experiments on Pascal VOC 2007 [1], which is one of the most widely used datasets for object detection tasks. 

**Additional experiments on Pascal VOC 2007.**  We compared our method to some of the best baselines such as HYDRA and OMP on YOLOv4 [2] which uses ImageNet pre-trained ResNet-18 as the backbone. The outcomes are presented in Table R3. **Our method achieves {3.78%, 2.67%} higher AP** than {HYDRA, OMP} at 90% sparsity level, which demonstrates the superiority of our method on object detection. We add the results to the revised paper in Table A6. Due to the time limitation, we provide outcomes only at two sparsity levels, more sparsity levels are promised in our final version. 

Table R3. AP comparison of our method, HYDRA, and OMP on YOLOv4 with ImageNet pre-trained ResNet-18 backbone and Pascal VOC 2007.
| Method | 50% sparsity | 90% sparsity |
| :----------: | :----------: | :----------: |
| HYDRA | 35.25 | 32.74 |
| OMP | 35.01 | 33.85 |
| VPNs (ours) | 38.37 | 36.52 |

[1] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010.

[2] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020.