Paper: ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models (ICLR-2024)

Reviewer comment (Novelty):
The method in this paper is relatively simple, and novelty is insufficient.

Author response:
> **C1. It lacks a theoretical analysis in the method introduction section to explain the effectiveness of the proposed method. And the method in this paper is relatively simple, and novelty is insufficient.**


We respectfully disagree that theoretical analysis is necessary for our introduction since we already provided clear motivations/challenges for tackling efficient pruning for large vision-language models. And, we propose a new method the coarse-to-fine layer-wise pruning approach upon these motivations suggested :

1) layer-wise one-shot pruning used for large models cannot figure out the optimal pruning rate per layer due to the lack of global (i.e., entire model) information (please see Introduction and Section 3.2).

2) In large multi-modal models, training modules for different modalities show a significant disparity in their weight and gradient scales. This distributional gap in different modules makes layer-wise one-shot pruning further challenging, and often fails to preserve the model performance with a higher sparsity ratio (please see Figure 1 and Figure 4).


From the perspective of novelty, our ECoFLaP takes unique advantages from both layer-wise and global pruning methods. EcoFLaP preserves the performance of full-precision models by computing intra-/inter weight importance (global pruning), yet this process is rapid and efficient (layer-wise pruning) through **our proposed two-step pruning procedures with zeroth-order approximation**. We utilize zeroth-order gradients for global information and bypass the need to construct a backpropagation graph, and thus **our approach can use a similar amount of GPU memory as layer-wise pruning**. To the best of our knowledge,**there has been no previous work similar to our idea for network pruning** and we believe this novel approach sufficiently contributes to the large and/or multimodal model pruning fields with clear insights.

To demonstrate the effectiveness of the proposed method, we provide extensive empirical evidence/analyses with multiple architectures (BLIP2, FlanT5, ViT, LLaMA) and tasks (VQA, Captions, MMLU, ImageNet, WikiText-2, etc), and we also report the results on CLIP in the next response. In those experiments, our approach outperforms the baseline by a margin in the high-sparsity regime, and we believe that **compared to theoretical analysis, the actual experimental results are the most straightforward and useful way to demonstrate the effectiveness of our approach**.

Reviewer comment (Theory):
The paper's writing could be improved as it lacks a theoretical analysis in the method introduction section to explain the effectiveness of the proposed method. Additionally, there is a scarcity of formulas in the paper, and many details are not adequately clarified.

Author response:
> **C2. there is a scarcity of formulas in the paper, and many details are not adequately clarified.**

We politely disagree with the notion that using many formulas necessarily makes the paper clearer. And we believe we already have included all the essential equations required to explain the algorithm effectively. 
Could you please help specify which parts you think are unclear, so we can address them more clearly/thoroughly in our rebuttal?

Reviewer comment (Experiments):
The experiments conducted were not comprehensive enough.

Author response:
> **C3. The experiments conducted were not comprehensive enough. a) The validation of the models in the experiments was also insufficient. Only the compression effects on the BLIPs model were tested, and the widelyused CLIP model, which is a mainstream multimodal model, was not evaluated.**

Thank you for your suggestions on more experiments on CLIP. However, as we mentioned in the Introduction and Section 3.1, in the paper, we focused on the modularized vision-language model, in which the vision and language components are trained independently. We also found this kind of modularized VL model (Flamingo, MiniGPT series, LLaVA, Qwen-VL, etc) becomes more common as the strong LLM emerges, and therefore our study will be useful for these VL models. On the other hand, we didn’t conduct experiments on CLIP because its vision and language transformers are trained together, which makes it not in our target models. However, we still perform the experiments on CLIP during the rebuttal, and show the results in the following table. We use 11 classification tasks to evaluate the zero-shot performance of CLIP before and after pruning. We ran Wanda and ECoFLaP over the sparsity 0.3 to 0.5, and we observed that ECoFLaP outperforms Wanda in the three sparsity ratios (we only report the results on sparsity 0.4 to make the reply succinct). In the 0.4 sparsity, **our approach shows a 5% to 18% improvement over Wanda in the 11 tasks**. We have included this experiment in Table 8 and Appendix F of our revision.

| Methods                |  caltech101 | dtd | eurosat | fgvc_aircraft | food101 | imagenet | oxford_flowers | oxford_pets | stanford_cars | sun397 | ucf101 |
|------------------------|---|---|---|---|---|---|---|---|---|---|---|
| Full Model             |  92.9 | 44.5 | 47.8 | 24.8 | 86.1 | 66.7 |    71.3  | 89.1 | 65.3 | 62.6 |66.8 |
| Wanda @ 0.4 sparsity   | 78.9 | 23.9 | 21.1  | 5.9  | 50.0 | 36.8 |    30.4 | 56.0  | 24.6  | 42.9  | 48.2|
| ECoFLaP @ 0.4 sparsity | **86.6** | **31.7** | **27.1** | **12.9** | **65.0** | **48.7** |   **46.8**  | **74.6**  | **40.1**  | **53.9** | **57.7** |

Reviewer comment (Experiments):
The validation of the models in the experiments was also insufficient. Only the compression effects on the BLIPs model were tested, and the widely used CLIP model, which is a mainstream multimodal model, was not evaluated. Furthermore, the comparison with the UPop model was only conducted on the NLVR2 and COCO Caption datasets, without comparisons on other datasets such as Flickr30k and VQA2.0.

Author response:
> **C4. Furthermore, the comparison with the UPop model was only conducted on the NLVR2 and COCO Caption datasets, without comparisons on other datasets such as Flickr30k and VQA2.0.** 

We also ran the Flickr30k experiments to compare with UPop and show the results in the following Table. We show that ECoFLaP outperforms Wanda in the no-fine-tuning setting, and outperforms UPop after the fine-tuning with almost no performance drop. This trend is the same as what we found in the NLVR2 and COCO experiments. We are running the VQA experiments and will release the results soon.


| Methods                   | Flickr30k (TR@1/IR@1) |
|---------------------------|-----------------------|
| Full model                | 96.8/86.9             |
| Wanda (w/o fine-tuning)   |  85.3/72.3    (-11.5/-14.6)           |
| ECoFLaP (w/o fine-tuning) | 90.2/79.5  (-6.6/-7.4)              |
| UPop (w/ fine-tuning)     | 94.0/82.0  (-2.8/-4.9)             |
| ECoFLaP (w/ fine-tuning)  |   **96.8**/**85.6** (-0.0/-1.3)            |

Reviewer comment (Experiments):
The ablation study experiments were somewhat simplistic and did not substantiate the fundamental reasons for the effectiveness of the proposed method. It would be beneficial to enhance the content of the ablation study experiments to provide a more detailed analysis and strengthen the overall credibility of the article.

Author response:
> **C5. The ablation study experiments were somewhat simplistic and did not substantiate the fundamental reasons for the effectiveness of the proposed method. It would be beneficial to enhance the content of the ablation study experiments to provide a more detailed analysis and strengthen the overall credibility of the article.**

To show the benefits of our main novelty in **“layer-wise pruning with global dynamic sparsity ratios”**, we demonstrated that our proposed approach is stronger than the plain layer-wise pruning methods on various models and tasks in Table 1, Figure 4, and Figure 7. **We believe these experiments have proved the effectiveness of our hypothesis on “dynamic layer ratios are critical for layer-wise pruning”**, and this is the fundamental reason that our approach outperforms the baselines. Moreover, in Table 1, we also show that **our approach uses a lot less GPU memory than other methods using first-order gradients**, and this justifies our idea of replacing first-order gradients with forward-only zeroth-order gradients.

For the ablation studies, we **comprehensively presented the hyperparameter choice of the zeroth-order optimization**, which is the main component of our approach. We showed that the results are quite **robust to the hyperparameters of the zeroth-order optimization**. In the last paragraph of Section 6.2 (ablation study) and Figure 6, we also visualized the loss landscape of BLIP-2 and found the landscape is bell-shaped with a smooth basin and a single local minimum. **This loss landscape also justifies the success of utilizing zeroth-order optimization** as we avoid sampling weights located in other basins that may cause an inaccurate gradient estimation.

In sum, we believe that our experiments have adequately justified the main reason for the effectiveness of our approach and hypothesis. Could you please be more specific about which part of the experiments is unclear or what hypothesis needs to be clarified more, so we try to address them during the rebuttal?

Reviewer comment (Evaluation):
In addition to the numerical comparison, can you show some example results that compare the before and after results?

Author response:
We conducted an analysis comparing the predictions of the Full model (before pruning) and the model pruned by ECoFlaP (with 0.5 sparsity) on the GQA datasets, and presented three illustrative examples in Table 9 (in Appendix G) of our revision. Please refer to them for the details.

We found one interesting observation: out of 12578 questions, the Full model correctly answered 1219 questions that our model did not, yet our model also accurately responded to 1072 questions where the Full model failed to respond. **This indicates that pruning does not always lead to performance degradation; in some instances, it may even enhance model robustness for certain questions** [1]. During our review of incorrect predictions by either our model or the Full model, we noticed that most were conceptually similar to the ground truth. This suggests that the pruned model still retains most of the capability. Furthermore, in some cases, such as with the 'horses running' example, both models provided answers that might be considered correct by many humans, even though they differ from the ground truth.

[1] Zhangheng Li, Tianlong Chen, Linyi Li, Bo Li, Zhangyang Wang, Can Pruning Improve Certified Robustness of Neural Networks?

Reviewer comment (Evaluation):
While it is not easy to measure the real performance improvement, can you discuss about it with the proposed pruning method?

Author response:
In this paper, we evaluate the pruning approaches on large pre-trained models (BLIP2, FlanT5, ViT, LLaMA) with downstream tasks in a zero-shot setting, and this approach is one of the most common ways to benchmark pre-trained models. With this evaluation approach, we demonstrated that ECoFLaP has an improvement over the compared approaches on various architectures and downstream tasks in Table 1, Figure 4, and Figure 7. While we have tried our best to evaluate the models in this way, could you elaborate more on the method in your mind that can measure the real performance improvement, so we can conduct the method during the rebuttal phase?

Reviewer comment (Evaluation):
Based on the importance score, what are important layers? What's the distribution of scores across all layers?

Author response:
Our sparsity ratios for layers are computed by the importance scores and thus the ratios can reflect the importance, for example, the higher sparsity for one layer denotes that this layer is less important for the tasks. In Figure 5, we can see that the sparsity ratio generally is lower for the vision transformer and higher for the language model, and this suggests that the language model is more prunable and has more redundancy for the vision-language tasks.

Reviewer comment (Reproducibility):
There is a need for a more detailed description of Algorithm 1.

Author response:
Sorry to hear that you feel further details are needed in Algorithm 1. However, we made effort to providing  extensive explanations of our algorithm using comments in Algorithm 1 and also used the equations in Section 4.1 as support. Could you please elaborate more detail which part is unclear so we can improve it during the rebuttal phase?

Reviewer comment (Evaluation):
Several metrics are used for importance scores, including weight magnitude, the multiplication of gradient and weight magnitude, gradient only, and sensitivity analysis. It would be beneficial to provide a comprehensive comparison of these metrics.

Author response:
Thank you for your suggestion. We presented a table to compare the approaches in Table 1 in terms of whether they are global or not and the used importance measure. We have included this table in our revision (Please see Table 6 in the Appendix). 

| Methods                  | Global or Layer-wise | Importance Measure                                                                |
|--------------------------|----------------------|-----------------------------------------------------------------------------------|
| Global Magnitude Pruning | Global               | $\| W \|_{ij}$                                                                   |
| Iterative OBD Pruning    | Global               | $ \| W \| _{ij} \cdot \| dL / dW \| _{ij}$                                               |
| SparseGPT                | Layer-wise           | $[\|W\|^ 2 / diag (XX^T + λI)^{−1}]_{ij}$                                         |
| Wanda                    | Layer-wise           | $\|W_{ij}\| \cdot \|X_{j}\|^2 $                                                   |
| ECoFLaP (zeroth-order)   | Mixed     | First Stage: $ \| dL / dW \| _{ij} $ Second Stage: $\|W _{ij}\| \cdot \|X _{j}\|^2 $ |

Reviewer comment (Writing):
There are some typos in this paper: (1) In Section 4.1, "and then convert the scores to sparsity by three steps" should be "and then convert the scores to sparsity by four steps"? (2) In Section 4.2, "like less than 5B parameters" should be "Like less than 5B parameters"?

Author response:
Thank you for catching the typos. We have incorporated them in the revision.

Reviewer comment (Experiments):
The sparsity ratios in the experiments cover a range from 0.1 to 0.6. Is it possible to extend this range to include a wider spectrum of sparsity ratios?

Author response:
Thank you for your suggestion on evaluation with a wider sparsity range. During the rebuttal period, we have compared Wanda and ECoFLaP at 0.7 sparsity in the following table. 

As shown, ECoFLaP consistently outperforms Wanda by a significant margin, especially on NoCaps and Flickr30k (30% up improvement). This is because recent layer-wise pruning approaches can only compute local (i.e., within-layer) importance, which leads to suboptimal compression, while our proposed method can efficiently and accurately prune model weights based on estimated global importance. In this case, our approach **assigns lower pruning ratios on the vision transformer, and this prevents a serious performance drop at such a high sparsity**. 

| **Methods @ 0.7 sparsity** | **VQA (Acc)** | **OK-VQA (Acc)** | **GQA (Acc)** | **NoCaps (CIDEr/SPICE)** | **Flickr30k (TR@1/IR@1)** |
|--------------------|---------------|------------------|---------------|--------------------------|---------------------------|
| Wanda                      | **4.76**          | 3.76             | 2.96          |  8.13/5.97         | 7.6/22.76                 |
| ECoFLaP (zeroth-order)     | **4.76**          | **5.05**             | **4.14**          | **38.28**/**8.74**         | **78.1**/**68.34**      |

Reviewer comment (Experiments):
Can this method be combined with SparseGPT?

Author response:
Thank you for your constructive suggestion. As the key contribution of our method for efficient estimation of layer-adaptive pruning rate for layer-wise pruning is orthogonal to SparseGPT, ECoFLaP can also be combined with SparseGPT. As shown in the table below, EcoFLaP+SparseGPT consistently achieves superior performance over multiple tasks. 
 
Additionally, the performance improvement seems relatively smaller compared to ECoFLaP with Wanda (i.e., original EcoFLaP). We hypothesize that this is because the reweighting mechanism in SparseGPT is somewhat correlated to our dynamic sparsity ratios, where both of them are important to preserve the performance in high sparsity ratios. We have added this discussion and results in Appendix E and Table 7 of our revision.

| **Methods @ 0.6 sparsity** | **VQA (Acc)** | **OK-VQA (Acc)** | **GQA (Acc)** | **NoCaps (CIDEr/SPICE)** | **Flickr30k (TR@1/IR@1)** |
|--------------------|---------------|------------------|---------------|--------------------------|---------------------------|
| SparseGPT    | 48.68  | 27.88      | 35.10       |  101.04/13.41       | 95.2/84.98 |
| ECoFLaP on SparseGPT (zeroth-order)     | **50.36**          | **27.93**             | **36.26**          | **105.25**/**13.96**   | **96.3**/**86.12**      |

Reviewer comment (Theory):
The paper claims that pruning multi-modal large models differs from other large models, but lacks mathematical or experimental support for this assertion.

Author response:
> **C2.The paper claims that pruning multi-modal large models differs from other large models, but lacks mathematical or experimental support for this assertion.**

We politely argue that this is a misunderstanding. we clearly provided analyses and results that pruning multi-modal models differs from pruning single-modal models in Section 3.2, and **conventional single-modal layer-wise pruning methods suffer from significant performance degeneration in multi-modal model in high sparsity regime (Please see Figure 4(a))**. In Section 3.2 and Figure 1(c), we demonstrated that **local score estimation is imbalanced for vision and language models**, which causes inaccurate estimation of the pruning ratios for each layer in the model. Also, the local score of each parameter cannot infer the importance of the final loss, and that is one reason that layer-wise pruning only uses a fixed pruning ratio for layers.

Reviewer comment (Novelty):
The use of layer-wise pruning due to the computational cost of calculating the inverse of the Hessian is common in other large model pruning approaches.

Author response:
> **C3. The use of layer-wise pruning due to the computational cost of calculating the inverse of the Hessian is common in other large model pruning approaches.**

We believe there is a misunderstanding in our contribution. 
Our contribution is not on layer-wise pruning itself. As described in the second paragraph of Section 3.2, we clarify the critical challenges in pruning large vision-language multi-modal models, and suggest a simple yet powerful pruning approach using the **memory-efficient zeroth-order optimization to estimate dynamic pruning ratio per layer with minimal computations**. Our EcoFLaP successfully mitigates severe performance drops even with a high degree of model compression (Figure 4), by overcoming the limitations of general layer-wise pruning methods.

Reviewer comment (Experiments):
The assumption of having sufficient GPU resources in the pruning scenario is not adequately discussed.

Author response:
> **C4. The assumption of having sufficient GPU resources in the pruning scenario is not adequately discussed.**

In Table 1, we showed that ECoFLaP only uses slightly more GPU memory (0.06GB) than Wanda and even less memory than SparseGPT. Furthermore, ECoFLaP also only uses 0.5GB more memory than Magnitude Pruning (which directly uses weight magnitude as the importance score and does not require any computation). We believe that the efficiency of ECoFLaP makes it a strong method as layer-wise pruning in the low computational cost regime.

Reviewer comment (Experiments):
The use of calibration datasets in experiments raises concerns about fairness and the significance of employing zeroth-order optimization to save memory if calibration datasets are used for fine-tuning.

Author response:
> **C6. The use of calibration datasets in experiments raises concerns about fairness and the significance of employing zeroth-order optimization to save memory if calibration datasets are used for fine-tuning.**

There is a clear misunderstanding of the basics of one-shot pruning. We first note that using small calibration data is a standard approach for one-shot (layer-wise) pruning [2,3,4]. And we also followed the same setting for our approach and didn’t re-train the model using the calibration set after pruning. 

[2]  Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, S., and Soudry, D. Accelerated sparse neural training: A provable and efficient method to find N:M transposable masks.

[3] Elias Frantar, Dan Alistarh, SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot

[4] Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter, A Simple and Effective Pruning Approach for Large Language Models

Reviewer comment (Theory):
that finds adaptive sparsity per layer by leveraging the global importance score approximated via first-order gradients. (should be zeroth-order gradients)

Author response:
> **C7. that finds adaptive sparsity per layer by leveraging the global importance score approximated via first-order gradients. (should be zeroth-order gradients)**

Thanks for catching the typo. We have reflected this in the revision.

Reviewer comment (Writing):
Note that our proposed method is computationally efficient by leveraging the first-order gradient to obtain a global importance score without Hessian operations (should be zeroth-order gradients)

Author response:
> **C8. Note that our proposed method is computationally efficient by leveraging the first-order gradient to obtain a global importance score without Hessian operations (should be zeroth-order gradients)**

This part is not a typo. We first show another version of our approach with the first-order gradient (we also show the result of this version in Table 1), which is cheaper than the second-order Hessian method, and then propose a more efficient version with the zeroth-order gradient.