Paper: Enhancing One-Shot Pruned Generative Pre-training Language Models through Sparse-Dense-Sparse Mechanism (ICLR-2024)

Reviewer comment (Experiments):
**Comparing to stronger methods like wandb:** Wandb (Sun et al., 2023) is a method that performs better than SparseGPT on one-shot pruning, and the authors should introduce it as a baseline and add more discussions around it. I believe that the framework is independent from the base pruning method, thus doing experiments on top of a strong existing method is highly recommended.

Author response:
**Response to weakness 1**: Thanks so much for your suggestion, Wanda is a very suitable one-shot pruning method as our baseline. Therefore, we construct SDS optimization experiments with Wanda as the basic pruning method.

In the comparative analysis, SDS shows a significant improvement over Wanda, with an average perplexity of about 33.33, which is lower than Wanda's 42.79. In the downstream tasks, SDS also obtains an average performance improvement of 3.1% compared to Wanda. The above results illustrate that SDS enables the model performance to be improved.

In fact, the Wanda method is less effective than SparseGPT for pruning undersized (non-redundant / more fully trained) PLMs. On the one hand, the idea of Wanda to protect outlier activations is less effective on undersized PLMs with insignificant outliers. On the other hand, Wanda fails to compensate for the pruning error as SparseGPT although it can find an effective sparse mask.

A brief introduction to Wanda, experimental results, and more analysis can be found in **Section A.5**. *Additional comments are provided for your convenience to refer to the experimental results*.

Reviewer comment (Experiments):
**Performing experiments on stronger base models:** A clear trend that is disclosed by comparing SparseGPT and Wanda in Sun et al., 2023 is that the stronger the base model is, the more the pruning process hurts the performance of the model. For example, in the SparseGPT paper, pruning retains the model performance on OPT models; however, when Sun et al., 2023 evaluates on LLaMA based models, the performance degradation is way more significant. Such observations make intuitive sense, as the more stronger the base model is, the more information each parameter carries, and the more the model performance gets hurt when the parameter gets pruned. Given this observation, I suggest the authors test on stronger base models like LLaMA to give a more accurate account of how practical one-shot pruning is for real applications.

Author response:
**Response for weakness 2**: As you mentioned, LLaMA is a model that is difficult to compress, and constructing experiments using LLaMA as a base model is better for exploring the validity of SDS.

We take the 7b-sized models of LLaMA and LLaMA2 with 2:4 sparse as the pruning configuration and SparseGPT/Wanda as the base pruning method to verify the language modeling perplexity and multi-task performance of the SDS-optimized pruned models. 

Experimental results show an average perplexity reduction of approximately 10.90% and an accuracy increase of around 2.46%. These findings demonstrate the SDS's effectiveness in enhancing larger and stronger pruned PLMs' performance.

For the LLaMA model, the language comprehension perplexity is impaired on a doubling scale after one-shot pruning with SparseGPT or Wanda, proving that LLaMA is indeed a PLM that is difficult to compress. LLaMA, being trained on large-scale data, is considered a non-redundant model even though it already has quite a few parameters. The fact that pruning of redundant parameters is somewhat unachievable on LLaMA elevates the difficulty of model compression and the need to perform SDS optimization.

The detailed experimental results and analysis can be found in **Section A.5**. *Additional comments are provided for your convenience to refer to the experimental results*.

Reviewer comment (Evaluation):
**The extra step leads to diminishing returns in performance as the model scales up:** From table 2, it’s clear that as the model scales up and becomes stronger, the performance of the the re-dense and r-=prune process leads to minimal improvement compared to simply using one step of pruning.

Author response:
**Response for weakness 3**: As the volume of PLM grows, the difficulty and inadequacy of its training grows rapidly. In the case of OPT-2.7b, its redundancy will be much higher than that of the smaller version OPT-125m, such that the implementation of one-shot pruning alone will already ensure very little degradation of its performance. Taking the premise that the theoretical upper bound on the performance of the SDS-optimized pruned model is on par with its dense counterpart, the effectiveness and necessity of performing SDS optimization for a highly redundant PLM is limited. However, as training techniques and high-quality data continue to be upgraded, newly born large-scale PLMs will increasingly exhibit low-redundancy characteristics, such as those of LLaMA. The SDS optimization will show more valid superiority ahead.

Reviewer comment (Experiments):
Will performing the re-dense and re-prune process in multiple iterations further improve performance?

Author response:
**Response for question 1**:   Keeping the calibration set constant, and iteratively performing the sparse-dense-sparse optimization will no longer improve the performance of the pruned PLMs because of the overfitting phenomenon that arises. Better performance will result if different data is used for each iteration. Even so, we still recommend using only a single SDS to avoid potential overfitting phenomena. Specific experimental and analytical procedures can be found in **Section A.4**.

Reviewer comment (Evaluation):
Although the results on OPT models look okay on paper, I believe they are not enough to judge the practical relevance of the proposed method. First of all, how much additional flops are being incurred to prune the models in three stages? Secondly, the performance of OPT class of models on pre-training and various downstream tasks is itself not good. So, are the gains reported in the paper statistically significant, or they lie within the standard deviation of the performance of OPT models on these tasks.

Author response:
The process of running SDS is efficient and memory-friendly: (i) only the layer being optimized needs to be in GPU memory; (ii) optimization between layers is parallelizable, so the device's power can be fully utilized for speedups. The following table shows the runtime of SDS for single-device serial, multi-device (our implementation, on 8 32GB Nvidia V100 GPUs) parallel, and extreme parallel (theoretical) cases (see **Section A.5** for more analysis).

|type|125M|1.3B|2.7B|7B|
|--|--|--|--|--|
|serial|~22min|~3.8hours|~7.4hours|~34.2hours|
|parallel|~4min|~25min|~1hours|~4.5hours|
|theoretical|<1min|~6min|~8min|~39min|

To highlight the contrast in outcomes, we introduce additional experimental comparisons including GPT, LLaMA as the base model, and Wanda as the base pruning method, and calculated the average accuracy on all downstream tasks (**Table 6, Table 7, and Table 9 in Section A.5**). Experimental results demonstrate that SDS outperforms SparseGPT and Wanda by decreasing language comprehension perplexity by an average of 6.4 and achieving an average accuracy improvement of 1.9% across multiple downstream tasks. Thus, the SDS pruning method can be proven effective.

*Additional comments are provided for your convenience to refer to the experimental results*.

Reviewer comment (Novelty):
Sparsity and pruning research is more relevant for larger models to reduce their inference time and the GPU/TPU memory footprint. But the proposed method only matches SparseGPT's performance for the larger models. Is the further Dense-Sparse pruning even necessary?

Author response:
In the revised manuscript, we included experiments involving the larger PLM LLaMA. Our findings demonstrate that SDS improves LLaMA's accuracy by 2.5% when compared to SparseGPT/Wanda. However, as the model size increases, redundancy also grows, allowing SparseGPT to achieve sufficiently good results independently. In such cases, SDS may not be necessary. However, for more recent models like LLaMa, which outperform OPT at similar sizes—highlighting LLaMA's compactness—SDS can still effectively enhance the performance of pruned PLMs.

Reviewer comment (Experiments):
The paper should also report results on the speedup obtained compared to the dense and SparseGPT models with varying model size and sparsity category (50%, 2:4, 4:8).

Author response:
Thanks for your interest in practical inference speedups for our pruned PLMs. Our approach is aimed at optimizing the performance of pruned PLMs under 50%/2:4/4:8 sparse configurations. The real-time speedup is contingent upon the support provided by sparse acceleration libraries or hardware capabilities. An instance of this is demonstrated by DeepSparse [1], showcasing a round 20% end-to-end speedup achieved through 50% unstructured sparsity on CPU. Nvidia [2] also demonstrates that a 2:4 sparse GEMM achieves an ideal twofold performance increase compared to an equivalent dense GEMM, leveraging Sparse Tensor Cores on GPUs with the Ampere architecture. Currently, 4:8 sparse acceleration is not supported. However, the experimental results in the manuscript will hold valuable implications for guiding future hardware designs.

[1] DeepSparse: https://github.com/neuralmagic/deepsparse

[2] Mishra A, Latorre J A, Pool J, et al. Accelerating sparse deep neural networks[J]. 2021.

Reviewer comment (Writing):
Is there a typo in Algorithm 1 in section A.1? The main text of the paper mentions that it uses $W_{l}^{sparse-2nd}$ to collect $X_{l - 1}$ during forward propagation, but line 8 in the $\textbf{Second pruning: sparse weight adjustment}$ sub-algorithm mentions $W_{i}^{sparse}$ for forward propagation.

Author response:
Thank you for pointing out the error in our formula. We have corrected it as follows: $X\leftarrow{W}_i^{\text{sparse-2nd}}X$ in the revised manuscript. We confirm that this correction does not impact our results or conclusions.

Reviewer comment (Experiments):
I believe the authors use the same subset of unlabeled data during the second and third phases of SDS. Did the authors observe any difference in using a different sample during these two stages?

Author response:
You are correct that we use the same samples throughout each SDS step. We appreciate your interest in using different samples within SDS, and the corresponding result is detailed in **row 11 of Table 4**. The efficacy of employing different samples within SDS closely mirrors that of the standard SDS, with only two tasks surpassing it. In the short term, this may be caused by the failure to learn sufficiently about each batch of samples as well as the varying quality of the samples. This finding confirms the advantage that SDS does not need to rely on additional samples.

Reviewer comment (Writing):
Typo in Section 2.2, third last line: it should be unlabeled data and not labeled data, right? since only $X$ input is being used and intermediate outputs collected during forward propagation?

Author response:
Yes, you are right. We intended to convey that our method utilizes only a small amount of unlabeled data. Acknowledging that the original phrasing (We did not use a large amount of labeled data) may have caused confusion, we have reworded the expression to be directly affirmative. We believe this change maintains the accuracy of our presentation while improving readability.

Reviewer comment (Novelty):
It is unknown whether this could be valid for pruning large language models. For pruning small language models, there are already many solutions. I wonder the advantage of pruning.

Author response:
**Response to weakness 1**: We chose the Llama model to further validate the effectiveness of the SDS framework on larger and stronger PLMs. We found that when pruning the model using conventional pruning methods (SparseGPT/Wanda) incurs a large performance loss, SDS is effective in optimizing the pruned model. Below are the experimental results and analysis, you can also find them in the second part of Section A.5 (Page 19 in our updated manuscript). 

We take the 7b-sized models of LLaMA and LLaMA2 with 2:4 sparse as the pruning configuration and SparseGPT/Wanda as the base pruning method to verify the language comprehension perplexity and multi-task performance of the SDS-optimized pruned models, main results are shown below:

|PLM|Method|Wiki.↓|COPA↑|BookQ.↑|PIQA↑|RTE↑|Story.↑|Wino.↑|Avg(acc.)↑|
|--------------|-------------------|--------|-------|---------|-------|------|---------|--------|-------------|
|Llama-7b|Dense|5.68|84|42.4|77.48|53.43|75.94|67.01|66.71|
|Llama-7b|SparseGPT|11.23|80|35.4|70.62|59.57|69.89|59.98|62.58|
|Llama-7b|SDS-SparseGPT|9.97|79|35.6|71.65|59.57|71.67|61.80|63.22|
|Llama-7b|Wanda|11.54|80|36.4|68.88|58.12|69.00|60.06|62.08|
|Llama-7b|SDS-Wanda|9.73|82|36.8|70.57|57.60|70.27|62.04|63.21|
|Llama2-7b|Dense|5.47|87|40.8|77.04|61.73|77.59|67.01|68.53|
|Llama2-7b|SparseGPT|10.82|83|35.6|71.33|63.90|71.23|63.30|64.73|
|Llama2-7b|SDS-SparseGPT|10.02|84|37.2|71.44|64.09|71.99|63.88|65.43|
|Llama2-7b|Wanda|12.14|79|34.6|70.08|59.57|69.70|60.22|62.20|
|Llama2-7b|SDS-Wanda|10.39|81|36.8|70.62|64.25|71.80|63.06|64.59|

The above results highlight that SDS achieves an average perplexity reduction of approximately 10.90% and an accuracy increase of around 2.46% over other one-shot pruning methods. This substantial improvement underscores the effectiveness of SDS in enhancing the performance of larger and stronger pruned PLMs.

Reviewer comment (Experiments):
Why couldn't SDSDS or SDSD...SDS achieve better performance? In general, an iterative SDS framework seems a good idea. Any ideas to make it work and find how many iterations to get the saturated performance.

Author response:
**Response to question 1**: The iterative implementation of the sparse-dense-sparse optimization failed to achieve better performance for the model, which we attribute to the overfitting phenomenon caused by overlearning the same batch or distribution of samples. In fact, optimizing with different samples in each iteration can produce higher-performance models. At the same time, we also found that SDS benefited more from multiple samples than SparseGPT. 

Specific experiments and more analysis can be found in Section A.4 (Page 17 in our updated manuscript).

Reviewer comment (Experiments):
Can SDS also work for **large** language models? Any insight?

Author response:
**Response to weakness 1**: We chose the Llama model to further validate the effectiveness of the SDS framework on larger and stronger PLMs. We found that when pruning the model using conventional pruning methods (SparseGPT/Wanda) incurs a large performance loss, SDS is effective in optimizing the pruned model. Below are the experimental results and analysis, you can also find them in the second part of Section A.5 (Page 19 in our updated manuscript). 

We take the 7b-sized models of LLaMA and LLaMA2 with 2:4 sparse as the pruning configuration and SparseGPT/Wanda as the base pruning method to verify the language comprehension perplexity and multi-task performance of the SDS-optimized pruned models, main results are shown below:

|PLM|Method|Wiki.↓|COPA↑|BookQ.↑|PIQA↑|RTE↑|Story.↑|Wino.↑|Avg(acc.)↑|
|--------------|-------------------|--------|-------|---------|-------|------|---------|--------|-------------|
|Llama-7b|Dense|5.68|84|42.4|77.48|53.43|75.94|67.01|66.71|
|Llama-7b|SparseGPT|11.23|80|35.4|70.62|59.57|69.89|59.98|62.58|
|Llama-7b|SDS-SparseGPT|9.97|79|35.6|71.65|59.57|71.67|61.80|63.22|
|Llama-7b|Wanda|11.54|80|36.4|68.88|58.12|69.00|60.06|62.08|
|Llama-7b|SDS-Wanda|9.73|82|36.8|70.57|57.60|70.27|62.04|63.21|
|Llama2-7b|Dense|5.47|87|40.8|77.04|61.73|77.59|67.01|68.53|
|Llama2-7b|SparseGPT|10.82|83|35.6|71.33|63.90|71.23|63.30|64.73|
|Llama2-7b|SDS-SparseGPT|10.02|84|37.2|71.44|64.09|71.99|63.88|65.43|
|Llama2-7b|Wanda|12.14|79|34.6|70.08|59.57|69.70|60.22|62.20|
|Llama2-7b|SDS-Wanda|10.39|81|36.8|70.62|64.25|71.80|63.06|64.59|

The above results highlight that SDS achieves an average perplexity reduction of approximately 10.90% and an accuracy increase of around 2.46% over other one-shot pruning methods. This substantial improvement underscores the effectiveness of SDS in enhancing the performance of larger and stronger pruned PLMs.

**Response to question 2**: SDS significantly enhances the performance of pruned undersized PLMs (OPTs) and larger PLMs (Llama). These models are characterized by low redundancy and they can be fully trained or trained on a sufficient number of samples. Conversely, models with notable redundancy, often comprising inadequately trained massive PLMs, can already be pruned by SparseGPT and Wanda with minimal or no loss. Consequently, the necessity to apply SDS and its optimization diminishes in such cases. SDS excels at efficiently optimizing pruned non-redundant PLMs, regardless of the model's volume (even though non-redundant PLMs often appear as undersized ones). As training techniques and high-quality data continue to be upgraded, newly born large-scale PLMs will increasingly exhibit low-redundancy characteristics, such as those of LLaMA. The SDS optimization will show more valid superiority ahead.