Paper: Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
on-Existent Discussion on Limitations: While the paper does provide a robust set of experiments and results, there is a lack of discussion regarding the limitations of the proposed methods. Understanding the boundaries of the method's applicability is crucial for both academic and industrial settings.

Author response:
**Response**: We appreciate your constructive suggestion. We add a discussion section and put it in the appendix section A.2 due to space constraints. Here is a shorter version:

Due to computational constraints, we have not yet evaluated Compresso on larger models like the 33B and 65B. However, Compresso is highly generalizable and can be scaled up to larger models. The effectiveness of our method is influenced by the quality of the pruning dataset. Currently, we use an instruction tuning dataset (GPT4alpaca) for pruning, due to the inaccessible and resource-intensive nature of original pre-training datasets for LLMs. While it's promising, but the use of instruction tuning dataset as pruning data is not the optimal. 
For instance, when pruning at higher sparsity ratios, we observe a noticeable performance decline.  Addressing this issue typically requires the original pre-training dataset to compenstate for the information loss of the pruned LLMs. As such, the generation of a high-quality pruning dataset that closely aligns the distribution of original pre-training dataset is a crucial consideration for further LLM pruning research.

Reviewer comment (Reproducibility):
clearly description of which parts are pruned are expected.

Author response:
**Response**: Thank you for your valuable suggestion. As shown in the Figure 3 of the original paper (now we move to Appendix due to space limit), we present the layer-wise remaining ratios of attention heads and FFN intermediate size, and compare it to LLM-Pruner. This figure demonstrates that Compresso learns different pruned structures compared to LLM-Pruner that adopts the uniform-sparsity setting. Specifically, (i) Compresso automatically learns a different layer-wise sparsity ratio; (ii) Compresso tends to preserve more heads in the first and middle layers, while it prunes more heads in the final layers. 

To provide more details on the pruned architectures, we list out the detailed model configurations in the following table (we also add a section in the Appendix section in the updated pdf). From the table, we can observe several interesting points:

1):  Across all three pruned models, {\sysname} exhibits a tendency to retain more heads in the first and middle layers, while pruning more heads in the final layers. This tendency, although less evident in the FFN intermediate, continues to persist.  Notably, in the 5.4B model, the heads and FFN intermediate size of certain middle layers are completely preserved.

2): Compresso generally prunes slightly more heads than FFN layers.

3): Only a few hidden dimensions are pruned.

| Model | Hidden     | Layer      | \# Heads | \# Intermediate|
| :--:   | :--:    |:--:        |:--:        |
| LLaMA-7B | 4096 | | 32| 11008|
|Compresso-5.4B| 4042 | Layer 1-8: |19, 16, 32, 32, 21, 32, 32, 32|7594, 7582, 11008, 11008, 7366, 11008, 11008, 11008|
| | |  Layer 9-16: |32, 32, 32, 30, 27, 32, 32, 32 | 11008, 11008, 11008, 7298, 7514, 11008, 11008, 11008 |
| | | Layer 17-24: | 27, 32, 32, 32, 26, 32, 6, 32 | 7514, 11008, 11008, 11008, 7546, 11008, 7931, 11008|
|||Layer 25-32: | 4, 10, 8, 6, 9, 8, 8, 8 | 7943, 7899, 7843, 7731, 7899, 7896, 7978, 8116|
|Compresso-5.4B| 4060 |  Layer 1-8: | 21, 22, 27, 25, 28, 25, 29, 24 | 8072, 8161, 8141, 7993, 7965, 7989, 7959, 7903|
||| Layer 9-16: | 26, 29, 30, 30, 30, 26, 31, 31 | 8001, 7933, 7862, 8055, 8133, 8278, 8260, 8391|
|||Layer 17-24: | 29, 28, 29, 25, 26, 22, 16, 19 | 8429, 8447, 8412, 8492, 8400, 8408, 8360, 8365|
||| Layer 25-32: | 19, 16, 16, 11,16, 14, 12, 13 | 8277, 8235, 8155, 8120, 8153, 8173, 8184, 8255|
|Compresso-4.5B| 4044|   Layer 1-8: | 14, 19, 24, 15, 17, 18, 15, 17| 8136, 8184, 8145, 7942, 7889, 7890, 7935, 7886|
|||Layer 9-16: | 22, 22, 22, 20, 26, 26, 26, 22|8014, 7970, 7979, 8067, 8237, 8266, 8279, 8351|
|||Layer 17-24: | 22, 24, 22, 17, 17, 14, 8, 11|8401, 8433, 8415, 8466, 8446, 8391, 8324, 8312|
|||Layer 25-32:| 5, 6, 7, 4, 9, 10, 7, 10|  8228, 8169, 8078, 8083, 8047, 8115, 8159, 8240|

Reviewer comment (Experiments):
if the choice of dataset for the pruning process can greatly impact the effectiveness of the technique as the ablation part indicates in this manuscript, then is the pruning still meaningful in LLMs, given that the instruction tuning dataset might always be insufficient?

Author response:
**Response**: Thank you for your insightful observation and question regarding the potential impact of pruning dataset choice on the effectiveness of our pruning technique. We appreciate your keen interest in understanding the significance of LLM pruning in the context of dataset-related challenges. To answer your questions, we would like to clarify the following points:

1. We acknowledge that the instruction tuning dataset, GPT4Alpaca, does not encompass all possible tasks. Despite this limitation, we opt for LLM pruning based on this dataset and we found that **the pruned models still exhibit a remarkable preservation of the original LLM's generalization ability across various tasks**. For example, in MMLU and BBH benchmarks, we identified 6 specific subtasks that are not covered in topics/domains from GPT4Alpaca. These tasks spanned topics and contexts beyond the dataset's coverage. Even in these uncovered tasks, our pruned 5.4B model maintains an average of 97.8% of the original LLaMA-7B's performance. The results demonstrate the robustness and generalizability of our pruned LLMs by the instruction tuning dataset. 

| Model | MMLU: philosophy| MMLU: prehistory     | MMLU: professional_psychology | BBH: penguins_in_a_table| BBH: dyck_languages| BBH: date_understanding|
| :--:   | :--:    |:--:        |:--:        |:--:        |:--:        |:--:        |
|LLaMA-7B| 45.0| 39.8| 34.2| 25.87| 23.88|26.31|
|Compresso-5.4B| 37.3| 36.4| 32.7| 24.48| 36.03| 23.89|


2. The primary reason we chose to use the instruction tuning dataset is due to the inaccessibility of the pre-training dataset and the substantial training resources required even if it were released. Despite its distribution differing from the pre-training data, the instruction tuning dataset is designed to better align with human intents. Some studies[1,2] have shown that fine-tuning pre-trained LLMs with it still exhibits strong cross-ability generalization on unseen data. Our above experiment results further validate this point. However, we acknowledge that the GPT4alpaca dataset we are currently using is not optimal. As discussed in our limitations and discussions (Appendix A.2), when pruning at higher sparsity ratios, the pruned LLM experiences a noticeable drop in performance. In such cases, a dataset with a distribution closer to the original pre-trained dataset is needed to recover the information loss. 

3. Our method, Compresso, and particularly the collaborative pruning paradigm, **is not limited to just the instruction tuning dataset. It can be applied to pre-trained text data**. For instance, in our ablation study, we used pure text data generated by the method proposed by LLM-QAT. Although the quality of the text data generated by LLaMA-7B is some problematic and its pruning performance is not as good as GPT4alpaca, it still outperforms other methods such as SparseGPT (structured), Wanda (structured), and LLM-Pruner as shown in the following Table. As for how to generate a high-quality dataset for efficient pruning, we leave it as future work.

| Method (5.4B) | Commonsense reasoning (0-shot) | Reading comprehension (0-shot) | MMLU (5-shot) | BBH (3-shot)|
| :--:   | :--:    |:--:        |:--:        |:--:        |
|SparseGPT|58.12 | 54.32| 26.80|27.83 |
|Wanda |58.06 | 55.54| 15.65| 16.08|
|LLM-Pruner| **59.14**|48.92 |24.86 |28.97 |
|Compresso with LLM-QAT generated data| 58.62|**55.18** |**27.89** |**29.65** |

[1] Dynamics of instruction tuning: each ability of large language models has its own growth pace,  https://arxiv.org/pdf/2310.19651.pdf

[2] Instruction Tuning with GPT4, https://arxiv.org/pdf/2304.03277.pdf

Reviewer comment (Experiments):
clearly description of which parts are pruned are expected. And how much acceleration benefit can be derived from the proposed pruning algorithm?

Author response:
**Response**: Thank you for your valuable suggestion. As shown in the Figure 3 of the original paper (now we move to Appendix due to space limit), we present the layer-wise remaining ratios of attention heads and FFN intermediate size, and compare it to LLM-Pruner. This figure demonstrates that Compresso learns different pruned structures compared to LLM-Pruner that adopts the uniform-sparsity setting. Specifically, (i) Compresso automatically learns a different layer-wise sparsity ratio; (ii) Compresso tends to preserve more heads in the first and middle layers, while it prunes more heads in the final layers. 

To provide more details on the pruned architectures, we list out the detailed model configurations in the following table (we also add a section in the Appendix section in the updated pdf). From the table, we can observe several interesting points:

1):  Across all three pruned models, {\sysname} exhibits a tendency to retain more heads in the first and middle layers, while pruning more heads in the final layers. This tendency, although less evident in the FFN intermediate, continues to persist.  Notably, in the 5.4B model, the heads and FFN intermediate size of certain middle layers are completely preserved.

2): Compresso generally prunes slightly more heads than FFN layers.

3): Only a few hidden dimensions are pruned.

| Model | Hidden     | Layer      | \# Heads | \# Intermediate|
| :--:   | :--:    |:--:        |:--:        |
| LLaMA-7B | 4096 | | 32| 11008|
|Compresso-5.4B| 4042 | Layer 1-8: |19, 16, 32, 32, 21, 32, 32, 32|7594, 7582, 11008, 11008, 7366, 11008, 11008, 11008|
| | |  Layer 9-16: |32, 32, 32, 30, 27, 32, 32, 32 | 11008, 11008, 11008, 7298, 7514, 11008, 11008, 11008 |
| | | Layer 17-24: | 27, 32, 32, 32, 26, 32, 6, 32 | 7514, 11008, 11008, 11008, 7546, 11008, 7931, 11008|
|||Layer 25-32: | 4, 10, 8, 6, 9, 8, 8, 8 | 7943, 7899, 7843, 7731, 7899, 7896, 7978, 8116|
|Compresso-5.4B| 4060 |  Layer 1-8: | 21, 22, 27, 25, 28, 25, 29, 24 | 8072, 8161, 8141, 7993, 7965, 7989, 7959, 7903|
||| Layer 9-16: | 26, 29, 30, 30, 30, 26, 31, 31 | 8001, 7933, 7862, 8055, 8133, 8278, 8260, 8391|
|||Layer 17-24: | 29, 28, 29, 25, 26, 22, 16, 19 | 8429, 8447, 8412, 8492, 8400, 8408, 8360, 8365|
||| Layer 25-32: | 19, 16, 16, 11,16, 14, 12, 13 | 8277, 8235, 8155, 8120, 8153, 8173, 8184, 8255|
|Compresso-4.5B| 4044|   Layer 1-8: | 14, 19, 24, 15, 17, 18, 15, 17| 8136, 8184, 8145, 7942, 7889, 7890, 7935, 7886|
|||Layer 9-16: | 22, 22, 22, 20, 26, 26, 26, 22|8014, 7970, 7979, 8067, 8237, 8266, 8279, 8351|
|||Layer 17-24: | 22, 24, 22, 17, 17, 14, 8, 11|8401, 8433, 8415, 8466, 8446, 8391, 8324, 8312|
|||Layer 25-32:| 5, 6, 7, 4, 9, 10, 7, 10|  8228, 8169, 8078, 8083, 8047, 8115, 8159, 8240|

Reviewer comment (Novelty):
The work lacks novelty as many parts are taken from existing works such as LoRA and differentiable masks.

Author response:
> The work lacks novelty as many parts are taken from existing works such as LoRA and differentiable masks.

**Response**: We thank you for your comment and appreciate the opportunity to clarify our contributions: 
 1. Our primary contributions are the introduction of the collaborative pruning LLM paradigm and the associated pruning prompt. Unlike conventional approaches treating LLMs passively, our paradigm makes the LLM an active participant in the pruning process, resulting in a substantial performance improvement. The behind insight is that recent state-of-the-art LLMs already have the capability to understand and adapt to complex operational instructions (i.e., encourage the LLM itself better adapt to the pruning process). We are pleased to note that Reviewers dFpf and VBbn have acknowledged and appreciated the novelty of this paradigm.
 2. Besides, our structured pruning algorithm is not a simple combination of existing techniques. In contrast, every design choice is purposeful and aimed at addressing specific challenges. For instance, regarding LoRA, typically applied in fine-tuning and instruction tuning, we innovatively integrate it into the pruning process. This integration allows us to significantly reduce the resources required for training-based pruning while maintaining promising pruning performance.  The use of differentiable masks in our pruning algorithm aligns perfectly with our proposed collaborative pruning prompt. This is a departure from existing pruning algorithms that rely on weight importance metrics to determine which weights to prune, a process that presents challenges when integrating with our collaborative pruning prompt. This optimally leverages the capabilities of both the pruning algorithm and the LLM itself to learn the optimal pruning decisions. 

Thank you for your consideration, and we look forward to your feedback.

Reviewer comment (Experiments):
The proposed collaborative prompt is interesting, but the empirical study fails to support it convincingly. The ablation study in Table 6 is inconsistent, and it is recommended to perform the Compresso without prompting on both training and inference. Additionally, the lack of analysis or discussion on collaborative prompts prevents readers from fully understanding it.

Author response:
> The ablation study in Table 6 is inconsistent, and it is recommended to perform the Compresso without prompting on both training and inference.  More analysis or discussions for pruning prompt. 

**Response (1/2)**: Thank you for your constructive suggestion. We have corrected the typos in Table 6 for consistency. Additionally, following your suggestion, we conduct an experiment to further evaluate the effectiveness of Compresso, where the pruning prompt is removed from the entire process, including training, fine-tuning and inference. The Table below presents the performance of 5.4B pruned LLMs under different settings. 

|Task| Compresso with Prompting | No Prompt in Inference| No Prompt in Training| *No Prompt in Training and Inference*| 
| :--:   | :--:    |:--:        |:--:        | :--:    |
|Commonsense Reasoning (0-shot) |60.09 | 56.07|57.36  | 57.09 |
|Reading Comprehension (0-shot) |60.35 |56.52 |56.62 | 55.84|
|MMLU (5-shot)| 31.90| 31.49|29.30 |29.10  |
|BBH (3-shot) | 31.47|30.57 | 28.08| 28.64| 

We observe three key findings as follows:

1) The results align with the original paper, indicating that removing the pruning prompt at any stage reduces the performance of pruned LLMs. 
2) We found that the impact of pruning prompt at different stages varies across benchmarks. For instance,  removing the pruning prompt at any stage affects zero-shot commonsense reasoning and reading comprehension. However, removing the pruning prompt during training has a greater impact on MMLU and BBH than during inference.
3) Even if the pruning prompt is removed from training, reintroducing it during inference can still improve the performance of tasks such as commonsense reasoning, reading comprehension, and MMLU, while it may decrease performance on BBH.

In summary, the additional ablation study further validate the effectiveness of our collaborative pruning paradigm. The experiment results highlight the significant role of our collaborative pruning prompt during the training phase. Specifically, the integration of the pruning prompt during the training-based pruning process is pivotal, fostering a collaborative learning approach where both the LLM and the pruning algorithm jointly optimize for better pruning decisions. 

**Response (2/2)**: Moreover, we would like to share more empirical experiments to validate the effectiveness of our pruning prompt. Following the suggestion by Reviewer dFpf, we broaden our experimental scope to include other sparse patterns such as unstructured and N:M sparse patterns. **We demonstrate that our pruning prompt can also improve the pruning performance under unstructured and N:M sparsity patterns**.

We utilize SparseGPT[1] for the experiment, which is a representative method for unstructured pruning and n:m sparsity. Specifically, we prune LLaMA-7B under a 50% sparsity ratio for unstructured pruning and under n:m (2:4) sparsity using SparseGPT. We then evaluate the pruned model under conditions with and without our pruning prompt for zero-shot commonsense reasoning and reading comprehension. As shown in the tables below, our pruning prompt significantly improves the performance on all benchmarks and tasks. 

|Model | Avg | StoryCloze| PIQA| HellaSwag| WinoGrande| ARC-e|ARC-c|OBQA|
| :--:   | :--:    |:--:        |:--:        |:--:        |:--:        |:--:        |:--:        |:--:        |
|2:4 sparsity | 51.46| 75.31| 70.18| 45.52| 58.33| 55.85| 31.4| 23.6|
|2:4 sparsity + our prompt| **53.97**| **76.16**| **71.0**| **47.56**| **60.85**| **59.3**| **33.7**| **29.2**| 
|unstructured (50% sparsity)| 54.59| 79.10| 71.93| 48.87| 62.04| 60.44| 34.13| 25.6|
|unstructured + our prompt (50% sparsity)|**57.25**|**79.69**| **73.12**| **50.58**| **64.33**| **66.2**|**36.86**|**30.0**| 

|Model | Avg | BoolQ| RACE-High|
| :--:   | :--:    |:--:        |:--:        |
|2:4 sparsity| 51.59| 67.00| 36.17|
|2:4 sparsity + our prompt| **53.55**| **69.11**| **37.99**|
|unstructured (50% sparsity)| 52.16| 67.19| 37.13|
|unstructured + our prompt (50 sparsity)| **54.09**| **70.95**| **37.22**|

[1] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Reviewer comment (Experiments):
The main experiment only includes one baseline, which makes the results less convincing.

Author response:
> Add more baselines 

**Response**: We are grateful for your suggestion to include more baselines for comparison. However, due to constraints such as the absence of open-source code or the requirement of substantial GPU resources, we have selected two one-shot baselines, SparseGPT[1] and Wanda [2], for comparison. They are originally designed for unstructured pruning, and can be extended for N:M sparsity. We further extend them to structured pruning, allowing the removal of entire attention heads and FFN intermediate size. All other settings follow the original papers. 

The results of our comparison are presented in the table below. More detailed numbers can be found in the updated PDF. Interestingly, we found that while SparseGPT and Wanda outperform LLM-Pruner on three of the benchmarks, they fall short on commonsense reasoning. Wanda performs better than SparseGPT at lower sparsity levels, but experiences a significant drop in accuracy at higher sparsity ratios (4.5B). One possible explanation is that Wanda relies on the activation and weights to conduct direct weight pruning, unlike SparseGPT which is dependent on second-order Hessian inverses and includes weight updates on pruned LLMs to recover information loss.

In contrast, our method significantly outperforms these baselines across all benchmarks, suggesting that our method is the most effective at preserving the generalization capability of the original LLaMA-7B. 

| Model | Method    | Commonsense Reasoning (0-shot)      | Reading Comprehension (0-shot) | MMLU (5-shot) | BBH (3-shot)|
| :--:   | :--:    |:--:        |:--:        |:--:        |:--:        |
|5.4B| *SparseGPT* | 58.12|54.32 |26.80 | 27.83|
| | *Wanda*| 58.06|55.54 | 24.50|28.96 |
| | LLM-Pruner | 59.14|48.92 | 24.86| 28.97|
| | **Compresso** | **60.09**|**60.35**| **31.90**|**31.47**|
|5.0B| *SparseGPT* |55.18 |51.35 |25.21 |29.40 |
| | *Wanda*|54.45 | 51.43| 23.32|24.86 |
| | LLM-Pruner | 56.37| 48.93| 23.22| 26.46|
| | **Compresso** |**57.05** |**56.58** |**27.68** |**31.27**|
|4.5B| *SparseGPT* |52.27 |50.39 | 22.30| 26.61|
| | *Wanda*|  34.33| 39.47| 15.65|16.08|
| | LLM-Pruner | 53.73| 47.70| 23.85|24.67 |
| | **Compresso** |**55.94** |**52.52**|**25.92**|**28.25** |

[1] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

[2] A Simple and Effective Pruning Approach for Large Language Models

Reviewer comment (Evaluation):
There needs to be more explanation or discussion of the performance improvement without post fine-tuning in Table 7.

Author response:
> There needs to be more explanation or discussion of the performance improvement without post fine-tuning in Table 7.

**Response**: Thank you for your valuable suggestion. Upon delving into the detailed numbers, we found that the slight performance decrease for the 4.5B model after fine-tuning is due to the less significant improvement on our proposed collaborative pruning prompt. As shown in the table below, we list out the BBH scores both with and without the collaborative pruning prompt. *The numbers in ( ) denote the BBH scores without the collaborative pruning prompt*. We observed that the average BBH, BBH on NLP, and BBH on algorithmic all improve after fine-tuning without the collaborative prompt, which aligns with common practice.  However, after fine-tuning, the score of BBH on algorithmic with the prompt only improves by 0.97, while it originally improves by 3.81 before the fine-tuning. 

| 4.5B on BBH | BBH Avg     | NLP      | Algorithmic | 
| :--:   | :--:    |:--:        |:--:        |
| No Fine-tuning | 29.56 (27.65)| 32.29 (32.90)| 27.37 (23.56)|
| After Fine-tuning| 28.25 (27.71) |32.62 (32.50) | 24.75 (23.78)|

Reviewer comment (Theory):
Demonstrating the theoretical underpinnings of Compresso would greatly enhance the paper.

Author response:
> Demonstrating the theoretical underpinnings of Compresso would greatly enhance the paper

**Response**: Thanks for your suggestion! We acknowledge that theoretical analysis can provide deeper insight into our proposed pruning process. However, the behavior of LLMs remains unclear and is still an open question in the literature, making it challenging to conduct a theoretical analysis of our proposed pruning paradigm, especially regarding why our collaborative pruning prompt works. Instead, we conduct extensive experiments to demonstrate the effectiveness and generalizability of our approach. Our experiments span diverse datasets and multiple models under different sparsity ratios (new results on LLaMA13b can be found in the Appendix A.1), and show very promising results. We leave the theoretical analysis as the future works.

Reviewer comment (Novelty):
This paper claims that LLM-Pruner is the only LLM structured pruning work. However, this claim appears to be open to question. The related works [1-7] that structurally prune LLMs are not mentioned or compared.

Author response:
> Compare Compresso with related works

**Response (1/2): Clarifications on the listed related works**: 
We greatly appreciate your insightful comments and the opportunity to clarify our work in relation to the cited works. 

Firstly, we would like to address the timing of the publication of some of LLM structured pruning works, such as [2][7]. These papers were published on Arxiv after ICLR deadline, specifically after October 10. At the time of our paper's submission, we were not aware of any other structured pruning works on state-of-the-art LLMs like LLaMA, apart from LLM-pruner. Consequently, we stated in our paper that LLM-pruner was the only LLM structured pruning work. We have revised this statement in the updated PDF and added the appropriate citations. 

Although [2][7] are concurrent works, we would like to highlight the differences between our work and [2][7] as follows:
Sheared LLaMA [2] utilizes RedPajama, a replicated pretraining dataset for LLaMA with 50B tokens, for both pruning and fine-tuning. However, it requires 16 A100s with 80GB of memory[2], which may not be feasible for public users. In contrast, our method uses an instruction tuning dataset with 52k instances and integrates LoRA in pruning. This allows for experiments on 4x 32GB V100s, a more efficient and practical approach acknowledged by Reviewer VBbn. Apart from its pruning dataset, the pruning algorithm of Sheared LLaMA does not present much novelty, as it only introduces minor modifications to the original CoFiPruning[8]. In contrast, our method proposes a new learning-based structured pruning paradigm, which leverages the LLM itself to conduct collaborative pruning, this differentiates our work from traditional pruning works. 

LoRAShear[7] is a one-shot structured pruning work. It initially creates dependency graphs over LoRA modules and analyzes the knowledge distribution. It then iteratively prunes based on this distribution. However, due to the unavailability of its source code and the limited time for rebuttal, we have not included it in our comparison.

Secondly, some of the related works you listed, like [4][5], are semi-structured pruning (n:m sparsity) that requires specified inference runtime and hardware for acceleration, while our focus is entirely on structured pruning that directly removes individual attention heads and reduces FFN intermediate size. [1] is specifically designed for dynamically pruning the structure of graph data, not on the pre-trained language model, while [6] is for BERT, not LLM. [3] only evaluated on BERT and GPT/GPT2. And there is no available open-source code for [3] and [6].

We hope this clarifies our position and the unique contributions of our work. We look forward to further discussions and feedback.

[1] Graph-to-Text Generation with Dynamic Structure Pruning

[2] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

[3] ZipLM: Inference-Aware Structured Pruning of Language Models

[4] Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs

[5] A Simple and Effective Pruning Approach for Large Language Models

[6] Gradient-Free Structured Pruning with Unlabeled Data

[7] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. 

[8] CoFiPruning: Structured Pruning Learns Compact and Accurate Models

Reviewer comment (Experiments):
Is it possible to compare Compresso with the related works (mentioned above)?

Author response:
> Compare Compresso with related works

**Response (2/2): Comparision results with two new baselines**:  We are grateful for your suggestion to include more baselines for comparison. However, due to constraints such as the absence of open-source code or the requirement of substantial GPU resources, we have selected two one-shot baselines, SparseGPT and Wanda [5], for comparison. They are originally designed for unstructured pruning and can be extended to N:M sparsity. We further extend them to structured pruning, allowing the removal of entire attention heads and FFN intermediate size. All other settings follow the original papers. 

The results of our comparison are presented in the Table below. More detailed numbers can be found in the updated PDF. Interestingly, we found that while SparseGPT and Wanda outperform LLM-Pruner on three of the benchmarks, they fall short on commonsense reasoning. Wanda performs better than SparseGPT at lower sparsity levels, but experiences a significant drop in accuracy at higher sparsity ratios (4.5B). One possible explanation is that Wanda relies on the activation and weights to conduct direct weight pruning, unlike SparseGPT which is dependent on second-order Hessian inverses and includes weight updates on pruned LLMs to recover information loss.

In contrast, our method significantly outperforms these baselines across all benchmarks, suggesting that our method is the most effective at preserving the generalization capability of the original LLaMA-7B. 

| Model | Method    | Commonsense Reasoning (0-shot)      | Reading Comprehension (0-shot) | MMLU (5-shot) | BBH (3-shot)|
| :--:   | :--:    |:--:        |:--:        |:--:        |:--:        |
|5.4B| *SparseGPT* | 58.12|54.32 |26.80 | 27.83|
| | *Wanda*| 58.06|55.54 | 24.50|28.96 |
| | LLM-Pruner | 59.14|48.92 | 24.86| 28.97|
| | **Compresso** | **60.09**|**60.35**| **31.90**|**31.47**|
|5.0B| *SparseGPT* |55.18 |51.35 |25.21 |29.40 |
| | *Wanda*|54.45 | 51.43| 23.32|24.86 |
| | LLM-Pruner | 56.37| 48.93| 23.22| 26.46|
| | **Compresso** |**57.05** |**56.58** |**27.68** |**31.27**|
|4.5B| *SparseGPT* |52.27 |50.39 | 22.30| 26.61|
| | *Wanda*|  34.33| 39.47| 15.65|16.08|
| | LLM-Pruner | 53.73| 47.70| 23.85|24.67 |
| | **Compresso** |**55.94** |**52.52**|**25.92**|**28.25** |

Reviewer comment (Novelty):
In Section 3.3, the authors claim that this paper introduces LoRA into pruning in a novel manner. Can you explain the main difference between LoRA used in fine-tuning LLMs and the way used in pruning in this paper?

Author response:
> Can you explain the main difference between LoRA used in fine-tuning LLMs and the way used in pruning?

**Response**: We appreciate your attention and have updated the PDF to address any confusion. Our initial goal was to emphansize that our method of optimizing differentiable masks naturally integrates with LoRA, as shown in equation (2). The implementation of LoRA used in fine-tuning and the way used in pruning in this paper are very similar, with the only differences being that both masks and LoRA modules are optimized together during pruning. 

However, we would like to emphasize that in the context of structured pruning based on training, traditional pruning methods and the recent Sheared LLaMA all require full gradient updates on the model parameters, which necessitate substantial training and GPU resources. By using LoRA to allow the update of a few model parameters during pruning, we can significantly reduce the training costs. For instance, we are able to run LLama13b on 4 V100 GPUs. However, this may also impact the pruning performance as only a few model parameters are allowed to update. Our Compresso demonstrates that it achieves superior performance despite these constraints.

Reviewer comment (Experiments):
Although Compresso is a one-shot pruning method, can the performance of LLMs be improved if we do Compresso iteratively?

Author response:
> Although Compresso is a one-shot pruning method, can the performance of LLMs be improved if we do Compresso iteratively?

**Response**: Thank you for your thoughtful consideration and question! I appreciate your question, but there seems to be slight misunderstanding about Compresso. Contrary to the assumption that it is a one-shot pruning method, Compresso employs a collaborative pruning paradigm and differentiable masks. This allows it to learn optimal pruning decisions, making it more than a traditional one-shot method. I hope this clarification provides a clearer understanding of Compresso and its capabilities. Please let us know if you have any further questions.

Reviewer comment (Experiments):
How to adjust the hyperparameters λ1 and λ2 using the AdamW optimizer? Can authors provide more details or an ablation study about that?

Author response:
> How to adjust the hyperparameters λ1 and λ2 using the AdamW optimizer? Can authors provide more details or an ablation study about that?

**Response**: Thank you for the question! We have two AdamW optimizers in our pruning process. The first optimizer is used for optimizing the inserted LoRA modules for updating model parameters, while the second is used to learn differentiable masks. Specifically, λ1 and λ2 are optimized in conjunction with the second AdamW optimizer and the masks. These parameters, as gradient-enabled NN.parameters, are passed to the second AdamW optimizer. Given our initial reglr=0.05, they work together with the first AdamW optimizer to learn the best pruning decision. The value of 'reglr' can impact λ1, λ2, and the overall pruning performance. Specifically, a larger 'reglr' tends to optimize the Lagrangian loss in Equation 5, ensuring that the model sparsity of the pruned LLM strictly reaches the target sparsity ratio during the entire training process. Conversely, a smaller 'reglr' makes the pruning process more inclined to optimize the model prediction loss, but it may not guarantee a well-aligned sparsity ratio during the entire pruning process.

To validate this, we conduct an ablation study. As shown in the table below, in our original paper's experiment, we empirically set 'reglr' to 0.05. Now we chose a larger 'reglr' of 0.1 and a slightly smaller 'reglr' of 0.02. These three different 'reglr' values all can ensure that the model sparsity reaches the target sparsity ratio after pruning. However, both a large and small 'reglr' can impact the pruning performance. In our experiments, we empirically set 'reglr' to 0.05, and it performs well on both LLaMA-7B and 13B.


|λ1, λ2| Commonsense Reasoning (0-shot) | Reading Comprehension (0-shot)|
| :--:   | :--:    |:--:        |
|0.1 | 56.10|56.13 |
|0.05 (used in our experiments)| 60.09|60.35 |
|0.02 | 58.54 | 56.36 |