Paper: BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation (ICLR-2024)

Reviewer comment (Evaluation):
It would be better to involve the computational cost of attention weight in Table 4.

Author response:
**A1:** We acknowledge the importance of providing information on the computational cost of running our pruned model in ViTCoD. In response to your suggestion, we have revised Table 4 to include the FLOPs (floating-point operations) of the pruned model. The updated table now offers a more comprehensive view of the efficiency gains achieved by our method. Please refer to the revised table below.

| Layer name | q_proj | k_proj | v_proj | o_proj | gate_proj | up_proj | down_proj |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dense Runtime | 4096 | 4096 | 4096 | 4096 | 10128 | 10128 | 10128 |
| Average Sparse Runtime | 2232.31 | 2230.50 | 2720.59 | 2698.53 | 5207.53 | 5125.0 | 6850.03 |
| Dense mFLOPs | 16.78 | 16.78 | 16.78 | 16.78 | 45.09 | 45.09 | 45.09 |
| Average Sparse mFLOPs | 8.55 | 8.27 | 9.29 | 10.88 | 37.36 | 35.34 | 27.63 |
| Speedup (Runtime) | 1.83x | 1.84x | 1.51x | 1.52x | 1.94x | 1.98x | 1.48x |

Table 4: Runtime (cycles), computational cost (FLOPs), and speedup across various layers in LLaMA-7B. The term "FLOPs" refers to floating-point operations while "cycles" denotes the number of instruction cycles necessary for the ViTCoD accelerator to perform the associated computational workloads.

Reviewer comment (Experiments):
There has been a lot of research on pruning in CV and NLP. The baselines are far from complete, such as "The Lottery Ticket Hypothesis" and its following works. While considering not much work on LLM, I would not penalize too much on it.

Author response:
**A3:** Thanks for the suggestion. In response, we have conducted experiments using three additional pruning techniques commonly employed in computer vision and natural language processing. These methods, namely iterative pruning (proposed in the Lottery Ticket Hypothesis [A]), threshold pruning [B], and direct mask learning, have been implemented in a blockwise manner for a fair comparison. For iterative pruning, we employ a step-wise approach where we iteratively prune the LLM block with a step of 10%. During each iteration, we update the weights with the reconstruction loss. The results are reported in the table under "Iterative Prune". Then for threshold pruning, we learn a threshold value for each layer within the block while maintaining the block sparsity at 50%. The results are reported in the table under "Threshold Prune". For direct mask learning, we directly learn a sparse mask for the LLM block. The results are reported in the table under "Direct Mask Learning". We opt for iterative pruning as it stands out as one of the most representative methods for pruning, incorporating weight updating. Additionally, we select threshold pruning for its cutting-edge capabilities, showcasing state-of-the-art performance in computer vision models. Lastly, we choose direct mask learning as it represents the most straightforward and directly differentiable pruning method. Notably, the original “Iterative prune” and “Threshold prune” methods use the weight value as a measure of importance. In contrast, we provide the versions that use the weight or Wanda importance metric.

The perplexity results on WikiText2, C4, and PTB datasets are presented below:

|  | BESA | Iterative prune (Wanda) | Iterative prune (Weight) | Threshold prune (Wanda) | Threshold prune (Weight) | Direct mask learning |
| --- | --- | --- | --- | --- | --- | --- |
| WikiText2 ppl | 6.86 | 6.88 | 123.48 | 2984.61 | 20351.98 | 1457.53 |
| C4 ppl | 8.96 | 9.08 | 102.11 | 2191.59 | 21203.34 | 371.47 |
| PTB ppl | 66.96 | 57.71 | 3929.01 | 4600.38 | 32474.87 | 3927.83 |

These results underscore the unique advantages of our method, particularly its ability to achieve substantial reductions in perplexity. We believe that this thorough exploration of differentiable methods provides a comprehensive understanding of the strengths and capabilities of BESA compared to other baselines.

[A] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.

[B] Kusupati, Aditya and Ramanujan, Vivek and Somani, Raghav and Wortsman, Mitchell and Jain, Prateek and Kakade, Sham and Farhadi, Ali. Soft Threshold Weight Reparameterization for Learnable Sparsity. Proceedings of the International Conference on Machine Learning, 2020.

Reviewer comment (Evaluation):
The paper does not provide a detailed analysis of the trade-offs between different pruning rates and their impact on model performance, which could be useful for understanding the optimal pruning strategy.

Author response:
We apologize for any confusion caused by the placement of ablation studies in our paper. On page 7, in Section 4.2, we discussed and conducted experiments with varying sparsities, which are visually represented in Fig. 3. However, we acknowledge that the figure is presented on page 12 due to the limit of space. From Fig.3, we can see that BESA consistently outperforms other baselines such as SparseGPT and Wanda. The gap is particularly pronounced at large sparsities.

Reviewer comment (Writing):
In page 3 BLOCK-WISE PRUNING equation (1), it is better to add the meaning of “W” together with “M, F, X, …”.

Author response:
Sorry for any confusion caused by our paper. To clarify, "W" represents all linear weights in a transformer block, "M" refers to the learnable binary mask of these weights, "X" means the input tokens, and "F" is the mapping function that gets the forward result of given "W" and "X".

Reviewer comment (Experiments):
Since the article mentioned that existing methods require meticulous hyperparameter tuning, adding the sensitivity of some vital hyperparameters of the proposed model and clarifying the advantage in the appendix will make this paper more convincing.

Author response:
Thank you for pointing it out. We provide a comprehensive ablation study of the hyperparameters used in our method on page 12, Appendix Sec. A, which is inclusive of the size of the calibration set, epochs, the step of sparsity candidates, learning granularity, and importance metric.

Reviewer comment (Experiments):
Considering the abstract mentions existing methods require meticulous hyperparameter tuning, it would be better to study the vital hyperparameter’s sensitivity of the proposed method and experiments are needed, which will make this paper more convincing.

Author response:
Thank you for pointing it out. We provide a comprehensive ablation study of the hyperparameters used in our method on page 12, Appendix Sec. A, which is inclusive of the size of the calibration set, epochs, the step of sparsity candidates, learning granularity, and importance metric.