Paper: The LLM Surgeon (ICLR-2024)

Reviewer comment (Experiments):
Since the authors adopt a multi-shot sparse approach, it would be beneficial to quantitatively compare the time costs and GPU memory consumption with SparseGPT.

Author response:
We agree with the reviewer that we should include a quantitative comparison between methods. Compared to other methods (e.g. SparseGPT), we are able to obtain better compression results but only at the expense of longer compression time. We will make sure this is absolutely clear from the main text. The table below shows compression time on Llama-v2 7B and 13B models. We will add these results to the paper appendices:

|           | SparseGPT  (baseline)           | Unstructured LLM-Surgeon (ours) |
| --------- | --------------------- | ------------------------ |
| Llama-7B  | <5m / 1 GPU (32GB)    | 2d8h16m / 4xH100 80 GB   |

|           | K-OBD   (baseline)              | Structured LLM-Surgeon (ours)   |
| --------- | --------------------- | ------------------------ |
| Llama-7B  | 16h58m / 4xH100 80 GB | 17h08m / 4xH100 80 GB    |
| Llama-13B | 1d6h5m / 8xH100 80 GB | 1d9h26m / 8xH100 80 GB   |

The method is highly parallelizable and more GPUs can be traded for a linear speed-up in time, if desired. Compression time measured using the required memory:

|                                       |Runtime|PPL 90%|PPL 80%|PPL 70%|PPL 60%|PPL 50%|
|---------------------------------------|-------|-------|-------|-------|-------|-------|
|Unstructured baseline (SparseGPT)      |**<5m**    |5.49   |5.58   |5.71   |5.94   |6.51   |
|Unstructured LLM Surgeon (ours)        |2d8h16m|**5.13**   |**5.20**   |**5.36**   |**5.66**   |**6.08**   |
|Structured baseline (K-OBD)            |**16h58m** |5.48   |9.14   |15.43  |28.03  |46.64  |
|Structured LLM Surgeon (ours)          |17h08m |**5.25**   |**6.18**   |**7.83**   |**10.39**  |**15.38**  |
|Structured baseline (K-OBD) Llama-13B  |**1d6h5m** |4.908  |6.294  |10.08  |13.06  |16.06  |
|Structured LLM Surgeon (ours) Llama-13B|1d9h26m|**4.692**  |**5.286**  |**6.207**  |**7.245**  |**9.428**  |

Further engineering efforts may further speed up compression, especially for unstructured pruning. As mentioned, our method does come with a higher computational cost compared to other approaches. Importantly, compression of LLMs in general only needs to happen once after which a pruned model can be deployed infinitely many times without further cost. This motivates our method which takes longer to run but provides much better final test performance.

Reviewer comment (Experiments):
Global rank ordering is a sound strategy, but there seems to be a lack of an ablation experiment, that is, whether the suggested method outperforms SparseGPT under the same layer-wise sparsity situation.

Author response:
We thank the reviewer for raising this point. Our proposed improved curvature approximations use gradient information that relates curvatures of individual layers to the final global objective. The gradient terms can be interpreted as the relative influence layer outputs have on the final objective. For this reason, our improved curvature estimates are especially useful in conjunction with global thresholding, which allocates weight to be pruned globally rather than for layers separately. Nevertheless, we will run an ablation study on this and include it in the appendix.

Reviewer comment (Theory):
The paper still uses approximations for computational tractability which limits pruning performance.

Author response:
Due to the high dimensionality of LLM models, the constraint optimization problem considered in this work is inherently intractable. This means that it can be mathematically proven that the problem can not be solved in an exact manner and the sheer use of an approximation should thus not be considered a 'weakness' but rather a necessity. In fact, most other methods (even magnitude pruning) can be seen as some sort of crude approximation to the Hessian of the global loss and are strictly more approximate than ours. Thus, compared to other methods we use the most accurate curvature approximation. We expect this to be a misunderstanding and will make this more clear in the main text to prevent confusion.

Reviewer comment (Evaluation):
Structured pruning leads to irregular sparsity patterns which are difficult to accelerate. The real inference speedup or memory savings after pruning is unknown.

Author response:
We thank the reviewer for raising this as the computational benefits of the considered pruning structures might not have been emphasised enough in the paper. It is not true that structured pruning results in irregular sparsity patterns. Rather the opposite, it provides the most regular sparsity patterns - hence the name 'structured' - and is very easy to accelerate. As complete rows and columns are removed, structured pruning of X% of weights means that matrix dimensions can directly be reduced by X%. The resulting smaller matrices have simply lower dimensions and are thus directly amenable to existing standard CUDA/and hardware-accelerated kernels. We recognize that we only very briefly mention this in Sec. 2. We thank the reviewer and will make sure this point is made more clear in the main text.

Reviewer comment (Evaluation):
Additional FLOPs for approximation and updating offsets gains during deployment are needed, while the detailed comparison and discussion might be missing.

Author response:
We are not exactly sure what is meant by 'updating offsets gains', but will assume the question is whether reported weight reduction relates to actual computational and memory savings at deployment. Unlike most other sparsity patterns, structured pruning allows for a direct reduction in the dimensionality of weight matrices by removing complete rows and columns. Therefore, a reduction of weights will result in the same reduction in the number of FLOPS. This is not generally true for most other sparsity patterns, and in fact an important benefit and in part the motivation for structured sparsity patterns. Below, we provide compression time of compressed models which we will add to the paper appendices.

Reviewer comment (Novelty):
Some related works might also be good to include [1]; [1] Yu, Shixing, et al. "Hessian-aware pruning and optimal neural implant." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

Author response:
We thank the reviewers for pointing out this work. We find the reference relevant and will add it to an updated related work section.

Reviewer comment (Experiments):
Could larger model sizes also be included and evaluated throughout?

Author response:
We agree that it would be interesting to assess performance on even larger models. Following the suggestions by the reviewers (zzzW, 2GDn), we now also include results for the larger Llama-v2 13B model and evaluated performance on downstream tasks (see reply to Reviewer 2GDn). We find that our method also outperforms baselines on this larger model, as well as on downstream tasks.