Paper: SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR-2024)

Reviewer comment (Evaluation):
The absence of inference time data for SparseGPT in the experiments makes it challenging to convincingly demonstrate the superior efficiency of SliceGPT.

Author response:
> _The absence of inference time data for SparseGPT in the experiments makes it challenging to convincingly demonstrate the superior efficiency of SliceGPT. (and Q2)_

This was a challenge for us: the SparseGPT authors do not provide a way to verify and kind of speedup using their method. See for example this [github issue](https://github.com/IST-DASLab/sparsegpt/issues/15).  The issue is that pytorch does not provide off-the-shelf support for structured sparsity. This means that at the moment, it’s not possible to verify end-to-end speedup for SparseGPT. Our work-around is to provide end-to-end results for SliceGPT, and apples-to-apples comparisons with SparseGPT on their terms. Please see main rebuttal for more details and forthcoming experiment.

Reviewer comment (Experiments):
The paper lacks a comparative analysis with state-of-the-art pruning methods such as low-rank approximation, unstructured sparsity, and block sparsity. The omission of these comparisons limits the paper's ability to establish the competitiveness of SliceGPT within the broader context of pruning techniques.

Author response:
> _lack of comparative analysis with state-of-the-art pruning methods such as low-rank approximation, unstructured sparsity, and block sparsity._ 

You’re right – we should add some rationale here. 

1. different sparsities. We compared to SparseGPT 2:4 sparsity because this is the SOTA method for LLMs. SparseGPT also provides unstructured sparsity and other sparsity structures, *but these provide little to no speedup*, which is the motivation the developers of  2:4 sparsity. This is why we chose SparseGPT 2:4 as a baseline, because it’s the strongest competitor. If we’re doing better than this, we’re doing well. 

2. Low rank approximations. We did make reference to low-rank works, which take each weight matrix and replace it with a pair of low-rank matrices. To our knowledge, no one has applied this idea to LLMs, and as we discussed in the text, it seems unlikely that they could improve the performance of LLMs. Of course, the [LoRA method]( https://arxiv.org/abs/2106.09685) has been very influential for fine-tuning, but we’re not aware of a method to use low-rank approximations in improving deployment of LLMs. 

3. Other works. We’ve added reference to [LLM-pruner](https://arxiv.org/pdf/2305.11627.pdf), a modern method for pruning LLMs. LLM-pruner provides less speedup than SliceGPT (see their table 3), requires fine-tuning post sparsity, and does not retain perplexity as well as our SliceGPT method (see their table 1). We’ve also added more context around other pruning methods, see response to reviewer 96nz.

Reviewer comment (Experiments):
Can you show the inference time of SparseGPT in comparsion to SliceGPT under the same experimental setup?

Author response:
> _The absence of inference time data for SparseGPT in the experiments makes it challenging to convincingly demonstrate the superior efficiency of SliceGPT. (and Q2)_

This was a challenge for us: the SparseGPT authors do not provide a way to verify and kind of speedup using their method. See for example this [github issue](https://github.com/IST-DASLab/sparsegpt/issues/15).  The issue is that pytorch does not provide off-the-shelf support for structured sparsity. This means that at the moment, it’s not possible to verify end-to-end speedup for SparseGPT. Our work-around is to provide end-to-end results for SliceGPT, and apples-to-apples comparisons with SparseGPT on their terms. Please see main rebuttal for more details and forthcoming experiment.

Reviewer comment (Writing):
I think section "layernorm transformers can be converted to RMSnorm" is not well motivated. Could the authors explain more in details the subtleties of this section and why it was written? I may have missed the point.

Author response:
Apologies if this wasn’t clear. SliceGPT depends on the computational-invariance trick. In turn, this depends on the commutation property of equation (2) – you can apply an orthogonal transform before-and-after RMSNorm without changing the model output. The trick does not work for LayerNorm, so we have to convert a network to RMSNorm before we apply SliceGPT.

Reviewer comment (Reproducibility):
Do the authors plan to release the code? I think open sourcing it is very important for the community.

Author response:
100%, we feel the same as you. It’s difficult for us to release pre-publication without de-anonymizing ourselves, but our code will be on github with an MIT license as soon as we’re published. If you would like to inspect the code this week, we’re happy to arrange that (though it will take a while for us to go through the code, to be certain that we’re not de-anonymizing ourselves).

Reviewer comment (Presentation):
The latex is broken, citations are not redirecting, I think your should recompile the pdf.

Author response:
Will do! We blame overleaf 😊

Reviewer comment (Experiments):
Could the authors comment on the use of a random projection (which is orthogonal in expectation, as in sketching methods) compared to $Q_\ell$ computed using by PCA, which is more expensive?

Author response:
We could indeed use a random projection, but it would break eq. 2. It’s not clear to us that random projections could help here, but we hope that you experiment with our code and let us know on release!

Reviewer comment (Reproducibility):
p8: what do the authors mean by "using dense kernels in our compressed models"? Did they code specific kernels for SliceGPT?

Author response:
Ah, this is something we haven’t communicated very well. Sparse models, like those produced by SparseGPT, are actually not easy to run. Pytorch does not have  off-the-shelf support (kernels) for 2:4 structured sparsity yet. In fact, those authors decline to provide actually accelerated models in this Github issue: https://github.com/IST-DASLab/sparsegpt/issues/15. We simply meant that the models produced by SliceGPT are easy to run in standard pytorch. We were trying to be diplomatic about their (very cool) contribution, whilst letting you know that our code is easy to run.

Reviewer comment (Theory):
"Theorem" is too strong for Theorem 1, I suggest "Proposition" or "Remark".

Author response:
Sure, fixed.

Reviewer comment (Evaluation):
The accuracy loss is not "negligible". With 25% sparsity, the perplexity of Llama-2-70B on WikiText2 increases from 3.32 to 4.89, which is similar to a dense Llama-2-13B. However, a 25% sparse Llama-2-70B has much more parameters than a dense Llama-2-13B.

Author response:
> _The accuracy loss is not neglibible_

We agree that this isn’t the right word – we’ve changed the text. We like to point out though that the accuracy loss is less than for SparseGPT (and we have an apples-to-apples latency comparison coming in the next day or so).

Reviewer comment (Evaluation):
The speedup is not impressive.

Author response:
> _The speedup is not impressive._
 
We respectfully disagree! For sparsity/pruning methods, our speedup is pretty good. For example, see the recent LLM-pruner paper. (2305.11627.pdf (arxiv.org) At 20% pruning of LLama 7b, their wikitext ppl is 17.39 at best (up from a baseline of12.62, see their Table 1), and the speedup is less than 15% (see their Table 3). We’re also adding experiments on batch-size (see main rebuttal), where we expect to see much larger throughput increases.

Reviewer comment (Novelty):
Compared to the quantization-based method, there is no advantage.

Author response:
> _Compared to the quantization-based method, there is no advantage (and Q2)_

We agree that quantization is absolutely key for deploying LLMs, and contributes a huge speedup. But pruning and quantization can work together to make LLM deployment cheaper and faster.  Most pipelines use pruning and quantization, along with finetuning/continued pre-training. See for example [Olive](https://microsoft.github.io/Olive/).

There is nothing to prevent a user from applying quantization on top of SliceGPT, and we anticipate that this will give the best speedup. We’d like to emphasize a key point of SliceGPT: that the activations are smaller, leading to less data movement on/between devices, which is something that quantization cannot do. The experiment in our main rebuttal will show this clearly.

Reviewer comment (Evaluation):
In Table 2, it is not fair to multiply the number of GPUs by the total latency and get "GPUms". Huggingface Transformers implements naive model parallelism (or device placement, or pipeline parallelism without pipelining) method to parallelize the models, which means that only one GPU is active at a time. A correct implementation of tensor parallelism or pipeline parallelism will give different results. Considering this, the latency speedup is less impressive.

Author response:
> _it is not fair to multiply the number of GPUs by the total latency_

You’re right. If we were to deploy using a more sophisticated deployment that allowed continuous batching, we’d see a speedup of by a factor of up-to the number of GPUs used. Since part of our speedup comes from reduced number of GPUs, the speedup will be less impressive. However, the total energy consumed will still be in proportion to the numbers in our text. We’ve updated the text to reflect this nuance, here’s what we added:

_Our HuggingFace-based testing does not enjoy continuous batching, where the next batch can be loaded onto the first GPU whilst the second GPU processes the current batch. This means that in terms of inference time, the dense-model could be improved more than our sliced model in terms of GPUms. Nonetheless, our measurements \emph{do} reflect the energy-usage per token in such a deployment._

Reviewer comment (Experiments):
The paper compares accuracy with 2:4 structured sparsity but does not provide head-to-head comparison with SparseGPT (2:4) in terms of latency.

Author response:
Hi Reviewer 7EAi,

You were concerned that we hadn’t provided a head-to-head comparison with SparseGPT, and that the memory savings over SparseGPT weren't clear. We think our main rebuttal fixes this: we show in experiment (1) that 25% slicing is the same speed as 50% sparsity, before we take improved data movement (and increased possible batch size) into account. Experiment (2) demonstrates the increase in batch size possible with SliceGPT leads to throughput of up to 6.26x that of the dense model, due to memory savings.

We also demonstrate the feasibility of applying varying slicing by layer - with performance improvements in OPT but the opposite in Llama models, in experiment (3).

Should you have any further questions, please let us know. If not, we would be very grateful if you were to consider increasing your score.

Reviewer comment (Experiments):
One of the premises of the paper is memory saving, but going through the results it is not clear how the memory savings are in comparison to 2:4 sparsity. Showing a trade-off possibly can clarify this point.

Author response:
Hi Reviewer 7EAi,

You were concerned that we hadn’t provided a head-to-head comparison with SparseGPT, and that the memory savings over SparseGPT weren't clear. We think our main rebuttal fixes this: we show in experiment (1) that 25% slicing is the same speed as 50% sparsity, before we take improved data movement (and increased possible batch size) into account. Experiment (2) demonstrates the increase in batch size possible with SliceGPT leads to throughput of up to 6.26x that of the dense model, due to memory savings.

We also demonstrate the feasibility of applying varying slicing by layer - with performance improvements in OPT but the opposite in Llama models, in experiment (3).

Should you have any further questions, please let us know. If not, we would be very grateful if you were to consider increasing your score.

Reviewer comment (Experiments):
Provide insights on which operation/layer is more sensitive to sparsification. Consider whether you have thought of not uniformly sparsifying all the layers, and whether looking into the range of values in the weight matrices could provide insights on how to apply the sparsification (both degree and pattern).

Author response:
Hi Reviewer 7EAi,

You were concerned that we hadn’t provided a head-to-head comparison with SparseGPT, and that the memory savings over SparseGPT weren't clear. We think our main rebuttal fixes this: we show in experiment (1) that 25% slicing is the same speed as 50% sparsity, before we take improved data movement (and increased possible batch size) into account. Experiment (2) demonstrates the increase in batch size possible with SliceGPT leads to throughput of up to 6.26x that of the dense model, due to memory savings.

We also demonstrate the feasibility of applying varying slicing by layer - with performance improvements in OPT but the opposite in Llama models, in experiment (3).

Should you have any further questions, please let us know. If not, we would be very grateful if you were to consider increasing your score.