Paper: SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR-2024)

Reviewer comment (Evaluation):
*Evaluation Benchmarks:* While the authors have acknowledged this in their limitations, the absence of performance evaluations on trendy benchmarks for generative models (such as MMLU, mathematical reasoning performance like GSM8K, or coding abilities such as humaneval) represents a weakness. Evaluating at least one of these benchmarks is crucial to better ascertain the proposed algorithm's effectiveness.

Author response:
**A1**: To address this concern, we provide MMLU and GSM-8K results below. We believe that perplexity measures are still very important: Previous work [1] showed that perplexity evaluations have a very strong correlation (-0.94) with performance on a diverse set of zeroshot tasks. This means that the perplexity evaluations generalize to many different tasks.

Following your suggestion we evaluate our method with MetaMath-7B, a recent version of Llama-2 specialized on math problem-solving, and  CodeLlama-7b, a popular code generation model. One can see that SpQR is close to fp16 model performance with 4 bit quantization while outperforming GPTQ by ~2\%. Drops on HumanEval dataset are more significant, but SpQR is still much better than GPTQ at the same bitwidth. These results correlate well with perplexity measure differences. We do not report pass@100 score, as it appears to be extremely noisy.  RTN quantization turns out to be numerically unstable and produces models with random performance.

Table 1. GSM-8k results

|	model	| method | wbits | Accuracy (%) |
|:-----------:|:------:|:-----:|:------------:|
| MetaMath-7B |  None  |   16  | 	66.9 	|
|         	|   RTN  |   4   |  	NaN 	|
|         	|  GPTQ  |   4   | 	64.7 	|
|         	|  SpQR  |  3.98 | 	66.5 	|

Table 2. HumanEval results

| 	model	| method | wbits | pass@1 | pass@10 |
|:------------:|:------:|:-----:|:------:|:-------:|
| CodeLLama-7b |  None  |   16  |  40.0  |   58.0  |
|          	|   RTN  |   4   |	NaN  |	NaN	|
|          	|  GPTQ  |   4   |  32.1  |   52.2  |
|          	|  SpQR  |  3.98 |  36.3  |   54.6  |

Reviewer comment (Experiments):
*Quantization Baselines:* SpQR demonstrates good accuracy using a very small group size. Therefore, I kindly suggest comparing it with baselines like RTN, GPTQ [1] and AWQ [2] by applying a group size. While it's important to compare accuracy at identical bit-width levels, it's equally crucial to evaluate inference latency at the same accuracy level. In this context, SpQR might offer reduced memory usage. Since SpQR introduces a new representation by combining dual quantization with sparse outliers, it's essential to provide a presentation that underscores the advantages of SpQR in relation to conventional methodologies.

Author response:
**A2**: The performance of GPTQ is very competitive at a group size of 128 and we were not able to improve performance by much using a smaller group size. This is why we kept running the experimental settings from the main GPTQ experimental setup. We will run these experiments and will provide these results. In terms of latency the overhead of loading groups comes mainly from the memory footprint of the scales and zeropoints and not the group size itself. As such, the average bit-footprint translates almost 1:1 to inference latency.

Reviewer comment (Experiments):
*Inference Time:* SpQR employs a two-stage quantization process to utilize a very small group size, allowing for fine-grained representation and enhancing accuracy through the use of sparse representation for outliers. While it is evident that this proposed method will yield very positive results in terms of accuracy, there is a growing concern that the associated latency overhead needs to be better managed. In other words, the method presents a well-analyzed problem statement combined with easily understandable techniques that have high accuracy. However, it seems to have a weakness in the latency overhead of the two-stage quantization with such a small group size and the latency overhead for the sparse representation of outliers. Therefore, it would have been more beneficial if there had been a more varied presentation of the latency results, which arise from combining sparse matrix multiplication with quantization matrix multiplication. Specifically, in the "Inference Time" section and paragraph, the speed of the end-to-end proposed algorithm is measured. However, it would have been a better presentation if the latencies without sparse outliers and with sparse outliers were both disclosed, allowing readers to recognize the performance trade-off easily. To delve into more specifics, when sparse outliers are excluded, there should be a new baseline (such as existing prior kernels [1, 2] for weight-only quantization) comparing the latency of the quantized model where the proposed two-stage quantization has been applied. Furthermore, discussing the additional overhead when introducing sparse outliers after such a comparison would have been an excellent addition.

Author response:
**A3**: Small group sizes do not affect latency for our implementation. Since every warp of threads on the GPU processes multiple values in the matrix multiplication, we can load multiple groups with each warp of threads. The total values loaded remain the same independent of group size. This means that inference speed is not dependent on the group size but just on the average amount of bits in the representation. The latency is affected by the sparse representations, but with the right implementations we can still generate about 115 tokens per second from a 7B model on a single A100 GPU. As such, our implementation is still fast enough to be practical despite the sparse representations.

Reviewer comment (Evaluation):
Ideally, to assess the performance of SpQR, it would enhance clarity if the author could specify the difference in accuracy when comparing the proposed SpQR to 4-bit group-wise methods like RTN, GPTQ [1], and AWQ [2]. At the same time, understanding the inference latency of SpQR, especially given its very small group size in comparison to these 4-bit group-wise weight-only quantization methods, would be beneficial. Lastly, highlighting its memory efficiency would further improve the clarity.

Author response:
**A5**: As suggested, in addition to comparison at the same bitwidth we perform comparison at the same accuracy level. The table below shows that GPTQ requires 0.5-0.6 more bits per parameter to match the performance of SpQR configuration with 3.5 bits per parameters on average, and 0.3-0.4 to match lossless compression configuration. This difference can be critical in some cases.
For example, with 3.5 bit per parameter one can fit Llama-2-70b on a single V100 with 32Gb
and have some space for KV cache, which would be impossible for GPTQ quantization with the same accuracy without weight offloading.

| Model | Method | wbits | wikitext2 |  c4  |
|:-----:|:------:|:-----:|:---------:|:----:|
|   7   |  GPTQ  |  4.07 |	5.88   | 7.28 |
|   	|  SpQR  |  3.45 |	5.88   | 7.33 |
|   	|  GPTQ  |  5.07 |	5.72   | 7.12 |
|   	|  SpQR  |  4.63 |	5.73   | 7.13 |
|   13  |  GPTQ  |  4.04 |	5.26   | 6.76 |
|   	|  SpQR  |  3.45 |	5.25   | 6.77 |
|   	|  GPTQ  |  5.04 |	5.13   | 6.64 |
|   	|  SpQR  |  4.63 |	5.13   | 6.64 |
|   30  |  GPTQ  |  4.02 |	4.28   | 6.11 |
|   	|  SpQR  |  3.49 |	4.29   | 6.11 |
|   	|  GPTQ  |  5.01 |	4.14   | 6.01 |
|   	|  SpQR  |  4.69 |	4.14   | 6.01 |
|   65  |  GPTQ  |  4.01 |	3.71   | 5.74 |
|   	|  SpQR  |  3.52 |	3.71   | 5.73 |
|   	|  GPTQ  |  5.01 |	3.58   | 5.65 |
|   	|  SpQR  |  4.71 |	3.57   | 5.64 |


**A6**: We acknowledge this point. The memory efficiency is determined by the average amount of bits which we report. For example, 3.4 bits per parameter for a 70B model would be 70B*(3.4 bits / 8 bits) = 70B*(0.425 bytes) = 29.75 GB.

Reviewer comment (Reproducibility):
In the "Ablations" paragraph of section 5, the mention of "16-bit statistics" seems to require clarification regarding the group size.

Author response:
**A7**: Thank you, we overlooked this detail and will add this information.

Reviewer comment (Experiments):
The experiment results related to large vision model (such SAM/DINOv2) are expected also.

Author response:
We believe we have a good range of language models that we evaluate. Adding vision models can help to test the robustness of our approach. On the other hand, vision models are relatively small compared to LLMs and since the main advantage of our method is the memory savings while maintaining inference speed, it would not be very useful for vision models since they are often not limited by memory.

Reviewer comment (Evaluation):
Are there experimental results that also quantify the activation?

Author response:
Currently, we do not do activation quantization. We could use the outlier mask also for the activations which would solve some problems observed in earlier work. However, we quantizing the activations to 4-bit or less would be a new and very difficult problem. We think outliers + 8-bit inputs with SpQR quantized weights could work well, but we have not run these experiments.

Reviewer comment (Evaluation):
The proposed method has limitations and may not be easily applicable to activation quantization in LLM. During inference, the memory accesses for activations also take a large amount of time.

Author response:
**A1**: If the batchsize is 1, for example LLM use for a single user, the activations for Llama 2 7B are 4000x smaller than the weight matrix. So for this case, the activation just take 0.025% of overhead. For larger batch sizes, the activations are a larger overhead, but even for batch sizes common in deployments, like ChatGPT, loading the activations for a 7B model would only account for at most about 7% of overhead. As such, this limitation is not a big one.

Reviewer comment (Experiments):
The unstructured sparse pattern of the outliers is not very friendly to GPUs, and this will impose significant overhead on the computation.

Author response:
**A2**: We measured the overhead and still provide faster inference than a unquantized model. So yes, there is an overhead, but inference is still fast. With the right implementation, for a 7B model, we could generate about 115 tokens per second on a single RTX 4090 consumer GPU. As such, this would still be fast enough to be practical. Thus, there is no significant practical disadvantage to having the sparse structure.

Reviewer comment (Reproducibility):
In section 4.1, high-sensitivity outliers, there are two types of outliers, i.e., group-wise outliers and individual outliers. In section 4.2, the weight outliers are stored with the Run-Length Coding, which saves the relative index. Are the group-wise outliers used in the proposed method? Or only save individual outliers? For the two types of outliers, do they both use the same format to store?

Author response:
**A3**: We use the run-length coding for all outliers. This means both the grouped outliers and sparse ones identified in our analysis are quantized in the same way. The group-wise outliers only make up about 0.1 to 0.2% of total weights. As such, having a separate encoding just for these outliers is not very beneficial since the efficiency gains would be minimal.

Reviewer comment (Experiments):
As for the weights in multi-head attention and the MLP, is there any difference when quantizing those weights? Or they are all the same?

Author response:
**A4**: We did not study this question in this work, but we studied this before (unpublished results) and we found that essentially the performance per weights is approximately the same. Since MLP layers have in total more weights than all attention layers, this means that MLP layers as a whole have slightly more importance. But for individual parameters in the weights, the importance is roughly the same. As such, there is no major difference between quantizing multi-head attention and MLP layer weights. We see some difference in outliers distribution, but these are small and only account for less than 1% of all quantized values.

Reviewer comment (Experiments):
Did you compare the model accuracy and the real speedup with the 8-bit weight quantization methods? I just wanna if it is worth quantizing the weights to 4-bit with such a large overhead from the unstructured sparse outliers.

Author response:
**A5**: Our method should be about 2x slower than 8-bit methods, but reduces memory by more than 2x times. As such, this method is best used for very large models that are difficult to use. For example, our method is ideal for a 7B model on mobile devices, or a 13B model on laptops, or a 65B model on consumer GPUs. 8-bit methods are more suitable when one runs large batch sizes, such when a company deploys LLMs in production since they usually do not have memory problems because LLMs are distributed across 8 GPUs.

Reviewer comment (Experiments):
Can you compare the performance between the quantized GEMM with 4-bit SpQR and the quantized GEMM with a normal 4-bit weight quantization method? It is better to do such ablation to show the performance overhead of the grouped quantization and the unstructured sparse outliers.

Author response:
**A6**: The currently best methods for 4-bit quantization are about 3.5x faster than 16-bit methods. SpQR is 1.15x faster than 16-bit methods. As such 4-bit quantization without outliers is up to 3.04x faster than SpQR. However, the performance of SpQR is much better while also compressing the model more strongly, that is close to 3.4 bits per parameter on average. For example, at 3.4 bit the memory savings compared to 4-bit techniques are about 1.3x better and the performance gap in terms of perplexity between 16-bit and 4-bit methods is halved. Taking these factors into consideration, SpQR is pretty close in speed to other methods, because one would need more bits (~5 bits) to achieve similar performance which is about 50% more GPU memory. So for example, SpQR enables to run Falcon 180B on a single GPU, but to get the same performance with regular quantization methods one would need to use two GPUs to fit this model. Since models keep increasing in scale over time, the memory benefit is an important factor.

Reviewer comment (Novelty):
SpQR's methodology, which focuses on outlier management and mixed-precision quantization. Prior works have already delved into the impact of outliers on quantization [\[1\]](https://arxiv.org/pdf/2306.03078.pdf) and the use of mixed-precision quantization [\[2\]](https://openaccess.thecvf.com/content_CVPR_2019/papers/Wang_HAQ_Hardware-Aware_Automated_Quantization_With_Mixed_Precision_CVPR_2019_paper.pdf) to optimize resource use. Hence, SpQR represents an evolutionary improvement in engineering work on LLM weight compression, refining rather than making some novel change in the field.

Author response:
Compared to prior work mentioned we study large language models (LLMs) which have very different outliers patterns than the smaller models common 3 years ago. Please see LLM.int8()[1], for a discussion on how LLM outliers differ from small scale networks. We extend the analysis in [1] to more complex outliers structures. This analysis is novel and goes beyond the structures discovered in earlier work.

Reviewer comment (Evaluation):
While the paper claims efficiency improvements, the actual gains may depend on specific model architectures and deployment scenarios, and there could be overheads associated with managing the sparse representations.

Author response:
Yes our approach is more involved, but so are comparable approaches. The further we compress LLMs the more complex the approach will be. We believe research on LLM compression works in waves: first complicated approaches reveal new insights, then the research challenge is to simplify these approaches while maintaining performance. The fact that our approach is complex does not diminish the insights and strong results that our work provides. As such, it is a strong basis for any future work on simpler methods.

Reviewer comment (Reproducibility):
The benefits of SpQR are maximized with specific hardware capabilities, which may not be universally available.

Author response:
Since the sparse representations only vary between 0.35% to 1.0% the fraction of outliers does not change much and the performance is almost the same for all architectures. The biggest difference that we see is for Falcon 180B which requires more outliers than other models, about 1%, while Llama v1/v2 require ~0.35-0.45%. This also demonstrates that differences between architectures are very minimal. Please see “Sparse GPU Kernels for Deep Learning”[2) for a discussion on how sparsity level relates to performance of sparse matrix multiplication. Note that for SpQR, our implementation is more effective compared to the implementation presented in [2].