Paper: SqueezeLLM: Dense and Sparse Quantization (ICLR-2024)

Reviewer comment (Novelty):
There are some missing references. Regarding sparse encoding of the outliers, [1] propose an approach that bears some similarities and would be worth mentioning. Similarly, regarding the measurement of the importance using the Hessian matrix, pruning techniques have previously used similar estimates [2,3] and in particular [4]. Including these elements would benefit the paper.

Author response:
We appreciate the reviewer’s valuable feedback and the key references. We will include these references and comparisons in the final version of our paper to provide a more comprehensive context.

Reviewer comment (Evaluation):
The results are great, so there is no need to avoid highlighting other methods when they perform on par with the proposed SqueezeLLM.

Author response:
We appreciate the reviewer’s valuable feedback and the key references. We will include these references and comparisons in the final version of our paper to provide a more comprehensive context.

Reviewer comment (Reproducibility):
It is unclear how the authors would fine-tune such a compressed model without losing the benefits of the compression technique, i.e., how to fold a LoRA or another similar approach.

Author response:
This is indeed an interesting question given that finetuning LLMs is emerging as an interesting research field. We envision that SqueezeLLM could be effectively employed as a backbone quantization method in parameter-efficient finetuning scenarios similar to QLoRA [1] – where the main model would be quantized for the forward pass, and only the smaller adapter modules would undergo weight updates – without losing the benefits of our compression technique. While it's common in practice [1, 2] to retain the LoRA module after fine-tuning, how to fold them will be an interesting avenue for future research.

[1] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." *arXiv preprint arXiv:2305.14314* (2023).

[2] Chai, Yuji, et al. "INT2. 1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation." *arXiv preprint arXiv:2306.08162* (2023).

Reviewer comment (Reproducibility):
I have reservations about the kernel implementation. Introducing such custom kernel, as described in your method, appears to disrupt the established high-performance computing framework. Although an unstructured or non-uniform structure might enhance model performance, it could also negatively impact acceleration performance or complicate it. Hence, a more detailed exposition about the kernel is essential to substantiate the novelty of your method.

Author response:
This is a fair point. We had originally focused on lower-grade GPUs partly due to the need for enabling inference on more affordable GPUs, and partially due to our limited access to A100 systems for our experiments. However, in light of the reviewer’s insightful comments, we have conducted additional experiments using our kernels on an A100 system to run the linear layers of LLaMA model for generating a sequence length of 128. As demonstrated in the table below, our kernel implementation still attains **1.5-2.5x performance speedups** relative to the fp16 matrix-vector multiply kernel across different model sizes, similar to what we have observed with our previous experiments. This is despite any additional optimizations or tuning.  

Therefore, our evaluation results indicate that SqueezeLLM **does not introduce any negative impact on acceleration performance** due to our unstructured or non-uniform structure. Furthermore, we would like to highlight that the implementation of SqueezeLLM does not introduce complexity. The core of SqueezeLLM is non-uniform quantization which only involves look-up-table operations, which can be implemented as *simply and efficiently* as those in uniform quantization.  
\
\
**Matrix-Vector Multiply Kernel Runtime for Generating 128 Tokens**  

| LLaMA Model | Baseline (FP16) | SqueezeLLM (4-bit) | SqueezeLLM (4-bit, 0.45% outliers) | SqueezeLLM (3-bit) | SqueezeLLM (3-bit, 0.45% outliers) |  
| ----------- | --------------- | ------------------ | ---------------------------------- | ------------------ | ---------------------------------- |  
| 7B          | 1.21            | 0.83               | 1.09                               | 0.56               | 0.83                               |  
| 13B         | 2.32            | 1.52               | 1.87                               | 0.97               | 1.32                               |  
| 30B         | 5.56            | 3.66               | 4.25                               | 2.26               | 2.86                               |

Reviewer comment (Experiments):
While your results indicate an improved model performance, the absence of a dedicated kernel might undermine its effectiveness.

Author response:
This is a fair point. We had originally focused on lower-grade GPUs partly due to the need for enabling inference on more affordable GPUs, and partially due to our limited access to A100 systems for our experiments. However, in light of the reviewer’s insightful comments, we have conducted additional experiments using our kernels on an A100 system to run the linear layers of LLaMA model for generating a sequence length of 128. As demonstrated in the table below, our kernel implementation still attains **1.5-2.5x performance speedups** relative to the fp16 matrix-vector multiply kernel across different model sizes, similar to what we have observed with our previous experiments. This is despite any additional optimizations or tuning.  

Therefore, our evaluation results indicate that SqueezeLLM **does not introduce any negative impact on acceleration performance** due to our unstructured or non-uniform structure. Furthermore, we would like to highlight that the implementation of SqueezeLLM does not introduce complexity. The core of SqueezeLLM is non-uniform quantization which only involves look-up-table operations, which can be implemented as *simply and efficiently* as those in uniform quantization.  
\
\
**Matrix-Vector Multiply Kernel Runtime for Generating 128 Tokens**  

| LLaMA Model | Baseline (FP16) | SqueezeLLM (4-bit) | SqueezeLLM (4-bit, 0.45% outliers) | SqueezeLLM (3-bit) | SqueezeLLM (3-bit, 0.45% outliers) |  
| ----------- | --------------- | ------------------ | ---------------------------------- | ------------------ | ---------------------------------- |  
| 7B          | 1.21            | 0.83               | 1.09                               | 0.56               | 0.83                               |  
| 13B         | 2.32            | 1.52               | 1.87                               | 0.97               | 1.32                               |  
| 30B         | 5.56            | 3.66               | 4.25                               | 2.26               | 2.86                               |

Reviewer comment (Experiments):
There was no effort to implement and optimize the kernel specifically for the A100. Even though the A100/H100 might be a costly choice, they can serve as a foundational hardware for large language models, particularly in tandem with NVIDIA's inference software. It's noteworthy that the memory bandwidth for the RTX series is below 1TB/s.

Author response:
This is a fair point. We had originally focused on lower-grade GPUs partly due to the need for enabling inference on more affordable GPUs, and partially due to our limited access to A100 systems for our experiments. However, in light of the reviewer’s insightful comments, we have conducted additional experiments using our kernels on an A100 system to run the linear layers of LLaMA model for generating a sequence length of 128. As demonstrated in the table below, our kernel implementation still attains **1.5-2.5x performance speedups** relative to the fp16 matrix-vector multiply kernel across different model sizes, similar to what we have observed with our previous experiments. This is despite any additional optimizations or tuning.  

Therefore, our evaluation results indicate that SqueezeLLM **does not introduce any negative impact on acceleration performance** due to our unstructured or non-uniform structure. Furthermore, we would like to highlight that the implementation of SqueezeLLM does not introduce complexity. The core of SqueezeLLM is non-uniform quantization which only involves look-up-table operations, which can be implemented as *simply and efficiently* as those in uniform quantization.  
\
\
**Matrix-Vector Multiply Kernel Runtime for Generating 128 Tokens**  

| LLaMA Model | Baseline (FP16) | SqueezeLLM (4-bit) | SqueezeLLM (4-bit, 0.45% outliers) | SqueezeLLM (3-bit) | SqueezeLLM (3-bit, 0.45% outliers) |  
| ----------- | --------------- | ------------------ | ---------------------------------- | ------------------ | ---------------------------------- |  
| 7B          | 1.21            | 0.83               | 1.09                               | 0.56               | 0.83                               |  
| 13B         | 2.32            | 1.52               | 1.87                               | 0.97               | 1.32                               |  
| 30B         | 5.56            | 3.66               | 4.25                               | 2.26               | 2.86                               |

Reviewer comment (Evaluation):
The majority of the model performance results focus on PPL outcomes. It would be beneficial to include MMLU or CSR results for larger models within the main content, rather than relegating it to the appendix.

Author response:
Thank you for your constructive feedback. We acknowledge your point regarding the inclusion of MMLU and CSR results for larger models in the main body of the content. As can be seen in Table A.4 and A.7, our method *consistently outperforms* other methods both in terms of perplexity as well as MMLU accuracy on *larger models*, showing the strong transferability of the results in the smaller models to the larger models. The decision to place these results in the appendix was due to the page constraints, and we will include it in the main content in the final version. Reviewer 1 also asked us to perform 5-shot MMLU results which are included in response to their question (please see W3 of Reviewer U5Sb)

Reviewer comment (Experiments):
SqueezeLLM uses a small sample of the training dataset to execute end-to-end forward and backward passes for gradient computations. This process appears to be more resource-intensive than other methods like RTN, GPTQ, or AWQ. It would be beneficial to understand the time and resources required for SqueezeLLM's dense-and-sparse quantization.

Author response:
That is correct and SqueezeLLM involves an end-to-end forward and backward pass over a small set of sampled data. This method differs from other methods such as RTN, which does not require any forward or backward pass, and GPTQ or AWQ, which requires isolated gradient calculation over individual layers, resulting in lower overhead. However, it is important to note that the overhead of SqueezeLLM is quite manageable and, crucially, our method consistently outperforms RTN, GPTQ, and AWQ as shown in the paper (Table 1).
To provide a clearer understanding of SqueezeLLM’s resource requirements, we conducted a comprehensive analysis of the time and memory requirements for (1) Fisher information computation and (2) sensitivity-aware k-means clustering for nonuniform quantization as below.
\
\
**1. Fisher Information Computation:**

| LLaMA Model | End-to-End Latency (s) | Peak Memory (GB) |
| --- | -- | -- |
| 7B   | 20     | 33   |
| 13B   | 35     | 61               |
| 30B         | 80   | 149              |
| 65B         | 150     | 292              |


We assessed the end-to-end latency and peak memory usage for computing Fisher information on an A100 system across different LLaMA models. The results indicate that the latency for calculating Fisher information (i.e. the sum of squares of gradients) is pretty fast and quantizing the 65B model can be performed on 4 A100 GPUs to address the peak memory usage.
\
\
**2. Sensitivity Aware K-means Clustering:**

| LLaMA Model | End-to-End Latency (min) |
| -- | ----- |
| 7B  | 18    |
| 13B   | 33  |
| 30B  | 71  |
| 65B  | 120 |


Another key component of SqueezeLLM is the sensitivity-aware K-means clustering applied to each weight matrix. This process can be executed efficiently in parallel across different layers, possibly using multiple cloud CPU instances. Our latency measurements on 8 Intel Xeon Platinum 8380 systems show that the time overhead for this clustering process is quite manageable.
We will include these data in the final version of the paper to offer a comprehensive view of SqueezeLLM's time and resource demands.

Reviewer comment (Experiments):
In the discussion about the need to minimize overall perturbations for the final loss term in section 4.1's "Sensitivity-Based K-means Clustering," the paper should also compare its performance with methods like AdaRound [1] or FlexRound [2]. These methods utilize a small calibration set and employ layer-wise or block-wise post-training quantization techniques. Notably, since FlexRound reports results on LLaMA using uniform weight-only quantization, it would strengthen the paper's claim about optimization with the final loss.

Author response:
That is a great question. Below in the table, we provide a comparison with AdaRound and FlexRound. We first have to kindly emphasize that it is unfortunately not feasible to do a direct comparison with FlexRound due to their closed-source code and models, as well as a lack of experimental details for their perplexity evaluation on WikiText. However, to address the reviewer’s question to the best of our ability,  we have utilized the results published in the FlexRound paper (for AdaRound and FlexRound) to assess the amount of post-quantization perplexity degradation of each method across different LLaMA models. As one can see, SqueezeLLM consistently exhibits a smaller drop in perplexity compared to AdaRound and FlexRound.

We should also kindly emphasize that we have performed a comprehensive comparison with the *latest weight-only quantization schemes* designed for LLMs, such as AWQ and SpQR, which share the same goal of minimizing output activation perturbation as FlexRound. In these comparisons, our method consistently demonstrates superior performance, both with 4-bit and 3-bit. Therefore, we maintain that our sensitivity-based k-means clustering that optimizes the final loss provides a more effective post-training quantization solution compared to other existing quantization strategies.
\
\
**Perplexity on Wikitext**

| LLaMA Model | AdaRound Perplexity Drop (baseline →  quantized) | FlexRound Perplexity Drop (baseline →  quantized) | SqueezeLLM Perplexity Drop (baseline →  quantized) |
| ----------- | -- | -- | --- |
| 7B          | 0.79 (8.90 → 9.69)        | 0.28 (8.90 → 9.18)    | **0.09 (5.68 → 5.77)**     |
| 13B         | 0.34 (7.73 → 8.07)    | 0.16 (7.73 → 7.90)   | **0.09 (5.09 → 5.18)**     |
| 30B         | 0.53 (6.35 → 6.88)     | 0.28 (6.35 → 6.63)     | **0.08 (4.10 → 4.18)**   |

Reviewer comment (Experiments):
It would be beneficial to display five-shot performance results on the MMLU benchmark using LLaMA, as this would offer a more comprehensive comparison with other methodologies.

Author response:
That is an excellent question. We have included five-shot performance results on the MMLU benchmark for Vicuna-v1.1 in the table below and also included a comparison with AWQ with the same model size. Similar to the zero-shot results that we had included in the paper, we can see that SqueezeLLM **consistently achieves higher-quality results**. At the moment, we are also running a comparison with GPTQ but we expect GPTQ to have a similar performance as AWQ. We will include these results in the final version of the paper beside Table 2 where we compared instruction following capabilities of the different models.
\
\
**MMLU 5-shot Accuracy**

| Vicuna-v1.1 | Baseline (FP16) | SqueezeLLM (3.24 bit) | AWQ (3.24 bit) |
| --| -- | -- | --------------------------------------- |
| 7B          | 45.3%  | **42.2%**  | 41.4%|
| 13B         | 50.0%  | **48.2%**   | 46.3% |

Reviewer comment (Evaluation):
Is there a relationship between outliers and sensitive weights? Specifically, are most of the sensitive weights outliers?

Author response:
From our extensive experiments conducted on Vicuna, LLaMA, and LLaMA2 models, we found no direct correlation between outliers and sensitive weights. In fact, we frequently observed that weights with smaller values can also be sensitive (as in Figure 2, Left). This is expected since the sensitivity measured by the Fisher information is not necessarily correlated with the magnitude of the weight value, but more with the curvature of the loss function with respect to that weight value.

Reviewer comment (Experiments):
Have you experimented with combining a uniform quantization scheme with the sparse decomposition concept?

Author response:
Yes, we have indeed explored integrating uniform quantization with the dense-and-sparse decomposition while developing SqueezeLLM. Specifically, we applied dense-and-sparse decomposition in combination with channel-wise uniform quantization on LLaMA 7B and 13B models. In the table below, we compared the performance of uniform quantization and our sensitivity-aware non-uniform quantization, both paired with the dense-and-sparse decomposition. The results indicate that non-uniform quantization based on our sensitivity-aware K-means method yields significantly better perplexity.
It is worth noting that the performance of uniform quantization could potentially be improved through grouped quantization strategies, where a small set of weights (e.g. 128 as in the table below) are grouped together to have their own scaling factor. While grouped quantization, as demonstrated in techniques like GPTQ and AWQ, is a viable approach, the SqueezeLLM approach offers a much simpler implementation by avoiding the complexities of fine-grained grouping and it still outperforms uniform quantization methods with grouping.
\
\
**Perplexity on C4**

| LLaMA Model | Uniform | Dense-and-Sparse + Uniform | Dense-and-Sparse + Uniform with grouping = 128 | Dense-and-Sparse + Non-uniform |
| ----------- | ------- | -------------------------- | ---------------------------------------------- | ------------------------------ |
| 7B          | 28.26   | 9.25                       | 8.32                                           | **7.56**                       |
| 13B         | 13.24   | 7.96                       | 7.30                                           | **6.92**                       |

Reviewer comment (Experiments):
In Table 1, there's a comparison with AWQ's latency performance. However, such a comparison is missing in Table 3 (section 5.4). Since the AWQ kernel is well showcased, it would be more convincing to show a comparison of the proposed kernel with AWQ.

Author response:
We appreciate the suggestion to include a comparison with AWQ's public kernel. However, as we outlined in Appendix A.3, we chose not to compare our kernels directly with those from AWQ for a couple of key reasons: (1) The 3-bit kernels of AWQ are not publicly available; and (2) AWQ has incorporated optimizations that are unrelated to quantization, such as LayerNorm and positional embedding. These optimizations, while effective, are universally applicable to various quantization methods including SqueezeLLM and GPTQ.