Paper: QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR-2024)

Reviewer comment (Evaluation):
The proposed method has a significant advantage *only* under very low-bit settings like W4A4. However, under W4A4, all methods (including QLLM) cannot achieve a reasonable accuracy. Take Table 1 as an example, the W4A4 LLaMA-65B accuracy of QLLM is only 59.83% (average), which is even lower than the FP16 accuracy of LLaMA-7B, which is 62.23%. The huge drop in accuracy makes the setting impractical. On the other hand, the proposed method does not outperform existing work like OmniQuant under higher precisions (e.g., W6A6). Therefore, it is questionable whether the proposed method has a practical advantage.

Author response:
We would like to emphasize that the advantages of our QLLM extend beyond low-bit settings. It is crucial to comprehensively evaluate LLM performance using a variety of metrics, including perplexity (PPL) and zero-shot accuracy, across different benchmarks. Notably, in W6A6 quantization settings, QLLM consistently outperforms OmniQuant in terms of lower PPL on both WikiText2 and C4 across all models, as shown in Tables 1 and A. 

Additionally, our method falls within the post-training quantization settings, where we need to quantize a pre-trained LLM using a small amount of calibration data with little computational overhead. This setup is in line with mainstream LLM quantization methods (Xiao et al., 2023, Wei et al., 2023, Shao et al., 2023) and inherently faces some performance constraints due to the smaller scale of data and training. Notably, in W4A4 settings, QLLM significantly outperforms the concurrent state-of-the-art quantization method, OmniQuant, by a large margin, as shown in Tables 1 and A. Moreover, our QLLM is complementary to large-scale fine-tuning to further improve performance. For example, by using 40 Attention-FFN Blocks for reconstruction and more calibration samples, our 4-bit quantized LLaMA-1-65B achieves **61.04% average accuracy on five zero-shot tasks**, which is **1.21%** higher than the result in the previous submission. Given the time constraints of this rebuttal period, we believe there is potential for more performance gains.

Reviewer comment (Reproducibility):
The proposed disassembly and assembly method will lead to inevitable inference overhead when there is LayerNorm or activation functions, since extra layers are inserted in the forward graph (also confirmed in Table 4, 27% overhead compared to INT8 under context length 256, which is negligible). Furthermore, it is unclear what kind of configurations (e.g., disassembly ratios) are used for the INT8 setting in Table 4, since there is no accuracy number reported under this setting. If we use a larger disassembly ratio, will the overhead be larger?

Author response:
We would like to correct a misconception about the inference cost associated with our QLLM. The actual additional inference cost, when compared to INT8 at a context length of 256, is only 15% rather than 27%, as shown in Table 4 in the initial submission. To further reduce this overhead, we implement our channel disassembly and assembly operation using Triton [A], optimizing these processes for enhanced efficiency. Additionally, the channel expansion ratios determined by our adaptive strategy are kept within a small range (as detailed in Q3), ensuring that the increase in inference overhead remains small. For detailed inference overhead, please refer to Q1 of the general response.

**Reference**

[A] Triton: an intermediate language and compiler for tiled neural network computations. MAPL 2019.

Reviewer comment (Evaluation):
It seems from Table 6 that directly updating the quantized weights (i.e., QAT) is always better than the LoRA tuning. Does it mean if we have enough GPUs for parallelized fine-tuning (e.g., FSDP), we should use QAT for the best accuracy?

Author response:
While TQW might be preferable in situations where ample GPU resources are available, our EEC shines in resource-limited settings, delivering performance that is on par with TQW. A notable example of EEC's efficiency is its ability to quantize the LLaMA-1-65B model using only a single 24GB consumer-grade GPU, such as the NVIDIA RTX 4090. 
This efficiency makes EEC especially beneficial for researchers facing resource constraints, markedly improving the accessibility of quantization technology for LLMs. For a comparison that underscores the benefits of EEC over TQW, please refer to Q3 of the general response.

Reviewer comment (Experiments):
While I acknowledge the efficiency of CHANNEL DISASSEMBLY and CHANNEL ASSEMBLY in the form of $y=xW^{l-1}W^{l}$, as explained in "MORE DETAILS ABOUT THE EFFICIENT IMPLEMENTATION FOR CHANNEL REASSEMBLY", I have reservations regarding its applicability to scenarios involving multiple inputs $X=\{x_1,x_2,...,x_K\}\in\mathbb{R}^{K\times M}$ and a non-linear transformation $y=\phi(xW^{l-1})W^{l}$, where $\phi(\cdot)$ represents a normalization layer followed by an activation function like GELU. Despite the authors' claim that "the channel indexes for decomposition and aggregation are calculated offline using calibration data, which only introduces a minor extra overhead", I remain unconvinced that the overhead is negligible.

Author response:
**A1**. Please refer to Q1 of the general response for the inference overhead of channel reassembly. With our efficient Triton kernel implementation, our channel reassembly only incurs **4% additional runtime overhead** compared with W4A4.

Reviewer comment (Theory):
It would be beneficial to provide a proof demonstrating that the approximation error introduced by Equation (4), in conjunction with the subsequent quantization error, is indeed smaller than the quantization error resulting from direct quantization.

Author response:
**A2**. To further show the effectiveness of our channel reassembly (CR), we compare the average block-wise reconstruction error across the entire network before and after applying CR and show the results on a calibration set with 128 randomly selected 2048-token segments from WikiText2 in Table VIII. The results clearly demonstrate that using CR significantly lowers the reconstruction error, and thus improves the performance of the quantized models. We have included these results and the corresponding discussions in Section G of the supplementary material.

Table VIII. Block-wise reconstruction error before and after channel reassembly (CR).
| Model       | Method | Reconstruction Error |
| ----------- | :----: | :------------------: |
| LLaMA-1-7B  | w/o CR |         4.71         |
| LLaMA-1-7B  | w/ CR  |       **2.74**       |
| LLaMA-1-13B | w/o CR |         7.67         |
| LLaMA-1-13B | w/ CR  |       **1.71**       |

Reviewer comment (Theory):
I am interested in whether the proposed CHANNEL DISASSEMBLY and CHANNEL ASSEMBLY methods can be extended to handle multiple inputs, specifically in the case of $Y=XW$ where $X\in\mathbb{R}^{batchsize\times M}$. I have some concerns regarding the effectiveness of Equations (3) and (4) on $X\in\mathbb{R}^{batchsize\times M}$ if the outliers differ among $X_{i,:}$, as the derivation relies on the approximation of the summation of scalars. Additionally, the authors mention that "with the channel indexes for decomposition and aggregation calculated offline using calibration data". Did the authors imply that the outliers often occur at the same index across different inputs? Otherwise, the effectiveness of the proposed methods may vary.

Author response:
**A5**. Our channel disassembly and assembly are indeed designed to work with different inputs, effectively identifying outliers and similar channels. As we have highlighted in Section 1, drawing on recent findings from studies (Dettmers et al., 2022; Xiao et al., 2023; Wei et al., 2023), activation outliers are often found to cluster within specific channels across a variety of inputs. This property is reflected in similar channels for channel assembly as well, which tend to exhibit consistent patterns irrespective of the input. Such uniformity allows for the identification of these channels with a small set of calibration data. We have included the discussions in Section 4.3 of the revised manuscript.

Reviewer comment (Experiments):
It would be beneficial to include the baseline results of $\gamma=0$ in Table 2.

Author response:
**A7**. We would like to clarify that the baseline results for $\gamma=0$ have been included in the first row of Table 2. To enhance clarity and prevent any potential misunderstandings, we have updated the presentation of these results in our revised manuscript as follows.

|  CD   |  CA   |  CP   | Adaptive | $\gamma$ | WikiText2 |  PTB   |   C4   |  Avg.  |
| :---: | :---: | :---: | :------: | :------: | :-------: | :----: | :----: | :----: |
|   ✓   |       |       |          |    0     |  189.35   | 539.59 | 303.45 | 344.13 |

Reviewer comment (Reproducibility):
Did the FP results reported in Table 4 refer to FP32 or FP16 results?

Author response:
**A10**. The FP results reported in Table 4 refer to FP16 results. We have revised our manuscript.

Reviewer comment (Reproducibility):
Could the authors provide pseudo-codes to provide more detailed explanations of the "additional disassembly and assembly layers that are designed to decompose and aggregate channels during runtime"?

Author response:
**A12**. To provide a clearer understanding, we include the pseudo-codes of our disassembly and assembly during runtime in PyTorch style as follows: 

```python
def channel_disassembly(x, num_split):
""" 
    x: input with shape of [batch, tokens, channels]
    num_split: the number of sub-channels for each channel with shape of [channels]
"""
     
    B, N, C = x.shape
    x = x.view(B * N, C)
    scaling = 1.0 / num_split   # compute the scaling factor of each channel
    x = x / scaling                    # scale each channel
    x = torch.repeat_interleave(x, num_split, dim=1) # perform channel decomposition
    C = x.shape[1]
    x = x.view(B, N, C)
    return x
```

``` python
def channel_assembly(x, src_idx, dst_idx):
""" 
    x: input with shape of [batch, tokens, channels] 
    src_idx: the channel index that will be merged in set A with shape of [#num_merged_channels]
    dst_idx: the channel index that will be merged in set B with shape of [#num_merged_channels]
"""
    B, N, C = x.shape
    ori_src_idx = torch.arange(0, C, 2, device=x.device)  # get index for set A
    ori_dst_idx = torch.arange(1, C, 2, device=x.device)  # get index for set B
    src, dst = x[..., ori_src_idx], x[..., ori_dst_idx]    # divide the channels into two sets A and B
    src_C = src.shape[-1]    # get the channel number in set A
    dst_C = dst.shape[-1]    # get the channel number in set B
    
    # A mask that indicates whether a channel is merged
    channel_mask = torch.ones(C, device=x.device, dtype=x.dtype)
    m_idx = ori_src_idx[src_idx]
    channel_mask[m_idx] = 0.0

    n, t1, c = src.shape
    sub_src = src.gather(dim=-1, index=src_idx.expand(n, t1, r))   # get channels that will be merged in set A 
    dst = dst.scatter_reduce(-1, dst_idx.expand(n, t1, r), sub_src, reduce=mode)  # merge channels
    src = src.view(B, N, src_C, 1)
    dst = dst.view(B, N, dst_C, 1)

    # concat set A and set B
    if src_C == dst_C:
        merged_x = torch.cat([src, dst], dim=-1).view(B, N, C)
    else:
        merged_x = torch.cat([src[..., :-1, :], dst], dim=-1).view(
            B, N, src_C + dst_C - 1
        )
        merged_x = torch.cat([merged_x, src[..., -1, :].reshape(B, N, 1)], dim=-1).view(
            B, N, src_C + dst_C
        )
    # remove the merged channels
    merged_x = merged_x.index_select(-1, (channel_mask != 0).nonzero().squeeze())
    return merged_x
```
We have included the above pseudo-codes in Section D of the supplementary material.

Reviewer comment (Evaluation):
Could the authors further elaborate on why OmniQuant consistently performs well on W6A6 but performs extremely poorly on LLaMA-2-70B with W4A4?

Author response:
**A13**. The varied performance of OmniQuant can be attributed to the architecture of LLaMA-2-70B, which employs grouped-query attention (Ainslie et al., 2023) where each group of queries shares a single key and value head. Such architecture makes the learnable equivalent transformation in OmniQuant incompatible with grouped-query attention. In W6A6 settings, the impact of activation outliers is relatively minor, enabling partial learnable equivalent transformations to suffice in maintaining performance. However, in the W4A4 settings, the effect of activation outliers becomes more prominent. Under these conditions, the partial learnable equivalent transformation is insufficient to address the outlier issue, leading to notably poorer performance. We have included the analysis in Section E of the appendix.

Reviewer comment (Experiments):
It is worth noting that "DIFFERENT NUMBERS OF CALIBRATION SAMPLES" play a crucial role in performance. Did the authors follow the same setting as other baseline methods?

Author response:
**A14**. Yes, we use the same number of calibration samples as OmniQuant, as already mentioned in the implementation details of Section 5.

Reviewer comment (Experiments):
While the algorithm perf. experiments are conducted with W6A6 and W4A4, inference efficiency experiments are only conducted with W8A8. I think implementing W4A4 kernels on some NVIDIA GPUs and demonstrating better efficiency-algo. perf. trade-off compared with W8A8 and existing W4A4 will make this paper stronger.

Author response:
**Q1**. I think implementing W4A4 kernels on some NVIDIA GPUs and demonstrating better efficiency-algo. perf. trade-off compared with W8A8 and existing W4A4 will make this paper stronger.

**A1**. Thanks for your valuable suggestion. Please refer to Q1 of the general response for the inference time results.

Reviewer comment (Experiments):
Table 4 shows QLLM has inference overhead compared to methods without channel reassembly. Does this overhead come from the additional disassembly and assembly layers for non-linear layers? A thorough breakdown of this overhead for differently-sized models (larger models, more compressed models such as 4bit) will help illustrate whether the method is really practical.

Author response:
**Q2**. Does the overhead come from the additional disassembly and assembly layers for non-linear layers? A thorough breakdown of this overhead for differently-sized models (larger models, more compressed models such as 4bit) will help illustrate whether the method is really practical.

**A2**. Yes, the additional inference overhead comes from the additional disassembly and assembly layers after the non-linear layers. Please refer to Q1 of the general response for a thorough breakdown of the inference overhead for differently-sized models. Additionally, we further show the inference overhead of QLLM for 4-bit LLaMA-2-70B. From Table V, our QLLM introduces only a **2% additional overhead** compared with 4-bit fixed-point baseline.

Table V. Inference throughput (tokens/s) comparisons of different models. The throughput is measured with a 2048-token segment on 8 NVIDIA RTX 4090 GPUs.
| Model | Method | Inference Throughput (tokens/s) |
| - | - | :-: |
| LLaMA-2-70B | FP16 | 824 |
| LLaMA-2-70B | W8A8 | 1406 |
| LLaMA-2-70B | W4A4 | 1677 |
| LLaMA-2-70B | W4A4 | 1638 |

Reviewer comment (Experiments):
As the channel-assembly process introduces additional approximations, only applying the channel-disassembly process results in the best perplexity (which is shown in Table 2). So, to further demonstrate the necessity of applying channel assembly to keep the channel number fixed, the authors can show the inference efficiency when using the expanded channel number (only apply CD).

Author response:
**Q3**. To further demonstrate the necessity of applying channel assembly to keep the channel number fixed, the authors can show the inference efficiency when using the expanded channel number (only apply CD).

**A3**. Please refer to Q1 of the general response for the inference efficiency with and without channel assembly. Our results show that applying both channel disassembly and assembly leads to higher throughput compared to using disassembly alone as it is more hardware-friendly.

Reviewer comment (Experiments):
I'm curious how much influence the reassembly process still has as the training data amount goes up. I.e., do the authors compare with using only reconstruction-based LoRA tuning without CD, CA, and CP, under exactly the same training settings but maybe with more calibration data (e.g., 256 instead of 128).

Author response:
**Q4**. I'm curious how much influence the reassembly process still has as the training data amount goes up. I.e., do the authors compare with using only efficient error correction (EEC) without CD, CA, and CP, under exactly the same training settings but maybe with more calibration data (e.g., 256 instead of 128).

**A4**. Please refer to Q2 of the general response for the detailed results. Using EEC only without channel reassembly results in poor performance as it suffers from activation outlier issues.

Reviewer comment (Experiments):
To raise more interest in using this method, I recommend doing some evaluations on chat models.

Author response:
**Q5**. A suggestion instead of a weakness: To raise more interest in using this method, I recommend doing some evaluations on chat models.

**A5**. Following your suggestion, we apply QLLM to quantize LLaMA-2-7B-Chat and LLaMA-2-13B-Chat to 4-bit. We compared our models with the concurrent state-of-the-art quantization method, OmniQuant. We use GPT-4 to assess the performance of the quantized models on a set of 80 sample questions in the Vicuna benchmark [A]. To eliminate the potential position bias [B], we conducted the comparisons in both orders (a vs. b and b vs. a) for each pair, amounting to a total of 160 trials. From Table VI, our QLLM consistently achieves much better performance than OmniQuant, demonstrating the effectiveness of our method on chat models. We have included the results and corresponding discussions in Section F of the revised manuscript.

Table VI. Performance comparisons between QLLM and OmniQuant for chat models.
| Model            |        Case        | Former Win |  Tie  | Former Lost |
| ---------------- | :----------------: | :--------: | :---: | :---------: |
| LLaMA-2-7B-Chat  | QLLM vs. OmniQuant |  **137**   |  19   |      4      |
| LLaMA-2-13B-Chat | QLLM vs. OmniQuant |  **116**   |  24   |     20      |


**Reference**

[A] Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. https: //lmsys.org/blog/2023-03-30-vicuna/.

[B] Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv 2023.

Reviewer comment (Theory):
Sec 4.1.1 uses $\max(x_M)$ to determine the number of disassembly channels $T$ according to $\theta$, why don't we use $\max(|x_M|)$?

Author response:
**Q6**. Sec 4.1.1 uses $\mathrm{max}(x_{M})$ to determine the number of disassembly channels $T$ according to $\theta$, why don't we use $\mathrm{max}(| x_{M} |)$?

**A6**. We appreciate your attention. To clarify, we indeed utilize $\mathrm{max}(| x_{M} |)$ to determine the number of disassembly channels $T$ based on $\theta$. We have updated our manuscript accordingly.

Reviewer comment (Experiments):
I wonder whether the CD results in Table 2 use the LoRA tuning? I would like to see the results of using CD without LoRA tuning as a reference.

Author response:
**Q7**. I wonder whether the CD results in Table 2 use efficient error correction (EEC)? I would like to see the results of using channel disassembly without EEC as a reference.

**A7**. The channel disassembly (CD) results presented in Table 2 indeed incorporate our EEC. To further demonstrate the effectiveness of CD, we apply CD without EEC to obtain 4-bit LLaMA-1-13B and show the results in Table VII. We observe that the absence of both CD and EEC leads to a significant decline in the performance of the quantized model. Notably, using CD alone substantially reduces the performance degradation associated with quantization. Moreover, increasing the channel expansion ratio $\gamma$ further improves the model's performance, which strongly shows the benefits of using CD to decompose the outlier channels. By incorporating both CD and EEC, the performance improvement is even more pronounced, underscoring the efficacy of EEC in conjunction with CD. We have included the results and discussions in Section H of the supplementary material.

Table VII. Perplexity results of channel disassembly (CD) with and without efficient error correction (EEC). “$\gamma$” is the channel expansion ratio. We report the perplexity of W4A4 LLaMA-1-13B on WikiText2, PTB and C4.
| CD  |  EEC  | $\gamma$ | WikiText2 |    PTB    |    C4     |   Avg.    |
| --- | :---: | :------: | :-------: | :-------: | :-------: | :-------: |
|     |       |    -     |  1702.34  |  1853.58  |  1159.41  |  1571.78  |
| ✓   |       |   0.01   |   19.34   |   45.36   |   23.25   |   29.32   |
| ✓   |   ✓   |   0.01   | **8.31**  | **14.44** | **10.74** | **11.16** |
| ✓   |       |   0.03   |   12.11   |   24.73   |   14.38   |   17.07   |
| ✓   |   ✓   |   0.03   | **8.01**  | **13.52** | **10.27** | **10.60** |
| ✓   |       |   0.05   |   11.4    |   23.53   |   13.62   |   16.18   |
| ✓   |   ✓   |   0.05   | **7.85**  | **13.38** | **10.13** | **10.45** |
| ✓   |       |   0.07   |   11.13   |   23.47   |   13.45   |   16.02   |
| ✓   |   ✓   |   0.07   | **7.81**  | **13.35** | **10.11** | **10.42** |

Reviewer comment (Presentation):
W6A6 results are included but only referenced a few times, mostly used in a sentence like "QLLM is comparable with other methods at W6A6." Given that authors in general would like to impress readers by "being better than others" rather than "being comparable to others," W6A6 really doesn't make a strong point in the main article. Plus W6A6 has almost no real HW support. The author might want to consider adding a few more comments/discussion on W6A6 results or moving them to appendix and make the main article more concise.

Author response:
Thanks for your advice. Following your suggestion, we expanded our discussions on W6A6 results. We acknowledge that W6A6 has limited hardware support in real-world applications. However, our QLLM still demonstrates performance benefits in these settings. For example, QLLM consistently surpasses the state-of-the-art quantization method in terms of lower perplexity across all models on both WikiText2 and C4 and achieves comparable accuracy on 5 zero-shot tasks, as shown in Tables 1 and A. While the absolute performance gains with 6-bit quantization might seem modest, this is partly due to the less pronounced effect of activation outliers at this bitwidth. Our approach's benefits are more pronounced in lower bitwidth scenarios, especially under W4A4 scenarios, as demonstrated in Tables 1, 5, and A. We have included the discussions in Section 5.1 of the revision.

Reviewer comment (Evaluation):
A little clarification about the inference time benchmark would be helpful. For example, readers might be interested in comparing the QLLM results with other quantization implementations, like weight-only quantization W4A16. Take auto-gptq (https://github.com/PanQiWei/AutoGPTQ/tree/main#inference-speed) as a reference, the "speed" usually is in the unit of token/sec or msec/token. Table 4 only says "inference time" and the unit is ms, which is a little unclear.

Author response:
Following your suggestion, we compare our QLLM with weight-only quantization W4A16. Please refer to Q1 of the general response for the inference time results. Notably, even when compared to the already efficient W4A16, our QLLM still achieves a **1.1$\times$ speedup**, underscoring its superior efficiency.