Paper: OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The major weakness of the paper is the noticeable absence of experimental comparisons with Outlier Suppression+ (OS+) (Wei et al., 2023). Despite OmniQuant's learnable equivalent transformation being conceptually similar to OS+, the paper does not provide a direct comparison or a detailed discussion highlighting the distinctions between the two methods. Such a comparison would be invaluable for readers and would significantly augment the paper's credibility and depth.

Author response:
**Q1:** The major weakness of the paper is the noticeable absence of experimental comparisons with Outlier Suppression+ (OS+) (Wei et al., 2023).

**A1:** We have added the comparisons with OS+ in Table 2 of our manuscript. A comparison overview is as follows.

| W4A4        | Quantization | Average Acc. |
| ----------- | ------------ | ------------ |
| LLaMa-1-7B  | OS+          | 48.43        |
| LLaMa-1-7B  | OmniQuant    | 52.65        |
| LLaMa-1-13B | OS+          | 49.86        |
| LLaMa-1-13B | OmniQuant    | 54.37        |
| LLaMa-1-30B | OS+          | 52.62        |
| LLaMa-1-30B | OmniQuant    | 56.63        |
| LLaMa-1-65B | OS+          | 52.52        |
| LLaMa-1-65B | OmniQuant    | 59.23        |

Reviewer comment (Novelty):
The novelty of the proposed learnable equivalent transformation is limited as the main idea — learning channel-wise shifting and scaling — is similar to Outlier Suppression+ (OS+) (Wei et al., 2023). A comparative discussion elucidating the distinctions between OmniQuant and OS+ would be beneficial for readers. Additionally, the absence of experimental comparisons with OS+ is a notable omission that should be addressed.

Author response:
**Q2:** Distinctions with Outlier Supression+. 

**A2:** OmniQuant differs from Outlier Suppression+ in several aspects, including parameter optimization through gradient updates, block-wise optimization, expansion of equivalent transformation to attention operations, and the introduction of learnable weight clipping (LWC). Details are as follows.

- Both OmniQuant and OS+ try to find the optimal equivalent transformation parameters. OS+ solves this problem by pre-defined shifting strength and grid-searched scaling parameters. However, OmniQuant obtains both shifting and scaling parameters through gradient optimization.

- OmniQuant leverages block-wise optimization while OS+ executes grid searching through a mix of single linear layer and multiple linear layers objective(Eq.6 and Eq.7 in their paper). Block-wise optimization considers interaction within a block and can produce better performance as demonstrated in BRECQ. However, it is a challenge for OS+ to expand to block-wise quantization, which would significantly enlarge the solution space and increase the time of grid searching.

- LET expends the equivalent transformation to attention operation (Eq. (5) in our paper), but OS+ only does the equivalent transformation on linear layers. 

- OmniQuant also introduces learnable weight clipping (LWC), which further facilitates the low-bits (2,3) weight-only quantization, while Outlier Suppression only supports weight-activation quantization.

We have added this discussion in Section A2 in our revision. You can also refer [Genera Response]() for more discussion about the novelty of our paper.

Reviewer comment (Novelty):
The proposed LWC learns a clipping strength instead of clipping threshold in PACT (Choi et al., 2018) and LSQ (Esser et al., 2019). However, the paper lacks a clear articulation of the advantages of LWC over PACT and LSQ, particularly in scenarios where it is combined with LET and weights are frequently changed. A more thorough explanation of the benefits and underlying mechanics of LWC in such contexts would be beneficial. Additionally, an investigation into whether an iterative application of LWC and LET would yield performance improvements could provide valuable insights.

Author response:
**Q3:** The proposed LWC learns a clipping strength instead of clipping threshold in PACT (Choi et al., 2018) and LSQ (Esser et al., 2019). However, the paper lacks a clear articulation of the advantages of LWC over PACT and LSQ, particularly in scenarios where it is combined with LET and weights are frequently changed. A more thorough explanation of the benefits and underlying mechanics of LWC in such contexts would be beneficial.

**A3:** To clarify the advantages of LWC, we have included Figure A5 in the appendix, illustrating the dynamic change in weight ranges during training with different clipping-based methods. The figure emphasizes that LET, by dramatically altering weight ranges, poses a challenge for traditional absolute clipping thresholds or step size learning. In contrast, LWC calculates relative clipping strength, utilizing extremums after LET as proxies. This approach enables LWC to adapt to changing weight ranges more naturally, addressing the optimization of difficulty introduced by LET.

Reviewer comment (Experiments):
In Section 3.1, the authors delineate the incorporation of learnable parameters, denoted as $\gamma$ and $\beta$, to learn the clipping threshold. While the methodology is clear, the experimental section does not furnish a thorough illustration of these parameters' distribution across layers. An inclusion of this visualization would strengthen the paper.

Author response:
**Q5**: Visualization of $\gamma$ and $\beta$ distribution.

**A5:** Indeed, we have plotted the distribution of learned \gamma and \beta in Figure A1 and presented the analysis in Section A5 in the Appendix. Through the visualization in Figure A1, we can find that LWC can learn more pronounced clipping when meeting with lower quantization precision (lower bits or greater group size). Such a phenomenon makes sense due to that lower quantization precision has smaller representation space, and consequently is expected to clip more weights to obtain better trade-offs.

Reviewer comment (Experiments):
The authors apply the LET to all linear layers, with the notable exception of the second linear layer of the FFN within the proposed method. This selective application raises an intriguing question: Do all instances of LET actively contribute to the model's final performance? An investigation into the individual and cumulative impact of LET on each linear layer could provide deeper insights into the efficacy and necessity of LET across different layers of the model.

Author response:
**Q6:** An investigation into the individual and cumulative impact of LET on each linear layer could provide deeper insights into the efficacy and necessity of LET across different layers of the model.

**A6:** Good question. In our paper, we exclude the LET of the second linear layer due to the high sparsity of features after the non-linear layer leads to unstable gradients. Therefore, we have four LET pairs, represented as [ln1, (q_proj, k_proj, v_proj)], [v_proj, out_proj], [Q, K], and [ln2, fc1]. To validate the effect of each part, we respectively remove the LET from each layer and report the average perplexity and the average accuracy on 6 zero-shot tasks like Table 2. As shown in the following table, we can find that all four LETs can improve the performance, specially for the [ln1, (q_proj, k_proj, v_proj)] pair. Such results also demonstrate that the activation outliers are more serious after layer normalization layers. We have added this experiment in Table A5 of our revision.

| LLaMa-1-7B                              | Average PPL | Average Acc. |
| --------------------------------------- | ----------- | ------------ |
| FP                                      | 6.38        | 64.09        |
| W4A4 (ours)                             | 12.87       | 52.65        |
| W4A4  - [ln1, (q_proj, k_proj, v_proj)] | 19.87       | 46.79        |
| W4A4  -[v_proj, out_proj]               | 13.03       | 51.68        |
| W4A4  -[Q, K]                           | 13.34       | 51.47        |
| W4A4  -[ln2, fc1]                       | 14.47       | 51.04        |

Reviewer comment (Experiments):
In the experiments, the authors mention retaining the Softmax output at full-precision owing to its long-tail distribution. It would be insightful to know the implications of quantizing the Softmax output to 8-bit. How does this quantization impact the overall model performance and accuracy?

Author response:
**Q7:** In the experiments, the authors mention retaining the Softmax output at full-precision owing to its long-tail distribution. It would be insightful to know the implications of quantizing the Softmax output to 8-bit. How does this quantization impact the overall model performance and accuracy?

**A7**: We have conducted experiments on quantizing the Softmax output, reporting the impact on model performance. As shown in the following table, we can find that quantizing the softmax into 8-bit and 6-bit brings acceptable performance degeneration, which demonstrates that block-wise calibration can compensate for the loss of 8-bit and 6-bit Softmax quantization. However, 4-bit Softmax quantization brings significant performance loss, which requires further exploration and additional trick such as log2 quantization in RepQViT[1]. We have added this experiment in Table A7 of our revision.

| LLaMa-1-7B           | Average PPL | Average Acc. |
| -------------------- | ----------- | ------------ |
| FP                   | 6.38        | 64.09        |
| W4A4 + Softmax 16bit | 12.87       | 52.65        |
| W4A4 + Softmax 8bit  | 12.91       | 51.93        |
| W4A4 + Softmax 6bit  | 13.20       | 51.70        |
| W4A4 + Softmax 4bit  | 18.80       | 48.52        |

Reviewer comment (Experiments):
In the experimental section, the authors mention initializing the channel-wise scaling factor using SmoothQuant (Xiao et al., 2023) and the channel-wise shifting factor with Outlier Suppression+. A pertinent question arises: Is the proposed method sensitive to these initializations? It would be elucidative to explore the effects when the channel-wise scaling factor is initialized to 1 and the channel-wise shifting factor to 0. How does this affect the quantization performance?

Author response:
**Q8:**  Sensitivity to channel-wise scaling and shifting initialization.

**A8**: Good advice, sorry for the missing of this ablation. As shown in the following table, we can find that both careful initialization of scaling and shifting can improve the final performance. Specifically, scaling initialization is more important than shifting due to scaling plays the main role in alleviating outliers. We have added this experiment in Table A6 of our revision.

| LLaMa-1-7B W4A4          | Average PPL | Average Acc. |
| ------------------------ | ----------- | ------------ |
| careful initialization   | 12.87       | 52.65        |
| initialize scaling as 1  | 13.64       | 51.37        |
| initialize shifting as 0 | 12.95       | 52.22        |

Reviewer comment (Experiments):
While I highlight the advantage of LWC over AWQ, PACK, and LSQ, this evaluation for PACK and LSQ is lacking.

Author response:
As described in Algorithm 1 in Appendix, OmniQuant firstly leverages LET to transfer magnitude from activation to weight, and then quantizes the weight after LET with LWC. To demonstrate the advantage of LWC, we replace LWC in our pipeline with PACT (train the clipping threshold directly) and LSQ (train the quantization step size directly) in Table A14, showcasing that LWC consistently outperforms PACT and LSQ. Additionally, Figure A5 in the Appendix further clarifies the advantages of LWC by illustrating its adaptability to changing weight ranges introduced by LET, addressing optimization difficulties not addressed by traditional methods. Please also see Q2/A2 in responses to WDuc.

Reviewer comment (Presentation):
Authors should present Figure 1 (a) in a quantitative way. It lacks actual numbers for cost of time and performance.

Author response:
Thank you for the suggestion. We have incorporated quantitative results into Figure 1(a) to provide a more comprehensive understanding of the cost of time and performance.

Reviewer comment (Novelty):
Learnable Equivalent Transformation (LET) isn't novel and can be seen in common quantization kernels such as LLM.int8. I suggest authors to elaborate more on comparison with PACK and LSQ instead of LET.

Author response:
As mentioned in Q2, LET is the first one to introduce gradient optimization into the equivalent transformation of large language models, which significantly pushes the limitation of low-bits quantization. To demonstrate the superiority of the proposed method, we have compared it with state-of-the-art weight-only (GPTQ, AWQ) and weight-activation (LLM-QAT, Outlier Suppression +) quantization methods in our paper. Additionally, we have transferred PACT and LSQ to the weight quantization in Table A14, showcasing that LWC outperforms previous clipping-based methods. As for the activation quantization, we leverage MinMax quantization due to that directly leverages LSQ in activation quantization leading to poor performance than MinMAx quantization as shown in Table 4 of the LLM-QAT paper. You can also refer [Genera Response](https://openreview.net/forum?id=8Wuvhh0LYW&noteId=D6esYYnEi2) for more discussion about the novelty of our paper.

Reviewer comment (Experiments):
The latency benchmark only contain fp16 and OmniQuant. Strong baseline such as GPTQ is lacking.

Author response:
OmniQuant, GPTQ, and AWQ are all uniform quantization methods, allowing their quantized models to be deployed with the same latency. In our paper, the weight-only quantization kernel is not our primary focus, and we leverage existing MLC-LLM to validate the practical deployment of our proposed methods. As shown in the following table, we present the latency compared between MLC-LLM, GPTQ kernel, and AWQ kernel. Note that all of these three manners can be used to deploy OmniQuant’s quantized models.

|                  | LLaMa-7B      | LLaMa-13B     |
| ---------------- | ------------- | ------------- |
| FP               | 69.2 token/s  | 52.5 token/s  |
| 4bit-MLC-LLM     | 134.2 token/s | 91.3 token/s  |
| 4bit-GPTQ kernel | 91.30 token/s | 63.2 token /s |
| 4bit-AWQ kernel  | 155.4 token/s | 94.86 token/s |

Reviewer comment (Experiments):
How does OmniQuant runtime efficiency compare to GPTQ and AWQ?

Author response:
OmniQuant, GPTQ, and AWQ are all uniform quantization methods, allowing their quantized models to be deployed with the same latency. In our paper, the weight-only quantization kernel is not our primary focus, and we leverage existing MLC-LLM to validate the practical deployment of our proposed methods. As shown in the following table, we present the latency compared between MLC-LLM, GPTQ kernel, and AWQ kernel. Note that all of these three manners can be used to deploy OmniQuant’s quantized models.

|                  | LLaMa-7B      | LLaMa-13B     |
| ---------------- | ------------- | ------------- |
| FP               | 69.2 token/s  | 52.5 token/s  |
| 4bit-MLC-LLM     | 134.2 token/s | 91.3 token/s  |
| 4bit-GPTQ kernel | 91.30 token/s | 63.2 token /s |
| 4bit-AWQ kernel  | 155.4 token/s | 94.86 token/s |

Reviewer comment (Experiments):
What is the performance when compared to PACK and LSQ?

Author response:
As described in Algorithm 1 in Appendix, OmniQuant firstly leverages LET to transfer magnitude from activation to weight, and then quantizes the weight after LET with LWC. To demonstrate the advantage of LWC, we replace LWC in our pipeline with PACT (train the clipping threshold directly) and LSQ (train the quantization step size directly) in Table A14, showcasing that LWC consistently outperforms PACT and LSQ. Additionally, Figure A5 in the Appendix further clarifies the advantages of LWC by illustrating its adaptability to changing weight ranges introduced by LET, addressing optimization difficulties not addressed by traditional methods. Please also see Q2/A2 in responses to WDuc.

Reviewer comment (Theory):
The paper lacks a necessary detailed explanation for the motivation and effectiveness of the weight clipping method. In 3.2, the paper claims that directly employing prior LSQ and PACT would produce unsatisfactory performance, as demonstrated in LLM-QAT. However, LLM-QAT says that the outliers for activation have a notable impact, bringing difficulty for clipping while this method works on weights here. Also, how can the proposed Eq. (2) solve the problem of learning clipping thresholds for outliers? In other words, what is the optimization difficulty (concept given in 3.2) of previous techniques, and how can Eq.(2) solve it? More explanation about the motivation is preferred.

Author response:
**A1:** We appreciate your clarification regarding LLM-QAT. Indeed, the confusion arises from our initial claim about the unsatisfactory performance of LSQ and PACT in weight quantization in Sec. 3.2 and Table A14. We have modified the expression to address any potential confusion. Thanks for the correction.

Reviewer comment (Theory):
In the appendix, the paper says that LET would decline the convergence of LSQ and PACT because LET alters the weight distribution. However, weight distribution altering is a common case for LSQ and PACT in QAT. Also, combined with LET, the \gamma and \beta in the proposed Eq. (2) can also go up and down during learning as it optimizes the ratio of the changeable weight.

Author response:
**A2:** Thanks for your question. We claim that LWC reduces the optimization difficulty in learning clipping thresholds because our LWC reparameterizes the learnable  clipping threshold into the product of a learnable 0~1 clipping strength and the  maximum or minimum of weights. The benefit is that no matter how the maximum and minimum of weights change caused by LET with the training going, our LWC would always learn a clipping strength under the reference of the current maximum and minimum of weights . Such a reparameterization technique has been utilized in various areas for better training properties. For example, batch normalization decomposes the neural activation into a variable approximately following normal distribution and learnable scale and shift parameters, which improves the training stability.

Here we provide more empirical evidence to clarify the motivation and advantages of LWC. As shown in Figure A5 in the appendix, it presents the dynamic change in weight ranges (maximun minus mininum) during training with different clipping-based methods. We can see that the weight range varies a lot as the training goes on (see the curve of Min-Max) when training our OmniQuant, which is caused by our proposed another component LET. It poses a challenge for traditional absolute clipping thresholds (PACT) or step size learning (LSQ). As we can see from Figure A5 (a & b), no matter how the weight range of Min-Max increases, PACT and LSR would always retain a relatively small weight range, which would clip many regular weights values to the threshold. In contrast, LWC calculates relative clipping strength and enables it to adapt to varying weight ranges, as shown in Figure A5 (c).

 We hope that our explanations and visualizations will address your concerns.

Reviewer comment (Novelty):
What is the core novelty of the LET? I find it looks similar to Outlier Suppression+. While the paper says that Outlier Suppression+ takes a pre-defined migration strength, but this method does not and proposes to optimize the output. I'd like to point out that Outlier Suppression+ did not take a pre-defined strength and proposed to optimize the output for channel-wise scaling parameters earlier than this paper. Meanwhile, the paper also states that AWQ adopts a grid-searched channel-wise scaling, which also seems relevant to the technique in this paper. Therefore, can the paper compare these different designs and explain why the proposed way is the best? I did not find these and this could help us better understand the effectiveness.

Author response:
**A3:** We have provided a detailed comparison of LET with Outlier Suppression+ (OS+) and AWQ in the table presented below. 

- For the equivalent transformation, AWQ only considers scaling operation, while OS+ and our LET consider both scaling and shifting operation.

- For the execution position, AWQ and OS+ only carry equivalent transformation on linear layers, while LET also considers the matrix multiplication within attention (Eq. (5) in our paper). This point enlarges the solution space of equivalent transformation and brings further improvements as demonstrated in Table A5 of Appendix.

-  For the optimization, OS+ leverages pre-defined shifting, both AWQ and OS+ find the scaling factors through grid searching based on some heuristic strategy. However, LET optimized all equivalent transformation parameters through end-to-end gradient descent, which significantly improve the performance. As acknowledged by reviewer ZyAK, making the learning of equivalent transformation stable and effective is a good contribution. Additionally, OmniQuant leverages block-wise optimization objectives, while both AWQ and OS+ take layer-wise optimization objectives. Block-wise optimization considers interaction within a block and can produce better performance as demonstrated in BRECQ. However, it is a challenge for OS+ and AWQ to expand to block-wise quantization, which would significantly enlarge the solution space and increase the time of grid searching.

|      | operation     | position                 | optimization                                          |
| ---- | ------------- | ------------------------ | ----------------------------------------------------- |
| AWQ  | scale         | linear layer             | grid searching                                        |
| OS+  | scale & shift | linear layer             | grid searching for scaling & pre-defined for shifting |
| LET  | scale & shift | linear layer & attention | gradient decent                                       |

We have added this discussion in Section A2 of our revision.  In Figure A3 of Appendix, we also compare the quantization error of each block. We can find that OmniQuant significantly reduce the quantization loss compared with the grid-searching based method such as AWQ and Outlier Suppression +. You can also refer [Genera Response](https://openreview.net/forum?id=8Wuvhh0LYW&noteId=D6esYYnEi2) for more discussion about the novelty of our paper.

Reviewer comment (Experiments):
Experiments shall be compared with the paper Outlier Suppression+ because you and they work on the same quantization problem, take the same equivalent transformation, and have similar optimization designs.

Author response:
**A4:**  We have incorporated comparisons with OS+ in Table 2 of our manuscript, and offered an overview of the W4A4 quantization with six zero-shot tasks as follows. We can find that OmniQuant outperforms OS+ in a range from 4.01% to 6.68% accuracy.

| W4A4        | Quantization | Average Acc. |
| ----------- | ------------ | ------------ |
| LLaMa-1-7B  | OS+          | 48.43        |
| LLaMa-1-7B  | OmniQuant    | 52.65        |
| LLaMa-1-13B | OS+          | 49.86        |
| LLaMa-1-13B | OmniQuant    | 54.37        |
| LLaMa-1-30B | OS+          | 52.62        |
| LLaMa-1-30B | OmniQuant    | 56.63        |
| LLaMa-1-65B | OS+          | 52.52        |
| LLaMa-1-65B | OmniQuant    | 59.23        |

Reviewer comment (Novelty):
To conclude, I find the proposed two techniques are not novel and the paper lacks the necessary explanation and comparison. I see the challenge of the two techniques is how to combine the two kinds of learning together as they influence each other. However, current techniques seem can not solve this problem well. Thus, I think it would be better if the paper gives more description, and design consideration to the combination part, which might increase the novelty. For example, maybe alternately train these two techniques.

Author response:
**A6:** Thanks for the suggestion. We want to emphasize that in our approach, LWC and LET are trained simultaneously, and we have also explored an iterative training approach by iterations or epochs. The results, as presented in the following table, clearly indicate that training LWC and LET simultaneously yields the best performance. As aforementioned in Q2, LWC leverages the extremums after LET as the reference, allowing it to capture the dynamic changes in the weight range introduced by LET. This synergy between LET and LWC creates a progressive process, where both techniques reinforce each other rather than interfere. To further support this statement, we conducted an additional experiment (last row in the table), training LWC and LET iteratively with double the epochs. The results show that simultaneous training with 20 epochs achieves comparable performance to iterative training with 40 epochs. This demonstrates the effectiveness and efficiency of training LWC and LET simultaneously. We have added this experiment in Table A8 of our revision.

| LLaMa-1-7B/ W4A4/ Iterative LWC and LET | Average PPL | Average Acc. |
| --------------------------------------- | ----------- | ------------ |
| simultaneous training (OmniQuant)       | 12.87       | 52.65        |
| each iteration                          | 13.56       | 50.91        |
| each epoch                              | 13.51       | 52.06        |
| each epoch + double training epochs     | 12.80       | 52.50        |



We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you and run more experiments for any pieces of your interests in our work.

Reviewer comment (Novelty):
The novelty is limited. The overall framework is based on two existing methods.

Author response:
**Q1**: The novelty is limited. The overall framework is based on two existing methods.

**A1:** Thanks for the comment. We argue that although optimally determining clipping threshold and equivalent parameters have been explored in existing literature, our proposed OmniQuant is still novel enough by the optimal design. We clarify our novelty from $4$ aspects. 1) the first differentiable quantization framework for LLMs; 2) the novelty of LET; 3) the novelty of LWC; 4) the seamless combination of LWC and LET. Please refer [General Response](https://openreview.net/forum?id=8Wuvhh0LYW&noteId=D6esYYnEi2) for more details.

Reviewer comment (Novelty):
The learnable idea is not new. In Outlier Suppression+, the scaling has been designed to be learned via a scheme that does not depend on gradient.

Author response:
**Q2:** The learnable idea is not new. In Outlier Suppression+, the scaling has been designed to be learned via a scheme that does not depend on gradient.

**A2:** Respecting your observation, we highlight that the novelty of our learnable approach lies in differential gradient updates for various quantization parameters. As illustrated in Figure A3, OmniQuant's differential gradient-based learning significantly reduces the quantization error compared with the grid-searching method employed in Outlier Suppression+. In experiments, Table 2 in the paper demonstrates the superiority of OmniQuant to OS+.

Reviewer comment (Novelty):
Both outlier suppression+ and this paper highlight the scaling to be learned. An in-depth comparison needs to be provided, including the experimental perspective and the theoretical perspective.

Author response:
**Q3:** An in-depth comparison needs to be provided, including the experimental perspective and the theoretical perspective.

**A3:**  OmniQuant differs from Outlier Suppression+ in several aspects, including parameter optimization through gradient updates, block-wise optimization, expansion of equivalent transformation to attention operations, and the introduction of learnable weight clipping (LWC). Details are as follows.

- Both OmniQuant and OS+ try to find the optimal equivalent transformation parameters. OS+ solves this problem by pre-defined shifting strength and grid-searched scaling parameters. However, OmniQuant obtains both shifting and scaling parameters through gradient optimization.

- OmniQuant leverages block-wise optimization while OS+ executes grid searching through a mix of single linear layer and multiple linear layers objective(Eq.6 and Eq.7 in their paper). Block-wise optimization considers interaction within a block and can produce better performance as demonstrated in BRECQ. However, it is a challenge for OS+ to expand to block-wise quantization, which would significantly enlarge the solution space and increase the time of grid searching.

- LET expends the equivalent transformation to attention operation (Eq. (5) in our paper), but OS+ only does the equivalent transformation on linear layers. 

- OmniQuant also introduces learnable weight clipping (LWC), which further facilitates the low-bits (2,3) weight-only quantization, while Outlier Suppression only supports for weight-activation quantization.

We also incorporated a comparison with Outlier Suppression+ in Table 2 . An overview is as follows.

| W4A4        | Quantization | Average Acc. |
| ----------- | ------------ | ------------ |
| LLaMa-1-7B  | OS+          | 48.43        |
| LLaMa-1-7B  | OmniQuant    | 52.65        |
| LLaMa-1-13B | OS+          | 49.86        |
| LLaMa-1-13B | OmniQuant    | 54.37        |
| LLaMa-1-30B | OS+          | 52.62        |
| LLaMa-1-30B | OmniQuant    | 56.63        |
| LLaMa-1-65B | OS+          | 52.52        |
| LLaMa-1-65B | OmniQuant    | 59.23        |

Reviewer comment (Experiments):
The optimization based on little data and backward propagation makes the learning easy to be overfitted. More validation should be conducted to prove the generalization ability of this learning.

Author response:
**Q4:** The optimization based on little data and backward propagation makes the learning easy to be overfitted. More validation should be conducted to prove the generalization ability of this learning.

**A4:** Acknowledging the potential for overfitting, we highlight that OmniQuant mitigates this issue through a constrained solution space. Specifically, optimization focuses on LET and LWC rather than all weights. Our results in Table A11 of the Appendix demonstrate OmniQuant's superior performance even with only 16 calibration samples. Additionally, we present performance on MMLU to further showcase the generalization ability of our approach.

| LLaMa-1-7B (FP: 38.41 ) | W4A16g128 | W3A16g128 | W2A16g128 | W4A4  |
| ----------------------- | --------- | --------- | --------- | ----- |
| RTN                     | 37.37     | 33.43     | 22.55     | 23.31 |
| GPTQ                    | 35.39     | 30.53     | 23.83     | -     |
| AWQ                     | 37.71     | 35.43     | 22.58     | -     |
| Outlier Suppression+    | -         | -         | -         | 25.72 |
| OmniQuant               | 37.50     | 35.60     | 26.03     | 26.93 |

Reviewer comment (Experiments):
There are some new kinds of ways to decompose the outliers, e.g., https://arxiv.org/abs/2310.08041. Comprehensive experiments are suggested to further enrich the validation.

Author response:
**Q5:** There are some new kinds of ways to decompose the outliers, e.g., https://arxiv.org/abs/2310.08041. Comprehensive experiments are suggested to further enrich the validation.

**A5:** Thanks for your suggestion. Your referred paper QLLM (https://arxiv.org/abs/2310.08041) alleviates outlier channels by disassembling instead of scaling. In my view, Outier disassembling only considers the outlier channels. However, LET in our paper, as shown in Figure A2, can also adjust the range of normal channels to achieve a more quantization-friendly distribution. Therefore, the combined disassembling operation of QLLM and LET of our paper may further boost the performance of quantization models.  We will undertake this exploration after QLLM releases its code and engage in discussions with the QLLM team for further insights.

We sincerely appreciate the time and efforts you have dedicated to reviewing our paper. Should you have any further inquiries, please let us know and we are more than delighted to discuss with you.

[1] Brecq: Pushing the limit of post-training quantization by block reconstruction

Reviewer comment (Experiments):
It would be beneficial to have additional experiments on more complex tasks. I am wondering how OmniQuant impacts the reasoning ability of LLMs, which can be evaluated by MMLU. GPT-4 evaluation is a bit ad-hoc nowadays, and there are also several better benchmarks to measure the instruction-tuned models performance, such as MT-Bench or AlpacaEval (correction: should be AlpacaEval instead of AlpacaFarm). Evaluating some stronger chatbots like Vicuna-v1.5 on them should be conducted.

Author response:
**Q1**: Results on MMLU.

**A1:** As shown in the following table, we report the average accuracy on MMLU, showcasing that OmniQuant consistently achieves comparable or superior performance in reasoning ability. We have added this experiment in Table A16 of our revision.

| LLaMa-1-7B (FP: 38.41 ) | W4A16g128 | W3A16g128 | W2A16g128 | W4A4  |
| ----------------------- | --------- | --------- | --------- | ----- |
| RTN                     | 37.37     | 33.43     | 22.55     | 23.31 |
| GPTQ                    | 35.39     | 30.53     | 23.83     | -     |
| AWQ                     | 37.71     | 35.43     | 22.58     | -     |
| Outlier Suppression+    | -         | -         | -         | 25.72 |
| OmniQuant               | 37.50     | 35.60     | 26.03     | 26.93 |



**Q2:**  Evaluating instruction-tuned models on MT-Bench or AlpacaFarm and evaluating stronger Chatbot.

**A2:** We have quantized Vicuna-v1.5 using W3A16g128 quantization and compared the performance of different methods through MT-Bench. The results reveal that OmniQuant consistently outperforms RTN and AWQ. 

| Vicuna-v1.5-7B-W3A16g128 | OmniQuant Win | Tie  | OmniQuant Lost | Win rate |
| ------------------------ | ------------- | ---- | -------------- | -------- |
| Omniquant v.s. RTN       | 101           | 239  | 20             | 83.4 %   |
| Omniquant v.s. AWQ       | 70            | 225  | 65             | 51.8%    |

Reviewer comment (Experiments):
Some strong related work is not discussed or compared, such as SpQR [1] and SqueezeLLM [2]. For instance, SqueezeLLM outperforms the proposed approach for wiki and c4 perplexity on LLaMA v1 7b and 13b under 3-bit and 4-bit weight-only quantization settings (see table 1 in their paper). Additional discussion and results should be added to compare OmniQuant with them.

Author response:
**Q3:** Discussion and comparison with related work- SpQR and SqueezeLLM

**A3:** Thanks for your suggestion, we have included the discussion and comparison with SqQR and SqueezeLLM in Section A7. However, we would like to emphasize two crucial points. Firstly, SpQR and SqueezeLLM are concurrent works, both available on arXiv in June 2023, which falls within four months of the ICLR submission deadline (Please refer to the policy in the last Q&A at https://iclr.cc/Conferences/2024/ReviewerGuide ). Secondly, Although OmniQuant may perform slightly worse than SqueezeLLM, our OmniQuant focuses on uniform (INT) quantization which offers simplicity and flexibility and has strong hard-ware support. Moreover, OmniQuant achieves competitive performance in both weight-only quantization and weight-activation quantization. In contrast, SpQR and SqueezeLLM only support weight-only quantization. We believe this distinction adds valuable context to the comparison.