Paper: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (ICLR-2024)

Reviewer comment (Experiments):
The paper overlooks in-depth comparisons with established methods like AdaRound and FlexRound.

Author response:
1 Your concern is quite reasonable.  As mentioned earlier, we haven't compared with them due to several issues. However, we have demonstrated that sign gradient descent can perform well in various scenarios without needing to tune any optimizer hyperparameters. Additionally, in many instances of W4G-1 and W4G128, we have achieved results that are quite comparable to FP16's with the assistance of v2. We hypothesize that even if other methods are superior in some scenarios, their advantages in these scenarios are minimal and likely slower

Reviewer comment (Experiments):
The performance comparisons between GPTQ and the proposed method are unfair since act-order was not enabled. This omission potentially skews the results, leading to instances where GPTQ underperforms compared to rounding-to-nearest (RTN) in some cases (W4 in Table 1).

Author response:
**4 Concerns of GPTQ act-order**

Thank you for the information, and we apologize for not incorporating the update introduced in June 2023. We had cloned the code some time back, and misunderstood the no grouping setting. We have re-run the experiments at W4G-1 and have included the new data in Tables 1, 3, 8, and 9. The results consistently highlight the advantages of SignRound. We plan to include additional data in the future.

Reviewer comment (Evaluation):
GPTQ, by utilizing second-order information, offers an efficient solution to the weight quantization problem. It would benefit for the authors to provide a clearer distinction of the advantages offered by the proposed method, especially when there are instances where it lags behind GPTQ, as evidenced in Tables 3, 4, and 5. Additionally, omitting a comparative analysis on training time introduces ambiguity, making it challenging to ascertain the relative efficiency of the two methods.

Author response:
**5 Questions about distinction of the advantages**

Although GPTQ is well motivated by optimal brain surgery, it is our understanding that it inherits two disadvantages. For example, let's split the tensor of weight into A and B.  After quantizing A, GPTQ adjusts B to minimize the error introduced by the quantization of A. However, if quantizing A results in a significant error, such as when quantizing A to 1 bit, it becomes challenging to adjust B. In contrast, SignRound keeps reducing the error of quantized A in mind. Additionally, after quantizing B, there are no remaining weights available to adjust and compensate for this error, whereas SignRound does not face this issue.

Given the numerous models and scenarios, we believe that SignRound could be beneficial, and we think in many scenarios, SignRound could be combined with other SOTA methods.



**6 Questions about running time**

We have included additional data in Table 13. Since our code has been optimized,  we use the latest data. For instance, when quantizing llama-65b, SignRound  takes 1.85 hours.

Reviewer comment (Evaluation):
Referring to Table 4, it is noteworthy that the proposed method underperforms compared to RTN for both W4G128 LLaMA-7B and W3G128 LLaMA-7B configurations. A more in-depth exploration or justification for this discrepancy would enhance the paper's clarity.

Author response:
**7 Questions about scenarios inferior to RTN**

To be honest, we were unable to determine the root cause of the issue. We have two hypotheses that may explain the problem. Firstly, the output reconstruction task is  be a proxy task, and lower error may not always provide an advantage for the real task. Secondly, the limited size of the calibration dataset might introduce some bias.

Reviewer comment (Experiments):
The comparisons presented in Table 8 between AWQ and the proposed method appear unfair due to disparities in the calibration datasets. For a comprehensive assessment of the proposed method's efficacy, it would be better for the authors to provide fair comparisons.

Author response:
**8 Concerns about fair comparison with AWQ** 

we have update the data of AWQ calibrated with pile-10k in Appendix B

Reviewer comment (Writing):
before equation 6, the equation is missing an "s"

Author response:
Apologies for any confusion, but to our understanding, equation 6 should not include 's', while equation 7 should incorporate 's'.

Reviewer comment (Evaluation):
why is there such high C4 perplexity for transformer block quantization for W3G128? Do the numbers indicate some form of instability during sign SGD?

Author response:
With the observation that the c4 perplexity of GPTQ for Llamav2 is reasonable, while its ptb perplexity is shown as NAN in Table 16, and SignRound has the same issue,  we hypothesize this is likely because perplexity is highly sensitive to outliers. The equation is represented as $exp(-(sum(log p))/seqlen)$, where a low probability p for a single token results in a high perplexity value.

It appears that SignRound introduces a higher number of outliers, and at present, we haven't been able to pinpoint the underlying cause. However, this issue could be  mitigated by incorporating minmax tuning as shown in Appendix and Table 3. Alternatively, tuning the hyperparameters, which we have confirmed to be effective in some outlier scenarios, could also alleviate this issue.

Reviewer comment (Novelty):
The paper essentially seems to apply signed gradient descent (which is not new) to the standard layer-/block-wise rounding problem considered by various LLM PTQ papers. Hence, the overall novelty is low.

Author response:
We have proposed a straightforward yet effective solution for the problem at hand, utilizing existing tools. This suggests that our approach, which may have been overlooked by the community, offers a novel solution. Furthermore, considering that GPTQ and equivalent transformations like AWQ are still the mainstream methods for weight-only quantization of LLMs, we are confident that our solution will be valuable to the community and could inspire further advancements.

Reviewer comment (Experiments):
GPTQ Activation-reordering can also be performed without any impact on inference performance (see the official GPTQ repo, option `--static-groups`). Further, if there is no grouping, reordering has no impact on inference. LLaMa1-7B and OPT-66B are known to be GPTQ outliers, for which reordering should be enabled to conduct a fair comparison.

Author response:
Thank you for the information, and we apologize for not incorporating the update introduced in June 2023. We had cloned the code some time back, and misunderstood the no grouping setting. We have re-run the experiments at W4G-1 and have included the new data in Tables 1, 3, 8, and 9. The results consistently highlight the advantages of SignRound. We plan to include additional data in the future.

Reviewer comment (Experiments):
The paper argues that signed gradient descent is preferable over standard straight-through QAT (applied to the layer-wise quantization problem, like ZeroQuant) for this application, but does not provide any ablation studies supporting that point.

Author response:
We have added the ablation study in Section 4.3 and Table 6. 
The motivation is: The boundary of V in Eq.2 is constrained, within the range of [-0.5, 0.5], where only the value of altering the rounding is significant. These characteristics offer three advantages for signed gradient descent.  Firstly, the optimal value is not a single float but typically a large region, negating the need for the gradient magnitude to converge to an exact point. Secondly, due to the limited boundary, we can traverse this space within a constrained number of steps, while the gradient magnitude may vary significantly, making it challenging to determine an appropriate step size within a limited number of iterations. Thirdly, signed gradient descent is inherently intuitive, allowing for easy adjustment of the step size (learning rate). For instance, we employed the same optimizer hyperparameters in all of our experiments, which include 400 steps and a learning rate of 0.0025, with linear weight decay. This ensures that 400*0.0025/2=0.5 covers the range of [-0.5, 0.5]. Furthermore, the signed gradient descent method is quite lightweight, which is crucial since LLMs demand significant resources.

Reviewer comment (Evaluation):
Based on Table 5, it appears that for the largest and most interesting models for compression applications, SignRound seems to perform very similar to GPTQ and in some cases even worse.

Author response:
Larger models typically exhibit greater resilience to compression, and recent state-of-the-art methods often deliver results that closely match those of FP16, leaving little room for improvement.

We present the average improvements of all the quantization configurations introduced by SignRound (without minmax tuning) over RTN, GPTQ (choosing the best without actorder and with actorder if we have the data), and FP16, respectively, as tuples of (ours-RTN, ours-GPTQ, ours-FP16):

- Llama30b (0.32, 0.07, -0.89)
- Llama60b (0.01, -0.40, -1.67)
- opt30b-ours-seq512 (2.67, -0.10, -0.82)
- opt66b (30.84, 0.15, -1.14).

It is important to note that SignRoundv2 (with minmax tuning) could achieve better results in most scenarios, as demonstrated in Table 1 and Table 9.

Furthermore, given that our method is based on lightweight training and our default hyperparameters are identified through experiments on smaller models, there is potential for further tuning.

Reviewer comment (Reproducibility):
The code is not available in the Supplementary material.

Author response:
We have uploaded the code

Reviewer comment (Experiments):
Is GPTQ using the same amount of samples as SignRound in your comparisons?

Author response:
We use the default 128 samples for a fair comparison. In GPTQ, the batch size is equivalent to the total samples, and scaling it to 512 samples would lead to a 1X increase in runtime. In contrast, our runtime is comparable to their default setting, as demonstrated in table 13 of Appendix F.

Reviewer comment (Evaluation):
Do you have any explanation for the surprisingly poor perplexity performance on Llama" models in Table 3? Related to that, how comes that those models still exhibit comparable or better ZeroShot performance.

Author response:
With the observation that the c4 perplexity of GPTQ for llamav2 is reasonable, while its ptb perplexity is shown as NAN in Table 16, and SignRound has the same issue,  we hypothesize this is likely because perplexity is highly sensitive to outliers. The equation is represented as $exp(-(sum(log p))/seqlen)$, where a low probability p for a single token results in a high perplexity value. On the other hand, the accuracy does not exhibit such issues, as each value falls within the range of [0,1].

It appears that SignRound introduces a higher number of outliers, and at present, we haven't been able to pinpoint the underlying cause. However, this issue could be  mitigated by incorporating minmax tuning as shown in Appendix A and Table 3. Alternatively, tuning the hyperparameters, which we have confirmed to be effective in some outlier scenarios, could also alleviate this issue.

Reviewer comment (Experiments):
The runtime comparisons in Table 12 stop at 13B size, how long does your method take for the largest models?

Author response:
We have added supplementary data to Table 13. Our code has been optimized, allowing us to utilize the latest data. For example, when quantizing llama-65b, SignRound only takes 1.85 hours.