Paper: QuantEase: Optimization-based Quantization for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
All the experiments were based on per-channel quantization. Nevertheless, SOTA quantization research (GPTQ and AWQ) set the baseline with sub-channel quantization with group size of 128. Author justified their choice of comparing per-channel baseline for computational efficiency, yet there were no quantitative support for the argument. Providing a runtime benchmark would be a good support.

Author response:
1. **Quantization with grouping** - Our claim that grouping can lead to slower inference is based on Table 3 of [3], which shows GPTQ with 128 grouping can be about an order of magnitude slower (in terms of inference latency) compared to ordinary GPTQ. Nevertheless, we had explored the performance of QuantEase with grouping, before deciding to not use grouping due to its overhead. These results are reported below. In these experiments, we use 3-bit quantization with either 128 or 256 grouping. Here, we report perplexity on wikitext2 datasets.  The first table below shows the results for the OPT family, and the second one for the BLOOM family.

| Algorithm  | 350m | 1.3b | 2.7b | 6.7b |
| --- | ----------- | -----------  | ----------- | ----------- |
| GPTQ (g128) | $27.69_{0.27}$ |$16.29_{0.12}$ | $13.37_{0.06}$ |$11.37_{0.03}$|
| QuantEase (g128) | $27.09_{0.07}$ | $16.01_{0.07}$| $13.54_{0.03}$ |$11.27_{0.01}$ |
| GPTQ (g256) | $29.29_{0.32}$ | $17.22_{0.04}$ | $13.84_{0.04}$ | Did not run |
| QuantEase (g256) | $27.89_{0.22}$ |$16.70_{0.06}$ | $13.79_{0.10}$ | Did not run |

We note that for the OPT family, except for the case of OPT-2.7b with 128-grouping, QuantEase outperforms GPTQ. 

| Algorithm  | 560m | 1b1 | 1b7 | 3b |
| --- | ----------- | -----------  | ----------- | ----------- |
| GPTQ (g128) | $25.77_{0.02}$ | $20.06_{0.02}$ | $17.30_{0.04}$ | $14.64_{0.01}$   |
| QuantEase (g128) | $25.43_{0.03}$ | $19.66_{0.06}$ | $16.76_{0.02}$ |$14.59_{0.02}$ |
| GPTQ (g256) |  $26.89_{0.06}$ | $20.79_{0.01}$ | $17.83_{0.04}$ |$14.81_{0.03}$ |
| QuantEase (g256) | $26.38_{0.08}$ | $20.23_{0.03}$ | $17.13_{0.03}$ |$14.79_{0.02}$  |

For the BLOOM family, we note that QuantEase outperforms GPTQ in all cases.

Reviewer comment (Theory):
Coordinate decent is often treated as approximation of Hessian based optimization. This work demonstrated CD performs better than GPTQ's Hessian. While it is encouraging, we'd like to see some explanation why it is the case.

Author response:
2. **CD vs. Hessian-based optimization**- We use cyclic CD where we optimize one coordinate at a time instead of all variables in one shot. Our single coordinate optimization does involve the hessian. CD is well-known to perform very well compared to gradient/hessian based methods when the optimization along a single coordinate is cheap, which is the case for our problem (see section 3.2 of the paper for the efficiency of our CD updates). In this sense, QuantEase is similar to, and motivated by Liblinear [1] and Improved Glmnet [2] which are classic methods from machine learning where CD works extremely effectively.

Reviewer comment (Evaluation):
Please explain why CD outperforms GPTQ's Hessian based optimization.

Author response:
3. **Why does Coordinate Descent (CD) outperform GPTQ** - This is an interesting but difficult question. Here we provide some points that differentiate the operation characteristics of CD-based QuantEase and GPTQ, which helps us to better understand the difference in performance.
* First, as we note in Section 2, CD forms a descent procedure that decreases the objective value (i.e., the quantization error) over iterations, while ensuring the solution remains feasible (i.e., quantized). Specifically, after we complete a pass of CD over all coordinates, we achieve a feasible and quantized solution However, as CD updates preserve feasibility of the solution, we can perform another pass of CD on the coordinates, which by construction of CD, decreases (or at least, does not increase) the objective. This allows us to further decrease the quantization error over multiple passes, which is what we do. On the other hand, we note that GPTQ, by construction, stops after looping once over all columns of weights. GPTQ updates can also break the feasibility of the solution which shows GPTQ is not a descent procedure.
* Second, GPTQ is based on Optimal Brain Surgeon (OBS)[4] and Optimal Brain Compression (OBC)[5] updates that were originally derived for neural network sparsification and pruning, rather than quantization. However, due to the discrete quantization structure, GPTQ uses an approximation to the exact OBS updates that have been derived before (this is discussed in details in Appendix A.1 of the paper). This can partially explain why GPTQ can lose in optimization performance. We note that as a by-product of these OBS-based updates, GPTQ requires the inversion of the Hessian, while our CD-based does not need that.

In the end, we would like to ask the reviewer to kindly revisit their evaluation and consider increasing the score.

Reviewer comment (Experiments):
I know it may be difficult getting compute resources, but if at all possible I would have liked to see results for AWQ and GPTQ on OPT-66b rather than just “OOM”. GPU memory constraints in quantization are not common, in my view. I don’t want to fault the authors if they don’t have access to larger compute resources.

Author response:
- **AWQ/GPTQ memory issues** - As mentioned in the paper, both AWQ and GPTQ run out of memory on V100 GPUs when quantizing the OPT-66b model, owing to a limited memory of 32GB. Based on feedback from the reviewer, we re-ran the experiments on OPT-66b using A100 GPUs (we have limited availability of these GPUs, which made running these experiments difficult). Following are the results. As we can see, QuantEase outperforms both GPTQ and AWQ for this model.

| Algorithm + bits | PPL (wikitext2) |
| --- | ----------- |
| GPTQ3 | $14.13_{0.43}$|
| AWQ3 | $17.94_{0.18}$ |
| QuantEase3 | $13.08_{0.38}$ |
| GPTQ4 | $9.58_{0.05}$|
| AWQ4 | $9.58_{0.01}$ |
| QuantEase4 | $9.47_{0.02}$|

Reviewer comment (Experiments):
I am aware of another paper on arXiv called "QuIP: 2-Bit Quantization of Large Language Models With Guarantees" claiming 2 bit quantization for LLM models like OPT and LLama2. I know it was released recently over the summer, but could the authors comment on this work? How does AffineQuant perform at 2 bits?

Author response:
- **Comparison to QuIP and 2-bit quantization** - Thanks for bringing our attention to this interesting paper. It seems the best results in [1] are achieved when incoherence processing is used, for example, in Table 1 of [1], the best performance is always achieved by an algorithm that uses incoherence processing. This is especially true for 2-bit quantization, where non-incoherence methods seem to perform poorly. However, incoherence processing needs projection steps---see line 3 of Algorithm 2 of [1] where quantized weights $W$ are rescaled using two orthogonal matrices $U,V$. As authors note in Section 4 of [1], producing and applying such projections requires a delicate bookkeeping of $U,V$ and possible computational overheads in the inference time.  As far as we could tell, the runtime implications of such overheads have not been explored numerically. This makes the practical applicability of incoherence processing for 2-bit quantization less clear. On the other hand, QuantEase performs standard quantization, which is commonly used in practice. 
However, we note that it seems incoherence processing can be treated as a module and integrated to any existing quantization method, as it is applied to GPTQ in [1]. As we see in [1], incoherence processing helps with boosting the performance of GPTQ, and as we show here, QuantEase can perform better than GPTQ. Therefore, we believe adding incoherence projection to QuantEase can lead to better quantization accuracy, although such a study is out of the scope of our paper.