Paper: LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR-2024)

Reviewer comment (Theory):
The authors assert the efficiency of LQ-LoRA based on empirical evidence, yet lack theoretical backing. To strengthen the credibility of the algorithm, a comparison might be beneficial, especially with methods that initialize the Q(W) + L1L2 matrix in a manner that closely mirrors the original pretrained matrix W. Consider, for instance, the use of GPTQ as a compensatory mechanism.

Author response:
**Comparison with GPTQ-based Methods**

Thanks for the question! We actually did implement and compare against GPTQ-LoRA. We called this "LREC" in the paper following this paper (https://arxiv.org/pdf/2306.08162.pdf). We find that GPTQ-LoRA underperforms both QLoRA and LQ-LoRA (Table 1).

(We now realize that this naming caused some confusion. Thanks for pointing that out --- we will clarify this in the paper.)

Reviewer comment (Novelty):
It appears that this paper serves as an expanded or refined rendition of the Q-LoRA paper. As such, it seemingly inherits the same limitation, notably the inference overhead, given that this approach must fail to integrate the LoRA layer into an existing linear layer.

Author response:
**Limitations from QLoRA**

This is a very good point! But we want to point out that inference with quantized + low-rank is not necessarily slower, especially in batch = 1 setting. Smaller quantized matrices allow us to reduce data movement between SRAM and HBM of the GPU. For example, in the latest releases of `bitsandbytes` (the low-level engine behind QLoRA) [1], they claimed their matrix-vector multiplication between quantized matrix and dense vector (the one needed during inference) could be competitive and even outperformed dense matrix-vector multiplication. 

[1] https://github.com/TimDettmers/bitsandbytes/releases

*Integrating LoRA Layers Into Linear Layers*

Given the potential inference speed-ups, we think that practitioners could use the "quantized + low-rank" models without merging. Such models could be just as fast (as discussed above) as merged dense models and more memory-efficient. In that regard, you could (loosely speaking) consider this as an alternative PTQ method.

Based on this potential of LQ-LoRA as an alternative to PTQ, we performed additional experiments by using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. Specifically, we train the LQ-LoRA (Fisher, 2.75-bit, 64-rank) for one epoch on two C4 partitions and WikiText-2. We then further quantize the LoRA components themselves to 8-bit (NF8). Please see the Table below for the results, where the numbers other than LQ-LoRA are from OmniQuant [1]. Note that we use the term "effective bits" to denote bits per parameter that treat the LoRA components as "overheads" for comparing with methods without LoRA; LoRA components (with NF-8 quantization) amount to about 0.1-0.2 extra bits per parameter.

| Base Model | Method                           | Effective Bits (7B, 70B) | C4 (7B) | C4 (70B) | WikiText2 (7B) | WikiText2 (70B) |
| -------- | ---------------------------------- | ----------- | ---- | ---- | ---- | ---- |
| LLaMA-2 | Dense                               | 16          | 6.97 | 5.52 | 5.47 | 3.31 |
| LLaMA-2 | RTN                                 | 3.15        | 8.40 | 6.02 | 6.66 | 3.97 |
| LLaMA-2 | GPTQ            | 3.15        | 7.89 | 5.85 | 6.29 | 3.85 |
| LLaMA-2 | AWQ               | 3.15        | 7.84 | -    | 6.24 | -    |
| LLaMA-2 | OmniQuant  | 3.15        | 7.75 | 5.85 | 6.03 | 3.78 |
| LLaMA-2 | LQ-LoRA (Fisher)   | 2.95, 2.85 | 7.60 | 5.88 | 5.67 | 3.65 |

[1] https://arxiv.org/abs/2308.13137

These results indicate that LQ-LoRA can also be used as a state-of-the-art PTQ method. Given the promise of LQ-LoRA as a potential alternative to PTQ, we will include this comparison against PTQ results in the next iteration of the paper.

Reviewer comment (Novelty):
Similarly, I would like to raise a query about the paper's novelty. While this method undeniably enhances the current approach (Q-LoRA), from a PEFT perspective, there could be superior methods, particularly concerning inference challenges. On the topic of novelty, I await the insights of fellow reviewers.

Author response:
**Novelty**

*Similarly, I would like to raise a query about the paper's novelty. While this method undeniably enhances the current approach (Q-LoRA), from a PEFT perspective, there could be superior methods, particularly concerning inference challenges. On the topic of novelty, I await the insights of fellow reviewers.*

We want to reiterate three main contributions on top of QLoRA:
1. We adopt a matrix decomposition view of initializing the quantized and low-rank components of LoRA adaptation. (Note that QLoRA does not perform explicit matrix decomposition)
2. Our ILP formulation enables us to search for the mixed-configuration quantization that fits specific resource requirements (e.g., 2.75 bits per parameter).
3. We extended the two methods above to a data-aware setting by incorporating the sensitivities of parameters.

Reviewer comment (Novelty):
The idea of decomposing a weight matrix into a low-rank component and a quantized component had already been proposed in QLoRA (Dettmers et. al, 2023).

Author response:
**Novelty with respect to QLoRA**

We want to note that QLoRA does not perform explicit matrix decomposition. It performs quantization of the original weight matrix and learns additive low-rank updates. In contrast, we perform explicit matrix decomposition of the original matrix into low-rank and quantized components via a matrix reconstruction objective.

Reviewer comment (Experiments):
A few ablations / baselines could be added, to make clearer where the gains of the method come from.

Author response:
**Ablations**

*How important is the ILP to LQ-LoRA? Can you show the performance of LQ-LoRA without the ILP?*

One of the main usages of ILP is to flexibly quantize the model to meet specific memory constraints (e.g., 2.75 bits/parameter). That being said, LQ-LoRA is effective even without the ILP. For example, Table 4 shows LQ-LoRA without ILP. Here, we use NF3 quantization configuration --- NF3 reuses the configuration of the QLoRA's NF4 quantization with 3-bit first-level quantization. 

*Can you show the performance of the regular LoRA method (no quantization), and also quantization (at different bit-rates) without LoRA, in Table 2?*

We want to note that the QLoRA paper already demonstrated little performance loss between regular LoRA and QLoRA (4bit). Hence, we can think of QLoRA (4bit) performance as a proxy for regular LoRA.

However, based on your suggestion/question, we performed additional experiments using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. Specifically, we train the LQ-LoRA (Fisher, 2.75-bit, 64-rank) for one epoch on two C4 partitions and WikiText-2. We then further quantize the LoRA components themselves to 8-bit (NF8). Please see the Table below for the results, where the numbers other than LQ-LoRA are from OmniQuant [1]. Note that we use the term "effective bits" to denote bits per parameter that treat the LoRA components as "overheads" for comparing with methods without LoRA; LoRA components (with NF-8 quantization) amount to about 0.1-0.2 extra bits per parameter.

| Base Model | Method                           | Effective Bits (7B, 70B) | C4 (7B) | C4 (70B) | WikiText2 (7B) | WikiText2 (70B) |
| -------- | ---------------------------------- | ----------- | ---- | ---- | ---- | ---- |
| LLaMA-2 | Dense                               | 16          | 6.97 | 5.52 | 5.47 | 3.31 |
| LLaMA-2 | RTN                                 | 3.15        | 8.40 | 6.02 | 6.66 | 3.97 |
| LLaMA-2 | GPTQ            | 3.15        | 7.89 | 5.85 | 6.29 | 3.85 |
| LLaMA-2 | AWQ               | 3.15        | 7.84 | -    | 6.24 | -    |
| LLaMA-2 | OmniQuant  | 3.15        | 7.75 | 5.85 | 6.03 | 3.78 |
| LLaMA-2 | LQ-LoRA (Fisher)   | 2.95, 2.85 | 7.60 | 5.88 | 5.67 | 3.65 |

[1] https://arxiv.org/abs/2308.13137

These results indicate that LQ-LoRA can also be used as a state-of-the-art PTQ method. Given the promise of LQ-LoRA as a potential alternative to PTQ, we will include this comparison against PTQ results in the next iteration of the paper.

Reviewer comment (Experiments):
How important is the ILP to LQ-LoRA? Can you show the performance of LQ-LoRA without the ILP?

Author response:
*How important is the ILP to LQ-LoRA? Can you show the performance of LQ-LoRA without the ILP?*

One of the main usages of ILP is to flexibly quantize the model to meet specific memory constraints (e.g., 2.75 bits/parameter). That being said, LQ-LoRA is effective even without the ILP. For example, Table 4 shows LQ-LoRA without ILP. Here, we use NF3 quantization configuration --- NF3 reuses the configuration of the QLoRA's NF4 quantization with 3-bit first-level quantization.

Reviewer comment (Experiments):
Can you show the performance of the regular LoRA method (no quantization), and also quantization (at different bit-rates) without LoRA, in Table 2?

Author response:
*Can you show the performance of the regular LoRA method (no quantization), and also quantization (at different bit-rates) without LoRA, in Table 2?*

We want to note that the QLoRA paper already demonstrated little performance loss between regular LoRA and QLoRA (4bit). Hence, we can think of QLoRA (4bit) performance as a proxy for regular LoRA.

However, based on your suggestion/question, we performed additional experiments using LQ-LoRA as a potential alternative to PTQ methods by training on a larger calibration dataset. Specifically, we train the LQ-LoRA (Fisher, 2.75-bit, 64-rank) for one epoch on two C4 partitions and WikiText-2. We then further quantize the LoRA components themselves to 8-bit (NF8). Please see the Table below for the results, where the numbers other than LQ-LoRA are from OmniQuant [1]. Note that we use the term "effective bits" to denote bits per parameter that treat the LoRA components as "overheads" for comparing with methods without LoRA; LoRA components (with NF-8 quantization) amount to about 0.1-0.2 extra bits per parameter.

| Base Model | Method                           | Effective Bits (7B, 70B) | C4 (7B) | C4 (70B) | WikiText2 (7B) | WikiText2 (70B) |
| -------- | ---------------------------------- | ----------- | ---- | ---- | ---- | ---- |
| LLaMA-2 | Dense                               | 16          | 6.97 | 5.52 | 5.47 | 3.31 |
| LLaMA-2 | RTN                                 | 3.15        | 8.40 | 6.02 | 6.66 | 3.97 |
| LLaMA-2 | GPTQ            | 3.15        | 7.89 | 5.85 | 6.29 | 3.85 |
| LLaMA-2 | AWQ               | 3.15        | 7.84 | -    | 6.24 | -    |
| LLaMA-2 | OmniQuant  | 3.15        | 7.75 | 5.85 | 6.03 | 3.78 |
| LLaMA-2 | LQ-LoRA (Fisher)   | 2.95, 2.85 | 7.60 | 5.88 | 5.67 | 3.65 |

[1] https://arxiv.org/abs/2308.13137

These results indicate that LQ-LoRA can also be used as a state-of-the-art PTQ method. Given the promise of LQ-LoRA as a potential alternative to PTQ, we will include this comparison against PTQ results in the next iteration of the paper.

Reviewer comment (Experiments):
I found it unusual that while the quantization configurations were optimized extensively (chosen via ILP), the rank of the low-rank components was kept fixed at 64 for the vast majority of experiments (except for Table 4). Perhaps the rank could also be chosen with the ILP?

Author response:
**ILP for Rank Selection**

This is a very good/interesting suggestion! We have looked into this by ILP-searching the bits and ranks together. One practical challenge is measuring the "storages/errors"  of the ranks of low-rank components. This is important to trade off bits of the quantized matrices and ranks of LoRA. Notice that LoRA components will be fine-tuned; searching the ranks using errors at initialization will unnecessarily favor more bits at the cost of lower ranks. We might be able to formulate this as a bi-level optimization problem (first searching the ranks and then searching the bits), but we will leave this as future work.

Reviewer comment (Novelty):
Is the only difference between QLoRA+ILP, and LQ-LoRA, the initialization?

Author response:
*Is the only difference between QLoRA+ILP, and LQ-LoRA, the initialization?*

Yep!

Reviewer comment (Evaluation):
Does the ILP budget, as well as the "bits per param" column, also consider the low-rank components?

Author response:
*Does the ILP budget, as well as the "bits per param" column, also consider the low-rank components?*

This table does not take into account low-rank components since we use rank = 64 for our QLoRA and LREC (GPTQ + LoRA) baselines. However, the PTQ table above **does** take into account the contribution from the low-rank components to ensure fair comparison against other PTQ methods. We find that the LoRA components (if quantized) add about ~0.2 bits. We will clarify this further!

Reviewer comment (Evaluation):
One of the contributions of this work is the ILP formulation for dynamic bit allocation across layers. This dynamic configuration of quantization parameters (e.g., bit-width, block size) subject to a total target memory budget is quite interesting. Hardware-wise, the proposed strategy necessitates mixed-precision compute (i.e., different bits for different layers). Even without the ILP, the $Q + L_1L_2$ decomposition requires handling $Q$ in low-precision format, whereas $L_1$ and $L_2$ is high (original)-precision format (e.g., $16$-bit). Moreover, the ILP formulation outputs bit-budget allocation at quite fine resolutions like $2$ or $3$ bits. My concern is that such low precision is not easily available as current hardware primitives; i.e., we can find a $4$-bit GPU, but can we find a $2$-bit GPU? I understand that the simulations are done in PyTorch that provide the flexibility of finer precisions (the authors also mention this on Page 5, "Implementation"). It would be worth discussing that this is a significant bottleneck in the deployment of this scheme for actual benefits in hardware. Please note again that I do not see this as a significant drawback of this paper, but it is important that the authors acknowledge this.

Author response:
**Mixed Precision & Hardware**

We note that our work is in the **weight only** quantization regime (as are QLoRA, GPTQ, etc.), where only the weights (and not the activations) are quantized to lower bits. The actual matmul is done in full precision after dequantization. 

Concretely, we use just-in-time de-quantize the (quantized) matrix $\mathbf{Q}$, execute matrix operations (e.g., `matmul(X, Q) ==> matmul(X, dequantize(Q))`), and throw away the de-quantized matrix. This supports arbitrary quantization, and the main technical difficulty is how to do de-quantization quickly. We outlined more details in the appendix.

Although weight-only quantization cannot make use of (faster and more energy-efficient) lower-precision matmuls, weight-only quantization still has two practical benefits. First, at a macro level, having a smaller model (due to quantization) allows us to increase the batch size. This reduces the need for expensive data/model parallelism, which requires cross-device or even cross-node communication. Second, at a micro level, smaller quantized matrices allow us to reduce data movement between SRAM and HBM of the GPU. As an example, QLoRA's matrix multiplication implementation could be faster than dense matrix multiplication [1].

[1] https://github.com/TimDettmers/bitsandbytes/releases

Reviewer comment (Theory):
There is a recent work on joint low-rank and quantized decomposition of a matrix: "Matrix Compression via Randomized Low Rank and Low Precision Factorization" (Saha, Srivastava & Pilanci) (https://arxiv.org/abs/2310.11028). This work derives Frobenius norm error bounds for the low-rank decomposition of a matrix, in which the low-rank factors are also quantized. LQ-LoRA considers the low-rank factors to be in high-precision. This work is complementary in the sense that the low-rank factors of the LQ-LoRA decomposition can also be quantized (in case the hardware is limited to low-precision only). This will also help in circumventing the mixed-precision hardware issue mentioned in point 1 above (i.e., now $W$, $L_1$, and $L_2$ — all three can be in the same precision). The analysis techniques proposed in this work can also be used to upper bound the Frobenius norm error of LQ-LoRA in order to make it more theoretically principled.

Author response:
**Related paper on Joint Low-rank and Quantized Matrix Decomposition**

Thanks for the suggestion; this is an interesting paper! We will discuss this paper in the related works section.

We primarily considered the cases in which $\mathbf{L}_1$ and $\mathbf{L}_2$ are floating-point parameters. This is necessary because we want to fine-tune these parameters. That being said, the techniques introduced in that paper paper might be helpful for other (future) use cases. For example, we could post-training merge $\mathbf{Q}, \mathbf{L}_1, \mathbf{L}_2$, and re-decompose them into three separate quantized matrices for inference.

Reviewer comment (Writing):
The authors mention: "LoRA obviates the need to allocate memory for storing gradients and optimizer states" — shouldn't it be stated as LoRA does not require us to store gradients for all parameters of the LLM, but only for the low-rank adapters, for which the number of parameters can be a fraction of the total number of LLM parameters?

Author response:
**Question 1**

You are correct -- we will clarify this; thanks!

Reviewer comment (Experiments):
Why did the authors choose NormalFloat instead of a simple RTN quantization scheme? In principle, it seems that LQ-LoRA can be extended with any quantization scheme, if I'm not mistaken? And RTN has benefits over NF, such as no Gaussian modeling assumptions on the weights?

Author response:
**Question 2**

You are correct in that the techniques in this paper are not specific to NF. QLoRA paper demonstrated the superior quality of such a scheme compared to classical integer quantization. Our preliminary experiments confirmed this, and hence, we chose NF.

We conducted the following experiment to be more concrete about NF vs INT. We ran LQ (i.e., Section 3.1) on LLaMA-2 7B's dense parameters with `rank=64` and NF3/INT3/NF4/INT4 quantization configuration (i.e., `b0=3/4, b1=8, b2=fp32, B0=64, B1=256`), and measure the reconstruction errors (i.e., $||\mathbf{W} - (\mathbf{Q} + \mathbf{L}_1 \mathbf{L}_2) ||_2^2$). The table below shows that NF indeed performed better.

| Method      | Reconstruction error (unit: $10^4$) |
| ----------- | ----------- |
| LQ with NF3      | 7.12       |
| LQ with INT3     | 11.6        |
| LQ with NF4      | 1.42       |
| LQ with INT4     | 2.02        |

Reviewer comment (Theory):
Where does the value $\delta = \frac{1}{2}\left(\frac{1}{30} + \frac{1}{32}\right)$ come from? Was it proposed in the NF paper? Does this value remain the same if more flexible quantization resolutions are considered (as is done in this paper in the ILP, but not probably in the NF paper)?

Author response:
**Question 3**

This is the same $\delta$ in QLoRA, and we treated it as a constant for all settings. We will clarify further in the next iteration of the paper.

Reviewer comment (Theory):
Page 3: How is NF quantization **lossless**?

Author response:
**Question 4**

We _defined_ $\mathbb{Q}$ as the set of matrices that can be quantized (Equation 1 is a minimization problem, not an exact decomposition problem). Thanks for pointing out the confusion; we will clarify this.

Reviewer comment (Writing):
In Alg. 1 pseudocode — Does $B$ denote the quantization budget or the total number of blocks (as used in the main text)?

Author response:
**Question 6**

Good catch -- it's meant for "budget" but shared similar symbols with $B_0$ and $B_1$.

Reviewer comment (Writing):
Page 6: What does it mean by: "weight matrix $\bf F$ has homogeneous rows or columns"? Please clarify in the main text.

Author response:
**Question 7**

We say a matrix $\mathbf{F}$ has homogenous rows or columns when either rows or columns of the matrix have identical values. We will clarify this.

Reviewer comment (Presentation):
Is Table 4 the data-aware or the agnostic variant? Also in Table 4: LQ-LoRA with rank $64$ on C4 has ppl $7.93$. Unless I'm mistaken, shouldn't there be a corresponding $7.93$ value in Table 2 as well?

Author response:
**Question 8**

Table 4 used the quantization configuration of NF3/NF4. Specifically, they use the block sizes and second-level quantization as the original NF4, but with the first-level quantization set to 3/4-bit. Table 2, however, used ILP to search for the quantization configuration, hence the difference.

We made such decisions for Table 4 to remove the possible confounding variable of using ILP to search for the quantization configuration (i.e., we just wanted to see the effect of higher rank LoRA components). We will clarify!