Paper: LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The result of the LoftQ algorithm is a quantised weight matrix ($Q_T$) as well as the LoRA matrices ($A_T$, $B_T$). An interesting ablation would be to discard $A_T$ and $B_T$ and use the default LoRA initialisation instead. This would tell us more about the importance of initialising $A_T$ and $B_T$ differently.

Author response:
LoftQ enables $Q_T + A_T B_T$ as close to the pre-trained weight $W$ as possible. It does not guarantee $Q_T$ alone would be a better initialization than $Q_0$ which is used by QLoRA. We verify this in two ways: (1) we compare the F norm $||W-Q_T||_F$ and $||W-Q_0||_F$; (2) we take experiments on $Q_T$ with default LoRA initialization, as you suggest. The results are listed below:

(1). We measure the fc1 matrix of the 1st encoder layer in BART-large.

|                         | t=0  | t=1  | t=2  | t=3  | t=4  |
| -------------------------------- | ---- | ---- | ---- | ---- | ---- |
| $ \|\|W - Q_t\|\|_F$                  | 10.50 | 10.69 | 10.99 | 11.28 | 11.55 |
| $ \|\|W-(Q_t + A_t B_t^{\top})\|\|_F$ | 10.05   | 9.73   | 9.56   | 9.44  | 9.36   |

From the table, we can see  $||W - Q_t||_F$ actually increases throughout more iterations while $||W - (Q_t + L_t R_t^{\top})||_F$ decreases.

(2). We also take experiments on GSM8K
|    Model    | $Q_T + A_T B_T$ |  $Q_T$ w/ zero init  |
| ----------- | --------------  | ----------- |
| LLAMA-2-7b  |      35.0       |     33.4    |
| LLAMA-2-13b |      45.0       |     41.6    |

Reviewer comment (Experiments):
One of the findings in the QLoRA paper is that it is crucial to add LoRA adapters to every linear layer of the model (Figure 2 in the QLoRA paper). It could be interesting to run a similar ablation with your method. Given your improved initialisation, maybe it is sufficient to add LoRA adapters to fewer layers.

Author response:
This is an interesting question, but it is beyond the scope of our work. It remains open on how to achieve the best trade off between the number of layers with LoRA adapters and the performance. We would suggest referring to [AdaLoRA](https://arxiv.org/abs/2303.10512), which allocates dynamic ranks to different frozen matrices.

Reviewer comment (Experiments):
It could be interesting to study the difference in initialisation of the low-rank matrices more. Does your work provide insights into what makes a good LoRA initialisation and could these insights be potentially applied to non-quantised LoRA as well?

Author response:
The insight of LoftQ is that the full low-rank adapters are used to bridge the gap brought by quantizing pre-trained weights. We, unfortunately, cannot apply this logic to non-quantized LoRA because it does not have quantization discrepancy issues.

Reviewer comment (Writing):
Introduction, second paragraph: "It is predicated on the hypothesis ..." — You might want to use "based" instead of predicated.

Author response:
Thank you for pointing our typos and writing suggestions, we will improve it in the next version.

Reviewer comment (Writing):
Discussion, LoftQ better than full precision LoRA: "Such zero initialisation could cause the fine-tuning unstable" — This sentence needs rewriting.

Author response:
Thank you for pointing our typos and writing suggestions, we will improve it in the next version.

Reviewer comment (Experiments):
It might be better to put higher priority and conduct more experiments on decoder-based (or encoder decoder) models for generative tasks. It seems that quantized lora (whether with or without intialization) lacks too much in classification tasks with encoders, to the extent that pratictionars probably won't want to train quantized lora models on these tasks.

Author response:
We appreciate your suggestion on conducting more experiments on decoder-only models. We are currently working on LLAMA-2-70b. Due to the time limit, we may be not able to present comprehensive results in rebuttal within a short response period. However, we will add more experiments in the next version.

Reviewer comment (Evaluation):
It would be nice to show the memory footprint for 2-bit quantized models during training.

Author response:
The training memory footprint varies by task, batch size, and input sequence length. Therefore, we only provide examples. 
| Model       | Bit  | Rank | Task  | Seq length | Batch Size | GPU Memory |
| ----------- | ---- | ---- | ----- | ---------- | ---------- | ---------- |
| LLAMA-2-7b  | 4    | 64   | GSM8K | 384        | 1         | 15GB       |
| LLAMA-2-13b | 4    | 64   | GSM8K | 384        | 1         |  24GB   |

Reviewer comment (Experiments):
Would the quantized lora initialization in turn help full quantized fine-tuning?

Author response:
We are not sure about what you mean by “full quantized fine-tuning”. If it means quantization-aware full fine-tuning, our method is orthogonal to this setting because quantization-aware full fine-tuning does not have low-rank adapters. However, in terms of the initialization of the quantization-aware full fine-tuning, which also starts with a quantized model, our method could provide valuable insight: simply add the $A_T B_T$ back to $Q_T$, but how to add it back so that it achieves the least quantization discrepancy still remains an open question.

Reviewer comment (Experiments):
For the XSUM and GSM8k tasks, LoftQ gets better accuracy than full-precision LoRA. I wonder how the FP LoRA was tuned? Maybe 4 bit quantization does implicit regularization, and FP LoRA was not regularized well enough? This would especially make a difference if the tasks are low dimensional. In other words, if a high capacity LLAMA 13B model is fine-tuned LoRA style on GSM8k, how did the authors ensure that the model was not overfitted?

Author response:
Thanks for the advice of taking experiments on regularized FP LoRA. The FP LoRA in our paper is fine-tuned without any regularization but we have properly tuned the hyper-parameters, including lora_alpha, learning rate, batch size, etc. We did observe overfitting: as the training epoch increases, the accuracy on the test set drops after a certain epoch. We use early stopping to prevent overfitting. We stop the training by the 6th epoch. 

Adding weight decay to the LoRA adapters does help the LLAMA-2-13b on GSM8K, but it does not help LLAMA-2-7b on GSM8K. Specifically, full precision LLAMA-2-13b with LoRA achieves 46.0% now, instead of 43%, but LLAMA-2-7b drops from 36.9% to 34.4% on GSM8K. We will add the regularization results to the latest paper.

Reviewer comment (Experiments):
It would be nice to analyze number of epochs, and training steps required for baseline full precision LoRA and LoftQ.

Author response:
We have listed the required epochs in Table 11 and Table 12 in the Appendix. We choose the same epochs for both FP LoRA and LoftQ in all experiments.

Reviewer comment (Experiments):
LoRA's original motivation stems from "training efficiency" while maintaining the inference cost the same as the base model. Conversely quantization's main motivation is inference efficiency. Keeping training efficiency aside, a good baseline maybe quantization aware fine-tuning (i.e. no LoRA), to establish upper bound on accuracy for LoftQ.

Author response:
Thank you for your insightful comments. Allow me to address a potential misunderstanding regarding the motivation of LoRA and quantization. The original motivation of LoRA is efficient multi-task learning as introduced in the second paragraph of Section 1 in LoRA paper. The motivation of weights-only quantization is model compression, allowing LLMs to run on memory-limited devices.

To the best of our knowledge and the NLP literature, it is not clear whether quantization-aware training (QAT) would be an upper bound for LoftQ or QLoRA, given two reasons: (1) the initial weights are quantized and therefore there exists a significant gap between quantized initialization and the pre-trained weights in QAT, while LoftQ can reduce such a gap. (2) it is difficult to update the non-continuous quantized parameters, which is merely possible to achieve full precision fine-tuning performance, let alone given severely degraded initial weights. On the other hand, LoftQ fine-tunes full precision LoRA adapters, making it easy to fine-tune.

Reviewer comment (Reproducibility):
It wasn't very fully clear but are the LoRA adaptors, A and B, quantized as well in LoftQ?

Author response:
The LoRA adapters, A and B, are not quantized. They remain full precision throughout the training. Maintaining full precision or for the LoRA adapters cause minor extra memory compared to quantized LoRA adapters.