Paper: QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
I wonder if larger model where the need of this technique is crucial, e.g., 30B-60B, could be discussed.

Author response:
Nice suggestion! In the original version, we provide experiments on the 33B and 65B models of LLaMA in Table 1, and QA-LoRA achieves improvement over the baseline, QLoRA with GPTQ. Upon these results, we discuss the need for QA-LoRA in large models in the following aspects. (1) QA-LoRA reduces the computational costs of fine-tuning large models, *e.g.*, using QA-LoRA, only 1 and 2 V100 GPUs are needed for fine-tuning the 33B and 65B models. (2) When larger models are used, there can be increasing needs for low-bit (*e.g.*, INT3 and INT2) quantization, especially when the large models are to be deployed to edge devices. QA-LoRA shows significant advantages in such scenarios. **In the revised paper, the above contents are added to Section 4.2 where the experimental results are discussed.**

Reviewer comment (Presentation):
Figure 3 legend should have QA-LoRA instead of A-LoRA.

Author response:
Sorry for our carelessness. **It has been fixed in the revised version.**

Reviewer comment (Presentation):
In Algorithm 1, the function `merge_with_quantization` is defined but never used.

Author response:
>**A1.** Sorry for misleading. Due to the space limit, we did not add the `main` function in the paper, which calls `qalora_training` (which calls `qalora_forward`) and `merge_with_quantization`. The `merge_with_quantization` function is called *after* the entire QA-LoRA training procedure for merging the LoRA weights $s\cdot\mathbf{A}\mathbf{B}$ into the quantized weight matrix $\tilde{\mathbf{W}}$. It is done by updating the $\beta$ parameter into $\beta_\mathrm{new}$. **In the revised paper, we have added an explanation in the main article, after the reference to Algorithm 1.**

Reviewer comment (Presentation):
Page 6 is just results - Page 6 is just results without much to interpret. These are a bunch of numbers. Please consider presenting this table in a better manner. Can this be presented as a graph for readers? Just numbers are hard to read.

Author response:
>**A3.** Thanks for the suggestion. We followed QLoRA and other recent works to report these exact numbers to ease the comparison against existing papers. Actually, the main results of Table 6 (5-shot MMLU on the Alpaca dataset) have been summarized in Figure 1. **In the revised paper, we plot other results (0-shot MMLU and the FLAN v2 dataset) as similar figures and add them to Appendix B.**

Reviewer comment (Experiments):
In Table 2, you indicate that the number of parameters in the method are lesser than QLoRA — almost by 2x. Why is this the case? This seems like an unfair comparison to QLoRA. What is the time taken in hours for fine-tuning with similar number of parameters?

Author response:
>**A4.** The reason behind the fewer amounts of parameters, compared to QLoRA, lies in the reduction of the dimensionality of $\mathbf{A}$. Compared to LoRA and QLoRA where $\mathbf{A}$ has $D_\mathrm{in}\times D_\mathrm{int}$ parameters, QA-LoRA reduces the number to $L\times D_\mathrm{int}$ where $L$ is the group size and $L\ll D_\mathrm{in}$. This reduces the number of parameters in QA-LoRA by around $1/2$ (originally, $\mathbf{A}$ and $\mathbf{B}$ have similar numbers of parameters). We would like to stress that this is a good feature of QA-LoRA which allows it to achieve higher fine-tuning accuracy with fewer parameters.  
>Regarding the fine-tuning time, it is *not* largely impacted by the parameters because the amount of LoRA parameters is much smaller than that of the LLM itself (*e.g.*, 89M or 160M *vs.* 7B). To verify this point, we double $D_\mathrm{int}$ which also doubles the number of parameters, surpassing that of QLoRA, but QA-LoRA is still much faster than QLoRA (see the table below). Essentially, QA-LoRA is faster in training because it uses INT4 computation while QLoRA uses NF4 computation which is not well optimized by CUDA.  
>**In the revised version, we make this point clear by adding the above contents to Section 4.2 (the paragraph starting with "the efficiency of QA-LoRA").**  

|Method|#Params|Time|  
|:-|:-:|:-:|  
|QLoRA ($D_\mathrm{int}=64$)|160M|40.0h|  
|QA-LoRA ($D_\mathrm{int}=64$)|89M|21.5h|  
|QA-LoRA ($D_\mathrm{int}=128)$|178M|21.8h|  

Table. A comparison of the number of parameters and training time.

Reviewer comment (Theory):
Section 3.3 explains the method and the reasoning behind on why the rank degenerates to 1. But the explanation is not comprehensive.

Author response:
>**A5.** First of all, please note that the final step of QA-LoRA is to fuse $\tilde{\mathbf{W}}$ (the quantized matrix during training) and $s\cdot\mathbf{A}\mathbf{B}$ (the adaptation weights) into a new matrix $\mathbf{W}'=\tilde{\mathbf{W}}+s\cdot\mathbf{A}\mathbf{B}$. For the efficiency of inference, we hope that $\mathbf{W}'$ is still in low-bit quantization. This means that all values in $\tilde{\mathbf{W}}$ and $\mathbf{W}'$ must come from an equidistant discrete set (see Footnote 1).  
>As an example, let a column of $\tilde{\mathbf{W}}$ contain the numerical values from the set of $\{0.3,1.5,2.7,3.9\}$, where we can use INT2 quantization ($\alpha=1.2$, $\beta=0.3$). Now, it is added by the corresponding column in $s\cdot\mathbf{A}\mathbf{B}$ (both $\mathbf{A}$ and $\mathbf{B}$ contain continuous numerical values), and we require that the summation is still INT2-quantizable. To guarantee this in a continuous optimization procedure, the only known way is to constrain the added values to be constant, which derives that all row vectors of $\mathbf{A}$ are identical, and hence $\mathrm{rank}(\mathbf{A})=1$. Actually, we can choose not to say $\mathrm{rank}(\mathbf{A})=1$ but simply say that the flexibility of adaptation is largely constrained because the row vectors of $\mathbf{A}$ must be identical.  
>We hope that we have made the explanation comprehensive. Of course, we are open to further discussions. Thanks.