Paper: NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR-2024)

Reviewer comment (Experiments):
Could the authors also provide the performance of the GPT-M, L models *without any fine-tuning* on the tasks considered? This will give the readers an idea of how much improvement is being achieved.

Author response:
This is an interesting suggestion. we looked at the literature and could not find any prior work reporting zero-shot results for this E2E NLG benchmark. We tried it ourselves and got very low accuracy. We believe we need to engineer a prompt for this task since a non-finetuned model does not know the task. We will do this for the camera ready version. 

Meanwhile, we found the following paper [b] that shows zero-shot results on a related dataset, DART. They too report very low accuracy for the zero-shot setting: Their BLEU score is 13.26 for zero-shot and 33.41 for 3-shot. We also report the results for DART in the appendix, but our results are not directly comparable with theirs since our experiments are with GPT-M and GPT-L while they use GPT-XL. 

[b] Keymanesh, Moniba, Adrian Benton, and Mark Dredze. "What makes data-to-text generation hard for pretrained language models?." arXiv preprint arXiv:2205.11505 (2022).

Reviewer comment (Experiments):
The experiments provided in the vision experiments section (3.3) are a bit weak because the ViT models pre-trained on ImageNet are already pretty powerful, and fine-tuning these models on much smaller and easier datasets such as CIFAR may not be the best way to demonstrate usefulness.

Author response:
We agree and thank the reviewer for the suggestion. We perform experiments with NOLA and LoRA on various more challenging datasets. The results are reported in the following table and also in Table 6 of the revised paper. These are more challenging settings given an ImageNet pretrained model since the accuracy for some cases is around $30$% to $40$%. Interestingly, NOLA is on-par or better than LoRA in all cases with almost 3 times smaller number of parameters, so our conclusion holds on these more challenging datasets too.  

**5 shot:** 

| Method   | \# Train | Aircraft 5 Shot | Food101 5 Shot | Pets 5 Shot | SUN397 5 Shot |
|----------|----------|------------------|-----------------|--------------|----------------|
| NN       |          | 24.6             | 48.4            | 82.3         | 44.4           |
| Linear   |          | 29.7 (2.3)        | 53.2 (0.3)      | 88.4 (0.4)   | 38.0 (0.8)      |
| Full-FT  | 82M      | 31.4 (2.4)        | 48.6 (5.1)      | 91.1 (0.5)   | 45.0 (3.3)      |
| LoRA     | 0.141M   | 32.4 (1.4)        | 60.8 (1.6)      | 91.6 (0.5)   | 51.6 (0.4)      |
| NOLA-MLP | 0.047M   | 33.7 (2.2)        | 64.5 (0.8)      | 88.0 (0.6)   | 50.5 (0.4)      |

**10 shot:**

| Method   | \# Train | Aircraft 10 Shot | Food101 10 Shot | Pets 10 Shot | SUN397 10 Shot |
|----------|----------|------------------|-----------------|--------------|----------------|
| NN       |          | 27.1             | 54.2            | 86.2         | 51.5           |
| Linear   |          | 36.9 (2.1)        | 61.5 (0.1)      | 91.3 (0.3)   | 42.7 (0.4)      |
| Full-FT  | 82M      | 43.2 (2.0)        | 65.8 (2.7)      | 91.1 (0.5)   | 52.6 (0.3)      |
| LoRA     | 0.141M   | 43.8 (1.5)        | 73.1 (0.6)      | 91.6 (0.5)   | 55.6 (0.3)      |
| NOLA-MLP | 0.047M   | 43.3 (1.4)        | 72.6 (0.4)      | 92.2 (0.3)   | 55.5 (0.3)      |

Reviewer comment (Experiments):
Can the authors try their techniques on more challenging datasets?

Author response:
We agree and thank the reviewer for the suggestion. We perform experiments with NOLA and LoRA on various more challenging datasets. The results are reported in the following table and also in Table 6 of the revised paper. These are more challenging settings given an ImageNet pretrained model since the accuracy for some cases is around $30$% to $40$%. Interestingly, NOLA is on-par or better than LoRA in all cases with almost 3 times smaller number of parameters, so our conclusion holds on these more challenging datasets too.  

**5 shot:** 

| Method   | \# Train | Aircraft 5 Shot | Food101 5 Shot | Pets 5 Shot | SUN397 5 Shot |
|----------|----------|------------------|-----------------|--------------|----------------|
| NN       |          | 24.6             | 48.4            | 82.3         | 44.4           |
| Linear   |          | 29.7 (2.3)        | 53.2 (0.3)      | 88.4 (0.4)   | 38.0 (0.8)      |
| Full-FT  | 82M      | 31.4 (2.4)        | 48.6 (5.1)      | 91.1 (0.5)   | 45.0 (3.3)      |
| LoRA     | 0.141M   | 32.4 (1.4)        | 60.8 (1.6)      | 91.6 (0.5)   | 51.6 (0.4)      |
| NOLA-MLP | 0.047M   | 33.7 (2.2)        | 64.5 (0.8)      | 88.0 (0.6)   | 50.5 (0.4)      |

**10 shot:**

| Method   | \# Train | Aircraft 10 Shot | Food101 10 Shot | Pets 10 Shot | SUN397 10 Shot |
|----------|----------|------------------|-----------------|--------------|----------------|
| NN       |          | 27.1             | 54.2            | 86.2         | 51.5           |
| Linear   |          | 36.9 (2.1)        | 61.5 (0.1)      | 91.3 (0.3)   | 42.7 (0.4)      |
| Full-FT  | 82M      | 43.2 (2.0)        | 65.8 (2.7)      | 91.1 (0.5)   | 52.6 (0.3)      |
| LoRA     | 0.141M   | 43.8 (1.5)        | 73.1 (0.6)      | 91.6 (0.5)   | 55.6 (0.3)      |
| NOLA-MLP | 0.047M   | 43.3 (1.4)        | 72.6 (0.4)      | 92.2 (0.3)   | 55.5 (0.3)      |

Reviewer comment (Presentation):
Although the number of parameters for the linear layer baseline depends on the dataset, it would be good to have this information visible in the Table.

Author response:
We report the information on number of parameters of the linear layer here and in the caption of Table 5 of the revised version. The size is: 0.01M, 0.1M, 0.2M, 0.1M parameters for CIFAR-10, CIFAR-100, CUB and Caltech respectively. Similarly, the linear classifier sizes for the additional datasets considered in the revised submission are: 0.03M, 0.2M, 0.1M, 0.3M for Aircraft, Food101, Pets and SUN397 datasets respectively. We did not previously include this in the table since the linear layer classifier is constant for all methods and makes the table cluttered.

Reviewer comment (Experiments):
What is the distribution used to generate the random basis matrices? Did the authors experiment with a few different choices?

Author response:
We simply use i.i.d. uniform distribution. We did not experiment with any other option. Studying what distribution should or may work better is an interesting research problem that we leave for the future work.

Reviewer comment (Novelty):
The work may need more rationales upfront to motivate the problems (i.e. the rank one lower bound present in LoRA). Given that mainstream GPUs have tens of GB of memory, it is reasonable to reduce the memory requirements from tens of GB to tens of MB at the expense of model quality through LoRA, as this can indeed reduce resource consumption and greatly reduce LLM transition overhead during inference. However, I don't think it makes much sense to further reduce memory requirements to several MBs at the expense of model quality.

Author response:
This is a great question that we have addressed in subsection named "Why Fewer Parameters Matter?" in the Introduction of the original submission. At the time of submission, we were envisioning that there will be several finetuned LLMs that need to be stored and used for inference. Interestingly a version of this vision is already here with the introduction of "GPTs" by OpenAI a couple of weeks ago. Task-specific GPTs are very easy to generate by the users, so we expect to see a large collection of them soon in the market place. Hence, handling the storage of those models and more importantly transitioning between them at the inference time is an interesting challenge that we suggest to solve. We argue that if the number of parameters are very small (e.g., in NOLA), we can store the weights for several GPTs in the GPU memory itself reducing the latency due to transferring them from CPU memory to GPU as soon as a new query for a specific GPT arrives. Then, even a small reduction in the size of the model compared to LoRA can become important due to the high cost of GPU memory and growing number of GPTs.

Moreover, note that the sizes are smaller in these GPT-2 experiments compared to more advanced LLMs. Implementing LoRA and NoLA on larger LLMs will result in much bigger number of parameters. Hence, any reduction in size compared to LoRA will be important. Note that LoRA has a lower bound on the number of parameters corresponding to $r=1$ while NOLA is flexible and can trade accuracy with the number of parameters independent of the rank or feature dimension.


Interestingly, a very recent paper [a] published a couple of weeks ago on arXiv (following) suggests that handling many task-specific LLMs is an important challenge and introduces a computer architecture solution. We are addressing a similar challenge by reparameterization at the learning time.

[a] Sheng, Ying, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou et al. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." arXiv preprint arXiv:2311.03285 (2023).

Reviewer comment (Novelty):
The technical novelty is relatively minor with the overall idea being a combination of prior works PRANC and NOLA. While this seems enough to provide empirical improvement, the approach itself is not that big of an innovation over prior works.

Author response:
PRANC reparameterizes CNN models for visual recognition tasks while training from scratch. We use a similar reparameterization but with several key distinctions: 

(a) We introduce a similar reparameterization to both A and B matrices of LoRA method and show on-par results with much smaller number of parameters. 

(b) We learn the residual model instead of learning it from scratch as in PRANC. 

(c) Moreover, PRANC is computationally expensive and has a large memory footprint due to the large basis matrix. We resolve this issue in NOLA since the big matrix is factorized into two smaller matrices. In the appendix, we show NOLA outperforms PRANC on vision tasks from scratch too.

(d) In addition, in Figure 2, we show that the factorization in NOLA offers much higher coverage of the subspace compared to PRANC with similar number of parameters.

Reviewer comment (Experiments):
While the prior approach PRANC is directly modified by the authors in this work there are no direct comparisons with it in either the language or vision tasks used to evaluate the proposed approach. There is a comparison of training loss in Section 3.4 and a comparison of the rank of possible solutions of the two approaches in Section 3.5 but without a direct comparison of test accuracy it is unclear if this approach is indeed an improvement over the baseline that it directly modifies.

Author response:
In the Appendix of the original submission, we directly compare NOLA with PRANC on vision tasks (Table 8 for CIFAR-10 and Table 9 for ImageNet-100). NOLA outperforms PRANC with on-par or smaller number of parameters. We report this in Appendix due to space limitations. PRANC is not suitable for language tasks due to its high cost of computation and memory for large models. We present the comparison with PRANC on ImageNet-100 below. The `Trained model' in the table refers to training the entire ResNet-18 model from scratch.

| **Method**           | **\# Params** | **Acc.**    |
|-|-|-|
| trained model        | 11,227,812    | **82.1\%**  |
|                   |           |         |
| HashedNet Chen et al. (2015) | 129,200 | 52.96\% |
| PRANC                | 119,200       | 61.08\%     |
| LoRA                 | 150,000       | 63.50\%     |
| NOLA                 | 109,600       | 64.66\%     |

Reviewer comment (Evaluation):
Why is the training time of NoLA with shared random basis similar to that of LoRA when the training time of NOLA with a unique random basis is higher? Aren't the number of coefficients being trained, the same in both cases?

Author response:
Yes, when using non-shared basis, the training is almost $7$% slower compared to using shared basis. In shared basis case, the same basis are used across all the layers. We believe the extra training time is due to the latency of the random matrix generator since in the non-shared case, we call it once for each layer at each iteration. Table 2 shows that shared basis version works as well as the non-shared one.

Reviewer comment (Presentation):
Please highlight the entries in Table 5 with the best performance for a given scenario. Currently there are too many entries, and it is too difficult to figure out which method is better for which case.

Author response:
Thanks for the suggestion. Note that simply highlighting the best is not enough since we are not claiming that NOLA is consistently better than LoRA in terms of accuracy. We want to show that they are on-par while NOLA is smaller in size. Hence, in the revised Table 5, we "boldface" the best number and "underline" any method within one point accuracy difference of the best method to show that NOLA and LoRA are close. Hope this helps.

Reviewer comment (Theory):
If each matrix in PRANC has size $d^2$ then why do we need multiple matrices to cover the rank of the original $\Delta W$ matrix (which also has size $d^2$)?

Author response:
Since PRANC matrices are initialized with random values and are not optimized, we need to combine several of them linearly (with learned coefficients) to represent $\Delta W$.

Reviewer comment (Evaluation):
The results lack clarity and explanatory depth, failing to provide substantial insights into how the method operates and leaving readers without a clear understanding of the underlying mechanisms.

Author response:
Thanks for your valuable feedback.

We argue that there are many possible local minima for $A$ and $B$ matrices in LoRA method. Training LoRA results in finding one of those local minima. Inspired by PRANC, our main intuition is that one of those local minima may live in a low dimensional random subspace. Hence, in learning, we directly look for that local minimum instead of searching for the whole space. Since the subspace is low-dimensional, we save on the number of parameters. Our experiments support our intuition empirically as we do not loose accuracy compared to original LoRA while reducing the number of parameters and decoupling them from the rank in LoRA method. We will clarify this in the text.

Reviewer comment (Presentation):
Tables 1 and 5 are presented as mere lists of numbers without necessary context or explanation, making it difficult for the audience to derive meaningful conclusions from the data.

Author response:
As stated, our main conclusion from the experiments is that our method can reduce the number of parameters while getting on-par results compared to LoRA. Since the numbers are on-par with LORA, we do not specifically highlight the best numbers on Table 1. We believe a part of confusion for the reviewer on Table 1 is due to showing that different variants (MLP, QV, etc) of our method achieves on-par results too. We will move some of those variants to the appendix or ablation section in the camera ready.

Reviewer comment (Evaluation):
GPT2-L and GPT-2M seem to perform the same for LoRA, and there is no explanation provided for why this is the case.

Author response:
1. Yes, they are similar on LoRA paper too. One explanation may be that GPT-2 M has enough capacity for these tasks, but we are not sure about a clear explanation. This may need to be studied as future work.

Reviewer comment (Presentation):
The presentation of results precedes the training details in Section 3.1, which seems to be an inadvertent oversight. Reversing the order, placing training details before the results, would enhance the logical flow of the content.

Author response:
2. That's a good idea. We changed the order in the revised version to make it more clear.