Paper: One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning (ICLR-2024)

Reviewer comment (Presentation):
The presentation of the paper could be improved, especially when comparing with previous PEFT methods. The authors could draw figures or list tables to show how existing methods can be integrated into GLoRA framework, e.g. what are the specifications of the A/B/C/D/E support tensors in Eq. (10).

Author response:
We thank the reviewer for the suggestion and we have worked on improving the overall presentation of the paper. Here we present a table which shows how existing methods can be integrated into GLoRA framework. We have also added this table to the main paper.. We set different support tensors to choices presented in the search space (Eq. (11)) to approximately mimic the behavior of existing methods. 

|  Method     |   A    |   B   |   C   |   D   |   E   | 
|--------|:-------:|:-------:|:-------:|:-------:|:--------------:|
| LoRA | LoRA   | None  | None  | None  | None  |
| VPT | None   | None  | Vector| None  | None  |
| SSF | Vector | None  | Vector| Vector| None  |
|RepAdapter | LoRA   | None  | None  | Vector| None  |

Reviewer comment (Experiments):
I wonder how much training time (supernet training and subnet searching) does GLoRA cost, such that to compare with existing methods more clearly from the perspective of training efficiency.

Author response:
When compared to other adaptation methods, the benefits gained with GLoRA in terms of performance improvement as well as generalization across datasets significantly outweigh the additional time spent in the adaptation process.

We demonstrate the superiority of GLoRA by analyzing the performance improvement per unit training time of GLoRA vs. other counterparts. Quantitatively, GLoRA requires an additional 5.6 folds of training time compared to a single run of LoRA amounting to a total of ~142 minutes for each VTAB-1k task. The GPU memory consumption of GLoRA is 13 GB compared to 9 GB for LoRA. Most of it is primarily because GLoRA requires roughly 5 times more epochs than LoRA for appropriate convergence and the additional time is spent on the evolutionary search process. This extra time of GLoRA leads to an average increase of 4.5 % accuracy across 19 vision tasks as compared to LoRA. Beyond this, the biggest benefit comes in terms of generalization across datasets. Additionally, we consider 5 best-performing methods on the VTAB-1k dataset - LoRA, NOAH, FacT, SSF and RepAdapter; and pick the task-specific best-performing models across them. We denote the training time of LoRA as x and the corresponding training time of NoAH, FacT, SSF and RepAdapter are 6x, x, 1.2x and 0.9x respectively. GLoRA's total training time including architecture search is 6.6x. The table below shows the performance and training time of Best-5 methods and GLoRA. This is to indicate that the combined training time of 5 best existing methods is more than that of GLoRA and yet it delivers a superior performance over its counterpart. It is important to note that this gain is reported assuming that the other adaptation methods do not require any hyperparameter search. However, unlike GLoRA, needing minimal hyperparameter search (see Appendix C), some of the other adaptation methods require thorough data-specific hyperparameter search for optimal performance. If we consider this aspect, then the training time required by GLoRA would be significantly lower in a relative sense. We have mentioned the actual training time and memory of GLoRA in the appendix of the main paper.
| Method   | Training Time | Inference Time | Natural | Specialized | Structured | Average |
|----------|:---------:|:-------------:|:------------:|:---------:|:-------:|:-------:|
| NOAH | 6x | &#8593; | 80.3 | 84.9 | 61.3 | 75.5 | 
| Best-5 | 10.1x | &#8593; | 82.5    | 86.7        | 63.3       | 77.5    |
| GLoRA | 6.6x | - | 83.6    | 87.0        | 63.3       | **78.0**    |

Reviewer comment (Writing):
There exist some typos: 1) "PETL"(maybe PEFT?) at the end of page 3 (first line of Sec. Limitations); 2) 4th line of page 4: "wieght" -> weight.

Author response:
We thank the reviewer for pointing out these typos and we have updated the draft accordingly.

Reviewer comment (Experiments):
How about the performance if we do not add the weight/bias scaling term: W_0 x A and D x b_0 in Eq. (10)? Or else, which of the five tensors are really necessary in terms of efficiency and efficacy?

Author response:
In the construction of GLoRA architectures, each support tensor plays an important role and is necessary to achieve the optimal performance. Among the various supports, A is utilized to scale the weight. B has the role to scale the input and shift the weight. C is the layer-wise prompt serving a similar function of VPT-Deep, D and E are used to scale and shift the bias, respectively. In the table below, we show the performance on CIFAR dataset by setting some support tensors to None, and it is clear that the performance dip is observed for all cases. Clearly, omitting any of these tensors gives sub-optimal performance, implying that each of these is important.

| Settings | Remove A and D | Remove C | Keep All |
|--------------|:---------:|:------:|:------:|
| **Accuracy** | 76.0    | 75.9 | 76.5 |

In terms of inference efficiency, GLoRA enjoys zero computational overhead due to the complete structural re-parameterization framework (as shown in Table 4). In terms of training efficiency, it is governed by the set of supports that form the GLoRA architecture. Among LoRA, Vector, Scalar and None attributes, LoRA is most computationally most expensive due to matrix multiplication involved in it, Vector being the next due to dot product operations and Scalar is the most efficient due to simple scalar multiplication. Additionally, since D and E cannot be LoRA, these are the most efficient support tensors. Further, C is the least efficient one due to the absence of scalar multiplication. We have also added these details in the updated draft.

Reviewer comment (Novelty):
It seems that GLoRA is not general, since it has an evolutionary search procedure to obtain the suitable components. The idea is similar to Neural Prompt Search [1]. GLoRA is not a fixed design as existing modules, which might limit its practicality.

Author response:
A1. We understand the concern of the reviewer, however, our use of 'general' for GLoRA does not refer to one set of adapters that can be plugged on top of a model. As we have shown, the performance of such a method varies significantly across datasets. Rather, GLoRA is general in the sense that the combination of supernet training and subnet search which form an inherent part of the method, result in well performing data-specific architectures. Although GLoRA is a search based method similar to [1], the core novelty of GLoRA lies in the unified re-parameterizable formulation as presented in Eq. 10. [1] attempts to search for VPT, LoRA and Adapter in a single framework, by defining them distinctly in the supernet whereas GLoRA doesn't define any existing adapter/module in the network explicitly but proposes a much 'general formulation' which implicitly mimics the behavior of many existing works. 

Regarding the limitation in terms of practicality, GLoRA identifies the right combination of support tensors/modules/prompts for each dataset. Any optimized network obtained from GLoRA can be absorbed into the base architecture using structural re-parameterization rendering superior or the same practical throughput as existing works. Thus, we do not see any limitations of the approach compared to the other baseline methods such as LoRA, SSF, etc.

Reviewer comment (Evaluation):
GLoRA has a large search space, which might yield huge time costs. However, the authors have not mentioned the actual training time and memory cost of GLoRA, which is very important for PEFT modules.

Author response:
A2. As highlighted by the reviewer, the large search space of GLoRA indeed increases the associated training time compared to the other adaptation methods. However, it is important to note that the overall time spent on the adaptation of models using GLoRA is still substantially very small when compared to fine-tuning the full model. Further, when compared to other adaptation methods, the benefits gained with GLoRA in terms of performance improvement as well as generalization across datasets significantly outweigh the additional time spent in the adaptation process.

We demonstrate the superiority of GLoRA by analyzing the performance improvement per unit training time of GLoRA vs. other counterparts. Quantitatively, GLoRA requires an additional 5.6 folds of training time compared to a single run of LoRA amounting to a total of ~142 minutes for each VTAB-1k task. The GPU memory consumption of GLoRA is 13 GB compared to 9 GB for LoRA. Most of it is primarily because GLoRA requires roughly 5 times more epochs than LoRA for appropriate convergence and the additional time is spent on the evolutionary search process. This extra time of GLoRA leads to an average increase of 4.5 % accuracy across 19 vision tasks as compared to LoRA. Beyond this, the biggest benefit comes in terms of generalization across datasets. Additionally, we consider 5 best-performing methods on the VTAB-1k dataset - LoRA, NOAH, FacT, SSF and RepAdapter; and pick the task-specific best-performing models across them. We denote the training time of LoRA as x and the corresponding training time of NoAH, FacT, SSF and RepAdapter are 6x, x, 1.2x and 0.9x respectively. GLoRA's total training time including architecture search is 6.6x. The table below shows the performance and training time of Best-5 methods and GLoRA. This is to indicate that the combined training time of 5 best existing methods is more than that of GLoRA and yet it delivers a superior performance over its counterpart. It is important to note that this gain is reported assuming that the other adaptation methods do not require any hyperparameter search. However, unlike GLoRA, needing minimal hyperparameter search (see Appendix C), some of the other adaptation methods require thorough data-specific hyperparameter search for optimal performance. If we consider this aspect, then the training time required by GLoRA would be significantly lower in a relative sense. We have mentioned the actual training time and memory of GLoRA in the appendix of the main paper.
| Method   | Training Time | Inference Time | Natural | Specialized | Structured | Average |
|----------|:---------:|:-------------:|:------------:|:---------:|:-------:|:-------:|
| NOAH | 6x | &#8593; | 80.3 | 84.9 | 61.3 | 75.5 | 
| Best-5 | 10.1x | &#8593; | 82.5    | 86.7  | 63.3   | 77.5    |
| GLoRA | 6.6x | - | 83.6    | 87.0| 63.3  | **78.0**    |

Reviewer comment (Reproducibility):
The authors introduce multiple PEFT modules including AdaptFormer, LoRA, etc. So could GLoRA simulate all of these modules? As far as I know, these modules are applied on different layers (LoRA in multi-head self-attention layers, while AdaptFormer in MLP layers). And which layer is GLoRA applied in practice?

Author response:
A3. Indeed GLoRA is capable of simulating multiple existing PEFT modules which include SSF, VPT, LoRA and RepAdapter. For example, in equation 10, setting all support tensors but A to None and setting A as LoRA would inherently give the classic LoRA model. To mimic SSF, A, D and E can be set to vector and other all support tensors to none. Similarly, C simulates the learning mechanism of a layerwise trainable prompt similar to VPT. Note that GLoRA cannot simulate AdaptFormer since it has a non-linearity in the parallel block inhibiting structural re-parameterization which is a core aspect of GLoRA. In the table below, we show how GLoRA is able to approximately mimic the behavior of many existing works by setting support tensors to specific attributes -

|  Method     |   A    |   B   |   C   |   D   |   E   | 
|--------|:-------:|:-------:|:-------:|:-------:|:--------------:|
| LoRA | LoRA   | None  | None  | None  | None  |
| VPT | None   | None  | Vector| None  | None  |
| SSF | Vector | None  | Vector| Vector| None  |
|RepAdapter | LoRA   | None  | None  | Vector| None  |

For vision tasks GLoRA is applied to all linear layers (MLP as well as the attention modules). In case of NLP tasks, for a fair comparison to LoRA, we apply GLoRA only in the multi-head self-attention layers. These details are mentioned in Section 3.2 of the main paper. However, in general, GLoRA can also be applied to the MLP layers, and we expect that similar to LoRA, it will improve the performance further [2].

[1] Neural Prompt Search.

[2] QLoRA: Efficient Finetuning of Quantized LLMs