Paper: Batched Low-Rank Adaptation of Foundation Models (ICLR-2024)

Reviewer comment (Theory):
The transition from Eqn 4 to 5 is not immediately clear. It would be helpful to provide intermediate steps.

Author response:
> Transition from Eq 4 to Eq 5

We will include more intermediate steps such as the transformation from vector to matrix operators  in the revision.

Reviewer comment (Theory):
P5: In the sentence, "Secondly, in configurations where the model has fewer hidden units but an increased number of layers, FLORA tends to outperform LORA due to the smaller value of d in the denominator of Eq. (7)." How is the increased number of layers important given that Eq (7) contains only the dimensionality of the hidden units d and the rank r?

Author response:
> How is the increased number of layers important given that Eq (7) contains only the dimensionality of the hidden units d and the rank r?

Thank you for pointing this out. We assume the number of parameters of a LLM is fixed, then a deeper LLM would indicate less number of hidden units. This helps Inequality. 7 hold. We will make this clearer in the revision.

Reviewer comment (Experiments):
Table 2: Is there any reason why FLoRA underperforms LoRA for Marathi? What is the amount of fine-tuning data for each language?

Author response:
> Is there any reason why FLoRA underperforms LoRA for Marathi? What is the amount of fine-tuning data for each language?

The statistics for each language can be found here https://commonvoice.mozilla.org/en/datasets. Each language does have a different number of hours of fine-tuning data. For Marathi, we observed instability in the training loss of fLoRA, which may have contributed to its relative underperformance.

Reviewer comment (Reproducibility):
The whole mechanism and the algorithm isn't mentioned clearly. e.g., how to choose the batch size for real situations, how to make each example corresponding to its appropriate adapters during inference. The paper over-concentrates on Fomulation and Computational Efficiency, while the high-level algorithm--the whole process is not quite clear.

Author response:
Thank you for pointing this out. We will make sure to include more algorithmic details in the revision. For the experiment in Sec 4.1, the vLLM framework does not use batch size. Instead, it uses max number batched tokens to control how many tokens can be batched in one forward pass. For the throughput experiment, we set max_num_batched_tokens to be 8192, following the convention of using higher batch size for throughput benchmarking. Assuming the average input length of the testing dataset (the chat dataset to fine-tune the Vicuna model) is 512 then the batch size would be 16. For the latency experiment, we set max_num_batched_tokens to be 2560 which is the default setting of the vLLM framework for online serving.  

We assume the simplest case in the serving experiment where all possible adapters have been loaded in the memory. We also assume that each example (each request) in the batch is associated with an adapter id, then the adapter can be chosen by calling torch indexing.

Reviewer comment (Evaluation):
What's the memory comsumption of FLoRA compared with other methods?

Author response:
In the serving scenario, we did not observe any memory difference between fLoRA and LoRA. The vLLM framework will fit as many tokens as possible (but less than max_num_batched_tokens), utilizing all GPU memory. In the fine-tuning stage, when the rank is small, the memory consumption of fLoRA is roughly the same as LoRA. However, when the rank gets bigger (say rank > 7), the memory consumption of fLoRA is larger than LoRA. Notice that the self-attention layer accounts for the major memory consumption which fLoRA and LoRA don't touch. We will clarify this in the revision.

Reviewer comment (Theory):
Can you further explain "FLORA has the same expressive power as LORA by its construction"?

Author response:
We use "expressive power" to denote the rank of the modulation matrix. Both fLoRA and LoRA have the capability to modulate the weight matrix up to any desired rank. In this perspective, they have the same expressive power. We will make this clearer in the revision.

Reviewer comment (Theory):
The reason for changing "addition" of low-rank adapters in LoRA to "multiplication" in FLoRA is only for computational efficiency or for something else?

Author response:
The transition from addition to multiplication is primarily driven by computational efficiency. The multiplication operator enables the batching mechanism.

Reviewer comment (Writing):
I suggest removing references from the abstract.

Author response:
We also appreciate the feedback on how to improve the paper’s readability such as removing references from the abstract and better framing in Fig 1. We will make these adjustments in the revision.

Reviewer comment (Writing):
Could you explicitly clarify the definition of “expressive power” in the paper?

Author response:
We use "expressive power" to denote the rank of the modulation matrix. Both fLoRA and LoRA have the capability to modulate the weight matrix up to any desired rank. In this perspective, they have the same expressive power. We will make this clearer in the revision.

Reviewer comment (Evaluation):
About contribution 3 (Introduction): since fLoRA allows task-specific adapters for fine-tuning, wouldn’t you expect a higher, rather than simply equivalent, accuracy compared to fLoRA? In which scenarios do you expect that fLoRA could have sacrificed accuracy compared to LoRA?

Author response:
In our experiments (Sec. 4.2 and Sec. 4.3), we also trained task specific LoRA heads for each language in order for a fair comparison. Hence, one wouldn’t expect fLoRA outperform LoRA in terms of accuracy on the tasks we considered. In practice, the **average task accuracy** of fLoRA might be higher LoRA assuming LoRA has to use the same adapter in a majority of samples. However, our experiments did not account for this scenario so we did not claim a higher accuracy from fLoRA.

Reviewer comment (Presentation):
Fig 1: The figure is useful, but framing the different sections (1,2,3,4), or at least avoid overlapping among them would help clarity. Also, 4 task descriptions are indicated at point 1, and the corresponding 4 results are shown at point 4, while only 2 adapters and weights computations are shown at point 2 and 3. In my view, it would be clearer to keep the number of examples consistent across the sub-figures.

Author response:
We also appreciate the feedback on how to improve the paper’s readability such as removing references from the abstract and better framing in Fig 1. We will make these adjustments in the revision.

Reviewer comment (Theory):
In LoRA the weight matrix of the adapted foundation model is expressed by the SUM of W0 and DeltaW, while in fLoRA the weight matrix specific for each example is calculated as the element-wise MULTIPLICATION of W0 and DeltaWi. Is this correct?

Author response:
This is correct.

Reviewer comment (Theory):
On paragraph 3.2 you say that “fLoRA exhibits a lower computational cost than bmm LoRA whenever the above inequality holds true”. Could you elaborate more about scenarios when you expect (7) to be lower than 1?

Author response:
Equation 7 will not hold true when the rank (r) is large. This is supported by the data shown in Figure 2, where LoRA's performance begins to exceed that of fLoRA as the rank increases beyond a certain threshold. We will elaborate on this in the revised version.