Paper: LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition (ICLR-2024)

Reviewer comment (Experiments):
The choice of using FLAN-T5-Large as the base model is questionable as a model pre-trained on unsupervised text might have been more relevant for fine-tuning / LoRA fine-tuning on upstream tasks from the FLAN collection.

Author response:
Thanks for your feedback. We would like to clarify that the selection of FLAN-T5-large as our main base model was a strategic choice based on several considerations and explain them in detail below.

Firstly, the choice of FLAN-T5 as our base model is grounded in the pursuit of language models with exceptional few-shot capabilities, making it a robust baseline for our study. FLAN-T5 not only excels in both zero-shot and few-shot scenarios but also boasts strong problem-solving capabilities, positioning it as a formidable candidate for our consideration.

Secondly, the choice of FLAN-T5 as our primary base model is crucial to ensuring a fair comparison with zero-shot learning. By utilizing LoRA modules trained on the FLAN collection, our intention is to insulate our method's performance from the influence of additional datasets. Given that the FLAN collection is already seen during FLAN-T5's training, we can reasonably assert that the true efficacy lies in the composition of LoRA modules rather than the introduction of extra datasets. Selecting an unsupervised model, such as T5, as the base would present an unfair advantage for our method, as it would indirectly leverage datasets not encountered during T5's pre-training.

Lastly, we genuinely appreciate and understand your concern. To address it comprehensively, we have also provided an analysis of the performance of our method with T5 as the base model in both Section 5 (`Can LoraHub work well on non-instruction-tuning models`) and Appendix B. These sections explicitly showcase the efficacy of our method when applied to models trained exclusively on unsupervised text, underscoring the generality of our approach.

Reviewer comment (Experiments):
The paper lacks clarity in explaining the rationale behind maintaining the same rank for the composed LoRA module and could benefit from exploring higher rank matrices when composing.

Author response:
Thanks for your question! In fact, we have investigated the impact of different ranks in Section 5 (`Will the rank of LoRA modules impact the performance of LoraHub learning`) and Appendix B. Our analysis indicates that, for FLAN-T5, the choice of rank has minimal impact. However, for T5, it still exerts some influence. Empirical findings reveal that, in comparison to rank values of $4$ or $64$, a rank value of $16$ consistently demonstrates superior performance across different runs, both in terms of average and best performance.

| Task | Rank 4$_{avg}$ | Rank 4$_{best}$ | Rank 16$_{avg}$ | Rank 16$_{best}$ | Rank 64$_{avg}$ | Rank 64$_{best}$ |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Average Performance Per Task | 16.1 | 24.2 | 20.8 | 30.7 | 14.8 | 21.4 |

We hope that this exploration addresses your concerns and brings satisfaction. We anticipate that future research endeavors will contribute to a deeper understanding of the factors influencing these effects.

Reviewer comment (Experiments):
The selection of 20 LoRAs for unseen tasks seems arbitrary and might limit the method’s performance. An iterative procedure or justification for this selection would have been beneficial.

Author response:
Thank you for pointing this out! In our initial experiment, the selection of the $20$ LoRA module candidates is conducted randomly, serving as the primary source of randomness in our method.

Acknowledging the need for improvement in the aspect, we explored a straightforward approach ($\rm LoraHub_{filter}$) in the selection of LoRA module candidates. Specifically, we first identified $20$ LoRA module candidates with the lowest loss on the few-shot examples, with the goal of minimizing introduced variance and fostering a more consistent performance. The experimental results demonstrate that this approach contributes to a slight improvement in the average performance. These results have been included in Appendix D for further reference. We hope the updated experiment addresses your concerns.

| $\rm Task$                                   | $\rm LoraHub_{avg}$ | $\rm LoraHub_{filter}$ |
|-------------------------------------------|-------------|----------------|
| Boolean Expressions                       | 55.5        | 60.0           |
| Causal Judgement                          | 54.3        | 52.9           |
| Date Understanding                        | 32.9        | 33.3           |
| Disambiguation                            | 45.2        | 62.7           |
| Dyck Languages                            | 1.0         | 0.0            |
| ... | | |
| Tracking Shuffled Objects (three objects) | 29.0        | 32.7           |
| Web of Lies                               | 53.0        | 46.0           |
| Word Sorting                              | 1.1         | 1.3            |
| **Avg Performance Per Task**                  |**34.7**       | **35.4**          |

Reviewer comment (Experiments):
The absence of certain baselines, such as the average performance for BBH's top 5 upstream tasks, leaves gaps in the evaluation.

Author response:
Thank you for your valuable suggestion. We acknowledge the importance of incorporating such a baseline, and as a response, we have included new experimental results in our study. Concretely, we evaluate the performance on BBH tasks of the top 5 generally useful LoRA modules presented in Table 2 (More details of these LoRA modules can be found in the paper). The additional experimental results are outlined below.

| Task                                      | WIQA: Last | RACE: Right | WIQA: First | AdversaraialQA: BiDAF  | WebQuestions: Answer |
|-------------------------------------------|------------|-------------|------------|-------|-------|
| Boolean Expressions                       | 52.7       | 58.0        | 52.7       | 54.7  | 53.3  |
| Causal Judgement                          | 55.2       | 63.2        | 55.2       | 57.5  | 57.5  |
| Date Understanding                        | 17.3       | 19.3        | 17.3       | 16.7  | 15.3  |
| Disambiguation                            | 0.0        | 0.0         | 0.0        | 0.0   | 0.0   |
| Dyck Languages                            | 0.7        | 0.7         | 0.7        | 1.3   | 1.3   |
| Formal Fallacies                          | 51.3       | 51.3        | 51.3       | 51.3  | 51.3  |
| Geometric Shapes                          | 8.0        | 13.3        | 8.0        | 6.7   | 7.3   |
| Hyperbaton                                | 16.7       | 44.0        | 16.7       | 1.3   | 6.0   |
| Logical Deduction (five objects)          | 23.3       | 28.0        | 23.3       | 19.3  | 20.7  |
| Logical Deduction (seven objects)         | 22.0       | 26.0        | 22.0       | 10.7  | 12.0  |
| Logical Deduction (three objects)         | 0.7        | 9.3         | 0.7        | 0.0   | 0.0   |
| Movie Recommendation                      | 63.3       | 62.7        | 63.3       | 56.7  | 63.3  |
| Multistep Arithmetic                      | 0.7        | 0.7         | 0.7        | 0.7   | 0.7   |
| Navigate                                  | 47.3       | 50.0        | 47.3       | 47.3  | 47.3  |
| Object Counting                           | 34.7       | 34.0        | 34.7       | 35.3  | 35.3  |
| Penguins in a Table                       | 45.7       | 41.3        | 45.7       | 39.1  | 43.5  |
| Reasoning about Colored Objects           | 40.0       | 37.3        | 40.0       | 31.3  | 30.7  |
| Ruin Names                                | 22.0       | 21.3        | 22.0       | 17.3  | 22.7  |
| Salient Translation Error Detection       | 36.7       | 34.7        | 36.7       | 32.7  | 37.3  |
| Snarks                                    | 52.6       | 55.1        | 52.6       | 47.4  | 52.6  |
| Sports Understanding                      | 56.0       | 58.7        | 56.0       | 55.3  | 55.3  |
| Temporal Sequences                        | 16.7       | 17.3        | 16.7       | 12.7  | 17.3  |
| Tracking Shuffled Objects (five objects)  | 12.0       | 12.0        | 12.0       | 10.7  | 12.0  |
| Tracking Shuffled Objects (seven objects) | 6.7        | 6.7         | 6.7        | 6.7   | 6.7   |
| Tracking Shuffled Objects (three objects) | 20.7       | 30.7        | 20.7       | 10.7  | 25.3  |
| Web of Lies                               | 54.7       | 54.0        | 54.7       | 54.0  | 54.0  |
| Word Sorting                              | 1.3        | 1.3         | 1.3        | 1.3   | 1.3   |
| **Avg Performance per Task**              | **28.1**    | **30.8**     | **28.1**    | **25.1** | **27.0** |
| **$\Delta$ FLAN-T5-large**                    | **1.1**     | **3.8**      | **1.1**     | **-1.9** | **0.0**  |

Notably, the findings indicate that, in most cases, these top LoRA modules perform similarly as the original FLAN-T5-large. And there is only one module "RACE: Right" stands out as significantly surpassing FLAN-T5-large. However, it's important to highlight that this individual performance doesn't reach the level achieved by LoraHub. These results reinforce the conclusion that module compositions can indeed enhance the overall performance. We appreciate your insightful suggestion, as it has enabled us to more effectively demonstrate the efficacy and necessity of our approach. Your perspective is invaluable to our work.

Reviewer comment (Experiments):
The baseline corresponding to the retrieval of a trained LoRAs, when given a handful of examples from unseen tasks, is missing. For example, see https://arxiv.org/abs/2302.03202

Author response:
Thank you for pointing out the absence of a baseline corresponding to the retrieval of trained LoRAs given a handful of examples from unseen tasks. We sincerely appreciate your sharing of the related work (https://arxiv.org/abs/2302.03202), and we acknowledge its interesting and novel methods.

In response to your feedback, we have included a discussion of this work in Section 5 ("Does composing LoRA modules extend beyond the single module’s benefits?") in the revised paper. And we conducted an experiment involving the LoRA retrieval method as our baseline. Notably, in order to ensure a fair comparison, we designed a LoRA retrieval mechanism based on the loss derived from few-shot examples. Specifically, we ranked all LoRA module candidates based on their loss on the few-shot examples and evaluated the best candidate on the test set of the unseen task. The performance comparison is as below.

| $\rm Task$                                    | $\rm LoRA  Retrieval$ | $\rm LoraHub_{avg}$ | $\rm LoraHub_{best}$ |
|-----------------------------------------|-------------|-----------------|------------------|
| Boolean Expressions                     | 53.3 | 55.5            | 60.7             |
| Causal Judgement                        | 63.2  | 54.3            | 63.2             |
| Date Understanding                      | 20.7 | 32.9            | 45.3             |
| Disambiguation                       | 0           | 45.2            | 68               |
| Dyck Languages                          | 0.7 | 1.0               | 2.7              |
|Formal Fallacies                        | 51.3 | 52.8            | 59.3             |
| Geometric Shapes                        | 9.3 | 7.4             | 18.7             |
| Hyperbaton                              | 57.3 | 62.8            | 72.7             |
| Logical Deduction (five objects)          | 26          | 36.1            | 40               |
| Logical Deduction (seven objects)         | 36.8 | 36.8            | 46               |
| Logical Deduction (three objects)         | 10.0          | 45.7            | 52.7             |
| Movie Recommendation                    | 63.3 | 55.3            | 62               |
| Multistep Arithmetic Two                | 1.3 | 0.4             | 1.3              |
| Navigate                                | 49.3 | 47.1            | 51.3             |
| Object Counting                         | 37.3 | 33.7            | 36.7             |
| Penguins in a Table                  | 41.3 | 35.9            | 47.8             |
| Reasoning about Colored Objects         | 40          | 40              | 44.7             |
| ruin_names                              | 21.3 | 24.4            | 28.7             |
| Salient Translation Error Detection     | 46          | 36              | 42.7             |
| Snarks                                  | 50          | 56.9            | 61.5             |
| Sports Understanding                    | 54.7 | 56.7            | 62.7             |
| Temporal Sequences                      | 17.3 | 18.2            | 21.3             |
| Tracking Shuffled Objects (five objects)  | 12.0          | 12.3            | 16.7             |
| Tracking Shuffled Objects (seven objects) | 6.7 | 7.7             | 15.3             |
| Tracking Shuffled Objects (three objects) | 30.7 | 29.2            | 31.3             |
| Web of Lies                             | 55.3 | 50.1            | 57.3             |
| Word Sorting                            | 0.7 | 1.1             | 1.3              |
|  **Average Performance**                                       | **31.7** | **34.7**        | **41.2**         |

As depicted in the table above, the performance of LoRA retrieval is notably impressive, positioning it as a strong baseline. However, in comparison to LoraHub, the performance of LoRA retrieval is relatively less favorable. This underscores the imperative and effectiveness of LoRA module composition.

We extend our sincere gratitude to the reviewer for providing insightful perspectives that have significantly enriched the comprehensiveness of our evaluation.

Reviewer comment (Theory):
Can you elaborate on the interpretation of negative coefficients for the LoRA weights?

Author response:
We apologize if our paper's description did not accurately convey our approach. In the equation: $\hat{m} = (w_1A_1+w_2A_2)(w_1B_1+w_2B_2)$, when the weight parameter ($w$) assumes a negative value, the operation remains consistent—subtracting the corresponding parameters of LoRA at the specified positions. No additional adjustments are made for negative coefficients. We hope this clarification provides a clearer understanding of the process.

Reviewer comment (Experiments):
Why was the decision made to maintain the same rank for the composed LoRA module? Have higher rank matrices been explored and if so, what were the findings?

Author response:
Thanks for your question! In fact, we have investigated the impact of different ranks in Section 5 (`Will the rank of LoRA modules impact the performance of LoraHub learning`) and Appendix B. Our analysis indicates that, for FLAN-T5, the choice of rank has minimal impact. However, for T5, it still exerts some influence. Empirical findings reveal that, in comparison to rank values of $4$ or $64$, a rank value of $16$ consistently demonstrates superior performance across different runs, both in terms of average and best performance.

| Task | Rank 4$_{avg}$ | Rank 4$_{best}$ | Rank 16$_{avg}$ | Rank 16$_{best}$ | Rank 64$_{avg}$ | Rank 64$_{best}$ |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Average Performance Per Task | 16.1 | 24.2 | 20.8 | 30.7 | 14.8 | 21.4 |

We hope that this exploration addresses your concerns and brings satisfaction. We anticipate that future research endeavors will contribute to a deeper understanding of the factors influencing these effects.

Reviewer comment (Experiments):
Is the selection of 20 LoRAs for unseen tasks fixed or is there an iterative procedure to this selection? How does this choice impact the method’s performance on unseen tasks?

Author response:
Thank you for pointing this out! In our initial experiment, the selection of the $20$ LoRA module candidates is conducted randomly, serving as the primary source of randomness in our method.

Acknowledging the need for improvement in the aspect, we explored a straightforward approach ($\rm LoraHub_{filter}$) in the selection of LoRA module candidates. Specifically, we first identified $20$ LoRA module candidates with the lowest loss on the few-shot examples, with the goal of minimizing introduced variance and fostering a more consistent performance. The experimental results demonstrate that this approach contributes to a slight improvement in the average performance. These results have been included in Appendix D for further reference. We hope the updated experiment addresses your concerns.

| $\rm Task$                                   | $\rm LoraHub_{avg}$ | $\rm LoraHub_{filter}$ |
|-------------------------------------------|-------------|----------------|
| Boolean Expressions                       | 55.5        | 60.0           |
| Causal Judgement                          | 54.3        | 52.9           |
| Date Understanding                        | 32.9        | 33.3           |
| Disambiguation                            | 45.2        | 62.7           |
| Dyck Languages                            | 1.0         | 0.0            |
| ... | | |
| Tracking Shuffled Objects (three objects) | 29.0        | 32.7           |
| Web of Lies                               | 53.0        | 46.0           |
| Word Sorting                              | 1.1         | 1.3            |
| **Avg Performance Per Task**                  |**34.7**       | **35.4**          |

Reviewer comment (Experiments):
I believe the strength of the method lies in cases where there are a handful of examples from unseen tasks. If it's otherwise, that it is beneficial to use a larger number of examples, it would make sense to compare against methods like IA3 [https://arxiv.org/abs/2205.05638], which fine-tunes efficiently on few-shot examples from unseen tasks.

Author response:
Thank you for your valuable suggestion. We have incorporated your feedback by updating Table 1 to provide a more focused comparison between our method and IA3, a representative efficient few-shot fine-tuning method. The experimental results are as below.

| $\rm Task$                                      |  $\rm Zero$    | $\rm ICL_{avg}$ | $\rm IA3_{avg}$ | $\rm LoraHub_{avg}$ |
|-------------------------------------------|-----------|-------------|-------------|-----------------|
| Boolean Expressions                       | 54.0      | 59.6        | 56.2        | 55.5            |
| Causal Judgement                          | 57.5      | 59.4        | 60.2        | 54.3            |
| Date Understanding                        | 15.3      | 20.4        | 20.0        | 32.9            |
| Disambiguation                            | 0.0       | 69.1        | 0.0         | 45.2            |
| Dyck Languages                            | 1.3       | 0.9         | 4.2         | 1.0             |
| Formal Fallacies                          | 51.3      | 55.3        | 51.5        | 52.8            |
| Geometric Shapes                          | 6.7       | 19.6        | 14.7        | 7.4             |
| Hyperbaton                                | 6.7       | 71.8        | 49.3        | 62.8            |
| Logical Deduction§ (five objects)         | 21.3      | 39.1        | 32.7        | 36.1            |
| Logical Deduction§ (seven objects)        | 12.7      | 40.7        | 33.8        | 36.8            |
| Logical Deduction§ (three objects)        | 0.0       | 51.6        | 8.5         | 45.7            |
| Movie Recommendation                      | 62.7      | 55.8        | 61.8        | 55.3            |
| Multistep Arithmetic                      | 0.7       | 0.7         | 0.7         | 0.4             |
| Navigate                                  | 47.3      | 45.3        | 46.2        | 47.1            |
| Object Counting                           | 34.7      | 32.4        | 35.1        | 33.7            |
| Penguins in a Table                       | 43.5      | 41.3        | 45.0        | 35.9            |
| Reasoning about Colored Objects           | 32.0      | 40.2        | 40.7        | 40.0            |
| Ruin Names                                | 23.3      | 19.3        | 24.4        | 24.4            |
| Salient Translation Error Detection       | 37.3      | 47.3        | 37.1        | 36.0            |
| Snarks                                    | 50.0      | 54.2        | 53.9        | 56.9            |
| Sports Understanding                      | 56.0      | 54.7        | 55.1        | 56.7            |
| Temporal Sequences                        | 16.7      | 25.1        | 18.2        | 18.2            |
| Tracking Shuffled Objects (five objects)  | 12.0      | 12.0        | 12.0        | 12.3            |
| Tracking Shuffled Objects (seven objects) | 6.7       | 6.7         | 6.7         | 7.7              |
| Tracking Shuffled Objects (three objects) | 24.7      | 31.1        | 30.7        | 29.2            |
| Web of Lies                               | 54.0      | 53.8        | 54.2        | 50.1            |
| Word Sorting                              | 1.3       | 0.5         | 1.3         | 1.1             |
| **Avg Performance Per Task**              | **27.0**  | **37.3**    | **31.6**    | **34.7**        |
| **Avg Tokens Per Example**                | **111.6** | **597.8**   | **111.6**   | **111.6**       |
| **Gradient-based Training**               | **No**    | **No**      | **Yes**     | **No**          |


The experimental results indicate that our method exhibits competitive, and even superior, average performance compared to IA3. We appreciate your insightful suggestion, which has led to a more targeted evaluation of our approach in comparison to IA3.

Reviewer comment (Evaluation):
How does the absolute value of LoRA weight not exceeding 1.5 relate to the method's performance in section 4.2, and is there a particular significance to this threshold?

Author response:
Thank you for question! In fact, the final LORA parameters will all be within 1. The choice of 1.5 is more to adapt to the implementation of parameter-free optimization methods, providing a good initial step size. This parameter plays a role similar to the learning rate in the deep learning process. Choosing other parameters may increase the number of steps required for convergence (used to adjust the step size).

Reviewer comment (Experiments):
It would be beneficial to include parameter efficient fine-tuning and traditional fine-tuning performance in Table 1, especially with the setup of a limited number of examples.

Author response:
Thank you for your constructive feedback regarding Table 1. We have thoroughly reviewed your suggestions, as well as those from other reviewers, and made the necessary adjustments to improve the comprehensiveness of our experimental results. In response to your suggestion, we have conducted additional experiments on LoRA tuning (LoRA), full fine-tuning (FFT), and report their average (avg) and best (best) performance across three different runs. Below is a summary of the results obtained from these experiments:

| $\rm Task$                                      | $\rm ICL_{best}$ | $\rm IA3_{best}$ | $\rm LoRA_{best}$ | $\rm FFT_{best}$ | $\rm LoraHub_{best}$ | $\rm ICL_{avg}$ | $\rm IA3_{avg}$ | $\rm LoRA_{avg}$ | $\rm FFT_{avg}$ | $\rm LoraHub_{avg}$ |
|-------------------------------------------|--------------|--------------|---------------|--------------|------------------|-------------|-------------|--------------|-------------|-----------------|
| Boolean Expressions                       | 62.7         | 58.0         | 60.7          | 65.3         | 60.7             | 59.6        | 56.2        | 56.0         | 62.2        | 55.5            |
| Causal Judgement                          | 59.8         | 62.1         | 57.5          | 60.9         | 63.2             | 59.4        | 60.2        | 55.6         | 57.5        | 54.3            |
| Date Understanding                        | 21.3         | 20.7         | 40.7          | 67.3         | 45.3             | 20.4        | 20.0        | 35.8         | 59.3        | 32.9            |
| Disambiguation                            | 69.3         | 0.0          | 68.7          | 70.7         | 68.0             | 69.1        | 0.0         | 68.0         | 68.2        | 45.2            |
| Dyck Languages                            | 2.0          | 4.7          | 25.3          | 33.3         | 2.7              | 0.9         | 4.2         | 22.2         | 19.5        | 1.0             |
| Formal Fallacies                          | 59.3         | 52.0         | 56.7          | 56.0         | 59.3             | 55.3        | 51.5        | 53.6         | 54.0        | 52.8            |
| Geometric Shapes                          | 20.0         | 15.3         | 28.7          | 39.3         | 18.7             | 19.6        | 14.7        | 24.0         | 31.1        | 7.4             |
| Hyperbaton                                | 72.7         | 49.3         | 57.3          | 82.0         | 72.7             | 71.8        | 49.3        | 55.3         | 77.3        | 62.8            |
| ... |  | | | | | | | | 
| Tracking Shuffled Objects (seven objects) | 6.7          | 6.7          | 12.0          | 10.0         | 15.3             | 6.7         | 6.7         | 10.0         | 9.8         | 7.7             |
| Tracking Shuffled Objects (three objects) | 31.3         | 30.7         | 32.0          | 36.0         | 31.3             | 31.1        | 30.7        | 30.9         | 32.0        | 29.2            |
| Web of Lies                               | 54.0         | 54.7         | 55.3          | 54.0         | 57.3             | 53.8        | 54.2        | 52.7         | 48.2        | 50.1            |
| Word Sorting                              | 0.7          | 1.3          | 5.3           | 6.0          | 1.3              | 0.5         | 1.3         | 4.9          | 4.9         | 1.1             |
| **Average Performance Per Task**            | **38.4**     | **32.1**     | **40.9**      | **46.2**     | **41.2**         | **37.3**        | **31.6**        | **37.7**         | **42.1**        | **34.7**            |
| **Avg Tokens Per Example**       |      **597.8**  | **111.6**  | **111.6**   | **111.6**  | **111.6**      |      **597.8**  | **111.6**  | **111.6**   | **111.6**  | **111.6**      |
| **Gradient-based Training**             |     **No**     | **Yes**    | **Yes**     | **Yes**    | **No**     |      **No**     | **Yes**    | **Yes**     | **Yes**    | **No**     |

Reviewer comment (Experiments):
**Optimization of $w$ for many tasks and robustness of `LoRAHub`:** It's not clear to me how the optimization methods scale to a higher number of upstream LoRA modules either in terms of cost (see previous point) or performance: In **Figure 4**, we see that increasing the number of LoRA module does not always improve performance but strongly affects the variance of the outputs. This suggests that the optimization procedure is either noisy and/or does not converge well. As a consequence, it means that the number of candidate upstream LoRA modules, $N$, must be carefully selected (and the optimal $N$ even seems to be task dependent from **Figure 4**) which introduces an additional hyperparameter. This can be an important limitation for real-life applications.

Author response:
We are grateful for the thoughtful feedback provided by the reviewer, especially regarding alternative methods to address variance in the optimization process. We highly appreciate the suggestions for exploring a hierarchical or curriculum-like strategy to enhance robustness beyond the prefiltering approach.

While these suggestions are intriguing, we would like to clarify that, based on our in-depth analysis, the primary source of variance in our current method is not related to gradient-free optimization algorithms but rather associated with the LoRA candidate modules. In other words, once the candidates are determined, random seeds have minimal impact on the final performance. Hence, we posit that the observed instability primarily arises from the inherent challenge of balancing the `quantity` and `quality` of the LoRA module candidates. This hypothesis receives partial support from our analysis in Section 5, where attempts to increase the number of LoRA modules led to a noticeable rise in model performance volatility.

In response to this observation, we explored a straightforward approach in the selection of LoRA module candidates. Specifically, we first identified $20$ LoRA module candidates with the lowest loss on the few-shot examples, with the goal of minimizing introduced variance and fostering a more consistent performance. This strategic selection process, named as $\rm LoraHub_{filter}$, has notably contributed to a slight enhancement in average performance compared to  $\rm LoraHub_{avg}$. Below we show a brief summary of the results, and the detailed results can be found in Appendix D.

| $\rm Task$                                   | $\rm LoraHub_{avg}$ | $\rm LoraHub_{filter}$ |
|-------------------------------------------|-------------|----------------|
| Boolean Expressions                       | 55.5        | 60.0           |
| Causal Judgement                          | 54.3        | 52.9           |
| Date Understanding                        | 32.9        | 33.3           |
| Disambiguation                            | 45.2        | 62.7           |
| Dyck Languages                            | 1.0         | 0.0            |
| ... | | |
| Tracking Shuffled Objects (three objects) | 29.0        | 32.7           |
| Web of Lies                               | 53.0        | 46.0           |
| Word Sorting                              | 1.1         | 1.3            |
| **Avg Performance Per Task**                  |**34.7**       | **35.4**          |


However, the method has also introduced additional reasoning overhead, as the losses of all LoRA module candidates need to be calculated. We have also discussed the variance in Appendix G. We appreciate your insights and remain open to further discussion on refining our methodology. We intend to explore additional strategies in future investigations.

Reviewer comment (Experiments):
On the topic of **optimizing $w$ for many tasks**, I am wondering if authors have considered alternative techniques which might be more robust to optimization noise (beyond the prefiltering strategy mentioned in Section 7): e.g. a hierarchical approach (optimize $w$ for multiple random subsets of 20 LoRA candidates, then learn $w'$ for these agglomerated modules) or a curriculum like approach (gradually drop some of the candidates when optimizing $w$ if they are consistently given very low weights) ?

Author response:
We are grateful for the thoughtful feedback provided by the reviewer, especially regarding alternative methods to address variance in the optimization process. We highly appreciate the suggestions for exploring a hierarchical or curriculum-like strategy to enhance robustness beyond the prefiltering approach.

While these suggestions are intriguing, we would like to clarify that, based on our in-depth analysis, the primary source of variance in our current method is not related to gradient-free optimization algorithms but rather associated with the LoRA candidate modules. In other words, once the candidates are determined, random seeds have minimal impact on the final performance. Hence, we posit that the observed instability primarily arises from the inherent challenge of balancing the `quantity` and `quality` of the LoRA module candidates. This hypothesis receives partial support from our analysis in Section 5, where attempts to increase the number of LoRA modules led to a noticeable rise in model performance volatility.

In response to this observation, we explored a straightforward approach in the selection of LoRA module candidates. Specifically, we first identified $20$ LoRA module candidates with the lowest loss on the few-shot examples, with the goal of minimizing introduced variance and fostering a more consistent performance. This strategic selection process, named as $\rm LoraHub_{filter}$, has notably contributed to a slight enhancement in average performance compared to  $\rm LoraHub_{avg}$. Below we show a brief summary of the results, and the detailed results can be found in Appendix D.

| $\rm Task$                                   | $\rm LoraHub_{avg}$ | $\rm LoraHub_{filter}$ |
|-------------------------------------------|-------------|----------------|
| Boolean Expressions                       | 55.5        | 60.0           |
| Causal Judgement                          | 54.3        | 52.9           |
| Date Understanding                        | 32.9        | 33.3           |
| Disambiguation                            | 45.2        | 62.7           |
| Dyck Languages                            | 1.0         | 0.0            |
| ... | | |
| Tracking Shuffled Objects (three objects) | 29.0        | 32.7           |
| Web of Lies                               | 53.0        | 46.0           |
| Word Sorting                              | 1.1         | 1.3            |
| **Avg Performance Per Task**                  |**34.7**       | **35.4**          |


However, the method has also introduced additional reasoning overhead, as the losses of all LoRA module candidates need to be calculated. We have also discussed the variance in Appendix G. We appreciate your insights and remain open to further discussion on refining our methodology. We intend to explore additional strategies in future investigations.

Reviewer comment (Evaluation):
**Figure 3 and variance**: I think **Figure 3** would be much more convincing with error bars or some notion of variance. This figure's aim is to illustrate that "LoRA with few samples does not work as well as LoraHub's few-shot learning", however the results are only available for 3 tasks, for which the assumption only holds until 20 few-shot samples; therefore it's not clear how the insight generalizes to more few-shot settings.

Author response:
In response to your insightful feedback, we have dedicated efforts to conduct a thorough analysis of various methods, including LoRA tuning (LoRA), full fine-tuning (FFT), and IA3 fine-tuning (IA3) as suggested by Reviewer 8M9N. Surprisingly, upon averaging the performance of LoRA tuning, our conclusions differ from the observations presented in Figure 3. Essentially, our new experimental results, as presented in Table 1 in response to Question 2, suggest that the previously highlighted instability of LoRA tuning, especially in scenarios with extremely limited examples, may have been overstated. As it stands, the overall performance of LoRA tuning appears to be great.

In light of these findings, we have taken the step of removing statements about the instability of LoRA tuning in Section 2, Section 4.2 and Section 5. Additionally, we have temporarily removed Figure 3 from the paper to prevent the dissemination of potentially inaccurate conclusions. In the final version, we are committed to enhancing the comprehensiveness of our presentation by expanding Figure 3 to encompass all tasks in BBH with varying numbers of examples. This extension aims to provide a more nuanced and thorough representation of our research findings. Your feedback has been invaluable in refining the precision and accuracy of our paper.

Reviewer comment (Evaluation):
The main weakness of the paper is its performance compared to in-context learning (ICL), as highlighted in Table 1. The authors acknowledge this in the paper but justify it by saying that their method uses fewer tokens in their fine-tuning process compared to in-context learning.

Author response:
## Reason 1. Inference Efficiency
---

While we acknowledge the effectiveness of in-context learning (ICL) for generalization, we wish to emphasize that our proposed LoraHub is not intended to replace ICL but rather to offer a complementary strategy with performance-efficiency trade-offs. An essential benefit of LoraHub, highlighted in our paper, is its focus on improving inference efficiency by utilizing fewer prompt tokens per example. As illustrated in Table 1, LoraHub demonstrates a significant advantage with a reduced token requirement per example — `111.6` tokens compared to the `597.8` tokens needed for ICL.

The efficiency of inference is paramount for real-world deployments, particularly when dealing with lots of inference examples. With fewer tokens per example during inference, our method significantly reduces computational overhead and enables faster responses. The commitment to optimizing inference efficiency aligns seamlessly with a broader research trend, as evidenced by several recent studies actively exploring approaches to reduce the number of input tokens [1,2,3,4,5,6]. For example, as stated in [1], in-context learning can be inefficient because it makes the input prompt much longer, consuming valuable space in the context window and leading to larger computational costs.

[1]. Wangchunshu Zhou, Yuchen Eleanor Jiang, Ryan Cotterell, Mrinmaya Sachan, Efficient Prompting via Dynamic In-Context Learning. ArXiv abs/2305.11170 (2023) 

[2]. Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, Furu Wei, In-context Autoencoder for Context Compression in a Large Language Model. ArXiv abs/2307.06945 (2023)

[3]. Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, Adapting Language Models to Compress Contexts. ArXiv abs/2305.14788 (2023)

[4]. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu, LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ArXiv abs/2310.06839 (2023)

[5]. Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, Compressing Context to Enhance Inference Efficiency of Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

[6]. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu, LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing