Paper: LoRA ensembles for large language model fine-tuning (ICLR-2024)

Reviewer comment (Experiments):
The current set of experiments considered is limited to multiple choice questions (predicting only a single token). While the method is indeed compatible with generative tasks, there are no such tasks considered in the experiments. The results would be more convincing with generative tasks since LLMs are commonly used for complex tasks.

Author response:
The main reason we choose to stick to multiple-choice QA is that the metric for measuring calibration and uncertainty quantification is well-defined. On open-ended generative tasks, how to measure output's calibration and uncertainty remains an open question. For example, [3] shows that there are multiple ways to quantify the uncertainty in open-ended QA problems while each metric could have different behaviors. We consider combining these advanced metrics with LoRA ensemble as a future research problem.

Reviewer comment (Experiments):
LoRA ensemble requires finetuning datasets. However, LLMs are commonly used in zero or few-shot ways in many scenarios. How does the method perform if only a few finetuning data is available?

Author response:
When there is only a limited amount of fine-tuning data available, the fine-tuned model would demonstrate severe overconfidence, as we can see from the experiment results from MMLU social sciences and STEM experiments, where there are only 397 and 411 training samples available: The single model (LoRA M=1) shows high ECE and NLL while LoRA ensembling significantly resolves the overconfidence. Our results show that the performance can be further improved by combining regularization with LoRA ensemble.

Reviewer comment (Presentation):
The presentation of experiment results are scattered. It would be easier to compare different methods if they are all listed in the same table/figure. Currently, comparisons to different baselines are in separate tables/figures, making it a bit hard to read.

Author response:
Thanks for the suggestion, we plan to add a table that summarizes the results from all different methods and baselines in the appendix for easier comparison.

Reviewer comment (Evaluation):
In Figure 3, 4, 5, and 6, why ECE increases as number of epoch increases? In this case, isn't the method making calibration worse by LoRA finetuning?

Author response:
Note that the training objective for LoRA fine-tuning is cross-entropy loss, therefore the fine-tuned model would tend to keep reducing the training loss by making the predictive distribution sharper, this is fine on the training set, as the model can reach %100 accuracy, however on the test set, the model would suffer from *increasing* calibration error (ECE) and negative log-likelihood (NLL) as the model is making overconfident and wrong predictions. Our results show that this can be alleviated by using LoRA ensemble, while weight decay regularization can further prevent the ECE from growing up.

Reviewer comment (Experiments):
In many cases, finetuning on specific dataset can compromise the model's performance on other tasks. Does LoRA ensemble suffer the same problem?

Author response:
The top row in Figure. 5 shows the performance on MMLU from the model fine-tuned on commonsense QA. We can see that, while both the single LoRA and LoRA ensembles show decreased accuracy as fine-tuning proceeds, LoRA ensembles still show improvement of performance upon the single fine-tuned model, we believe this implies that LoRA ensembles suffer less from performance degradation.

Reviewer comment (Evaluation):
Another concern is that no computational costs is reported in this paper. I understand the inference costs of LoRA ensemble is much lower than traditional finetuning ensemble, but it is good to demonstrate this.

Author response:
## Computational cost

We provide a summarization of loading time and memory cost below under Llama-13b and LoRA of rank 8. Note that the memory requirements for full-model ensembling are beyond the capacity of an 80GB A100 GPU, therefore at test time, one may need to load and unload the models for ensembling, causing extra overhead.

|              | Full model ensemble (M=5) | LoRA ensemble (M=5) |
|--------------|---------------------------|---------------------|
| Loading time | ~30s                      | ~6.5s               |
| Memory usage | ~129GB                     | ~26GB               |

Reviewer comment (Presentation):
The title in the paper doesn't exactly match the title in the openreview system.

Author response:
## Discrepancy in title
We apologize for the mistake, we will fix the title in later revisions.

Reviewer comment (Experiments):
The focus of the paper is only on prediction ensembles, which neglects the important weight ensemble methods [1,3,4,6]. The paper argues that maintaining an ensemble of, for instance, 5 LLMs in memory can be challenging in certain scenarios. However, it's worth noting that weight ensembles require the maintenance of just one model. Recent papers adopt online ensemble methods [3] that continuously average weight parameters.

Author response:
Thanks for pointing out the related literature on weight averaging. We will add discussion to weight space averaging methods in later revisions.  It is important to clarify, though, that we use the term "ensemble" to represent output space ensembling following classic literature ([2]), we are unaware of the potential confusion with weight space averaging during writing, which we will further clarify in later revisions.

We opted not to include weight averaging methods as a baseline for several reasons. Firstly, in studies focusing on ensembling to enhance neural networks' calibration and uncertainty quantification, such as [4, 5, 6], weight averaging methods like SWA [7] are rarely used as a baseline. The reason is that weight space averaging aims at solving problems fundamentally different from output space ensembling: Weight space averaging methods target acquiring models with **better generalization**, which is usually measured by accuracy. In contrast, output space ensembling methods, including Bayesian neural networks, MC dropout, and deep ensembles, aim to address issues of overconfidence and poor calibration, evaluated using metrics like NLL and ECE. Notably, weight-averaging methods are **not** recognized for resolving calibration and overconfidence issues, with calibration error and out-of-distribution detection performance seldom serving as evaluation metrics in weight-averaging studies.

Moreover, an examination of Tables 3, 4, and 5 in the appendix of [1] reveals that SWA's calibration error is consistently surpassed by output space ensembling methods. To further investigate the impact of weight averaging on LLM fine-tuning, we applied SWA to LoRA by averaging weights across five iterations. Our results, detailed below, indicate that while SWA does enhance accuracy, it fails to improve NLL and ECE. This suggests that SWA is not effective in addressing the overconfidence and calibration problems central to our study."


| Method              | Accuracy | Negative log-likelihood (NLL) | Expected calibration error (ECE)|
|---------------------|----------|-------------------------|----------------------------|
| LoRA (M=1)          | 0.808    | 0.948                   | 0.153                      |
| LoRA (M=1, SWA)     | 0.813    | 1.18                    | 0.156                      |
| LoRA Ensemble (M=5) | **0.832**    | **0.581**                   | **0.064**                      |

Reviewer comment (Experiments):
The method's evaluation is restricted to small datasets, and its scalability remains unverified. Furthermore, the absence of actual ensemble baselines is notable. For example, [1,3] employ ensemble techniques while training the model only once, which is highly relevant to the task addressed in this paper.

Author response:
It is unclear to us what type of "scalability" the reviewer refers to. The biggest dataset we used is commonsense QA, which has 9.8K training samples. With LoRA ensemble, we are also able to perform large-scale ensembling experiments using 15 ensemble components (Fig. 6), which is not possible for full model fine-tuning. We also did not see any obstacles in applying LoRA ensembles to larger datasets or models without trouble. Essentially, if LoRA can be applied for fine-tuning, LoRA ensemble can be applied to boost the performance. We are happy to incorporate more experiments if the reviewer could elaborate more on scalability.

In the "Ensembling of Neural Networks" and "Ensembling in LLMs" paragraphs in the related work section, we discussed works on ensembling methods. However, these methods only consider training ensembles from scratches and it is unclear how one can apply these methods for LLM fine-tuning: Adapting these methods to LLM fine-tuning is beyond the scope of our paper. We are also not aware of any works that perform (output space) ensembling for LLM fine-tuning, we are happy to include these methods as baselines if the reviewer could kindly point out references to these works.