Paper: FedLoRA: When Personalized Federated Learning Meets Low-Rank Adaptation (ICLR-2024)

Reviewer comment (Experiments):
The concern of overfitting associated with the training of the full-rank matrix needs to be adequately addressed. This aspect could affect the generalization capability of the aggregated parameters.

Author response:
Thanks for your review.

We feel that there might be some miscommunication here. In fact,  **FedLoRA has the same amount of shared parameters compared with the baseline method FedAvg**. Therefore, FedLoRA does not bring additional risks of over-fitting. Here are detailed explanations:

First, we would like to clarify that our intention is not to use LoRA for training large language models (LLMs) in Federated Learning (FL). Instead, we draw inspiration from the concept of LoRA, which was originally conceived for fine-tuning LLMs, and adapt it to the context of Personalized Federated Learning (PFL). Consequently, the full-rank model employed in FedLoRA essentially resembles a traditional neural network commonly used in the FL literature. Therefore, FedLoRA has **the same amount of shared parameters** compared to the majority of FL methods that upload models, such as FedAvg. As a result, our FedLoRA does not bring additional risks of over-fitting.

Reviewer comment (Experiments):
The method's scalability seems constrained by model size. As model size increases, transferring the full-rank matrix to the server becomes impractical.

Author response:
Thanks for your review.

This question has been partially addressed in our response to Q2. In our reply to Q2, we emphasize that **FedLoRA maintains an equivalent number of shared parameters compared to the baseline method**, FedAvg. Consequently, in comparison to the baseline algorithm, FedAvg, our approach does not incur additional communication costs when applied to a large-sized model.

Reviewer comment (Experiments):
Is it possible to extend the experiments to use an ImageNet-pretrained ResNet to finetune on CIFAR10/CIFAR100? I assume in this paper we need to learn $\sigma$ because we are training from scratch and we still need to learn the feature extractor. I expect that if the feature extractor is already trained (as an ImageNet-pretrained model), we should expect that $\tau$ is learned more often than $\sigma$. ($\Delta \sigma$ should be much smaller than $\Delta \tau$). This study could be a strong support on the client-specific knowledge statement.

Author response:
Thanks for your review. As requested by the reviewer, we add experiments about how the update for global part and personalized part (denoted as $\Delta \sigma$ and $\Delta \tau$) affect the final performance. Specifically, we initialize the $\sigma$ with ImageNet pre-trained weights and conduct experiment on the CIFAR-100 dataset in the Dirichlet non-IID scenario with $\alpha=0.1$. 

We control the value of $E_{lora}$ (higher $E_{lora}$ means update $\tau$ more often, and results in larger $\Delta \tau$) at different levels. The results are summarized in the table below.

| $E_{lora}$      | 0     | 1     | 2     | 3     | 4     |
| --------------- | ----- | ----- | ----- | ----- | ----- |
| Accuracy        | 39.35 | 50.37 | 71.47 | 72.00 | 72.42 |
| $\Delta \sigma$ | 37.99 | 31.59 | 22.12 | 16.16 | 10.57 |
| $\Delta \tau$   | 0.00  | 26.48 | 89.68 | 98.26 | 99.63 |

From this table, we can conclude that **when $\Delta \tau$ is larger than $\Delta \sigma$, FedLoRA achieves better results**. This indicates that with the pre-trained weights, the $\tau$ should be updated more often than $\sigma$ to learn client-specific knowledge. This aligns with the reviewer's expectation and can serve as strong support for the client-specific knowledge statement.

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Reviewer comment (Experiments):
The paper lacks experiments and analysis about training and communication cost, since the new training strategy needs to train two times for full-rank matrix and low-rank matrix, and the cost for transmitting full-rank matrixes may be huge because they are much larger than low-rank matrix.

Author response:
Thank you for your feedback. We feel that there might be some miscommunication here. We would like to emphasize that FedLoRA has **the same communication overhead** and **lower computation overhead** than the baseline method FedAvg. Here are the detailed explanations:

First, we would like to clarify that our intention is not to use LoRA for training large language models (LLMs) in Federated Learning (FL). Instead, we draw inspiration from the concept of LoRA, which was originally conceived for fine-tuning LLMs, and adapt it to the context of Personalized Federated Learning (PFL). Consequently, the full-rank model employed in FedLoRA essentially resembles a traditional neural network commonly used in the FL literature. As a result, FedLoRA shares the same communication overhead as the majority of FL methods since these methods, including FedAvg, all involve uploading the full-rank model.

In terms of computational cost, it's worth noting that FedLoRA actually incurs a lower computational overhead compared to many traditional FL methods, including FedAvg. This advantage can be attributed to our proposed alternating training technique. For example, in a FL setting where all clients need to train locally for $E$ epochs before uploading parameters, then for FedAvg, each epoch requires training a standard full-rank model; but for FedLoRA, it trains $E_{lora}$ rounds of the low-rank part (fewer trainable parameters) and $E - E_{lora}$ rounds of the full-rank part.  Therefore, the overall computational overhead is actually reduced compared to FedAvg.

Reviewer comment (Evaluation):
This paper does not include the evaluation results of each client which may lead to sacrificing the performance of some clients for improvement.

Author response:
Thank you for your valuable insights. We have performed additional experiments to demonstrate that our proposed FedLoRA, which incorporates personalized low-rank parameter matrices, does not result in any degradation of individual client performance.

The experimental details are as follows. We test FedLoRA on CIFAR-100 and Tiny-ImageNet datasets in the Dirichlet non-IID scenario with $\alpha=0.1$. Specifically, we plot the mean / minimum / maximum of accuracy improvements of FedLoRA for each client, compared with FedAvg and Individual Training (i.e., each client trains the model locally without collaboration).

The results are summarized in the following table.

| CIFAR-100 | Mean Value of Improvement | Minimum Value of Improvement | Maximum Value of Improvement |
|-| - | - | - |
| Compared with FedAvg | 27.97 | 15.00| 45.00|
| Compared with Individual Training | 16.07| 3.00| 31.00|

| Tiny Imagenet | Mean Value of Improvement | Minimum Value of Improvement | Maximum Value of Improvement |
| - | - | - | - |
| Compared with FedAvg | 25.29 | 18.50 | 35.50 |
| Compared with Individual Training | 17.86 | 5.50 | 25.00 |

For more detailed statistics for each client, please refer to Appendix D in our revised paper.

Notice that the minimum improvement value across all clients when employing FedLoRA is greater than 0, affirming that the use of FedLoRA does not lead to any deterioration in individual client performance.

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Reviewer comment (Experiments):
The convergence analysis is missing in this paper, and there are no experiments and analysis about the proposed new training stage to support the intuition in Fig 3.

Author response:
Thanks for the comment. We conduct additional experiments to further validate the intuition in Fig 3. Here is the detailed experiment information.

The motivation for proposing alternating training is to reduce the impact of data heterogeneity on the shared part. That is, **reduces the deviation of shared parameters to the local minimum point of the client**. Thus, we expect that using alternating training should reduce the difference in shared parameters among clients during local training. We conduct experiments to analyze the change in model distance before and after using alternating training. The results are shown in the following table: 

|                             | CIFAR-100 | Tiny Imagenet |
| --------------------------- | --------- | ------------- |
| Simultaneous Training       | 19.05     | 12.25         |
| Alternating Training (ours) | 11.74     | 6.59          |

For more detailed statistics including the trend of model difference during the training process, please refer to Appendix C in our revised paper. It is evident that the utilization of the alternating training strategy results in a significantly smaller disparity in the shared components of the model. This observation aligns with our intuition, as illustrated in Fig. (3).

We intend to incorporate these experiments into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Reviewer comment (Theory):
The proposed alternative training seems heuristic and lacks theoretical justification.

Author response:
While we reserve the theoretical justification for future work, we have taken your comment into consideration and conducted empirical experiments to validate the effectiveness of the proposed alternating training strategy.

As a quick reminder, the primary objective of alternating training is to mitigate the impact of data heterogeneity on the shared parameters, essentially **reducing the deviation of shared parameters to the local minimum point of the client**. Consequently, employing alternating training should lead to a reduction in the discrepancies among shared parameters across clients during their local training phases. To validate the effectiveness of alternating training in achieving this goal, we carried out additional experiments to compare the disparities in shared parameters among clients when using alternating training as opposed to not using it. The results are presented in the table below. It's evident from the data that, across both datasets, the utilization of alternating training significantly diminishes the differences in the shared parameters among clients.

|| CIFAR-100 | Tiny Imagenet |
| - | - | - |
| Simultaneous Training| 19.05| 12.25|
| Alternating Training (ours) |11.74| 6.59|

For a more comprehensive set of statistics, including the trend of model differences throughout the training process, please refer to Appendix C in our revised paper.

Reviewer comment (Experiments):
Experiments are conducted on three small image datasets. More large datasets, especially in different modalities, should be used.

Author response:
In response to your comment, we have conducted additional experiments on both a larger image dataset and a natural language processing (NLP) dataset.

- For the larger image dataset, we selected a subset from ImageNet, consisting of 400 classes with a total of 80,000 samples. We utilized the ResNet-10 model architecture, with each client having 2,000 training samples generated following the Dirichlet distribution with $\alpha=0.1$.
- For the NLP dataset, we opted for AG_NEWS, a text classification dataset with 120,000 samples. We employed the Transformer model architecture, with each client having 3,000 training samples generated following the Dirichlet distribution with $\alpha=1.0$. Additionally, for the Transformer model, we applied model decomposition to the weights in the self-attention modules and fully connected weights in the classifier module.

| | FedAvg | FedPer | FedRoD | FedLoRA   |
| -| - | - | - | - |
| AG_NEWS | 89.36  | 90.76  | 91.38  | **91.79** |
| ImageNet (subset) | 18.55  | 29.37  | 32.45  | **35.67** |

The table above displays the test accuracy results for these two datasets. It's evident that FedLoRA consistently outperforms other state-of-the-art methods on both datasets. We intend to incorporate these experiments on the new datasets into a revised version of the paper, and we are committed to making all the source code publicly available in the future.

Reviewer comment (Experiments):
Why not try larger datasets, and datasets beyond image?

Author response:
In response to your comment, we have conducted additional experiments on both a larger image dataset and a natural language processing (NLP) dataset.

- For the larger image dataset, we selected a subset from ImageNet, consisting of 400 classes with a total of 80,000 samples. We utilized the ResNet-10 model architecture, with each client having 2,000 training samples generated following the Dirichlet distribution with $\alpha=0.1$.
- For the NLP dataset, we opted for AG_NEWS, a text classification dataset with 120,000 samples. We employed the Transformer model architecture, with each client having 3,000 training samples generated following the Dirichlet distribution with $\alpha=1.0$. Additionally, for the Transformer model, we applied model decomposition to the weights in the self-attention modules and fully connected weights in the classifier module.

| | FedAvg | FedPer | FedRoD | FedLoRA   |
| -| - | - | - | - |
| AG_NEWS | 89.36  | 90.76  | 91.38  | **91.79** |
| ImageNet (subset) | 18.55  | 29.37  | 32.45  | **35.67** |

The table above displays the test accuracy results for these two datasets. It's evident that FedLoRA consistently outperforms other state-of-the-art methods on both datasets. We intend to incorporate these experiments on the new datasets into a revised version of the paper, and we are committed to making all the source code publicly available in the future.