Paper: Improving LoRA in Privacy-preserving Federated Learning (ICLR-2024)

Reviewer comment (Experiments):
Unclear how the approach performs under other challenges like adversarial attacks, concept drift, and personalization.

Author response:
> Unclear how the approach performs under adversarial attacks, concept drift, and personalization.

The main objective of our paper is on the challenges when we consider the task of fine-tuning of LLMs, and impact of . Although these questions are important in the context of Fed Fine tuning LLMs. They are beyond the scope of this paper.

Reviewer comment (Experiments):
The paper only evaluates NLP tasks with text data. Unclear if the benefits of FFA-LoRA generalize to other data types like image, speech, etc.

Author response:
> The paper only evaluates NLP tasks with text data. Unclear if the benefits of FFA-LoRA generalize to other data types like image, speech, etc.

It would be interesting to see the performance of such fine-tuning in the context of computer vision and speech. We running experiments on these tasks and will update the results here once they are ready. We note that fine-tuning LLMs with DP is computationally intensive, and we need more time to finish the experiments.

We here provide an additional dataset on the computer vision task.
We use the pre-trained vision transformer (https://huggingface.co/google/vit-base-patch16-224-in21k), and consider the task of fine-tuning on the Food-101 dataset (as shown on https://huggingface.co/datasets/food101) for image classification.

For context, we provide performance reported on huggingface as baseline. A centralized, fine-tuned model has an accuracy of 0.8539.

We first report the results in our centralized experimental setting in the table below.In this case there is no significant performance discrepancy between the two methods, implying that FFA-LoRA and vanilla has similar performance without consideration of DP and FL. This also aligns with Theorem 1 in Author's response.

In terms of the federated case, we first report the iid setting. In this setting, under similar hyper-parameters, we provide the learning curve of both FFA-LoRA and LoRA (https://imgur.com/5xnXWSu). as well as a last k-iterated (k=50) averaged accuracy. 

It can be seen that compared to LoRA, FFA-LoRA has both (a) better convergence and (b) less fluctuations in training. The findings align with our findings in language-related tasks, showing that the properties of LoRA being discussed in our paper are not limited to language tasks only.

The non-iid setting as well as the DP setting are more time-consuming, and we are still running more experiments. Due to the time limit, we will add them to the final version of our paper if the paper is accepted.


| Method          | Accuracy |
|-----------------|----------|
| Baseline        | 0.8539   |
| Centralized LoRA | 0.8618   |
| Centralized FFA-LoRA | 0.8583   |
| FL iid LoRA | 0.8133   |
| FL iid FFA-LoRA | 0.8210   |

Reviewer comment (Theory):
The theoretical analysis and intuitions provided are informal. No formal convergence or privacy proofs given.

Author response:
> The theoretical analysis and intuitions provided are informal. No formal convergence or privacy proofs given.

Regarding the **scaling factor** $\alpha$ for FFA-LoRA and LoRA, we have updated our draft to include more formal statements, and provide an additional theorem to formally show that FFA-LoRA is in fact equivalent to LoRA with $\alpha \rightarrow \infty$. 
>> **Theorem 1 (Scaling Factor):** For local updates with the same initial condition on $\mathbf{W}$, vanilla LoRA update with scaling factor $\alpha\_{LoRA}$ produces trajectory $ \\{ W\_{\alpha\_{LoRA}}^k \\}\_{k\in [K]}$, and FFA-LoRA with scaling $\alpha\_{FFA}$ produces trajectory $\\{W\_{\alpha\_{FFA}}^k\\}\_{k\in [K]}$. Then we have
>> $$\lim_{\alpha_{LoRA} \rightarrow \infty} W_{\alpha_{LoRA}}^k = W_{\alpha_{FFA}}^k, \text{ for all } k, \alpha_{FFA}.$$

Regarding the **convergence study** of our proposed method. We first provide the following statements for some additional context on the theoretical aspects behind PEFT on LLMS. 
>> **Theorem 2 (Smoothness conditions):** Assume that the loss function given weight and dataset is denoted $F(\mathbf{W}, D)$. For a low-rank decomposition on model parameter $\mathbf{W}$ such that $\mathbf{W}(\mathbf{A}, \mathbf{B}) = \mathbf{W}\_0 + \mathbf{B} \mathbf{A}$ satisfying Equation (1). We have the following properties.
>> 1. If $\mathbf{B} $ is trainable, $\mathbf{A}$ is fixed with $||\mathbf{A}|| \leq C$ and $F(\mathbf{W}, D)$ is Lipschitz smooth with respect to $\mathbf{W}$ with factor $L$. The loss function $F(\mathbf{W}(\mathbf{A}, \mathbf{B}))$ is Lipschitz smooth with respect to $\mathbf{B}$ with factor $LC^2$.
>> 2. If both $\mathbf{A}$ and $\mathbf{B}$ are trainable and $F(\mathbf{W}, D)$ is Lipschitz smooth with respect to $\mathbf{W}$ with factor $L$, the loss function $F(\mathbf{W}(\mathbf{A}, \mathbf{B}))$ has no Lipschitz smoothness guarantees with respect to $\[\mathbf{A}, \mathbf{B}\]$.
>>
>> All the smoothness notions are defined with respect to the matrix Frobenius norm, which is denoted as $||\cdot||$.

These results show that if the objective function loss defined on full fine-tuning with $\mathbf{W}$ satisfies technical conditions such as Lipschitz smoothness, the same assumptions would also hold for FFA-LoRA, but not for Vanilla LoRA.

Suppose that the objective function $F$ satisfies the conditions presented in existing works to ensure convergence of full federated fine-tuning [1] or DP tuning [2] on $\mathbf{W}$, then we can get similar **convergence results for FFA-LoRA using the theorem above**.

Regarding **differential privacy**, we formally state our corollary for privacy guarantee. Our analysis below is based on Theorem 1 of [3] and the parallel composition and resistance to post-processing of DP.
>>**Corollary 3 (Privacy Guarantee):** Given Theorem 1 with moments accountant in [3],  the parallel composition and resistance to post-processing of DP, the mechanism updating FFS-LoRA with locally ran DP-SGD and FedAvg can satisfy $(\epsilon, \delta)$-DP given $\forall i, q=\frac{|B_i|}{|N_i|}$, the number of total local updates $T$ of each client and  $\sigma = O(\frac{q\sqrt{T\log(1/\delta)}}{\epsilon})$. 

(The exact $\sigma$ is computed by the Pytorch's Opacus package [4] numerically given $q, T, \epsilon, \delta$).

We refer to our "**Response to All**" for detailed proof.

---
[1] *Lian, Xiangru, Yijun Huang, Yuncheng Li, and Ji Liu.* "Asynchronous parallel stochastic gradient for nonconvex optimization." *Advances in neural information processing systems 28 (2015).*

[2] *Wang, Di, Changyou Chen, and Jinhui Xu.* "Differentially private empirical risk minimization with non-convex loss functions." *In International Conference on Machine Learning, 2019.*

[3] *Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and
Li Zhang.* Deep learning with differential privacy. *2016 ACM SIGSAC
conference on computer and communications security*

[4] <https://github.com/pytorch/opacus>

Reviewer comment (Evaluation):
The benefit of FFA-LoRA on differential privacy (DP) is not very well backed by empirical evaluation. The performance gap between the vanilla LoRA and the proposed FFA-LoRA remains the same across various privacy budgets $\epsilon$, including $\epsilon = 0$. Such an empirical result suggests that the impact of DP noise is the same on both the vanilla LoRA and the proposed FFA-LoRA.

Author response:
We note that the exact performance of the algorithms are In Table 1, although there are counter-examples, the performance gap between FFA and vanilla is still evident, with the largest gap occurring $\epsilon = 1$ when taking average over the 4 tasks. Therefore we can say that FFA is more suitable for higher noise. 
The individual performance of one task, $SST-2$ in this case, depends on many factors such as the difficulty of the task. 

We further point out that our claim on noise amplification is more apparent in Table 4. Where for rank as low as 2 or 4, Lora can not converge to any meaningful results when $\epsilon = 1$.

Additionally, when noise is small (no significant noise amplifying), the impact of noise is not as big, as shown in Figure 1 of our paper. The simulation in Figure 1 also aligns with our observation in Table 1 and 4.

Additionally, existing literature [1] (Figure 4) has shown that the relationship between performance gap and $\epsilon$ is not always significant, and varies between tasks. This observed behavior could be due to robustness of DPSGD-style algorithms. Currently, the dynamics of DPSGD optimization in deep neural networks is still not fully understood.

Reviewer comment (Theory):
I do not see why the proposed FFA-LoRA is free from tuning the hyper-parameter $\alpha$. In Section 4, the authors claim that "FFA-LoRA does not rely on $\alpha$, and is equivalent to LoRA with $\alpha = \infty$". Such a claim, in fact, suggests that the $\alpha$ is fixed in FFA-LoRA. Then, in Theorem 1, the theoretical result suggests that tuning $\alpha$ is equivalent to tuning the learning rate $\eta$. I'm not able to fully follow the discussion here.

Author response:
We would like to first reiterate that for the proposed FFA-LoRA algorithm, as confirmed by Theorem 1 in our paper, $\alpha_{FFA}$ can be seen as fixed. However, this does not make FFA-LoRA equivalent to vanilla LoRA with some given $\alpha_{LoRA}$. As we have stated, FFA-LoRA with any $\alpha$ in fact has the same dynamic as LoRA with $\alpha_{LoRA} \rightarrow \infty$, we provide the details below:

In order to reduce confusion and provide additional theoretical robustness to our paper, we provide an additional theorem as a more formal statement on the relationship between FFA-LoRA and vanilla LoRA. We refer to our "**Response to All**" for a more detailed explanation.

Reviewer comment (Theory):
Is the FFA-LoRA approach more sensitive to random initialization? Suppose a bad initialization sets $\mathbf{A} = \mathbf{0}$; is the model still trainable?

Author response:
This is a really good question. In our experiments, all the $A_0$ matrices were randomly initialized under random Kaiming distribution ( a default initialization method is deep learning and is consistent with previous works in LoRA) and according to Theorem 1, the scaling on the $A_0$ is not important for FFA-LoRA. 
And we briefly discuss the impact of other types of initialization below.

We know that for a zero-initialized A matrix, neither LoRA nor FFA-LoRA are able to train any meaningful results. However, suppose that we have A to be full rank (which is also satisfied for any random initialization in general), there are a number of different initializations that we could utilize, such as orthogonal random initialization and using the top $r$ singular vectors of $W_0$ as matrix A. 

We provide some initial results here (without DP), it can be seen from the plot that matrix $A$ with orthogonal initialization seems to perform slightly better than the existing approach. 
However, the performance gap is not significant enough for a definitive answer.

| Method   | QNLI mean | QNLI variance |
|---|---|----|
| Kaiming Init.    | 91.84\%   | 0.38\% |
| Orthogonal Init. | 92.16\%   | 0.83\%  |
| SVD Init.    | 91.50\%   | 0.59\%   |

However, although this is a great question with many interesting potential directions, we mainly consider the result of freezing A after initialization, and the choices of initialization is beyond our initial scope of this work. Although all these methods use SGD as the back-bone optimization algorithm, the key ideas and procedures for training are very different from DP. Thus, each of these challenges may deserve an independent project to be investigated. We will mentions these problems as future directions in our revision.

Reviewer comment (Evaluation):
What's the variance in the experiment?

Author response:
In many of the experiments (especially for RoBERTa), we find that the variance is typically low, and the performance is consistent, therefore we mainly consider the performance. We conducted additional experiments (20 reruns with different seeds) the QNLI experiment, under the same setting as Table 1, and provide the mean and variance of the results here. We note that some rare cases where the algorithm failed to converge has been removed from the statistics.

| Privacy Budget + Method   | QNLI mean | QNLI variance |
|----|---|---|
| Non Private + LoRA   | 91.56\%   | 0.43\%  |
| Non Private + FFA-LoRA    | 91.84\%   | 0.38\%  |
| $\epsilon = 6$ + LoRA   | 86.55\%   | 1.02\% |
| $\epsilon = 6$ + FFA-LoRA | 87.42\%   | 0.80\%  |
| $\epsilon = 3$ + LoRA     | 85.66\%   | 1.20\%  |
| $\epsilon = 3$+ FFA-LoRA  | 86.18\%   | 1.01\%  |
| $\epsilon = 1$ + LoRA   | 80.86\%   | 0.28\%     |
| $\epsilon = 1$+ FFA-LoRA  | 83.63\%   | 1.32\%      |

Reviewer comment (Theory):
The motivation is straightforward and intuitive, without theoretical insights.

Author response:
We appreciate the reviewer's comment on our motivation being straightforward and intuitive. We provide some additional theoretical results regarding the relationship between FFA-LoRA and LoRA, a theorem regarding the convergence properties of FFA-LoRA, and lastly a guarantee on differential privacy. All of which will be added to our revised draft.
We refer to our "**Response to All**" for a more detailed discussion.

Reviewer comment (Evaluation):
Why rank 16 for MNLI is worse than rank 8 in Table 2?

Author response:
We report all our experiments truthfully. It has been shown that even for vanilla LoRA, there is an optimal rank for a specific fine-tuning task, and the performance of LoRA could decrease if rank is too large, the exact optimal rank is task-dependent [1]. This observation can be potentially explained by a similar reason.

We thank the reviewer for the careful observation, and will add statements to address this in the revised version of the paper.

Reviewer comment (Experiments):
Another intuitive variant is to alternative optimize the two LoRA weights. How would this perform compare with the proposed method?

Author response:
This is a great question. We have conducted experiments regarding alternate updates on the weights. The results show slower convergence and no significant performance gains compared to vanilla LoRA. 
Empirically, this means that alternating update is hard to tune: with a smaller learning rate similar to LoRA, it converges slower; with a larger learning rate similar to FFA-LoRA, it is not robust and often unable to converge. For the case of non-iid agents, this algorithm is unable to converge under 5 different random seeds and multiple learning rates $\eta \in \\{0.05, 0.1, 0.2\\}$. (For this experiment, the best LR for LoRA and FFA-LoRA is $\eta=0.05$ and $\eta = 0.5$, respectively.)
This can be confirmed by both accuracy scores as well as observations of the learning curve during training.
Additionally, with the consideration of DP, the number of gradient calculations is doubled for alternative update methods, which could result in a higher noise added to ensure privacy. In this case, the drawbacks overshadow the benefits. We will provide a brief discussion on this potential solution in our revision. 

We also include one result from the alternate update case here demonstrating its performance in the Table below.

Reviewer comment (Evaluation):
Why non-iid performance is similar to iid performance in Table 3?

Author response:
As we stated in Section 5.1, for the heterogeneous setting, we split data based on their labels. Due to the heterogeneity presented to these methods is not strong enough, there is not a strong separation between iid and non-iid experiments for some tasks. Therefore, a performance gap exists between FFA-LoRA and vanilla LoRA, but it is not very significant. To demonstrate the impact of heterogeneity, we included an additional experiment with even stronger heterogeneity.

In the previous MNLI results we ran in the original paper, the non-iid case was formulated using a data label split of $[0.6, 0.2, 0.2], [0.2, 0.6, 0.2], [0.2, 0.2, 0.6]$, which we denote as mild non-iid here. We consider a stronger hetergeneity here, with data label split ratio of $[0.9, 0.05, 0.05], [0.05, 0.9, 0.05], [0.05, 0.05, 0.9]$. The performance degradation is clearly shown, with FFA-LoRA significantly better than LoRA.


| Privacy Budget + Method  | MNLI matched | MNLI mismatched |
|----------------------------|--------------|-----------------|
| I.I.D. + LoRA              | 86.90\%      | 87.15\%         |
| I.I.D. + FFA-LoRA          | 87.13\%      | 87.21\%         |
| I.I.D. + Alt. update       | 86.15\%      | 86.98\%         |
| Mild Het. + LoRA           | 87.01\%      | 87.33\%         |
| Mild Het. + FFA-LoRA       | 87.04\%      | 87.36\%         |
| Severe Het. +  LoRA        | 83.95\%      | 84.51\%         |
| Severe Het. +  FFA-LoRA    | 86.28\%      | 86.71\%         |
| Severe Het. +  Alt. update | 35.45\%      | 35.22\%         |