Paper: Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning (ICLR-2024)

Reviewer comment (Experiments):
There are still some baselines that are not compared in the paper, for example, the methods in paper [1],[2],[3].  
[1] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.  
[2] Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference on Machine Learning, pages 12073–12086. PMLR, 2021.  
[3] Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lv. Target: Federated class-continual learning via exemplar-free distillation, 2023.

Author response:
**Weakness 1**

*Comparison with Other Baselines:* LwF [1] is a centralized continual learning algorithm proposed in 2017. While determining baselines from centralized CL, our approach involved a thorough examination of both the SOTA methodologies and well-established earlier works. Notably, FedLwF is compared with FedCIL in [A] and shown to perform much worse. Given the inclusion of FedCIL in our baseline, and show that FOT has superior performance in comparison, we have chosen to exclude FedLwF from our analysis.

While categorized under Continual Federated Learning, it's crucial to recognize that FedWeIT [2] addresses a fundamentally distinct problem. It is primarily interested in learning individual local tasks, not learning a global model but utilizing the information of other clients. It employs federated learning not to train a global model but to enhance the continual learning process for individual local tasks by leveraging information from other clients.  Due to the distinct objective of this paper in comparison to ours, we haven’t included it in our baseline. 

We mention TARGET [3] in our related work section (Section 2 and Appendix B). Unfortunately, due to its publication date being in close proximity to the submission deadline, we were unable to incorporate it into our baselines.
Following the reviewer’s advice, we did an initial comparison of TARGET with FOT on mini-Imagenet dataset. At the end of each task, the generator in TARGET is trained on the server side in a data-free way.  Then using that generative model, the server generates synthetic samples (750 samples per task in our experiments) and sends them to the clients. Clients perform knowledge distillation between the current model and the stored old-task model with the synthetic samples to prevent forgetting.

|  | IID |  | nonIID |  |
|---|---|---|---|---|
|  | ACC(%) | FGT(%) | ACC(%) | FGT(%) |
| FedAvg | 50.43 | 33.08 | 41.00 | 32.92 |
| TARGET | 58.45 | 25.12 | 53.67 | 26.31 |
|FedCIL | 55.98 | 28.90 | 51.12 | 25.66 |
| FOT | 69.07 | 0.19 | 62.06 | 0.17 |

Similar to FedCIL, TARGET achieves a better forgetting and accuracy compared to vanilla FedAvg, but is outperformed by our FOT approach. We would like to highlight that these only represent initial comparisons with TARGET and we plan to extend to a more thorough comparison on other datasets in the revised manuscript in Appendix G.7.  Given your suggestions, we have also extended our related work discussion in the manuscript to highlight [1] and [2].

Reviewer comment (Evaluation):
In FOT, the global update was generated by projecting the aggregated updates onto the orthogonal subspace for each layer in the model. Intuitively, this will make the convergence slower, which is directly related to potential extra communication costs. The authors did not provide a discussion about this.

Author response:
**Weakness 2 & Question 2**

In the pursuit of fairness, we empirically identified the optimal number of rounds for the base case, namely FedAvg, and employed this determined round number consistently across all other methods.

*Convergence of FOT:* Thank you for asking this question. FOT does not exhibit slower convergence; However, it is crucial to highlight that in subsequent tasks, convergence is directed toward the optimal point within a restricted convex set through a subspace projection. While this projection doesn't slow down the convergence speed, it does lead to a slight decrease in that task’s accuracy due to regularization by the target convex set. 
In the updated version of the manuscript (Appendix G.5), we discuss the convergence behaviors of FOT and share the accuracy-round plots. As we can see from these plots, the convergence behavior of FOT is similar to FedAVG. While FOT may exhibit lower accuracy than FedAvg in a specific task, FOT maintains this accuracy in later tasks. On the other hand, the accuracy of other methods tends to decline below FOT's performance as new tasks are introduced. That’s why, at the end of training a sequence of different tasks, the average accuracy of FOT is higher and the forgetting is less than other methods.

Reviewer comment (Experiments):
On page 6, Theorem 1, the author mentioned that "for sufficiently large n, the principal column space of Y recovers the low-rank column space of A up to rank k with a negligible error.". In my understanding, n corresponds to the number of data in every client. Then what is the number of data in your experiments? For the 100 clients' experiments, is there enough data? If not, how do you explain the effectiveness of the FOT?

Author response:
**Question 1**

In the server, after aggregating all local $A_{t,i}^\ell$, the server gets global  $A_{t}^\ell$ (see equation 10). This new matrix is equal to the multiplication of global input matrix $X^\ell$ and random Gaussian matrix $G$. Therefore, dimension n from the theorem corresponds to the second dimension of $X^\ell$ which is equal to the total number of samples distributed among clients. The value of n remains constant for the task whether we simulate a task with 100 clients or 50 clients. For the specific quantity of data for each task and dataset, refer to Table 8 in the manuscript, specifically examining the ' # training/task' rows.

Reviewer comment (Experiments):
Authors said that they used the same round number for different methods. How did you decide the round number? Did other methods converge earlier than the round number?

Author response:
**Weakness 2 & Question 2**

In the pursuit of fairness, we empirically identified the optimal number of rounds for the base case, namely FedAvg, and employed this determined round number consistently across all other methods.

*Convergence of FOT:* Thank you for asking this question. FOT does not exhibit slower convergence; However, it is crucial to highlight that in subsequent tasks, convergence is directed toward the optimal point within a restricted convex set through a subspace projection. While this projection doesn't slow down the convergence speed, it does lead to a slight decrease in that task’s accuracy due to regularization by the target convex set. 
In the updated version of the manuscript (Appendix G.5), we discuss the convergence behaviors of FOT and share the accuracy-round plots. As we can see from these plots, the convergence behavior of FOT is similar to FedAVG. While FOT may exhibit lower accuracy than FedAvg in a specific task, FOT maintains this accuracy in later tasks. On the other hand, the accuracy of other methods tends to decline below FOT's performance as new tasks are introduced. That’s why, at the end of training a sequence of different tasks, the average accuracy of FOT is higher and the forgetting is less than other methods.

Reviewer comment (Experiments):
According to the description of the FOT, to me, it seems like that FOT would also work well if we only apply this to several layers in the model. Have the authors tried this?

Author response:
**Question 3**

Thank you for asking this question. We could select some layers and apply FOT only on those layers. If we exclude some of the layers from FOT, we should choose whether they will be frozen in the later tasks or continue to be trained. If we freeze these layers, FOT still works well as we show in section 5.2.2. In that experiment, we freeze the first layer and exclude it from FOT and the result is still as good as the original FOT. However, if we choose to continue training the excluded layers in the later tasks, FOT might not work well. The intuition behind this is that, when we fine-tune the excluded layers, these layers’ activations in response to inputs from earlier tasks could change drastically and this change can propagate to the activation response in later layers in the model. 

Following the reviewer’s advice, we use FOT on only a subset of layers. In the first experiment, we apply FOT on the last half of the layers and fine-tune the early half of the layers. In the second experiment, we still exclude the first half layers from FOT but we freeze them after the first task.  We also do the same experiment where we apply FOT only on the first half of the layers and fine-tune/freeze the second half. The experiments are done in mini-imagenet and the results are shared below. (A)-(B) denotes that method (A) is applied to the first half of the model and method (B) is applied to the second half.

|  | FOT-FOT  | FOT-frozen  | FOT-finetune  | frozen-FOT  | finetune-FOT |
|---|---|---|---|---|---|
| ACC(%) | 69.07 | 67.48 | 49.79 | 67.58 | 47.31 |
| FGT(%) |  0.19 |  0.14 | 28.15 |  0.12 | 33.41 |

When we do not freeze the excluded layers, the average accuracy is low which is in-line with our hypothesis. If we freeze these layers, we still get comparable results but the average accuracy is lower. This is because when we freeze some layers, we restrict the learning of representations for subsequent tasks which degrades the performance when optimizing for these later tasks.

We included the additional experiments described above in Appendix G.6 of the revised manuscript.

Reviewer comment (Experiments):
Communication overhead represents a significant weakness in the methodology presented in the paper. However, the paper's analysis is somewhat superficial, lacking a comparison with baseline experiments.

Author response:
**Communication Overhead:**
We would like to emphasize that the additional communication cost incurred by FOT is negligible. First, it is important to note that this additional cost is associated only with the GPSE step, and this step consists of just one round, conducted once at the end of each task. In this round, the server transmits $\{ O_\ell \}^L_{\ell=1}$ to the clients, and the clients send only $\{ A_\ell \}^L_{\ell=1}$ to the server. Importantly, the communication cost of GPSE remains independent of the amount of data and is scaled only with the model size. Furthermore, while the communication complexity aligns with the model size, the actual size is observed to be smaller than the model itself as shown in Table 5.

As a numerical example from our experiments, if we train Alex-net on one Cifar100 task for 1000 federated learning rounds by communicating the model, then the FOT cost and FedAVG incur exactly the same cost (2840 MB in total) over the 1000 model training rounds because FOT only communicates the model between clients and the server (as in FedAvg).  At the end of 1000 rounds, GPSE uses a communication load less than or equal to only a single round which is much smaller than rounds used for model aggregation because GPSE communicates $\{ O_\ell \}^L_{\ell=1}$  and $\{ A_\ell \}^L_{\ell=1}$ and their total size (0.29 mb - See table 5) is less than model size (1.42 mb - See table 5 ). As a result, the additional communication cost is less than 0.1% of the total training cost for a single task. That’s why we claim the additional communication overhead is minimal.

In contrast, recent federated continual learning work -  FedCIL [1] has a huge overhead in terms of communication. FedCIL communicates a generative model with the original classifier in each training round (compared to FOT which only does additional communication only once per task). Given the generative model is bigger (1.60 mb) than the classifier, it increases the communication cost by more than 100%. 

We provide the total communication cost (in MB) of one client at the end of each Cifar100 task in the below table. We give a more extensive description of the additional communication cost in Appendix G.4 in the updated version of the paper.

| Task | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| FedAvg | 2840.0 | 5680.0 | 8520.0 | 11360.0 | 14200.0 | 17040.0 | 19880.0 | 22720.0 | 25560.0 | 28400.0 |
| FOT | 2840.0 | 5680.3 | 8520.6 | 11360.9 | 14201.2 | 17041.4 | 19881.7 | 22722.0 | 25562.3 | 28402.6 |
| FedCIL | 2840.0 | 8880.0 | 14920.0 | 20960.0 | 27000.0 | 33040.0 | 39080.0 | 45120.0 | 51160.0 | 57200.0 |

Reviewer comment (Theory):
The argument regarding the minimized communication overhead is not sufficiently elaborated upon.

Author response:
**Communication Overhead:**
We would like to emphasize that the additional communication cost incurred by FOT is negligible. First, it is important to note that this additional cost is associated only with the GPSE step, and this step consists of just one round, conducted once at the end of each task. In this round, the server transmits $\{ O_\ell \}^L_{\ell=1}$ to the clients, and the clients send only $\{ A_\ell \}^L_{\ell=1}$ to the server. Importantly, the communication cost of GPSE remains independent of the amount of data and is scaled only with the model size. Furthermore, while the communication complexity aligns with the model size, the actual size is observed to be smaller than the model itself as shown in Table 5.

As a numerical example from our experiments, if we train Alex-net on one Cifar100 task for 1000 federated learning rounds by communicating the model, then the FOT cost and FedAVG incur exactly the same cost (2840 MB in total) over the 1000 model training rounds because FOT only communicates the model between clients and the server (as in FedAvg).  At the end of 1000 rounds, GPSE uses a communication load less than or equal to only a single round which is much smaller than rounds used for model aggregation because GPSE communicates $\{ O_\ell \}^L_{\ell=1}$  and $\{ A_\ell \}^L_{\ell=1}$ and their total size (0.29 mb - See table 5) is less than model size (1.42 mb - See table 5 ). As a result, the additional communication cost is less than 0.1% of the total training cost for a single task. That’s why we claim the additional communication overhead is minimal.

In contrast, recent federated continual learning work -  FedCIL [1] has a huge overhead in terms of communication. FedCIL communicates a generative model with the original classifier in each training round (compared to FOT which only does additional communication only once per task). Given the generative model is bigger (1.60 mb) than the classifier, it increases the communication cost by more than 100%. 

We provide the total communication cost (in MB) of one client at the end of each Cifar100 task in the below table. We give a more extensive description of the additional communication cost in Appendix G.4 in the updated version of the paper.

| Task | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| FedAvg | 2840.0 | 5680.0 | 8520.0 | 11360.0 | 14200.0 | 17040.0 | 19880.0 | 22720.0 | 25560.0 | 28400.0 |
| FOT | 2840.0 | 5680.3 | 8520.6 | 11360.9 | 14201.2 | 17041.4 | 19881.7 | 22722.0 | 25562.3 | 28402.6 |
| FedCIL | 2840.0 | 8880.0 | 14920.0 | 20960.0 | 27000.0 | 33040.0 | 39080.0 | 45120.0 | 51160.0 | 57200.0 |

Reviewer comment (Theory):
Remark 1 highlights that the convergence analysis in the paper is relatively straightforward and lacks comprehensive theoretical underpinnings.

Author response:
**Convergence Analysis:** The paper presents an approach for enabling continual learning in a federated learning setting. We rely on an orthogonalization approach, however, we take careful consideration in allowing this without violating the privacy requirements and the convergence guarantees in FL. Our proposed approach allows us to achieve protection against forgetting using projection into a convex set which allows us to rely on convergence guarantees in the literature for ProximalSGD. We would argue that this simplicity is a strength of the proposed method as it allows us to guarantee theoretical convergence with privacy guarantees and little communication overhead in addition to observed empirical benefits. 

We would also like to point out that the central theoretical challenge in the considered problem and our proposed approach is enabling the learning of principal subspaces and the application of orthogonal projection in a distributed privacy-preserving setting.

Reviewer comment (Theory):
I have a question of how to understand the correctness of the orthogonal process in this paper (intuitively)?

Author response:
**Intuition of Orthogonal Process:** To understand why the orthogonal process works intuitively, let's say we train a model on a given task t. When we move to the next task t+1, the data of task t will disappear. That’s why, while training task t+1, the new model updates will disrupt what the model learns from task t. With an orthogonal process, we restrict the update of the model such that the disruption to the layer outputs (activations) for task t will be very small. 

We examine the model layer by layer and we will apply the orthogonal process to each layer. Let $X_t$ be the input of a layer at task t and $W$ be the optimal weight of that layer at the end of task t. When we move to the next task, $W$ will be updated as we do training. So, the model’s new response to previous task input is $(W + \Delta W)X_t$. Our goal is to make the change in the layer response to task t input as minimal as possible (ideally zero). If the response of the new model is only minimally changed with respect to the old model at all layers, then the final model output remains the same (or very close), which means no (minimal) forgetting. 
An easy way to visualize this is if we ideally make $\Delta W X_t = 0$, then $(W + \Delta W)X_t = W X_t$, meaning training of t+1 didn’t affect task t. 
Therefore, we aim to make  $\Delta W X_t = 0$ ideally or more practically, we want to make  $|| \Delta W X_t|| \approx  0$. 
To achieve it, we should make the column space of $X_t$ orthogonal to the row space of $\Delta W$, which is the crux of the orthogonal process presented in the paper. 

To realize this objective in a federated learning setup, we need 2 things:  

1- Extracting  principal column subspace of $X_t$ correctly

2-Ensuring updates are orthogonal to that subspace.
 
GPSE achieves 1 in a privacy-preserving, communication, and computation-friendly way and FedProject achieves 2 in the server side.