Paper: Variational Federated Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
How do you compare with this work? [1] Guo, H., Greengard, P., Wang, H., Gelman, A., Kim, Y., & Xing, E. P. (2023). Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach. arXiv preprint arXiv:2302.04228.

Author response:
**Comparison with "Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach" (FedEP).**

We compare FedEP with ours in terms of problem setting and method.

Problem setting: FedEP studies the classical federated learning problem, where all clients learn a task together, where all clients jointly learn a task, and the task definition is static and predefined. In our problem setting, each client learns sequentially on a private sequence of tasks, and clients learn different tasks. Our problem is more challenging, and the challenges include catastrophic forgetting due to continual learning, as well as heterogeneity across client tasks.

Method: FedEP is a centralized federated learning approach, which output is a single global model. Our approach is personalized, and it learns a personalized model for each client. The same point is that both FedEP and ours use variational inference to solve the approximate posterior.

Thanks for providing this work, and we have added it to the related work.

Reviewer comment (Theory):
My main issue is with the global posterior aggregation method (Section 3.3). Equation 10 makes sense to me (using a mixture of local approximate posterior distributions). However, Equation 11 does not seem related to Equation 10: Equation 11 looks like you have just simply multiplied together all the various local approximate (mean-field) Gaussians. IE it is a *product* of local posterior distributions. Am I misunderstanding something? If this is indeed the case, why write it through Equation 10 instead of just saying you are multiplying together the distributions?

Author response:
**In global posterior aggregation, Eq. (11) does not seem related to Eq. (10).**

The method described in Eq. (10) is a straightforward way to obtain a global posterior, however, it is intractable due to the non-i.i.d. client data. Although it is possible to approximate the solution using Monte Carlo sampling, the huge computational overhead and the need for a training dataset make it infeasible in practice.

Therefore, we introduce the conflation, which fuses multiple distributions into a single distribution, the aggregated distribution is with density proportional to the product of the densities of input distribution, and Eq. (11) describes how the parameters in the product of Gaussian distributions are calculated, i.e., the parameters of the final aggregated distribution.

We have revised this section to provide a more smooth transition and added a detailed explanation of global posterior aggregation in Appendix E.

Reviewer comment (Theory):
Although it is nice to see an attempt at theoretical analysis, I did not see what benefit this brings to the paper or method. Could the authors perhaps discuss how such analysis might be helpful for current or future analysis? The three bullet points at the end of Section 4 all seemed fairly obvious to me (but perhaps that is the point? But then what use is this section?).

Author response:
**What benefits does theoretical analysis bring to the paper ?**

As a reminder, the key contribution of our approach is to integrate historical and global knowledge into the mixture prior and migrate the knowledge to the current model, mitigating local overfitting and catastrophic forgetting.

This theoretical analysis proves that the mixture prior can reduce the upper bound of generalization error (as $\lambda_{p}$ increases, the overall error decreases), and clarifies the effectiveness of the mixture prior, which is the core contribution of our work.

Further, we also experimentally (see Table 1) verify the positive effect of the mixed prior on the model performance, which is consistent with the findings of the theoretical analysis. In addition, we also find that too strong prior constraints, while reducing model forgetting, also reduce the model's adaptability to new tasks, ultimately undermining the model's learning ability.

We have revised this section of the article to provide a clearer explanation of the utility of theoretical analysis for method.

Reviewer comment (Evaluation):
The experiments only reported one number per method. It is important to run many times (at least 3 or 5) with a mean and std deviation across runs to get an idea of if these results are significant in some way.

Author response:
**The experiments need to be run several times to validate the effectiveness of the method.**

Our comparison experiments were run 3 times. We modified tables 1,2, and added the deviation bar (due to space constraints, we only added it on "Avg.", which represents the average of the results of the rows in the table.).

Reviewer comment (Writing):
Why are the first 2 tasks called 'warm-up tasks'?

Author response:
**Why are the first 2 tasks called 'warm-up tasks' ?**

In incremental learning, when the model learns the first few tasks, the model generalization performance will be poor due to less exposure to data, and the adaptation to new tasks is slow, which is reflected in the low accuracy of the new task learning, so we call the first few tasks of incremental learning 'warm-up tasks' .

Reviewer comment (Evaluation):
It is not clear exactly what is being reported in Tables 1 and 2. Is it the average accuracy after training on task t (where t is the number in the first row), or is it the final accuracy on task t's data after training on the last task? What is the Avg over? These details should ideally go in the caption (and the main text).

Author response:
**What are the evaluation metrics in Tables 1 and 2 ?**

The metrics under 1-10 in the table are the average accuracies of all the tasks that have been learned, e.g. after learning the 3rd task, the accuracy under 3 is the average of the model's accuracies on tasks 1,2,3.

"Avg." represents the average of the results of the rows in the table, which is the same as in "Federated Class-Incremental Learning".

We revised the caption of the tables and added a detailed description of the evaluation metrics in Appendix A.

Reviewer comment (Writing):
In the second paragraph of the Introduction, the authors talk about previous methods (regularisation based and knowledge distillation based), which apparently use global knowledge / models. But then the authors say these methods do not combat local overfitting. I thought the point of use global knowledge was to combat local overfitting, so I did not understand why these other methods do not do this? I think the writing could be made clearer here.

Author response:
**Why we say previous methods do not combat local overfitting ?** 

Although existing FCL works have utilized the global model as the initialization for local training, the local model will gradually diverge from the initialized global model after starting local training, and the limited local data of the client will lead to local overfitting, which reduces the generalization ability and convergence rate. Existing methods focus on solving the catastrophic forgetting without explicitly preventing the divergence of the client local learning process, so we argue that these works do not address the overfitting problem.

Our method uses a personalized model on the client, and instead of replacing the local model with the weights of the global model, global knowledge is fused into the mixture prior, preventing local overfitting through regularization during the entire local training process, which can also be interpreted as passing global knowledge to the local model.

We have revised the introduction to provide a clearer explanation of why previous work ignored the overfitting problem.

Reviewer comment (Experiments):
The empirical setting studied in this paper is a simplified version of the FCL setting in FedWeIT where clients have different task sequences. In this case, however, the clients are assumed to have the same task sequence, which is reducible to a pure CL setting: (1) existing CL parameterization can be adopted "as is" for the common model architecture; and (2) existing FL techniques can be applied to handle the federation of data. This ignores the essence of federated CL where there can be another cause for catastrophic forgetting, which is client interference. This happens when clients have different task sequences and a naive attempt to aggregate their models (solving different tasks) can also lead to forgetting.

Author response:
**Do the empirical setting ignore the catastrophic forgetting brought by client interference (a key challenge of FCL) ?**

We agree that the catastrophic forgetting caused by client interference is a key challenge of FCL. However, we are afraid that the reviewer misunderstood our experimental setup. We would like to clarify that the client task sequences in our experiments are different across clients. In addition, we conducted experiments on the benchmark datasets according to the reviewer's suggestions, and the results validated the effectiveness of our method (see later answer for more details).

We provide a detailed explanation of how the client private task sequences in our experiments were constructed in Appendix A. First, the original dataset is split into several tasks that contain no overlapping classes. Second, the split tasks are randomly selected as the client's task and and the classes of tasks included are also randomly selected. Private task sequences from different clients may have similar (i.e., overlapping classes), unrelated, or interfering tasks. Clients utilize the knowledge of other clients in the learning process to improve the performance of the local model while preventing interference from the knowledge of other clients.

In FedWeIT, client interference is addressed through parameter decoupling and selective knowledge migration, while our approach addresses this challenge through personalization and client selection. We describe the client personalization in detail in Appendix A.2, the weights of the backbone as well as the structure of the classifiers are personalized for different clients, which allows VFCL to learn on personalized client task sequences. Client selection is described in Algorithm 1, where the server coordinates the learning of clients with similar tasks in a synchronized manner, which speeds up the convergence rate and reduces client interference.

Reviewer comment (Theory):
The proposed method has not addressed the challenge of client interference because the global posterior aggregation here assumes local posteriors are derived from the same task (albeit with different data distributions).

Author response:
**Do the empirical setting ignore the catastrophic forgetting brought by client interference (a key challenge of FCL) ?**

We agree that the catastrophic forgetting caused by client interference is a key challenge of FCL. However, we are afraid that the reviewer misunderstood our experimental setup. We would like to clarify that the client task sequences in our experiments are different across clients. In addition, we conducted experiments on the benchmark datasets according to the reviewer's suggestions, and the results validated the effectiveness of our method (see later answer for more details).

We provide a detailed explanation of how the client private task sequences in our experiments were constructed in Appendix A. First, the original dataset is split into several tasks that contain no overlapping classes. Second, the split tasks are randomly selected as the client's task and and the classes of tasks included are also randomly selected. Private task sequences from different clients may have similar (i.e., overlapping classes), unrelated, or interfering tasks. Clients utilize the knowledge of other clients in the learning process to improve the performance of the local model while preventing interference from the knowledge of other clients.

In FedWeIT, client interference is addressed through parameter decoupling and selective knowledge migration, while our approach addresses this challenge through personalization and client selection. We describe the client personalization in detail in Appendix A.2, the weights of the backbone as well as the structure of the classifiers are personalized for different clients, which allows VFCL to learn on personalized client task sequences. Client selection is described in Algorithm 1, where the server coordinates the learning of clients with similar tasks in a synchronized manner, which speeds up the convergence rate and reduces client interference.

Reviewer comment (Experiments):
The experiment is too limited in comparison to the setting that was investigated under FedWeIT. For a thorough comparison, its exact same setting should have been adopted here. Also, the current reported performance is without deviation bar and has a relatively thin margin between best and second best methods, which is not very conclusive.

Author response:
**Limited experimental setting compared with FedWeIT.**

We have restated our experimental setup in the first reply, our setup is equally challenging with FedWeIT, the client task sequences are heterogeneous, capable of responding the client interference.

To further validate the effectiveness of our method, we adopted the reviewer's suggestion to rerun the experiments on benchmark datasets Overlapped-CIFAR100 and NonIID-50 (repeated 3 times to obtain the mean and deviation bar), and added the detailed results to Appendix F.1, where the experimental results show that our method outperforms the others.


**Experimental results that lack deviation bar are not conclusive.**

All of our comparison experiments were repeated 3 times and averaged, and we revised Tables 1,2 to add deviation bar to the results. Due to space constraints, we only added deviation bar on "Avg.", and "Avg." denotes the average of the results in a single row of the table.

Reviewer comment (Theory):
It is unclear how the developed theory is specific to continual and federated learning. Its theoretical setup is entirely oblivious to the FCL setting, so its implications for this context are uncertain.

Author response:
**Relationship between theoretical analysis and method ?**

In FCL, multiple tasks are learned sequentially, and each task is learned by a separate federated learning (FL) process. Since all FL processes are isomorphic, we analyze the upper bound of the generalization error of a single FL process to analyze the overall FCL process.

The theoretical analysis guided our approach as follows: First, the upper bound of the generalization error of our method is affected by sample size, model capacity, and mixture prior (which is the core innovation of our work). Second, mixture prior is theoretically shown to reduce the upper bound on generalization error. Third, we obtain results in our experiments that are consistent with the theoretical analysis, verifying the positive effect of the mixture prior on the model performance.

Reviewer comment (Writing):
While the writing is mostly clear, several parts remain unclear. For example, background on conflation is missing, making it hard to see what is the loss function that replaces (10) in Section 3.3. The FL characterization in Eq. (1) is also strange as it suggests the optimization of all clients is decoupled (even though it is not supposed to be the case). Additionally, it is unclear whether the mixing weights in Eq. (5) are learnable.

Author response:
**Lack of background on conflation.**

The goal of global aggregation is to find a distribution which can integrate the posterior distributions of all clients. Conflation can fuse information from multiple distributions into a single distribution, and the aggregated distribution is with density proportional to the product of the densities of input distribution. Eq. (11), which is the calculation of the parameters of the global posterior distribution, replaces the optimization process in Eq. (10).

We revised this section and added a detailed description of conflation in Appendix E.

**The FL characterization in Eq. (1) is strange.**

The client models in our approach are personalized, so the final result obtained in Eq. (1) is all personalized models $\{\boldsymbol\theta_c\}_{c=1}^{C}$. In contrast, traditional centralized federated learning outputs only one global model, we thought this distinction might bother reviewers.

**Are the mixing weights in Eq. (5) learnable ?**

The mixing weights in Eq. (5) are not learnable, they can be set manually (in our paper) or calculated using existing client evaluation methods.

Reviewer comment (Theory):
The reviewer believes the paper focuses on a setting that assumes away a key challenge of federated CL and requests clarification if this is a misunderstanding.

Author response:
**Does the server need to know all the task sequences for each client in advance?**

No, the server only needs to know the task that the client is currently learning, because in practice it is not possible to determine the sequence of tasks for each client in advance. In practice, the server asks for information about the tasks of all currently available participating clients, and then selects clients that learning similar tasks for federated learning.



**Does the client learn in the order of the local task sequence?**

Yes, clients learn in the order of local continual learning. The client locally utilizes a task queue (FIFO) to manage the tasks to be learned, and the server selects clients based on the current task to be learned (i.e., the task at the head of the queue). When a client participates in training, the current task at the head of the queue is popped up, and then the head of the queue is replaced with the next task.

The experimental setup is the same for us and several other FCL works:

- "Federated Continual Learning with Weighted Inter-client Transfer"
- "Federated Class-Incremental Learning"
- "Continual Federated Learning Based on Knowledge Distillation"
- "Federated Reconnaissance: Efficient, Distributed, Class-Incremental Learning"

and all clients learn in the order of the local task sequence.



**Does the server forget what other clients have already learned?**

No, the server maintains a global posterior at all times into which the knowledge of all clients that have participated is integrated, so the server does not forget the knowledge of tasks that have already been learned. When the client performs learning, it downloads the latest global knowledge as a prior for local training, thus migrating the existing knowledge to the local model, so it can utilize the knowledge that has already been learned.



**Further explanation**

Client selection is an optional item in our approach, and our approach can also disable client selection and deal with the challenges posed by client interference through personalization alone. The advantage of using client selection is that it reduces client interference and speeds up the convergence of individual tasks, but its disadvantage can make unselected clients wait for the next collaborative task. And not using client selection will reduce client waiting, but will bring more client task interference.

We also verified performance without client selection in our experiments. In the experiments conducted on the benchmark datasets introduced by FedWeIT, we disabled client selection to maintain the same experimental setup as in FedWeIT, i.e., learning each client's tasks simultaneously, which are potentially similar, unrelated, or interfering. The experimental results are supplemented in Appendix F.1. The results show that we can still achieve superior performance over the comparison methods through personalization without applying client selection.

Moreover, as a kind reminder, client selection is not the core contribution of our work, our main contribution is the integration of global and historical knowledge in a unified framework, we innovatively propose the mixture prior that stores both global and historical knowledge to simultaneously address local overfitting and catastrophic forgetting. Furthermore, to the best of our knowledge, our approach is the first BNN-based solution for FCL, providing a new technological path that is different from previous approaches.

Reviewer comment (Experiments):
Experiments should be re-run on the benchmark data introduced in https://proceedings.mlr.press/v139/yoon21b/yoon21b.pdf, and all experiment results should include deviation bars.

Author response:
**Limited experimental setting compared with FedWeIT.**

We have restated our experimental setup in the first reply, our setup is equally challenging with FedWeIT, the client task sequences are heterogeneous, capable of responding the client interference.

To further validate the effectiveness of our method, we adopted the reviewer's suggestion to rerun the experiments on benchmark datasets Overlapped-CIFAR100 and NonIID-50 (repeated 3 times to obtain the mean and deviation bar), and added the detailed results to Appendix F.1, where the experimental results show that our method outperforms the others.


**Experimental results that lack deviation bar are not conclusive.**

All of our comparison experiments were repeated 3 times and averaged, and we revised Tables 1,2 to add deviation bar to the results. Due to space constraints, we only added deviation bar on "Avg.", and "Avg." denotes the average of the results in a single row of the table.

Reviewer comment (Theory):
The authors need to elaborate more on their theoretical results and explain, if possible, how they specifically account for the continual and federated learning setup.

Author response:
**Relationship between theoretical analysis and method ?**

In FCL, multiple tasks are learned sequentially, and each task is learned by a separate federated learning (FL) process. Since all FL processes are isomorphic, we analyze the upper bound of the generalization error of a single FL process to analyze the overall FCL process.

The theoretical analysis guided our approach as follows: First, the upper bound of the generalization error of our method is affected by sample size, model capacity, and mixture prior (which is the core innovation of our work). Second, mixture prior is theoretically shown to reduce the upper bound on generalization error. Third, we obtain results in our experiments that are consistent with the theoretical analysis, verifying the positive effect of the mixture prior on the model performance.