Paper: Concept Matching: Clustering-based Federated Continual Learning (ICLR-2024)

Reviewer comment (Novelty):
There are some more related works [1, 2] about clustered federated learning not included in this paper.

Author response:
We added them in the updated version. Let us note that these two works are related only in the sense that they apply clustering in federated learning. They do not tackle any FCL challenges as our proposed approach.

Reviewer comment (Experiments):
The authors cite several related works about FCL but compare only one of them in experiments.

Author response:
In addition to vanilla FL, our evaluation adapts two baselines. One baseline (FedWeIT) is the most cited FCL work published recently in a top machine learning conference. The other baseline is to apply a classic CL method (EWC) in FL. We believe they are representative of the state-of-the art.

Reviewer comment (Experiments):
This paper claims that CM can handle both class-incremental and task-incremental FCL but the experiments only include class-incremental setting.

Author response:
In 5.1 experimental setup of the original submission, we mention that in task-incremental scenarios, the difference is that the clients do not have to perform concept matching every round, because they know the task IDs and the concept drift due to the transition of different tasks. For the same experimental setting, task-incremental training would achieve the same model performance with lower computation overhead at the clients.  

We run additional experiments under the task-incremental setting. In the updated paper Appendix C.6 Figure 4, the learning curves of both the task-incremental and class-incremental settings exhibit smooth learning progress, and little difference in terms of model performance. The paper refers to this figure in 5.1 experimental setup when discussing the difference between task-incremental training and class-incremental training.  

In the updated paper, we added the clarification in Section 4.3 that in both the task-incremental and class-incremental scenarios, the server does not know the concepts of clients data, and it shall perform clustering and concept matching every round. We rephrased “the clients do not have to perform concept matching every round” into “the clients only need to perform concept matching at the initial rounds”  for the task incremental-scenarios. In other words, the difference of CM adapted in the class-incremental and task-incremental is that the client in the task-incremental scenarios can use the mapping after it encounters all tasks. The other operations proposed in this study are all necessary for both task-incremental and class-incremental.

Reviewer comment (Experiments):
Are there experimental results on the datasets of FedWeIT under its setting?

Author response:
We evaluate the proposed approach on the same data settings of FedWeIT. However, as we mention in 5.2 results  of the original submission, the learning settings of FedWeIT are on a very small scale and unrealistic (i.e. 5 clients per round, 5 classes to train per client, and non-overlapping classes over clients). In addition, the server in FedWeIT knows the class subset labels from the clients. Since our goal is to evaluate under realistic assumptions and on a larger scale, these learning settings are different from the original FedWeIT paper.

Reviewer comment (Evaluation):
Can we regard it as privacy risk to update and upload only selected concept models after local training?

Author response:
In the original submission, we explain in 4.3 design discussion that in terms of privacy, the CM framework is the same as vanilla FL (i.e., the clients send only their model weights to the server). In the updated paper, we further added the explanation that an adversarial server might seek to extract information from the client model using the same techniques from vanilla FL, which is beyond the scope of this paper.

Reviewer comment (Theory):
The problem background is unreasonable and does not align with practical needs. The requirement for each client to train with different datasets each time is not realistic in real-world scenarios.

Author response:
In the introduction and overview, we state the motivation background for our problem as FCL for IoT and mobile devices, where it is difficult to train with the entire dataset on-device at every round due to resource constraints, and the data not only accumulate over time, but also change their distributions.  We provide human activity recognition (HAR) as an example, and its distribution change can be due to the subsets of activities, the locations of the activities, or the health status of the user. 
It seems the reviewer only focused on the scenario in Figure 1 where each client trains with different image classes each time. Nevertheless, we added another example of such a real-world scenario in the updated paper. For the photos a client takes in daily life, the concepts can be different types of the place of interest visited (e.g., museum vs. national park), or different types of meals (i.e. breakfast, lunch, dinner).

Reviewer comment (Novelty):
The method lacks innovation and can be seen as a mere patchwork of existing client aggregation approaches.

Author response:
The novelty was further clarified in the updated paper. There are two main novelties in the design. The first novelty is the clustering-based framework to alleviate catastrophic forgetting and interference among clients, and consequently achieve good model performance in FCL. The second novelty is our effective and theoretically grounded concept matching algorithms. Because the server does not know which cluster model is responsible for which concept, clustering alone will not solve FCL.

Reviewer comment (Experiments):
The experimental comparisons lack persuasiveness as the baselines should be diverse in their configurations.

Author response:
- In the updated submission, we added the comparison results between vanilla FL and CM where all the data is separated (i.e., treating each dataset as a concept without mixing or splitting the datasets). Similar to the orignal settings, Figure 3 in Appendix C.5 illustrates the smooth learning curves for CM over vanilla FL for all concepts.Compared with vanilla FL, the weighted average accuracy over the number of samples for all the concepts is improved from 85.5% to 88.0% as well.
- We do not test the scenario with  all the data combined, because it is not a CL setting. 
- In the original submission, we explain in Appendix C.1 that we split FaceScrub into two concepts to verify whether CM can differentiate them successfully. We mix the three MNIST datasets together to make it more difficult, because MNIST datasets are easy to learn. The purpose is to stress-test CM rather than favoring the proposed method.

Reviewer comment (Theory):
The model's implicit assumption that the number of concepts is smaller than the number of global concept models introduces a limitation. When there is a large number of concepts, it becomes evident that the algorithm's maintained models may encounter difficulties in effectively handling this situation. This limitation is a result of the algorithm's design.

Author response:
The paper does not assume the number of concepts is smaller than the number of global concept models. On the contrary, our evaluation shows CM has good resilience when configured with numbers of concept models that are different (either smaller or larger) from the number of concepts. The results are in the Appendix C.6 due to the page limit.  Even when the number of concepts is larger than the number of global concept models,  the average model accuracy achieves 89.5% and 88.9% respectively, and beats vanilla FL (86.7%).

Reviewer comment (Novelty):
The overall design of the model lacks novelty, as there are many similar methods available. The author simply transfers existing methods from other domains to the problem of federated continual learning without introducing any substantial innovation.

Author response:
The novelty was further clarified in the updated paper. There are two main novelties in the design. The first novelty is the clustering-based framework to alleviate catastrophic forgetting and interference among clients, and consequently achieve good model performance in FCL. The second novelty is our effective and theoretically grounded concept matching algorithms. Because the server does not know which cluster model is responsible for which concept, clustering alone will not solve FCL.

Reviewer comment (Experiments):
The experiments should be more comprehensive and diversified. It is recommended to include baselines that cover scenarios where all the data is combined or where all the data is separated. This will ensure a fair comparison and prevent experiments from being solely designed to favor the proposed method.

Author response:
- In the updated submission, we added the comparison results between vanilla FL and CM where all the data is separated (i.e., treating each dataset as a concept without mixing or splitting the datasets). Similar to the orignal settings, Figure 3 in Appendix C.5 illustrates the smooth learning curves for CM over vanilla FL for all concepts.Compared with vanilla FL, the weighted average accuracy over the number of samples for all the concepts is improved from 85.5% to 88.0% as well.
- We do not test the scenario with  all the data combined, because it is not a CL setting. 
- In the original submission, we explain in Appendix C.1 that we split FaceScrub into two concepts to verify whether CM can differentiate them successfully. We mix the three MNIST datasets together to make it more difficult, because MNIST datasets are easy to learn. The purpose is to stress-test CM rather than favoring the proposed method.

Reviewer comment (Theory):
In terms of equations, many equations are not written clearly and normatively. Specifically, Eq.(2) is wrong where k* has missed n in Eq.(2), and the argmin operation of the loss function is not presented. Aggregate function in Eq.(4) is not expressed. \Theta and \omega are not unified in \Theta^t={\omega_1,\omega_2,...,\omega_J}.

Author response:
- Eq.(2) is a client operation, k* is a client local value unrevealed to the server. Thus, n is omitted in the original submission.  Due to the confusion, we added n in the updated paper. 
- We do not find any equation with loss function in the original submission where the argmin operation is absent. They are all presented when expressing the optimization of the loss function, including eq(1) and eq(6). It will be helpful if the reviewer could indicate the specific equation in the paper where the argmin operation of the loss function is not presented. 
- From the abstract and throughout the paper, we state the CM framework is flexible to use any aggregation function. Thus, we did not specifically express any aggregation function. 
-  \Theta (aggregated cluster model set) is used to differentiate the global model set \Omega, even though its elements are also models. Thus, we used a different symbol.

Reviewer comment (Writing):
In terms of the writing, this submission has not been written well. What is the meaning of the concept in the image experiment of this paper? This abstract vocabulary has not been explained clearly. More reasons for experiments should be analyzed. Many equations are not clear. In Algorithm 1, the \omega_j and \omega_k are repetitive in the inner and outer loop.

Author response:
- In the original submission, we define the image concepts as different sets of classes from the first time we mention image in section2 overview.
- We updated the abstract. We would greatly appreciate it if the reviewer could provide more specific details regarding the aspects in which the abstract is unclear.
- The first paragraph in evaluation of the original submission specifies the main goals of the experiments as well as minor goals for Appendix. Our experiments are designed to target these goals. 
- \omega_j and \omega_k are elements in different sets. We fixed the minor issue due to the confusion.

Reviewer comment (Experiments):
The authors claim that two issues in Federated Continual Learning (FCL), catastrophic forgetting and interference among clients, can be greatly diminished. It will be better to prove that claim through conducting some experiments.

Author response:
Figure 2 in the original submission illustrates the negative effect of catastrophic forgetting and interference among clients on the learning curve and the model performance in vanilla FL, and CM can effectively smoothen the learning curve and improve the model performance. We will investigate how to further quantify catastrophic forgetting and interference among clients in the next version.

Reviewer comment (Novelty):
The novelty is limited, as there are some Federated Learning (FL) works adopting the clustering method. however, the authors apply the clustering method to FCL at a single time step, which does not reflect the unique design of CL.

Author response:
The reviewer misunderstood the proposed approach. Throughout the paper, we indicate clustering is applied every FCL training round. Clustering alone will not solve FCL, because the server does not know which cluster model is responsible for which concept. Our additional novelty lies in our effective and theoretically grounded concept matching algorithms.