Paper: Client-centric Federated Learning (ICLR-2024)

Reviewer comment (Writing):
$t_k$ should be defined the first time it is used in "Distributed Estimation" rather than later in "Clusters Updating''.

Author response:
Answer to weakness 1: Thank you for identifying this issue. To rectify this, we have made the necessary adjustments accordingly in the section titled "Distributed Estimation" (highlighted in red).

Reviewer comment (Reproducibility):
In Algo 2: Server Update subroutine, should the first condition be $t-\tau > \tau_0$ instead? Client Thread: where is local timestamp $t_m$ updated before sending to the server?

Author response:
Response to weakness 2: Thank you for spotting this error; it should indeed be $t-\tau > \tau_0$. We have corrected this error in the paper (highlighted in red). Additionally, concerning the update of the local timestamp $t_m$, please refer to the last line of the `Client()` thread. It gets updated when the client receives the most recent timestamp from the server.

Reviewer comment (Theory):
For the weights $u_{mk}^t$ to sum to 1 in Algo 1, we need $c_1, c_2$ to satisfy $c_1 \sum_i \ell_i + c_2 \sum_i d_{1i} + (1-c_1-c_2) \sum_i d_{2i} = K$. Is it necessarily true that such $c_1,c_2$ will exist?

Author response:
Answer to weakness 3: Yes, the way the distribution $u_{mk}^t$ is computed is designed such that $\sum_k u_{mk}^t =1 $ holds for arbitrary values of $c_1, c_2$ such that $c_1 \in [0, 1]$, $c_2 \in [0, 1]$ and $c_1+c_2 \in [0, 1]$.

Reviewer comment (Writing):
What does "fluid dynamics" mean? And fluid dynamics is a completely different field in itself, so I would suggest rephrasing.

Author response:
Answer to weakness 4: We apologize for any potential confusion arising from our terminology. By 'dynamics', we specifically refer to the changes in clients' data distributions. To avoid any misinterpretation, we have now removed the term 'fluid dynamics' from the conclusion section.

Reviewer comment (Theory):
My main concern with the work is its practical relevance. The authors claim to put clients at the forefront. But, if the server estimates the distribution of the clients, this is a violation of the client data privacy? How can this be reconciled since privacy is one of the privacy motivations for doing FL in the first place?

Author response:
Answer to main concern: Thank you for raising this insightful question. We want to clarify that the focus of client-centrism considered in this paper is to enhance the performance and efficiency of clients in FL, but not particularly on privacy. Nevertheless, we emphasize that the potential information leakage in our framework remains the same with that of a canonical FL framework (e.g., FedAvg). Specifically, in our design, a client uploads its local model trained on private data to the server and nothing more, which is identical to what is done in canonical FL frameworks. Therefore, our design does not amplify the risk of privacy violation, and any information about private data that can be inferred in our framework can also be inferred by curious server in canonical FL. We acknowledge that privacy is certainly a critical aspect in FL, and how to further enhance clients' data privacy in our client-centric framework is an interesting and important future research question.

Reviewer comment (Writing):
Assum. 1: What is $F$? Do you mean $F_k$ for all $k$?

Author response:
Answer to question 2: Yes. To make it more clear, we have now restated Assumption 1 (highlighted in red).

Reviewer comment (Theory):
Theorem 1: the authors have not commented on the result. Why is the constant $\epsilon$ needed? Why does the convergence for any $k$ depend on $\sum_i a_i$? Most importantly, can you show explicitly for what choice of the learning rate $\gamma$ can we get convergence, and how do $S_k$ and $H$ affect it?

Author response:
Answer to question 5: We apologize for the oversight in not discussing the final convergence theorem. We have now included additional discussions on our convergence analysis (highlighted in red) for your reference. 
* We need the small constant $\epsilon$ to further upper bound the right hand side of inequality (7), where we need a lower bound of $\mathbb{E}[\langle \nabla H_k(w_{h-1}; \mathbf{w}_\tau), \nabla h_m(w _{h-1}; \mathbf{w} _\tau)\rangle]$. We can show that there exists a positive constant $\epsilon$ such that for $\rho \geq  \frac{2V_1+\frac{1}{2}\left\|w _{h-1}-\mathbf{w} _\tau\right\|^2+\sqrt{4\left\|w _{h-1}-\mathbf{w} _\tau\right\|^2(1+V_1)\epsilon}}{2\left\|w _{h-1}-\mathbf{w} _\tau\right\|^2}$ and all possible $w _{h-1}, \mathbf{w} _\tau$, $\mathbb{E}[\langle \nabla H_k(w _{h-1}; \mathbf{w} _\tau), \nabla h_m(w _{h-1}; \mathbf{w} _\tau)\rangle]$ can be lower bounded by $\epsilon\left\|\nabla F_k(w _{h-1})\right\|^2$, then we can upper bound the right hand side of inequality (7) by $\epsilon\left\|\nabla F _k(w _{h-1})\right\|^2$.
* To the question why the convergence for any k depends on $\sum_i{a_i}$, it is because the convergence bound of model $k$ contains a term $\left\| \mathbf{w} _\tau - \mathbf{w} _{s-1}\right\|$ , which is shown in (13)-(18) to be upper bounded by $\left(2\sum _{i=1}^K a  _i+(2K+1)a _k+K\right)\Delta$. Intuitively speaking, the model $\mathbf{w} _\tau$ is aggregated by all cluster models $\mathbf{w} _1, ..., \mathbf{w} _K$, so $\left\|\mathbf{w}  _\tau-\mathbf{w}  _{s-1}\right\|$ can be seen as the distance from an aggregation of all cluster models at epoch $\tau$, namely the aggregation of $\mathbf{w} _i^\tau, i=1, ... K$, to the cluster $k$'s last model $w _{s-1}$. Since we assume for $\forall i \in [K]$, $\left\|\mathbf{w} _i\right\| \leq a _i \Delta$, thus $a _i$ can help bound the distance from cluster $k$'s model to other clusters' model $\mathbf{w} _i$. As a result, $\sum _i a _i$ is needed here to bound the $l _2$-norm distance between $\mathbf{w} _\tau$ and cluster $k$'s last model $\mathbf{w} _{s-1}$, namely $\left\|\mathbf{w} _\tau-\mathbf{w} _{s-1}\right\|$, and finally contributes to the upper bound of the convergence of distribution $k$. 
* For the choice of $\gamma, S _k, H _{min}$ and $H _{max}$, as shown in the proof in Appendix A.2, the upper bound of $\mathbb{E}[\left\|\nabla F _k(w)\right\|^2]$ is proven to be $\frac{\mathbb{E}[F _k(\mathbf{w} _{0})-F _k(\mathbf{w} _{S _k})]}{\alpha _0\gamma\epsilon S _k H _{min}}+\frac{\left(\frac{L _k}{2}+\rho H _{max}+\frac{\rho H _{max}^2}{2}\right)\gamma H _{max}V_2}{\epsilon H _{min}} +\frac{\sqrt{V _1} \left(2\sum _{i=1}^K a _i+(2K+1)a _k+K\right)\Delta}{\gamma\epsilon H _{min}}+\frac{\left(\frac{L _k}{2}+\rho\right)\left(2\sum _{i=1}^K a _i+(2K+1)a _k+K\right)^2\Delta^2}{\gamma\epsilon H _{min}}$. Here we do not specify particular values for variables $H _{min}$, $H _{max}$, $S _k$, and $\gamma$. One possible choice of these variables, as suggested in [1], is that $\gamma = \frac{1}{\sqrt{S _k}}$ and $S _k = H _{min}^5$, leading to an upper bound of $\mathcal{O}\left(\frac{1}{\epsilon H^3 _{min}}+\frac{\delta^3}{\epsilon\sqrt{H _{min}}}+\frac{K\delta}{\epsilon\sqrt{H _{min}}}+\frac{K^2\delta^2\Delta}{\epsilon\sqrt{H^5 _{min}}}+\frac{K^2\delta^2\Delta^2}{\epsilon\sqrt{H^3 {min}}}\right)$ with $\delta = \frac{H _{max}}{H _{min}}$. Generally speaking, the upper bound increases with $H _{max}$, and decreases with $S _k$ and $H _{min}$. Intuitively, $S _k$ is the iteration counts of learning cluster $k$, thus more iterations at the server lead to a smaller upper bound of $\mathbb{E}[\left\|\nabla F _k(w)\right\|^2]$. $H _{min}$, $H _{max}$ are respectively the least and most local iteration counts before a single client updates to the server in one cycle, then bigger $H _{min}$ indicates a more sufficient local training, and smaller $H _{max}$ reduces the influence of overfitting.
*  In conclusion, this theorem indicates that if a client's model $w$ undergoes continuous training on data from distribution $k$, meaning that a portion of the client's data consistently originates from distribution $k$, then the $l_2$-norm of the gradient of the model loss on cluster $k$ will converge to a specific point (always less than $\infty$). For any data distribution $k$ continuously sampled by a client, the proposed algorithm guarantees the client's model to have good performance on this particular distribution $k$.

[1] Xie, C., Koyejo, S., Gupta, I., 2019. Asynchronous federated optimization. arXiv preprint
arXiv:1903.03934 .

Reviewer comment (Evaluation):
Table 1: FedSoft-Async ACC for FMNIST(6). How come cluster accuracy is worse than everything else, even client bfr and local ACC.?

Author response:
Response to question 6: Thank you for your meticulous observation. To address the query regarding accuracy, please refer to Figure 9 in Appendix A.3, titled "Timeflow accuracy of clients and clusters on FashionMNIST (6 clusters)". As depicted in the figure, lower accuracy for cluster accuracy arises from certain significantly low-performing clusters, such as cluster 1. This occurrence can be attributed to the limitations of `FedSoft` in accurately estimating distributions, particularly when dealing with a relatively large number of clusters. Previous experiments, as detailed in [1], primarily focused on 2 or 3 clusters. However, in the case of 6 clusters, the distribution estimation by `FedSoft` was notably inaccurate, sometimes approximating an equal distribution of 1/6 for each cluster. Consequently, this inaccurate estimation led to clusters being updated by models that did not align with their corresponding distributions, thereby resulting in lower cluster accuracy over time.

[1] Ruan, Y., Joe-Wong, C., 2022. Fedsoft: Soft clustered federated learning with proximal local updating, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8124–8131.

Reviewer comment (Theory):
The global adjustments are claimed to mitigate adversarial behavior. But, if the server has data from all distributions, won't it recognize adversarial behavior anyway?

Author response:
Answer 9: We sincerely apologize for any confusion caused by the use of the term "adversarial". It's essential to clarify that in our framework, we assume all clients to be honest. The global adjustments we introduce serve to *"filter out"* models that might have a detrimental impact on the global cluster models. Due to the non-iid nature of the clients' data, certain trained models may not enhance the overall model performance and could, in some cases, even reduce global model accuracy. In such instances, the global adjustments effectively prevent these potentially harmful models from impacting the server cluster accuracy. We have rectified this improper use of the term "adversarial behavior" in the relevant section of our paper (highlighted in red) for your reference.

Reviewer comment (Theory):
Although the paper provides a convergence result in Section 4.3, there is no discussion about it. Does this result elucidate anything about the behavior of the algorithm under the given assumptions? I admit I did not check the math carefully.

Author response:
Response to weakness 1: Thank you for highlighting this concern. We have now added discussions regarding Theorem 1's implication after the theorem statement (indicated in red). Specifically, the theorem indicates that if a client's model $w$ undergoes continuous training on data from distribution $k$, meaning that a portion of the client's data consistently originates from distribution $k$, then the $l_2$-norm of the gradient of the model loss on cluster $k$ will converge to a specific point (always less than $\infty$). For any data distribution $k$ continuously sampled by a client, the proposed algorithm guarantees the client's model to have good performance on this particular distribution $k$.

Reviewer comment (Evaluation):
The presented experiments don't account for variance across repetitions. None of the plots have error bars, and the error bars in Table 1 represent variance across clients in one single repetition. This would make the results more robust, particularly when the results are close (e.g., in Table 1, when the number of clusters is 4).

Author response:
Response to weakness 2: Thank you for highlighting the absence of error bars, which could potentially affect the robustness of our experiments. We have incorporated error bars, indicated by shaded areas, based on three repetitions for each experiment in Figure 3 in the main body of the paper, Figures 6-17 in Appendix A.3. Table 1 is also updated with average accuracy across 3 trials.

Reviewer comment (Experiments):
There is no discussion on how to choose the number of clusters or where the public cluster data would come from.

Author response:
Response to weakness 3: Thank you for highlighting the missing information. The total number of clusters, denoted as $K$, should indeed be considered prior knowledge, which the service provider may decide based on the type of service (e.g., skincare maintenance), or deduce from its available data, possibly encompassing various user attributes such as genders or ethnicities. The public cluster data can be collected initially by the service provider, and only small amount can help to achieve high cluster and client performance: see experiment on different sizes of public datasets in Figure 18 in Appendix A.4. With only 1000 data points at the server, a client accuracy of 80\% and a cluster accuracy of 82\% can be achieved for FashionMNIST dataset. We have also provided further clarification on this aspect in our problem definition (Section 3), highlighted in red for your convenience.

Reviewer comment (Theory):
I enjoy the client-centric philosophy presented by the paper, but I am not sure if the method truly adheres to this philosophy. I understand this client-centrism is exemplified by the asynchronous nature of the framework, giving clients autonomy over when to send updates to the server. Similarly, the philosophy is reflected by the fact that the updates result in a personalized model for the clients. However, I struggle to reconcile the idea of a client-centric framework with the fact that the server is still a key element and bottleneck in this algorithm. In fact, the paper recognizes that more tasks have been placed on the server compared to previous work (e.g., distribution estimation). This is the main reason for my rating, as I believe the manuscript should either be more careful in addressing this tension, or be more specific in their title/motivation.

Author response:
Answer to Question 1: We sincerely regret any confusion and misunderstanding that may have arisen. To establish a client-centric service, we have envisioned the server's role as an assistant to the clients. In addressing our problem setting where clients' data exhibit a mixture of distributions that dynamically change over time, we opted for a clustered FL framework to resolve this challenge. It's noteworthy that existing clustered FL methodologies burden clients with all computation and communication loads, which doesn't align with the client-centric paradigm we strive for. Specifically, **our notion of client-centrism is for the client to achieve high performance with minimum computation and communication complexities**. Therefore, we restructured the clustered FL system, moving the major computation burden of distribution estimation onto the server, without requiring communicating all cluster models to the client, thereby significantly reducing communication and computation loads on clients. This redesign adheres to our goal of performant and lightweight clients. Importantly, a client-centric approach doesn't entail transferring all computation to clients but rather ensures they attain optimal performance with minimal resource utilization. To clarify this point, we have now revised portions of our introduction (highlighted in red) to better explain this motivation.

Reviewer comment (Writing):
In the first paragraph of Section 4.2, there is a reference to "aggregation of global models". I assume this refers to the cluster models? This particular term was confusing when I first read the manuscript.

Author response:
Response to question 2: We sincerely apologize for this confusion. To clarify, yes, our reference to "aggregation of global models" indeed refers to the process of combining cluster models. We have revised the first paragraph of Section 4.2 (highlighted in red) to make this clear.

Reviewer comment (Experiments):
When explaining the experimental set-up in Section 5.1, there is no detailed description of how the clients' data changes over time. If this is in the appendix, please include a reference.

Author response:
Answer to question 3 : Thanks for your careful reading of the our experiments. In response to your feedback, we have now included in Appendix A.1 (highlighted in red) detailed descriptions on how clients' data change over time in our experiments.

Reviewer comment (Experiments):
Just to clarify: in Figure 4(a), you know the true distribution of the clients because you track how their data changes through time?

Author response:
Answer to question 4: Yes, we monitored the evolution of clients' data throughout our experiment to facilitate a comparative analysis between our estimation method and the baseline.

Reviewer comment (Experiments):
The technical contributions can be difficult to follow, although the paper seems to suggest their main contribution is to have a client centric update process a lot of the focus is on clustered FL and client level distribution shifts making the exact setting and problem a bit contrived. It would be good to isolate these factors better both in the motivation of the method and in the experiments.

Author response:
Thank you for your valuable suggestion regarding the isolation of these factors. We have revised the introduction section (highlighted in red) to clearly describe the relationship between client-centric FL, clustered FL, and distribution shifts. Specifically, we now clearly describe the setting of the considered client-centric FL: a client's local data is sampled from a mixture of distributions whose mixing ratio may change over time, hence the client may need to update its model with the server to accommodate the new distribution. We also provide real-life examples to illustrate the significance of the considered client-centric FL problem. On the other hand, as now clearly stated in the introduction, clustered FL, in which a group of $K$ cluster models (one for each distribution) are hosted at the server, and are updated using the uploaded data from clients, is adopted in this paper as a basic framework to tackle the challenges in the considered client-centric FL problem. Our main contribution of this paper is to formulate the client-centric FL problem, and propose a clustered FL based protocol for this problem (where when to do model update is autonomously decided by each individual client) with high cluster and client accuracies, and minimum communication and computation costs at clients.

Reviewer comment (Novelty):
How does client level distribution shift relate to the area of continual federated learning.

Author response:
We regret any confusion that may have arisen regarding the relationship between distribution shift in our setting and Federated Continual Learning (FCL). It's critical to note that our setting is drastically distinct from FCL. FCL primarily focuses on mitigating catastrophic forgetting induced by arrivals of new tasks at each client, striving to achieve proficient performance across both past and current tasks. Conversely, our focus lies solely on maintaining high performance on the current task. We aim to provide clients with models better suited to their update-to-date data when their data distribution undergoes a shift. In order to avoid any confusion, we have revised the related work section to explicitly compare our setting with FCL (highlighted in red).

Reviewer comment (Experiments):
There are a lot of hyperparameters, how would these be decided in practice.

Author response:
Thank you for your inquiry. For comprehensive insights into our experiments, we have meticulously documented all the specific hyperparameters utilized in our study in Table 2 in Appendix A.1. Additionally, we have conducted ablation studies involving variations on the number of clients, different regularization terms represented by $\rho$, and various adjustment thresholds in Section 5.3 Ablation study and Appendix A.4. If you require more detailed information about particular hyperparameters and their impact on the algorithm's performance, please feel free to specify your query. We would be more than willing to provide further clarification and elaboration on these aspects.

Reviewer comment (Experiments):
How would this framework perform in a setting without clusters or assumptions of local distribution shifts.

Author response:
Thank you for the question. Our framework is designed as a comprehensive solution, encompassing the mentioned cases as special instances. We have also included experiments on FashionMNIST (1 cluster) in the Appendix A.3, as depicted in Figure 6, and CIFAR-100 (1 cluster), depicted in Figure 10. In these two experiments,  the local distribution would shift after the clients update their local model. What's more, we included experiments on FashionMNIST where clients' data distribution stay the same as shown in Appendix A.4, Figure 19, 20, 21. This additional experiment illustrates that our method consistently outperforms `FedSoft-Async` concerning cluster and client accuracy.