From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of client drift in federated learning, by decomposing the network and gradients into two parts of low-level and high-level features, thereby improving gradient similarity.

Unfortunately some concerns remained on the given explanations relating to the client drift phenomenon (which is alleviated in several existing methods, but doesn't solve the studied problem in deep learning). It is not fully explained why the proposed technique should improve generalization. Another main concern was that experiments were conducted only on synthetic small datasets and not realistic federated ones.

Finally consensus among the reviewers remained that the paper in its current form remains below the bar even after the discussion phase, mainly due to the mentioned concerns.

We hope the detailed feedback helps to strengthen the paper for a future occasion.
justification_for_why_not_higher_score: Concern remained after discussion, on both theory and experimental contributions
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper4317/Reviewer_3s3d
title: Acknowledging your answers
comment: Thanks for your answers, the paper looks more clear than before, but I still think the theory is not strong enough, however, the experimental results are nice. I raise my score from 3 to 5.

From: ICLR.cc/2023/Conference/Paper4317/Reviewer_fFS7
title: thanks for the repsonse
comment: Sorry for the late reply. I will keep my initial rating after carefully reading the response.

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Window for draft updating is closing
comment: 
Dear Reviewer eaTq,

Thanks very much for your time and valuable comments. We understand you're busy. But as the window for paper revision is closing, would you mind checking our response ([a brief summary](https://openreview.net/forum?id=bp6Lr0TmmUS&noteId=OTsT8le_9J2), and [details](https://openreview.net/forum?id=bp6Lr0TmmUS&noteId=qLMzGASbg19)) and confirm whether you have any further questions?

Best regards and thanks,

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Window for draft updating is closing
comment: 
Dear Reviewer 3s3d,

Thanks a lot for your efforts in reviewing paper and your constructive comments. We understand you're busy. But as the window for paper revision is closing, would you mind checking our response ([a brief summary](https://openreview.net/forum?id=bp6Lr0TmmUS&noteId=izCxBCedtj), and [details](https://openreview.net/forum?id=bp6Lr0TmmUS&noteId=vyBQGWlD9f)) and confirm whether you have any unaddressed concerns?

Best regards and thanks,

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Thanks for your supports
comment: Dear Reviewer fFS7,

We have updated our draft following your suggestions. Please let us know if you feel other changes are needed to address your concerns. We are very glad to provide answers and revision to your further questions.

Best regards and thanks,

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: A brief summary of our response
comment: 
Dear Reviewer 3s3d,

Thanks again for your time and efforts in reviewing our paper. To save your time, we provide a brief summary of our previous response:
- Regarding the confusing notation problem, we carefully revised the notations in the paper, and we add more clarification on the confused points. 
- Regarding the remarks after Th4.1, we have revised the paper following your constructive suggestion.
- Regarding the definition of the worst-case margin, we have revised it to show that it can be a good measure of generalization.
- Regarding the privacy concerns, we provided the clarificaation of how it does not introduce extra privacy leakage than FedAvg.

As the window for draft updating is closing, please reach us out if you have any further concerns.

Best regards and thanks,

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: A brief summary of our response
comment: 
Dear Reviewer eaTq,

Thanks again for your time and efforts in reviewing our paper. We provide a summary of our previous response to save your time, as our reply might be a bit long:
- Regarding the notation problem, we have carefully revised paper to clear the points that may confuse you. 
- Regarding the presentation of the theoretical part and the generalization, we added more clarification and the explanation of how the proposed algorithm exploits the theoretical conclusions to harness client drift.

We humbly expect you could check our response and confirm whether our response has addressed your concerns. Please reach us out if you have any further concerns.

Best regards and thanks,

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: looking forward your further outstanding questions
comment: 
Dear reviewer fFS7,

Thank you again for taking the time to read our paper and finding our theoretical result is sound and our algorithm is novel and effective. According to your insightful comments, we have carefully revised the paper to answer the questions.

We hope that our response can solve your concerns. And we look forward to further discussing with you if you have any further questions.


Best regards

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: looking forward your further outstanding questions
comment: 
Dear reviewer 3s3d,

We appreciate that you spent time reviewing our paper and the valuable comments. We have endeavored to answer your questions and revise the paper in response to your comments. The responses are summarized as follows: (1) we restate the remark after Th4.1 for more clear presentation and clarification; (2) the worst-case margin is explained in more detail; (3) we make more clarification about the Gaussian distribution and the privacy concerns.

Since the discussion due is approaching and we have addressed most similar questions of the **Reviewer UyvV**, would you mind checking our response and revision to confirm where you have any further questions? We look forward to seeing your suggestions about the concrete places that we can improve.

Best regards

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: looking forward your further outstanding questions
comment: 
Dear reviewer eaTq,

Thanks for your great efforts in reviewing and insightful comments, according to which we have tried our best to answer the questions and carefully revise the paper. Specifically, we revised the paper for the notation problems and the presentation of the theoretical part, and provided more explanations about our algorithm following your constructive suggestions.

Since the discussion due is approaching, would you mind checking our response to confirm whether you have any further questions? We are looking forward to your reply.

Best regards

Authors of #4317

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Response to Reviewer UyvV Part [1/3]
comment: We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our theoretical concepts are novel and the experimental evaluation is quite extensive and done accurately. Please see our detailed feedback below.


**Q1**: ''In the distributed case without multiple local SGD steps, the variance can be eliminated via known control variate techniques, and even with local steps, it is not a problem from an optimization viewpoint.''

**A1**: The authors totally agree with your in-depth understanding of the problem. Namely, we agree with that it is not a problem from an optimization viewpoint, so we introduce a generalization viewpoint. Note that, when meeting the Non-IID data distribution, local-SGD with control variate techniques [1][2] cannot fully eliminate such a variance, but only reduce it. This leads to a severe slow convergence problem[3]. How to eliminate such a gradient variance is still under exploring in the literature from an optimization viewpoint [4][5].


**Q2**: "The current theory is not able to explain the bad performance of these methods in the setting of training non-convex deep neural networks. So, it is hard for me to agree with the original problem formulation."

**A2**: Thank you for pointing out this critical problem. The distribution discrepancy among local training can be large, thus leading to poor model performance. This is consistent with our theoretical conclusion. Meanwhile, the proposed decoupling strategy mitigates the distribution discrepancy, so improving the model performance. Moreover, we believe all of us prefer a theorem that can explain the bad performance of existing methods, but it is out-of-scope of a conference paper.


**Q3**: ''The authors make an assumption that feature distributions can be approximated by a Gaussian referring to some previous works. Why is it the case? Was this assumption tested in practice e.g. via a statistical test? Is it the case only for a particular dataset and model architecture?''

**A3**: 1) To approximate feature distributions, we have to make some assumptions to approximate the distribution of the feature distribution, as the true distribution is unknown. 2) We employ the maximum mean discrepancy (MMD) and two sample test to verify the assumption. The results show that the p-value calculated using MMD is much higher than 0.05 on CIFAR10 dataset, demonstrating the rationality of our assumption. 3) The p-value on rounds 0, 400, and 900 are 0.75, 0.9, and 0.95, respectively, showing the real features are highly similar to the estimated Gaussian distribution.


**Q4**: "Can the proposed method be combined with standard federated learning techniques like differential privacy and secure aggregation?"

**A4**: It is an exciting and promising direction to explore, but it is out-of-scope of this work. Nevertheless, for the security problem, we added the noise of different degrees on the shared estimation parameters $\mu$ and $\sigma$ to see how noise influences performance. The experiment results in Table 7 in Appendix show that our methods can perform well with large noise.


**Q5**: "Important baselines like FedAdam/FedAdagrad are missing.They were shown to tackle well the natural heterogeneity in federated datasets. Can the method from this work be combined with adaptive optimizers?"

**A5**: Our method is orthogonal to these methods. Specifically, we reduce the gradient dissimilarity through the decoupling strategy and sampling features from the estimated feature distribution. Our method does not change the procedures of the gradient descent. Thus, other FL optimizers can be combined with our method.


**Q6**: Experiments were conducted only on synthetically created small federated versions of centralized datasets. It is not clear whether the conclusions can be generalized to natural federated datasets like Federated Extended MNIST, Stackoverflow, etc.

**A6**: Thanks for these suggestions about experiments. We have added the experiments of the Federated Extended MNIST to test our methods. The results are shown in Figure 11 and Table 8 in the Appendix of the revised paper, showing our method outperforms other FL optimizers.

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Response to Reviewer UyvV Part [2/3]
comment: **Q7**: Experimental results are obtained for single runs which limits their credibility. In addition, the curves in Figure 2 (b) are almost indistinguishable, and the benefits of the proposed approach are not clear.

**A7**: Thanks for indicating this experiment limitation. We have run experiments multiple times to verify our methods. And we modify Figure 2 (b) to make the curves more clear.



**Q8**: Important baselines like FedAdam/FedAdagrad are missing. They were shown to tackle well the natural heterogeneity in federated datasets. Can the method from this work be combined with adaptive optimizers?

**A8**: Our method only samples new data from the estimated share feature distribution. And our method does not modify the gradients of local training and the aggregation on the sever-sides. Therefore, our method can be directly combined with other FL optimizers, including FedAdam/FedAdagrad.



**Q9**: ''What do $\rho(\cdot)$ and $y_m$ look like mathematically? These objects (their dimension and the space they come from) were not properly defined.''

**A9**: We have revised our paper to clearly define notations. Suppose there are $C$ classes in all datasets $\cup_{m\in\mathcal{M}}\mathcal{D}_m$ indexed by $[C]$. A sample is $\mathcal{D}_m$ is denoted by $(x,y)\in \mathcal{X}\times[C]$, where $x$ is a model input in the space $\mathcal{X}$ and $y$ is its corresponding label. $\rho(\theta;x): \mathcal{X} \to \mathbb{R}^C$ represents the machine learning model, which maps the data $x$ to the predicted output.



**Q10**: ''Definitions 1, and 2 need much more detail and proper explanation. As far as I understand they are not standard in the community and I have not found these exact objects in the referenced papers, (Koltchinskii & Panchenko, 2002, Franceschi et al., 2018).''

**A10**: The Definition 1, worst-case margin, measures the expected largest distance between the data $x$ with label $y$ and the data $x'$ that is mis-classified by the model $\rho$. Thus, a smaller margin means the higher possibility to mis-classify the data $x$. If the margin is equal to zero, the model mis-classifies almost all samples of the given distribution. The similar definition can be found in [9].


**Q11**: ''What does notation $\Delta:\mathbf{L}(\mathcal{D}_m)$ in equation (2) mean? What is $\mathcal{J}$ from definition 2? How $\hat{f}$ on (page 5) is defined? How is equation (6) obtained?''

**A11**: The $\Delta:\mathbf{L}(\mathcal{D}\_m)$ means the pseudo gradient obtained by applying a learning algorithm $\mathbf{L}(\mathcal{D}\_m)$ to a distribution $\mathcal{D}\_m$. And $\mathcal{J}$ means a joint data distribution of model inputs $(\mathcal{D}|y, \mathcal{D}'|y)$. $\hat{f}$ equal to $f(\theta;\varphi_{\theta_{low}}(\xi_m)) + f(\theta; \hat{h})$ as shown in equation (6). We propose to optimize equation (6) by decomposing training the whole model into low-level and high-level parts. The high-level model is trained based on both the outputs from the low-level model and the sampled features from $\mathcal{H}$, thus reducing the gradient dissimilarity of the high-level model.


**Q12**: ''The sentence on page 6 “Intuitively, we prefer the client where the generalization contribution can be lower bounded.” is not clear to me. Why is it actually the case?''

**A12**: The lower bounded quantification of the generalization contribution $\mathbb{E}_{\Delta:\mathbf{L}(\mathcal{D}_m)}  W_d(\rho(\theta + \Delta), \mathcal{D} \backslash \mathcal{D}_m))$ means that updating model $\theta$ with the pseudo gradient $\Delta$ obtained by applying a learning algorithm $\mathbf{L}(\cdot)$ to a distribution $\mathcal{D}_m$ can **improve** the model generalization performance on other client distributions $\mathcal{D}\backslash \mathcal{D}_m$. Thus, client $m$ is preferred to be chosen.

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Response to Reviewer UyvV Part [3/3]
comment: 
**Q13**: ''The client drift is defined as $\frac{1}{|\mathcal{S}^r|}\sum_{i \in \mathcal{S}^r}  \| \bar{\theta} - \theta_i\|$ and referred to the work of Karimireddy et al., 2019. However, in the original SCAFFOLD paper, this term is defined differently.''

**A13**: Thanks for indicating this crucial mistake we made. Yes, the $\frac{1}{|\mathcal{S}^r|}\sum_{i \in \mathcal{S}^r}  \| \bar{\theta} - \theta_i\|$ is different from the SCAFFOLD paper.  The metric that we use should be an approximation of gradient dissimilarity[3][6] rather than the client drift defined in SCAFFOLD paper. We have modified it in the revision. This does not influence our experimental conclusion of Figure 2 (b), that our method can reduce the gradient dissimilarity.


**Q14**: ''Conclusion on “reducing the high-level gradient dissimilarity is more important than the low-level” on page 9 is drawn from one experimental example (ResNet-18 trained on CIFAR-10) and needs more evidence.''

**A14**: Thanks for your valuable suggestions. We have add more experiments to check the layer-wise divergence with ResNet-18 on FMNIST and SVHN, and ResNet-50 on CIFAR-100. The new experimental results are shown in Figure 10 in the Appendix D.3 of the revised paper. Results show that the weight divergence of low-level parts of the model shrinks faster than the high-level parts.



**Q15**: ''In Appendix Table 6 shows improved results for baseline methods when using momentum. It was previously shown (Hsu et al., 2019) that momentum can greatly help in the more synthetically heterogeneous case and parameters need to be adjusted for various heterogeneity levels. I would like to ask why the authors did not compare their approach to the stronger baselines.''

**A15**: Table 6 in the Appendix shows decreased results compared with baseline methods when using client-side momentum. The client-side momentum in FL training does not always commit better convergence because the momentum introduces larger local updates, increasing the client drift, which is also observed in [7]. And the server-side momentum (Hsu et al., 2019) may improve the performance. The compared algorithms including FedAvg, FedProx, SCAFFOLD, FedNova do not use the server-side momentum. For the fair comparisons, we did not use the server-side momentum for all algorithms.


**Q16**: ''Minor Comments and typos. Is the problem setting limited only to classification? Typos: "variance", missed work, SCAFFOLD reference.''

**A16**: Thanks for the contrustive comments and typos. Our method is not limitted only to the classification problem, as our method only needs to estimate the feature distribution. We have fixed the typos in the paper and modified the SCAFFOLD reference.


[1] SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. Karimireddy et al. In ICML 2020.

[2] Tackling the objective inconsistency problem in heterogeneous federated optimization. Wang et al. In NeurIPS 2020.

[3] Minibatch vs Local SGD for Heterogeneous Distributed Learning. Woodworth et al. In NeurIPS 2020.

[4] Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology. Huang et al. In ICML 2022.

[5] On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond. Yuan et al. In NeurIPS 2022.

[6] Federated Optimization in Heterogeneous Networks. Li et al. In MLSys 2020.

[7] FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks. He et al. In FL-AAAI-22. 

[8] LEAF: A Benchmark for Federated Settings. Caldas et al. Workshop on Federated Learning for Data Privacy and Confidentiality  2019

[9] Large Margin Deep Networks for Classification. Elsayed et al. In NeurIPS 2018.

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Response to Reviewer fFS7
comment: We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our theoretical result of local training generalization and methods are sound, and our idea is novel and effective. Please see our detailed feedback below.

**Q1**: ''The communication cost should be reduced (e.g., quantizations) (although authors talked in the limitations.)''

**A1**: Thanks for indicating this important and practical problem. We reduce the communication costs by estimating the feature distribution rather than broadcasting all features. However, the extra communication costs of the estimated parameters still exist. The suggested quantizations from the reviewer may practically reduce the communication costs further. We will explore this promising strategy in the future work.


**Q2**: "The theoretical result of local training generalization is sound and answers the question related to local training."

**A2**: The authors appreciate that the reviewer recognizes the key contribution of our work. Specifically, we investigate the generalization contribution of local training and employ the theoretical conclusion to promote the performance of FL models. We believe that your correct understanding and interpretation of the theorem can corroborate **Reviewer 3s3d**'s misunderstanding of our work.

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Response to Reviewer 3s3d
comment: We would like to thank the reviewer for taking the time to review our work. We appreciate that you find our work provides a nice and interesting idea and the experiments are thorough. Please see our detailed feedback below.

**Q1**: ''some statements are just wrong: the remark after Th 4.1 the authors say that Th4.1 implies 'the generalization contribution of the training distribution is expected to be large on the training set', this is not implied by Th4.1. You simply expect from any good algorithm to perform well on data similar to what it was trained on.''

**A1**: Thanks for pointing out this potentially confusing probelm. The remark after Th4.1 is used for highlighting the conclusion of Th4.1. Following your valuable suggestion, we have revised the paper as follows: Theorem 4.1 implies that three terms are related to the generalization contribution. The first and second terms are intuitive, showing that the generalization contribution of a distribution $\mathcal{D}\_m$ is expected to be large on and similar to a training dataset $\tilde {\mathcal{D}}\_{m}$. The last term is also intuitive, which implies that promoting the generalization performance requires constructing similar conditional distributions.


**Q2**: '' Also in theorem 4.1, I would expect the learning algorithm to depend on $\mathcal{\tilde{D}}\_{m}$ not on $\mathcal{D}\_{m}$, because if I understood correctly, you will be training on the sampled distribution $\mathcal{\tilde{D}}\_{m}$.''

**A2**: Unfortunately, it might be a misunderstanding of our work that leads to the comments. We train models using sampled distributions, but we use both the training distribution and the training dataset in our theorem and proof. Similar approaches can be found in [1, 2].


**Q3**: "One last thing is in the definition of the worst-case margin, you should explain why this would be a good measure of generalization."

**A3**: Thanks for your constructive suggestion. We have revised the paper following the suggestion. The Definition 1, worst-case margin, measures the expected largest distance between the data $x$ with label $y$ and the data $x'$ that is mis-classified by the model $\rho$. Thus, smaller margin means higher possibility to mis-classify the data $x$. If the margin is equal to zero, the model mis-classifies almost all samples of the given distribution. 


**Q4**: ''the theory, in my opinion, is not very strong. (maybe because of the notation, but there is also $\mathcal{D}\_{m}$ and $\mathcal{\tilde{D}}\_{m}$''

**A4**: We have revised our notations and the remarks of Theorem 4.1. Please see **A1** and **A2**.


**Q5**: "you are also proving one thing and using a different thing in the experiments (the Gaussian approximation is not included in your theory)."

**A5** We believe it is the misunderstanding of our work that leads to the comments. Our theoretical results suggest that the distribution discrepancy is theoretically related to the generalization contribution. To mitigate the distribution discrepancy, we construct a globally share distribution, which is composed of several distributions that are approximated by Gaussian distributions.


**Q6**: ''you proclaim that your approach preserves privacy but you don't prove it.''

**A6**: Thanks for pointing out this potentially confusing problem. We have highlighted that the proposed methods do not introduce any **extra** privacy leakage than FedAvg, as the estimated Gaussian distribution contains the same statistics as BatchNormlization opperation [1]. Specifically, we estimate the mean and std of the features, which are also used in the BatchNorm Layers and will be transmitted in the FedAvg strategy. Thus, our method does not expose more information than FedAvg.


**Q7** "there is domain adaptation that has similar goals to the generalization part in your paper (training on one dataset and testing on another), so I think it will be useful for you to take a look at it."

**A7** Thanks for your suggestion. The challenge in FL is that local training on one client cannot use the data of other clients. Accordingly, we cannot apply the mentioned domain adaptation methodology in FL. Thus, we provide a theorem, i.e., Th4.1, to promote Fl with domain adaptation methodology. This is the reason why **Reviewer UyvV** recognizes the novelty of our theory and **Reviewer fFS7** thinks the theoretical result sound. Please see **A2**. 

[1] Batch normalization: Accelerating deep network training by reducing internal covariate shift. Ioffe et al. ICML, 2015.

From: ICLR.cc/2023/Conference/Paper4317/Authors
title: Response to Reviewer eaTq
comment: 
We would like to thank the reviewer for taking the time to review our work. We appreciate that you find the proposed question is interesting and the in-training metric is interesting and helpful. Please see our detailed feedback below.


**Q1**: ''Data heterogeneity and client drift might exist due to numerous issues: covariate shift, **model shift**, or both. It seems learning a shared model for features can handle covariate shift, but it **might fail** in the presence of model shift.''

**A1**: Would you mind providing **any references** for the mentioned 'model shift' in federated learning, as **we fail to find the concept in comprehensive surveys [1, 2, 3]**? Moreover, employing appropreate algorithms can contribute to covariate shift [4], conditional shift [5], and dataset shift [6].


**Q2**: ''... each client comes from a Gaussian distribution with mean $0$ and covariance $\Sigma$ (shared among all clients), but a different regressor $\theta_i$ ... I am not sure how drift among high level parts can be mitigated.''

**A2**: We have tried our best to understand the mentioned case. Unfortunately, no distribution shift happens in the case, as the distribution among clients are the same, i.e., $\mathcal{N}(0, \Sigma)$. Thus, no need is required to mitigate the shift. We look forward to your outstanding questions.

**Q3**: ''I might be mistaken mostly due to the fact that authors use the same notation $\mathcal{D}$ when referring to both data distribution (feature, label) pair and solely feature distribution and further clarification would be very helpful.''

**A3**: Thanks for your kind suggestion. We would like to highlight that $\mathcal D_m$ stands for the data distribution, $\varphi_{\theta}(x)$ represents the features that are calculated based on model input $x$, $(x,y)\sim \mathcal{D}_m$. We have revised the notations in the paper for the clarity.


**Q4**: ''It is not clear how the proposed algorithm can improve the generalization theoretically. While the decoupling idea is inspired from definition of conditional Wasserstein, but it would be nice to see ...''


**A4**: Thanks for your kind suggestion. As mentioned in your comments, the method is inspired by our theoretical analysis. That is, the generalization performance is related to the discrepancy between distributions. Accordingly, we propose to train models on distributions having small discrepancy by decoupling neural networks, i.e., our decoupling strategy.


**Q5**: "the presentation of the theoretical part can be improved by using better notation and adding more clarification"


**A5**: Thanks for your kind suggestions. We have revised the paper following your valuable suggestions.


**Q6** "in the statement of Thm 4, I originally thought $\Delta$ is missing but when I checked the proof I realized the authors abbreviated it"

**A6** Thank you for pointing this out. We abbreviated it in the proof for simplicity, so it does not affect the conclusion. To make the paper clear, we have revised the paper following your suggestions, i.e., replacing $W_d(\rho, \cdot)$ with $W_d(\rho(\theta + \Delta), \cdot)$.

**Reference:**

[1] Federated Learning: Challenges, Methods, and Future Directions. Li et al. 2019.

[2] Advances and Open Problems in Federated Learning, Kairouz et al. 2021.

[3] Federated Learning Challenges and Opportunities: An Outlook. Ding et al. 2022.

[4] Domain-adversarial training of neural networks. Ganin et al. 2016.

[5] Domain adaptation with conditional transferable components. Gong et al. 2016.

[6] Transfer feature learning with joint distribution adaptation. Long et al. 2013.

From: ICLR.cc/2023/Conference/Paper4317/Reviewer_eaTq
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: One of the key challenges in learning from private data sources in a collaborative manner as in federated learning is the data heterogeneity (non-IIDness) among data shards. This causes drift among local models which slows down the convergences or entails a poor generalization on local distributions. This issue is further exacerbated in the presence of infrequent synchronization which is widely used in resource-limited scenarios such as low communication bandwidth to reduce the number of communication rounds. This led to an enormous amount of efforts such as clustered learning, personalization, or explicit drift reduction using control variates or regularization (often dynamic) techniques. 

This paper focuses on improving the generalization by proposing an effective gradient correction method. To do so, the authors view local training as a generalization contribution problem, how local training of a specific client contributes to the generalization performance of the global model with respect to distribution of other clients. This can provide a mechanism to understand the contribution of each client at each communication round.  The authors quantify this by the conditional Wasserstein distance between clients’ distributions. The authors propose a decoupling idea to jointly learn  a low-level model (a feature extractor network) and a high-level one (head or classifier)to construct a shared identical distribution in the feature space. From a theoretical standpoint, the authors show that the effect of non-IID ness (gradient dissimilarity) on generalization and convergence can be decoupled based on two distributions: features and labels. Then it is shown that estimating and sharing feature distributions improves the generalization by  reducing gradient dissimilarity.

strength_and_weaknesses: To understand the contribution of each client at each communication round, the basic idea is to  generalize the margin based discrepancy to incorporate the gradient (pseudo gradient between two consecutive communication rounds for each client to be precise) as defined in Eq. 2. Lower bounding this error can guarantee the update will benefit the other clients if the drift is properly mitigated. The analysis leads to the algorithmic contribution of the paper where authors propose to learn/iteratively update a common feature distribution (Guassian in their case) among all clients.

The paper studies with an interesting question, and proposes an interesting in-training metric that can help to quantify and guide the algorithm to mitigate data heterogeneity issues. However, in my opinion the proposed algorithm and execution of ideas suffers from serious issues that prevents me from giving it a high score:

1. Data heterogeneity and client drift might exist due to numerous issues: covariate shift,  model shift, or both. It seems learning a shared model for features can handle covariate shift, but it might fail in the presence of model shift. For example, consider the case that data at each client comes from a Gaussian distribution with mean 0 and covariance \Sigma (shared among all clients), but a different regressor \theta_i. In this case I am not sure how drift among high level parts can be mitigated. Note that the original metric considers discrepancy in both covariates and labels. I might be mistaken mostly due to the fact that authors use the same notation \mathcal{D} when referring to both data distribution (feature, label) pair and solely feature distribution and further clarification would be very helpful. 

2. It is not clear how the proposed algorithm can improve the generalization theoretically.  While the decoupling idea is inspired from definition of conditional Wasserstein and decomposition of inter-client gradient dissimilarity into low and high level parts, but it would be nice to see how the iterative adaptation of these parameters and heuristics used to make the algorithm practical (such as Guassian) can indeed entail a good generalization. 

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and discusses the main idea in the context of existing works, but the presentation of the theoretical part can be improved by using better notation and adding more clarification. For example, in the statement of Thm 4, I originally thought \Delta is missing but when I checked the proof I realized the authors abbreviated it. Also, it would be much nicer to be specific when referring to different distributions (features, or pair of features  and label).

summary_of_the_review: Overall, the paper studies an interesting question, proposes a novel metric to quantify in-training generalization, but IMHO is poorly executed and  the quality of exposition seems to be low which makes me lean toward rejection.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4317/Reviewer_3s3d
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper addresses the problem of client drift in Federated Learning, a problem well-known in the literature to diminish the efficacy of FL methods. The authors propose to use a decoupling of features into low and high-level ones, they show that by grouping and sharing the low-level features among clients, gradient dissimilarity can be reduced, and thus performance gains can be obtained.
In practice, the low-level distribution is approximated by a Gaussian distribution to avoid high communication costs and make the algorithm feasible.

The authors also proclaim that this process preserves privacy.
strength_and_weaknesses: Strengths: the idea of decoupling features is not 100% new, but sharing the low-level features in the fashion proposed in the paper in addition to approximating the low-level feature distribution is a nice idea. Also, the experiments (I haven't gone through all the details of them) seem thorough, and most importantly they show that this method works better than other already-known methods.

Weaknesses: The writing of the paper is not clear sometimes, and some statements are just wrong, for example, the remark after Th 4.1 the authors say that Th4.1 implies "the generalization contribution of the training distribution is
expected to be large on the training set", this is not implied by Th4.1 you simply expect from any good algorithm to perform well on data similar to what it was trained on, the second implication is also not implied by Th4.1.  The notations are hard to follow (the Gaussian distribution mean is denoted by sigma and the variance by mu is an example, another example is $\zeta,y$ I would say $\zeta = x,y$ but you keep using both, so in the distance do you include the label or it is just the distance between inputs). Also in theorem 4.1, I  would expect the learning algorithm to depend on $\tilde{D}_m$ not on $D_m$, because if I understood correctly, you will be training on the sampled distribution $\tilde{D}_m$ (it would have been better to introduce this before the theorem and explain exactly what you are doing: we sample $\tilde{D}_m$ from $D_m$ and use it for training and so on...). One last thing is in the definition of the worst-case margin, you should explain why this would be a good measure of generalization (I would say you can make a link with SVMs, but you should explain it).

Also, the theory, in my opinion, is not very strong (maybe because of the notation, but there is also $\tilde{D}_m$ and $D_m$). you are also proving one thing and using a different thing in the experiments (the Gaussian approximation is not included in your theory). Also, you proclaim that your approach preserves privacy but you don't prove it (because there can still be information leakages)

One final point about the theory: there is domain adaptation that has similar goals to the generalization part in your paper (training on one dataset and testing on another), so I think it will be useful for you to take a look at it. 
clarity,_quality,_novelty_and_reproducibility: The paper is not very clear, the notations make it hard to follow, and some statements are not logically correct (see above).

The proposed training algorithm is to the best of my knowledge new.

The code is available and all the details (to the best of checking abilities) are included.

summary_of_the_review: The paper has some new and interesting ideas, but the way it was written makes it hard to follow, also the theory needs some adjustments.
 In general, the idea is interesting and I would encourage the authors to polish the paper, make it more clear, and resubmit. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4317/Reviewer_fFS7
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper analyzes the generalization contribution of local training in Federated learning. Their results show the key to promote generalization (conditional Wasserstein distance) and therefore this paper proposes decoupling the deep models for harnessing client drift and protecting privacy. Experimental results show FL with decoupled gradient dissimilarity is robust to data heterogeneity.


strength_and_weaknesses: 
Strength: This paper is well written and the topic of client drift is important and interesting. The theoretical result of local training generalization is sound and answers the question related to local training. 

Weakness: The communication cost should be reduced (e.g., quantizations) (although authors talked in the limitations.)

clarity,_quality,_novelty_and_reproducibility: The paper talks about interesting and important topics in FL with good quality and clarity. The idea is novel as far as I know. 

summary_of_the_review: Overall, I think that the studied local training generalization contribution in this paper is well-motivated. The idea behind the proposed algorithm is novel and effective.  Therefore, I recommend “accept”.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4317/Reviewer_UyvV
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the problem of data heterogeneity in federated learning. It suggests mathematical analysis based on the “worst-case margin” theory to measure the generalization contribution of clients’ FedAvg updates. Based on theoretical results authors propose decoupling the neural network into 2 parts: feature extractor and classifier in a desire to decrease gradient dissimilarity. Experimental evaluation shows the benefits of the proposed method on (synthetic) heterogeneous federated datasets.
strength_and_weaknesses: ## Strengths

1. The paper is trying to address a critical problem in federated learning.

2. Experimental evaluation is quite extensive and done accurately.

3. Theoretical concepts suggested for analysis seem novel in this setting.


## Weaknesses

1.  Problem origin: it is said that FedAvg suffers from client drift caused by heterogeneous data. The issue is that the so-called “client drift” is a consequence of the optimization method which relies on local training. In the distributed case without multiple local SGD steps, the variance can be eliminated via known control variate techniques, and even with local steps, it is not a problem from an optimization viewpoint. The current theory is not able to explain the bad performance of these methods in the setting of training non-convex deep neural networks. So, it is hard for me to agree with the original problem formulation.

2. The authors make an assumption that feature distributions can be approximated by a Gaussian referring to some previous works. Why is it the case? Was this assumption tested in practice e.g. via a statistical test? Is it the case only for a particular dataset and model architecture? Can the proposed method be combined with standard federated learning techniques like differential privacy and secure aggregation?

3. Experiments were conducted only on synthetically created small federated versions of centralized datasets. It is not clear whether the conclusions can be generalized to natural federated datasets like Federated Extended MNIST, Stackoverflow, etc.

4. Experimental results are obtained for single runs which limits their credibility. In addition, the curves in Figure 2 (b) are almost indistinguishable and the benefits of the proposed approach are not clear.

5. Important baselines like FedAdam/FedAdagrad are missing. They were shown to tackle well the natural heterogeneity in federated datasets. Can the method from this work be combined with adaptive optimizers?
clarity,_quality,_novelty_and_reproducibility: Section 4 is hard to follow and not clear. The ideas look original in the federated learning context. However, I think that this part needs a lot of clarification to make it ready for publication. Next, I try to list and unriddle some of my concerns.

- What do $\rho(\cdot)$ and $y_m$ look like mathematically? These objects (their dimension and the space they come from) were not properly defined. Is the guess that $y_m$ is a one-hot encoded vector right? Then, I have a question for definition 1: how $\rho(\theta; \xi’)$ can possibly be equal to $y$ in practice except in the case when the network is absolutely sure about the prediction and the scores for all other classes equal to zero. 

- Definitions 1, and 2 need much more detail and proper explanation. As far as I understand they are not standard in the community and I have not found these exact objects in the referenced papers, (Koltchinskii & Panchenko, 2002, Franceschi et al., 2018). 

- What does notation $\Delta : \mathbf{L}(\mathcal{D}_m)$ in equation (2) mean? What is $\mathcal{J}$ from definition 2? How $\hat{f}$ on (page 5) is defined? How is equation (6) obtained?


- The sentence on page 6
>“Intuitively, we prefer the client where the generalization contribution can be lower bounded.” 

is not clear to me. Why is it actually the case?

- The client drift is defined as $\frac{1}{\left|\mathcal{S}^{r}\right|} \sum_{i \in \mathcal{S}^{r}}\left\|\bar{\theta}-\theta_{i}\right\|$
 and referred to the work of Karimireddy et al., 2019. However, in the original SCAFFOLD paper, this term is defined differently.

- Conclusion on “reducing the high-level gradient dissimilarity is more important than the low-level” on page 9 is drawn from one experimental example (ResNet-18 trained on CIFAR-10) and needs more evidence.

- In Appendix Table 6 shows improved results for baseline methods when using momentum. It was previously shown (Hsu et al., 2019) that momentum can greatly help in the more synthetically heterogeneous case and parameters need to be adjusted for various heterogeneity levels. I would like to ask why the authors did not compare their approach to the stronger baselines.
summary_of_the_review: The main part of the paper lacks a lot of details which makes it very hard to understand clearly the theoretical contributions. That is why it is hard to judge their significance based only on text claims.

The empirical section lacks important baselines and relies on a very restrictive assumption of gaussian features distribution.

Although, if my concerns are addressed and questions are clarified I am open to raising my recommendation.


**Minor comments and typos**

- Is the problem setting limited only to classification?
- Typos: “varaince” -> variance on pages 6, 16
    - “Learnign” -> learning on page 20
- Looks like a missed word on page 7 in the sentence “We conjecture that CIFAR-100 datasets … , **leading to the results**”.
- SCAFFOLD work was already published and could be referred to as an ICML paper.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold