Paper: Which mode is better for federated learning? Centralized or Decentralized (ICLR-2024)

Reviewer comment (Novelty):
Many related works are missing, and a more thorough literature review is necessary.

Author response:
Thank you for pointing out this and we are sorry for causing you misunderstanding. We do not attribute the DFedAvg framework solely to any single work because it has been studied for a long time. [3] summarizes and provides an optimization convergence on the decentralized heterogeneous dataset in the FL framework which is explicitly targeted at DFL scenarios under the general assumptions widely used in the current FL community.

The mentioned paper [2] has been cited in our paper (page 3 paragraph "Decentralized federated learning"). The mentioned paper [1] mainly focuses on the optimization convergence of local-update-SGD under different paradigms. Our paper mainly focuses on the generalization difference between CFL and DFL. We will add [1] to the related work to further summarize the development of the decentralized approaches.


[1] Cooperative SGD: A Unified Framework for the Design and Analysis of Local-update SGD Algorithms (JMLR2021)

[2] A Unified Theory of Decentralized sgd with Changing Topology and Local Updates (ICML2020)

[3] Decentralized Federated Averaging (TPAMI)

Reviewer comment (Novelty):
Several related studies are not included, necessitating a more comprehensive literature review.

Author response:
Thank you for pointing out this and we are sorry for causing you misunderstanding. We do not attribute the DFedAvg framework solely to any single work because it has been studied for a long time. [3] summarizes and provides an optimization convergence on the decentralized heterogeneous dataset in the FL framework which is explicitly targeted at DFL scenarios under the general assumptions widely used in the current FL community.

The mentioned paper [2] has been cited in our paper (page 3 paragraph "Decentralized federated learning"). The mentioned paper [1] mainly focuses on the optimization convergence of local-update-SGD under different paradigms. Our paper mainly focuses on the generalization difference between CFL and DFL. We will add [1] to the related work to further summarize the development of the decentralized approaches.


[1] Cooperative SGD: A Unified Framework for the Design and Analysis of Local-update SGD Algorithms (JMLR2021)

[2] A Unified Theory of Decentralized sgd with Changing Topology and Local Updates (ICML2020)

[3] Decentralized Federated Averaging (TPAMI)

Reviewer comment (Presentation):
Table 1 is presented without an explanation of its significance and implications in the introduction, making it necessary for readers to go through the entire paper to understand the table.

Author response:
Thank you for pointing out this and we are very sorry for the inconvenience with Table 1. We will add a paragraph in the introduction section to describe them in detail.

Actually, Table 1 summarizes the excess risk of FedAvg and DFedAvg. From the excess risk perspective, the best partial participation ratio in CFL is $\frac{n}{m}=\mathcal{O}(m^{-\frac{\mu L}{1+2\mu L}})$ and the best topology in DFL satisfies $\kappa_\lambda=0$. Under their best selections respectively, the generalization in CFL achieves $\mathcal{O}(m^{-\frac{1+\mu L}{1+2\mu L}})$ and the generalization in DFL achieves $\mathcal{O}(m^{-\frac{1}{1+\mu L}})$.

Reviewer comment (Novelty):
This work seems to lack a thorough review of related literature. For example, attributing the DFedAvg framework solely to Sun et al. (2022) is inaccurate. Multiple studies have examined the scenario of local updates in decentralized training, including:  
- Wang, Jianyu, and Gauri Joshi. "Cooperative SGD: A unified framework for the design and analysis of local-update SGD algorithms." The Journal of Machine Learning Research 22.1 (2021): 9709-9758.  
- Koloskova, Anastasia, et al. "A unified theory of decentralized sgd with changing topology and local updates." International Conference on Machine Learning. PMLR, 2020.

Author response:
Thank you for pointing out this and we are sorry for causing you misunderstanding. We do not attribute the DFedAvg framework solely to any single work because it has been studied for a long time. [3] summarizes and provides an optimization convergence on the decentralized heterogeneous dataset in the FL framework which is explicitly targeted at DFL scenarios under the general assumptions widely used in the current FL community.

The mentioned paper [2] has been cited in our paper (page 3 paragraph "Decentralized federated learning"). The mentioned paper [1] mainly focuses on the optimization convergence of local-update-SGD under different paradigms. Our paper mainly focuses on the generalization difference between CFL and DFL. We will add [1] to the related work to further summarize the development of the decentralized approaches.


[1] Cooperative SGD: A Unified Framework for the Design and Analysis of Local-update SGD Algorithms (JMLR2021)

[2] A Unified Theory of Decentralized sgd with Changing Topology and Local Updates (ICML2020)

[3] Decentralized Federated Averaging (TPAMI)

Reviewer comment (Novelty):
Numerous studies have explored DSGD, its model consistency, and have proposed corrective methods, including:  
- Yuan, Kun, et al. "On the influence of bias-correction on distributed stochastic optimization." IEEE Transactions on Signal Processing 68 (2020): 4352-4367.  
- Shi, Wei, et al. "Extra: An exact first-order algorithm for decentralized consensus optimization." SIAM Journal on Optimization 25.2 (2015): 944-966.  
- Alghunaim, Sulaiman A., and Kun Yuan. "A unified and refined convergence analysis for non-convex decentralized learning." IEEE Transactions on Signal Processing 70 (2022): 3264-3279.  
- Mishchenko, Konstantin, et al. "Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally!." International Conference on Machine Learning. PMLR, 2022.  
- Nguyen, Edward Duc Hien, et al. "On the performance of gradient tracking with local updates." arXiv preprint arXiv:2210.04757 (2022).  
- Alghunaim, Sulaiman A. "Local Exact-Diffusion for Decentralized Optimization and Learning." arXiv preprint arXiv:2302.00620 (2023).

Author response:
I am very grateful to the reviewer for mentioning several extensions of decentralized approaches. 

First of all, I would like to clarify the main target of our paper. We do not focus on proposing a new decentralized method. Our study mainly attempts to answer a question mentioned in the introduction: "Which framework is better for federated learning? Centralized or decentralized?". Several studies have learned the convergence rate in decentralized approaches could achieve a comparable optimization convergence rate as the centralized approaches. However, its performance in empirical studies is always not as good as we expected. This motivated us to explore the generalization performance. Therefore, we focus the related work targeted more on the generalization and stability analysis in decentralized methods. But all mentioned papers [1,2,3,4,5,6] focus more on the convergence analysis in their papers.

In order to dispel the reviewer's concerns, we will add a new paragraph in related work to introduce the development of both the centralized and the decentralized approaches including all missed work mentioned in the rebuttal. Thank you again for pointing out this issue.


[1] On the influence of bias-correction on distributed stochastic optimization

[2] Extra: An exact first-order algorithm for decentralized consensus optimization

[3] A unified and refined convergence analysis for non-convex decentralized learning

[4] Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally!

[5] On the performance of gradient tracking with local updates

[6] Local Exact-Diffusion for Decentralized Optimization and Learning

Reviewer comment (Novelty):
How does the generalization analysis presented in this study differ from that of DSGD without local updates as detailed in:  
- Taheri, Hossein, and Christos Thrampoulidis. "On generalization of decentralized learning with separable data." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.  
- Bars, Batiste Le, Aurélien Bellet, and Marc Tommasi. "Improved Stability and Generalization Analysis of the Decentralized SGD Algorithm." arXiv preprint arXiv:2306.02939 (2023).  
- Zhu, Tongtian, et al. "Topology-aware generalization of decentralized sgd." International Conference on Machine Learning. PMLR, 2022.

Author response:
We summarize the difference as follows.

(1) All analysis in [1] only is suitable for the **convex** function (Assumption 2 in their paper.) In their conclusions, they mentioned "We believe our work opens several directions, with perhaps the most exciting one being the analysis of non-convex objectives.". And, Our analysis targets the smooth and **non-convex** objectives.

(2) To compare to [2], we provide a table in the following.
|       |   Lipschitz Function Required   | Related to Spectrum |
|:-----:|:-----:|:-----:|
| [2] | $\sqrt{}$ | $\times$ | 
| our paper | $\times$ | $\sqrt{}$ |

First of all, in our proofs, we remove the strong assumption of the Lipschitz function (which indicates the gradient must be upper bounded) in the stability analysis. Bounded gradients force each iteration to be still upper bounded. However, many previous studies learn that this assumption may not always hold, especially in current deep learning models. Our proofs indicate that even the iterations are not necessarily bounded, the final stability is still bounded.

Furthermore, in their Theorem 3.1, 3.2, and 3.3, the stability bound **is not related to the spectral gap or information of topology**. One of the main targets of the research in decentralized methods is to understand the impact of topology on the final performance. Our paper provides the stability analysis with the topology coefficient, which is dominated by $\mathcal{O}(\left(\frac{1+6\sqrt{m}\kappa_\lambda}{m}\right)^\frac{1}{1+\mu L})$, where $m$ is the number of the total clients and $\kappa_\lambda$ is the coefficient related to the spectral gap $\lambda$. Furthermore, we derive the dominant factors of the partial participation in FL to be $\mathcal{O}(\frac{n^\frac{\mu L}{1+\mu L}}{m})$. These theorems in our paper can fairly compare the performance differences between CFL and DFL under the same assumptions.

(3) [3] not only must rely on the strong assumption of the Lipschitz function, but also require a additional assumption: "the weight differences at the $t$-th iteration are multivariate normally distributed, $w_{1,k}^t-w_{2,k}^t\sim\mathcal{N}(\mu_k^t,\sigma_{t,k}^tI_d)$" (Assumption A.4 in their paper). This assumption is too strong for the stability analysis because it forces the parameters of all layers to be normalized bounded. Compared with their work, we do not require any extra assumptions on the stability analysis, which could be more general in the current deep learning community.

[1] On generalization of decentralized learning with separable data

[2] Improved Stability and Generalization Analysis of the Decentralized SGD Algorithm

[3] Topology-aware generalization of decentralized sgd

Reviewer comment (Theory):
The bounds presented in theorems 1 and 2 demand a comprehensive explanation. A clearer breakdown of the influence of each parameter is necessary. Specifically, the intuition behind equation (9) where the bound seems to deteriorate with increasing values of T and K remains elusive (to readers not quite familiar with generalization analysis).

Author response:
Thank you for pointing out this issue and we will add more details to demonstrate our theorem 1 and 2. Actually, the most important part is the active coefficient part (related to $n$, $m$, and $\kappa_\lambda$), which reflects the essential difference between CFL and DFL. Our five main conclusions are also derived from the derivation according to this part. Sorry for the impact on reading, we will add some texts to explain the results of stability analysis, which is proposed in [1].

[1] Train faster, generalize better: Stability of stochastic gradient descent (ICML2016)

Reviewer comment (Evaluation):
In the scenario of a fully connected network, DFedAvg reduces to Fed-Avg. However, table 1 indicates disparate generalization errors for both under this condition. Could you provide clarification on this discrepancy?

Author response:
Sorry for the misunderstanding here. In the scenario of a fully connected network, DFedAvg reduces to FedAvg. From our **analysis of generalization**, we can see:
|   mode    |   CFL (Eq.(9) in our paper)   |  DFL (Eq.(11) in our paper) |
|:-----:|:-----:|:-----:|
| general generalization | $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{n^{\frac{\mu L}{1+\mu L}}}{m}\right)\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ | $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{1+6\sqrt{m}\kappa_\lambda}{m}\right)^\frac{1}{1+\mu L}\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ |
| when fully connected | $n=m$ |  $\kappa_\lambda=0$ |
|generalization under fully connected| $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{1}{m}\right)^\frac{1}{1+\mu L}\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ | $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{1}{m}\right)^\frac{1}{1+\mu L}\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ |

Therefore, the full-connection topology in DFL shows the same generalization performance as all participantion in CFL.

**What is the meaning of Table 1?**

**Table.1 considers the joint performance of excess risk, which is, the sum of optimization errors and generalization errors.** From the perspective of excess risk, we provide the best selection of active ratio ($n$ in CFL) and best topology (in DFL) respectively in Table 1. Under the best selections of each framework, we also show their generalization performance. Table 1 indicates that in FedAvg, from the excess risk perspective, we do not need all clients to participate in per communication round ($n< m$). Only when $n=m$, FedAvg is equal to the DFedAvg under full-connection topology.

Reviewer comment (Novelty):
There are many existing works studying the generalization of centralized and decentralized federated learning algorithms. However, this paper missed a lot of them. Then, it is not clear what new contributions this paper has. e.g., is the bound of this paper comparable with existing ones?  

[1] https://openreview.net/pdf?id=-EHqoysUYLx  

[2] https://arxiv.org/abs/2306.02939

Author response:
Thank you very much for your review and valuable comments on our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring. We are also very honored to share some of our understandings with you.

## 1. About the question of "it is not clear what new contributions this paper compared with [2,3]":
(1) **Our contributions**: Firstly thank you for pointing out these related works. There are really many independent studies based on centralized or decentralized approaches. However, to our best knowledge, there is no complete conclusion to indicate which one of them performs better in the FL framework. Since [1] learns D-PSGD could achieve comparable convergence as C-PSGD, it is generally believed that the decentralized methods will achieve the same performance as the centralized methods. However, decentralized experimental performance has always been poor. Therefore, we want to learn their stability under the same general assumptions via the same analysis framework to fairly compare their generalization performance. Our paper completes the further exploration of the comparison within these two modes in FL. Meanwhile, we also find some novel and interesting conclusions based on the stability and excess risk analysis. We summarize them in the following table.

| Main Conclusions from our Proofs |
| :-----|
| 1. In CFL, from the excess risk perspective, the best training mode per round is partial participation.  |
| 2. In CFL, when the local interval increases, we should decrease the active ratios per round. |
| 3. In DFL, a better topology must satisfy the spectrum coefficient is small enough. |
| 4. In DFL, to avoid performance collapse, the spectrum coefficient must be lower than $\mathcal{O}(\sqrt{m})$. |
| 5. From a stability perspective, DFL always generalizes worse than CFL. |
1,2: page 6; 3,4: page 7; 5: Table.3


(2) Let us summarize a simple comparison between our work and the mentioned work [2,3]. Though they focus on the exploration of centralized or decentralized stability, they are still largely different from our work.

|       |   Lipschitz Function Required   |  Additional Assumptions Required | Comparison between CFL and DFL |
|:-----:|:-----:|:-----:|:-----:|
| [2] | $\sqrt{}$ | $\sqrt{}$ |  $\times$ |
| [3] | $\sqrt{}$ |  $\times$ | $\times$ |
| our paper | $\times$ | $\times$ | $\sqrt{}$ |

First of all, in our proofs, we remove the strong assumption of the Lipschitz function (which indicates the gradient must be upper bounded) in the stability analysis. Bounded gradients force each iteration to be still upper bounded. However, many previous studies learn that this assumption may not always hold [4,5,6,7], especially in current deep learning models. Our proofs indicate that even the iterations are not necessarily bounded, the final stability is still bounded.

[2] introduces several additional assumptions to analyze the unparticipated clients without bounded losses, but this analysis does not guide us in understanding whether CFL or DFL is better. [3] provides an analysis of D-SGD, but in their Theorem 3.3, the stability bound is not related to the spectral gap or information of topology. One of the main targets of the research in decentralized methods is to understand the impact of topology on the final performance. Our paper provides the stability analysis with the topology coefficient, which is dominated by $\mathcal{O}(\frac{1+6\sqrt{m}\kappa_\lambda}{m})$, where $m$ is the number of the total clients and $\kappa_\lambda$ is the coefficient related to the spectral gap $\lambda$. Furthermore, we derive the dominant factors of the partial participation in FL to be $\mathcal{O}(\frac {n^\frac{\mu L}{1+\mu L}}{m})$. These two theorems in our paper can fairly compare the performance differences between CFL and DFL under the same conditions.


[1] Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic
Gradient Descent (NIPS2017)

[2] Generalization Bounds for Federated Learning: Fast Rates, Unparticipation Clients and Unbounded Losses (ICLR2023)

[3] Improved Stability and Generalization Analysis of the Decentralized SGD Algorithm (arXiv2023)

[4] The Lipschitz Constant of Self-attention (ICML2021)

[5] Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness (ICML2021)

[6] Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradients: Convergence, Divergence and Limitations of Its Continuous Approximation. (arXiv)

[7] Beyond Uniform Lipschitz Condition in Differentially Private Optimization (ICML2023)

Reviewer comment (Evaluation):
This paper does not show how the heterogeneity affects the generalization error.

Author response:
## 2. About the question of "This paper does not show how the heterogeneity affects the generalization error.":
Thank you for pointing out this question and it may be a misunderstanding on the stability analysis.

(1) The work [1] mentioned by the reviewer also does not show the heterogeneity impacts. Furthermore, many existing generalization studies in FL also do not involve the heterogeneity impacts [2,3,4].

(2) The stability analysis mainly explores the potential model differences when it is trained on two different datasets. The heterogeneity measures the quality and similarity of the local dataset, which mainly affects the optimization convergence, that is, the efficiency to achieve the optimal state. While stability does not care about the final state. In other words, even if the model converges very poorly on a very poor dataset, it may still maintain good stability because the stability is largely decided by the training algorithms. And the final test accuracy is decided by both the optimization errors and stability errors. Therefore, the quality of the dataset is not important for the stability analysis. That's why most of the previous studies of the generalization analysis in FL do not involve heterogeneity.


[1] Generalization Bounds for Federated Learning: Fast Rates, Unparticipation Clients and Unbounded Losses (ICLR2023)

[2] A Robust Federated Learning: The Case of Affine Distribution Shifts (NIPS2020)

[3] Generalized Federated Learning via Sharpness Aware Minimization (ICML2022)

[4] Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization (NIPS2023)

Reviewer comment (Theory):
When the communication graph is fully connected, DFedAvg becomes FedAvg. But the generalization error of this paper does not have this relationship.

Author response:
## 3. About the question of "When the communication graph is fully connected, DFedAvg becomes FedAvg. But the generalization error of this paper does not have this relationship.":
Thank you for pointing out this and this is a misunderstanding. We will provide a table to introduce their relationships in the following.

|   mode    |   CFL (Eq.(9) in our paper)   |  DFL (Eq.(11) in our paper) |
|:-----:|:-----:|:-----:|
| general generalization | $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{n^{\frac{\mu L}{1+\mu L}}}{m}\right)\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ | $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{1+6\sqrt{m}\kappa_\lambda}{m}\right)^\frac{1}{1+\mu L}\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ |
| when fully connected | $n=m$ |  $\kappa_\lambda=0$ |
|generalization under fully connected| $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{1}{m}\right)^\frac{1}{1+\mu L}\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ | $\frac{4}{S}\left(\frac{\sigma_lG}{L}\right)^{\frac{1}{1+\mu L}}\left(\frac{1}{m}\right)^\frac{1}{1+\mu L}\left(UTK\right)^{\frac{\mu L}{1+\mu L}}$ |

When the topology is fully connected in DFL, all clients will aggregate the model after the local training. Therefore, it is equivalent to all participation in CFL which indicates $n=m$. Their generalization is the same. Hope this table can avoid this misunderstanding.

Reviewer comment (Novelty):
There are many existing works studying the generalization of centralized and decentralized federated learning algorithms. However, this paper missed a lot of them. Then, it is not clear what new contributions this paper has. e.g., is the bound of this paper comparable with existing ones?  

[1] https://openreview.net/pdf?id=-EHqoysUYLx  

[2] https://arxiv.org/abs/2306.02939

Author response:
Thank you very much for your review and valuable comments on our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring. We are also very honored to share some of our understandings with you.

## 1. About the question of "it is not clear what new contributions this paper compared with [2,3]":
(1) **Our contributions**: Firstly thank you for pointing out these related works. There are really many independent studies based on centralized or decentralized approaches. However, to our best knowledge, there is no complete conclusion to indicate which one of them performs better in the FL framework. Since [1] learns D-PSGD could achieve comparable convergence as C-PSGD, it is generally believed that the decentralized methods will achieve the same performance as the centralized methods. However, decentralized experimental performance has always been poor. Therefore, we want to learn their stability under the same general assumptions via the same analysis framework to fairly compare their generalization performance. Our paper completes the further exploration of the comparison within these two modes in FL. Meanwhile, we also find some novel and interesting conclusions based on the stability and excess risk analysis. We summarize them in the following table.

| Main Conclusions from our Proofs |
| :-----|
| 1. In CFL, from the excess risk perspective, the best training mode per round is partial participation.  |
| 2. In CFL, when the local interval increases, we should decrease the active ratios per round. |
| 3. In DFL, a better topology must satisfy the spectrum coefficient is small enough. |
| 4. In DFL, to avoid performance collapse, the spectrum coefficient must be lower than $\mathcal{O}(\sqrt{m})$. |
| 5. From a stability perspective, DFL always generalizes worse than CFL. |
1,2: page 6; 3,4: page 7; 5: Table.3


(2) Let us summarize a simple comparison between our work and the mentioned work [2,3]. Though they focus on the exploration of centralized or decentralized stability, they are still largely different from our work.

|       |   Lipschitz Function Required   |  Additional Assumptions Required | Comparison between CFL and DFL |
|:-----:|:-----:|:-----:|:-----:|
| [2] | $\sqrt{}$ | $\sqrt{}$ |  $\times$ |
| [3] | $\sqrt{}$ |  $\times$ | $\times$ |
| our paper | $\times$ | $\times$ | $\sqrt{}$ |

First of all, in our proofs, we remove the strong assumption of the Lipschitz function (which indicates the gradient must be upper bounded) in the stability analysis. Bounded gradients force each iteration to be still upper bounded. However, many previous studies learn that this assumption may not always hold [4,5,6,7], especially in current deep learning models. Our proofs indicate that even the iterations are not necessarily bounded, the final stability is still bounded.

[2] introduces several additional assumptions to analyze the unparticipated clients without bounded losses, but this analysis does not guide us in understanding whether CFL or DFL is better. [3] provides an analysis of D-SGD, but in their Theorem 3.3, the stability bound is not related to the spectral gap or information of topology. One of the main targets of the research in decentralized methods is to understand the impact of topology on the final performance. Our paper provides the stability analysis with the topology coefficient, which is dominated by $\mathcal{O}(\frac{1+6\sqrt{m}\kappa_\lambda}{m})$, where $m$ is the number of the total clients and $\kappa_\lambda$ is the coefficient related to the spectral gap $\lambda$. Furthermore, we derive the dominant factors of the partial participation in FL to be $\mathcal{O}(\frac {n^\frac{\mu L}{1+\mu L}}{m})$. These two theorems in our paper can fairly compare the performance differences between CFL and DFL under the same conditions.


[1] Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic
Gradient Descent (NIPS2017)

[2] Generalization Bounds for Federated Learning: Fast Rates, Unparticipation Clients and Unbounded Losses (ICLR2023)

[3] Improved Stability and Generalization Analysis of the Decentralized SGD Algorithm (arXiv2023)

[4] The Lipschitz Constant of Self-attention (ICML2021)

[5] Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness (ICML2021)

[6] Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradients: Convergence, Divergence and Limitations of Its Continuous Approximation. (arXiv)

[7] Beyond Uniform Lipschitz Condition in Differentially Private Optimization (ICML2023)

Reviewer comment (Evaluation):
This paper does not show how the heterogeneity affects the generalization error.

Author response:
## 2. About the question of "This paper does not show how the heterogeneity affects the generalization error.":
Thank you for pointing out this question and it may be a misunderstanding on the stability analysis.

(1) The work [1] mentioned by the reviewer also does not show the heterogeneity impacts. Furthermore, many existing generalization studies in FL also do not involve the heterogeneity impacts [2,3,4].

(2) The stability analysis mainly explores the potential model differences when it is trained on two different datasets. The heterogeneity measures the quality and similarity of the local dataset, which mainly affects the optimization convergence, that is, the efficiency to achieve the optimal state. While stability does not care about the final state. In other words, even if the model converges very poorly on a very poor dataset, it may still maintain good stability because the stability is largely decided by the training algorithms. And the final test accuracy is decided by both the optimization errors and stability errors. Therefore, the quality of the dataset is not important for the stability analysis. That's why most of the previous studies of the generalization analysis in FL do not involve heterogeneity.


[1] Generalization Bounds for Federated Learning: Fast Rates, Unparticipation Clients and Unbounded Losses (ICLR2023)

[2] A Robust Federated Learning: The Case of Affine Distribution Shifts (NIPS2020)

[3] Generalized Federated Learning via Sharpness Aware Minimization (ICML2022)

[4] Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization (NIPS2023)

Reviewer comment (Theory):
Authors claimed the study focuses on nonconvex functions (e.g., Table 1), but to determine the final excess risk (Corollary 1.2 and 2.2), the additional PL-condition is imposed, so all in all the study and follow-up discussion comparing CFL and DFL in fact focuses on PL functions only, which restricts the scope of the research. Even though I may agree PL is a bit common in nonconvex literature, but all in all it is still pretty strong (similar to strong convexity though), not to mention the mismatch with deep learning models you used in the experiment part. I think the authors should clearly clarify this point in the theory part, and present it at the beginning of the paper.

Author response:
## 1. About the question of "All discussions must rely on PL-condition:"
(1) We first would like to clarify the misunderstanding about the PL-condition and our paper. Our main theoretical contribution is to fairly compare the generalization performance between CFL and DFL via the same analysis framework under the same assumptions. Though we mentioned comparing their excess risk, however, most of our conclusions provided in our paper are based on the stability comparison, **which never adopts the PL-condition.** So we think the comment "so all in all the study and follow-up discussion comparing CFL and DFL in fact focuses on PL functions only" is very inaccurate. Here we make a table in the following to summarize our main novel conclusions and the conditions required.

| Main Conclusions from our Proofs | Use PL-condition or Not |
| :-----| :----: |
| 1. In CFL, from the excess risk perspective, the best training mode per round is partial participation.  | $\sqrt{}$ |
| 2. In CFL, when the local interval increases, we should decrease the active ratios per round. | $\times$ |
| 3. In DFL, a better topology must satisfy the spectrum coefficient is small enough. | $\times$ |
| 4. In DFL, to avoid performance collapse, the spectrum coefficient must be lower than $\mathcal{O}(\sqrt{m})$. | $\times$ |
| 5. From a stability perspective, DFL always generalizes worse than CFL. | $\times$ |
1,2: page 6; 3,4: page 7; 5: Table.3

From the above table, we can clearly see that in the five main conclusions in our paper, four of them do not rely on the PL-condition. Therefore, we object to the statement that our discussion is limited to the PL-condition only.


(2) In the excess risk, the test error is bounded by the sum of optimization error and generalization error, which is $\varepsilon_e=\varepsilon_{gen}+\varepsilon_{opt}=\mathbb{E}[F(w^T)-f(w^T)] + \mathbb{E}[f(w^T)-f(w^\star)]$. Current non-convex analysis only provides the $\frac{1}{T}\sum_t\Vert\nabla f(w^t)\Vert^2$ term is upper bounded. Therefore, in many studies of analyzing the excess risk, to approximate the bound of $\varepsilon_{opt}=\mathbb{E}[f(w^T)-f(w^\star)]$ term, many additional assumptions are adopted. We follow the studies [1,2] to select the PL-condition to provide the $\varepsilon_{opt}$ term.

[1] Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond (NIPS2021)

[2] Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization (NIPS2023)

Reviewer comment (Novelty):
In terms of the theoretical proof, as far as I can see, the proof heavily relies on those in Hardt et al., 2016, with some modification in the later recursion to fit the finite-sum structure. Even though the story of the proof is interesting, the theoretical novelty is restricted a bit regarding the resemblance.

Author response:
## 3. About the question of "the proof heavily relies on those in [2] with some modification in the later recursion":
We disagree with this point. Uniform stability analysis in deep learning is proposed by [1,2] and has been widely used in a lot of previous studies to analyze various algorithms [3,4,5,6]. As a good analysis framework, many classic conclusions in generalization are obtained based on this analysis. Furthermore, we must emphasize our additional contributions in this work:

(1) To our best knowledge, this is the first work to prove uniform stability without adopting the assumption of Lipschitz continuous function (bounded gradient assumption). Many previous studies have learned the Lipschitz function assumption is too strong in deep models which may not always hold. Our work further refines the uniform stability analysis.

(2) We focus on fairly understanding the difference in generalization performance between CFL and DFL with the help of a uniform stability framework. This is not a simple incremental work. We not only prove that the generalization performance of DFL is always worse than CFL but also find a novel conclusion, that is, in the DFL scenarios, the topology still has a theoretical minimum requirement to avoid the collapse (summarized in the Table in Answer.1). We also conduct empirical studies to validate the conclusions proposed in our paper. Both theoretical analysis and experiments confirm our conclusions.

(3) Beyond [2], we analyze how the topology affects the generalization in DFL, and how active ratio affects the generalization in CFL. In our analysis, we can effectively compare the generalization efficiency of CFL and DFL and give strict theoretical differences in the stability and generalization performance.


[1] Stability of Randomized Learning Algorithms. (JMLR 2005)

[2] Train Faster, Generalize Better: Stability of Stochastic Gradient Descent (ICML2016)

[3] Uniform Stability for First-Order Empirical Risk Minimization (NIPS2018)

[4] Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent (ICML 2020)

[5] Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond (NIPS2021)

[6] Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization (NIPS2023)

Reviewer comment (Writing):
Several typos with $||f()-f()||$, which should be revised to absolute values for scalars.

Author response:
## 4. About the question of "some typos":
We will fix all $\Vert f() - f()\Vert$ to $\vert f() - f()\vert$ and add a notations table in Section 3 to summarize all marks in this paper. Thank you again for pointing out this issue.

Reviewer comment (Writing):
Some notations are not defined in the main body. For example, The matrix U in Theorem 1 is shown before it is formally defined in the Appendix. It would be kind of confusing to readers.

Author response:
## 1. About the question of "Some notations are not defined in the main body":
Thank you for pointing this and we are very sorry that some missing notation definitions have caused difficulty in reading and we will add a comprehensive notation table to clarify each symbol.

Reviewer comment (Theory):
The paper seems to require the sample gradient to be Lipschitz, according to Lemma 6. However, Assumption 1 requires that the full gradient is Lipschitz. Is this a typo?

Author response:
## 2. About the question of "The paper seems to require the sample gradient to be Lipschitz":
We are very grateful to the reviewer for checking our proof section and we will fix Assumption 1 to a sample gradient formation. Lemma 6 in our paper refers to Lemma 1.1 in [1] and the expansion of Lemma 3.6 in [2]. In the study of stability analysis, Lemma 6 in our paper is a very general recursive formula and many classical works adopt this property to upper bound the difference on the same data sample. 


[1] Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond (NIPS2021)

[2] Train faster, generalize better: Stability of stochastic gradient descent (ICML2016)