From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper studies the convergence rate of FedAvg under the interpolation setting. It proved the linear convergence rate of FedAvg for both centralized and decentralized settings.

The reviewers think that the results are incremental and not significant enough. Most of the reviewers do not support its publication at ICLR. The reviewer pointed out that a recent work has already provided the convergence rate of the distributed optimization algorithm in the interpolation regime. Moreover, the validation relying on simple benchmark datasets and models did not show promising empirical evidences, given the popularity of large datasets (ImageNet etc.) and models (VGG etc.). 

The PL-condition is a well-known condition that could help non-convex has the same convergence rate as strongly convex. SGD under interpolation regime would make SGD convergence rate the same as linear convergence rate of GD in the strong convex cases. Therefore, the result is not really surprise. Although the authors note that SGD cannot straightforward extend to FL in general, these two assumptions are strong. 

I agree with the reviewers that the paper is not ready for the publication at ICLR. Please take the comments and suggestions from the reviewers in their detail reviews to improve the manuscript for the future submission. 

justification_for_why_not_higher_score: - The PL-condition is a well-known condition that could help non-convex has the same convergence rate as strongly convex. SGD under interpolation regime would make SGD convergence rate the same as linear convergence rate of GD in the strong convex cases. Therefore, the result is not really surprise.
- The validation relying on simple benchmark datasets and models did not show promising empirical evidences, given the popularity of large datasets (ImageNet etc.) and models (VGG etc.). 
- The reviewer pointed out that a recent work has already provided the convergence rate of the distributed optimization algorithm in the interpolation regime.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3806/Reviewer_qmgS
title: Thanks for your response
comment: Thanks for the authors' response. 

However, a recent work (https://openreview.net/forum?id=mwIPkVDeFg) has already provided the convergence rate of the distributed optimization algorithm in the interpolation regime. Extending that algorithm to the periodic communication setting is trivial. Thus, the novelty of this work is quite limited.

From: ICLR.cc/2023/Conference/Paper3806/Reviewer_oTp9
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the convergence of FedAvg --- a widely used federated learning optimization algorithm --- in the interpolation regime, e.g., optimizing overparameterized neural networks. Under this interpolation regime, the authors provide *linear* convergence guarantees for FedAvg under both *server* and *decentralized* settings. The authors conduct experiments on several benchmark datasets that corroborate the theoretical results.
strength_and_weaknesses: Strength:

1. Given FedAvg is a widely used algorithm in FL and the interpolation regime is an important and realistic regime in FL, this paper makes important contributions to provide *linear* convergence rates of FedAvg in this setting.


Weaknesses:

1. The models in the experiment section do not satisfy the assumptions in Theorem 1 and Theorem 2. I agree it is interesting to investigate the convergence performance of FedAvg under the settings in Sec 4. I would like to see the empirical performance of FedAvg (Alg 1, Alg 2) for solving a problem where all the assumptions hold, e.g., overparameterized linear regression (or a more complex case), which could better corroborate the theoretical results and examine the tightness of the convergence analysis.


clarity,_quality,_novelty_and_reproducibility: (Clarity) This work is well presented. 

(Quality) High quality work, makes important contributions.

(Novelty) Novel.

(Reproducibility) Good.

===============================================================

Typo: FedAvg (Karimireddy et al., 2020b) (s) -> Scaffold (Karimireddy et al., 2020b) (s) in Table 1.
summary_of_the_review: This paper resolves an important problem in FL optimization (convergence rate of FedAvg under interpolation regime), I would recommend acceptance. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3806/Reviewer_qmgS
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper studied the convergence rate of FedAvg under the interpolation setting. In particular, it proved the linear convergence rate of FedAvg for both centralized and decentralized settings. The experimental results confirm the effectiveness of the proposed methods. 
strength_and_weaknesses: Pros:
1. This paper established the linear convergence rate of FedAvg under the interpolation setting.
2. The writing is clear. It is easy to follow.

Cons:
1. This paper claims that the convergence rate is established under the interpolation setting. But I didn't find where Assumption 1 is used in the paper. 
2. The algorithms employ the minibatch SGD. How does the batch size affect the convergence rate? Typically, a large batch size could reduce the variance. But there is no assumption regarding the variance. So how to relate the batch size to the convergence rate?
3. For the sublinear convergence rate, FedAvg can achieve the linear speedup regarding the number of devices. How about this linear convergence rate?
4. For the decentralized variant, it is not clear how the spectral gap affects the convergence rate.

======== after rebuttal========

a recent work (https://openreview.net/forum?id=mwIPkVDeFg) has already provided the convergence rate of the distributed optimization algorithm in the interpolation regime. Extending that algorithm to the periodic communication setting is trivial. Thus, the novelty of this work is quite limited. Considering this existing work, I have to lower down my rating. 
clarity,_quality,_novelty_and_reproducibility: Clarity: neutral
Quality: good
Novelty: incremental. It is not surprising to extend the single-machine results to FL. The authors did not discuss the challenges to do this extension. 
Reproducibility: neutral
summary_of_the_review: This paper studied the convergence rate of FedAvg under the interpolation setting. But the novelty is incremental and some parts are not clear. More discussions are needed for the theoretical results. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3806/Reviewer_A4jy
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper revisited federated and decentralized learning and showed the linear convergence rate for both of them when the objective loss satisfied the PL condition without generic assumptions such as bounded variance. Specifically, the authors leveraged the sample-wise smoothness of the local loss functions to capture the effect of heterogeneity. To validate the theory presented in this paper, the authors used a few benchmark datasets with simple models to show the model performance and claimed that the experimental results matched the theoretical conclusions well.
strength_and_weaknesses: Strength:

This paper presents some new analytical results to show that the FedAvg and Decentralized SGD are able to achieve linear convergence rate with only smoothness assumptions when the objective functions satisfy the PL condition. It seems like this work relaxed the generic assumptions used in most other existing work to show the convergence rate of these two sorts of algorithms. The paper is well written and easy to follow.

Weaknesses:

1. The only contribution of this work to me is to relax the assumption and give the linear convergence rate with PL conditions. It has been well know that when the objective loss satisfies PL conditions, its nature is very close to strong convexity, although it is not necessarily convex. Compared to general non-convex objectives, PL condition is a very strong condition. I understand the authors would like to show some interesting results for a class of objective losses. For sure, some objective losses can approximately meet this requirements. But focusing on more general non-convex functions can be more applicable. Also, the title is a bit confusing. Since base on that, it would give an impression that the conclusions drawn from this work apply to all non-convex functions.

2. As the authors mentioned that the proof techniques used in the decentralized SGD can be independent of interest to the community, I think the difference should be pointed in the main content since that can serve as a novelty. The authors cannot just ask the readers to figure it out themselves. 

3. The experimental results to me are not promising and convincing. First, why did the overparameteried model outperform the underparameterized one? There is no discussion in the paper. Is it because overparameterized model satisfied more PL condition? What caused the gaps? Second, it is not obvious to observe the linear convergence from the loss plots. Since they are similar to the plots where sublinear rate is there. Third, there is no baseline comparison in the plot. In Table 1, the authors have shown many existing methods. Why not just using at least a couple of them to show case the improvement of convergence rate? It is a bit difficult to convince readers that the theoretical analysis works. Even though some parameters need to be designed carefully in the experiments. That is also fine. Fourth, the model architectures are too small. Bigger models should be used to validate the theory. Fifth, for decentralized learning, different topologies are also critical. A simple ring topology is not sufficient. To summarize, the current experimental results didn't play a good role in validating the theory established in this work.

***************************Post-rebuttal******************************
I would like to thank the authors for their detailed responses, additional results, and revisions to the paper. After carefully reviewing the responses, I think most of my concerns have been addressed. Though I am willing to raise my score, the overall theoretical novelties to me are not that significant. The validation relying on simple benchmark datasets and models didn't show very promising empirical evidences, given the popularity of large datasets (ImageNet etc.) and models (VGG etc.).
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this work are good. While the theoretical contributions look decent, the experimental evidence cannot support the theory. More work is required to make the paper more technically solid and sound.
summary_of_the_review: This paper revisited federated and decentralized learning and showed the linear convergence rate for both of them when the objective loss satisfied the PL condition without generic assumptions such as bounded variance. The authors theoretically analyzed the convergence rate for these two learning setting. They also showed some simple numerical result to validate the theory. In terms of algorithm design, they have been existing. The only novelty is the new analysis. However, the experimental results failed supporting well the theoretical claims, which requires additional amount of work.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3806/Reviewer_PYE9
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies convergence rates of decentralized FedAvg for nonconvex problems. The paper considers an interpolation regime where there is a model with zero error on any example, which is the setting of overparameterization. The paper considers both a centralized setting and a decentralized setting, for both of which a linear convergence is established. Experimental results are given to verify the theoretical finding of the paper.
strength_and_weaknesses: **Strength**

Federated learning is a popular algorithm with various applications in addressing complex datasets. The paper develops a linear convergence rate for FedAvg in nonconvex problems. According to Table 1, it seems that this is the linear convergence of FedAvg in nonconvex problems.

The results are derived without bounded variance and bounded heterogeneity conditions.

**Weakness**

1. In both Theorem 1 and Theorem 2, there are complicated constraints on the step size. It seems that step size under this assumption would not be used in practice. Therefore, there is a gap between theory and practice.

2. The communication rounds are of the order $\tilde{O}(T^{3/2}/N)$ in the centralized setting and $\tilde{O}(T^2/N)$ in the decentralized setting. According to these results, we require more communication rounds as we increase $T$. Therefore, the best choice of $T$ becomes $T=1$. In this case, the algorithm becomes SGD and decentralized SGD, respectively. Therefore, the theoretical analysis does not show an advantage of performing local SGD updates in local nodes. In Table 1, there are several convergence rates $O(1/T)$ (ignoring other factors), which show the benefit of local SGD update. The authors should give some discussions on the effect of $T$ here.

clarity,_quality,_novelty_and_reproducibility: It is not clear to me the novelty of theoretical analysis. For example, in the literature there is some existing work on the linear convergence of SGD in the interpolation setting under PL condition without bounded variance assumptions (Bassily et al 2018). There is also existing work on the convergence of FedAvg on both centralized and decentralized settings. It seems that the technique in the paper builds the existing techniques. I would suggest the authors explain the novelty in the analysis.

The paper is clearly written. There are some typo as follows

In Algorithm 1 and Algorithm 2: there is $S_k\subset B_k^{r,t}$ . However, it seems that $S_k$ is not used in the algorithm

Page 7: "his procedure" should be "this procedure"

Page 8: "result in 1" should be "result in Theorem 1"

Page 8: "functions non-convex functions"

Appendix C.1: "to prove 1" should be "to prove Theorem 1"

In the last second equation of the proof of Lemma 2: $\Phi_k(\underline{w}^{r,\tau-1})$ should be $\Phi_k(w_k^{r,\tau-1})$?
summary_of_the_review: The paper shows a linear convergence of FedAvg under interpolation and PL setting. However, the analysis does not show the benefit of local SGD update in each local node. Furthermore, it is not clear to me the contribution from the technical side.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold