Paper: A New Theoretical Perspective on Data Heterogeneity in Federated Averaging (ICLR-2024)

Reviewer comment (Theory):
**Lack of Clarity in Technical Challenges and Limited Technical Contributions**: The authors do not sufficiently show the technical challenges associated with the convergence analysis based on the new assumption. The paper could benefit from a more detailed statement of challenges from the new Lipschitz assumption, especially in comparison to the conventional local Lipschitz assumption (assumption 1). This lack of clarity leaves room for misinterpretation and raises questions about the significance of the proposed approach. As I understand, the main difference in the proof seems to revolve around using this new assumption to bound the discrepancy between the aggregated gradients and the groundtruth gradients (eq 11). In contrast, other analyses have traditionally employed the local Lipschitz assumption (assumption 1). A clearer statement of how this change significantly enhances the convergence analysis would be beneficial.

Author response:
We thank Reviewer wiji for the questions and the suggestions. Our responses are as follows.

1. **About the technical novelty.** We agree with Reviewer wiji that the main technical novelty in the proof for Theorem 4.3 and 4.5 is about developing new techniques to incorporate Assumption 4.1 and 4.2. In addition, we also develop Proposition 5.1-5.3 and Theorem 5.5 to demonstrate the usefulness and the effectiveness of Assumption 4.1 and 4.2. We summarize our technical novelty as follows.

(1) We need to develop new techniques to incorporate Assumption 4.1 and 4.2. In the proof of Theorem 4.3 shown in the Appendix, we need to characterize the difference between local gradients. In the literature, this is done by applying the local Lipschitz constant in shown in Assumption 3.1 in the main paper. In our paper, since Assumption 3.1 is replaced by our newly introduced assumption 4.2, the proof techniques in the literature cannot be applied. It requires developing new proof techniques to use Assumption 4.2 as shown in the proof of Lemma A.1-A.3. For example, in Lemma A.1, due to the application of Assumption 4.1 and 4.2, we have to deal with a new term, the local gradient deviation $\\|\frac{1}{N}\sum_{i=1}^N \nabla F_i(\mathbf{x}_i) - \nabla F_j(\mathbf{x}_j) \\|^2$, which is not shown in other techniques. Another example is in the proof of Theorem 4.5. Due to that we only use the global Lipschitz gradient assumption, we have to derive a new method to bound and incorporate the sampling related term 

$$\mathbb{E}\\|\frac{1}{M}\sum_{i\in S_r}\sum_{k=0}^I \mathbf{g}_i(\mathbf{x}_r^{r,k}) \\|^2,$$

which can be seen from (A.27) to (A.35).

(2) In addition to using Assumption 4.2 to characterize the new convergence rate of FedAvg, we also validate this assumption from the theoretical perspective. We develop the proof for Proposition 5.1-5.3 in Appendix.

(3) In Theorem 5.5, since we use iteration-by-iteration analysis in the proof, the learning rate is not a function of $I$. It can be seen that in the literature, such as Theorem IV in [1], for quadratic objective functions, the learning rate is upper bounded by $I$. The advantage of that $\gamma$ is not a function of $I$ can be explained as follows. In Theorem 5.5, to obtain the optimal dependence on $R$ and $I$, we can choose $\gamma = \frac{1}{\sqrt{RI}}$. This requires that $\frac{1}{\sqrt{RI}}\le \frac{1}{L_g}$, which means that $I$ can be as large as possible. However, if $\gamma \le \frac{1}{IL_g}$, we will have $\frac{1}{\sqrt{RI}}\le\frac{1}{IL_g} $ such that $I\le \frac{R}{L_g^2}$, which means that to achieve the convergence rate of $\mathcal{O}(\frac{1}{\sqrt{RI}})$, $I$ cannot be arbitrarily large. Therefore, the range of the learning rate in Theorem 5.5 significantly enhances the convergence analysis.

Reviewer comment (Experiments):
**Experiment Results Below Expectation**: The experimental results presented in the paper fall short of expectations, particularly in comparison to prior works. In many previous studies, training and test accuracy consistently reach high levels, such as 90% for MNIST and 80% for CIFAR10 datasets. The paper's results, on the other hand, demonstrate lower performance. This discrepancy might be attributed to the specific models employed, namely linear regression and simple CNN. The choice of these relatively basic models may limit the generalizability of the findings, and more comprehensive experiments with diverse model architectures could provide a more robust assessment of the proposed approach. In addition, I believe the estimation of Lipschitz is a highlight for experiments. So I hope the author can provide a more in-depth discussion on how Lipschitz constants are estimated, the justification, and present results across a broader range of datasets and models to reinforce the paper's findings.

Author response:
We thank Reviewer wiji for the questions and the suggestions. Our responses are as follows.

2. **About the experiments.** We would like to clarify the following points for our experiments.

(1) About the accuracy. It is common in the literature on the theoretical analysis of FL algorithms [1-4] that accuracy cannot reach $80\\%$ for CIFAR-10 with a highly non-IID partition. This is because that in the literature of theoretical analysis of FL algorithms, the goal of the experiments is mainly to show the effect of some parameters, such as $I$, while other parameters are kept the same instead of being fine-tuned. For the MNIST dataset, the test accuracy can achieve $90\%$ when the percentage of heterogeneous data is $50\\%$. The experimental results for MNIST dataset can be found in Appendix C.

(2) About more experiments with different models and datasets. During these days, we run more experiments with the CINIC-10 dataset and VGG-16. We have added the new experimental results in Appendix C, which can be found in the new PDF file. 

(3) About estimating $\tilde{L}$, $L_g$ and $L_h$. We provide the methods of estimating $L_h$, $L_g$ and $\tilde{L}$ in Appendix C. 

We will provide more experimental results for (2) and (3) in the subsequent version if the paper is accepted.

[1] Wang, Jianyu, et al. "Tackling the objective inconsistency problem in heterogeneous federated optimization." Advances in neural information processing systems 33 (2020): 7611-7623.

[2] Jhunjhunwala, D., Sharma, P., Nagarkatti, A., \& Joshi, G. (2022, August). Fedvarp: Tackling the variance due to partial client participation in federated learning. In Uncertainty in Artificial Intelligence (pp. 906-916). PMLR.

[3] Khaled, Ahmed, Konstantin Mishchenko, and Peter Richtárik. "Tighter theory for local SGD on identical and heterogeneous data." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.

[4] Reddi, Sashank J., et al. "Adaptive Federated Optimization." International Conference on Learning Representations. 2020.

Reviewer comment (Theory):
After Theorem 4.3 (in "New insights about the effect of data heterogeneity"), the authors write "*A key message is that when $\zeta^2$ is large, as long as $L_h^2$ is small enough, the error caused by local updates can still be small.*" First of all, "small enough" should be precisely quantified. But more importantly, the choice of $\gamma$ in Corollary 4.4 completely nullifies the role of $L_h^2$ in the convergence bound. The bound in Corollary 4.6 is also independent of $L_h^2$. This looks a bit strange (especially after reading the aforementioned statement).

Author response:
1. **About the choice of $\gamma$.** First, we quantify $L_h$ in Theorem 4.3 as follows. When $L_h$ is small enough such that $\gamma^2L_h^2\zeta^2\ll \frac{\mathcal{F}}{\gamma \eta R}$, the convergence bound will become smaller when increasing $I$. However, it should be noted that this does not imply that $I$ can be arbitrarily large. As we will show in the following, the upper bound of $I$ depends on $L_h$. 

Second, in Corollary 4.4, choosing $\gamma = \frac{1}{IL_h}$ does not mean that the role of $L_h$ is nullified. Instead, a smaller $L_h$ implies that we can choose a larger $I$ in Corollary 4.4. The explanation is as follows. 
In Theorem 4.3, the condition for $\gamma$ is $\gamma \le \frac{1}{30(L_g+L_h)I}$. By this condition, when $L_h \le \frac{1}{I}$, we have
\begin{align}
    \gamma = \frac{1}{L_h I}\cdot \frac{1}{\sqrt{RIN}} \le \frac{1}{\sqrt{RIN}} \le \frac{1}{30(L_g+L_h)I}.
\end{align}
Rearranging the above inequality, we have $I \le \frac{RN}{900(L_g+L_h)^2}$. Combined with $L_h\le \frac{1}{I}$, we obtain the upper bound of $I$. That is

$$
I \le \min \left\\{\frac{1}{L_h}, \frac{RN}{900(L_g+L_h)^2}\right\\}.
$$

It can be seen that a smaller $L_h$ means we can choose a larger $I$ such that Corollary 4.4 still holds.

Reviewer comment (Novelty):
A result similar to Theorem 4.5 (and Theorem 4.4) is presented in Theorem 1 of [1] (cited below). They use the per-client smoothness assumption (Assumption 3.1 in this paper) though. The authors claim that they "develop new techniques" to obtain Theorem 4.5 but Appendix B does not mention what are the new techniques. Can the authors please elaborate on this? It seems that this paper obtains prior results with better constants but it is not clear to me how significant the technical challenges are in doing so.

Author response:
2. **About the technical novelty.** It is not straightforward to apply Assumption 4.1 and 4.2 to the convergence analysis for both full participation and partial participation. The technical novelties are summarized as follows.

(1) We need to develop new techniques to incorporate Assumption 4.1 and 4.2. In the proof of Theorem 4.3 shown in the Appendix, we need to characterize the difference between local gradients. In the literature, this is done by applying the local Lipschitz constant in shown in Assumption 3.1 in the main paper. In our paper, since Assumption 3.1 is replaced by our newly introduced assumption 4.2, the proof techniques in the literature cannot be applied. It requires developing new proof techniques to use Assumption 4.2 as shown in the proof of Lemma A.1-A.3. For example, in Lemma A.1, due to the application of Assumption 4.1 and 4.2, we have to deal with a new term, the local gradient deviation $\\|\frac{1}{N}\sum_{i=1}^N \nabla F_i(\mathbf{x}_i) - \nabla F_j(\mathbf{x}_j) \\|^2$, which is not shown in other techniques. Another example is in the proof of Theorem 4.5. Due to that we only use the global Lipschitz gradient assumption, we have to derive a new method to bound and incorporate the sampling related term 

$\mathbb{E}\\|\frac{1}{M}\sum_{i\in S_r}\sum_{k=0}^I \mathbf{g}_i(\mathbf{x}_r^{r,k}) \\|^2,$

which can be seen from (A.27) to (A.35).

(2) In addition to using Assumption 4.2 to characterize the new convergence rate of FedAvg, we also validate this assumption from the theoretical perspective. We develop the proof for Proposition 5.1-5.3 in Appendix.

(3) In Theorem 5.5, since we use iteration-by-iteration analysis in the proof, the learning rate is not a function of $I$. It can be seen that in the literature, such as Theorem IV in [1], for quadratic objective functions, the learning rate is upper bounded by $I$. The advantage of that $\gamma$ is not a function of $I$ can be explained as follows. In Theorem 5.5, to obtain the optimal dependence on $R$ and $I$, we can choose $\gamma = \frac{1}{\sqrt{RI}}$. This requires that $\frac{1}{\sqrt{RI}}\le \frac{1}{L_g}$, which means that $I$ can be as large as possible. However, if $\gamma \le \frac{1}{IL_g}$, we will have $\frac{1}{\sqrt{RI}}\le\frac{1}{IL_g} $ such that $I\le \frac{R}{L_g^2}$, which means that to achieve the convergence rate of $\mathcal{O}(\frac{1}{\sqrt{RI}})$, $I$ cannot be arbitrarily large. Therefore, the range of the learning rate in Theorem 5.5 significantly enhances the convergence analysis.

Reviewer comment (Theory):
The class of quadratics in Theorem 5.5 (i.e., $A_i = A$ for all $i$) is too restrictive for me. It would have been better to obtain a general result with $A_i \neq A$ and then state how small $L_h = 2 \max_{i} |\lambda(A_i - A)|$ needs to be so that local SGD is better than mini-batch SGD.

Author response:
3. **About the quadratic results.** We would like to clarify the following points for our quadratic results in Theorem 5.5.

(1) Developing Theorem 5.5 is not straightforward, and it provides a better range for $I$. As mentioned in our response to the question about the technical novelty, we use the techniques which are different from those used in the proof for Theorem 4.3. In Theorem 5.5, the range of the learning rate is given by $\gamma\le \frac{1}{|\lambda(\mathbf{A})|}$, which is not a function of $I$. This is not shown in previous works.
The advantage of the range of the learning rate $\gamma\le \frac{1}{|\lambda(\mathbf{A})|}$ can be shown as follows. To obtain the optimal dependence on $R$ and $I$, we can choose $\gamma = \frac{1}{\sqrt{RI}}$. By the range of the learning rate, this only requires $\frac{1}{\sqrt{RI}}\le \frac{1}{|\lambda(\mathbf{A})|}$, which means we can choose arbitrarily large $I$. Extending $\gamma\le \frac{1}{|\lambda(\mathbf{A})|}$ to non-convex quadratic objectives with $\mathbf{A} \neq \mathbf{A}_i$ can be very challenging since current techniques require $\gamma < \frac{1}{IL_g}$ as shown in Theorem 4.3. This requires us developing new techniques, which could be one of our future work.

(2) The theoretical result in Theorem 4.3 is general and we can use Theorem 4.3 to make a comparison between local SGD and mini-batch SGD when $L_h>0$. Later we can see that the implication and the intuition are similar to what we have observed in Theorem 5.5.

The convergence upper bound for local SGD is
$$
    \min_r \mathbb{E}\left\\| \nabla f(\bar{\mathbf{x}}^r) \right\\| = \mathcal{O}\left(\frac{\mathcal{F}}{\gamma\eta RI} + \frac{\gamma\eta L_g \sigma^2}{N}+\gamma^2L_h^2I^2\zeta^2 + \gamma^2 L_h^2 I\sigma^2 \right).
$$
The convergence upper bound for mini-batch SGD is
$$
    \min_r \mathbb{E}\left\\| \nabla f(\bar{\mathbf{x}}^r) \right\\| = \mathcal{O}\left(\frac{\mathcal{F}}{\alpha R}+ \frac{\alpha L_g \sigma^2}{NI} \right).
$$
Here, $\alpha$ denotes the learning rate of mini-batch SGD, distinguishing it from local SGD. Let $\gamma\eta = \alpha$. If $L_h$ is small enough, such that $\gamma^2L_h^2(I^2\zeta^2 + I\sigma^2)\le \frac{\gamma\eta L_g\sigma^2}{N}$, when $\sigma \to 0$, the convergence rate of local SGD goes to $\mathcal{O}(\frac{1}{RI})$ while the convergence rate of mini-batch SGD goes to $\mathcal{O}(\frac{1}{R})$.  Later we can see that the implication and the intuition are similar to what we have observed in Theorem 5.5.

The convergence upper bound for local SGD is
$$
    \min_r \mathbb{E}\left\\| \nabla f(\bar{\mathbf{x}}^r) \right\\| = \mathcal{O}\left(\frac{\mathcal{F}}{\gamma\eta RI} + \frac{\gamma\eta L_g \sigma^2}{N}+\gamma^2L_h^2I^2\zeta^2 + \gamma^2 L_h^2 I\sigma^2 \right).
$$
The convergence upper bound for mini-batch SGD is
$$
    \min_r \mathbb{E}\left\\| \nabla f(\bar{\mathbf{x}}^r) \right\\| = \mathcal{O}\left(\frac{\mathcal{F}}{\alpha R}+ \frac{\alpha L_g \sigma^2}{NI} \right),
$$
where we use $\alpha$ to denote the learning rate of mini-batch SGD to distinguish from local SGD. Let $\gamma\eta = \alpha$. If $L_h$ is small enough such that $\gamma^2L_h^2(I^2\zeta^2 + I\sigma^2)\le \frac{\gamma\eta L_g\sigma^2}{N}$, when $\sigma \to 0$, the convergence rate of local SGD goes to $\mathcal{O}(\frac{1}{RI})$ while the convergence rate of mini-batch SGD goes to $\mathcal{O}(\frac{1}{R})$.

Reviewer comment (Theory):
Regarding Weakness 1, perhaps the authors should choose a $\gamma$ that is *not* inversely proportional to $L_h$?

Author response:
1. **About the choice of $\gamma$.** First, we quantify $L_h$ in Theorem 4.3 as follows. When $L_h$ is small enough such that $\gamma^2L_h^2\zeta^2\ll \frac{\mathcal{F}}{\gamma \eta R}$, the convergence bound will become smaller when increasing $I$. However, it should be noted that this does not imply that $I$ can be arbitrarily large. As we will show in the following, the upper bound of $I$ depends on $L_h$. 

Second, in Corollary 4.4, choosing $\gamma = \frac{1}{IL_h}$ does not mean that the role of $L_h$ is nullified. Instead, a smaller $L_h$ implies that we can choose a larger $I$ in Corollary 4.4. The explanation is as follows. 
In Theorem 4.3, the condition for $\gamma$ is $\gamma \le \frac{1}{30(L_g+L_h)I}$. By this condition, when $L_h \le \frac{1}{I}$, we have
\begin{align}
    \gamma = \frac{1}{L_h I}\cdot \frac{1}{\sqrt{RIN}} \le \frac{1}{\sqrt{RIN}} \le \frac{1}{30(L_g+L_h)I}.
\end{align}
Rearranging the above inequality, we have $I \le \frac{RN}{900(L_g+L_h)^2}$. Combined with $L_h\le \frac{1}{I}$, we obtain the upper bound of $I$. That is

$$
I \le \min \left\\{\frac{1}{L_h}, \frac{RN}{900(L_g+L_h)^2}\right\\}.
$$

It can be seen that a smaller $L_h$ means we can choose a larger $I$ such that Corollary 4.4 still holds.

Reviewer comment (Novelty):
The insight that the average of local models may be close to the centralized model even with the presence of heterogeneity is also observed in Wang et al., 2022, i.e., to instead measure the average drift at optimum (Eq. (15) in Wang et al., 2022). The corresponding discussion seems limited to that Wang et al., 2022 studies convex functions whereas this paper studies non-convex functions, without much comments on how they differ / relate in measuring heterogeneity. As a result, I'm not quite convinced by the statement "both works cannot capture the information on the data heterogeneity contained in the local Lipschitz constant as shown in our paper so their convergence upper bounds can be worse compared to ours."

Author response:
2. **About the difference between Wang et al., 2022 and our paper.** In Wang et al., 2022, a new metric for data heterogeneity, $\rho$, the average drift at optimum, is proposed. The definition of $\rho$ is
\begin{align}
\left\\|\frac{1}{\gamma I}\left(\frac{1}{N}\sum_{i=1}^N\mathbf{x}_i^{r,I} - \bar{\mathbf{x}}^r \right)\right\\|= \rho.
\end{align}
The detailed comparison between Wang et al., 2022 and our paper is in the following.


(1) The new metric $\rho$ in Wang et al., 2022 focuses on the difference on models while in our paper, we still focus on the difference on the gradients. The key insight in Wang et al., 2022 is that since $\rho$ is small, when the global model is $\mathbf{x}^*$, after multiple local updates, the averaged model does not change significantly. 
In our paper, the key insight is that since $L_h$ can be small, the difference between the current global gradient and the current averaged local gradients can be small.
In Wang et al., 2022, the gradient divergence is not used in the analysis.
In our analysis, we still use the gradient divergence jointly with the proposed $L_h$ to measure the data heterogeneity.

(2) In Wang et al., 2022, it is only empirically shown that $\rho$ can be small. In our paper, we not only empirically show that $L_h$ can be small, but also provide an analytical example. Our quadratic example can be non-convex, a case which $\rho$ cannot cover. 


(3) One weakness of using $\rho$ is that in the convergence bound in Wang et al., 2022, the convergence error shown by $\rho$ cannot vanish. This means that by choosing $\gamma = \frac{1}{\sqrt{R}}$, when $R$ goes to infinity, the convergence bound cannot guarantee that FedAvg can converge to the local minima of the global objective function. On the contrary, our convergence bound can guarantee the convergence to the local minima of the global objective function, which is shown by Corollary 4.4.

Reviewer comment (Theory):
The analysis in the paper seems to me more numerical than theoretical, as the improvement in convergence rates comes purely from improved constants, which are mostly measured numerically. The theoretical arguments that the proposed constants are smaller are also limited to quadratics.

Author response:
3. **About the proposed $L_h$.** Using $L_h$ can address the unsolved problem in the literature. We observe that existing theoretical analyses are pessimistic about the convergence error caused by local updates; however, in practice, local updates can improve the convergence rate and reduce communication costs. By employing $L_h$, we can bridge the gap between the pessimistic theoretical results and the positive experimental outcomes for federated algorithms, which is \textbf{unsolved} in the literature. This is our main contribution. 

Other theoretical contributions include: (i) as shown in Proposition 5.1, we rigorously prove our assumptions are weaker; (ii) there are technical novelties in our proof (see the response point 1 above.)
(iii) in Theorem 5.5, by extending the discussion to $L_h=0$, we show that local SGD can outperform mini-batch SGD for non-convex quadratic objectives, which is not shown in the literature.

Reviewer comment (Theory):
An improved version of Assumption 3.3 would be to assume bounded gradient divergence at optima only, as in Glasgow et al., 2022, or as Assumption 2.3 in Patel, Kumar Kshitij, et al., 2023. How would this assumption affect the comparison / relation of the proposed measure to previous measures?

Author response:
4. **About the question for the bounded gradient divergence on the optima only.** The bounded gradient divergence on the optima is 

$$
 \\|\nabla f(\mathbf{x}^*) - \nabla F_i(\mathbf{x}^*) \\|^2 = \\| \nabla F_i(\mathbf{x}^*)\\|^2 \le \zeta_*^2.
$$

We observe that $\zeta_*$ is often used to substitute $\zeta$ in the analysis for convex objective functions in the literature [2-4]. To the best of our knowledge, there is no literature applying $\zeta_*$ in the analysis for non-convex objective functions. The possible reason could be that the convergence upper bound for non-convex objective functions can only guarantee the convergence to one of the local minima. Only assuming gradient divergence on the global minima might not help the convergence analysis. However, we believe that it is possible to use both $L_h$ and $\zeta_*$ in the convergence analysis for convex objectives.  For example, in the proof for heterogeneous data of [2], there are also steps transforming the gradient difference to the model divergence, such as (58) and (59) in [2], where we can use $L_h$ to substitute $\tilde{L}$. Therefore, applying Assumption 4.1 and 4.2 to the convergence analysis for convex objectives can be one of our future work.

Reviewer comment (Novelty):
The contribution of the paper is limited. The bound is the same as [1]. The only difference is that the Lipschitz constants are replaced, but the bound has the same order and the same limitations as [1] with respect to $R$, $I$, and $N$.

Author response:
We thank Reviewer NMhc for the questions and the suggestions.
Our responses are as follows.

1. About the contributions of our paper. We respectfully disagree with Reviewer NMhc about our contributions on the heterogeneity-driven Lipschitz constant, $L_h$. The reasons are as follows.

(1) Using $L_h$ can address an open problem in the literature. We observe that existing theoretical analyses are pessimistic about the convergence error caused by local updates; however, in practice, local updates can improve the convergence rate and reduce communication costs. By employing $L_h$, we can bridge the gap between the pessimistic theoretical results and the positive experimental outcomes for federated algorithms, which is **unsolved** in the literature. This is our main contribution.

(2) Using $L_h$ in the convergence analysis is not straightforward. We develop new techniques and lemmas in the proof. In the proof of Theorem 4.3 shown in the Appendix, we need to characterize the difference between local gradients. In the literature, this is done by applying the local Lipschitz constant in shown in Assumption 3.1 in the main paper. In our paper, since Assumption 3.1 is replaced by our newly introduced assumption 4.2, the proof techniques in the literature cannot be applied. It requires developing new proof techniques to use Assumption 4.2 as shown in the proof of Lemma A.1-A.3. For example, in Lemma A.1, due to the application of Assumption 4.1 and 4.2, we have to deal with a new term, the local gradient deviation $\\|\frac{1}{N}\sum_{i=1}^N \nabla F_i(\mathbf{x}_i) - \nabla F_j(\mathbf{x}_j) \\|^2$, which is not shown in other techniques. Another example is in the proof of Theorem 4.5. Due to that we only use the global Lipschitz gradient assumption, we have to derive a new method to bound and incorporate the sampling related term 

$$\mathbb{E} \\| \frac{1}{M}\sum_{i\in S_r}\sum_{k=0}^I \mathbf{g}_i(\mathbf{x}_r^{r,k}) \\|^2,$$

which can be seen from (A.27) to (A.35).

(3) We believe that our contributions of demonstrating the usefulness and effectiveness of $L_h$ are overlooked by the reviewer. One advantage of using $L_h$ is that we rigorously show that Assumption 4.1 and 4.2 are weaker than the widely used Lipschitz gradient assumption in the literature. Another advantage is that using the quadratic example with $L_h=0$, we show that with non-convex quadratic objectives, local SGD can outperform mini-batch SGD, which is not shown in the literature. These two advantages are overlooked by the reviewer.

(4) We can obtain a better convergence rate by using $L_h$.
The convergence rate in our paper is $\mathcal{O}\left(\frac{1}{\sqrt{RIN}} + \frac{1}{RIN} \right)$ while the convergence rate in [1] is $\mathcal{O}\left(\frac{1}{\sqrt{RIN}}+ \frac{1}{R} \right)$ . It can be seen that our convergence rate is better for second term $\frac{1}{RIN}$. The advantage of our convergence rate is that we can show when fixing the product of $R$ and $I$, if we increase $I$ and decrease $R$, we can achieve the same accuracy. However, in [1], this is not true since the second term will become larger if we increase $I$ and decrease $R$.
A quick example of showing this property is in Theorem 5.5.
In Theorem 5.5, we showed that we can keep the product of $R$ and $I$ and choose any $R\ge 1$ to achieve the same accuracy. This is one of the advantages of using Assumptions 4.1 and 4.2.

Reviewer comment (Theory):
The paper claims based on the bounds, FedAvg works with large $I$ and small $R$. However, this doesn't happen in practice. Also, in order for the bound to hold, we should have $\eta \gamma \le \frac{1}{4I L_g}$, and in the Corollary 4.4 the bounds $\eta \gamma = \sqrt{\frac{4FN}{RI L_g \sigma^2}}$, which means $O(IN) \le R$. Therefore, with the increase of number of local steps, more rounds are needed for the bound to hold.

Author response:
3. First, we respectfully disagree with Reviewer NMhc that ``The paper claims based on the bounds, FedAvg works with large $I$ and small $R$. However, this doesn't happen in practice.'' In fact, the main advantage of FedAvg is that using more local updates (a larger $I$) can reduce the communication cost (a smaller $R$). In Section 5, we have provided a quadratic example which even does not require any experiments to show that we can choose $R=1$ and $I$ can be arbitrarily large.

Second, we agree that when choosing the learning rates in Corollary 4.4, with $\eta\gamma\le \frac{1}{4IL_g}$, we have
$$
\gamma\eta = \sqrt{\frac{4\mathcal{F}N}{RIL_g\sigma^2}}\le \frac{1}{4IL_g}.
$$
We can rewrite the above inequality as 
\begin{align}
I\le \frac{R\sigma^2}{64\mathcal{F}NL_g}.
\end{align}
This means the range of $I$ is $[1,\frac{R\sigma^2}{64\mathcal{F}NL_g}]$. When $I$ is in this range, increasing $I$ does not mean that we have to increase $R$. Therefore, keeping the product of $R$ and $I$, we can still choose a smaller $R$ and a larger $I$ as long as the above inequality still holds.