Paper: Impact of Agent Behavior in Distributed SGD and Federated Learning (ICLR-2024)

Reviewer comment (Novelty):
There have been many studies on communication topology in existing work, e.g., [Koloskova et al. (2020), Wang et al. (2021)]. Generally speaking, as long as assumption 2.5 is made, the consistency of the distributed learning algorithm can be guaranteed, so the GD-SGD algorithm designed in this paper is not novel.

Author response:
We acknowledge that Assumption 2.5 is indeed a common condition to guarantee the consistency of various distributed learning algorithms. However, the primary aim of our paper is not to introduce a novel distributed learning algorithm per se. Rather, our focus is on providing a detailed asymptotic analysis within the unified framework of *generalized* decentralized SGD (GD-SGD), by showing the consistency (almost sure convergence) as well as asymptotic normality via CLT, with which we can single out the impact of each agent on the overall performance. The essence of our contribution lies in elucidating the effects of different sampling strategies employed by individual agents in a distributed learning setting. This aspect of our research is particularly significant as it addresses nuances that are not captured by the current non-asymptotic analysis frameworks, as highlighted in Table 1 of our paper.

Reviewer comment (Experiments):
Logistic regression is a toy model, it is better to further consider other real-world models.

Author response:
We understand the importance of demonstrating the applicability of our GD-SGD algorithm to more complex, real-world scenarios. In response to this, we would like to clarify that we have also performed the simulation with a non-convex objective function in Appendix G in our original submission. In this additional simulation, the GD-SGD algorithm has not yet reached the asymptotic regime, and the communication patterns (e.g., size of partial client sampling set, communication interval, and communication matrix) still significantly influence the MSE. Notably, the core finding from our study remained consistent: enhanced sampling strategies employed by a subset of agents resulted in faster convergence rates across almost all time periods. This result is particularly insightful as it suggests that employing effective sampling strategies at the level of individual agents can mitigate performance losses that might arise from less frequent aggregation. Such a strategy is valuable in reducing communication costs, whether among agents or between agents and a central server.

Reviewer comment (Theory):
I feel that technical comparison to existing work is lacking. While the table summarizes the existing results and what settings they operate in, it does not discuss what are precisely the bounds obtained by papers such as Doan et al. (2017). As a result, it is unclear whether these bounds actually fail to capture the effect of sampling on all the clients.

Author response:
Answer: When the reviewer mentions [Doan et al. (2017)], we guess it is [Doan et al. (2019)] in our reference. Regarding the technical comparison to existing works, we refer the reviewer to bullet point #2 of the response to (Q1) in the ‘response to all reviewers’ part.

Regarding assumption 2.3, since most of the previous papers focus on the constant communication interval only, we only need assumption 2.3(i), where the usual polynomial step size $γ_n=1/n^a$  for $a∈(0.5,1]$ is assumed. Moreover, the only paper we are aware of talking about the increasing communication interval scheme is [Li et al. (2022)] and our assumption 2.3(ii) is slightly stringent than theirs in order to cover more decentralized settings and Markovian sampling, and we have already included the discussion of this assumption in Remark 1 in our original submission.

Reviewer comment (Theory):
All the results in this paper require Assumption 2.3. Was that required by the previous papers as well?

Author response:
Answer: When the reviewer mentions [Doan et al. (2017)], we guess it is [Doan et al. (2019)] in our reference. Regarding the technical comparison to existing works, we refer the reviewer to bullet point #2 of the response to (Q1) in the ‘response to all reviewers’ part.

Regarding assumption 2.3, since most of the previous papers focus on the constant communication interval only, we only need assumption 2.3(i), where the usual polynomial step size $γ_n=1/n^a$  for $a∈(0.5,1]$ is assumed. Moreover, the only paper we are aware of talking about the increasing communication interval scheme is [Li et al. (2022)] and our assumption 2.3(ii) is slightly stringent than theirs in order to cover more decentralized settings and Markovian sampling, and we have already included the discussion of this assumption in Remark 1 in our original submission.

Reviewer comment (Novelty):
Can the authors comment on technical comparison to related works, as I mentioned above?

Author response:
Answer: When the reviewer mentions [Doan et al. (2017)], we guess it is [Doan et al. (2019)] in our reference. Regarding the technical comparison to existing works, we refer the reviewer to bullet point #2 of the response to (Q1) in the ‘response to all reviewers’ part.

Regarding assumption 2.3, since most of the previous papers focus on the constant communication interval only, we only need assumption 2.3(i), where the usual polynomial step size $γ_n=1/n^a$  for $a∈(0.5,1]$ is assumed. Moreover, the only paper we are aware of talking about the increasing communication interval scheme is [Li et al. (2022)] and our assumption 2.3(ii) is slightly stringent than theirs in order to cover more decentralized settings and Markovian sampling, and we have already included the discussion of this assumption in Remark 1 in our original submission.

Reviewer comment (Reproducibility):
What is the step-size schedule in the experiments?

Author response:
Answer: Thank you for highlighting the omission regarding the step-size schedule in our experiments. Your observation has helped us improve the clarity of our simulation setup. In our experiments, we employ a decreasing step size $γ_n=1/n^{0.9}$. This choice is made to ensure compliance with Assumption 2.3 and thus consistency with the theoretical framework we have established. We have now incorporated this detail into the revised version of our paper.

Regarding the increasing communication interval, our findings depicted in Figure 2(b) reveal that the convergence rate is not adversely affected by this scheme. Contrary to the suggestion that increasing the number of local steps may be unnecessary, our results actually indicate a different conclusion. The implementation of an increasing communication interval scheme effectively reduces the frequency of aggregations, either with neighboring agents or a central server. This strategy leads to a reduction in communication costs, which is a significant consideration in distributed learning scenarios. Importantly, we found that this approach has minimal impact on the convergence speed in the asymptotic regime. This insight is particularly valuable as it suggests that we can achieve cost-effective communication without compromising the efficiency of convergence in distributed learning algorithms.