Paper: Communication-efficient Random-Walk Optimizer for Decentralized Learning (ICLR-2024)

Reviewer comment (Experiments):
Given that a mini-batch is introduced to reduce the communication cost by a factor of K, why is it necessarily the case that using this variant of ADAM which costs "1" to communicate as opposed to "3" (original ADAM-based random-walk), better than using ADAM + random walk with a mini-batch size of 3K?

Author response:
We suppose $K$ refers to the number of local steps in the inner loop of Algortihm 3, whose influence is studied in Figure 7, Section 4.3. While using a larger $K$ can reduce the communication cost, setting it too large can lead to worse models, which agrees with Theorem 3.7 and Corollary 3.8. Therefore, solely using more local steps may not always lead to better performance, while removing the communication of auxiliary parameters (momentum or preconditioner) can be helpful in such case.

Reviewer comment (Evaluation):
From the simulations, it seems that ADAM performs worse on the training data set (counting only computations). Can the authors explain why this seems to be the case?

Author response:
From Figure 2-5, it seems that Adam has almost the same performance as Gossip, FedAvg or Localized SAM (comparing the orange line with purple, gray and pink lines).

Reviewer comment (Experiments):
Since SAM is integrated to prevent potential overfitting, the experiment should present this effect compared with its counterpart that does not have the perturbed model. The lack of this experiment comparison would question the necessity of incorporating SAM in the proposed Localized framework.

Author response:
> Since SAM is integrated to prevent potential overfitting, the experiment should present this effect compared with its counterpart that does not have the perturbed model. The lack of this experiment comparison would question the necessity of incorporating SAM in the proposed Localized framework. 

There might be some misunderstandings. Indeed, we have conducted ablation study on the choice of different hyper-parameter $\rho$’s in Figure 8, Section 4.3. Setting $\rho = 0$ reduces to the case where we do not have perturbation, and its performance is worse than more appropriate settings (e.g., $\rho = 0.05$).

Reviewer comment (Evaluation):
The simulation only shows the loss performance of the proposed algorithms and the benchmarks; however, in practice, we would be more interested to see the classification accuracy.

Author response:
> The simulation only shows the loss performance of the proposed algorithms and the benchmarks, however, in
practical, we would be more interested to see the classification accuracy. 

We have added these results in Appendix A.1, as are also shown here. These results are generally consistent with the loss performances (Figure 2-5 in the main text): SGD and Walkman generally achieve slightly worse final performances than other methods. Among all methods that are based on adaptive optimizers, Localized SAM achieves the best overall performance.

| Graph type         |      | Ring |      |       |3-regular |     |
|--------------------|------|------|------|------|-------|------|
| # clients          | 10   | 20   | 50   | 10   | 20        | 50   |
| SGD                | 82.3 | 81.7 | 81.4 | 83.2 | 82.5      | 81.9 |
| Adam               | 85.2 | 85.0 | 84.6 |  88.8    | 88.1 | 87.4   |
| Walkman            | 81.6 | 78.2 | 78.9 | 85.7 | 85.3 | 84.8  |
| Adam (no momentum) |   85.1 | 84.8 | 85.2 | 88.2 | 87.7 | 87.2   |
| Gossip             |  85.6 | 85.3 | 84.9 | 88.5 | 88.2 | 87.8  |
| FedAvg             |  85.4 | 85.5 | 85.0 | 88.3 | **88.6** | 87.6  |
| Localized SAM      |**85.8**   |  **86.0**  |  **85.4**    |  **89.0**    |  **88.6**    |**88.0**  |

Reviewer comment (Experiments):
The proposed algorithm is compared with FedAvg; however, for FedAvg, not all agents are communicating all the time, which does not make sense in the setting that FedAvg does not need to consider communication. That means, I suppose that if all agents in FedAvg communicate all the time, the performance of FedAvg might be much better than all the other methods, since there exists a coordinator, although the communication cost would be very high. The figures presented, however, show that Localized SAM is always better than FedAvg in the random sample setting in both performance and communication, which is not a fair comparison.

Author response:
> The proposed algorithm is compared with FedAvg, however, for FedAvg, not all agents are communicating all
the time, which does not make sense in the setting that FedAvg does not need to consider communication. 

There might be some misunderstandings. In our experiments, note that only the gossip method require all agents to communicate in
each iteration. FedAvg only requires some of the agents to communicate with the central server in each iteration, and all
the other methods (which are all from random-walk decentralized optimization) only requires one agent to communicate
in each iteration. The communication cost of FedAvg is analyzed in experiments (Figure 2-5). Its communication cost is
relatively small compared to the gossip method or Adam optimizer in random-walk setting, but still higher than Adam (no
momentum) (Triastcyn et al. 2022) and our method Localized SAM.

> That means, I suppose that if all agents in FedAvg communicate all the time, the performance of FedAvg might
be much better than all the other methods, since there exists a coordinator, although the communication cost would
be very high. The figures presented, however, show that Localized SAM is always better than FedAvg in the random
sample setting in both performance and communication, which is not a fair comparison. 

For FedAvg, we follow the
original setting of FedAvg and only uses some of the agents in each iteration, and the communication cost is computed
based on the number of agents. For the proposed Localized SAM (and similarly other methods based on random-walk
decentralized optimization), it only activates one client in each iteration, which updates the model using its local training
data and sends the updated model to the next agent. Both methods do not require all agents to communicate in each iteration,
and the comparison is fair across different methods. Moreover, in Figure 2-5, the performance of FedAvg are similar to
Localized SAM with respect to the number of weight updates, but has a slightly larger communication cost.

Reviewer comment (Novelty):
Page 2, second paragraph in Related Work, the Walkman algorithm (Mao et al., 2020) is solved by ADMM, with two versions, one is to solve a local optimization problem, the other is to solve a gradient approximation. Therefore, it is not accurate to say that “However, these works are all based on the simple SGD for decentralized optimization.”

Author response:
> Page 2, second paragraph in Related Work, the Walkman algorithm (Mao et al., 2020) is solved by ADMM, with
two versions, one is to solve a local optimization problem, the other is to solve a gradient approximation. Therefore,
it is not accurate to say that “However, these works are all based on the simple SGD for decentralized optimization.”

Thank you for raising this question. We have changed this sentence to “However, these works do not consider adaptive
learning rates or momentum in decentralized stochastic optimization.”

Reviewer comment (Writing):
Section 3, first paragraph, in “It can often have faster convergence and better generalization than the SGD-based Algorithm 1, as will be demonstrated empirically in Section 4.1.” The “it” does not have a clear reference.

Author response:
> Section 3, first paragraph, in “It can often have faster convergence and better generalization than the SGD-based
Algorithm 1, as will be demonstrated empirically in Section 4.1.” The “it” does not have a clear reference. 

Thank you
for mentioning this. We have replaced “it” with a more direct reference “Algorithm 2”.

Reviewer comment (Writing):
In Section 3.1, you introduced $\boldsymbol{u}_k$, which was not defined previously and did not show up after Algorithm 3.

Author response:
> In Section 3.1, you introduced uk , which was not defined previously and did not show up after Algorithm 3.

Thank you for mentioning this. We have revised this error in the updated version.

Reviewer comment (Novelty):
Each component (e.g. local updates, SAM) has been studied before in different contexts. The novelty lies more in the combination.

Author response:
While we agree that local updates or SAM has been studied before in different contexts, we would like
to emphasize that a key component of our method is to use localized momentum and pre-conditioner, which has not been
explored in previous works (Sun et al. 2022; Triastcyn et al. 2022) on random-walk decentralized optimization. Please also
see our responses regarding novelty where we have listed our contributions in points.

Reviewer comment (Novelty):
Missing References: "Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent".

Author response:
Thank you for mentioning these works. We have added these related works into our submission.

Reviewer comment (Novelty):
Missing References: "Asynchronous decentralized parallel stochastic gradient descent" where multiple local model updates is used in asynchronous decentralized training.

Author response:
Thank you for mentioning these works. We have added these related works into our submission.

Reviewer comment (Experiments):
Have you experimented with more complex network topologies beyond ring and expander graphs? There is a "chord network" topology in "Asynchronous decentralized parallel stochastic gradient descent" that may be beneficial.

Author response:
Thank you for your suggestion. As suggested, we have added experiments on the chord network with 20 clients.
Results are shown in Figure 10 in Appendix A.2, and are generally similar to the ring graph. In general, methods based
on adaptive optimizers outperform SGD or Walkman, regardless of the type of communication (centralized, random-walk,
or gossip). The gossip method has a much higher communication cost than both centralized and random-walk methods.
Walkman achieves slightly better performance than SGD with the same communication cost, but its performance is worse
than that of adaptive optimizers, especially for the more difficult CIFAR-100 data set. Directly using Adam in random-walk
optimization leads to higher communication cost than the other methods, and using Adam without momentum reduces
communication cost while maintaining good performance. The proposed Localized SAM optimizer achieves similar
convergence rate as other methods based on adaptive optimizers, while its communication cost is significantly smaller than
all other methods.

Reviewer comment (Theory):
Does the algorithm work with dynamic network topology where each iteration has a different communication topology?

Author response:
Thank you for raising this interesting question. Our algorithm can be directly combined with dynamic network
topology, as it only needs local neighborhood information in each iteration. Theoretical analysis may require a little more
work as we may need to assume the communication topology does not drastically change. We have also added some
discussions on this extension in Appendix C.

Reviewer comment (Theory):
Is it possible to combine your approach with compression of model parameters for further communication reduction without making the convergence rate worse?

Author response:
Thank you for this question. It should be straight-forward to
combine our method with model compression for further communication reduction, and the convergence rate should also
match methods that also employ model/gradient compression (Lin et al. 2018; Vogels et al. 2019). We have also added
some discussions on this extension in Appendix C.

References:

Lin et al. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. ICLR 2018.

Vogels et al. PowerSGD: Practical low-rank gradient compression for distributed optimization. NeurIPS 2019.

Reviewer comment (Novelty):
Limited novelty. The idea of using local SGD with local optimizer states is quite standard in the field of federated learning and distributed training. The same comment could be applied to include SAM for FL. See e.g., [1, 2] and the related work section of [1].

Author response:
### Limited novelty. 

> The idea of using local SGD with local optimizer states is quite standard in the field of
federated learning and distributed training. 

We agree that using local SGD with local optimizer states have been used
in federated learning and distributed training (McMahan et al. 2017; Reddi et al. 2021). Indeed, we also mentioned this
in Section 3.1. However, note that they all require a central server to aggregate updates from local optimizers, and the
optimizer states are stored in the central server, not clients. On the contrary, our focus is on extending local SGD and local
optimizer to random-walk decentralized optimization, which does not need any central server, and this has not been explored
in the random-walk decentralized optimization literature (such as (Sun et al. 2022; Triastcyn et al. 2022)). In this paper, we
demonstrate that the communication of optimizer states can be removed without performance degradation.

Reference: Reddi et al. Adaptive federated optimization. ICLR 2021.

> The same comment could be applied to include SAM for FL. See e.g., [1, 2] and the related work section of [1].:

Although they all use SAM in distributed learning, these works focus on different types of optimization algorithms. [1]
is based on gossip decentralized algorithms, where all clients must always be active during the training process, and [2]
considers centralized federated learning, i.e., a central server can coordinate the whole learning process. Moreover, most of
related works discussed in [1] either focus on centralized federated learning (Qu et al., 2022; Sun et al., 2023) or gossip
decentralized algorithms (Sun et al. 2022; Dai et al. 2022), which are all different from our method. Our method focuses on
random-walk decentralized optimization that does not need a central server and only require one client to be active in each
iteration. As such, it has much weaker requirements on the communication network. Please also see our general responses
on novelty part.

References:

Qu et al. Generalized federated learning via sharpness aware minimization. ICML 2022.

Sun et al. Decentralized federated averaging. IEEE TPAMI 2022.

Dai et al. DisPFL: Towards communication-efficient personalized federated learning via decentralized sparse training.
ICML, 2022.

Sun et al. Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency
and Smooth Landscape. ICML 2023.

Reviewer comment (Experiments):
Some advanced decentralized communication topologies were not considered for the evaluation, e.g., the exponential graph [3, 4, 5]. Besides, the considered relative communication cost could be questionable, as it counts each sending of the model parameter as 1, instead of taking the network bandwidth / wallclock-time into account.

Author response:
> Some advanced decentralized communication topologies were not considered for the evaluation, e.g., the
exponential graph [3, 4, 5]. 

Thank you for your comment. As suggested, we have added experiments on exponential
graphs with 20 clients. Results are shown in Figure 9 in the revised Appendix A.2, and are generally similar to ring or
3-regular expander graphs. In general, methods based on adaptive optimizers outperform SGD or Walkman, regardless of
the type of communication (centralized, random-walk, or gossip). The gossip method has a much higher communication
cost than both centralized and random-walk methods. Walkman achieves slightly better performance than SGD with the
same communication cost, but its performance is worse than that of adaptive optimizers, especially for the more difficult
CIFAR-100 data set. Directly using Adam in random-walk optimization leads to higher communication cost than the other
methods, and using Adam without momentum reduces communication cost while maintaining good performance. The
proposed Localized SAM optimizer achieves similar convergence rate as other methods based on adaptive optimizers, while
its communication cost is significantly smaller than all other methods.

> Besides, the considered relative communication cost could be questionable, as it counts each sending of the
model parameter as 1, instead of taking the network bandwidth / wallclock-time into account. 

Counting the total
amount of transmitted data are common practice for measuring communication cost in literature (Sun et al. 2022; Triastcyn
et al. 2022; Wang et al. 2022; Wu et al. 2022; Gao et al. 2021). As for the network bandwidth or wallclock time, note that
the wallclock time depends on the network bandwith as well as the amount of data sent in each iteration. The network
bandwidth should be assumed same across different algorithms for fair comparison, hence the wallclock time solely depends
on the amount of data sent in each iteration, as is measured by the relative communication cost.

References:

Wang et al. ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training.
ICML 2022.

Wu et al. SmartIdx: Reducing Communication Cost in Federated Learning by Exploiting the CNNs Structures. AAAI 2022.

Gao et al. On the Convergence of Communication-Efficient Local SGD for Federated Learning. AAAI 2021

Reviewer comment (Experiments):
A carefully controlled evaluation should be provided. The considered baseline methods cannot guarantee a fair evaluation, as the community has a line of research on improving communication efficiency. It looks strange to directly compare an improved random walk decentralized optimizer with the other standard-form distributed optimizer.

Author response:
> A carefully controlled evaluation should be provided. The considered baseline methods cannot guarantee a
fair evaluation, as the community has a line of research on improving communication efficiency. It looks strange
to directly compare an improved random walk decentralized optimizer with the other standard-form distributed
optimizer. 

Other techniques that improves communication efficiency include model quantization (Alistarh et al. 2017;
Nadiradze et al. 2021), top-K sparsification (Wang et al. 2023) and gradient compression (Lin et al. 2018; Vogels et
al. 2019). Note that our work focuses on reducing the communication cost by removing the communication of auxiliary
parameters in adaptive optimizers. Our experiments verify that removing these auxiliary parameters does not harm the final
performance, while significantly reducing the communication cost. Thus, these existing techniques are orthogonal to the
proposed method and can be straightforward combined. We have also added some discussions on potential future works in
Appendix C that combine the proposed method with other techniques.

References:

Alistarh et al. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. NIPS 2017

Nadiradze et al. Asynchronous Decentralized SGD with Quantized and Local Updates. NeurIPS 2021

Wang et al. CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks. ICML 2023

Lin et al. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. ICLR 2018

Vogels et al. PowerSGD: Practical low-rank gradient compression for distributed optimization. NeurIPS 2019