From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The authors propose SWIFT: a new asynchronous decentralized training algorithm enriched with local updates. Compared to previous works, the authors removed the bounded delay assumption in the analysis, and hence obtained better theoretical results. In order to ensure the convergence, a new technique is introduced that allows for non-symmetric non-doubly stochastic mixing matrices.Experiments on CIFAR10 dataset with a cluster of machines validate the superiority of the proposed algorithm.

All four reviewers recommended acceptance, and I agree with their views. Almost all issues raised were addressed by the authors. I recommend these explanations and improvements to be implemented in the final version of the paper.

Extra issue:  Recently, several authors removed bounded delay assumptions in analysis if SGD: https://arxiv.org/abs/2206.08307 https://arxiv.org/abs/2206.07638 (not my works!) These works absolutely need to be mentioned & compared to in the camera ready version of the paper.

AC
justification_for_why_not_higher_score: All scores are borderline accept; and the work does not seem to be of a notable technical or conceptual novelty.
justification_for_why_not_lower_score: All four reviewers suggested an accept (6).
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Thank You For Great Discussion!
comment: Dear Reviewer 1Mgi,

Thank you again for your time and feedback. We appreciate the discussion, and will use your feedback to improve our current paper. Namely, we will (1) add a discussion of communication delays within the Appendix, (2) overhaul the description of SWIFT and reformulate Algorithm 1 to better detail how we overcome computational delays, and (3) create a SWIFT algorithm that describes how SWIFT runs for a single client.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_1Mgi
title: Reply
comment: Communication: Yes, that's what I meant. AD-PSGD is blocking if they don't use gradient queue. 

About computation delay: I think the confusion comes from that algorithm 1 does not perfectly match the description before it. The description says that global iteration $t$ is determined after the local gradient is done (Line 15), but the current algorithm presents it as "at iteration $t$ SWIFT pick $i_t$" and **then** "at iteration $t+1$ SWIFT pick $i_{t+1}$". I think the authors' intended algorithm is that $i_{t+1}$ can be picked before $i_t$ but just finishes after $i_t$. It would be much clearer if you can reformulate the algorithm. Nevertheless, I think the description is correct and I will increase my scores.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Reviewer 1Mgi Follow-up (Part 1)
comment: Dear Reviewer 1Mgi, 

Thanks for your speedy reply.

## Resolving Issue Regarding Communication Delay

> If the communication is "instantaneous", then in their algorithm 2, the computation thread does not have to wait for the communication thread. 

We believe there may be a misunderstanding regarding the computation thread waiting for the communication thread in Line 5 of Algorithm 2. We note the reviewer's point that if fetching a neighboring model is "instantaneous", then the computation thread does not need to wait (granted the doubly-stochastic averaging is atomic). However, AD-PSGD would become blocking **if "instantaneous" communication is not assumed**.

For example, let's consider the two-client case for AD-PSGD. We note that at the beginning of training $g=0$ for both clients. The computation threads of Clients 1 and 2 begin running Algorithm 2, while the communication threads of Clients 1 and 2 begin running Algorithm 3. 

In Algorithm 2, the computation threads on both clients begin by computing a stochastic gradient. In Algorithm 3, the communication threads begin by immediately searching for a neighbor to average with (since $g=0$ initially and there is no available gradient to update its local model). In the setting we are considering, communication is extremely slow. As such, the communication threads begin communicating their models with each other very slowly. 

Meanwhile, the computational threads finish computing their stochastic gradients and do not wait at Line 5 of Algorithm 2 since no gradient is initially stored in the local buffer ($g=0$). In Line 6 of Algorithm 2, the computation threads store their computed stochastic gradient in the local buffer as $g$. The computational threads then begin Algorithm 2 again, pulling the local model and updating it with the gradient stored in the local buffer (Lines 2-3). Now, a second stochastic gradient is computed with the updated model. However, once computed, the computational threads **must wait at Line 5 of Algorithm 2 until the communication threads are finished with their communication**. Therefore, without “instantaneous” communication, AD-PSGD would be blocking.

### Waiting in Line 5 of Algorithm 2 in AD-PSGD

Algorithms 2 and 3 in AD-PSGD are constructed to support the practical implementation of Algorithm 1 (which assumes "instantaneous" communication in Line 5 of Algorithm 1 in AD-PSGD paper). The use of waiting in Line 5 of Algorithm 2 is to ensure that communication delays do not disrupt the algorithm in practice. When communication is "instantaneous", Line 5 vanishes and Algorithms 2 and 3 become equivalent to Algorithm 1. It is stated in AD-PSGD that their convergence results involve Algorithm 1 and not Algorithms 2 and 3: "This implementation avoids deadlock but still fits in the general algorithm Algorithm 1 we are analyzing".

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Reviewer 1Mgi Follow-up (Part 2)
comment: ## Handling Delayed Computations

> In the SWIFT algorithm, line 15 says client $i_t$ can use the neighbor's most up-to-date model for averaging, compute the gradient at the latest local model, and then update $x_{i_t}^{t+1}$.

There may be confusion due to the SWIFT algorithm being a global algorithm and not a specific client algorithm. In the SWIFT algorithm, Line 9 is where the gradient is computed for the active client. Line 12 denotes where the model fetching and averaging occurs. Line 15 is where the gradient from Line 9 and the averaged model from Line 12 are combined to form the client's next model. Once this is complete, the client will immediately broadcast its new model and begin computing its next gradient update.

> Then in iteration $t+1$, if client $i_t$ is the neighbor of active client $i_{t+1}$, then already $x_{i_t}^{t+1}$ can be used by $i_{t+1}$.
 
 As mentioned above (and in responses to other reviewers), we assume that the averaging, update, and subsequent model broadcast all happen in an atomic fashion. Therefore, once $i_t$ updates its model, its updated model will have been sent to all of its neighbors for them to use. 
 
 > But it means the gradients are computed in 1 iteration? How do you handle the delayed computation?

Delayed computations are handled by SWIFT through its communication matrix. When a client $i$ is performing gradient computations (Line 9), no other client $j$ can overwrite client $i$'s model when client $j$ performs model averaging. **This only occurs due to our novel non-doubly stochastic communication matrix**. Therefore, client $i$ will never be computing gradients with a delayed model.

SWIFT deals with a slow client $i$ (having delayed computations) by allowing a faster client $j$ to reuse a stored model of client $i$ until client $i$ finally finishes its model update. The stored model is still up to date (and not delayed) since client $i$ has not finished its gradient computations, updated its model, and sent out its updated model to neighboring clients (as it experiences delayed computations). If a client has not received a message from a neighbor, then their stored model for that neighbor is still its most up-to-date model.

In summary, the reviewer's argument would be correct if we enforced a doubly-stochastic communication matrix. As a novel contribution, SWIFT uses a non-doubly stochastic communication matrix which allows averaging amongst clients with their most up to date models.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_1Mgi
title: Follow up
comment: I would kindly disagree that AD-PSGD assumes "instantaneous". If the communication is "instantaneous", then in their algorithm 2, the computation thread does not have to wait for the communication thread. The update rule of AD-PSGD (at the bottom of page 6) happens **after** the communication is over, no matter whether the communication is "instantaneous" or not, c.f. their algorithm 3. Note that the queuing technique is to let the client do computation while waiting for updates to arrive, but it does not make communication instantaneous.

But even if we assume the communication is "instantaneous", the SWIFT algorithm looks strange. In the SWIFT algorithm, line 15 says client  $i_t$ can use the neighbor's most up-to-date model for averaging, compute the gradient at the latest local model, and then update $x_{i_t}^{t+1}$. Then in iteration t+1, if client $i_t$ is the neighbor of active client $i_{t+1}$, then $x_{i_t}^{t+1}$ already can be used by $i_{t+1}$. But it means the gradients are computed in 1 iteration? How do you handle the delayed computation?

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Reviewer 1Mgi Follow-up
comment: We greatly appreciate your feedback and engagement!

## Resolving Issue Regarding Communication Delay

**Response:**

> AD-PSGD does not assume these steps are instantaneous. 

We respectfully disagree  that AD-PSGD [R1] does not assume model fetching is instantaneous. We begin laying out our case by writing  the update rule of AD-PSGD (detailed in Algorithm 1 and the bottom of page 6 in [R1]) as:
$$
X_{k+1} = X_k W_k - \gamma \partial g(\hat{X}_k;\xi_k^{i_k},i_k).
$$

Before continuing, we would like to point out that $X_k W_k$ is the **model averaging of *the most up to date client models* and not previous client models**.

> For example, suppose the bandwidth is extremely small, the communication will be much slower than computation and there will be delayed models. They did not provide any measure to guarantee "instantaneous" but rather uses the bounded delay assumption.

Within Algorithm 3 of [R1], Line 5 states that the communication thread of worker $i$ *fetches $x^j$ from a neighbor $j$*. This model $x^j$ is then averaged with the current model $x^i$. Let's consider the same setting that is floated by the reviewer above: communication is much slower than computation. In this scenario, worker $i$ would be fetching **and averaging with** a model from worker $j$ that is delayed (since worker $j$ would have already performed a new model update due to slow communication). However, as shown in the update rule, *AD-PSGD is assuming averaging between clients with their most up to date models*. This leads us to final conclusion: ***AD-PSGD is assuming instantaneous fetching of neighboring client models.*** The bounded delay assumption in AD-PSGD regards computational delays between workers and *not communication delays*.


> In fact, in their algorithm 2, the computation thread has to wait for the communication thread if the batch size is fixed to M.

As the reviewer mentioned, AD-PSGD has the computational thread *wait* for the communication thread in certain scenarios (though they propose a gradient queue to try to get around this). **We do not have to deal with these computational wait times in SWIFT**. This is an ***important novelty of our work***, and a big improvement over AD-PSGD. 

In conclusion, SWIFT utilizes the same fetching assumptions as AD-PSGD without having to deal with computational wait times (and subsequent bounded delay assumptions).

We hope this clarifies the doubt. We are happy to clarify further if there are any concerns. If not, we would like to request the reviewer  to consider raising the scores.

## Citations
[R1] Lian, Xiangru, et al. "Asynchronous decentralized parallel stochastic gradient descent." International Conference on Machine Learning. PMLR, 2018.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_1Mgi
title: Reply
comment: Sorry for my late reply.

I have read your responses to my review and Cc5h's review. However, I am not convinced that your responses address my concern about asynchrony. I think the main confusion comes from the author's discussion on AD-PSGD in Cc5h's thread
> If not, we remark that models are fetched right before averaging them and completing an update. Like AD-PSGD, these steps (averaging and finalizing the update) are assumed to be done instantaneously in SWIFT. AD-PSGD specifically states in Section 3.3.1 and Appendix A that the averaging step is "atomic".

First, the "atomic" step in AD-PSGD is irrelevant to our discussion. AD-PSGD uses "atomic" averaging to avoid deadlock caused by gossip. But there is no deadlock in SWIFT.

Second, AD-PSGD does not assume these steps are instantaneous. For example, suppose the bandwidth is extremely small, the communication will be much slower than computation and there will be delayed models. They did not provide any measure to guarantee "instantaneous" but rather uses the bounded delay assumption. In fact, in their algorithm 2, the computation thread has to **wait** for the communication thread if the batch size is fixed to M. 

So my question can be somehow simplified as follows: when the bandwidth is very small (compared to computation power), do you wait at line 11 of algorithm 1 or do you use the locally stored model? It seems to me that if you wait, then algo 1 is blocking and can be unbounded. If you use a locally stored model, then the delayed model is used and Eq. (4) is incorrect --- $X^0$, $X^1$, ..., $X^t$ should appear on the right-hand side of Eq. (4).

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Reviewer Cc5h Follow-up on Atomic Averaging
comment: Dear Reviewer Cc5h,

We appreciate your speedy reply and feedback.

## Atomic Averaging is a Common Assumption

Below, we would like to provide some examples of how commonplace the atomic averaging assumption is within FL/decentralized optimization.

We begin by mentioning that AD-PSGD [R1], our closest related work (asynchronous decentralized FL), utilizes this assumption. In fact, like other synchronous FL results, AD-PSGD assumes that the averaging between *multiple* clients is atomic (Appendix A):
> "The theoretical guarantee of AD-PSGD relies on the the doubly stochastic property of matrix W. The implication is the averaging of the weights between two workers should be atomic."

In SWIFT, we only assume that averaging is atomic for one worker (since we do not require multiple workers to average simulatneously).

The works [R2, R3, R4, R5] are some of the first examples that came to mind of asynchronous decentralized or FL works which *explicitly* mention that they assume atomic model averaging with neighbors. 

In asynchronous parallel distributed optimization [R6] and FL communication protocol [R7] surveys, communication is often assumed as perfect in FL [R7] and involve atomic operations [R6]. In fact, it is stated within [R6] that:
> Asynchronous operations, on the other hand, are usually implemented in one of two ways: by mutual exclusion (mutex ) locks or atomic operations.

Finally, there has been some recent literature in FL [R8, R9, R10] which aim to achieve convergence under imperfect communication. The novelty of these works lie in their ability to strip away the *common assumption* that communication is perfect and reliable within FL settings. These works are extremely interesting, and provide an avenue for future work: guaranteeing theoretical convergence of SWIFT under imperfect communication settings.


## Citations
[R1] Lian, Xiangru, et al. "Asynchronous decentralized parallel stochastic gradient descent." International Conference on Machine Learning. PMLR, 2018.

[R2] Luo, Qinyi, et al. "Prague: High-performance heterogeneity-aware asynchronous decentralized training." Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 2020.

[R3] Nadiradze, Giorgi, et al. "Asynchronous decentralized SGD with quantized and local updates." Advances in Neural Information Processing Systems 34 (2021): 6829-6842.

[R4] van Dijk, Marten, et al. "Asynchronous federated learning with reduced number of rounds and with differential privacy from less aggregated gaussian noise." arXiv preprint arXiv:2007.09208 (2020).

[R5] Lu, Yucheng, Jack Nash, and Christopher De Sa. "Mixml: A unified analysis of weakly consistent parallel learning." arXiv preprint arXiv:2005.06706 (2020).

[R6] Assran, Mahmoud, et al. "Advances in asynchronous parallel and distributed optimization." Proceedings of the IEEE 108.11 (2020): 2013-2031.

[R7] Zhao, Zihao, et al. "Towards Efficient Communications in Federated Learning: A Contemporary Survey." arXiv preprint arXiv:2208.01200 (2022).

[R8] Ye, Hao, Le Liang, and Geoffrey Ye Li. "Decentralized federated learning with unreliable communications." IEEE Journal of Selected Topics in Signal Processing 16.3 (2022): 487-500.

[R9] Wei, Xizixiang, and Cong Shen. "Federated learning over noisy channels: Convergence analysis and design examples." IEEE Transactions on Cognitive Communications and Networking (2022).

[R10] Fan, Xin, et al. "Joint optimization of communications and federated learning over the air." IEEE Transactions on Wireless Communications (2021).

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_Cc5h
title: Reply
comment: Dear Authors,

Thanks for your clarification. As you assume the atomic operation, your algorithm is indeed asynchronous. All of my previous examples are related to the case that the fetch operations may not be atomic.

However, I am not sure if this assumption really makes sense in the FL setting: in the data center setting, I guess you can assume this, but for FL, communication is always the issue. Because of my lack of expertise in the system-wise applications of FL/decentralized optimization, I am not able to judge this assumption, and I will now keep the score and confidence and explicitly flag the discussion to AC and other reviewers to discuss. May raise the score later.

Minor: delayed update globally does not mean asynchronous? For sample, variance reduction methods like SVRG and SAGA also have the "delayed" update in some sense but are definitely synchronous.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Reviewer Cc5h Follow-up
comment: Dear Reviewer Cc5h,

Thank you for your reply and engagement! We greatly appreciate your feedback.

## SWIFT Asynchrony

**Response:**

Asynchronous algorithms allow the global model $x^{t-\\tau}$ stored by a client at time $t-\\tau$ to become stale or outdated as the client computes its gradient. The model becomes stale as other clients finish their updates, thereby changing the current model by the time the client is finished with its update at time $t$. **The "classical" asynchronous literature needs a bounded delay assumption in order to deal with the gradient update using a previous and outdated version $x^{t-\\tau}$:**
$$
x^{t+1} = x^t - \\eta \\nabla f(x^{t-\\tau}).
$$

Within the decentralized Federated Learning (FL) setting, clients store their own personal models and do not use a globally shared model. The global model equivalent in this setting is the average model amongst a client and its neighbors. In the case of synchronous decentralized FL, this average is recovered during the synchronization (model averaging) process and then subsequently used to perform the gradient update:
$$
\bar{x}^t = \frac{1}{n}\sum_{i=1}^n x_i^t,
$$
$$
x_i^{t+1} = x_i^t - \eta \nabla f (\bar{x}^t).
$$

The global model update can be found by taking the sum over all $n$ neighboring clients in the equation above
$$
\frac{1}{n}\sum_{i=1}^n x_i^{t+1} = \frac{1}{n}\sum_{i=1}^n x_i^t - \eta \nabla f (\bar{x}^t),
$$
$$
\bar{x}^{t+1} = \bar{x}^t - \eta \nabla f (\bar{x}^t).
$$

This phenomena ***does not occur in the asynchronous decentralized setting***, since synchronization is removed. Instead, we find that the global model is indeed being updated with a decayed gradient. Now, like other asynchronous algorithms, we will show that SWIFT does indeed update the global model using this decayed gradient.

Lets say we have two clients running SWIFT, Clients 1 and 2. Both start with the same model $x_1^0 = x_2^0$ at $t=0$. Client 2 is slow, and thus Client 1 performs the first three updates ($t=1,2,3$). Client 2 finally finishes an update, the 4th total update. The SWIFT update for this 4th update is detailed as:
$$
x^{3.5}_2 = \frac{x_1^{3} + x_2^{3}}{2} = \bar{x}^{3}, \text{ where } x_2^{3} = x_2^{0},
$$
$$
x_2^{4} = x_2^{3.5} - \eta \nabla f (x_2^{3}).
$$

As one can see, this is a typical gradient update with respect to Client 2's own local model. However, this local update can be rewritten as:
$$
x_2^{4} = \bar{x}^{3} - \eta \nabla f (\bar{x}^0).
$$

Now, one can immediately find the decayed update when looking at the global model update:
$$
\bar{x}^{4} - \bar{x}^{3} = \frac{x_1^{4} + x_2^{4} - x_1^{3} - x_2^{3}}{2} = \frac{ x_2^{4} - x_2^{3}}{2} = \frac{\bar{x}^{3} - \eta \nabla f (\bar{x}^0) - \bar{x}^0}{2},
$$
$$
\longrightarrow \quad \bar{x}^{4} = \bar{x}^{3} - \eta \Big(\frac{\nabla f (\bar{x}^0)}{2} - \frac{\bar{x}^3 - \bar{x}^0}{2\eta}\Big).
$$

As one can now see, the global model is updated with gradient information from a decayed model. Therefore, **the asynchrony in SWIFT is seen at the global level and not the local level.**

We would also like to add that the definition of a wait-free (and thereby asynchronous) algorithm is that **all operations must finish in a finite number of steps even if other threads are stopped**. All operations finish in a finite number of steps in SWIFT, even if other threads are stopped. In the two client example again, Client 1 still performs updates and makes progress even if Client 2 stops working. Client 1 would continually fetch the last model Client 2 sent before stopping, and use it during model averaging. Client 2 plays no role in keeping Client 1 from reaching $T$ total iterations. Therefore, SWIFT is a wait-free and asynchronous algorithm by definition.

Finally, the reviewer mentioned that for any asynchronous algorithm, it is possible for Client 2 to perform an update between the time Client 1 fetches its model and performs model averaging. We believe that the reviewer perhaps meant to float the scenario where Client 2 performs an update while Client 1 performs its gradient update (as we discuss at the beginning of our response). If not, we **remark** that models are fetched right before averaging them and completing an update. Like AD-PSGD, these steps (averaging and finalizing the update) are assumed to be done instantaneously in SWIFT. AD-PSGD specifically states in Section 3.3.1 and Appendix A that the averaging step is "atomic". This is a reasonable assumption, as model averaging only requires simple addition of models and finalizing the update requires subtraction of the already computed gradient. Thus, the scenario mentioned by the reviewer will never occur in SWIFT since once models are fetched the update is virtually complete.

## Thank You!

Thank you again for your feedback, and please let us know if we have cleared up the confusion and addressed your concerns. If so, we would appreciate it if you could update your score. If not, we would love to engage further.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_Cc5h
title: Further comments
comment: Besides, in your reply to other reviewers, I found this sentence: **If a client has not received a message from a neighbor, then their stored model for that neighbor is still its most up-to-date model.** This means that the broadcast is very fast and can be done in a single round? If not, then there must be the case that the fetched model may not be the most up-to-date model? However if you assume that the broadcast if very fast, then it is just not the asynchronous setting.

Sry again for the late reply. I don't know when the rebuttal ends (or if you cannot modify the pdf but can still discuss). Hope you can still clarify your settings.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_Cc5h
title: Still confusing
comment: Dear authors,

Thanks for your rebuttal and sorry for the late reply.

After reading your rebuttal, I am not convinced that SWIFT is an asynchronous algorithm. I totally understand how you define the round. However, in algorithm 1, it clearly says that if client 1 is the active client at round t, then client 1 will fetch the model in round t (line 11), mixing (line 12). However for the asynchronous algorithm, it should handle the case that although client 1 may be the active client at round $t$, it may fetch the model of client 2 at round maybe $t'$, and client 2 updates its model at some round $t''$ where $t' < t'' < t$. That's why the "classical" asynchronous literature needs the bounded delay assumption. This may happen if the broadcast from client 2 at t'' takes time and the message does not arrive at client 1 before round t, or just because computations go on, client 2 updated.

I also have a quick look at your proof, and it seems that the proof corroborates my understanding of algorithm 1. In Eq. 19 you write out the descent lemma, and the second term is exactly the mixing term. Then since you "assume" that the active client can fetch the model and update in the single round, you can then cancel the mixing term out in Eq. 20. The proof strategy seems totally the same as the synchronous decentralized optimization (but its much complicated/harder as it is the "partial participation" in the synchronous setting). However, if you consider the case I mentioned before: if the model is updated after client 1 fetch and before client 1 finishes, then you cannot go from Eq. 19 to Eq. 20 (or basically there is no Eq. 19).

Please point out if my understanding is wrong, and maybe directly point out the equations where you handle the case I mentioned before.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Paper Revision
comment: Dear Reviewer BZ9x,

Thank you for your reply. We are indeed able to submit a revision during the discussion period, and have just done so. We included the references in the revision as well as the rationale behind using a ring of cliques topology in Appendix A.4.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_BZ9x
title: thanks for the clarification
comment: Thanks for the clarification. I will keep the positive score.
I also recommend to add these references in the revision. I think the authors could submit revisions during the discussion period. Is that right?

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Rebuttal Follow-up
comment: Dear reviewers,

Thank you again for your thoughtful reviews. Have our responses addressed your questions? We would appreciate the opportunity to engage further.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer BZ9x (Part 1)
comment: # Response to Reviewer BZ9x

We want to thank the reviewer for their valuable reviewing service and dedication of time to the community. Below, we address all questions and misunderstandings the reviewer has.

## Ring of Clique (ROC) Topology

> In the experiments, 2 network topologies including ROC and ROC-xC are used. Are these 2 types of topologies common for the FL scenario? If so, is there any reference to support that? Otherwise, I would recommend the authors to explain why these 2 types of topologies are chosen for the experiments, or adding more experiments with other types of topologies.

**Response:**

There is a misunderstanding regarding the ring of clique topology. ROC-xC refers to the ring of clique topology with a given number $x$ clusters C. Thus, ROC-3C denotes a ring of cliques topology with 3 cliques. ROC and ROC-xC are the same topology (ROC-xC just describes the number of clusters).

We believe that the ring of cliques network topology is a realistic topology in the decentralized setting. In many real-world setting, like one's home (smart appliances, smart speakers/displays, phones, etc.), devices are connected together in a small clique. Only a small amount of these devices have connections to other devices outside the cluster (like a phone or router). We wanted to utilize this network topology due to this realistic nature.

Furthermore, there are papers that utilize this topology [R1, R2]. In fact, [R2] shows that a ring of clique topology can be used in the decentralized setting to reduce the  impact of label distribution skew.

Finally, we want to mention that we also utilize a ring topology within our experiments. This is a standard baseline topology used within many Federated Learning papers. 


## Affect of Topology on Convergence

> How is the convergence affected by the network topology, or the properties of the mixing matrix W (e.g., eigenvalues of its graph Laplacian)?

**Response:**

The connectivity of a topology absolutely affects the convergence rate of SWIFT. As discussed in the second paragraph of Section 5 as well as Appendix C, the 2nd largest eigenvalue of the expected client communication matrix $(\bar{W}^t)^T\bar{W}^t$ is denoted as $\rho$ ($\rho < 1$). A smaller $\rho$ results in information spreading faster throughout the network ($\rho = 0$ in centralized FL settings). In our analysis we use the constant $\rho_\nu \geq 775/4$, which contains $\rho$ (as well as the parameter $\nu$ which is specific to column stochastic matrices such as our client communication matrix $W_{i_t}^t$). Specifically, within Theorem 1, we show that SWIFT obtains the following convergence:
$$
\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E} ||\nabla f(\bar{x}^{t})||^2 \leq  \frac{2\sqrt{\Delta_f}}{T} + \frac{2\sqrt{L \Delta_f}\bigg(1 + \frac{1921}{20}\rho_\nu \big( \sigma^2 + 6\zeta^2 \sqrt{M} \big) \bigg)}{\sqrt{TM}}.
$$

When the network topology is well-connected, $\rho$ is very small and subsequently $\rho_\nu$ is close to 775/4. When the network topology is sparse, $\rho$ becomes closer to 1 and $\rho_\nu$ grows to become a larger constant. One can immediately see that having a sparse topology will increase the constant factor within our convergence result, and thus the number of iterations required to converge.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer BZ9x (Part 2)
comment: ## Expected Client-Communication Matrix

> The communication pattern of SWIFT actually have 2 kinds of uncertainty: 1. inactive clients; 2. randomly selected active client for communication. However, I could only find 1 expectation, namely the client-communication matrix. Does the expected client-communication matrix resolve the uncertainty for both inactive clients and random communication? If so, I believe that the expected client-communication matrix indicates the expected delay for all the clients. Given the expected delay, we surely do not need the upper bound of the delay for the theoretical analysis. I think this is a simple trick, but the authors' overclaim of the theoretical analysis without the bounded delay assumption may get the readers confused. I would recommend the authors to clarify it.

**Response:**

In SWIFT, *all* clients work simultaneously. An iteration occurs **whenever any client finishes its update**. Thus, each iteration of SWIFT is mapped to the active client which performed it. Overall, multiple clients are performing updates simultaneously at differing speeds, and the iteration count is updated whenever a client finishes its own update.

Due to this, SWIFT experiences uncertainty only in which client will perform the $t$-th iteration (i.e. uncertainty with which client will be the active client). Therefore, as mentioned in Section 5, we bound any delays by taking the expectation over the active client. The probability of each client $i$ being the active client is simply its sampling probability $p_i$. We therefore assume that each client $i$ is expected to perform updates at its prescribed sampling probability rate $p_i$.

### Citations

[R1] Jeon, Beomyeol, et al. "Privacy-preserving decentralized aggregation for federated learning." IEEE INFOCOM 2021-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 2021.

[R2] Bellet, Aurélien, Anne-Marie Kermarrec, and Erick Lavoie. "D-Cliques: Compensating for Data Heterogeneity with Topology in Decentralized Federated Learning." (2021).

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer KnPQ (Part 1)
comment: # Response to Reviewer KnPQ

We want to thank the reviewer for their valuable reviewing service and dedication of time to the community. Below, we address all questions and misunderstandings the reviewer has.

## Asynchronous Contribution

> Asynchronous decentralized training is already a mature technique. As the authors mentioned in the paper, several papers already proposed, such as AD-PSGD, and [1]. After reading the paper, it is unclear why do we need a new asynchronous algorithm. I understand that it is an important contribution to remove the bounded delay assumption. But it should be clear whether the algorithmic changes or new analysis techniques enable this theory advancement. The authors are supposed to highlight the insights on how and why.

**Response:**

We agree that asynchronous decentralized training is already shown to be effective and that one of SWIFT's novel contributions is to improve upon these current methods by removing the bounded delay assumption. However, we want to also spotlight Table 1 within our paper which shows that SWIFT also: (1) uses only local memory, (2) allows for asynchronous neighborhood averaging, and (3) allows for periodic communication unlike AD-PSGD [R1] and other methods.

Other asynchronous decentralized methods use gossip-style communication (like AD-PSGD). Another novel contribution of SWIFT, is that we carefully construct our communication matrix and protocol to allow for neighborhood averaging while still ensuring convergence. This form of communication no longer requires two clients to exchange models simultaneously as in AD-PSGD. As a result, SWIFT is wait-free and greatly reduces communication time (by removing the idle time in waiting for slow clients) as shown in our experiments.

## Definition of Communication Time per Round

> The authors mentioned several places (e.g. in the abstract) that SWIFT reduces the communication time per round. But this statement is not accurate. As an asynchronous algorithm, SWIFT does not change the communication time but just reduce/remove the idle time in waiting for slow workers. I suggest the authors to refine the related statements.

**Response:**

Within our work, we count waiting for communications from slower clients as part of the communication process. Therefore, we include this idling as part of the communication time. We agree that as an asynchronous algorithm, SWIFT removes the the idle time in waiting for slow workers. We will highlight that the reduction in communication time stems from the reduction in idling time in the final version of our manuscript.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer KnPQ (Part 2)
comment: ## Active Client Confusion

> I did not get why SWIFT needs to randomly select an active client at each round. If so, there must be some idle time at some clients, as they are not selected but may already finish computation. In practice, the active clients should be determined by the randomness in the client computation time. Once a client finishes local computation, it can trigger a communication and be labeled as the active clients. Server cannot control this process. The authors are supposed to provide clarifications on this.

**Response:**

The misunderstanding arises from the difficulties of describing an asynchronous process algorithmically (Algorithm 1 in [R1], AD-PSGD, also suffers from this issue). Algorithm 1 in our paper comes from the prespective of the *active client*, or the specific client that finishes the $t$-th global iteration. Each iteration $t$ corresponds to *one* client finishing its update (while all other clients are still working on their own updates at that moment in time).

Therefore, in SWIFT, *all* clients work simultaneously.  Clients perform updates at differing speeds (we detail this phenomenon in Figure 1). In our analysis, an iteration occurs **whenever any client finishes its update**. Thus, each iteration in Algorithm 1 is mapped to the active client which performed it.

In summary, this does **not** mean that only one client is working at a time. Multiple clients are performing updates simultaneously at differing speeds, and the iteration count is updated whenever a client finishes its own update.

To provide an example of what we state above, let's say we have 3 clients: Client 1 performs one update every 0.6 seconds, Client 2 performs one update every 1.1 seconds, and Client 3 performs one update every 1.6 seconds. When SWIFT begins, all clients begin computing their updates simultaneously. After 0.6 seconds, Client 1 finishes its first update, and is therefore designated as the *active client* for the first iteration $t=1$ (Client 1 completed the first iteration). Client 1 then immediately begins its second update. After 1.1 seconds, Client 2 finishes its first update and is designated as the *active client* for the second iteration $t=2$. Client 2 then immediately begins its second update. After 1.2 seconds, Client 1 finishes its second update and is designated as the *active client* for the third iteration $t=3$. After 1.6 seconds, Client 3 finishes its first update and is designated as the *active client* for the fourth iteration $t=4$. This process repeats until convergence.

### Citations
[R1] Lian, Xiangru, et al. "Asynchronous decentralized parallel stochastic gradient descent." International Conference on Machine Learning. PMLR, 2018.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer Cc5h (Part 1)
comment: We want to thank the reviewer for their valuable reviewing service and dedication of time to the community. Below, we address all questions and misunderstandings the reviewer has.

## Asynchronous Setting Misunderstanding

> I think the setting in this paper considered is not the asynchronous setting, at least from the theory part. This is more like the "partial participation" case in the distributed optimization/federated learning but not the asynchronous setting, and thus I still think that it is a synchronous algorithm (or at least the analysis is for the synchronous setting). The reason is that, this paper seems to require that only a single agent/client can do the update during each round, but the asynchronous setting does not require this and the other clients can start updating even before the other client ends. Nonetheless, partial participation/client sampling is still interesting for decentralized optimization algorithms. From an empirical point of view, this partial participation/client sampling can indeed make communication faster (because do not need to wait for all), but from the theory side, it is very different from the asynchronous setting.

**Response:** SWIFT *does* consider the asynchronous setting. The misunderstanding arises from the difficulties of describing an asynchronous process algorithmically (Algorithm 1 in [R1], AD-PSGD, also suffers from this issue). Algorithm 1 comes from the prespective of the *active client*, or the specific client that finishes the $t$-th global iteration. Each iteration $t$ corresponds to *one* client finishing its update (while all other clients are still working on their own updates at that moment in time).

Therefore, in SWIFT, *all* clients work simultaneously. As mentioned by the reviewer, clients in the asynchronous setting can begin performing updates even before other clients end (we detail this phenomenon in Figure 1). In our analysis, an iteration occurs **whenever any client finishes its update**. Thus, each iteration in Algorithm 1 is mapped to the active client which performed it.

In summary, this does **not** mean that only one client is working at a time. Multiple clients are performing updates simultaneously at differing speeds, and the iteration count is updated whenever a client finishes its own update.

To provide an example of what we state above, let's say we have 3 clients: Client 1 performs one update every 0.6 seconds, Client 2 performs one update every 1.1 seconds, and Client 3 performs one update every 1.6 seconds. When SWIFT begins, all clients begin computing their updates simultaneously. After 0.6 seconds, Client 1 finishes its first update, and is therefore designated as the *active client* for the first iteration $t=1$ (Client 1 completed the first iteration). Client 1 then immediately begins its second update. After 1.1 seconds, Client 2 finishes its first update and is designated as the *active client* for the second iteration $t=2$. Client 2 then immediately begins its second update. After 1.2 seconds, Client 1 finishes its second update and is designated as the *active client* for the third iteration $t=3$. After 1.6 seconds, Client 3 finishes its first update and is designated as the *active client* for the fourth iteration $t=4$. This process repeats until convergence.

## Experimental Setup

> It is not clear how the experiment is done (even in the appendix): for example if there is a notion of a round in the implementation. If yes (and it is exactly what is show in the algorithm), then the implementation is also not asynchronous but synchronous, and it should be compared to the synchronous algorithms.

We believe that the asynchronous setting clarification above may resolve the major questions that the reviewer has with our experiments. The iteration counter $t$ is incremented any time a client completes a local gradient update. All clients asynchronously perform local updates and neighborhood averaging until convergence.

We chose to compare SWIFT with D-SGD, as it is the foundational and standard decentralized federated baseline. Furthermore, many other decentralized FL algorithms compare their performance against D-SGD. This is especially cruical because we were unable to reproduce AD-PSGD (as no working code exists). Therefore, we had to compare relative performance between AD-PSGD and SWIFT to D-SGD. We agree with the reviewer that comparing SWIFT with D2 would be interesting, especially in the heterogeneous data settings. Adding more baseline comparisons would continue to showcase the run-time superiority of SWIFT as compared to non wait-free algorithms.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer Cc5h (Part 2)
comment: ## Bound on Total Iterations $T$

> For theory, $T$ needs to be larger than $n^2$, which seems not tight. Why the theory needs $T \geq \Omega(n^2)$. Naively thinking, $\mathcal{\tilde{O}}(n)$rounds should be equivalent to a single round in the synchronous setting, and thus intuitively, $T$ should only larger than $\Omega(n)$?

We agree that the bound on $T$ may not be as tight as possible. However, we tighten the bound further compared to the state-of-the-art asynchronous decentralized AD-PSGD (as mentioned in Corollary 1). In future work, we aim to further decrease the dependency on $n$ and make the bound tighter.

## Data Heterogeneity

> The theory needs the bounded dissimilarity between each client/agent, and thus may not handle arbitrary heterogeneity.


The use of a inter-client gradient variance bound is common within many decentralized FL works.  Currently, no asynchronous decentralized FL algorithm avoids such a constraint. We agree that achieving a convergence result without such bound, as in D2, would be a significant theoretical result. We aim to achieve such a result in future work.


### Rebuttal Conclusion
We again want to thank the reviewer for their time and effort in the reviewing process. We hope to have alleviated the misunderstanding surrounding the asynchronous nature of SWIFT. We believe that we have addressed all of the reviewer's concerns, and would be happy to discuss any further questions that remain.

### Citations
[R1] Lian, Xiangru, et al. "Asynchronous decentralized parallel stochastic gradient descent." International Conference on Machine Learning. PMLR, 2018.

From: ICLR.cc/2023/Conference/Paper2216/Authors
title: Response to Reviewer 1Mgi
comment: We want to thank the reviewer for their valuable reviewing service and dedication of time to the community. Below, we address all questions and misunderstandings the reviewer has.

## Algorithm 1 Asynchronous Misunderstanding


>Algorithm 1 is not wait-free according to Line 11 and 12. As far as I can see, worker i has to wait for the models from all of its neighbors before averaging. If any update of its neighbor is delayed, which could be an unbounded delay, then worker i is blocked. Then in the future communication rounds, all neighbors of i are blocked as they have to wait for worker i for updates. I didn't find how the asynchrony is actually addressed in the algorithm.

**Response:** 

SWIFT is a wait-free algorithm due to its non-blocking and asynchronous model communication. At the beginning of every local update, each client asynchronously broadcasts its local model to their neighbors (Line 7 of Algorithm 1). Each client stores a copy of their neighbors' model. Before performing model averaging, a client checks for any new model updates from its neighbors (Line 11). **If a client has not received a message from a neighbor, then their stored model for that neighbor is still its most up-to-date model**.


Thus, the model fetching and averaging process within Lines 11 and 12 **do *not* require** a client to wait for all of their neighboring client models. This ensures that there is no waiting for slower clients, and thus no parallelization delays. Specifically, the asynchronous averaging process is detailed mathematically in Equation (4) given as:
$$
X^{t+1} =  X^t W^t_{i_t} - \gamma G(x^t_{i_t}, \xi^t_{i_t}).
$$

Within the equation above, the gradient matrix $G$ is zero everywhere except for the column corresponding to the gradient update of the active client $i_t$. The client-communication matrix $W_{i_t}$ is neither doubly-stochastic nor symmetric, allowing clients to average without having to wait for their neighbors. $W_{i_t}$ is defined in Equation (5) as:
$$
W^t_{i_t} := I_n + (w_{i_t}^t - e_{i_t})  e_{i_t}^\intercal, \;
    w_{i_t}^t := [w_{1, i_t}^t, \ldots,  w_{n, i_t}^t]^\intercal \in \mathbb{R}^n, \; \sum_{j=1}^n w_{j,i} = 1, \; w_{i,i} \geq 1/n \; \forall i.
$$

If the asynchronous averaging process defined above was synchronous, then $G$ would be a fully dense matrix containing the gradient updates to *all* clients at once.
Furthermore, the client-communication matrix $W_{i_t}$ would be symmetric and doubly-stochastic. This would require all model averaging to be done together in a synchronized fashion.

## Theoretical Benefit (Linear Speed-up)
 

>There is no theoretical benefit to use SWIFT. The convergence rate is  which does not scale with the number of workers. It means SWIFT is not benefiting from the collaborative training and only have the same rate as training alone.

**Response:** We emphasize that SWIFT *does* obtain a linear speed-up with respect to the number of clients. We obtain this speed-up in the same way as AD-PSGD [R1]. The proof of this is summarized in Remark 4 of [R1], and we expand upon it further below:

An algorithm achieves an $\epsilon$-approximate solution if
$$
\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}||\nabla f (\bar{x}^t)||^2 \leq \epsilon.
$$

SWIFT's convergence rate in Theorem 1 (as well as AD-PSGD's) is consistent with SGD and is on the order of (dominated by) $1/\sqrt{T}$. Therefore, the total computational complexity to achieve an $\epsilon$-approximate solution is bounded by $\mathcal{O}(\frac{1}{\epsilon^2})$. As the number of clients does not affect the total complexity, a single client only shares a computational complexity of $\mathcal{O}(\frac{1}{n\epsilon^2})$. Therefore, ***linear speed-up with respect to the number of clients is achieved by SWIFT in terms of computational complexity***. We will highlight this in the final version of our manuscript.

### Rebuttal Summary
We again want to thank the reviewer for their time and effort in the reviewing process. We hope to have alleviated the misunderstanding surrounding the asynchronous nature of SWIFT, where clients store the most recent neighboring models and communicate in a non-blocking and wait-free manner. Finally, we wanted to clarify the linear speed-up which SWIFT (following along the lines of other Federated algorithms) obtains with respect to the number of clients. As mentioned in these other works, having more clients will provide more gradient updates and allow the total computational complexity to be reached faster. 


We believe that we have addressed all of the reviewer's concerns, and would be happy to discuss any further questions that remain.


### Citations

[R1] Lian, Xiangru, et al. "Asynchronous decentralized parallel stochastic gradient descent." International Conference on Machine Learning. PMLR, 2018.

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_KnPQ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a new asynchronous decentralized training algorithm with local updates, named SWIFT. Compared to previous works in this direction, the authors removed the bounded delay assumption in analysis, and hence, obtained better theoretical results. In order to ensure the convergence, a new technique is introduced to allow non-symmetric non-doubly stochastic mixing matrices. At the end, experiments on CIFAR10 dataset with a cluster of machines validate the superiority of the proposed algorithm.
strength_and_weaknesses: ## Strength
1. The authors removed the bounded delay assumption in nearly all previous works when analyzing asynchronous decentralized training algorithms.
1. In order to ensure convergence, several new techniques are introduced to deal with non-symmetric and non-doubly stochastic mixing matrices.
1. The authors are able to run experiments on a real cluster of machines, instead of just simulating runtimes on a single machine. So the experimental results are more convincing.


## Weakness
1. Asynchronous decentralized training is already a mature technique. As the authors mentioned in the paper, several papers already proposed, such as AD-PSGD, and [1]. After reading the paper, it is unclear why do we need a new asynchronous algorithm. I understand that it is an important contribution to remove the bounded delay assumption. But it should be clear whether the algorithmic changes or new analysis techniques enable this theory advancement. The authors are supposed to highlight the insights on how and why. 
2. The authors mentioned several places (e.g. in the abstract) that SWIFT reduces the communication time per round. But this statement is not accurate. As an asynchronous algorithm, SWIFT does not change the communication time but just reduce/remove the idle time in waiting for slow workers. I suggest the authors to refine the related statements.
3. I did not get why SWIFT needs to randomly select an active client at each round. If so, there must be some idle time at some clients, as they are not selected but may already finish computation. In practice, the active clients should be determined by the randomness in the client computation time. Once a client finishes local computation, it can trigger a communication and be labeled as the active clients. Server cannot control this process. The authors are supposed to provide clarifications on this.
4. If the above point is true, then that means the actual running SWIFT is different from what the authors analyze. The authors model the randomness in computing time by randomly selecting active clients. This can be a very rough approximation. The authors should justify why this theoretical approximation make sense and why this assumption is better than bounded delay assumption. Also, it should be made clear in the paper that their analysis is applied to a variant (possibly simplified version) of SWIFT.
clarity,_quality,_novelty_and_reproducibility: Overall the paper is well-written but I found that some details are unclear. While asynchronous decentralized training is not new, it is interesting to see a new variant, which does not need bounded delay assumption.
summary_of_the_review: I believe this paper make a good contribution in advancing asynchronous decentralized training algorithm, but some technical details are unclear such that I cannot accurately evaluate its significance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_1Mgi
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper considers asynchronous training in the absence of a central server. They propose a novel algorithm SWIFT to address the problem and does not rely on the common bounded delay assumption. Convergence rates and empirical validations are provided 
strength_and_weaknesses: There are two aspects that concern me the most:

- Algorithm 1 is not wait-free according to Line 11 and 12. As far as I can see, worker i has to wait for the models from all of its neighbors before averaging. If any update of its neighbor is delayed, which could be an unbounded delay, then worker i is blocked.  Then in the future communication rounds, all neighbors of i are blocked as they have to wait for worker i for updates. I didn't find how the asynchrony is actually addressed in the algorithm.

- There is no theoretical benefit to use SWIFT. The convergence rate is $O(1/sqrt{T})$ which does not scale with the number of workers $n$.  It means SWIFT is not benefiting from the collaborative training and only have the same rate as training alone.
clarity,_quality,_novelty_and_reproducibility: There are two important questions that concern me. 
summary_of_the_review: I have listed two questions that concern me. I recommend rejection if the authors cannot address them. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_BZ9x
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper propose a novel wait-free decentralized FL algorithm, which is a decentralized SGD algorithm with heterogeneous local updates and asynchronous local model synchronization. Theoretical analysis is provided for both iid and non-iid cases. The empirical results show good performance.
strength_and_weaknesses: Strength:
1. This paper propose a novel wait-free decentralized FL algorithm, which is a decentralized SGD algorithm with heterogeneous local updates and asynchronous local model synchronization. 
2. A novel client-communication weight selection algorithm is proposed.
3. Theoretical analysis is provided for both iid and non-iid cases. 
4. The empirical results show good performance.

Weakness (concerns and questions):
1. In the experiments, 2 network topologies including ROC and ROC-xC are used. Are these 2 types of topologies common for the FL scenario? If so, is there any reference to support that? Otherwise, I would recommend the authors to explain why these 2 types of topologies are chosen for the experiments, or adding more experiments with other types of topologies.
2. How is the convergence affected by the network topology, or the properties of the mixing matrix W (e.g., eigenvalues of its graph Laplacian)?
3. The communication pattern of SWIFT actually have 2 kinds of uncertainty: 1. inactive clients; 2. randomly selected active client for communication. However, I could only find 1 expectation, namely the client-communication matrix. Does the expected client-communication matrix resolve the uncertainty for both inactive clients and random communication? If so, I believe that the expected client-communication matrix indicates the expected delay for all the clients. Given the expected delay, we surely do not need the upper bound of the delay for the theoretical analysis. I think this is a simple trick, but the authors' overclaim of the theoretical analysis without the bounded delay assumption may get the readers confused. I would recommend the authors to clarify it.
clarity,_quality,_novelty_and_reproducibility: In overall, I think this is a good paper with thorough theoretical and empirical analysis. 
The supplementary material provides the source code and scripts to reproduce the experiments.
Though I would recommend the authors to clarify how they get rid of the bound-delay assumption in the theoretical analysis, I don't think it is a big issue.
summary_of_the_review: In overall, I think this is a good paper with thorough theoretical and empirical analysis. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2216/Reviewer_Cc5h
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes SWIFT, which is claimed to be a wait-free asynchronous decentralized optimization algorithm. This paper provides a theory for the convergence rate and uses experiments to show the effectiveness of the proposed algorithm.
strength_and_weaknesses: Strength:
1. The proposed algorithm uses the ideas from local steps, which is well applied in federated learning scenarios.
2. In the theory part, it tries to remove the bounded delay assumption used by most other asynchronous optimization algorithms. (See the weakness part for more discussion on this)
3. The numerical experiment result seems interesting if the authors can justify their experiment is actually in the asynchronous setting. (See the weakness part for more discussion)

Weakness:
1. (Major) I think the setting in this paper considered is not the asynchronous setting, at least from the theory part. This is more like the "partial participation" case in the distributed optimization/federated learning but not the asynchronous setting, and thus I still think that it is a synchronous algorithm (or at least the analysis is for the synchronous setting). The reason is that, this paper seems to require that only a single agent/client can do the update during each round, but the asynchronous setting does not require this and the other clients can start updating even before the other client ends. Nonetheless, partial participation/client sampling is still interesting for decentralized optimization algorithms. From an empirical point of view, this partial participation/client sampling can indeed make communication faster (because do not need to wait for all), but from the theory side, it is very different from the asynchronous setting.
2. (Major) It is not clear how the experiment is done (even in the appendix): for example if there is a notion of a round in the implementation. If yes (and it is exactly what is show in the algorithm), then the implementation is also not asynchronous but synchronous, and it should be compared to the synchronous algorithms.
3. (Minor) Why not compare to the best synchronized decentralized algorithms like D2? D2 can better handle the data heterogeneity issue than D-SGD.
4. (Minor) The theory needs the bounded dissimilarity between each client/agent, and thus may not handle arbitrary heterogeneity.
5. (Minor) For theory, T needs to be larger than $n^2$, which seems not tight.
clarity,_quality,_novelty_and_reproducibility: Clarity:
Based on the review and opinion in the weakness part, I don't think that the author did a great job making the setting of this paper clear (between the asynchronous and synchronous settings). The writing for other parts, e.g. proofs, algorithms, and experiments except the settings, is very clear to me.

Quality:
For the theory part, I don't think that the contribution is very significant because I don't think the setting is asynchronous. (However, the client sampling/partial participation in decentralized optimization also seems interesting to me, see the summary part for more discussions and questions). For the empirical part, the results and experiment seem interesting to me, but my concern about the setting still holds.

Novelty:
Even though the setting is a little bit confusing to me, it is still interesting to consider for theoretical and empirical people. However, the synchronous nature of the setting should make the analysis much easier than the real asynchronous setting.

Reproducibility:
Can be reproduced.
summary_of_the_review: Here I list the summary and my questions. I hope that the authors can point out my misunderstandings about the settings or clarify my questions during the author's response.

1. (Major) The setting. To me, the setting considered in this paper is the synchronous setting, not the asynchronous setting. The asynchronous setting considered in [1] allows clients to grab the model "before" another client ends the round. The author should discuss much more why the setting considered in this paper is also asynchronous setting, e.g., by giving more citations.
2. (Major) The author should also discuss how the experiment is done, e.g., if a client needs to wait until another client finishes its round. From the algorithm, it seems not but for the general understanding of the asynchronous algorithm, it should be yes.
3. For comparison, I suggest the authors to compare with the best synchronous optimization algorithms that can also deal with data heterogeneity, e.g., D2 [2].
4. Why the theory needs $T \ge \Omega(n^2)$. Naively thinking, $\tilde O(n)$ rounds should be equivalent to a single round in the synchronous setting, and thus intuitively, $T$ should only larger than $\Omega(n)$?

Given the questions above, I believe this paper does not reach the acceptance bar at this time. I am happy to adjust the score after the authors point out my conceptual misunderstanding or answer my questions.


Citations

[1] Asynchronous Stochastic Gradient Descent with Delay Compensation. Zheng et al., 2016

[2] D2 : Decentralized Training over Decentralized Data. Tang et al., 2018
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold