Original: Data-heterogeneity-aware Mixing for Decentralized Learning
Revised: Data-heterogeneity-aware Mixing for Decentralized Learning

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: Decentralized learning provides an effective framework to train machine learning models with data distributed over arbitrary communication graphs.
Revised: Decentralized learning provides an effective framework to train machine learning models with data distributed over arbitrary communication graphs.

Original: However, most existing approaches towards decentralized learning disregard the interaction between data heterogeneity and graph topology.
Revised: However, most existing approaches towards decentralized learning disregard the interaction between data heterogeneity and graph topology.

Original: In this paper, we characterize the dependence of convergence on the relationship between the mixing weights of the graph and the data heterogeneity across nodes.
Revised: In this paper, we characterize the dependence of convergence on the relationship between the mixing weights of the graph and the data heterogeneity across nodes.

Original: We propose a metric that quantifies the ability of a graph to mix the current gradients.
Revised: We propose a metric that quantifies the ability of a graph to mix the current gradients.

Original: We further prove that the metric controls the convergence rate, particularly in settings where the heterogeneity across nodes dominates the stochasticity between updates for a given node.
Revised: We further prove that the metric controls the convergence rate, particularly in settings where the heterogeneity across nodes dominates the stochasticity between updates for a given node.

Original: Motivated by our analysis, we propose an approach that periodically and efficiently optimizes the metric using standard convex constrained optimization and sketching techniques
Revised: Motivated by our analysis, we propose an approach that periodically and efficiently optimizes the metric using standard convex constrained optimization and sketching techniques.

Original: 1. Introduction
Revised: 1. Introduction

Original: Machine learning is gradually shifting from classical centralized training to decentralized data processing.
Revised: Machine learning is gradually shifting from classical centralized training to decentralized data processing.

Original: For example, federated learning (FL) allows multiple parties to jointly train an ML model together without disclosing their personal data to others [18].
Revised: For example, federated learning (FL) allows multiple parties to jointly train an ML model together without disclosing their personal data to others [18].

Original: While FL training relies on a central coordinator, fully distributed learning methods instead use direct peer-to-peer communication between the parties (e.g. personal devices, organization, or compute nodes inside a datacenter)
Revised: While FL training relies on a central coordinator, fully distributed learning methods instead use direct peer-to-peer communication between the parties (e.g. personal devices, organization, or compute nodes inside a datacenter)

Original: In decentralized learning, communication is limited to the network topology.
Revised: In decentralized learning, communication is limited to the network topology.

Original: The nodes can only communicate with their direct neighbors in the network in each round of (one hop) communication [47].
Revised: The nodes can only communicate with their direct neighbors in the network in each round of (one hop) communication [47].

Original: The convergence in decentralized learning with distributed SGD [D-SGD, 27] crucially depends on two factors (i) the spread of information, i.e. many rounds (hops) of communication are required to spread information to all nodes in the network [3, 48], and (ii) the heterogeneity of the data sources, i.e. when local data on each node is drawn from different distributions [6, 16, 19, 52].
Revised: The convergence in decentralized learning with distributed SGD [D-SGD, 27] crucially depends on two factors (i) the spread of information, i.e. many rounds (hops) of communication are required to spread information to all nodes in the network [3, 48], and (ii) the heterogeneity of the data sources, i.e. when local data on each node is drawn from different distributions [6, 16, 19, 52].

Original: The influence of the first factor is usually expressed through the spectral gap of the mixing matrix [3, 13, 35, 39, 56].
Revised: The influence of the first factor is usually expressed through the spectral gap of the mixing matrix [3, 13, 35, 39, 56].

Original: However, spectral gap only provides a worst case diffusion rate independent of the data heterogeneity.
Revised: However, spectral gap only provides a worst case diffusion rate independent of the data heterogeneity.

Original: The effect of data-heterogeneity is commonly incorporated through the variance of the gradients for the objectives across nodes [22], disregarding the relationship between nodes due to the underlying topology.
Revised: The effect of data-heterogeneity is commonly incorporated through the variance of the gradients for the objectives across nodes [22], disregarding the relationship between nodes due to the underlying topology.

Original: Instead of viewing these factors independently, we propose an analysis that incorporates the interaction between the graph topology and the data heterogeneity.
Revised: Instead of viewing these factors independently, we propose an analysis that incorporates the interaction between the graph topology and the data heterogeneity.

Original: Concretely, we refine the theoretical analysis of D-SGD to reveal precisely the tight interplay between the graph’s mixing matrix and the time-varying distribution of gradients across nodes.
Revised: Concretely, we refine the theo- retical analysis of D-SGD to reveal precisely the tight interplay between the graph’s mixing matrix and the time-varying distribution of gradients across nodes.

Original: We prove that the above interaction affects the convergence through a metric, that we call relative heterogeneity .
Revised: We prove that the above interaction affects the convergence through a metric, that we call relative heterogeneity .

Original: The metric quantifies the© . error between the average of the gradient across all the nodes and the updates obtained by each node when using the given mixing weights to average the gradients of its neighbours at the same value of the parameters.
Revised: The metric quantifies the error between the average of the gradient across all the nodes and the updates obtained by each node when using the given mixing weights to average the gradients of its neighbours at the same value of the parameters.

Original: Crucially, the metric doesn’t depend on the magnitude of the consensus and the stochastic noise in the gradients, allowing us to disentangle the precise effect of data heterogeneity on the convergence.
Revised: Crucially, the metric doesn’t depend on the magnitude of the consensus and the stochastic noise in the gradients, allowing us to disentangle the precise effect of data heterogeneity on the convergence.

Original: Motivated by our theoretical analysis, we propose an approach to optimize the mixing matrix to have low relative heterogeneity .
Revised: Motivated by our theoretical analysis, we propose an approach to optimize the mixing matrix to have low relative heterogeneity .

Original: In contrast to existing works attempting to tackle dataheterogeneity through specialized algorithms [28, 30, 38, 46], we investigate how the performance of D-SGD can be improved by a time-varying and data-aware design of the mixing matrix (while respecting the network topology).
Revised: In contrast to existing works attempting to tackle dataheterogeneity through specialized algorithms [28, 30, 38, 46], we investigate how the performance of D-SGD can be improved by a time-varying and data-aware design of the mixing matrix (while respecting the network topology).

Original: We show that the design of an optimal data-dependent mixing matrix can be described as a quadratic program that can efficiently be solved.
Revised: We show that the design of an optimal data-dependent mixing matrix can be described as a quadratic program that can efficiently be solved.

Original: Subsequently, we combine the above optimization with D-SGD to dynamically adapt the mixing matrix during training .While
Revised: Subsequently, we combine the above optimization with D-SGD to dynamically adapt the mixing matrix during training .While

Original: our approach only updates the mixing matrix, our metric can also be used to select different topologies for a given heterogeneity across nodes as well as different permutations of the nodes for a given network (see Appendix G).
Revised: our approach only updates the mixing matrix, our metric can also be used to select different topologies for a given heterogeneity across nodes as well as different permutations of the nodes for a given network (see Appendix G).

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: We provide a tighter convergence analysis of DSGD by introducing a new metric that captures the interplay between the communication topology and data heterogeneity in decentralized (and federated) learning.
Revised: We provide a tighter convergence analysis of DSGD by introducing a new metric that captures the interplay between the communication topology and data heterogeneity in decentralized (and federated) learning.

Original: We propose a communication and computation efficient algorithm to design data-aware mixing matrices in practice and verify its effectiveness through a set of extensive experiments on synthetic and real data.
Revised: We propose a communication and computation efficient algorithm to design data-aware mixing matrices in practice and verify its effectiveness through a set of extensive experiments on synthetic and real data.

Original: 2. Related Work
Revised: 2. Related Work

Original: Decentralized convex optimization over arbitrary network topologies has been studied in [13, 34, 47, 53] and decentralized versions of the stochastic gradient method (D-SGD) have been analyzed in [22, 26, 27, 49].
Revised: Decentralized convex optimization over arbitrary network topologies has been studied in [13, 34, 47, 53] and decentralized versions of the stochastic gradient method (D-SGD) have been analyzed in [22, 26, 27, 49].

Original: It was found that the convergence of D-SGD is strongly affected by heterogeneous data.
Revised: It was found that the convergence of D-SGD is strongly affected by heterogeneous data.

Original: Such impacts are not only observed in practice [16, 27], but also verified theoretically by theoretical complexity lower bounds [19, 22, 55].Concurrent to our work, [5] provided a similar analysis of the convergence rate of D-SGD for the smooth convex case using a metric quantifying the mixing error in gradients named “neighborhood heterogeneity”.
Revised: Such impacts are not only observed in practice [16, 27], but also verified theoretically by theoretical complexity lower bounds [19, 22, 55].Concurrent to our work, [5] provided a similar analysis of the convergence rate of D-SGD for the smooth convex case using a metric quantifying the mixing error in gradients named “neighborhood heterogeneity”.

Original: We discuss this work and other related works in more detail in Appendix D
Revised: We discuss this work and other related works in more detail in Appendix D

Original: 3. Setup
Revised: 3. Setup

Original: We consider optimizing the sum structured minimization objective distributed over n nodes or workers/clients:
Revised: We consider optimizing the sum structured minimization objective distributed over n nodes or workers/clients:

Original: [Equation]
Revised: [Equation]

Original: where the functions f i ( x ) =
Revised: where the functions f i ( x ) =

Original: E ξ ∼D i F i ( x , ξ i ) denote the stochastic objectives locally stored on every node i .
Revised: E ξ ∼D i F i ( x , ξ i ) denote the stochastic objectives locally stored on every node i .

Original: In machine learning applications, this corresponds to minimizing an empirical loss f averaged over all local losses f i , with D i being a distribution over the local dataset on node i .
Revised: In machine learning applications, this corresponds to minimizing an empirical loss f averaged over all local losses f i , with D i being a distribution over the local dataset on node i .

Original: We define a communication graph G =
Revised: We define a communication graph G =

Original: ( V, E ) with | V | = n nodes.
Revised: ( V, E ) with | V | = n nodes.

Original: Following a convention in  decentralized literature [e.g. 56], we define a mixing matrix W ∈ R n × n as a weighted adjacency matrix of G withthe weights w ij ∈ [0 , 1] , w ij > 0 only if ( i, j ) ∈ E and the matrix is doubly stochastic (cid:80)ni
Revised: Following the convention in the decentralized literature [e.g. 56], we define a mixing matrix W ∈ R n × n as a weighted adjacency matrix of G with w ij ∈ [0 , 1] , w ij > 0 only if ( i, j ) ∈ E and W being doubly stochastic, i.e. (cid:80)

Original: =1 w ij = 1 .
Revised: [Equation]

Original: In D-SGD, every worker i ∈ [ n ] maintains local parameters x ( t ) i ∈
Revised: In D-SGD, every worker i ∈ [ n ] maintains local parameters x ( t ) i ∈

Original: R d that are updated in each iteration with a stochastic gradient update (computed on the local function f i ) and by averaging with neighbors in the communication graph.
Revised: R d that are updated in each iteration with a stochastic gradient update (computed on the local function f i ) and by averaging with neighbors in the communication graph.

Original: It is convenient to compactly write the gradients in matrix notation:
Revised: It is convenient to compactly write the gradients in matrix notation:

Original: [Equation]
Revised: [Equation]

Original: where ξ ( t ) are independent random variables such that E (cid:2)
Revised: where ξ ( t ) are independent random variables such that E (cid:2)

Original: ∂F ( X ( t ) , ξ ( t ) ) (cid:3) =
Revised: ∂F ( X ( t ) , ξ ( t ) ) (cid:3) =

Original: ∂f ( X ( t ) ) .
Revised: ∂f ( X ( t ) ) .

Original: Similarly, we denote the mixing step as multiplication with the mixing matrix W .
Revised: Similarly, we denote the mixing step as multiplication with the mixing matrix W .

Original: This is illustrated through the following update rule for D-SGD:
Revised: This is illustrated through the following update rule for D-SGD,

Original: [Equation]
Revised: [Equation]

Original: where the mixing matrix W ( t ) is sampled from a distribution W ( t ) and G ( t ) =
Revised: where the mixing matrix W ( t ) is sampled from a distribution W ( t ) and G ( t ) =

Original: ∂F ( X ( t ) , ξ ( t ) ) 
Revised: ∂F ( X ( t ) , ξ ( t ) ) .

Original: 3.1. Standard Assumptions
Revised: 3.1. Standard Assumptions

Original: We use the following standard assumptions on objective functions, similar to several existing works [22, 27, 48, 50].
Revised: We use the following standard assumptions on objective functions, similar to several existing works [22, 27, 48, 50].

Original: Following standard notation we let L and µ denote the smoothness and strongconvexity constants of F i ( x , ξ ) and f i respectively (defined in Appendix A), wherever applicable.
Revised: Following standard notation we let L and µ denote the smoothness and strongconvexity constants of F i ( x , ξ ) and f i respectively (defined in Appendix A), wherever applicable.

Original: Assumption 1 (Bounded Variance) .
Revised: Assumption 1 (Bounded Variance) .

Original: We assume that there exists a constant σ such that ∀ x ∈ R d
Revised: We assume that there exists a constant σ such that ∀ x ∈ R d

Original: [Equation]
Revised: [Equation]

Original: For the convex case it suffices to assume a bound on the stochasticity at the optimum x ⋆ :=
Revised: For the convex case it suffices to assume a bound on the stochasticity at the optimum x ⋆ :=

Original: arg min f ( x ) .
Revised: arg min f ( x ) .

Original: We assume there exists a constant σ 2 ⋆ ≤ σ 2 , such that
Revised: We assume there exists a constant σ 2 ⋆ ≤ σ 2 , such that

Original: [Equation]
Revised: [Equation]

Original: Assumption 2 (Consensus Factor) .
Revised: Assumption 2 (Consensus Factor) .

Original: We assume that there exists a constant p ∈ (0 , 1] such that for all t ≥ 0 :
Revised: We assume that there exists a constant p ∈ (0 , 1] such that for all t ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: for all X ∈ R d × n and X :
Revised: for all X ∈ R d × n and X :

Original: = X 11 ⊤ n .
Revised: = X 11 ⊤ n .

Original: The factor p measures averaging abilities of the mixing matrix W : in case of  fully connected graph p = 1 , as we obtain a perfect average in one step; for the disconnected graph p = 0 .
Revised: The factor p measures averaging abilities of the mixing matrix W : in case of the fully connected graph p = 1 , as we obtain a perfect average in one step; for the disconnected graph p = 0 .

Original: 3.2. Gradient Mixing
Revised: 3.2. Gradient Mixing

Original: Prior work introduced various notions to measure the dissimilarity between the local objective functions [22, 27, 32, 46].
Revised: Prior work introduced various notions to measure the dissimilarity between the local objective functions [22, 27, 32, 46].

Original: We instead introduce a new assumption that quantifies the interaction between the data heterogeneity and graph topology.
Revised: We instead introduce a new assumption that quantifies the interaction between the data heterogeneity and graph topology.

Original: Assumption 3 (Relative Heterogenity) .
Revised: Assumption 3 (Relative Heterogenity) .

Original: We define ζ ′ as the smallest positive constant such that
Revised: We define ζ ′ as the smallest positive constant such that

Original: [Equation]
Revised: [Equation]

Original: For the convex case, it suffices to assume a bound at the optimum X ⋆ only.
Revised: For the convex case, it suffices to assume a bound at the optimum X ⋆ only.

Original: We assume define ζ ⋆ as the smallest positive constant, such that ∀ t ≥
Revised: We assume define ζ ⋆ as the smallest positive constant, such that ∀ t ≥

Original: [Equation]
Revised: [Equation]

Original: The above quantity measures the effectiveness of a mixing matrix in producing close to the global average of the gradients at each node.
Revised: The above quantity measures the effectiveness of a mixing matrix in producing close to the global average of the gradients at each node.

Original: 4. Convergence Result
Revised: 4. Convergence Result

Original: In this section, we present a refined analysis of the D-SGD algorithm [22, 27].
Revised: In this section, we present a refined analysis of the D-SGD algorithm [22, 27].

Original: We state our main convergence results below, whose proofs can be found in Appendix B.
Revised: We state our main convergence results below, whose proofs can be found in Appendix B.

Original: In all cases, we assume
Revised: In all cases, we assume

Original: L -smoothness of F i ( x , ξ i ) .
Revised: L -smoothness of F i ( x , ξ i ) .

Original: These results are stated in terms of the mean of the parameters across nodes x ( t ) :
Revised: These results are stated in terms of the mean of the parameters across nodes x ( t ) :

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =1 x ( t ) i .
Revised: =1 x ( t ) i .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Assumptions 1, 2 and 3 hold.
Revised: Let Assumptions 1, 2 and 3 hold.

Original: Then there exists a constant stepsize, such that DSGD needs the following number of iterations to achieve an ε error:
Revised: Then there exists a constant stepsize, such that DSGD needs the following number of iterations to achieve an ε error:

Original: Non-Convex: It holds 1 T +1 (cid:80)
Revised: Non-Convex: It holds 1 T +1 (cid:80)

Original: Tt =

Revised: Tt =

Original: E (cid:13)(cid:13) ∇ f ( x ( t ) )
Revised: E (cid:13)(cid:13) ∇ f ( x ( t ) )

Original: (cid:13)(cid:13) 2 2 ≤ ε after O 
Revised: (cid:13)(cid:13) 2 2 ≤ ε after O (cid:18) σ

Original: [Equation]
Revised: [Equation]

Original: pε (cid:19) · LF 0 iterations.
Revised: pε (cid:19) · LF 0 iterations.

Original: If we in addition assume convexity,
Revised: If we in addition assume convexity,

Original: Convex: Under convexity ( µ ≥ 0 ), the error
Revised: Convex: Under convexity ( µ ≥ 0 ), the error

Original: [Equation]
Revised: [Equation]

Original: Ef
Revised: E

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: f ⋆ ) [Equation] ε after O (cid:18)
Revised: f ⋆ ) ≤ ε after O (cid:18)

Original: [Equation]
Revised: [Equation]

Original: nε
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: √L
Revised: √

Original: [Equation]
Revised: [Equation]

Original: ζ
Revised: pε

Original: [Equation]
Revised: [Equation]

Original: p ) pε [Equation] [Equation]
Revised: /+

Original: Lpε (cid:19) · R 20 iterations, and if strong-convexity µ > 0 ,
Revised: Lpε (cid:19) · R 20 iterations, and if strong-convexity µ > 0 ,

Original: Strongly-Convex: then (cid:80) Tt =
Revised: Strongly-Convex: then (cid:80) T t =

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:19) iterations, where w t denote appropriately chosen positive weights, F 0 :
Revised: (cid:19) iterations, where w t denote appropriately chosen positive weights, F 0 :

Original: = f ( x 0 )
Revised: = f ( x 0 )

Original: − f ⋆ for f ⋆ = min x ∈ R d f ( x ) and R 0 =
Revised: − f ⋆ for f ⋆ = min x ∈ R d f ( x ) and R 0 =

Original: ∥ x 0 − x ⋆ ∥ denote the initial errors.
Revised: ∥ x 0 − x ⋆ ∥ denote the initial errors.

Original: 4.1. Discussion
Revised: 4.1. Discussion

Original: We note that our convergence rates in Theorem 1 resemble the ones in [22] but the old heterogeneity ζ (or ζ ⋆ ) described in Section 3.2 is replaced with the new relative heterogeneity measure ζ ′ (or ζ ′ ⋆ correspondingly).
Revised: We note that our convergence rates in Theorem 1 resemble the ones in [22] but the old heterogeneity ζ (or ζ ⋆ ) described in Section 3.2 is replaced with the new relative heterogeneity measure ζ ′ (or ζ ′ ⋆ correspondingly).

Original: As ζ ′ ≤ ζ ( ζ ′ ⋆ ≤ ζ ⋆ ), the convergence rates given in Theorem 1 are always tighter than previous works.
Revised: As ζ ′ ≤ ζ ( ζ ′ ⋆ ≤ ζ ⋆ ), the convergence rates given in Theorem 1 are always tighter than previous works.

Original: In Appendix C, we explain that ζ ′ can be significantly smaller than ζ .
Revised: In Appendix C, we explain that ζ ′ can be significantly smaller than ζ .

Original: 5. Heterogeneity-Aware Mixing
Revised: 5. Heterogeneity-Aware Mixing

Original: In this section, we build upon our novel theoretical insights developed above to improve the performance of D-SGD in practice.
Revised: In this section, we build upon our novel theoretical insights developed above to improve the performance of D-SGD in practice.

Original: 5.1. Motivation
Revised: 5.1. Motivation

Original: Theorem 1 predicts that small values of the relative heterogeneity parameter ζ ′ lead to improved convergence.
Revised: Theorem 1 predicts that small values of the relative heterogeneity parameter ζ ′ lead to improved convergence.

Original: More specifically, progress in each iteration is determined by the current data-dependent gradient mixing error (cid:13)(cid:13)
Revised: More specifically, progress in each iteration is determined by the current data-dependent gradient mixing error (cid:13)(cid:13)

Original: ∂f ( X ( t ) ) W ( t )
Revised: ∂f ( X ( t ) ) W ( t )

Original: − ∂f ( X ( t ) ) (cid:13)(cid:13) 2 which is upper bounded by ζ ′ (as defined in
Revised: − ∂f ( X ( t ) ) (cid:13)(cid:13) 2 which is upper bounded by ζ ′ (as defined in

Original: Assumption 3).
Revised: Assumption 3).

Original: This quantity depends both on the current iterate X ( t ) but also on the chosen mixing weights W ( t ) , thus suggesting to continually update the mixing matrix such that the gradient mixing error remains low, while the gradients evolve during training.
Revised: This quantity depends both on the current iterate X ( t ) but also on the chosen mixing weights W ( t ) , thus suggesting to continually update the mixing matrix such that the gradient mixing error remains low, while the gradients evolve during training.

Original: Thus, we can write the following time-varying optimization problem for the mixing weights
Revised: Thus, we can write the following time-varying optimization problem for the mixing weights

Original: W .
Revised: W .

Original: For current parameters X ∈ R d × n , X = X 1 n 11 ⊤ (we drop the time index) distributed over n nodes, we aim to solve
Revised: For current parameters X ∈ R d × n , X = X 1 n 11 ⊤ (we drop the time index) distributed over n nodes, we aim to solve

Original: [Equation]
Revised: [Equation]

Original: GME
Revised: GME

Original: [Equation]
Revised: [Equation]

Original: where M w = { W : 1 W = 1 , 1 ⊤ W = 1 ⊤ ; 0 ≤ w ij ≤ 1 ∀ i, j , w ij = 0 ∀ ( i, j ) ̸∈ E } is the set of allowed mixing matrices.
Revised: where M w = { W : 1 W = 1 , 1 ⊤ W = 1 ⊤ ; 0 ≤ w ij ≤ 1 ∀ i, j , w ij = 0 ∀ ( i, j ) ̸∈ E } is the set of allowed mixing matrices.

Original: The objective function comes from the definition of ζ ′ 2 in Equation (14).
Revised: The objective function comes from the definition of ζ ′ 2 in Equation (14).

Original: The first two conditions ensure double stochasticity of W , while the last condition respects edge constraints of the communication graph G .
Revised: The first two conditions ensure double stochasticity of W , while the last condition respects edge constraints of the communication graph G .

Original: Note that unlike the matrix corresponding to the optimal spectral gap, the optimal matrix obtained above could be asymmetric.
Revised: Note that unlike the matrix corresponding to the optimal spectral gap, the optimal matrix obtained above could be asymmetric.

Original: We call this optimization problem the exact Gradient Mixing Error (GME-exact).
Revised: We call this optimization problem the exact Gradient Mixing Error (GME-exact).

Original: 5.2. Proposed Algorithm
Revised: 5.2. Proposed Algorithm

Original: We can equivalently reformulate (GME-exact) as to more efficiently solve when the dimension d of the gradient vectors is large, compared to the number of nodes n .
Revised: We can equivalently reformulate (GME-exact) as to more efficiently solve when the dimension d of the gradient vectors is large, compared to the number of nodes n .

Original: Defining the gram matrix Γ := (cid:0) ∂f
Revised: Defining the gram matrix Γ := (cid:0) ∂f

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: ∂f
Revised: ∂f

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: ∂f ( X ) − ∂f ( X )
Revised: ∂f [Equation] X ) − ∂f ( X )

Original: [Equation]
Revised: [Equation]

Original: , we obtain:
Revised: , we obtain:

Original: [Equation]
Revised: [Equation]

Original: This is a quadratic program with linear constraints.
Revised: This is a quadratic program with linear constraints.

Original: The minimizer, i.e. resulting mixing matrix, of (GME-opt- Γ ) is the same as for (GME-exact).
Revised: The minimizer, i.e. resulting mixing matrix, of (GME-opt- Γ ) is the same as for (GME-exact).

Original: However, as the problem formulation depends only on the gram matrix Γ ∈ R n × n it can be solved more efficiently [9].
Revised: However, as the problem formulation depends only on the gram matrix Γ ∈ R n × n it can be solved more efficiently [9].

Original: Since the direct optimization of (GME-opt- Γ ) is infeasible due to the inability to access x and the full gradients, we propose to approximately solve an approximation of the above objective, only once every H ≥ 1 step.
Revised: Since the direct optimization of (GME-opt- Γ ) is infeasible due to the inability to access x and the full gradients, we propose to approximately solve an approximation of the above objective, only once every H ≥ 1 step.

Original: We summarise the resulting algorithm in Algorithm 1, which calls equation (GME-opt- Γ ) as a subproblem.
Revised: We summarise the resulting algorithm in Algorithm 1, which calls equation (GME-opt- Γ ) as a subproblem.

Original: To make our approach communication-efficien, we utilize sketching techniques and intermittent communication.
Revised: To make our approach communication-efficien, we utilize sketching techniques and intermittent communication.

Original: Sketching allows the nodes to communicate low dimensional projections of the gradients/parameters instead of the full parameter vectors.
Revised: Sketching allows the nodes to communicate low dimensional projections of the gradients/parameters instead of the full parameter vectors.

Original: This is illustrated in Algorithm 2 (CE-GME), and further explained in Appendix E, along with theoretical guarantees in Appendix E.3.
Revised: This is illustrated in Algorithm 2 (CE-GME), and further explained in Appendix E, along with theoretical guarantees in Appendix E.3.

Original: We provide additional justification for the design choices inAppendix
Revised: We provide additional justification for the design choices in

Original:  E.
Revised: Appendix E.

Original: Algorithm
Revised: Algorithm

Original: X (0) , stepsizes { η t } T − 1 t =0 , number of iterations T , communication graph G ,
Revised: X (0) , stepsizes { η t } T − 1 t =0 , number of iterations T , communication graph G ,

Original: GME optimization period H , initial mixing matrix W (0) X ( T ) , parameters of all the nodes for t in 0 . . .
Revised: GME optimization period H , initial mixing matrix W (0) . for t in 0 . . .

Original: T do in parallel on all workers 1:2:
Revised: T do in parallel on all workers 1:2:

Original: G ( t ) = ∂F ( X ( t ) , ξ ( t ) )
Revised: G ( t ) = ∂F ( X ( t ) , ξ ( t ) )

Original: ▷ stochastic gradients 3: if t mod H = 0 then 4:
Revised: ▷ stochastic gradients 3: if t mod H = 0 then 4:

Original: W ( t ) =
Revised: W ( t ) =

Original: CE-GME( G ( t ) ) 5: else 6:
Revised: CE-GME( G ( t ) ) 5: else 6:

Original: W ( t ) =
Revised: W ( t ) =

Original: W ( t − 1) 7: end if 8:
Revised: W ( t − 1) 7: end if 8:

Original: X ( t +1) =
Revised: X ( t +1) =

Original: ( X ( t ) − η t G ( t ) )
Revised: ( X ( t ) − η t G ( t ) )

Original: W ( t ) ▷ update & mixing 9: end parallel for
Revised: W ( t ) ▷ update & mixing 9: end parallel for

Original: Algorithm 2 CE-GME: Communication Efficient GME matrix G ∈ R d × n , distributed columnwise across n nodes, random seed s , dimension k W , mixing matrix minimizing the GME in parallel on n nodes do 1:2: sample A ∈ R k × d , a ij ∼ N (0 , 1)
Revised: Algorithm 2 CE-GME: Communication Efficient GME matrix G ∈ R d × n , distributed columnwise across n nodes, random seed s , dimension k W , mixing matrix minimizing the GME in parallel on n nodes do 1:2: sample A ∈ R k × d , a ij ∼ N (0 , 1)

Original: ▷ use the same random seed s on every node.
Revised: ▷ use the same random seed s on every node.

Original: S = 1 √ k AG ∈ R k × n ▷ compute sketches 4:
Revised: S = 1 √ k AG ∈ R k × n ▷ compute sketches 4:

Original: S = S 11 ⊤ n ▷ all-reduce-communication 5: Γ = (cid:0) S − S (cid:1) ⊤ (cid:0) S − S (cid:1)
Revised: S = S 11 ⊤ n ▷ all-reduce-communication 5: Γ = (cid:0) S − S (cid:1) ⊤ (cid:0) S − S (cid:1)

Original: ▷ sketched gram matrix 6:
Revised: ▷ sketched gram matrix 6:

Original: W = GME-opt(Γ) 7: end
Revised: W = GME-opt(Γ) 7: end

Original: [Figure]
Revised: [Figure]

Original: 1 n (cid:13)(cid:13)
Revised: 1 n (cid:13)(cid:13)

Original: X − X (cid:13)(cid:13) 2 F gradient mixing error (cid:13)(cid:13)∂F ( X ξ )
Revised: (cid:13)(cid:13)

Original:  W − ∂F  X, ξ ) (cid:13)(cid:13)  F vs the number of iterations for quadratic objectives.
Revised: ∂F ( X, ξ ) W − ∂F ( X, ξ ) (cid:13)(cid:13) 2 F vs. the number of iterations for quadratic objectives.

Original: “Optimal Spectral Gap” denotes the DSGD algorithm with mixing matrix optimized using the SLEM objective defined in [8].
Revised: “Optimal Spectral Gap” denotes the DSGD algorithm with mixing matrix optimized using the SLEM objective defined in [8].

Original: We report an average over a window of 5 iterations of corresponding quantity on each plot.
Revised: We report an average over a window of 5 iterations of corresponding quantity on each plot.

Original: 6. Experiments
Revised: 6. Experiments

Original: For all our experiments, we use the CVXPY [12] convex optimization library to perform the constrained optimization defined in the section.
Revised: For all our experiments, we use the CVXPY [12] convex optimization library to perform the constrained optimization defined in the section.

Original: We denote the number of nodes in the underlying topology and the time period for GME optimization by n and T respectively.
Revised: We denote the number of nodes in the underlying topology and the time period for GME optimization by n and T respectively.

Original: HA-DSGD refers to our proposed Alg. 1 with updates alternating between the weights obtained by the GME optimization and the Metropolis-Hastings weights, similarly as discussed in Appendix E.
Revised: HA-DSGD refers to our proposed Alg. 1 with updates alternating between the weights obtained by the GME optimization and the Metropolis-Hastings weights, similarly as discussed in Appendix E.

Original: We consider a simple setting of random quadratic objectives, with the objective for the i th client given by f i ( x ) =
Revised: We consider a simple setting of random quadratic objectives, with the objective for the i th client given by f i ( x ) =

Original: ∥
Revised: ∥

Original: A i x + b i ∥ 22 where x denotes a d dimensional parameter vector and both A i and b i contain entries sampled randomly from N (0 , 1) and fixed for each client.
Revised: A i x + b i ∥ 22 where x denotes a d dimensional parameter vector and both A i and b i contain entries sampled randomly from N (0 , 1) and fixed for each client.

Original: For our experiments, we set d = 10 .
Revised: For our experiments, we set d = 10 .

Original: We further introduce stochasticity to the gradients by adding random Gaussian noise with variance 0 .
Revised: We further introduce stochasticity to the gradients by adding random Gaussian noise with variance 0 .

Original: Figure 1 illustrates the improvements due to our approach across three metrics: the distance from the optimum, consensus error, as well as the gradient mixing error.
Revised: Figure 1 illustrates the improvements due to our approach across three metrics: the distance from the optimum, consensus error, as well as the gradient mixing error.

Original: In the
Revised: In the

Original: Appendix J, we provide additional results for Deep Learning Benchmarks and quadratic objectives along with details of the experiments
Revised: Appendix J, we provide additional results for Deep Learning Benchmarks and quadratic objectives along with details of the experiments

Original: Appendix A. Preliminary Definitions
Revised: Appendix A. Preliminary Definitions

Original: L -smoothness : Each local function F i ( x , ξ ): R d × Ω i
Revised: L -smoothness : Each local function F i ( x , ξ ): R d × Ω i

Original: → R , i ∈ [ n ] is differentiable for each ξ ∈ supp( D i ) and there exists a constant L ≥ 

Revised: → R , i ∈ [ n ] is differentiable for each ξ ∈ supp( D i ) and there exists a constant L ≥

Original: such that for each x , y ∈ R d , ξ ∈ supp( D i ) :
Revised: such that for each x , y ∈ R d , ξ ∈ supp( D i ) :

Original: [Equation]
Revised: [Equation]

Original: (7) ( µ -strong) convexity : Each function f
Revised: (7) ( µ -strong) convexity : Each function f

Original: i : R d → R , i ∈ [ n ] is µ -(strongly)
Revised: i : R d → R , i ∈ [ n ] is µ -(strongly)

Original: convex for constant µ ≥ 0 .
Revised: convex for constant µ ≥ 0 .

Original: That is, for all x , y ∈ R d :
Revised: That is, for all x , y ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: Appendix B. Proofs of Main Results B.1. Preliminaries
Revised: Appendix B. Proofs of Main Results B.1. Preliminaries

Original: We utilize the following set of standard useful inequalities:
Revised: We utilize the following set of standard useful inequalities:

Original: Lemma 1.
Revised: Lemma 1.

Original: Let g be an L -smooth convex function.
Revised: Let g be an L -smooth convex function.

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let Y ∈ R
Revised: Let Y ∈ R

Original: d
Revised: d

Original: × n be an arbitrary matrix and Y the matrix with each column containing the columnwise mean of Y i.e. Y =
Revised: × n be an arbitrary matrix and Y the matrix with each column containing the columnwise mean of Y i.e. Y =

Original: Y 11 ⊤ n .
Revised: Y 11 ⊤ n .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma 3.

Original: For arbitrary set of n vectors { a i } ni =1 , a i ∈ R d (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i
Revised: For arbitrary set of n vectors { a i } ni =1 , a i ∈ R d (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i

Original: = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: = a i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: For given two vectors a , b ∈ R d
Revised: For given two vectors a , b ∈ R d

Original: [Equation]
Revised: [Equation]

Original: This inequality also holds for the sum of two matrices A, B ∈ R n × d in Frobenius norm.
Revised: This inequality also holds for the sum of two matrices A, B ∈ R n × d in Frobenius norm.

Original: B.2. Recursion For Consensus
Revised: B.2. Recursion For Consensus

Original: The recursion for consensus, analyzed in Lemmas 9 and 12 of [22] relies on the following inequalities:
Revised: The recursion for consensus, analyzed in Lemmas 9 and 12 of [22] relies on the following inequalities:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: The above inequality, however, discards the fact that it is desirable for the update at each node to be close to the update to the mean.
Revised: The above inequality, however, discards the fact that it is desirable for the update at each node to be close to the update to the mean.

Original: Our analysis below instead incorporates the effect of the mixing of the gradient through the following lemma:
Revised: Our analysis below instead incorporates the effect of the mixing of the gradient through the following lemma:

Original: Lemma 5.
Revised: Lemma 5.

Original: The update to X ( t − 1) at the t th step of DSGD with mixing matrix W ( t − 1) can be reformulated as:
Revised: The update to X ( t − 1) at the t th step of DSGD with mixing matrix W ( t − 1) can be reformulated as:

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: Where in the last step we used the identity 11 ⊤ n W = 11 ⊤ n , valid for any doubly stochastic matrix
Revised: Where in the last step we used the identity 11 ⊤ n W = 11 ⊤ n , valid for any doubly stochastic matrix

Original: W , implying that X ( t − 1) W ( t − 1) =
Revised: W , implying that X ( t − 1) W ( t − 1) =

Original: X ( t − 1) and ∂F ( X ( t − 1) , ξ ( t − 1) ) W ( t − 1) = ∂F
Revised: X ( t − 1) and ∂F ( X ( t − 1) , ξ ( t − 1) ) W ( t − 1) = ∂F

Original: ( X ( t − 1) , ξ ( t − 1) ) .
Revised: ( X ( t − 1) , ξ ( t − 1) ) .

Original: For the sake of generality and consistency with [22], we prove our result under a generalization of the Assumption 3 on the gradient mixing error
Revised: For the sake of generality and consistency with [22], we prove our result under a generalization of the Assumption 3 on the gradient mixing error

Original: Assumption 4 (Relative Heterogenity with Growth) .
Revised: Assumption 4 (Relative Heterogenity with Growth) .

Original: We assume that there exist constants ζ ′ and
Revised: We assume that there exist constants ζ ′ and

Original: P ′ , such that ∀ X ∈ R
Revised: P ′ , such that ∀ X ∈ R

Original: d
Revised: d

Original: × n :
Revised: × n :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assumption 3 corresponds to a special case of the above assumption with P ′ = 0 .
Revised: Assumption 3 corresponds to a special case of the above assumption with P ′ = 0 .

Original: We now prove the following consensus recursion:
Revised: We now prove the following consensus recursion:

Original: Lemma 6.
Revised: Lemma 6.

Original: Let Ξ t = 1 n E t (cid:80)
Revised: Let Ξ t = 1 n E t (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: x ( t ) i − x ( t ) (cid:13)(cid:13)(cid:13) 2 denote the consensus distance at time t , and let e t = f
Revised: x ( t ) i − x ( t ) (cid:13)(cid:13)(cid:13) 2 denote the consensus distance at time t , and let e t = f

Original: ( x ( t ) ) − f ( x ⋆ ) for the convex case and e t = (cid:13)(cid:13) ∇ f ( x ( t ) )
Revised: ( x ( t ) ) − f ( x ⋆ ) for the convex case and e t = (cid:13)(cid:13) ∇ f ( x ( t ) )

Original: (cid:13)(cid:13) 2 2 for the non-convex case.
Revised: (cid:13)(cid:13) 2 2 for the non-convex case.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Where D = 36 L (1 − p ) + 4 L 8 − 7 p p for the convex case , 8 − 7 p p P ′ for the nonconvex case and
Revised: Where D = 36 L (1 − p ) + 4 L 8 − 7 p p for the convex case , 8 − 7 p p P ′ for the nonconvex case and

Original: A = 8 − 7 p p ( ζ ′ 2 ) + 3(1 − p ) σ 2 for the non-convex case and 16 − 14 p p ( ζ ′ 2 ) +
Revised: A = 8 − 7 p p ( ζ ′ 2 ) + 3(1 − p ) σ 2 for the non-convex case and 16 − 14 p p ( ζ ′ 2 ) +

Original: [Equation]
Revised: [Equation]

Original: 1 − p ) σ 2 for the convex case.
Revised: 1 − p ) σ 2 for the convex case.

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: Let Ξ t = 1 n E t
Revised: Let Ξ t = 1 n E t

Original: [Equation]
Revised: [Equation]

Original: ni =1 (cid:13)(cid:13)(cid:13) x ( t ) i − x ( t ) (cid:13)(cid:13)(cid:13) 2 denote the consensus distance at time t .
Revised: ni =1 (cid:13)(cid:13)(cid:13) x ( t ) i − x ( t ) (cid:13)(cid:13)(cid:13) 2 denote the consensus distance at time t .

Original: We have, using
Revised: We have, using

Original: Lemma 5: n
Revised: Lemma 5: n

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( t
Revised: ( t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( X ( t
Revised: ( X ( t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: ) (cid:17) W ( t
Revised: ) (cid:17) W ( t

Original: [Equation]
Revised: [Equation]

Original: 1) (cid:13)(cid:13)(cid:13)
Revised: 1) (cid:13)(cid:13)(cid:13)

Original: F = E
Revised: F = E

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: (cid:125)
Revised: (cid:125)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Where the last inequality follows from the fact that noise in the gradient is independent at each time step, and also from unbiased stochastic gradients E ξ ( t − 1) ∂F ( X ( t − 1) , ξ ( t − 1) )
Revised: Where the last inequality follows from the fact that noise in the gradient is independent at each time step, and also from unbiased stochastic gradients E ξ ( t − 1) ∂F ( X ( t − 1) , ξ ( t − 1) )

Original: =
Revised: =

Original: ∂f ( X ( t − 1) ) .
Revised: ∂f ( X ( t − 1) ) .

Original: We first observe that, using assumption 2, we have:
Revised: We first observe that, using assumption 2, we have:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: Furthermore, using equation (2)
Revised: Furthermore, using equation (2)

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: We then add and subtract the gradients at the mean point ∂F ( X ( t − 1) , ξ ( t − 1) ) and the corresponding mean ∂F ( X ( t − 1) ) to obtain:
Revised: We then add and subtract the gradients at the mean point ∂F ( X ( t − 1) , ξ ( t − 1) ) and the corresponding mean ∂F ( X ( t − 1) ) to obtain:

Original: [Equation]
Revised: [Equation]

Original: Using the L smoothness of each node’s objective
Revised: Using the L smoothness of each node’s objective

Original: [Equation]
Revised: [Equation]

Original: the first two terms can be bounded as follows:
Revised: the first two terms can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have:
Revised: Similarly, we have:

Original: [Equation]
Revised: [Equation]

Original: Subsequently, we utilize the assumptions on stochasticity 1 to bound the third term.
Revised: Subsequently, we utilize the assumptions on stochasticity 1 to bound the third term.

Original: We proceed separately for the Convex and non-convex cases:
Revised: We proceed separately for the Convex and non-convex cases:

Original: Convex Case : We add and subtract ∂F ( X ⋆ , ξ ( j ) ) and the corresponding mean ∂F ( X ⋆ ) to obtain:
Revised: Convex Case : We add and subtract ∂F ( X ⋆ , ξ ( j ) ) and the corresponding mean ∂F ( X ⋆ ) to obtain:

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 ≤
Revised: Lemma 1 ≤

Original: [Equation]
Revised: [Equation]

Original: Non-convex Case : We directly utilize the uniform bound on the stochasticity (assumption 1) to obtain:
Revised: Non-convex Case : We directly utilize the uniform bound on the stochasticity (assumption 1) to obtain:

Original: [Equation]
Revised: [Equation]

Original: The final bound on T 2 is therefore given by:
Revised: The final bound on T 2 is therefore given by:

Original: Convex case :
Revised: Convex case :

Original: [Equation]
Revised: [Equation]

Original: Non
Revised: Non

Original: -convex case
Revised: -convex case

Original: [Equation]
Revised: [Equation]

Original: We now bound T 1 as follows:
Revised: We now bound T 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: [Table]
Revised: [Table]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: The second term can be bounded by utilizing Equation 17 and Equation 2 as follows:
Revised: The second term can be bounded by utilizing Equation 17 and Equation 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: Finally, incorporating the bound on T
Revised: Finally, incorporating the bound on T

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We now choose β 1 such that (1 − p )
Revised: We now choose β 1 such that (1 − p )

Original: (1 + β 1 ) =
Revised: (1 + β 1 ) =

Original: (1 − 7 p 8 ) i.e. β 1 = p 8(1 − p )
Revised: (1 − 7 p 8 ) i.e. β 1 = p 8(1 − p )

Original: Subsequently, we choose β 2 such that ((1 − 7 p 8 )(1 + β 2 ) =
Revised: Subsequently, we choose β 2 such that ((1 − 7 p 8 )(1 + β 2 ) =

Original: (1 − 3 p 4 ) i.e. β 2 = p 8 − 7 p .
Revised: (1 − 3 p 4 ) i.e. β 2 = p 8 − 7 p .

Original: Then, assuming that the step size η t satisfies, η 2 t ≤
Revised: Then, assuming that the step size η t satisfies, η 2 t ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 6 p
Revised: 6 p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 − p ) (cid:17)
Revised: 1 − p ) (cid:17)

Original: L 2 , we obtain: n Ξ
Revised: L 2 , we obtain: n Ξ

Original: [Equation]
Revised: [Equation]

Original: F + (1 + β − 1 1 ) η 2 t E (cid:13)(cid:13)(cid:13)(cid:16) ∂F ( X ( t − 1) ) − ∂F ( X ( t − 1) (cid:17) W ( t − 1) (cid:13)(cid:13)(cid:13)
Revised: F + (1 + β − 1 1 ) η 2 t E (cid:13)(cid:13)(cid:13)(cid:16) ∂F ( X ( t − 1) ) − ∂F ( X ( t − 1) (cid:17) W ( t − 1) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Since p
Revised: Since p

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: 80 L 2 , we only require the step size to be O ( pL ) , same as [22].
Revised: 80 L 2 , we only require the step size to be O ( pL ) , same as [22].

Original: Thus the consensus distance decreases linearly, along with an error dependent on the diffusion of the gradients across nodes.
Revised: Thus the consensus distance decreases linearly, along with an error dependent on the diffusion of the gradients across nodes.

Original: Finally, substituting the assumption 4 for the non-convex case
Revised: Finally, substituting the assumption 4 for the non-convex case

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: For the convex case, we first bound the gradient mixing error at X in terms of that at X ∗ as follows:
Revised: For the convex case, we first bound the gradient mixing error at X in terms of that at X ∗ as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Lemma 2 ≤
Revised: Lemma 2 ≤

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: Where in the last step
Revised: Where in the last step

Original: [Equation]
Revised: [Equation]

Original: we used Equation 1.
Revised: we used Equation 1.

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: B.3. Convergence Rate
Revised: B.3. Convergence Rate

Original: We utilize the consensus recursion in Lemma 6 to bound an appropriately weighted sum of the consensus iterates as follows:
Revised: We utilize the consensus recursion in Lemma 6 to bound an appropriately weighted sum of the consensus iterates as follows:

Original: [Equation]
Revised: [Equation]

Original: Recursively substituting n Ξ t − 1 for t in [1 , · · · , T ] , we then obtain:
Revised: Recursively substituting n Ξ t − 1 for t in [1 , · · · , T ] , we then obtain:

Original: [Equation]
Revised: [Equation]

Original: Where in the last step we used w t ≤ w j for j ≥ t
Revised: Where in the last step we used w t ≤ w j for j ≥ t

Original: We thus obtain an Equation having the same form as Equation 18 of [22] :
Revised: We thus obtain an Equation having the same form as Equation 18 of [22] :

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: where η satisfies η ≤ (cid:113) pbD 2 B and the factor B is as defined in [21] for the different cases.
Revised: where η satisfies η ≤ (cid:113) pbD 2 B and the factor B is as defined in [21] for the different cases.

Original: The rest of the proof involves utilizing the descent lemma in [22] and choosing the appropriate step size following exactly the use of Equation 18 in [22].
Revised: The rest of the proof involves utilizing the descent lemma in [22] and choosing the appropriate step size following exactly the use of Equation 18 in [22].

Original: Finally, setting P ′ = 

Revised: Finally, setting P ′ =

Original: leads to the convergence rates provided in Theorem 1.
Revised: leads to the convergence rates provided in Theorem 1.

Original: Appendix C. Comparision with Koloskova et al. [22]
Revised: Appendix C. Comparision with Koloskova et al. [22]

Original: In this section, we discuss how ζ ′ relates to ζ defined in Koloskova et al.
Revised: In this section, we discuss how ζ ′ relates to ζ defined in Koloskova et al.

Original: Remark 1.
Revised: Remark 1.

Original: Using Assumption 4, we obtain:
Revised: Using Assumption 4, we obtain:

Original: [Equation]
Revised: [Equation]

Original: ∼W
Revised: ∼W

Original: [Equation]
Revised: [Equation]

Original: This implies that ζ ′ 2 ≤ (1 − p ) ζ 2 and ζ ′ 2 ⋆ ≤ (1 − p ) ζ 2 ⋆ .
Revised: This implies that ζ ′ 2 ≤ (1 − p ) ζ 2 and ζ ′ 2 ⋆ ≤ (1 − p ) ζ 2 ⋆ .

Original: Often ζ ′ can even be much smaller
Revised: Often ζ ′ can even be much smaller

Original: As a motivating example, we consider a ring topology with the Metropolis-Hasting mixing weights and a particular pattern on how the data is distributed across the nodes:
Revised: As a motivating example, we consider a ring topology with the Metropolis-Hasting mixing weights and a particular pattern on how the data is distributed across the nodes:

Original: Example 1.
Revised: Example 1.

Original: Consider a ring topology on n = 3 k nodes, k ≥ 1 , with uniform mixing among neighbors ( w i,i − 1 = w i,i = w i,i +1 = 13 ) and assume that D i
Revised: Consider a ring topology on n = 3 k nodes, k ≥ 1 , with uniform mixing among neighbors ( w i,i − 1 = w i,i = w i,i +1 = 13 ) and assume that D i

Original: =
Revised: =

Original: D i +3 mod n for all i and suppose there is an x ′ with ∇ f ( x ′ )
Revised: D i +3 mod n for all i and suppose there is an x ′ with ∇ f ( x ′ )

Original: = 0 , ∥∇ f 1 ( x ′ ) ∥ > 0 .
Revised: = 0 , ∥∇ f 1 ( x ′ ) ∥ > 0 .

Original: Then ζ ′ = 0 and ζ ̸ = 0 .
Revised: Then ζ ′ = 0 and ζ ̸ = 0 .

Original: It is easy to see that the relative heterogenity is ζ ′ = 0 .
Revised: It is easy to see that the relative heterogenity is ζ ′ = 0 .

Original: This holds, because uniform averaging of three neighboring gradients result in an unbiased gradient estimator:
Revised: This holds, because uniform averaging of three neighboring gradients result in an unbiased gradient estimator:

Original: [Equation]
Revised: [Equation]

Original: while in contrast
Revised: while in contrast

Original: [Equation]
Revised: [Equation]

Original: In Appendix F, we discuss another example provided by [6], where mixing within interconnected cliques (i.e., locally fully connected sets of nodes) having global label distribution leads to ζ ′ = 0 .]
Revised: In Appendix F, we discuss another example provided by [6], where mixing within interconnected cliques (i.e., locally fully connected sets of nodes) having global label distribution leads to ζ ′ = 0 .]

Original: Appendix D. Additional Related Work
Revised: Appendix D. Additional Related Work

Original: Several recent works have attempted to tackle the undesirable effects of data heterogeneity across nodes on the convergence of D-SGD through suitable modifications to the algorithm.
Revised: Several recent works have attempted to tackle the undesirable effects of data heterogeneity across nodes on the convergence of D-SGD through suitable modifications to the algorithm.

Original: D 2 /Exactdiffusion [46, 57, 58] apply variance reduction on each node.
Revised: D 2 /Exactdiffusion [46, 57, 58] apply variance reduction on each node.

Original: Gradient Tracking [23, 30, 31, 37, 40] utilizes an estimate of the full gradient at each node, obtained by successive mixing of gradients along with corrections based on updates to the local gradients.
Revised: Gradient Tracking [23, 30, 31, 37, 40] utilizes an estimate of the full gradient at each node, obtained by successive mixing of gradients along with corrections based on updates to the local gradients.

Original: However, these approaches have not been found to yield performances comparable to D-SGD in practice [28], despite superior theoretical properties [2, 23].
Revised: However, these approaches have not been found to yield performances comparable to D-SGD in practice [28], despite superior theoretical properties [2, 23].

Original: The undesirable effects of data heterogeneity persist also in the Federated Learning setting, which is a special case of the fully decentralized setting.
Revised: The undesirable effects of data heterogeneity persist also in the Federated Learning setting, which is a special case of the fully decentralized setting.

Original: Several algorithms have been designed to mitigate the undesirable effects of data heterogeneity [10, 19, 33, 51], yet extending them to the setup of decentralized learning remains challenging.
Revised: Several algorithms have been designed to mitigate the undesirable effects of data heterogeneity [10, 19, 33, 51], yet extending them to the setup of decentralized learning remains challenging.

Original: For optimizing convex functions, specialized variants such as EXTRA [45], decentralized primaldual methods [1] have been developed.
Revised: For optimizing convex functions, specialized variants such as EXTRA [45], decentralized primaldual methods [1] have been developed.

Original: With a focus on deep learning applications, [28, 59] propose adaptations of momentum methods.
Revised: With a focus on deep learning applications, [28, 59] propose adaptations of momentum methods.

Original: Bellet et al.
Revised: Bellet et al.

Original: [6] recently proposed utilizing a topology that minimizes the data-heterogeneity across cliques composed of clusters of nodes capturing the entire diversity of data distribution (DCliques).
Revised: [6] recently proposed utilizing a topology that minimizes the data-heterogeneity across cliques composed of clusters of nodes capturing the entire diversity of data distribution (DCliques).

Original: Our analysis does apply to their setting and can be used to theoretically explain the theoretical underpinnings behind D-Clique averaging (Appendix F)
Revised: Our analysis does apply to their setting and can be used to theoretically explain the theoretical underpinnings behind D-Clique averaging (Appendix F)

Original: Another line of work focuses on the design of (data-independent) mixing matrices with good spectral properties [56].
Revised: Another line of work focuses on the design of (data-independent) mixing matrices with good spectral properties [56].

Original: Another example is time-varying topologies such as the directed exponential graph [3] that allow for perfect mixing after multiple steps, or matchings [50].
Revised: Another example is time-varying topologies such as the directed exponential graph [3] that allow for perfect mixing after multiple steps, or matchings [50].

Original: Several theoretical works argue to perform multiple averaging steps between updates [24, 32, 43], though this introduces a noticeable overhead in practical DL applications.
Revised: Several theoretical works argue to perform multiple averaging steps between updates [24, 32, 43], though this introduces a noticeable overhead in practical DL applications.

Original: Vogels et al.
Revised: Vogels et al.

Original: [48] propose to replace gossip averaging with a new mechanism to spread information on embedded spanning trees.
Revised: [48] propose to replace gossip averaging with a new mechanism to spread information on embedded spanning trees.

Original: Several works have analyzed the relationship between graph topology, data-heterogeneity, and computational efficiency in settings such as Network Lasso [15, 17], Clustered Federated Learning [42], and Extreme Variational Inference [60].
Revised: Several works have analyzed the relationship between graph topology, data-heterogeneity, and computational efficiency in settings such as Network Lasso [15, 17], Clustered Federated Learning [42], and Extreme Variational Inference [60].

Original: However, such an analysis for the optimization in decentralized SGD has been lacking.
Revised: However, such an analysis for the optimization in decentralized SGD has been lacking.

Original: Appendix E. Designing good mixing matrices
Revised: Appendix E. Designing good mixing matrices

Original: One of the main advantages of our theoretical analysis is that it allows a principled design of good mixing matrices.
Revised: One of the main advantages of our theoretical analysis is that it allows a principled design of good mixing matrices.

Original: We identify in Theorem 1 two concurrent factors: on the one hand, the consensus factor p should be close to 1 , and on the other hand the relative heterogeneity parameter ζ ′ should be close to 0 .
Revised: We identify in Theorem 1 two concurrent factors: on the one hand, the consensus factor p should be close to 1 , and on the other hand the relative heterogeneity parameter ζ ′ should be close to 0 .

Original: Trying to find a mixing matrix satisfying both might seem a difficult task.
Revised: Trying to find a mixing matrix satisfying both might seem a difficult task.

Original: However, one can combine matrices that are good for either of the tasks.
Revised: However, one can combine matrices that are good for either of the tasks.

Original: Example 2.
Revised: Example 2.

Original: Suppose a mixing matrix W p has consensus factor p ≤ 1 , and a mixing matrix W ζ ′ has relative heterogeneity parameter ζ ′ .
Revised: Suppose a mixing matrix W p has consensus factor p ≤ 1 , and a mixing matrix W ζ ′ has relative heterogeneity parameter ζ ′ .

Original: Then W = W ζ ′ W p has consensus factor at least p and relative heterogeneity at most ζ ′ .
Revised: Then W = W ζ ′ W p has consensus factor at least p and relative heterogeneity at most ζ ′ .

Original: Proof By the mixing property of W p ,
Revised: Proof By the mixing property of W p ,

Original: [Equation]
Revised: [Equation]

Original: and similarly, 1 n (cid:13)(cid:13)
Revised: and similarly, 1 n (cid:13)(cid:13)

Original: ∂f ( X ) W ζ ′ W p − ∂f
Revised: ∂f ( X ) W ζ ′ W p − ∂f

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Where we used the double stochasticity of W ζ ′ which implies that (cid:13)(cid:13) ( W − 1 n 11 ⊤ ) (cid:13)(cid:13) 2 ≤ 1 (Proof in Proposition 5).
Revised: Where we used the double stochasticity of W ζ ′ which implies that (cid:13)(cid:13) ( W − 1 n 11 ⊤ ) (cid:13)(cid:13) 2 ≤ 1 (Proof in Proposition 5).

Original: In practice, we observe that two communication rounds are not necessary, alternating between mixing with W p and W ζ ′ works well and does not increase the communication costs.
Revised: In practice, we observe that two communication rounds are not necessary, alternating between mixing with W p and W ζ ′ works well and does not increase the communication costs.

Original: E.1. Justifying the Design Choices
Revised: E.1. Justifying the Design Choices

Original: Next we analyze the relationship between Algorithm 2 and GME-exact.
Revised: Next we analyze the relationship between Algorithm 2 and GME-exact.

Original: Effect of Periodic Optimization.
Revised: Effect of Periodic Optimization.

Original: In Algorithm 1, we optimize the mixing matrix W only once every H steps in order to reduce the computational cost.
Revised: In Algorithm 1, we optimize the mixing matrix W only once every H steps in order to reduce the computational cost.

Original: Below we show that for small H , if at step t
Revised: Below we show that for small H , if at step t

Original: + H
Revised: + H

Original: we apply the matrix W ( t ) found by GME at the step t , then this matrix would still give a good error ζ ′ .
Revised: we apply the matrix W ( t ) found by GME at the step t , then this matrix would still give a good error ζ ′ .

Original: To isolate the effect of periodic optimization, we assume that every H steps we solve
Revised: To isolate the effect of periodic optimization, we assume that every H steps we solve

Original: an original
Revised: an original

Original: GME-exact problem.
Revised: GME-exact problem.

Original: We perform optimization using the full gradients, moreover on line 4 of Algorithm 1 we solve an original (GME-exact) problem with full gradients on the averaged parameters, i.e. line 4 is replaced with W ( t ) =
Revised: We perform optimization using the full gradients, moreover on line 4 of Algorithm 1 we solve an original (GME-exact) problem with full gradients on the averaged parameters, i.e. line 4 is replaced with W ( t ) =

Original: GME( ∂f ( X ( t ) )) .
Revised: GME( ∂f ( X ( t ) )) .

Original: Proposition 1.
Revised: Proposition 1.

Original: [Equation]
Revised: [Equation]

Original: For the proof refer to the appendix.
Revised: For the proof refer to the appendix.

Original: Since the learning rate η t is usually small, the relative heterogeniety does not increase much for a small number of steps H .
Revised: Since the learning rate η t is usually small, the relative heterogeniety does not increase much for a small number of steps H .

Original: Effect of Stochastic Estimation.
Revised: Effect of Stochastic Estimation.

Original: In practice the full gradients are too expensive to compute, so we will resort to stochastic gradients instead.
Revised: In practice the full gradients are too expensive to compute, so we will resort to stochastic gradients instead.

Original: The following proposition controls the error due to the selection of the mixing matrix using stochastic gradients.
Revised: The following proposition controls the error due to the selection of the mixing matrix using stochastic gradients.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let W ∗ ( ξ ) be any mixing matrix dependent on the noise parameters ξ satisfying the given edge constraints.
Revised: Let W ∗ ( ξ ) be any mixing matrix dependent on the noise parameters ξ satisfying the given edge constraints.

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: Proof can be found in the appendix.
Revised: Proof can be found in the appendix.

Original: Setting W ∗ ( ξ ) =
Revised: Setting W ∗ ( ξ ) =

Original: arg min
Revised: arg min

Original: [Equation]
Revised: [Equation]

Original: reveals that minimizing GME with stochastic gradients would also lead to a small heterogeneity ζ up to additive stochastic noise.
Revised: reveals that minimizing GME with stochastic gradients would also lead to a small heterogeneity ζ up to additive stochastic noise.

Original: Sketching for Gram Matrix Estimation.
Revised: Sketching for Gram Matrix Estimation.

Original: The original GME-exact formulation requires transmitting the entire gradients.
Revised: The original GME-exact formulation requires transmitting the entire gradients.

Original: We instead propose to calculate the Gram matrix using sketched gradients, for improved communication efficiency.
Revised: We instead propose to calculate the Gram matrix using sketched gradients, for improved communication efficiency.

Original: Let A denote a random matrix with Gaussian entries and let U be an arbitrary matrix.
Revised: Let A denote a random matrix with Gaussian entries and let U be an arbitrary matrix.

Original: We observe that 1 k E ( UA ) ⊤ UA
Revised: We observe that 1 k E ( UA ) ⊤ UA

Original: = 1 k E U ⊤
Revised: = 1 k E U ⊤

Original: A ⊤ AU
Revised: A ⊤ AU

Original: =
Revised: =

Original: U ⊤ U .
Revised: U ⊤ U .

Original: Therefore, the above projection operation preserves the inner products in expectation.
Revised: Therefore, the above projection operation preserves the inner products in expectation.

Original: The approximation error of the above scheme can be bounded using standard concentration techniques.
Revised: The approximation error of the above scheme can be bounded using standard concentration techniques.

Original: We provide precise theoretical guarantees for sketching and the use of local gradients in section ??
Revised: We provide precise theoretical guarantees for sketching and the use of local gradients in section ??

Original: of the Appendix.
Revised: of the Appendix.

Original: E.2. Optimizing mixing of updates of arbitrary algorithms
Revised: E.2. Optimizing mixing of updates of arbitrary algorithms

Original: Our approach can be generalized to arbitrary additive updates to the parameters of the form x ( t +1)
Revised: Our approach can be generalized to arbitrary additive updates to the parameters of the form x ( t +1)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x ( t ) i + η u ( t ) i .
Revised: x ( t ) i + η u ( t ) i .

Original: Concretely, let U ( t ) denote the matrix with its i th column being u ( t ) i .
Revised: Concretely, let U ( t ) denote the matrix with its i th column being u ( t ) i .

Original: Then, the updates to X can be decoupled as follows:
Revised: Then, the updates to X can be decoupled as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the contribution to the deviation from the mean of the i th node due to the mixing of the updates is again given by (cid:80)
Revised: Therefore, the contribution to the deviation from the mean of the i th node due to the mixing of the updates is again given by (cid:80)

Original: [Equation]
Revised: [Equation]

Original: For example, replacing the gradients in the Algorithm 1 by the updates of the Adam algorithm [20] results in the minimization of the mixing error involved in decentralized Adam updates.
Revised: For example, replacing the gradients in the Algorithm 1 by the updates of the Adam algorithm [20] results in the minimization of the mixing error involved in decentralized Adam updates.

Original: We empirically verify the effectiveness of such an algorithm for an NLP task as discussed in Section J.2.
Revised: We empirically verify the effectiveness of such an algorithm for an NLP task as discussed in Section J.2.

Original: In Appendix I, we discuss other variations and extensions covered by our framework, such as directly optimizing the mixing of parameters.
Revised: In Appendix I, we discuss other variations and extensions covered by our framework, such as directly optimizing the mixing of parameters.

Original: E.3. Guarantees on Sketching and the use of Local Gradients
Revised: E.3. Guarantees on Sketching and the use of Local Gradients

Original: The bound on the error due to sketching is provided through the following extension of the Johnson–Lindenstrauss lemma:
Revised: The bound on the error due to sketching is provided through the following extension of the Johnson–Lindenstrauss lemma:

Original: Proposition 3.
Revised: Proposition 3.

Original: Let { u 1 , · · · , u m } ∈ R d .
Revised: Let { u 1 , · · · , u m } ∈ R d .

Original: Assume that the entries in A ⊂ R k × d are sampled independently from N (0 , 1) .
Revised: Assume that the entries in A ⊂ R k × d are sampled independently from N (0 , 1) .

Original: Then, for k = ω ( log ( mδ ) ε) , with probability greater than 1 − δ , we have:
Revised: Then, for k = ω ( log ( mδ ) ε) , with probability greater than 1 − δ , we have:

Original: [Equation]
Revised: [Equation]

Original: In our algorithm, the { u 1 , · · · , u m } ∈ R d correspond to the gradients across nodes, and are compressed using a the random projection generated independently at each period, using shared seeds.
Revised: In our algorithm, the { u 1 , · · · , u m } ∈ R d correspond to the gradients across nodes, and are compressed using a the random projection generated independently at each period, using shared seeds.

Original: Use of local X .
Revised: Use of local X .

Original: In our practical implementation we solve GME problem for gradients computed at the parameters X instead of X in GME-exact.
Revised: In our practical implementation we solve GME problem for gradients computed at the parameters X instead of X in GME-exact.

Original: We show that this leads to the minimization of the
Revised: We show that this leads to the minimization of the

Original: GME upto an additional term proportional to the consensus:
Revised: GME upto an additional term proportional to the consensus:

Original: Proposition 4.
Revised: Proposition 4.

Original: [Equation]
Revised: [Equation]

Original: Our analysis also provides an estimate of the decrease of consensus distance ∥ X − X ∥ 2 F .
Revised: Our analysis also provides an estimate of the decrease of consensus distance ∥ X − X ∥ 2 F .

Original: Thus, the small right hand side ensures the small relative heterogeneity.
Revised: Thus, the small right hand side ensures the small relative heterogeneity.

Original: E.4. Proof of Proposition
Revised: E.4. Proof of Proposition

Original: We first note that
Revised: We first note that

Original: [Equation]
Revised: [Equation]

Original: We further have:
Revised: We further have:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: Applying Lemma 2 to the second term in the RHS yields:
Revised: Applying Lemma 2 to the second term in the RHS yields:

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Finally, using Equation 22 and the L -smoothness of the objectives
Revised: Finally, using Equation 22 and the L -smoothness of the objectives

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: E.5. Proof of Proposition
Revised: E.5. Proof of Proposition

Original: We start by adding and subtracting the corresponding gradients at the mean parameters X :
Revised: We start by adding and subtracting the corresponding gradients at the mean parameters X :

Original: [Equation]
Revised: [Equation]

Original: E.6. Spectral Norm of Doubly Stochastic Matrices with Non-negative Entries
Revised: E.6. Spectral Norm of Doubly Stochastic Matrices with Non-negative Entries

Original: Proposition 5.
Revised: Proposition 5.

Original: Let W ∈ R n × n be possibly asymmetric doubly stochastic matrix with non-negative entries.
Revised: Let W ∈ R n × n be possibly asymmetric doubly stochastic matrix with non-negative entries.

Original: Then the spectral norm ∥ W ∥ 2 is bounded by 1 .
Revised: Then the spectral norm ∥ W ∥ 2 is bounded by 1 .

Original: Proof We note that W T W is itself a symmetric doubly-stochastic matrix and therefore has an eigenvector 1 √ n 1 with eigenvalue 1 .
Revised: Proof We note that W T W is itself a symmetric doubly-stochastic matrix and therefore has an eigenvector 1 √ n 1 with eigenvalue 1 .

Original: Perron-Frobenius theorem then implies that the largest eigenvalue of ( W ( t ) )
Revised: Perron-Frobenius theorem then implies that the largest eigenvalue of ( W ( t ) )

Original: ⊤ W ( t ) is bounded by 1 , completing the proof.
Revised: ⊤ W ( t ) is bounded by 1 , completing the proof.

Original: E.7. Proof of Proposition
Revised: E.7. Proof of Proposition

Original: The proof proceeds by introducing the stochastic gradients into the LHS as follows:
Revised: The proof proceeds by introducing the stochastic gradients into the LHS as follows:

Original: [Equation]
Revised: [Equation]

Original: Lemma 3 ≤
Revised: Lemma 3 ≤

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 ≤
Revised: Lemma 2 ≤

Original: [Equation]
Revised: [Equation]

Original: Since W ∗ ( ξ ) is doubly-stochastic
Revised: Since W ∗ ( ξ ) is doubly-stochastic

Original: [Equation]
Revised: [Equation]

Original: using Proposition
Revised: using Proposition

Original: [Equation]
Revised: [Equation]

Original: we obtain a bound on the spectral norm (cid:13)(cid:13) W ( ∗ ) (cid:13)(cid:13) 2 ≤ 1 .
Revised: we obtain a bound on the spectral norm (cid:13)(cid:13) W ( ∗ ) (cid:13)(cid:13) 2 ≤ 1 .

Original: Combining the bound on the spectral norm with the assumption on the variance yields:
Revised: Combining the bound on the spectral norm with the assumption on the variance yields:

Original: [Equation]
Revised: [Equation]

Original: E.8. Proof of Proposition
Revised: E.8. Proof of Proposition

Original: We utilize the following compression bound, that arises as a consequence of the concentration of χ 2 random variables, as often utilized in the proof of the Johnson–Lindenstrauss lemma [7]:
Revised: We utilize the following compression bound, that arises as a consequence of the concentration of χ 2 random variables, as often utilized in the proof of the Johnson–Lindenstrauss lemma [7]:

Original: Lemma 7.
Revised: Lemma 7.

Original: Let { u 1 , · · · , u m } ∈ R d .
Revised: Let { u 1 , · · · , u m } ∈ R d .

Original: Assume that the entries in A ⊂ R k × d are sampled independently from N (0 , 1) .
Revised: Assume that the entries in A ⊂ R k × d are sampled independently from N (0 , 1) .

Original: Then, for k ≥ 100( log ( mδ ) ε) , with probability greater than 1 − δ , we have, ∀ i, j ∈ [ m ] :
Revised: Then, for k ≥ 100( log ( mδ ) ε) , with probability greater than 1 − δ , we have, ∀ i, j ∈ [ m ] :

Original: [Equation]
Revised: [Equation]

Original: Slightly weaker bounds can be obtained in more general settings such as that of sub-Gaussian random variables but we restrict to the Gaussian case in the theory as well as implementations of our algorithm.
Revised: Slightly weaker bounds can be obtained in more general settings such as that of sub-Gaussian random variables but we restrict to the Gaussian case in the theory as well as implementations of our algorithm.

Original: Now, adding {− u 1 , · · · , − u m } to the set of points and applying Lemma 7 yields, ∀ i, j ∈ [ m ] :
Revised: Now, adding {− u 1 , · · · , − u m } to the set of points and applying Lemma 7 yields, ∀ i, j ∈ [ m ] :

Original: [Equation]
Revised: [Equation]

Original: Therefore, we bound the inner product as follows:
Revised: Therefore, we bound the inner product as follows:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we obtain the lower bound:
Revised: Similarly, we obtain the lower bound:

Original: [Equation]
Revised: [Equation]

Original: Appendix F. Using different matrices for Parameter and Gradient Mixing
Revised: Appendix F. Using different matrices for Parameter and Gradient Mixing

Original: An additional advantage of our analysis is that it decouples the effect of parameter and gradient mixing.
Revised: An additional advantage of our analysis is that it decouples the effect of parameter and gradient mixing.

Original: This allows our analysis to be extended to the case of use of different mixing matrices W p and W g for mixing the parameters and gradients at each step respectively.
Revised: This allows our analysis to be extended to the case of use of different mixing matrices W p and W g for mixing the parameters and gradients at each step respectively.

Original: Concretely, we consider the following algorithm:
Revised: Concretely, we consider the following algorithm:

Original: Algorithm 3 D ECENTRALIZED SGD WITH DECOUPLED MIXING
Revised: Algorithm 3 D ECENTRALIZED SGD WITH DECOUPLED MIXING

Original: X (0) , stepsizes { η t } T − 1 t =0 , number of iterations T , mixing matrix distributions W ( t ) p , W ( t ) g , t ∈ [0 , T ] for t in 0 . . .
Revised: X (0) , stepsizes { η t } T − 1 t =0 , number of iterations T , mixing matrix distributions W ( t ) p , W ( t ) g , t ∈ [0 , T ] for t in 0 . . .

Original: T do in parallel on all workers 1:2:
Revised: T do in parallel on all workers 1:2:

Original: G ( t ) = ∂F ( X ( t ) , ξ ( t ) )
Revised: G ( t ) = ∂F ( X ( t ) , ξ ( t ) )

Original: ▷ stochastic gradients 3:
Revised: ▷ stochastic gradients 3:

Original: W ( t ) p ∼ W ( t ) p , W ( t ) g ∼ W ( t )
Revised: W ( t ) p ∼ W ( t ) p , W ( t ) g ∼ W ( t )

Original: g ▷ sample mixing matrices 4:
Revised: g ▷ sample mixing matrices 4:

Original: X ( t +1) =
Revised: X ( t +1) =

Original: X ( t ) W ( t ) p − η t G ( t ) W ( t )
Revised: X ( t ) W ( t ) p − η t G ( t ) W ( t )

Original: g ▷ update & mixing 5: end parallel for
Revised: g ▷ update & mixing 5: end parallel for

Original: We now show that the above algorithm leads to convergence rates having the same dependence on p and ζ ′ as Theorem 1 but with these parameters defined as above in terms of W ( t ) u and W ( t ) g .
Revised: We now show that the above algorithm leads to convergence rates having the same dependence on p and ζ ′ as Theorem 1 but with these parameters defined as above in terms of W ( t ) u and W ( t ) g .

Original: For instance, for the Non-convex case, we obtain that
Revised: For instance, for the Non-convex case, we obtain that

Original: [Equation]
Revised: [Equation]

Original: ( t )
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: 2 2 ≤ ε after
Revised: 2 2 ≤ ε after

Original: [Equation]
Revised: [Equation]

Original: iterations.
Revised: iterations.

Original: Analogously, we obtain the corresponding convergence rates for the convex case with ζ ′ ⋆ defined at the optimum i.e. E W g ∼W ( t ) g 1 n (cid:13)(cid:13)
Revised: Analogously, we obtain the corresponding convergence rates for the convex case with ζ ′ ⋆ defined at the optimum i.e. E W g ∼W ( t ) g 1 n (cid:13)(cid:13)

Original: ∂f ( X ⋆ ) W g − ∂f ( X ⋆ ) (cid:13)(cid:13) 2 ≤ ζ ′ 2 .
Revised: ∂f ( X ⋆ ) W g − ∂f ( X ⋆ ) (cid:13)(cid:13) 2 ≤ ζ ′ 2 .

Original: Similar to Lemma 5, the update can then be expressed as
Revised: Similar to Lemma 5, the update can then be expressed as

Original: [Equation]
Revised: [Equation]

Original: Subsequently, analogous to the proof of Theorem 1, we obtain the following decomposition of the consensus iterates:
Revised: Subsequently, analogous to the proof of Theorem 1, we obtain the following decomposition of the consensus iterates:

Original: [Equation]
Revised: [Equation]

Original: Now, for p and ζ ′ satisfying:
Revised: Now, for p and ζ ′ satisfying:

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: we obtain the analogous consensus recursion:
Revised: we obtain the analogous consensus recursion:

Original: [Equation]
Revised: [Equation]

Original: 28) where D = 36 L +
Revised: 28) where D = 36 L +

Original: D-cliques [6] : Suppose that the graph can be divided into K cliques, such that the mean gradient for each clique equals the mean across the entire graph.
Revised: D-cliques [6] : Suppose that the graph can be divided into K cliques, such that the mean gradient for each clique equals the mean across the entire graph.

Original: Let the nodes be numbered such that the n k nodes belonging to the k th clique succeed the n k − 1 nodes belonging to the ( k − 1) th clique.
Revised: Let the nodes be numbered such that the n k nodes belonging to the k th clique succeed the n k − 1 nodes belonging to the ( k − 1) th clique.

Original: Then, we observe that utilizing a block matrix of the type 
Revised: Then, we observe that utilizing a block matrix of the type 

Original: [Equation]
Revised: [Equation]

Original:  leads to zero Gradient Mixing Error.
Revised:  leads to zero Gradient Mixing Error.

Original: This corresponds to the proposed algorithm in D-cliques [6] where W ( t ) g is set to a matrix performing uniform averaging within each clique, while W ( t ) u utilizes all the edges for mixing.
Revised: This corresponds to the proposed algorithm in D-cliques [6] where W ( t ) g is set to a matrix performing uniform averaging within each clique, while W ( t ) u utilizes all the edges for mixing.

Original: For unbiased cliques, we obtain ζ ′ =
Revised: For unbiased cliques, we obtain ζ ′ =

Original: Therefore, our analysis above provides an explanation for the improvements achieved by decoupled parameter mixing and cliqueaveraging [6] under the presence of unbiased cliques.
Revised: Therefore, our analysis above provides an explanation for the improvements achieved by decoupled parameter mixing and cliqueaveraging [6] under the presence of unbiased cliques.

Original: We further note that, unlike the algorithm presented in [6], our algorithm HA-DSGD with random sampling of mixing matrices does not involve the additional communication overhead for separately mixing the gradients at each timestep.
Revised: We further note that, unlike the algorithm presented in [6], our algorithm HA-DSGD with random sampling of mixing matrices does not involve the additional communication overhead for separately mixing the gradients at each timestep.

Original: Appendix G. Mixing Error under Permutations
Revised: Appendix G. Mixing Error under Permutations

Original: We now demonstrate how the “Gradient Mixing Error” can be used to guide the choice of the arrangement of a given set of nodes over a graph.
Revised: We now demonstrate how the “Gradient Mixing Error” can be used to guide the choice of the arrangement of a given set of nodes over a graph.

Original: Given a set of nodes having fixed data distributions,
Revised: Given a set of nodes having fixed data distributions,

Original: [Figure]
Revised: [Figure]

Original: the parameters controlling the Gradient Mixing Error (GME) is controlled by the choice of mixing weights as well as the graph topology.
Revised: the parameters controlling the Gradient Mixing Error (GME) is controlled by the choice of mixing weights as well as the graph topology.

Original: To illustrate the effects of the choice of topology on the convergence rates, we consider a toy setup of 4 nodes, having data distributions defined by quadratic objectives as in Section 6.
Revised: To illustrate the effects of the choice of topology on the convergence rates, we consider a toy setup of 4 nodes, having data distributions defined by quadratic objectives as in Section 6.

Original: To further illustrate the benefits of selecting an appropriate permutation, we consider a setup of 16 nodes distributed on a ring topology with the data distributions of exactly half of the nodes belonging to each one of the following class of objectives:
Revised: To further illustrate the benefits of selecting an appropriate permutation, we consider a setup of 16 nodes distributed on a ring topology with the data distributions of exactly half of the nodes belonging to each one of the following class of objectives:

Original: [Equation]
Revised: [Equation]

Original: where A denotes a fixed matrix with entries from N (0 , 1) .
Revised: where A denotes a fixed matrix with entries from N (0 , 1) .

Original: We simulate the noise in SGD, by adding random vectors ξ ( t ) ∼ N (0 , 0 . 001) to the gradient updates for each node We compare the performance of DSGD under the following two permutations and choices of the mixing matrices:
Revised: We simulate the noise in SGD, by adding random vectors ξ ( t ) ∼ N (0 , 0 . 001) to the gradient updates for each node We compare the performance of DSGD under the following two permutations and choices of the mixing matrices:

Original: Heterogenous pairing: As illustrated in Figure
Revised: Heterogenous pairing: As illustrated in Figure

Original: the nodes are ordered around the ring alternating between the data for objectives f 1 and f 2 .
Revised: the nodes are ordered around the ring alternating between the data for objectives f 1 and f 2 .

Original: Subsequently
Revised: Subsequently

Original: , every
Revised: , every

Original: node is paired with exactly one of its neighbours such that the mixing steps involve averaging between the members of the pairs with equal weights of 0 .
Revised: node is paired with exactly one of its neighbours such that the mixing steps involve averaging between the members of the pairs with equal weights of 0 .

Original: Random permutation: The nodes are randomly distributed on the ring with the mixing matrix corresponding to the maximal spectral gap.
Revised: Random permutation: The nodes are randomly distributed on the ring with the mixing matrix corresponding to the maximal spectral gap.

Original: We provide illustrations of the setup and the results in Figures 2 and 3 respectively, confirming the improvements in convergence due to the minimization of the GME.
Revised: We provide illustrations of the setup and the results in Figures 2 and 3 respectively, confirming the improvements in convergence due to the minimization of the GME.

Original: Appendix H. Effect of Varying Data-Heterogeneity
Revised: Appendix H. Effect of Varying Data-Heterogeneity

Original: Since ζ ′ ≤ ζ , reducing the heterogeneity of gradients across clients diminishes the role played by relative heterogeneity in the convergence rates.
Revised: Since ζ ′ ≤ ζ , reducing the heterogeneity of gradients across clients diminishes the role played by relative heterogeneity in the convergence rates.

Original: We verify this empirically on quadratic objectives under a similar setup as 6 with a random connected graph on 16 nodes having 60 edges.
Revised: We verify this empirically on quadratic objectives under a similar setup as 6 with a random connected graph on 16 nodes having 60 edges.

Original: [Figure]
Revised: [Figure]

Original: We consider local objectives of the form f i ( x ) =
Revised: We consider local objectives of the form f i ( x ) =

Original: ∥ (( I + A i ) x − ( 1 + b i ) ∥ 2 2 with I denoting the identity matrix, the dimension of the parameters x being d = 10 and A i , b i containing entries sampled randomly from N (0 , τ ) and fixed for each client.
Revised: ∥ (( I + A i ) x − ( 1 + b i ) ∥ 2 2 with I denoting the identity matrix, the dimension of the parameters x being d = 10 and A i , b i containing entries sampled randomly from N (0 , τ ) and fixed for each client.

Original: The parameter τ therefore controls the level of heterogeneity acrosss clients.
Revised: The parameter τ therefore controls the level of heterogeneity acrosss clients.

Original: In Figures 

Revised: In Figures

Original: and we observe that reducing the level of heterogeneity across clients limits the improvements achieved by HA-DSGD in the distance to the optimum, while the consensus error and the GME are still lower than those for DSGD
Revised: and we observe that reducing the level of heterogeneity across clients limits the improvements achieved by HA-DSGD in the distance to the optimum, while the consensus error and the GME are still lower than those for DSGD

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Appendix I. Possible Extensions
Revised: Appendix I. Possible Extensions

Original: Theorem 1 does not cover the just discussed case of alternating between two or more matrices.
Revised: Theorem 1 does not cover the just discussed case of alternating between two or more matrices.

Original: As our main focus in this work is on highlighting the benefits of relative heterogeneity, we just covered a simple case of time-varying mixing in the theorem (when all matrices are sampled from the same distribution).
Revised: As our main focus in this work is on highlighting the benefits of relative heterogeneity, we just covered a simple case of time-varying mixing in the theorem (when all matrices are sampled from the same distribution).

Original: However, it is possible to extend our analysis to deterministic sequences (such as alternating) with the derandomization technique presented in [22, Assumption 4, Theorem 2].
Revised: However, it is possible to extend our analysis to deterministic sequences (such as alternating) with the derandomization technique presented in [22, Assumption 4, Theorem 2].

Original: Our
Revised: Our

Original: [Figure]
Revised: [Figure]

Original: We report an average over a window of corresponding quantity analysis can also be extended to the case of optimizing parameter averaging and the use of two separate mixing matrices to mix parameters and gradients respectively (similar as in Bellet et al. 6) as discussed in sections I.2 and F respectively.
Revised: We report an average over a window of corresponding quantity analysis can also be extended to the case of optimizing parameter averaging and the use of two separate mixing matrices to mix parameters and gradients respectively (similar as in Bellet et al. 6) as discussed in sections I.2 and F respectively.

Original: I.1. Optimizing mixing of updates of arbitrary algorithms
Revised: I.1. Optimizing mixing of updates of arbitrary algorithms

Original: Our approach can be generalized to arbitrary additive updates to the parameters of the form x ( t +1)
Revised: Our approach can be generalized to arbitrary additive updates to the parameters of the form x ( t +1)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x ( t ) i + η u ( t ) i .
Revised: x ( t ) i + η u ( t ) i .

Original: Concretely, let U ( t ) denote the matrix with its i th column being u ( t ) i .
Revised: Concretely, let U ( t ) denote the matrix with its i th column being u ( t ) i .

Original: Then, the updates to X can be decoupled as follows:
Revised: Then, the updates to X can be decoupled as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the contribution to the deviation from the mean of the i th node due to the mixing of the updates is again given by (cid:80) nj =1 W ji u j − u .
Revised: Therefore, the contribution to the deviation from the mean of the i th node due to the mixing of the updates is again given by (cid:80) nj =1 W ji u j − u .

Original: For example, replacing the gradients in the Algorithm 1 by the updates of the Adam algorithm [20] results in the minimization of the mixing error involved in decentralized Adam updates.
Revised: For example, replacing the gradients in the Algorithm 1 by the updates of the Adam algorithm [20] results in the minimization of the mixing error involved in decentralized Adam updates.

Original: We empirically verify the effectiveness of such an algorithm for an NLP task as discussed in Section J.2.
Revised: We empirically verify the effectiveness of such an algorithm for an NLP task as discussed in Section J.2.

Original: In Appendix I, we discuss other variations and extensions covered by our framework, such as directly optimizing the mixing of parameters.
Revised: In Appendix I, we discuss other variations and extensions covered by our framework, such as directly optimizing the mixing of parameters.

Original: An alternate way of simultaneously maximizing the consensus factor p and the gradient mixing error is to directly optimize the mixing error of the parameters i.e. (cid:13)(cid:13)
Revised: An alternate way of simultaneously maximizing the consensus factor p and the gradient mixing error is to directly optimize the mixing error of the parameters i.e. (cid:13)(cid:13)

Original: ( X ( t ) − X ( t ) )
Revised: ( X ( t ) − X ( t ) )

Original: W (cid:13)(cid:13) 2 F .
Revised: W (cid:13)(cid:13) 2 F .

Original: Our theoretical analysis covers such a choice of mixing matrices as a special case that involves trying to obtain a mixing matrix having both small (1 − p ) and the gradient mixing error.
Revised: Our theoretical analysis covers such a choice of mixing matrices as a special case that involves trying to obtain a mixing matrix having both small (1 − p ) and the gradient mixing error.

Original: However, unlike the gradient mixing error that involves changes of the order η 2 as shown by Lemma 1, the distribution of the parameters across nodes can change rapidly due to the mixing.
Revised: However, unlike the gradient mixing error that involves changes of the order η 2 as shown by Lemma 1, the distribution of the parameters across nodes can change rapidly due to the mixing.

Original: Moreover, we found both approaches to yield similar improvements in practice and focus on the gradient mixing error since it covers a wider range of design choices such as mixing within unbiased cliques.
Revised: Moreover, we found both approaches to yield similar improvements in practice and focus on the gradient mixing error since it covers a wider range of design choices such as mixing within unbiased cliques.

Original: Appendix J. Additional Experiments and Details J.1. Quadratic Objectives
Revised: Appendix J. Additional Experiments and Details J.1. Quadratic Objectives

Original: Details of the sampling procedure for graphs : We generate a random connected graph ofnodes through the following procedure: starting from a fully connected graph having 16 nodes and 120 edges, we repeatedly construct graphs by deleted 60 of the edges chosen uniformly at random until we obtain a connected graph having 60 edges.
Revised: Details of the sampling procedure for graphs : We generate a random connected graph ofnodes through the following procedure: starting from a fully connected graph having 16 nodes and 120 edges, we repeatedly construct graphs by deleted 60 of the edges chosen uniformly at random until we obtain a connected graph having 60 edges.

Original: Experiments on 64 nodes : Similar to Section 6, we generate a random graph containingnodes with half of the edges randomly removed from a complete graph.
Revised: Experiments on 64 nodes : Similar to Section 6, we generate a random graph containingnodes with half of the edges randomly removed from a complete graph.

Original: Figure 6 shows the results in this setting for quadratic objectives, confirming the effectiveness of HADSGD on large graphs.
Revised: Figure 6 shows the results in this setting for quadratic objectives, confirming the effectiveness of HADSGD on large graphs.

Original: J.2. Deep Learning Benchmarks
Revised: J.2. Deep Learning Benchmarks

Original: In all our results, the period denotes the number of updates after which the mixing matrix is recomputed i.e. a period of 100 implies that the communication of the compressed gradients and the computation of the mixing matrix occurs only for a 1100 fraction of the updates.
Revised: In all our results, the period denotes the number of updates after which the mixing matrix is recomputed i.e. a period of 100 implies that the communication of the compressed gradients and the computation of the mixing matrix occurs only for a 1100 fraction of the updates.

Original: Furthermore, in
Revised: Furthermore, in

Original: Appendix J.4, we empirically verify that a sketching dimension of 100 − 1000 is sufficient for models having millions of parameters.
Revised: Appendix J.4, we empirically verify that a sketching dimension of 100 − 1000 is sufficient for models having millions of parameters.

Original: These two factors make our overhead in communication cost negligible compared to the baselines.
Revised: These two factors make our overhead in communication cost negligible compared to the baselines.

Original: In the Appendix J.3, we discuss utilizing the periodic global communication to perform averaging using the decoded sketched gradients.
Revised: In the Appendix J.3, we discuss utilizing the periodic global communication to perform averaging using the decoded sketched gradients.

Original: CIFAR10.
Revised: CIFAR10.

Original: We evaluate our approach on the CIFAR10 dataset [25] by training the Resnet model [41] with Evonorm [29] for 300 epochs for each model.
Revised: We evaluate our approach on the CIFAR10 dataset [25] by training the Resnet model [41] with Evonorm [29] for 300 epochs for each model.

Original: Following Sec.
Revised: Following Sec.

Original: I.1, we consider the extension of our algorithm to the mixing of Nesterov momentum updates, denoted by HA-DSGD (momentum) in Table 1, and compare against the corresponding version of DSGD with momentum.
Revised: I.1, we consider the extension of our algorithm to the mixing of Nesterov momentum updates, denoted by HA-DSGD (momentum) in Table 1, and compare against the corresponding version of DSGD with momentum.

Original: We also compare against the D 2 algorithm [46] and Gradient Tracking [23, 30, 31, 37, 40] for completeness.
Revised: We also compare against the D 2 algorithm [46] and Gradient Tracking [23, 30, 31, 37, 40] for completeness.

Original: The results show that our approach generally outperforms the baselines across three topologies, ring ( n = 16 ), torus ( n = 16 ), as well as the topology defined by the Davis Southern
Revised: The results show that our approach generally outperforms the baselines across three topologies, ring ( n = 16 ), torus ( n = 16 ), as well as the topology defined by the Davis Southern

Original: Women dataset as available in the Networkx library [14].
Revised: Women dataset as available in the Networkx library [14].

Original: Since both the Metropolis-Hastings and the optimal spectral gap mixing schemes lead to similar results, we only compare against the
Revised: Since both the Metropolis-Hastings and the optimal spectral gap mixing schemes lead to similar results, we only compare against the

Original: Metropolis-Hastings schemes in the subsequent tasks
Revised: Metropolis-Hastings schemes in the subsequent tasks

Original: [Table]
Revised: [Table]

Original: averaged over three random
Revised: averaged over three random

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Transformer on AG News.
Revised: Transformer on AG News.

Original: We evaluate the extension of our algorithm to the mixing of
Revised: We evaluate the extension of our algorithm to the mixing of

Original: Adam [20] updates on the NLP task of fine-tuning the distilbert-base-uncased model [54] on the AGNews dataset [61].
Revised: Adam [20] updates on the NLP task of fine-tuning the distilbert-base-uncased model [54] on the AGNews dataset [61].

Original: Table 3 verifies the applicability of our approach to Adam updates.
Revised: Table 3 verifies the applicability of our approach to Adam updates.

Original: Imagenet.
Revised: Imagenet.

Original: To evaluate our approach on a large-scale dataset, we consider the task of training a
Revised: To evaluate our approach on a large-scale dataset, we consider the task of training a

Original: Resnet18 model [41] with evonorm on the Imagenet dataset [11].
Revised: Resnet18 model [41] with evonorm on the Imagenet dataset [11].

Original: We use a larger period of 1000 for the optimization of the mixing matrix to account for the larger number of steps per epoch.
Revised: We use a larger period of 1000 for the optimization of the mixing matrix to account for the larger number of steps per epoch.

Original: We train each model using Nesterov momentum for 90 epochs using a ring topology defined on 16 nodes.
Revised: We train each model using Nesterov momentum for 90 epochs using a ring topology defined on 16 nodes.

Original: [Table]
Revised: [Table]

Original: Similar to other settings, our approach as shown in Table 2 outperforms DSGD, demonstrating its effectiveness under large period and dataset sizes.
Revised: Similar to other settings, our approach as shown in Table 2 outperforms DSGD, demonstrating its effectiveness under large period and dataset sizes.

Original: J.3. Effect of Periodic Averaging
Revised: J.3. Effect of Periodic Averaging

Original: While the use of sketching significantly reduces the communication cost, our algorithm is still not fully decentralized due to the requirement of periodically communicating the sketched gradients to a central server or node.
Revised: While the use of sketching significantly reduces the communication cost, our algorithm is still not fully decentralized due to the requirement of periodically communicating the sketched gradients to a central server or node.

Original: When the full gradient is utilized, or a suitable compression scheme is used
Revised: When the full gradient is utilized, or a suitable compression scheme is used

Original: [Table]
Revised: [Table]

Original: we can further utilize the central communication step to perform an averaging of the parameters
Revised: we can further utilize the central communication step to perform an averaging of the parameters

Original: [Table]
Revised: [Table]

Original: J.4. Effect of the Compression Dimension
Revised: J.4. Effect of the Compression Dimension

Original: Proposition 3 predicts that a low approximation error in the entries of the Gram matrix can be achieved through compression with dimension independent of the number of parameters and logarithmic in the number of nodes.
Revised: Proposition 3 predicts that a low approximation error in the entries of the Gram matrix can be achieved through compression with dimension independent of the number of parameters and logarithmic in the number of nodes.

Original: We empirically verify this for the CIFAR10 dataset using HADSGD with Nesterov momemtum and a period of 100 in table
Revised: We empirically verify this for the CIFAR10 dataset using HADSGD with Nesterov momemtum and a period of 100 in table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Appendix K. Limitations
Revised: Appendix K. Limitations

Original: Like other works in the optimization literature, our convergence analysis does not directly explain generalization.
Revised: Like other works in the optimization literature, our convergence analysis does not directly explain generalization.

Original: However, we empirically validate improvements in generalization performance on several deep learning benchmarks.
Revised: However, we empirically validate improvements in generalization performance on several deep learning benchmarks.

Original: Our work also assumes a fixed topology and incorporating timevarying and adapting topology is a promising direction for future work
Revised: Our work also assumes a fixed topology and incorporating timevarying and adapting topology is a promising direction for future work

Original: [Table]
Revised: [Table]

Original: Appendix L. Societal Impact
Revised: Appendix L. Societal Impact

Original: The demands for preserving privacy in machine learning training systems have been constantly growing over the past few years [18, 36].
Revised: The demands for preserving privacy in machine learning training systems have been constantly growing over the past few years [18, 36].

Original: We believe that Decentralized learning can play a major role in meeting such demands.
Revised: We believe that Decentralized learning can play a major role in meeting such demands.

Original: This can improve the trust towards Machine Learning applications as well as maintenance of data-ownership in society.
Revised: This can improve the trust towards Machine Learning applications as well as maintenance of data-ownership in society.

Original: Additionally, improvements in efficiency of decentralized optimization algorithms can reduce the environmental impact of training large machine learning models.
Revised: Additionally, improvements in efficiency of decentralized optimization algorithms can reduce the environmental impact of training large machine learning models.

Original: We believe that the focus of our work towards the setting of heterogeneous data makes it especially relevant for practical settings.
Revised: We believe that the focus of our work towards the setting of heterogeneous data makes it especially relevant for practical settings.

Original: Appendix M. Conclusion and Future Work
Revised: Appendix M. Conclusion and Future Work

Original: In this work, we extended the analysis of DSGD to incorporate the interaction between the mixing matrix and the data heterogeneity, leading to a novel technique for dynamically adapting the mixing matrix throughout training.
Revised: In this work, we extended the analysis of DSGD to incorporate the interaction between the mixing matrix and the data heterogeneity, leading to a novel technique for dynamically adapting the mixing matrix throughout training.

Original: Future work could involve extending our technique to algorithms designed for specific settings such as EXTRA [44] for convex non-stochastic cases, as well as approaches based on row-stochastic, column-stochastic matrices and time-varying topologies.
Revised: Future work could involve extending our technique to algorithms designed for specific settings such as EXTRA [44] for convex non-stochastic cases, as well as approaches based on row-stochastic, column-stochastic matrices and time-varying topologies.

Original: On the theoretical side, promising directions include extending our analysis to the mixing of momentum.
Revised: On the theoretical side, promising directions include extending our analysis to the mixing of momentum.