Communication-Efﬁcient Topologies for Decentralized Learning with O (1) Consensus Rate
Abstract
Decentralized optimization is an emerging paradigm in distributed learning in which agents achieve network-wide solutions by peer-to-peer communication without the central server.
Since communication tends to be slower than computation, when each agent communicates with only a few neighboring agents per iteration, they can complete iterations faster than with more agents or a central server.
However, the total number of iterations to reach a network-wide solution is affected by the speed at which the agents’ information is “mixed” by communication.
We found that popular communication topologies either have large maximum degrees (such as stars and complete graphs) or are ineffective at mixing information (such as rings and grids).
To address this problem, we propose a new family of topologies,
EquiTopo, which has an (almost) constant degree and a network-size-independent consensus rate that is used to measure the mixing efﬁciency.
In the proposed family, EquiStatic has a degree of Θ(ln( n )) , where n is the network size, and a series of time-dependent one-peer topologies, EquiDyn, has a constant degree of 1.
We generate EquiDyn through a certain random sampling procedure.
Both of them achieve an n -independent consensus rate.
We apply them to decentralized SGD and decentralized gradient tracking and obtain faster communication and better convergence, theoretically and empirically.
Our code is implemented through
BlueFog and available at https://github.com/kexinjinnn/EquiTopo .
Introduction
Modern optimization and machine learning typically involve tremendous data samples and model parameters.
The scale of these problems calls for efﬁcient distributed algorithms across multiple computing nodes.
Traditional distributed approaches usually follow a centralized setup, where each node needs to communicate with a (virtually) central server.
This communication pattern incurs signiﬁcant communication overheads and long latency.
Decentralized learning is an emerging paradigm to save communications in large-scale optimization and learning.
In decentralized learning, all computing nodes are connected with some network topology (e.g., ring, grid, hypercube, etc.)
in which each node averages/communicates locally with its immediate neighbors.
This decentralized setup allows each node to communicate with fewer neighbors and hence has a much lower overhead in per-iteration communication.
However, local averaging is less effective in “mixing” information, making decentralized algorithms converge slower
[Table]
than their centralized counterparts.
Therefore, seeking a balance between communication efﬁciency and convergence rate in decentralized learning is critical.
The network topology (or graph) determines decentralized algorithms’ per-iteration communication and convergence rate.
The maximum graph degree controls the communication cost, whereas the connectivity inﬂuences the convergence rate.
Intuitively speaking, a densely-connected topology enables decentralized methods to converge faster but results in less efﬁcient communication since each node needs to average with more neighbors.
Selecting an appropriate network topology is key to achieving light communication and fast convergence in decentralized learning.
1.1 Prior arts in topology selections
Gossip matrix and consensus rate.
Given a connected network of size n and its associated doublystochastic gossip matrix W ∈ R n × n (see the deﬁnition in § 2), its consensus rate β determines how effective the gossip operation W x is to mix information (see more explanations in § 2).
It is a long-standing topic in decentralized learning to seek topologies with both a small maximum degree and a fast consensus rate (i.e., a small β as close to 0 as possible).
Static graphs.
Static topologies maintain the same graph connections throughout all iterations.
The directedness, degree, and consensus rate of various common topologies are summarized in
Table 1.
The ring, grid, and torus graphs [22] are the simplest sparse topologies with Θ(1) maximum degree.
However, their consensus rates quickly approach 1 as network size n increases, which leads to inefﬁcient local averaging.
The hypercube graph [31] maintains a nice balance between degree and consensus rate since ln( n ) varies slowly with n .
However, this graph cannot be formed when size n is not the power of 2 .
The static exponential graph extends hypercubes to graphs with any size n , but its directed communications cannot enable symmetric gossip matrices required in well-known decentralized algorithms such as EXTRA [28], Exact-Diffusion [40], NIDS [16], D 2 [30].
Two widely-used random topologies, i.e., the Erdos-Renyi graph [20, 3] and the geometric random graph [4, 5], are also listed in Table 1.
It is observed that the Erdos-Renyi graph achieves a networksize-independent consensus rate with a Θ(ln( n )) averaged degree in expectation.
However, it is worth noting that the communication overhead in network topology is determined by the maximum degree.
Since some nodes may have much more neighbors than others in a random realization, the maximum degree in the Erdos-Renyi graph can be O ( n ) with a non-zero probability.
Moreover, the random graphs listed in Table 1 are undirected.
The may not be used in scenarios where directed graphs are preferred.
Dynamic graphs.
Dynamic graphs allow time-varying topologies between iterations.
When an exponential graph allows each node to cycle through all its neighbors and communicate only to a single node per iteration, we achieve the time-varying one-peer exponential graph [2, 37].
When the network size n is the power of 2 , a sequence of one-peer exponential graphs can together achieve periodic global averaging.
However, its consensus rate is unknown for other values of n .
A closely related work Matcha [34] proposed a disjoint matching decomposition sampling strategy when training learning models.
While it decomposes a static dense graph into a series of sparse graphs with small degrees, the consensus rates of these dynamic graphs are not established.
Finally, it is worth noting that the consensus rates of all graphs (except for the Erdos-Renyi graph) discussed above are either unknown or dependent on size n .
Their efﬁciency in mixing information gets less effective as n goes large.
1.2 Main results
Motivation.
Since existing network topologies suffer from several limitations, we ask the following questions.
Can we develop topologies that have (almost) constant degrees and network-sizeindependent consensus rates that admit both symmetric and asymmetric matrices of any size?
Can these topologies allow one-peer dynamic variants?
This paper provides afﬁrmative answers.
Main results and contributions.
This paper develops several novel graphs built upon a set of basis graphs in which the label difference between any pair of connected nodes are equivalent .
With a general name EquiTopo, these new graphs can achieve network-size-independent consensus rates while maintaining (almost) constant graph degrees.
Our contributions are:
• We construct a directed graph named D-EquiStatic that has a network-size-independent consensus rate ρ with a degree Θ(ln( n )) .
Furthermore, we develop an one-peer time-varying variant named OD-EquiDyn to achieve a network-size-independent consensus rate with degree 1 .
• We construct a undirected graph U-EquiStatic, which has a network-size-independent consensus rate ρ with degree Θ(ln( n )) .
It admits symmetric gossip matrices that are required by various important algorithms.
We also develop an one-peer time-varying and undirected variant named OU-EquiDyn to achieve a network-size-independent consensus rate with degree 1 .
• We apply the EquiTopo graphs to two well-known decentralized algorithms,
i
e., decentralized stochastic gradient descent (SGD) [6, 17, 12] and stochastic gradient tracking (SGT) [23, 36, 7, 35], to achieve the state-of-the-art convergence rate while maintaining Θ(ln( n ))
(with D/UEquiStatic) or 1 (with OD/OU-EquiDyn) degree in per-iteration communication.
The comparison between EquiTopo and other common topologies in Table 1 shows that the EquiTopo family (especially the one-peer variants) has achieved the best balance between maximum graph degree and consensus rate.
The comparison between EquiTopo and other common topologies when applying to DSGD and DSGT are listed in Tables 3 and 4 in Appendix C.2 and Table 2.
Note.
This paper considers scenarios in which any two nodes can be connected when necessary.
The high-performance data center cluster is one such scenario in which all GPUs are connected with high-bandwidth channels, and the network topology can be fully controlled.
EquiTopo may not be applied to wireless network settings where two remote nodes cannot be connected directly.
1.3 Other related works
In decentralized optimization, decentralized gradient descent [24, 6, 26, 39] and dual averaging [8] are well-known approaches.
While simple and widely used, their solutions are sensitive to heterogeneous data distributions.
Advanced algorithms that can overcome this drawback include explicit biascorrection [28, 40, 16], gradient tracking [23, 7, 25, 36], and dual acceleration [27, 33].
Decentralize
SGD is extensively studied in [6, 17, 2] to solve stochastic problems.
It has been extended to directed [2, 37] or time-varying topologies [12, 21, 34, 37], asynchronous settings [18], and dataheterogeneous scenarios [30, 35, 1, 11, 19] to achieve better performances.
Notations and Preliminaries
Notations.
We let 1 n ∈ R n be the all-ones vector and I ∈ R n × n be the identity matrix.
Furthermore, we deﬁne J = 1 n 1 n 1 Tn and Π =
I − J .
A matrix A =
[ a ij ] ∈ R n × n is nonnegative if a ij ≥for all 1 ≤ i, j ≤ n .
A nonnegative matrix A is doubly stochastic if A 1 n =
A T 1 n = 1 n .
Given a matrix A ∈ R m × n , (cid:107) A (cid:107) 2 is its spectral norm.
For x ∈ R n , (cid:107) x (cid:107) is its Euclidean norm.
We let [ n ] = { 1 , · · · , n } .
Throughout the paper, we deﬁne a mod operation that returns a value in [ n ] as i mod n = (cid:26) (cid:96) if i = kn + (cid:96) for some k ∈ Z and (cid:96) ∈
[ n − 1] , n if i = kn for some k ∈ Z .
Network.
Given a graph G ( V , E ) with a set of n nodes V and a set of directed edges E .
An edge ( j, i
) ∈
E means node j can directly send information to node i .
For undirected graphs, ( j, i )
∈ E if and only if ( i, j ) ∈ E .
Node i ’s degree is the number of its in-neighbors |{ j | ( j, i ) ∈
E}| .
A one-peer graph means that the degree for each node is at most 1.
Weight matrices.
To facilitate the local averaging step in decentralized algorithms, each graph is associated with a nonnegative weight matrix W =
[ w ij ] ∈ R n × n , whose element w ij is non-zero only if ( j, i )
∈ E or i = j .
One beneﬁt of an undirected graph is that it can be associated with a symmetric matrix.
Given a nonnegative weight matrix W ∈ R n × n , we let G ( W ) be its associated graph such that ( j, i ) ∈
E if w ij > 0 and i (cid:54) =
j .
Consensus rate.
For weight matrices { W ( t ) } t ≥ 0 ⊆
R n × n , the consensus rate β is the minimum nonnegative number such that for any t ≥ 0 and vector x ∈ R n with the average ¯ x = 1 n (cid:80) ni
=1 x i ,
E (cid:20)(cid:13)(cid:13)(cid:13) W ( t ) x
− ¯ x · 1 n (cid:13)(cid:13)(cid:13) 2 (cid:21) ≤ β 2 (cid:107) x − ¯ x · 1 n (cid:107) 2 , or equivalently, E (cid:104) (cid:107) Π W ( t ) x (cid:107) 2 (cid:105) ≤ β 2 (cid:107) Π x (cid:107) 2 .
For W ( t ) ≡ W , β essentially equals (cid:107) Π W (cid:107) 2 .
Directed EquiTopo Graphs 3.1 Basis weight matrices and basis graphs
Given a graph of size n , we introduce a set of doubly stochastic basis matrices { A ( u,n ) } n − 1 u =1 , where
A ( u,n ) =
[ a ( u,n ) ij ] ∈ R n × n with a ( u,n ) ij =   n − 1 n , if i =
( j + u ) mod n,n , if i = j, 0 , otherwise .
Their associated graphs {G ( A ( u,n ) ) } n − 1 u =1 are called basis graphs .
A basis graph G ( A ( u,n ) ) has degree one and the same label difference ( i − j ) mod n for all edges ( j, i ) .
The set of ﬁve basis graphs {G ( A ( u, 6) ) } 5 u =1 for n = 6 is shown in Fig.
When n is clear from the context, we omit it and write A ( u ) instead.
3.2 Directed static EquiTopo graphs (D-EquiStatic)
Our directed graphs are built on the above basis graphs, and a weight matrix has the form
[Equation]
where u i ∈ [ n − 1] and M > 0 is the number of basis graphs we will sample.
Throughout this paper, the multiset { u i } Mi =1 are called basis index .
It is possible that u
i = u j for some i (cid:54) = j .
Since each
A ( u ) has the form (2), the matrix W is doubly stochastic, and all nodes of the directed graph G ( W ) have the same degree that is no more than M .
Since G ( W ) is a directed static graph and built with M basis graphs, we name it D-EquiStatic.
The following theorem shows that we can construct a weight matrix W such that its consensus rate is independent of the network size n by setting M properly.
The proofs of all theorems are in the
Appendix.
Theorem 1 Let A ( u ) be deﬁned by (2) for any u ∈ [ n − 1] .
For any constant ρ ∈ (0 , 1) , we can choose a sequence of u 1 , · · · , u M from [ n − 1] with M = Θ (cid:0) ln ( n ) /ρ 2 (cid:1) and construct the
D-EquiStatic weight matrix W as in (3) such that the consensus rate of W is ρ , i.e.,
[Equation]
The graph G ( W ) has degree at most M .
In the following, we will just say that the degree is Θ(ln( n )) .
As ρ is tunable, we choose ρ as a constant, e.g., ρ = 0 . 5 .
A method of constructing D-EquiStatic weight matrix W can be found in Appendix A.1.
Remark 1 Compared to all common topologies listed in Table 1, D-EquiStatic achieves a better balance between degree and consensus rate.
Moreover, D-EquiStatic works for any size n ≥ 2 .
Different from the Erdos-Renyi random graph and the geometric random graph, whose degree cannot be predeﬁned before the implementation, we can easily specify the degree M for D-EquiStatic.
3.3 One- peer directed EquiTopo graphs (OD-EquiDyn)
While D-EquiStatic achieves a size-independent consensus rate with Θ(ln( n )) degree, we develop a one-peer dynamic variant to further reduce its degree.
Given a weight matrix W of form (3) and its associated basis matrix { A ( u i ) } Mi =1 , the one-peer directed variant, or OD-EquiDyn for short, samples a random A ( u ) per iteration and utilizes it as the one-peer weight matrix, see Alg. 1.
Since each node in G ( A ( u ) ) has exactly one neighbor, the graph G ( W ( t ) ) has degree one for every iteration t .
Note that W ( t ) is a random time-varying weight matrix.
Its consensus rate (in expectation) can be characterized as below.
Algorithm
Input: constant η ∈ (0 , 1) ; basis index { u 1 , u 2 , . . .
, u M } from a weight matrix W of form (3);
Pick v t from uniform distribution over the basis index { u 1 , u 2 , . .
, u M } ;
Produce basis matrix A ( v t ) according to (2);
Output: W ( t ) =
(1 − η ) I + η A ( v t )
Theorem 2 Let the one-peer directed weight matrix W ( t ) be generated by Alg.
It holds that
[Equation]
where ρ is the consensus rate of the weight matrix W (which can be tuned freely as in Theorem 1).
Remark 2 The OD-EquiDyn graph has a degree of 1 no matter how dense the input matrix W is.
When η = 1 / 2 , which is used in our implementations, it holds that E (cid:107) Π W ( t ) x (cid:107) 2 ≤ (1 + ρ ) / 2 (cid:107) Π x (cid:107) 2 for any x ∈ R n .
Thus, the OD-EquiDyn graph maintains the same Θ(1) degree as ring, grid, and the one-peer exponential graph but with a faster size-independent consensus rate.
Remark 3 For basis index { 1 , · · · , n − 1 } ( W = J ), Alg.
1 returns a sequence of OD-EquiDyn graphs with E (cid:107) Π W ( t ) x (cid:107) 2 ≤ 12 (cid:107) Π x (cid:107) 2 when η = 12 because (cid:107)
Π Jx (cid:107) = 0 implies ρ = 0 .
Remark 4 Although the Erdos-Renyi random graph also enjoys Θ(1) consensus rate and Θ(ln( n )) average degree (see Table 1), its maximum degree could be as large as Θ( n ) which implies that
Erdos-Renyi random graphs could be highly unbalanced.
Moreover, Erdos-Renyi random graphs are undirected graphs, while EquiStatic graphs can be both directed (Section 3) and undirected (Section 4).
In addition, the structure of EquiStatic allows simple construction of one-peer random graphs which preserve Θ(1) consensus, while it is still an open problem on whether Erdos-Renyi random graphs admit one-peer variants with Θ(1) consensus rate.
Undirected EquiTopo Graphs
The implementation of many important algorithms such as EXTRA [28], Exact-Diffusion [40],
NIDS [16], decentralized ADMM [29], and the dual-based optimal algorithms [27, 33, 13] rely on symmetric weight matrices.
Moreover, devices in full-duplex communication systems can communicate with one another in both directions, and undirected networks are natural to be utilized.
These motivate us to study undirected graphs.
4.1 Undirected static EquiTopo graphs (U-EquiStatic)
Given a D-EquiStatic weight matrix W and its associated basis matrices { A ( u i ) } Mi =1 , we directly construct an undirected weight matrix name U-EquiStatic by
[Equation]
(5) whose basis index are { u i , − u i } Mi =1 because A ( − u ) =
[ A ( u ) ] T .
Since (cid:102) W is built upon W and Π W = W Π , the following theorem follows directly from
[Equation]
Theorem 3 Let W be a D-EquiStatic matrix with consensus rate ρ and (cid:102) W be the U-EquiStatic matrix deﬁned by (5) .
It holds that
[Equation]
4.2 One-peer undirected EquiTopo graphs (OU-EquiDyn)
Constructing a one-peer undirected graph OU-EquiDyn is not as direct as U-EquiStatic because 12 ( A ( u ) + [ A ( u ) ] T ) admits a graph with degree 2, see Appendix B.1 for an illustration.
Alg. 2 shows a method to construct a series of OU-EquiDyn matrices with degree 1.
Starting from node s , we connect a node with the u th node after it, as long as both of them have not been connected to any other nodes.
Fig.
2 illustrates the process when s = 1 and u = 1 , · · · , 5 for a network of size 6 .
Some nodes have no neighbors at realizations u
= 2 and u = 4 .
This phenomenon is caused by the restriction that each node has no more than one neighbor.
For instance, when u = 2 , node 5 wants to connect with node 1 but node 1 has already been connected to node 3 .
Thus, there exist node pairs that are never connected when s = 1 .
To resolve this issue, we let the starting index s be sampled randomly from [ n ] .
Fig.
8 in Appendix B.1 illustrates the scenarios when s = 3 .
It is observed that the pairs { 3 , 5 } and { 4 , 6 } are now connected to each other.
The node version of OU-EquiDyn is illustrated in Alg. 4 of Appendix.
B.2.
Theorem 4 Let (cid:102) W be a U-EquiStatic matrix with consensus rate ρ , and (cid:102) W ( t ) be an OU-EquiDyn matrix generated by Alg.
2, it holds that
[Equation]
Remark 5 Theorem 4 implies that the OU-EquiDyn graph can achieve a size-independent consensus rate with a degree at most 1 .
When η = 1 / 2 , it holds that E (cid:107) Π W ( t ) x (cid:107) 2 ≤ [(2 + ρ ) / 3] (cid:107) Π x (cid:107) 2 .
When (cid:102) W = J and the basis index { 1 , · · · , n − 1 } are input to Alg.
2, we obtain an OU-EquiDyn sequence (cid:102) W ( t ) such that E (cid:107) Π
[Equation]
Remark 6 An alternative OU-EquiDyn matrix construction that relies on the Euclidean algorithm is in Appendix B.4.
Applying EquiTopo Matrices to Decentralized Learning
We consider the following distributed problem over a network of n computing nodes:
[Equation]
where f i ( x ) :
=
E ξ i ∼D i
[ F ( x ; ξ i ) (cid:3) .
The function f i ( x ) is kept at node i , and ξ i denotes the local data that follows the local distribution D i .
Data heterogeneity exists if local distributions {D i } ni = are not identical.
Throughout this section, we let x ( t ) i be node i ’s local model at iteration t , and ¯ x ( t ) = 1 n (cid:80) ni
=1 x ( t ) i .
Assumptions.
We make the following standard assumptions to facilitate analysis.
A.1 Each local cost function f i ( x ) is differentiable, and there exists a constant L > 0 such that (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) for all x , y ∈ R d .
A.2 Let g ( t ) i = ∇ F ( x ( t ) i ; ξ ( t ) i ) .
There exists σ 2 > 0 such that for any t and i E ξ ( t ) i
∼D i g ( t ) i = ∇ f i ( x ( t ) i ) , and E ξ ( t ) i ∼D i (cid:104) (cid:13)(cid:13) (cid:13) g ( t ) i − ∇ f i ( x ( t ) i ) (cid:13)(cid:13) (cid:13) 2 (cid:105) ≤ σ 2 .
A.3 (For DSGD only)
There exists b 2 such that 1 n (cid:80) ni =1 (cid:107)∇ f i
( x ) − ∇ f ( x ) (cid:107) ≤ b 2 for all x ∈ R d .
5.1 Decentralized stochastic gradient descent
The decentralized stochastic gradient descent (DSGD) [6, 17, 12] is given by
[Equation]
where the weight matrix W ( t ) =
[ w ( t ) ij ] can be time-varying and random.
Applying the EquiTopo matrices discussed in § 3-4, we achieve the following convergence results, whose proof follows
Theorem 2 in [12] directly, and is omitted here.
More results for DSGD are given in Appendix C.1.
Theorem 5 Consider the DSGD algorithm (8) .
Under Assumptions A.1-A.3, it holds that
[Equation]
• where β
= ρ with D-EquiStatic W or U-EquiStatic (cid:102) W ; • where β = (cid:112) (1 + ρ ) / 2 for OD-EquiDyn W ( t ) (Alg. 1 with η = 1 / 2 ), and β = (cid:112) (2 + ρ ) / 3 for OU-EquiDyn (cid:102) W ( t )
(Alg. 2 with η = 1 / 2 ).
For a sufﬁciently large T , the term O (1 / √ nT ) dominates the rate, and we say the algorithm reaches the linear speedup stage.
The transient iterations are referred to as those iterations before an algorithm reaches the linear-speedup stage.
We compare the per-iteration communication, convergence rate, and transient iterations of DSGD over various topologies in Tables 3 and 4 of the Appendix.
It is observed that OD/OU-EquiDyn endows DSGD with the lightest communication, fastest convergence rate, and smallest transient iteration complexity.
5.2 Decentralized stochastic gradient tracking algorithm
The decentralized stochastic gradient tracking algorithm (DSGT) [23, 7, 25, 36, 35] is given by
[Equation]
The following result of DSGT does not appear in the literature since it admits an improved convergence rate for stochastic decentralized optimization over asymmetric or time-varying weight matrices.
Existing works on DSGT assume weight matrix to be either symmetric [11, 1] or static [35].
Theorem 6 Consider the DSGT algorithm in (9) .
If { W ( t ) } t ≥ 0 have consensus rate β , then under
Assumptions A.1-A.2, it holds for T ≥ 1 1 − β that
[Equation]
When utilizing the EquiTopo matrices, the corresponding β is speciﬁed in Theorem 5.
Note that
DSGT achieves linear speedup for large T .
The per-iteration communication and convergence rate comparison of DSGT over different topologies is in Table 2.
OD/OU-EquiDyn endows DSGT with the lightest communication, fastest convergence rate, and smallest transient iteration complexity.
Numerical Experiments
This section presents experimental results to validate EquiTopo’s network-size-independent consensus rate and its comparison with other commonly-used topologies in DSGD on both strongly-convex problems and non-convex deep learning tasks.
More experiments for EquiTopo in DSGT and all implementation details are referred to Appendix D.
Network-size independent consensus rate.
This simulation examines the consensus rates of all four EquiTopo graphs.
We recursively run the gossip averaging x ( t +1) =
W ( t ) x ( t ) with x (0) ∈ R n initialized arbitrarily and W ( t ) ∈ R n × n generated as D-EquiStatic (Eq. (3)), OD-EquiDyn (Alg. 1),
U-EquiStatic (Eq.
(5)), and OU-EquiDyn (Alg. 2), respectively.
Fig.
3 depicts how the quantity (cid:107) x ( t ) − Jx (0) (cid:107) evolves when n ranges from 1000 to 10 , 000 with D-EquiStatic topology.
See
Appendix D for all other EquiTopo graphs, which also achieve network-size independent consensus rates when n varies.
These results are consistent with Theorems 1 - 4.
Topology
[Table]
Ring
[Table]
Torus
[Table]
Static
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
Comparison with other topologies.
We now compare EquiTopo’s consensus rate with other commonly-used topologies.
Fig.
4 illustrates the performance of several graphs with Θ(1) degree when running gossip averaging.
We set n = 4900 so that the grid graph can be organized as 70 × 70 .
OD/OU-EquiDyn is much faster than other topologies.
Note that each node in OD-EquiDyn, OUEquiDyn, and O.-P. Exp. has exactly one neighbor per iteration.
More experiments on graphs with Θ(ln( n )) degrees and on scenarios with smaller network sizes are in Appendix D.
DSGD with EquiTopo: least-square.
We next apply D/U-EquiStatic graphs to DSGD when solving the distributed least square problems.
In the experiment, we let n = 300 and set M =
so
that
D/U-EquiStatic has the same degree as the exponential graph.
Fig.
5 depicts that D/U-EquiStatic converges much faster than a static exponential graph, especially in the initial stages when the learning rate is large.
The U-EquiStatic performs slightly better than D-EquiStatic since its bi-directional communication enables the graph with better connectivity.
DSGD with EquiTopo: deep learning.
We consider the image classiﬁcation task with ResNet20 model [9] over the CIFAR-10 dataset [14].
We utilize BlueFog [38] to support decentralized communication and topology setting in a cluster of 17 Tesla P100 GPUs.
Fig.
6 illustrates how D/UEquiStatic compares with static exp., ring, and centralized SGD in training loss and test accuracy.
It is observed that D/U-EquiStatic has strong performance.
They achieve competitive training losses to centralized SGD but slightly better test accuracy.
Meanwhile, they also outperform static exponential graphs in test accuracy by a visible margin (D-EquiStatic: 92 % , U-EquiStatic: 91.7 % , Static Exp.: 91.5 % ).
Experiments with EquiDyn topologies and results on MNIST dataset [15] are in Appendix D.
[Figure]
Conclusion
This paper proposes EquiTopo graphs that achieve a state-of-the-art balance between the maximum degree and consensus rate.
The EquiStatic graphs are with Θ(ln( n )) degrees and n -independent consensus rates, while their one-peer variants, EquiDyn, maintain roughly the same consensus rates with a degree at most 1 .
EquiTopo enables decentralized learning with light communication and fast convergence.
To the best of our knowledge, we are not aware of any negative social impacts in our results.
Acknowledgement
We thank the anonymous reviewers for suggesting updated literature on E.-R. graphs.
Ming Yan was partially supported by the NSF award DMS-2012439.
Lei Shi was partially supported by Shanghai
Science and Technology Program under Project No. 21JC1400600 and No. 20JC1412700 and
National Natural Science Foundation of China (NSFC) under Grant No. 12171093.
Kexin Jin was supported by Alibaba Research Internship Program.
A Directed EquiTopo Graphs A.1 Construction of a D-EquiStatic graph
A practical method to construct a D-EquiStatic weight matrix W is provided in Alg.
We should mention that the “while" loop in the algorithm is adopted to guarantee (cid:107) Π W (cid:107) 2 ≤ ρ .
Algorithm
[-Generate basis weight matrices { A ( u i ) } Mi =
-]
A.2 Proof of Theorem
Before showing properties of W deﬁned by (3), we provide two lemmas as follows.
Referring to Theorem 1.6 of [32], we have the following result for a sequence of random matrices.
Lemma 1 (Matrix Bernstein) Consider a sequence of K independent random n × n matrices { M i } Ki =1 .
Assume that each random matrix satisﬁes
E [ M i ] = 0 , and (cid:107) M i (cid:107) 2 ≤ R almost surely .
Deﬁne
[Equation]
It holds that
[Equation]
Lemma
For any matrix B =
[ b ij ] ∈ R n × n , it holds that
[Equation]
Proof.
By deﬁnition,
[Equation]
By . Moverover, for all (cid:107) x (cid:107) ≤ 1 , (cid:107) y (cid:107) ≤ 1 , we have
[Equation]
Thus, Lemma 2 holds.
[Equation]
Theorem 1 (Formal restatement of Theorem 1) Let A ( u ) be deﬁned by (2) for any u ∈ [ n − 1] and the D-EquiStatic weight matrix W be constructed by (3) with { u i } Mi =1 following an independent and identical uniform distribution from [ n − 1] .
For any size-independent consensus rate ρ ∈ (0 , 1) and probability p ∈ (0 , 1) , if M ≥ 83 ρ 2 ln 2 np it holds with probability at least 1 − p that
[Equation]
[Equation]
Proof.
Notice that
E
[ A
[Equation]
u
[Equation]
=
J , [Equation] i ∈ [Equation]
[Equation]
, M } .
Since each A ( u i ) is doubly stochastic, it follows from Lemma 2 that
[Equation]
Consequently,
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Analogously, (cid:80) M i =1 E
[Equation]
A
[Equation]
J )
[Equation]
A
[Equation]
J )
[Equation]
≤ M. By Lemma 1,
P
[Equation]
W [Equation] = [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
i.e.,
[Equation]
Note that Π W = Π W Π .
If (cid:107) Π W (cid:107) 2 ≤ ρ , then
[Equation]
Therefore, the conclusion holds.
[Equation]
The relation M ≥ 83 ρ 2 ln(2 n/p ) is required for theoretical analysis, and it is very conservative.
In practice, we can set M to be far less than 83 ρ 2 ln(2 n/p ) and repeat the process described in the formal version of Theorem 1 until we ﬁnd a desirable W (see Alg. 3).
In addition, the veriﬁcation condition (cid:107) Π W (cid:107) 2 ≤ ρ in Alg. 3 can also be dropped in implementations so that we only conduct the “while” loop once.
We ﬁnd that these relaxations can still achieve W with an empirically fast consensus rate (see the illustration in the experiments).
Remark 7 We have much ﬂexibility in the choice of p , such as p = 1 / 2 or p = 1 /n .
If p ∈ (1 / poly (n) , 1) , the corresponding value of M will only differ in constants.
A.3 Proof of Theorem
Due to Π A ( v t )
=
A ( v t ) Π
= Π A ( v t ) Π , it follows from Alg. 1 that
[Equation]
(cid:105) = E (cid:104) (cid:13)(cid:13)(cid:13) Π (cid:16) (1 − η )
{+I + η A ( v t ) (cid:17) x (cid:13)(cid:13)(cid:13) 2 (cid:105) ≤ (1 − η ) 2 (cid:107) Π x (cid:107) 
 + 2 η (1 − η ) ( Π x ) T E (cid:2)+}
Π A ( v t ) (cid:3) Π x + η 2 E (cid:104) (cid:13) (cid:13)(cid:13)
A ( v t ) (cid:13) (cid:13)(cid:13) 2 2 (cid:105) (cid:107) Π x (cid:107)
Notice that E (cid:2) A ( v t ) (cid:3) =
W and (cid:107) Π W y (cid:107) ≤ ρ (cid:107) Π y (cid:107) , ∀ y ∈ R n .
Therefore,
[Equation]
The proof is completed.
[Equation]
B One-Peer Undirected EquiTopo Graphs (OU-EquiDyn) B.1 Illustration for basis graphs of OU-EquiDyn
The associated graph of (cid:98) A ( u ) = 12 ( A ( u ) + [ A ( u ) ] T ) with n = 6 are given in Fig.
Clearly, there exist non-one-peer graphs.
Consider n = 6 .
The OU-EquiDyn graphs generated by Alg.
2 (when s = 3 ) are presented in
B.2 Node version of Alg.
From the node’s perspective, an equivalent version of Alg. 2 is presented in Alg. 4.
In the remainder of Appendix B, we denote (cid:98) x (cid:99) as the largest integer no greater than x .
We also denote the traditional mod operator as
[Equation]
If v t = n/ 2 , every node i is connected to node ( i + n/ 2) mod n for Alg. 2, which is the same as
Alg. 4.
If v t < n 2 , then q = v t .
In Alg. 2, nodes { s t , s t + 1 , · · · , s t + q − 1 } mod n are connected to { s t + q, s t + q + 1 , · · · , s t + 2 q − 1 } mod n , respectively.
If j ∈ { s t + q, s t + q + 1 , · · · , s t + 2 q − 1 } mod n , then new edges cannot be added because they have been connected.
Similar process starts from connecting node ( s t + 2 q ) mod n with node ( s t + 3 q ) mod n , and as a result, for v t < n 2 , Alg. 2 can be interpreted as follows: divide [ n ] into q disjoint subsets:
[Equation]
Equivalently,
[Equation]
q (cid:23)(cid:27) .
For node i , we deﬁne k ( i ) =
mod( i − s t , n ) , d ( i ) = (cid:98) k
( i ) /q
(cid:99) and r ( i ) =
mod( k ( i ) , q ) .
Then, i ∈ C r
( i ) and i =
( s t + r
( i )
+ d ( i ) · q ) mod n .
In each C (cid:96) , s t + (cid:96) is connected with s t + (cid:96) + q , s t + (cid:96) + 2 q is connected with s t + (cid:96) + 3 q , ....
Thus, if | C (cid:96) | is even, every node in C (cid:96) has a neighbor.
If | C (cid:96) | is odd (equivalently, (cid:106) n − 1 − (cid:96) q (cid:107) is odd), the node (cid:96) + q · (cid:106) n − 1 − (cid:96) q (cid:107) is idle and the others has neighbors.
Deﬁne
[Equation]
(cid:107)(cid:27) , otherwise.
Then, node i has a neighbor if and only if it is in the set C (cid:48) r ( i ) .
In addition, for node i in C (cid:48) r ( i ) , it is connected to ( i + q ) mod n if d ( i ) is even; and connected to ( i − q ) mod n if d ( i ) is odd.
Input: η ∈ (0 , 1) ; basis index { u i , − u i } M i =1 from a weight matrix (cid:102) W ∈ R n × n of form (5);
Initialize A =
[ a ij ] =
I ; for node i = 1 to n (in parallel) do
Pick v t from { u i , − u i } M i =1 and s t ∈ [ n ] uniformly at random using the common random seed; if v t ≤ n/ 2 then q = v t ; k ( i )
=
mod( i − s t , n ) ; endelse q = n − v t ; k ( i ) =
mod( i − s t + q, n ) ; end r ( i )
=
mod( k ( i ) , q ) ; d ( i ) = (cid:98) k
( i ) /q
(cid:99) ; if (cid:98) ( n − 1 − r ( i )) /q
(cid:99) is odd or d ( i ) < (cid:98) ( n − 1 − r ( i )) /q
(cid:99) then if d ( i ) is even then j = ( i + q ) mod n ; endelse j = ( i − q ) mod n ; end a ij = ( n − 1)
/n ; a ii = 1 /n ; end end
Output: (cid:102) W ( t ) =
(1 − η ) I + η A
In Alg. 4, we compute r ( i ) and d ( i ) ﬁrstly, and then check whether node i is in C (cid:48) r ( i ) .
If i ∈ C (cid:48) r ( i ) and d ( i ) is even (odd) , then it is connected to ( i + q ) mod n ( ( i − q ) mod n ).
Otherwise, node i is idle, i.e., a ii = 1 .
This yields the equivalence between Alg.2 and Alg.
4 for the case of v t < n/ 2 .
If v t > n/ 2 , let q = n − v t < n/ 2 .
Then the nodes in { s t , s t + 1 , · · · , s t + q − 1 } mod n are connected to the nodes { s t + v t , s t
+ 1 + v t , · · · , s t + q − 1 + v t } mod n respectively.
Note that ( i + v t ) mod n is equivalent to ( i − q ) mod n .
Then, equivalently, nodes { s t , s t + 1 , · · · , s t + q − 1 } mod n are connected with { s t − q, s t − q + 1 , · · · , s t − 1 } mod n , respectively.
Similar process starts from connecting node ( s t + 2 q ) mod n with node ( s t + q ) mod n .
Then, the undirected graph generated with the starting point s t and the label difference v t > n/ 2 is equivalent to the undirected graph generated with the starting point ( s t − q ) mod n and the label difference q .
So by setting k ( i ) =
mod( i − ( s t − q ) , n ) , the proof follows by similar arguments for the case v t < n/ 2 .
B.3 Proof of Theorem
We ﬁrst provide the following three lemmas.
In the remainder of Appendix B, for any matrix
A =
[ a ij ] ∈ R n × n , we denote its edge set as
[Equation]
Denote the matrix A generated at the t -th iteration of Alg. 2 by (cid:101)
A ( v t ) =
[ (cid:101) a ( v t ) ij
] ∈ R n × n .
Note that (cid:101)
A ( v t ) is also stochastic even when v t is given since s t is randomly chosen from [ n ] .
Lemma
For any n ≥ 2 , it holds for (cid:101)
A ( v t ) deﬁned by Alg. 2 that it holds that
[Equation]
Proof.
Because |E ( (cid:101) A ( v t ) ) | is invariant with respect to s t , it sufﬁces to prove |E ( (cid:101)
A ( v t ) )
| ≥ 2 n/ 3 for s t = 1 .
For v t ≤ n/ 2 , we deﬁne m = (cid:98) n/ (2 v t ) (cid:99) and r = mod( n, 2 v t ) , then, m ≥ 1 .
Notice that node
i is connected with i + v t for any i ∈ { (cid:96) + 2 dv t : 1 ≤ (cid:96) ≤ v t , 0 ≤ d < m } .
If r ≤ v t , then the last r nodes are idle.
As m ≥ 1 , we have n = 2 v t m + r ≥ 2 v t + r ≥ 3 r.
Thus, | [Equation]
[Equation]
n = [Equation] n.
If r > v t , then node i in { 2 mv t + (cid:96) : 1 ≤ (cid:96) ≤ r − v t } is connected to i + v t .
As a result, only the nodes in { 2 mv t + (cid:96) : r − v t + 1 ≤ (cid:96) ≤ v t } are idle, i.e., 2 v t − r nodes are idle.
We have n = 2 mv t + r > 3 v t from m ≥ 1 and r > v t .
Consequently,
[Equation]
We have shown |E ( (cid:101) A ( v t ) )
| ≥ 23 n for v t ≤ n/ 2 .
For v t > n/ 2 , recall that we have shown in Appendix B.2 that the undirected graph generated with label difference v t and starting point s t equals the undirected graph generated with label difference q = n − v t < n/ 2 and starting point ( s t − q ) mod n .
Since the number of edges is invariant with the starting point, the case v t > n/ 2 has been reduced to v t < n/ 2 .
This completes the proof.
[Equation]
Lemma
For any symmetric matrix B =
[ b ij ] ∈ R n × n , if B 1 n = 0 n , then
[Equation]
Proof.
The i -th entry of Bx is
[Equation]
Hence,
[Equation]
Due to B T =
B , we have
[Equation]
Averaging the above equations yields the result.
[Equation]
Lemma
For any n ≥ 2 , it holds for (cid:101)
A ( v t )
that
[Equation]
Proof.
If v t = n/ 2 , then (cid:101)
A ( v t ) = A ( v t ) =
[ A ( v t ) ] T .
By Lemma 4, x T (cid:16) I − A ( v t ) (cid:17) x ≥ 0 , i.e.,
A ( v t ) (cid:22) I .
Thus, [Equation] (cid:16) A
[Equation] [Equation]
] T
[Equation]
Consider v t (cid:54) = n/ 2 .
Notice that E ( (cid:101)
A ( v t ) )
⊂ E (cid:0) A ( v t )
+ [ A ( v t ) ] T (cid:1) and |E ( A ( v t )
+ [ A ( v t ) ] T ) | ≤ 2 n .
By Lemma 3 and the fact that s t is from the uniform distribution over [ n ] , it holds for any ( i, j ) ∈
E ( A ( v t ) +
[ A ( v t ) ] T )
that
[Equation]
By the construction of (cid:101) A ( v t ) in Alg.
2 and A ( v t ) in (2), the non-diagonal and non-zero entries are n −n .
It follows from Lemma 4 that for any x ∈ R n , we have
[Equation]
Rearranging the terms, we derive
[Equation]
This completes the proof.
[Equation]
Proof of Theorem 4 It follows from Lemma 5 that
[Equation]
Consequently,
[Equation]
Combining the above two inequalities
[Equation]
we derive
[Equation]
Thus, the proof is completed.
[Equation]
B.4 An alternative construction of OU-EquiDyn
Alg. 5 provides a different way to construct one-peer undirected graphs which achieve a similar consensus rate as the graphs generated by Alg.
but with a different structure.
We denote the matrix A generated at the t -th iteration of Alg.
5 by A ( v t ) =
[ a ( v t ) ij ] ∈ R n × n .
Next, we explain the motivation of Alg. 5.
Denote gcd( a, b ) as the greastest common divisor of a and b .
Let d = gcd( v t , n ) and ˜ v t = v t /d , ˜ n = n/d .
Then, ˜ v t and ˜ n are coprime.
Firstly, we divide [ n ] into d disjoint subsets:
[Equation]
Output W ( t ) =
(1 − η ) I + η A
Clearly, (cid:12)(cid:12) C (cid:96) (cid:12)(cid:12) =
n/d = ˜ n , ∀ 0 ≤ (cid:96) < d .
We claim that
[Equation]
To proof the claim, we denote the RHS of (11) by (cid:101) C (cid:96) .
Since v t can be divided evenly by d , for any i ∈ (cid:101) C (cid:96) , it satisﬁes mod ( i, d ) =
mod(( s t + (cid:96) ) , d ) .
Combining with the fact that (cid:101) C (cid:96) ⊂ [ n ] , we have (cid:101) C (cid:96) ⊂ C (cid:96) .
Then, since ˜ n and ˜ v are coprime, for any 0 ≤ m 1 < m 2 < ˜ n , mod( m 1 ˜ v t , ˜ n ) (cid:54) = mod( m 2 ˜ v t , ˜ n ) .
Then, mod( m 1 v t , n ) (cid:54) = mod( m 2 v t , n ) .
Thus, (cid:12)(cid:12)(cid:12) (cid:101) C (cid:96) (cid:12)(cid:12)(cid:12) =
˜ n = (cid:12)(cid:12) C (cid:96) (cid:12)(cid:12) .
Combining with (cid:101) C (cid:96) ⊂ C (cid:96) , we have C (cid:96) = (cid:101) C (cid:96) .
The above analysis also implies that for each i ∈ C (cid:96) , there exists a unique 0 ≤ m ( i ) <
d such that i =
( s t + (cid:96) + m ( i ) v t ) mod n .
By (11), a natural way to construct one-peer undirected graphs is: in each C (cid:96) , connect s t + (cid:96) with s t + (cid:96) + v t , connect s t + (cid:96) +2 v t with s t + (cid:96) +3 v t , ....
Equivalently, i is connected with ( i + v t ) mod n if m ( i ) is even and connected with ( i − v t ) mod n if m ( i ) is odd.
In this way, if ˜ n is even, every node in C (cid:96) has a neighbor, ∀ 0 ≤ (cid:96) < d , i.e., every node in [ n ] has a neighbor.
Equivalently, a node i ∈ C (cid:96) has a neighbor if it is in the set C (cid:48) [Equation] = (cid:26) C (cid:96) , if ˜ n is even { ( s t + (cid:96) + mv
[Equation]
mod
[Equation]
[Equation]
− [Equation] , if ˜ n is odd (12) To give a practical way of the process described above from each node’s perspective, for each node i ∈ C (cid:96) , we provide a more efﬁcient way to compute the unique 0 ≤ m ( i ) <
˜ n such that i =
( s t + (cid:96) + m ( i ) v t ) mod n .
Firstly, for each i ∈ C (cid:96) , since (cid:96) = mod( i − s t , d ) , we have i =
( s t + (cid:96) + (cid:98) ( i − s t ) /d (cid:99) · d ) mod n.
Then, by B ´ e zout’s theorem, since ˜ n and ˜ v t are coprime, there exist integers 1 ≤ b ≤ ˜ n − 1 and b (cid:48) such that b · v t + b (cid:48) · n = d .
The pair ( b, b (cid:48) ) can be computed by the Euclidean algorithm in O (ln( n )) time.
Then,
[Equation]
Deﬁne m ( i ) =
mod( (cid:98) ( i − s t ) /d
(cid:99) · b, ˜ n ) , then, we have ( (cid:98) ( i − s t ) /d (cid:99) · d ) mod n = m ( i ) v t mod n,
Thus, i =
( s t + (cid:96) + m ( i ) v t ) mod n.
In Alg. 5, each node i computes ˜ n and its m ( i ) ﬁrstly.
If ˜ n is even, every node has a neighbor.
If ˜ n is odd but m ( i ) <
˜ n − 1 , then i ∈ C (cid:48) (cid:96) where (cid:96) = mod ( i − s t , d ) , i.e., i also has a neighbor.
In this way, each node can determine whether it has a neighbor in this iteration.
If node i has a neighbor, as we have described above, it is connected with ( i + v t ) mod n if m ( i ) is even and ( i − v t ) mod n otherwise.
The following lemma is used to prove Lemma 7.
Lemma 6 Let d = gcd( v t , n ) , then
[Equation]
Proof.
Deﬁne C (cid:96) and C (cid:48) (cid:96) as in (11) and (12).
If n can be divided by 2 d evenly, i.e., (cid:12)(cid:12) C (cid:96) (cid:12)(cid:12) =
˜ n is even for any 0 ≤ (cid:96) < d .
Then, every node has a neighbor, i.e., (cid:12)(cid:12)(cid:12) E
( A ( v t ) )
(cid:12)(cid:12)(cid:12)
=
n = (2 d ) · (cid:4) n 2 d (cid:5) .
If n cannot be divided by 2 d evenly, by (12), in each C (cid:96) , there is one node idle in this iteration.
Then, we also have (cid:12)(cid:12)(cid:12) E
( A ( v t ) )
(cid:12)(cid:12)(cid:12)
=
n − d = (2 d ) · (cid:4) n 2 d (cid:5) .
As d = gcd( v t , n ) , we have n = (2 k + 1) d , with k ∈ Z .
Since v t can be divided by d evenly and v t ≤ n − 1 , we have d < n .
Thus, k ≥ 1 , i.e., n ≥ 3 d .
Then, (cid:12)(cid:12)(cid:12) E ( A ( v t ) )
(cid:12)(cid:12)(cid:12)
= n − d ≥ n − n 3 = 2 n 3 .
Since the above analysis holds for arbitrary s t ∈ [ n ] , the lemma is proved.
(cid:3)
The following lemma follows by similar arguments with Lemma 5.
Lemma
For any n ≥ 2 , the output matrix A ( v t ) of Algorithm 2 satisﬁes
[Equation]
The following consensus rate for Alg. 5 is proved similarly to Theorem 4.
Theorem 7 Let (cid:102) W be a U-EquiStatic matrix with consensus rate ρ , and (cid:102) W ( t ) be an OU-EquiDyn matrix generated by Alg.
5, it holds that
[Equation]
C Applying EquiTopo Matrices to Decentralized Learning C.1 Convergence of DSGD for strongly convex cost functions
We assume that f i ( x ) is µ -strongly convex for any i , i.e., there exists a constant µ > 0 such that
[Equation]
As we have tested the performance of DSGD with EquiTopo matrices for strongly convex cost functions, we attach the following convergence result of the algorithm (8).
The proof follows by [12,
Theorem 2 ] (or Appendix A.4 therein) and is omitted here.
Theorem 8 Consider the DSGD algorithm
(8) utilizing the EquiTopo matrices, and f i being µstrongly convex for all i .
Under Assumptions A.1-A.3, it holds that
[Equation]
where κ = L/µ , ˜ O ( · ) hides constants and polylogarithmic factors, positive weights h ( t )
= (cid:0) 1 − µγ 2 (cid:1) t , and H T = (cid:80)
Tt =
h ( t ) .
Furthermore,
• β
= ρ with D-EquiStatic W or U-EquiStatic (cid:102) W ; • β
=
(cid:112) (1 + ρ ) / 2 for OD-EquiDyn W ( t ) (Alg. 1 with η = 1 / 2 ), and β = (cid:112) (2 + ρ ) / 3 for OU-EquiDyn (cid:102) W ( t )
(Alg. 2 with η = 1 / 2 ).
C.2 Transient iteration
The computation of transient iteration.
For nonconvex cost functions, the convergence rate of (8) is given by
[Equation]
To reach the linear speedup stage, the iteration T has to be sufﬁciently large so that the √ nT -term dominates, i.e., σ √ nT ≥ β 23 σ
T 23 (1 − β ) 13 , σ √ nT ≥ β 23 b
T 23 (1 − β ) 23 , and moreover, σ √ nT ≥ β T (1 − β ) .
Then
T ≥ β 4 n 3 (1 − β ) 2 σ 2 , T ≥ β 4 b 4 n 3 (1 − β ) 4 σ 6 , and T ≥ β 2 n (1 − β ) 2 σ 2 .
Substituting β into the inequalities, transient iterations under different networks can be computed.
Similar methods can be adopted for the transient iterations of the distributed gradient tracking algorithm.
Under different network topologies, for non-convex and strongly convex cost functions, convergence results and transient iterations are shown in Table 3 and 4.
The results indicate that the proposed networks are at faster rates.
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
Static Exp.
[Equation]
n ))
[Equation]
O.-P. Exp.
[Equation]
D
[Equation]
EquiStatic
[Equation]
C.3 Decentralized stochastic gradient tracking algorithm
We write local variables compactly into matrix form, for instance
[Equation]
[Table]
The matrices Y ( t ) , G ( t ) ∈
R n × d are deﬁned analogously.
We also denote ∇ F (cid:0) X ( − 1) (cid:1)
=
G ( − 1) = 0 for notational simplicity.
Clearly
[Table]
the DSGT algorithm can be simpliﬁed as (cid:18) X ( t +1) [Equation]
[Equation]
For simplicity, we deﬁne
[Equation]
and moreover, W ( j : k ) =
I for j > k .
Notice that
[Equation]
Consequently, it holds for all t ≥ 1 that
[Equation]
Moreover, we have two inequalities as follows.
Lemma
For any t ≥ 1 and β ∈ (0 , 1) , we have
[Equation]
where c 1 = 4 , c 2 = 16 .
Proof.
Deﬁne r ( x ) = x 2 β x − 1 , where x ≥ 1 .
Then, for the ﬁrst inequality, it sufﬁces to show that r ( x ) ≤ c 1 / (1 − β ) 2 for x ≥ 1 .
Due to r (cid:48) ( x ) =
xβ x − 1 (2 + x ln β ) , r ( x ) attains its maximum at x 0 = max (cid:110) 1 , − 2 ln β (cid:111) .
If − 2 ln β > 1 , by combining with the fact that ln β ≤ β − 1 < 0 for β ∈ (0 , 1) , we have
[Equation]
If − 2 ln β ≤ 1 , then
[Equation]
The second inequality follows by similar arguments and the fact that √ β ≤ 1+ β 2 .
The proof is completed.
[Equation]
The following lemma is a generalization of Cauchy-Schwartz inequality.
Its proof follows by using (cid:107) A + B (cid:107) 2F ≤ 1 α (cid:107) A (cid:107) 2F + 1 1 − α (cid:107) B (cid:107) 2F ( α ∈ (0 , 1)) repeatedly.
Lemma 9 Consider a sequence of matrices { B i } mi =1 .
If α 1 , α 2 , · · · , α m > 0 and (cid:80) mi
=
α i ≤ 1 , then (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
We deﬁne a potential function as
[Equation]
≥ 1 , [Equation] [Equation] and moreover,
[Equation]
The following Lemma 10 and Lemma 11 are used to prove Lemma 12.
Theorem 6 follows by combining Lemma 12 with the descent lemma (Lemma 14).
Lemma 10 Consider the DSGT (9) .
Let Assumptions A.1 and A.2 hold.
If (cid:110) W ( t ) (cid:111) t ≥ 0 have convergence rate β , i.e., E (cid:20)(cid:13)(cid:13)(cid:13)
Π W ( t ) y (cid:13)(cid:13)(cid:13) 2 (cid:21) ≤ β 2 (cid:107) Π y (cid:107) 2 for any y ∈ R n , then [Equation] Π [Equation]
t [Equation] [Equation] − β
[Equation]
Φ
[Equation]
t )
[Equation]
∀ t ≥ 0 .
Proof.
The case t = 0 follows by deﬁnition directly.
For t ≥ 1 , we deﬁne
[Equation]
j =
[Equation]
, and
[Equation]
F (cid:0) X ( j ) (cid:1) − G ( j − 1) + ∇ F (cid:0) X
( j − 1) (cid:1) (cid:17) .
Recalling (14) gives
X ( t ) =
Q ( t, 1) + Q ( t, 2) .
Then
E (cid:20)(cid:13) (cid:13)(cid:13)
Π X ( t ) (cid:13) (cid:13)(cid:13)
F (cid:21) ≤ 2 E (cid:20)(cid:13) (cid:13)(cid:13)
Π Q ( t, 1) (cid:13) (cid:13)(cid:13)
F (cid:21) + 2 E (cid:20)(cid:13) (cid:13)(cid:13)
Π Q ( t, 2) (cid:13) (cid:13)(cid:13)
F (cid:21) .
Rearranging Q ( t, 2) yields
[Equation]
By Assumption A.2 and the assumption on consensus rate
[Equation]
we have
[Equation]
(cid:13) (cid:13)(cid:13) F (cid:21) + γt −
[Equation]
(cid:88)
j = 4 ( t − j ) 2 β 2( t − j − 1) E (cid:20) (cid:13) (cid:13)(cid:13) G ( j ) − ∇ F (cid:0) X ( j ) (cid:1) (cid:13) (cid:13)(cid:13)
F (cid:21) ≤ 4 nγt (cid:88)
j = j 2 β 2( j − 1) σ 2 .
By Lemma 8, we have
[Equation]
As a result,
E (cid:20)(cid:13)(cid:13)(cid:13)
Π Q ( t, 2) (cid:13)(cid:13)(cid:13)
F (cid:21) ≤ 4 nc 1 γ(1 − β ) 3 σ 2 .
Moreover,
E (cid:20)(cid:13)(cid:13)(cid:13)
Π Q ( t, 1) (cid:13)(cid:13)(cid:13)
F (cid:21) ≤ (1 − β )
β t E (cid:20)(cid:13) (cid:13)(cid:13)
Π W (0:( t − 1))
X (0) (cid:13) (cid:13)(cid:13)
F (cid:21) + t − 1 (cid:88)
j = γ 2 ( t − j )(1 − β ) β t − j − [Equation] E (cid:20) (cid:13)(cid:13) (cid:13) Π W
( j :( t − 1)) (cid:16) [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] where the ﬁrst inequality follows by Lemma 9 and the fact that (1 − β ) (cid:80) tj =
β j < 1 , the second inequality is by the assumption on consensus rate
[Table]
and the third inequality is by Lemma 8.
Therefore, the conclusion holds by the deﬁnition of Φ ( t ) .
[Equation]
Lemma 11 Consider the DSGT (9) .
Let Assumptions A.1 and A.2 hold.
If γ ≤ 1 L , it holds for t ≥that
[Equation]
Proof.
Clearly,
[Equation]
It follows from Assumption A.1 that
[Equation]
Notice that by induction, (cid:80) ni =1 y ( t ) = (cid:80) ni
=1 g ( t ) .
Recalling (9) gives
[Equation]
i =
[Equation]
n
[Equation]
i
[Equation]
= γ n n (cid:88)
[Equation]
By Assumptions
A
[Equation]
1 and A.
[Equation]
we derive
[Equation]
2 (cid:21) + 2 γ 2 Ln
E (cid:20)(cid:13)(cid:13)(cid:13)
Π X ( t ) (cid:13)(cid:13)(cid:13)
F (cid:21) + γ 2 σn .
Due to γ ≤ 1 L , we have
E (cid:20) (cid:13)(cid:13) (cid:13)
¯ x ( t +1) − ¯ x ( t ) (cid:13)(cid:13) (cid:13) 2 (cid:21) ≤ 2 γ 2 E (cid:20) (cid:13)(cid:13) (cid:13) ∇ f (cid:0) ¯ x ( t ) (cid:1) (cid:13)(cid:13) (cid:13) 2 (cid:21) + 2 n E (cid:20) (cid:13)(cid:13) (cid:13) Π X ( t ) (cid:13)(cid:13)
(cid:13)
F (cid:21) + γ 2 σn .
Then the conclusion follows (17), (20) and (cid:13)(cid:13) (cid:13) ¯ X ( t +1)
− ¯ X ( t ) (cid:13)(cid:13) (cid:13) 2 F = n (cid:13)(cid:13)
¯ x ( t +1) − ¯ x ( t ) (cid:13)(cid:13) 2 .
(cid:3)
Lemma 12 Consider the DSGT (9) .
Let Assumptions A.1 and A.2 hold.
Suppose that (cid:110) W ( t ) (cid:111) t ≥have consensus rate β and y ∈ R n .
If 48 c 2 γ 2 L 2 (1 − β ) 4 ≤ 12 , then
T (cid:88) t =
E (cid:20)(cid:13) (cid:13)(cid:13)
Π X ( t ) (cid:13) (cid:13)(cid:13)
[Equation]
Proof.
By the deﬁnition of Φ ( t ) in (15), we have that for t ≥ 0 ,
[Equation]
Then, for t ≥ 0 , by Lemma 10 and (21)
[Equation]
we have
[Equation]
For t ≥ 1 , by Lemma 11, we derive
[Equation]
It follows by the deﬁnition of Φ ( t )
that
[Equation]
By Lemma [Equation] we obtain
[Equation]
Taking sum on both sides of (22) and noting that 48 c 2 γ 2 L
[Equation]
[Equation]
≤
[Equation]
Φ [Equation] +1) ≥ 0 , the lemma is proved.
(cid:3)
Lemma 13 is standard in the analysis of gradient tracking methods.
We attach its proof for completeness.
Lemma 13 Consider the DSGT (9) .
Let Assumptions A.1 and A.2 hold.
If γ ≤ 14 L , then
[Equation]
Proof.
By Assumption A.
[Equation]
we have
[Equation]
It follows from (9) that
[Equation]
[Table]
where the ﬁrst equality is by Assumption A.2 and (18); the second inequality is by Assumption A.
[Table]
Recalling (19) yields
[Equation]
The lemma is proved.
[Equation]
Referring to Lemma 26 of [11]
[Equation]
we have the following result.
Lemma 14 Let A, B, C, T and α be positive constants.
Deﬁne
[Equation]
Then
[Equation]
Proof of Theorem 6 Deﬁne f ∗ = inf x f ( [Equation] [Equation] = f (cid:0) ¯ x (0) (cid:1) − f
[Equation]
, C 0 = (cid:13)(cid:13)(cid:13) Π X (0) (cid:13)(cid:13)(cid:13) 2 F and D 0 = (cid:80) ni
=1 (cid:13)(cid:13)(cid:13) ∇ f i (cid:16) x (0)
i (cid:17)(cid:13)(cid:13)(cid:13) 2 .
We show that
[Equation]
Proof.
Let γ ≤ (1 − β ) 2 50 L to satisfy the conditions in Lemmas 12 and 13.
Then, we have
[Equation]
If γ ≤ 1 − β 10 L , then 192 c 2 γ 4 L 4 (1 − β )≤ 12 .
Then, we have
T +
T (cid:88) t = E
[Equation]
t
[Equation]
By (16) and c 1 = 4 , c 2 = 16 deﬁned in Lemma 8, if γ ≤ (1 − β ) 2 50 L and T ≥ 1 1 − β , we have
[Equation]
+
[Equation]
[Equation]
[Equation]
nT
[Equation]
2 F (cid:33) .
To meet the conditions of Lemma 12, Lemma 13, (24), (25), it sufﬁces to let γ ≤ (1 − β )
L .
Then, the result follows by setting g ( γ ) to be the RHS of (25), A = 8 (cid:0) f (cid:0) ¯ x (0) (cid:1) − f ∗ (cid:1) , B =
Ln σ 2 ,
C =
L 2 (1 − β ) 3 σ 2 and α =
(1 − β ) 2 50 L in Lemma 14.
(cid:3)
D Numerical Experiments D.1 Network-size independent consensus rate
In this experiment, we set M = 5 ln( n ) for D-EquiStatic and M = 2 ln( n ) for U-EquiStatic, which is consistent with Theorems 1 and 3.
For OD-EquiDyn and OU-EquiDyn, we set M = 5 ln( n ) and η = 0 .
Fig.
9 shows that the consensus rate is independent of the network size for all EquiTopo graphs.
The results are obtained by averaging over 3 independent random experiments.
[Figure]
D.2 Comparison with other topologies
We compare the consensus rate between topologies with one-peer or Θ(ln( n )) neighbors on networksize n = 300 and n = 4900 .
In the one-peer case, each topology has exactly one neighbor.
For
OD-EquiDyn and OU-EquiDyn, we set M = n − 1 and η = 0 .
In the Θ(ln( n )) neighbors case, we set M = 9 and M = 13 for n = 300 and n = 4900 , respectively, so that the number of neighbors is identical to the static exponential graph for a fair comparison.
The results are obtained by averaging over 10 and 3 independent random experiments for n = 300 and n = 4900 , respectively.
D.3 DSGD with EquiTopo
Least-square The distributed least square problems are deﬁned with f i ( x ) = (cid:107)
A i x − b i (cid:107) 2 , in which x ∈ R d and A i ∈ R K × d .
In the simulation, we let d = 10 and K = 50 .
At node i , we generate each element in A i following standard normal distribution.
Measurement b i is generated by b i =
A i x (cid:63) + s i with a given arbitrary x (cid:63) ∈ R d where s i ∼ N (0 , σ 2 s I ) is some white noise.
At
[Figure]
each iteration t , each node will generate a stochastic gradient via (cid:99) ∇ f i ( x )
= ∇ f i ( x ) + n i where n i ∼ N (0 , σ 2 n I ) is a white gradient noise.
By adjusting constant σ n , we can control the noise variance.
In this experiment, we set σ s = 0 .
1 and σ n = 1 .
The network size n is
[Figure]
and we set
M = 9 so that D/U-EquiStatic has the same degree as the static exponential graph.
After ﬁxing
M , we sample the basis until the second-largest eigenvalue of the gossip matrix is small enough.
The initial learning rate is 0 .
037 and decays by 1 .
every 40 iterations.
The results are obtained by averaging over 10 independent random experiments.
Deep learning
MNIST.
We utilize EquiTopo graphs in DSGD to solve the image classiﬁcation task with CNN over
MNIST dataset [15].
Like the CIFAR-10 experiment, we utilize BlueFog [38] to support decentralized communication and topology settings in a cluster of 17 Tesla P100 GPUs.
The network architecture is deﬁned by a two-layer convolutional neural network with kernel size 5 followed by two feed-forward layers.
Each convolutional layer contains a max pooling layer and a Rectiﬁed Linear Unit (ReLu).
We generate D/U-EquiStaic with M = 4 and sample OD/OU-EquiDyn with M = 16 and η = 0 .
The local batch size is 64 , momentum is 0 .
5 , the learning rate is 0 . 01 , and we train for 20 epochs.
Centralized SGD and Ring are included for comparison.
See Fig.
11 for the training loss and test accuracy of D/U-EquiStaic and OD/OU-EquiDyn graphs.
See Table 5 for the test accuracy calculated by averaging over last 3 epochs.
EquiTopo graphs achieve competitive train loss and test accuracy to centralized SGD.
CIFAR-10.
We use the ResNet-20 model implemented by [10].
In this experiment, we train for 130 epochs with local batch size 8 , momentum 0 .
9 , weight decay 10 − 4 , and the initial learning rate 0 .
01 , which is divided by 10 at 50th, 100th, and 120th epochs.
We follow the data augmentation from [14], a 4 × 4 padding followed by a random horizontal ﬂip and a 32 × 32 random crop.
We generate D/U-EquiStaic with M = 5 and sample OD/OU-EquiDyn with M = 16 and η = 0 .
See
Fig.
12 for the training loss and test accuracy of OD/OU-EquiDyn.
See Table 5 for the test accuracy calculated by averaging over last 5 epochs.
D.4 DSGT with EquiTopo
In addition to the DSGD experiments, we apply the OD/OU-EquiDyn graphs to the DSGT algorithm when solving logistic regression with non-convex regularizations, i.e., f i ( x ) =
1 L (cid:80) L(cid:96) =1 ln(1 + exp( − y i,(cid:96)
h Ti,(cid:96) x ))
+ R (cid:80) dj
=1 x 2[ j ] / (1 + x 2[ j ] ) where x [ j ] is the j -th element of x , and { h i,(cid:96) , y i,(cid:96) } L(cid:96) =
[Table]
is the data kept by node i .
Data heterogeneity exists when local data ξ i follows different distributions
D i in problem (7).
To control data heterogeneity across the nodes, we ﬁrst let each node i be associated with a local solution x (cid:63)i , and such x (cid:63)i is generated by x (cid:63)i = x (cid:63) + v i where x (cid:63) ∼ N (0 , I d ) is a randomly generated vector while v i ∼ N (0 , σ 2 h I d ) controls the similarity between each local solution.
Generally speaking, a large σ 2 h results in local solutions { x (cid:63) i } that are vastly different from each other.
With x (cid:63)i at hand, we can generate local data that follows distinct distributions.
At node i , we generate each feature vector h i,(cid:96) ∼ N (0 , I d ) .
To produce the corresponding label y i,(cid:96) , we generate a random variable z i,(cid:96) ∼ U (0 , 1) .
If z i,(cid:96) ≤ 1 + exp( − y i,(cid:96) h Ti,(cid:96) x (cid:63)i ) , we set y i,(cid:96) = 1 ; otherwise y i,(cid:96) = − 1 .
Clearly, solution x (cid:63)i controls the distribution of the labels.
This way, we can easily control data heterogeneity by adjusting σ 2 h .
Furthermore, to easily control the inﬂuence of gradient noise, we will achieve the stochastic gradient by imposing a Gaussian noise to the real gradient, i.e., (cid:99) ∇ f i ( x ) = ∇ f i ( x ) + s i in which s i ∼ N (0 , σ 2 n I d ) .
We can control the magnitude of the gradient noise by adjusting σ 2 n .
We let d = 10 , L = 1000 , n = 300 , R = 0 .
001 , and σ h = 0 .
2 in the simulation.
For OD/OUEquiDyn, we set M = n − 1 and η = 0 .
The learning rate for OD/OU-EquiDyn and O.-P. Exp.
isand 1 . 5 , respectively so that all of them converge to the same level of accuracy.
The left plot in Fig.depicts the performance of different one-peer graphs in DSGT.
The right plot in Fig.
13 illustrates how O.-P. Exp.
behaves if it has the same learning rate 3 as OU/OD-EquiDyn.
The gradient norm is used as a metric to gauge the convergence performance.
The results are calculated by averaging over 10 independent random experiments.
It is observed that OD/OU-EquiDyn converges faster than a one-peer exponential graph.
[Figure]