D ECENTRALIZED SGD WITH A SYNCHRONOUS , L OCAL , AND Q UANTIZED U PDATES
A BSTRACT
The ability to scale distributed optimization to large node counts has been one of the main enablers of recent progress in machine learning.
To this end, several techniques have been explored, such as asynchronous, decentralized, or quantized communication–which signiﬁcantly reduce the impact of synchronization, and the ability for nodes to perform several local model updates before communicating– which reduces the frequency of synchronization.
In this paper, we show that these techniques, which have so far been considered independently, can be jointly leveraged to minimize distribution cost for training neural network models via stochastic gradient descent (SGD).
We consider a setting with minimal coordination: we have a large number of nodes on a communication graph, each with a local subset of data, performing independent SGD updates onto their local models.
After some number of local updates, each node chooses an interaction partner uniformly at random from its neighbors, and averages a possibly quantized version of its local model with the neighbor’s model.
Our ﬁrst contribution is in proving that, even under such a relaxed setting, SGD can still be guaranteed to converge under standard assumptions.
The proof is based on a new connection with parallel load-balancing processes, and improves existing techniques by jointly handling decentralization, asynchrony, quantization, and local updates, and by bounding their impact.
On the practical side, we implement variants of our algorithm and deploy them onto distributed environments, and show that they can successfully converge and scale for large-scale image classiﬁcation and translation tasks, matching or even slightly improving the accuracy of previous methods.
I NTRODUCTION
Several techniques have been recently explored for scaling the distributed training of machine learning models, such as communication-reduction, asynchronous updates, or decentralized execution.
For background, consider the classical data-parallel distribution strategy for SGD (Bottou, 2010), with the goal of solving a standard empirical risk minimization problem.
Speciﬁcally, we have a set of samples S , and wish to minimize the d -dimensional function f : R d → R , which is the average of losses over samples from S , by ﬁnding x (cid:63) =
argmin x (cid:80) s ∈ S f s ( x ) / | S | .
We have n compute nodes which can process samples in parallel.
In data-parallel SGD, each node computes the gradient for one sample, followed by a gradient exchange.
Globally, this leads to the iteration:
[Equation]
where η t is the learning rate, x t is the value of the global parameter, initially 0 d , and ˜ g
it ( x t ) is the stochastic gradient with respect to the parameter obtained by node i at time t .
When executing at large scale, two major bottlenecks are communication , that is, the number of bits transmitted by each node, and synchronization , i.e., the fact that nodes need to wait for each other in order to progress across iterations.
Speciﬁcally, to maintain a consistent view of the parameter x t above, the nodes need to broadcast and receive all gradients, and need to synchronize globally at the end of every iteration.
Signiﬁcant work has been dedicated to removing these two barriers.
In particular, there has been progress on communication-reduced variants of SGD, which propose various gradient compression schemes (Seide et al., 2014; Strom, 2015;
Alistarh et al., 2017; Wen et al., 2017; Aji and Heaﬁeld, 2017; Dryden et al., 2016; Grubic et al., 2018; Davies et al., 2020), asynchronous variants, which relax the strict iteration-by-iteration synchronization (Recht et al., 2011; Sa et al., 2015; Duchi et al., 2015)), as well as large-batch or periodic model averaging methods, which aim to reduce the frequency of communication (Goyal et al., 2017; You et al., 2017) and (Chen and Huo, 2016; Stich, 2018)), or even decentralized variants, which allow each node to maintain its own, possibly inconsistent, model variant (Lian et al., 2017; Tang et al., 2018;
Koloskova et al., 2019).
(We refer the reader to the recent surveys of (Ben-Nun and Hoeﬂer, 2019;
Liu and Zhang, 2020) for a detailed discussion.)
Using such techniques, it is possible to scale SGD, even for complex objectives such as the training of deep neural networks.
However, for modern large-scale models, the communication and synchronization requirements of these parallel variants of SGD can still be extremely burdensome.
Contribution.
In this paper, we take a further step towards removing these scalability barriers, showing that all the previous scaling techniques—decentralization, quantization, asynchrony, and local steps—can in fact be used in conjunction .
We consider a highly decoupled setting with n compute agents, located at vertices of a connected communication graph, each of which can execute sequential SGD on its own local model, based on a fraction of the data.
Periodically, after some number of local optimization steps, a node can initiate a pairwise interaction with a uniform random neighbor.
Our main ﬁnding is that this procedure can converge even though the nodes can take several local steps between interactions, may perform asynchronous communication, reading a stale version of each others’ models, and may compress data transmission by quantizing.
However, we do observe trade-offs between convergence rate and degree of synchronization, in that the algorithm may need additional gradient steps in order to attain a good solution.
Our algorithm, called S WARM SGD, is decentralized in sense that each node maintains local version of the model, and two interacting nodes only see each others’ models.
We further allow that the data distribution at the nodes may not be exactly i.i.d.
Speciﬁcally, each node i is assigned a set of samples S i , and maintains its own parameter estimate x i .
Each node i performs local SGD steps on its model x i based on its local data, and then picks a neighbor uniformly at random to share information with, by averaging of the two models.
(To streamline the exposition, we ignore quantization and model staleness unless otherwise speciﬁed.)
Effectively, if node i interacts with node j , node i ’s updated model becomes
[Equation]
where t is the total number of interactions performed by all nodes up to this point, j is the interaction partner of i at step t + 1 , and the input models x it,H i and x jt,H j have been obtained by iterating the
SGD step H i and H j times, respectively, locally from the previous interaction of either node.
We assume that H i and H j are random variables with mean H , that is, each node performs H local steps in expectation.
The update for node j is symmetric, so that the two models match after the averaging step.
In this paper, we analyze variants of the above SwarmSGD protocol.
The main intuition behind the algorithm is that the independent SGD steps will allow nodes to explore local improvements to the objective function on their subset of the data, while the averaging steps provide a decentralized way for the models to remain in sync, albeit in a loosely coupled way.
We show that, as long as the maximum number of local steps is bounded, this procedure still converges, in the sense that gradients calculated at average value of models are vanishing as we increase the number of interactions.
Speciﬁcally, assuming that the n nodes each take a constant number of local SGD steps on average before communicating, we show that SwarmSGD has Θ( √ n ) speedup to convergence in the nonconvex case.
This matches results from previous work which considered decentralized dynamics but which synchronized upon every SGD step, (Lian et al., 2017; 2018).
Our analysis also extends to arbitrary regular graph topologies, non-blocking (delayed) averaging of iterates, and quantization.
Generally, we show that the impact of decentralization, asynchrony, quantization, and local updates can be asymptotically negligible in some parameter regimes.
On the practical side, we show that this algorithm can be mapped to a distributed system setting, where agents correspond to compute nodes, connected by a dense communication topology.
Specifically, we apply SwarmSGD to train deep neural networks on image classiﬁcation and machine translation (NMT) tasks, deployed on the Piz Daint supercomputer (Piz).
Experiments conﬁrm the intuition that the average synchronization cost of SwarmSGD per iteration is low: it stays around 10% or less of the batch computation time, and remains constant as we increase the number of nodes.
For example, using SwarmSGD deployed on 16 nodes, we are able to train a Transformer- [Table]
faster than a highly-optimized large
[Table] batch [Table]
baseline, and to slightly higher accuracy , without additional hyper-parameter tuning.
At the same time, our method appears to be faster and more accurate than the previous practical decentralized methods, e.g. (Lian et al., 2017; 2018; Assran et al., 2018), in the same setting.
Importantly, we also note a negative result: in less overparametrized settings such as training residual
CNNs (He et al., 2016) on ImageNet (Russakovsky et al., 2015), nodes do need to perform more iterations over the dataset relative to the baseline in order to recover full accuracy.
This is predicted by the analysis, and conﬁrms similar ﬁndings in previous work (Assran et al., 2018).
Overall, however, our family of methods should be well-suited to training very large modern models, or to very large-scale settings, where global synchronization among all nodes is prohibitively expensive.
Related Work.
The study of decentralized optimization algorithms dates back to Tsitsiklis (1984), and is related to the study of gossip algorithms for information dissemination (Kempe et al., 2003;
Xiao and Boyd, 2004; Boyd et al., 2006).
Gossip is usually studied in one of two models (Boyd et al., 2006): synchronous , structured in global rounds, where each node interacts with a randomly chosen neighbor, and asynchronous , where each node wakes up at times given by a local Poisson clock, and picks a random neighbor to interact with.
The model we consider can be seen as equivalent to the asynchronous gossip model.
The key differences between our work and averaging in the gossip model, e.g. Boyd et al.
(2006), are that that 1) we consider local SGD steps, which would not make sense in the case of averaging ﬁxed initial values; and 2) the gossip input model is static (node inputs are ﬁxed, and node estimates must converge to the true mean), whereas we study a dynamic setting, where models are continually updated via SGD.
Several optimization algorithms have been analyzed in this setting (Nedic and Ozdaglar, 2009; Johansson et al., 2009; Shamir and Srebro, 2014).
Tang et al.
; Koloskova et al.
(2019) analyze quantization in the synchronous gossip model.
Lian et al.
(2017; 2018); Assran et al.
(2018) consider SGD-type algorithms in gossip-like models.
Speciﬁcally, they analyze the SGD averaging dynamic in the non-convex setting but do not allow nodes to perform local updates or quantize .
In particular, nodes perform pairwise averaging upon every SGD step.
Table 2 in the Appendix summarizes their assumptions, results, and rates.
Their results are phrased in the synchronous gossip model, in which nodes interact in a sequence of perfect matchings, for which they provide O (1 / √ Tn ) convergence rates under analytical assumptions.
Lian et al.
(2018) extends these results to a variant of the gossip model where updates can be performed based on stale information, similarly to our non-blocking extension.
Upon careful examination, one can ﬁnd that their results can be extended to the asynchronous gossip setting we consider, as long as nodes are not allowed to perform local SGD updates to their models (corresponding to H = 1 ) or to quantize communication .
Extending the analysis of distributed SGD to allow for local steps is challenging even in centralized models, see for instance Stich (2018).
If we assume H = 1 , our technique yields similar or better bounds relative to previous work in the decentralized model, as our potential analysis is speciﬁcally-tailored to this dynamic interaction model.
For instance, for Assran et al.
, the speedup with respect to the number of nodes depends on a parameter C , which in turn, depends on 1) the dimension d of the objective function, 2) the number iterations for the graph given by edge sets of all matrices used in averaging to be connected, and the 3) diameter of the aforementioned connected graph.
In the dynamic interaction model we consider, the parameter C will be at least linear in the number of nodes n , which will eliminate any speedup from such an upper bound.
We present a systematic comparison in terms of assumptions in Appendix B.
In sum, relative to prior work on decentralized algorithms, our contributions are as follows.
We are the ﬁrst to consider the impact of local updates, asynchrony, and quantization in conjunction with decentralized SGD.
We show that the cost for the linear reduction in communication in H given by local steps is at worst a squared convergence decrease in the parameter H .
Our analysis technique relies on a ﬁne-grained analysis of individual interactions, which is different than that of previous work, and can yield improved bounds even in the case where H = 1 .
By leveraging the lattice-based quantization scheme of Davies et al.
, we also allow for communication-compression.
From the implementation perspective, the performance of our algorithm is superior to that of previous methods, notably D-PSGD (Lian et al., 2017), AD-PSGD (Lian et al., 2018) and SGP (Assran et al., 2018), mainly due to the ability to take local steps.
Wang and Joshi (2018) and Koloskova et al.
(2020) provide analysis frameworks for the synchronous version of decentralized SGD with local updates, and possibly changing topologies.
This is a different setting from ours, since it requires each agent to take an equal number of gradient steps before every interaction round, and therefore does not allow for agents to progress at different speeds (asynchrony).
Further, we support quantization, and validate our analysis at scale.
P RELIMINARIES
The Distributed System Model.
We consider a model which consists of a set of n ≥ 2 anonymous agents, or nodes, each of which is able to perform local computation.
We assume that communication network of nodes is a r -regular graph G with spectral gap λ 2 , which denotes the second smallest eigenvalue of the Laplacian of G .
This choice of communication topology models supercomputing and cloud networks, which tend to be regular, densely connected and low-diameter, mimicking regular expanders (Kim et al., 2008; Besta and Hoeﬂer, 2014).
The execution proceeds in discrete steps , where in each step we sample an edge of the graph G uniformly at random
and we allow the agents corresponding to the edge endpoints interact.
Each of the two chosen agents updates its state according to a state update function, speciﬁed by the algorithm.
The basic unit of time is a single pairwise interaction between two nodes.
Notice however that in a real system Θ( n ) of these interactions could occur in parallel.
Thus, a standard global measure is parallel time , deﬁned as the total number of interactions divided by n , the number of nodes.
Parallel time intuitively corresponds to the average number of interactions per node to convergence.
We note that our model is virtually identical to the population model of distributed computing (Angluin et al., 2006), or to asynchronous gossip models (Xiao and Boyd, 2004).
Stochastic Optimization.
We assume that the agents wish to minimize a d -dimensional, differentiable function f : R d → R .
Speciﬁcally, we will assume the empirical risk minimization setting, in which agents are given access to a set of m data samples S
= { s 1 , . . .
, s m } coming from some underlying distribution D , and to functions (cid:96) i : R d → R which encode the loss of the argument at the sample s i .
The goal of the agents is to converge on a model x ∗ which minimizes the empirical loss over the m samples, that is
[Equation]
In this paper, we assume that the agents employ these samples to run a decentralized variant of
SGD, described in detail in the next section.
For this, we will assume that each agent i has access to stochastic gradients (cid:101) g i of the function f , which are functions such that
[Equation]
Stochastic gradients can be computed by each agent by sampling i.i.d.
the distribution D , and computing the gradient of f at θ with respect to that sample.
(Our analysis can be extended to the case where each agent is sampling from its own partition of data, see Section H in the Appendix.)
We will assume a the following conditions about the objective function (One of the extensions removes the second moment bound):
Smooth Gradients :
The gradient ∇ f ( x ) is L -Lipschitz continuous for some L > 0 ,
i
e. for all x, y ∈
[Equation]
d :
[Equation]
Bounded Second Moment : The second moment of the stochastic gradients is bounded by some M 2 > 0 , i.e. for all x ∈ R d and agent i :
[Equation]
Note that throughout this paper for any random variable X , by E (cid:107) X (cid:107)
we mean E [ (cid:107) X (cid:107) 2 ] .
T HE S WARM SGD A LGORITHM
Algorithm Description.
We now describe a decentralized variant of SGD, designed to be executed by a population of n nodes, interacting over the edges of r -regular graph G .
We assume that each node i has access to local stochastic gradients (cid:101) g i , and maintains a model estimate
X i .
For simplicity, we will assume that this initial model is 0 d at each agent, although its value may be arbitrary.
Each agent performs SGD steps on its local estimate X i .
At random times given by a clock of Poisson rate, we pick two neighboring agents i and j uniformly at random from G , and have them average their estimates.
The interaction is precisely described in Algorithm 1.
For simplicity, the pseudocode is expressed sequentially, although in practice nodes perform their local SGD steps in parallel. Also, for simplicity, we have assumed a constant learning rate; we will detail the precise way to update the learning rate in the next section, as well as more complex variants of this basic update.
% Let G be r -regular graph.
% Sample an edge ( i , j ) of G uniformly at random.
Require: agents i and j chosen for interaction % choose H i and H j % agent i performs H i local SGD steps for q = 1 to H
i do
X i ← X i − η (cid:101) g
i ( X i ) end for % agent
j performs H j local SGD steps for q = 1 to H j do
X j ← X j − η (cid:101) g j ( X j ) end for % agents average their estimates coordinate-wise avg ← ( X i + X j ) /
X i ← avg
X j ← avg
T HE C ONVERGENCE OF S WARM SGD
We begin by analyzing the convergence of the baseline SwarmSGD algorithm.
Fix an integer H ≥ 1 .
First, we will consider a variant where H i and H j are independent, geometrically-distributed random variables, with mean H .
This corresponds to interaction times being chosen by a Poisson clock of constant rate.
To handle the fact that the number of local steps upon an interaction is a random variable, in this ﬁrst case we will require stochastic gradients to satisfy the bounded second moment assumption , speciﬁed above.
Intuitively, this is required since otherwise the “distance travelled” by a node could be virtually unbounded.
In this setting, we prove the following:
Theorem 4.1.
Let f be an non-convex, L -smooth function, whose stochastic gradients satisfy the bounded second moment assumption above.
Let the number of local stochastic gradient steps performed by each agent upon interaction be a geometric random variable with mean H .
Let the learning rate we use be η = n/ √ T .
Deﬁne µ t = (cid:80) n i =
X it /n , where X it is a value of model i after t interactions, be the average of the local parameters.
Then, for learning rate η = n/ √ T and any number of interactions T ≥ n 4 :
[Equation]
Discussion.
First, notice that each of the upper bound terms has an intuitive interpretation: the ﬁrst represents the reduction in loss relative to the initialization, and gets divided by the number of local steps H , since progress is made in this term in every local step; the second represents the inﬂuence of the variance of each individual local step multiplied by a term which bounds the impact of the graph topology on the convergence.
In particular, this term negatively impacts convergence for large values of H , L , and M , but gets dampened if the graph is well-connected (i.e. large λ 2 ).
For example, in the case of the complete graph, we have λ 2 = n .
Second, importantly, the time T in this bound counts the total number of interactions.
However, in practice Θ( n ) pairwise interactions will occur in parallel, as they are independent.
Therefore, we can replace T by nT in the above formula, to estimate the speedup in terms of wall-clock time.
At the same time, notice that this speedup is dampened in the second term by the non-trivial additional variance due to noisy local gradient steps, a fact which we will revisit in the experimental section.
Third, although the requirement T ≥ n 4 appears restrictive, some non-trivial dependency between n and T is necessary, as gradient information has to “mix” well in the graph before global optimization can occur.
Previous work requires stronger variants of this restriction: speciﬁcally, Lian et al.
(2018) require T ≥ n 6 , while Assran et al. (2018) requires T = Ω( nd 2 ) .
Proof Overview.
At a high level, the argument rests on two technical ideas.
The ﬁrst idea is to show that, due to the pairwise averaging process, and in spite of the local steps, the nodes’ parameters will have to remain concentrated around their mean µ t .
The second is to show that, even though stochastic gradients are taken at perturbed, noisy estimates of this mean, the impact of this noise on convergence can be bounded.
In particular, the main technical difﬁculty in the proof is to correctly “encode” the fact that parameters are well concentrated around the mean.
For this, we deﬁne the potential Γ t , which denotes the variance of models after t interactions.
Formally,
[Equation]
where µ t = (cid:80) ni
=
X it /n .
We bound the expected evolution of Γ t in terms of r , the degree of nodes in the interaction graph G , and λ 2 , the second smallest eigenvalue of the Laplacian of G .
For both algorithm variants we consider, our bound depends on the learning rate, number of local steps, and the bound provided by the assumption on the stochastic gradients (the bound M 2 and σ 2 ).
The critical point is that the upper bound on the expectation of Γ t does not depend on the number of interactions t .
Our approach is roughly similar to the analysis of static load balancing schemes, e.g. Berenbrink et al.
Two key elements of novelty in our case are that (1) for us the load balancing process is dynamic , in the sense that loads (gradients) get continually added; (2) the loadbalancing process we consider is multi-dimensional, whereas usually the literature considers simple scalar weights.
The complete argument is presented in the Appendix.
This technique is quite powerful, as it allows for a number of non-trivial extensions:
Extension 1: Removing the second-moment bound and allowing for non-i.i.d. local data.
In the ﬁrst extension of the algorithm, we assume that the number of local steps performed by each agent is ﬁxed and is equal to H .
In this case, we are able to remove the bounded second moment assumption, and are able to prove convergence under standard assumptions for non-i.i.d data. Speciﬁcally, in the non-i.i.d.
setting, we consider that each f i ( x ) is the local function of agent
i (computed over the samples available to i ).
We will require that 1) the function f i is L -smooth, and that 2) for each agent i , ˜ g i is unbiased estimate of f i and that 3) for any x , E [ (cid:107)
˜
g
i ( x ) − f i ( x ) (cid:107) 2 ] ≤ σ 2 .
We deﬁne f ( x ) = (cid:80) ni
{+=
+}
f i ( x ) /n and the bound (cid:80) ni =1 (cid:107)∇ f i ( x ) − ∇ f ( x ) (cid:107) 2 /n ≤ ρ 2 .
Theorem 4.2. Let f be an non-convex, L-smooth function whose minimum x (cid:63) we are trying to ﬁnd via the SwarmSGD procedure given in Algorithm 1.
Let local functions of agents satisfy conditions discussed above.
Let H be the number of local stochastic gradient steps performed by each agent upon interaction.
Deﬁne µ t = (cid:80) ni
=
X it /n , where X it is a value of model i after t interactions.
For learning rate η = n √ T and T =
O (cid:18)
[Equation]
r
[Equation]
22 + [Equation]
[Equation]
we have that: [Equation]
Please see Appendix H for the details of the proof; we note that we did not optimize for constants.
Relative to Theorem 4.1, we have the same quadratic dependency on the number of local steps H and on L , but now the second moment bound is replaced by the variance terms. We emphasize that for non-i.i.d data under second-moment bounds, the statement of Theorem 4.1 will hold with exactly the same bounds.
Extension 2: Non-blocking averaging.
Algorithm 1 is blocking , in that it requires both nodes to complete their local iterations at the same time before they can interact.
In practice, nodes can average their local updates without synchronizing, as follows.
Each node i keeps two copies of the model: the live copy X
i (on which local SGD iterations are applied) and the communication copy
Y i , which can be accessed asynchronously by communicating nodes.
When completing its local steps, a node i ﬁrst checks if some other node averaged against its communication copy Y i since its last communication step.
If the answer is yes, it simply applies its locally-generated update to its communication model Y i , updates the live copy so that X i
=
Y i , and proceeds with the next iteration of local computation.
If no other node has averaged against its communication copy, then the node actively seeks a random communication partner j , and averages its live copy against its model Y j , updating both to ( X i + Y j ) / 2 .
The node then proceeds with the next iteration of local computation.
Please see Appendix F for the precise deﬁnition of the algorithm, and for the formal convergence guarantee for this variant.
Extension 3: Quantization.
For large models, the cost of the averaging step can become signiﬁcant, due to bandwidth constraints.
To remove the bandwidth bottleneck, we allow the averaging step to be performed with respect to quantized versions of the two models.
While communicationcompression has been considered in a decentralized context before, e.g. Tang et al.
; Lu and
Sa (2020), our proposal approaches it differently.
Instead of modifying the algorithm or maintaining neighbor information at every node, we implement quantization by making use of the quantization scheme proposed by Davies et al.
(2020), which we adapt to our context.
The key issue when quantizing models is that for most known quantization schemes, e.g. (Alistarh et al., 2017), the quantization error depends on the norm of the inputs : in our setting, since models are not necessarily close to the origin, the quantization error at each step would depend on the norm of the models themselves, which would break our bound on Γ t .
Instead, we observe that the quantization scheme of Davies et al.
(2020) guarantees error which is bounded by the distance between inputs , rather than input norms.
Crucially, observe that Γ t can in fact be used to bound the distance between models.
Thus, we can bound quantization error in terms of Γ t at each steps, which allows us, with some care, to generalize the analysis to the case where the models are quantized, up to constants.
We provide a full description and proof of convergence in Appendix G.
E XPERIMENTAL R ESULTS
In this section, we validate our analysis, by applying the algorithm to training deep neural networks for image classiﬁcation and speech recognition.
We map the algorithm onto a multi-node supercomputing setting, in which we have a large number of compute nodes, connected by fast communication links.
The key overhead in this setting is synchronization : at large node counts, the cost of synchronizing all nodes so they execute in lock-step can be very high, see e.g. Li et al.
for numerical results on different workloads.
SwarmSGD mitigates this overhead, since nodes synchronize only sporadically and in pairs.
Harnessing the computational power of this large-scale distributed setting is still an underexplored area (Ben-Nun and Hoeﬂer, 2019).
Target System and Implementation.
We run SwarmSGD on the CSCS Piz Daint supercomputer, which is composed of Cray XC50 nodes, each with a Xeon E5-2690v3 CPU and an NVIDIA Tesla
P100 GPU, using a state-of-the-art Aries interconnect over a Slimﬂy topology.
Please see (Piz) for more details.
We implemented SwarmSGD in Pytorch and TensorFlow using NCCL and MPIbased primitives.
Both variants implement the version with non-blocking averaging.
The Pytorch implementation is on top of SGP framework of SGP (Assran et al., 2018), and uses SwarmSGD to train ResNets on the CIFAR-10/100 (Krizhevsky et al., 2014) and ImageNet (Russakovsky et al., 2015) datasets, while we use the TensorFlow implementation to train a much larger Transformer-XL model (Vaswani et al., 2017) on the WMT17 (En-Ge) dataset.
We note that all algorithms used the same communication topology (fully-connected with random pairings), and that SGP was run with overlap factor 1 , as suggested in Assran et al.
Training Process.
Our training methodology follows data-parallel training, with some differences due to decentralization, and is identical to previous work on decentralized and local SGD, e.g. Lian et al.
(2017); Assran et al.
Lin et al.
Training proceeds in epochs , each of which corresponds to processes collectively performing a full pass over the dataset.
At the beginning of each epoch, we re-shufﬂe the dataset and partition it among processes (Lin et al., 2018).
As noted in previous work (Lian et al., 2017; 2018; Assran et al., 2018) variants of decentralized
SGD are not always able to recover sequential SGD accuracy within the same number of epochs as this baseline.
This is justiﬁed by Theorems 4.1 and 4.2, which predict that the slower mixing (and higher local model variance) can affect convergence.
Thus, in some experiments, we will allow the decentralized schemes to execute for more epochs, by a constant multiplier factor between 1 and 3 . Once we have ﬁxed the number of epochs, we do not alter the other training hyperparameters : in
[Table]
[Figure]
particular, the learning rate schedule, momentum and weight decay terms are identical to sequential
SGD, for each individual model.
Accuracy and Speed.
We ﬁrst examined whether SwarmSGD can in fact recover full accuracy versus the sequential or large-batch SGD baselines.
In Table 1 we provide an overview of parameter values to recover or exceed large-batch SGD accuracy (following (Goyal et al., 2017)) using
SwarmSGD, on the ResNet/ImageNet/CIFAR tasks.
We execute for 32 nodes on ImageNet, andnodes on CIFAR-10.
(Local batch sizes are 256 for ResNet20 and ResNet18, and 128 for ResNet50.
Quantization is not applied.)
The results show that Swarm can recover or slightly exceed the accuracy of the large-batch baselines, and that it has lower practical communication cost relative to existing methods (see Figure 2(b), where we separate the average computation cost per batch, constant at ∼ 0 .
4 for all methods, from the communication cost).
However, Swarm requires signiﬁcant additional passes over the data (up to 2 .
7 × )
to achieve full accuracy, which negates its performance beneﬁts in this speciﬁc setting, relative to large-batch SGD.
(Please see Appendix Figure 5 for an end-to-end time comparison.
We do not take the cost of ﬁne-tuning the hyperparameters for largebatch SGD into account in this example.)
This ﬁnding is in line with previous work on decentralized methods (Assran et al., 2018).
Next, we examine accuracy for the WMT17 task.
The results are provided in Figure 1(a), in accuracy-vs-time format, for 16 and 32 nodes, executing for 10 global epochs.
Here, the large-batch
SGD (LB-SGD) baseline (BLEU score 26.1 at 16 nodes) is a poor alternative at high node counts: its throughput is very low, due to the size of the model (see Figure 1(b)).
At 16 nodes, Swarm slightly exceeds the baseline accuracy at 26.17 BLEU, for an end-to-end speedup of ∼ 1 . 5 × .
In the same setting, Swarm outperforms all other decentralized methods (the fastest previous method, ADPSGD, is 30% slower, and less accurate), both in terms of BLEU score, and in terms of end-to-end time.
At 32 nodes, all decentralized methods reach lower scores ( ∼ 23 . 5 ) after 10 epochs.
However, we observed experimentally that running Swarm for an additional 5 epochs at 32 nodes recovered a
BLEU score of ∼ 25 .
9 , 30% faster than the 16-node version in terms of end-to-end time (omitted for readability).
The objective loss has a similar behavior to the BLEU score; the graph is provided in Appendix Figure 7.
[Figure]
In addition, we investigated 1) the accuracy of the real average of all models throughout training: it is usually more accurate than an arbitrary model, but not signiﬁcantly, corroborating the claim that individual models tend to stay close to the mean; 2) the inﬂuence of the number of local steps on accuracy: perhaps surprisingly, we were able to recover baseline accuracy on ResNet18/ImageNet for up to 4 local steps (see Figure 2(a)); 3) the impact of quantization on convergence, where we were able to recover accuracy when applying 8-bit model quantization to Swarm.
We encourage the reader to examine the experimental report in the Appendix, which contains data on these experiments.
Discussion.
Generally, the performance and accuracy of SwarmSGD are superior to previous decentralized methods (see Figure 1 for an illustration, and the Appendix for a performance breakdown).
In particular, a closer examination of the execution times given in Figure 2(b) shows that time per node per batch (including communication and computation) is largely constant as we increase the number of nodes, which gives our method close-to-ideal scaling behaviour.
This advantage relative to previous schemes, notably AD-PSGD, comes mainly from the reduction in communication frequency :
Swarm communicates less often, and therefore incurs lower average communication cost.
The main shortcoming of Swarm is that, similar to previous decentralized methods, it may need additional data passes in order to fully recover accuracy at high node counts.
However, we also note that our method does not beneﬁt from or require the same level of hyperparameter tuning as largebatch SGD, e.g. (Goyal et al., 2017).
We ﬁnd it interesting that this accuracy issue is less prevalent in the context of large, over-parameterized models, such as the Transformer, where Swarm could be a practically-viable alternative to large-batch SGD within the same number of epochs.
C ONCLUSIONS AND F UTURE W ORK
We have analyzed the convergence of SGD in a decoupled model of distributed computing, in which nodes mostly perform independent SGD updates, interspersed with intermittent pairwise averaging steps, which may be performed in an inconsistent and noisy manner.
We have shown that SGD is able to still converge in this restrictive setting, and under considerable consistency relaxation, and moreover can still achieve speedup in the number of nodes, in terms of iteration time.
The empirical results in a supercomputing environment complement and conﬁrm our analysis, showing that this method can be competitive with standard solutions.
SwarmSGD provides good analytic and empirical properties, even though each model only directly observes a small fraction of the data.
Our work opens several avenues for extensions.
One natural but non-trivial direction is to further generalize the bounds to arbitrary communication graphs, or in terms of their the assumptions on the objective and on the system setting considered.
A S UMMARY OF THE A PPENDIX SECTIONS
Appendix contains the following sections:
• In Section B we compare SwarmSGD with some of the existing algorithms.
We list convergence bounds and the assumptions needed to achieve them.
• In Section C we provide crucial properties for the load balancing on the graph.
• In Section D we provide deﬁnitions for the local steps we use in the later sections.
• In Section E we provide the sketch of proof of Theorem 4.1, which shows the convergence of SwarmSGD assuming the second moment bound on the gradients.
Recall that the number of local steps in this case is a geometric random variable with mean H .
• In Section F we provide the proof for the non-blocking version of the swarm SGD algorithm.
We again assume the second moment bound on the gradients and that the number of local steps is a geometric random variable with mean H .
In Section G we provide the proof for the quantized version of the swarm SGD algorithm.
We again assume the second moment bound on the gradients and that the number of local steps is a geometric random variable with mean H .
In Section H we prove Theorem 4.2.
In this case we do not assume the second moment bound, data is not distributed identically and the number of the local steps performed by each agent is a ﬁxed number H .
In Section I we provide additional experimental results for SwarmSGD.
B C OMPARISON OF RESULTS
In this section we compare convergence rates of existing algorithms, while specifying the bounds they require for convergence.
In the tables T -corresponds to the parallel time and n is a number of processes.
We use the following notations for needed bounds (or assumptions):
σ 2 - bound on the variance of gradient.
M 2 - bound on the second moment of gradient.
d - bounded dimension.
λ 2 - bounded spectral gap of the averaging matrix (interaction graph in case of SwarmSGD).
τ - bounded message delay.
r - interaction graph is r -regular.
∆ - bounded diameter of interaction graph.
Algorithm
Assumptions
[Table]
σ
[Equation]
[Table]
Discussion.
We compare in more detail against Lian et al.
(2018) and Assran et al.
(2018), since these are the only other papers which do not require explicit global synchronization in the form of rounds.
(By contrast, e.g. Wang and Joshi (2018); Koloskova et al. (2020) require that nodes synchronize in rounds, so that at every point in time each node has taken the same number of steps.)
In Assran et al.
(2018), all nodes perform gradient steps at each iteration, but averaging steps can be delayed by τ iterations.
Unfortunately, in this case the mixing time depends on the dimension - d (more precisely, it contains a √ d factor!), on the delay bound τ , and on ∆ , deﬁned as the number of iterations over which the interaction graph is well connected.
Additionally, the analysis is not suitable for random interactions.
On the other hand, Lian et al.
(2018) consider random interaction matrices and do not require the agents to perform the same number of gradient steps.
Unlike our model, in their case more than two nodes can interact during the averaging step.
To circumvent the global synchronization issue, Lian et al.
(2018) allow agents to have outdated views during the averaging step.
Yet, we would like to emphasize that they require blocking during the averaging steps, while we allow some amount of non-blocking property.
By some amount we means that algorithm needs blocking only in the case when some node takes more than two consecutive iterations to complete it’s local gradient steps.
This means that for each node i to complete
H i local steps should not more take more then O ( n ) global steps (since each node interacts with probability 2 /n at each step), this assumption also holds for Lian et al.
In summary, our algorithm reduces the synchronization required by averaging steps, by considering pairwise interactions and by introducing local steps and providing non-blocking version of the algorithm as well (in SGP and AD-PSGD, agents perform one local step and one averaging step per iteration).
We would like to point out that we also allow a random number of local steps between interactions in the case when we have second moment bound on the stochastic gradient, which reduces synchronization costs even further.
Finally, our algorithm requires T ≥ O ( n 4 ) number of iterations to achieve the convergence rate of O (1 / √ Tn ) in the case of blocking algorithm and T = Ω( n 6 ) in general.
(By contrast, Lian et al.
(2018) requires T = Ω( n 6 ) .)
C P ROPERTIES OF THE L OAD B ALANCING
In this section provide the useful lemmas which will help as in the later sections.
We are given a simple undirected graph G , with n nodes (for convenience we number them fromto n ) and edge set E .
Each node is adjacent to exactly r nodes.
Each node i of graph G keeps a local vector model X it ∈
R d ( t is the number of interactions or steps); let X t =
( X 1 t , X 2 t , ..., X nt ) be the vector of local models at step t .
An interaction (step) is deﬁned as follows: we pick an edge e
= ( u, v ) of G uniformly at random and update the vector models correspondingly.
Let µ t
= (cid:80) ni
=
X it /n be the average of models at step t and let Γ t = (cid:80) ni =1 (cid:107)
X it − µ t (cid:107) 2 be a potential at time step t .
Let L be the Laplacian matrix of G and let let λ 2 be a second smallest eigenvalue of L .
For example, if G is a complete graph λ 2 = n .
First we state the following lemma from Ghosh and Muthukrishnan (1996):
Lemma C.1.
[Equation]
Now, we show that Lemma C.1 can be used to lower bound (cid:80) ( i,j
) ∈ E (cid:107) X
[Equation]
Lemma C.2.
[Equation]
Proof.
Observe that
[Equation]
Also, notice that Lemma C.1 means that for every vector v = ( v 1 , v 2 , ..., v n ) such that (cid:80) ni
=1 v i = 0 , we have:
[Equation]
Since (cid:80) ni =
( X it − µ t ) is a 0 vector, we can apply the above inequality to the each of d components of the vectors X 1 t − µ t , X 2 t − µ t , ..., X nt − µ t separately, and by elementary properties of 2 -norm we prove the lemma.
D D EFINITIONS FOR THE L OCAL S TEPS
In this section we provide the formal deﬁnition of the local steps performed by our algorithms.
Recall that X
it is a local model of node
i at step t .
Let H it be the number of local steps node i performs in the case when it is chosen for interaction at step t + 1 .
A natural case is for H it to be ﬁxed throughout the whole algorithm, that is: for each time step t and node i , H it =
H .
However, optimal choice H it depends on whether a second moment bound on gradients (5) is assumed.
Let:
[Equation]
and for 1 ≤ q ≤ H it let:
[Equation]
Note that stochastic gradient is recomputed at each step, but we omit the superscript for simplicity, that is: (cid:101) [Equation]
X [-it − (cid:80) q − 1 s =
-][Equation] = [Equation]
si [Equation] X it )) .
Further , for 1 ≤ q ≤ H it , let
[Equation]
be the expected value of (cid:101) h qi ( X it ) taken over the randomness of the stochastic gradient (cid:101) g i .
Let (cid:101) h
i ( X it ) be the sum of H it local stochastic gradients we computed:
[Equation]
Similarly, for simplicity we avoid using index t in the left side of the above deﬁnition, since it is clear that if the local steps are applied to model X it we compute them in the case when node i interacts at step t + 1 .
The update step in Swarm SGD (Algorithm 1) is (before averaging):
[Equation]
Notice that
[Equation]
E A NALYSIS UNDER SECOND MOMENT BOUND AND RANDOM NUMBER OF LOCAL STEPS
In this section we consider Algorithm 1, where for each node i , H
i is a geometric random variable with mean H .
We also assume a gradient second moment bound ( 5 ).
We provide only a sketch of the proof since the proof for the non-blocking version of algorithm in Section F is more general.
If nodes i and j interact at step t + 1 and their local models have values of X it and X jt
after step t .
Their new model values become:
[Equation]
Recall that µ t is average of the values of models after time step t and Γ t = (cid:80) ni
=1 (cid:107) X it − µ t {+(cid:107) 
+}
First of all we can prove that (see Lemma F.1, the result can be achieved even though algorithms differ):
[Equation]
We can further show that Lemma F.2, and therefore Lemmas F.3 and F.4, also hold, yielding:
[Equation]
and n
[Equation]
Next in the similar fashion as in the proof of Theorem F.8 we can show that :
[Equation]
This allows to show that:
Theorem 4.1.
Let f be an non-convex, L -smooth function, whose stochastic gradients satisfy the bounded second moment assumption above.
Let the number of local stochastic gradient steps performed by each agent upon interaction be a geometric random variable with mean H .
Let the learning rate we use be η = n/ √ T .
Deﬁne µ t = (cid:80) ni
=
X it /n , where X i t is a value of model i after t interactions, be the average of the local parameters.
Then, for learning rate η = n/ √ T and any number of interactions T ≥ n 4 :
[Equation]
Proof.
We again skip the calculations and follow steps from the proof of Theorem F (note that constants can be improved, but for simplicity we keep them the same).
After applying (13) and (14), this results in:
[Equation]
once we sum up the above inequality t = 0 to t =
T − 1 and massage terms we get (additionally recall that E [ f ( µ T )] ≥ f ( x ∗ ) ):
[Equation]
Finally since η = n/ √ T ≤ 1 n (because T ≥ n 4 ) we get the proof of the lemma.
Note that the difference between this theorem and Theorem F.8 is that we have lower bound of n 4 for T here, instead of n 4 ( n + 1) 2 . The reason is that we are not required to use Lemmas F.7 and F.5 since our algorithm allows blocking and this means that interacting agents do not have incomplete values for the models.
F A NALYSIS OF THE N ONBLOCKING V ARIANT , WITH S ECOND M OMENT B OUND AND R ANDOM N UMBER OF L OCAL S TEPS
First we deﬁne how the non-blocking property changes our interactions.
Let i and j be nodes which interact at step t + 1 , we set
[Equation]
and
[Equation]
where for each node k , if p kt + 1 is the last time interacting before and including step t :
[Equation]
Intuitively the last deﬁnition means that node k has computed X ip kt +1 / 2 but has not ﬁnished computing X p kt
+1 , hence when some other node tries to read X p kt
+1 , it reads the value which is missing local gradient update step, but it does not have to wait for node k to ﬁnish computing.
Since p kt +is the last step node k interacted we have that X kt =
X kp kt
+1 .
More formally: Algorithm j
% Let G be r -regular graph.
% Sample an edge ( i , j ) of G uniformly at random.
Require: agents i and j chosen for interaction, i is initiator % choose H i and H j % agent i performs H i local SGD steps
S i ←
X i for q = 1 to H i do
X i ← X i − η (cid:101) g
i ( X i ) end for % agent
j performs H j local SGD steps
S j ← X j for q = 1 to H j do
X j ← X j − η (cid:101) g j ( X j ) end for % agents update their estimates
[Equation]
Notice the differences between the main algorithm and non-blocking one: ﬁrst, local gradient steps are applied only after the averaging steps (this corresponds to term
X
i − S
i for node i ), and second, nodes get access to the model of their interacting partner, which might not be complete for the reasons described above (for example, node i is forced to use X j (cid:48) instead of X j in its averaging step).
If node i is the initiator of the interaction and its chosen interaction partner j is still computing the local gradients from its previous interaction, this algorithm allows node i not to wait for j to ﬁnish computation.
In this case, i simply leaves its value X i (cid:48) in j ’s memory.
Notice that since i is ﬁnished computation it does not need to pass its outdated model to j , but we assume the worst case.
We proceed by proving the following lemma which upper bounds the expected change in potential:
Lemma F.1.
For any time step t we have:
[Equation]
Proof.
First we bound change in potential ∆ t =
Γ t +1 − Γ t for some ﬁxed time step t > 0 .
For this, let ∆ i,jt be the change in potential when we choose agents ( i, j ) ∈ E for interaction (While calculating ∆ i,jt we assume that X t is ﬁxed).
Let R it = − η (cid:101) h
i ( X it ) + η (cid:101) h
j (cid:16) X j pjt (cid:17) and R jt =
[Equation]
have
[Equation]
[Equation]
This gives us that:
[Equation]
For k (cid:54) = i, j we get that
[Equation]
Hence:
[Equation]
Observe that:
[Equation]
After combining the above two equations
[Equation]
we get that:
[Equation]
Recall that R it = − η (cid:101) h
i ( X it )+ η (cid:101) h
j (cid:16) X j pjt (cid:17) and R jt = − η (cid:101) h
j
( X jt )+ η (cid:101)
h
i (cid:16) X ipit (cid:17) , Using Cauchy-Schwarz inequality we get that
[Equation]
(cid:107) (cid:101)
[Equation]
(cid:16) X
[Equation]
it (cid:17) (cid:107) 2 . Denote 2 η [-
-]2 (cid:107) (cid:101) h
i ( X it ) (cid:107) 2 + η 2 2 (cid:107) (cid:101) h i (cid:16)
X ip it (cid:17) (cid:107) 2 by S it and 2 η 2 (cid:107) (cid:101)
h
i ( X jt ) (cid:107)
+ η 2 2 (cid:107) (cid:101)
h j (cid:16) X j p jt (cid:17) (cid:107) 2 by S jt .
Hence (17) can be rewritten as:
[Equation]
Further:
[Equation]
This gives us:
[Equation]
( i,j ) ∈ E ∆ i,jt ≤ (cid:88) ( i
[Equation]
j )
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
Next, we use the above inequality to upper bound ∆ t in expectation:
[Equation]
n . Finally, we remove the conditioning:
[Equation]
By considering the deﬁnition of ∆ t and S it
[Equation]
we get the proof of the lemma.
Next
[Equation]
we upper bound the second moment of local updates , for any step t and node i :
Lemma F.2.
[Equation]
Proof.
[Equation]
Where in the last step we used
[Equation]
This allows us to upper bound the potential in expectation for any step t .
Lemma F.3.
[Equation]
Proof.
We prove by using induction.
Base case t = 0 trivially holds.
For an induction step step we assume that E [Γ t ] ≤ ( 40 r λ 2 + 80 r 2 λ 22 ) nη 2 H 2 M 2 r 2 .
We get that :
[Equation]
The next lemma allows us to upper bound (cid:80)
ni =1 E (cid:104)∇ f ( µ t ) , − (cid:101) h
i ( X it ) (cid:105) which will be used later once we apply L -smoothness to upper bound f ( µ t +1 ) .
The intuition is as follows: if for each i , (cid:101) h
i ( X it ) was just a sum of single stochastic gradient( H i = 1 ) by the unbiasedness property we would have to upper bound (cid:80) ni
=1 E (cid:104)∇ f ( µ t ) , −∇ f
( X it ) (cid:105) =
(cid:80) ni
=1 (cid:16) E (cid:104)∇ f ( µ t ) , ∇ f ( µ t ) − ∇ f ( X it ) (cid:105) −
E (cid:107)∇ f ( µ [Equation] (cid:17) , which can be done by using L -smoothness and then deﬁnition of Γ t .
Lemma F.4.
For any time step t .
[Equation]
Proof.
[Equation]
Using Young’s inequality we can upper bound E
[Equation]
{+s =
+}
η
[Equation]
h si
[Equation]
X it ))
[Equation]
by
[Equation]
Plugging this in the above inequality we get:
[Equation]
Next we use Cauchy-Schwarz inequality on E (cid:107) µ
[Equation]
X it +
[Equation]
h si
[Equation]
X it
[Equation]
n
[Equation]
i =
[Equation]
Term E (cid:107) (cid:80) q − 1 s =0 η (cid:101) h si ( X it ))
(cid:107) 2 can be upper bounded by q 2 M 2 using Cauchy-Schwarz and assumption (5).
Hence:
[Equation]
Note that:
[Equation]
[Equation]
Also:
[Equation]
Where in the last step we used (Recall that H it is a geometric random variable with mean H ):
[Equation]
By plugging inequalities (22) and (21) into inequality (20) we get the proof of the lemma.
Our next goal is to upper bound (cid:80)
[Equation]
Lemma F.5.
n
[Equation]
Proof.
The proof is very similar to the proof of lemma F.4, except that when we subtract and add term E |∇ f ( µ t ) (cid:107) 2 in the proof it will eventually end up with a positive sign (After using Young’s inequality it will have factor of 14 + 1 instead of factor of 14 − 1 ) and we have (cid:80) ni =1 E (cid:107) µ t − X ip it (cid:107)instead of Γ t = (cid:80) ni =1 E (cid:107) µ t − X it (cid:107) 2 .
Thus, we omit the proof in this case.
Next step is to upper bound (cid:80)
ni
=1 E (cid:107) µ t − X ip it (cid:107) 2 , for this we will need the following lemma:
Lemma F.6.
For any node i and time step t ,
[Equation]
Proof.
Notice that
[Equation] [Equation]
where we used Cauchy-Schwarz inequality in the lastt step.
Fix step s .
Let u and v be nodes which interact at step s + 1 .
We have that
[Equation]
We again used the Cauchy-Schwarz inequality since the expectation is taken only over the randomness of sampling and number of local steps.
We can use the approach from lemma F. 2 to upper bound E (cid:107) (cid:101) h u ( X us ) (cid:107) 2 , E (cid:107) (cid:101) h v ( X vs ) (cid:107) 2 , E (cid:107) (cid:101) h u ( X up us ) (cid:107) 2 and E (cid:107) (cid:101) h v ( X vp vs ) (cid:107) 2 .
(In the lemma we upper bound the sum of n similar terms, but with η 2 .)
Hence:
[Equation]
t − p it is a geometric random variable with mean n/ 2 (because the probability that node i interacts is 2 /n at every step).
Thus, E [( p it − t ) 2 ] = 2( E [ t − p it ]) 2 − E [ t − p it ] ≤ n 2 2 .
Thus, E (cid:107) µ t − µ p it (cid:107) 2 ≤ 10 η 2 H 2 M 2 .
Finally we can show that:
Lemma F.7.
For any node i and time step t ,
[Equation]
Proof.
Using Cauchy-Schwarz inequality we get:
[Equation]
Where the last inequality comes from Lemmas F.3 and F.6.
Now we are ready to prove the main theorem.
Theorem F.8.
Let f be an non-convex, L -smooth, function satisfying assumption 5, whose minimum x (cid:63) we are trying to ﬁnd via the non-blocking version of SwarmSGD procedure (See, algorithm 2).
Let the number of local stochastic gradient steps performed by each agent upon interaction be a geometric random variable with mean H .
Let the learning rate we use be η = n/ √ T .
Deﬁne µ t = (cid:80) ni
{+=
+}
X it /n , where X it is the value of model i after t interactions.
Then, for learning rate η = n/ √ T and any T ≥ n 4 ( n + 1) 2 :
[Equation]
Proof.
Let E t denote expectation conditioned on { X t 1 , X t 2 , ..., X tn } .
By L -smoothness we have that
[Equation]
After removing conditioning:
[Equation]
[Equation]
First we look at E [ µ t +1 − µ t ] .
If agents i and j interact, (which happens with probability 1 rn/ 2 ).
We have that µ t +
[Equation]
X j p jt ) .
Hence we get that
[Equation]
h i ( X ip it )
[Equation]
(cid:33) =
[Equation]
n
[Equation]
i =
E
[Equation]
[ (cid:101) h
i ( X i t )]
[Equation]
n (cid:88) i = (cid:101)
h
[Equation]
( X i p i t ) .
and
E
[Equation]
Next we look at E (cid:107) µ t +1 − µ t (cid:107) 2 .
If agents i and j interact, (which happens with probability 1 rn/ 2 ).
We have that µ t
[Equation]
1 −
[Equation]
= −
[Equation]
X it ) [Equation] X jt ) + [Equation]
2 n
[Equation]
i ( X ip it ) + η 2 n (cid:101) h j ( X j p jt ) .
Hence we get that
E t
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
and
[Equation]
Hence
[Equation]
we can rewrite (25) as:
[Equation]
Next [Equation] we use Lemmas F.4 and F.5:
[Equation]
We use Lemmas F.3 and F.7 to upper bound E
[Γ t ] and (cid:80) ni =1 E (cid:107) µ t − X ip it (cid:107) 2 respectively :
[Equation]
by summing the above inequality for t = 0 to t
= T −
[Equation]
we get that
[Equation]
From this we get that :
[Equation]
Note that E [ f ( µ T )] ≥ f ( x ∗ ) , hence after multiplying the above inequality by 4 n ηHT we get that
[Equation]
Observe that η = n/ √ T ≤ 1 n ( n +1) , since T ≥ n 4 ( n + 1) 2 .
This allows us to ﬁnish the proof:
[Equation]
G A NALYSIS OF Q UANTIZED A VERAGING , A SSUMING S ECOND M OMENT B OUND AND R ANDOM N UMBER OF L OCAL S TEPS
First we deﬁne how quantization of models changes our interactions.
Both the algorithm and the analysis in this case are similar to those of Section F. Let i and j be nodes which interact at step t
+ 1 , we set
[Equation]
and
[Equation]
where for each node k , X kt (cid:48) is a quantized version of the model X kt .
We use the quantization provided in Davies et al.
The key property of this quantization scheme is summarized below:
Lemma G.1.
Let q be a parameter we will ﬁx later.
If the inputs x u , x v at nodes u and v , respectively satisfy (cid:107) x u − x v (cid:107) ≤ q q d (cid:15) , then with probability at least 1 − log log( 1 (cid:15) (cid:107) x
u − x v (cid:107) ) · O ( q − d ) , the quantization algorithm of Davies et al.
(2020) provides node v with an unbiased estimate x u (cid:48) of x u , with (cid:107) x u (cid:48) − x u (cid:107) ≤ ( q 2 + 7) (cid:15) , and uses O (cid:0) d log( q (cid:15) (cid:107) x u − x v (cid:107) ) (cid:1) bits to do so.
For our purposes, x u and x v are the local models of the nodes u and v and d is their dimension (we omit the time step here).
In the following , we refer to the above lemma as the quantization lemma.
Recall that in section E, for each node k , if p kt + 1 is the last time interacting before and including step t :
[Equation]
Analysis Outline.
The crucial property we used in the analysis is that E [ (cid:101) h k
( X kp kt )] ≤ 2 H 2 M 2 η(see Lemma F.2).
We also used that for (cid:101)
h k
( X kp kt ) , we can use the smoothness property (4) in
Lemmas F.5 and F.7.
In our case we plan to use the quantization lemma above.
For this, ﬁrst notice that the estimate is unbiased: this means that E [ X kt (cid:48) ]
=
X kt , eliminating the need to use Lemma F.5 and subsequently F.7, since (cid:80) ni =1 E (cid:104)∇ f ( µ t ) , X it − X it (cid:48) (cid:105) = 0 in our case.
Secondly if we set ( q 2 + 7) (cid:15) = HηM we also satisfy Lemma F.2.
This means that entire analysis can be replicated, and even further as in the case of section E we will only need T ≥ n 4 (In one case we do not use Lemma F.5 at all, and in the second case upper bound can be replaced by 0, which is the same as not using it).
Now we concentrate on calculating the probability that (cid:107) x u − x v (cid:107) ≤ q q d (cid:15) (which we call the distance criterion) required by the quantization lemma to hold over T steps and each pair of nodes.
We also need to calculate the probability with which we fail to decode.
Assume that Lemma F.3 holds for step t , as in the proof of this lemma we will use induction and
Lemma F.1 (we omit the proof since it will be exactly the same given that the conditions we discuss above hold).
That is: E [Γ t ] ≤ ( 40 r λ 2 + 80 r 2 λ 22 ) nη 2 H 2 M 2 .
Notice that for a pair of nodes u and v , (cid:107) X ut − X vt (cid:107) 2 ≤ 2Γ t (Using Cauchy-Schwarz).
Hence we need to calculate the probability that Γ t ≥ ( q q d (cid:15) ) 2 / 2 .
Using Markov’s inequality, the probability of this happening is at most:
[Equation]
We set q = 2 + T 3 /d , this means that ( q q d ) 2 ≥ 2 T 3 .
So given that T ≥ n 4 , we have that Pr [Γ t ≥ ( q q d (cid:15) ) 2 / 2] ≤ O (1 /T 4 ) (note that r ≤ n − 1 and λ 2 = Ω(1 /n 2 ) since our graph is connected).
Hence the distance criterion is satisﬁed with probability 1 − O (1 /T 2 ) .
Given that it is satisﬁed, we also have failure probability log log( 1 (cid:15) (cid:107) X ut − X vt (cid:107) )
· O ( q − d ) =
O ( loglog T T) .
So, the total probability of failure, either due to contravening the distance criterion or by probabilistic failure, is at most O ( T − 2 ) .
Hence with probability 1 − O (1 /T 2 ) we can use Lemma F. 1 and prove that
E [Γ t +1 ] ≤ ( 40 r λ 2 + 80 r 2 λ 22 ) nη 2 H 2 M 2 .
What is left is to union bound over T steps (Note that we do not need to union bound over all pairs of u and v , since can assume that u and v are the ones which interact at step t + 1 ), and we get that with probability 1 − O (1 /T )
= 1 − O (1 /n 4 ) the quantization algorithm never fails and the distance criterion is always satisﬁed.
The total number of bits used per step is O
( d log q )
=
O ( d + log T ) .
With this we can state the main theorem:
Theorem G.2.
Let f be an non-convex, L -smooth function, whose stochastic gradients satisfy the bounded second moment assumption above.
Consider the quantized version of the algorithm 1.
Let the number of local stochastic gradient steps performed by each agent upon interaction be a geometric random variable with mean H .
Let the learning rate we use be η = n/ √ T .
Deﬁne µ t = (cid:80) ni
=
X it /n , where X it is a value of model i after t interactions, be the average of the local parameters.
Then, for learning rate η = n/ √ T and any number of interactions T ≥ n 4 , with probability at least 1 − O (1 /n 4 ) we have that:
[Equation]
and additionally we use O ( d + log T ) communication bits per step.
H F IXED NUMBER OF LOCAL STEPS WITH V ARIANCE B OUND AND NON - IDENTICALLY DISTRIBUTED DATA
We again deal with a non-convex L-smooth function, but we no longer require a second moment bound, and no longer assume that data is distributed identically.
We use a constant learning rate η and ﬁxed local steps sizes H it
=
H , for each node i and step t .
Each agent i has access to local function f
i such that: For each agent i , the gradient ∇ f i ( x ) is L -Lipschitz continuous for some L > 0 , i.e. for all x, y ∈ R d : [Equation] for every x ∈ R d :
[Equation]
For each agent i and x ∈ R d :
[Equation]
For each agent i and x ∈ R d there exist σ 2 such that:
[Equation]
For each x ∈ R d there exist σ 2 such that:
[Equation]
Notice that since data is not distributed identically, for 1 ≤ q ≤ H , we no longer have that
[Equation]
Instead,
[Equation]
We proceed by proving the following lemma which upper bounds the expected change in potential:
Lemma H.1.
For any time step t , we have:
[Equation]
Proof.
First we bound change in potential ∆ t =
Γ t +1 − Γ t for some ﬁxed time step t > 0 .
For this, let ∆ i,jt be the change in potential when we choose agents ( i, j ) ∈ E for interaction.
We have that:
[Equation]
This gives us that:
[Equation]
For k (cid:54) = i, j we get that
[Equation]
Hence
[Equation]
Observe that:
[Equation]
After combining the above two equations
[Equation]
we get that:
[Equation]
This gives us:
[Table]
(cid:101) h
i ( X it ) (cid:107)
[Equation]
+ (cid:88) ( i,j )
[Equation]
(cid:13)(cid:13)(cid:13) X it
[Equation]
t (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
jt
[Equation]
t (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
Next, we use deﬁnition of Γ t :
[Equation] [Equation]
Next, we use the above inequality to upper bound ∆ t in expectation:
[Equation]
[Equation]
Finally
[Equation]
we remove the conditioning:
[Equation]
n
[Equation]
By considering the deﬁnition of ∆ t
[Equation]
we get the proof of the lemma.
Lemma H.2.
For any 1 ≤ q ≤ H and step t , we have that
[Equation]
Proof.
n
[Equation]
Lemma H.3.
For any 1 ≤ q ≤ H and step T , we have that n (cid:88) i =
[Equation]
it
[Equation]
2 ≤ nσ 2 + 4 nρ+ 16 L 2 E [Γ t ] + n (cid:88) i = {+
+} L 2 η 2 E (cid:107) q − 1 (cid:88) s
= (cid:101) h si ( X it ) (cid:107) 2 + 4 n E (cid:107) n (cid:88)
i = h qi ( X it ) /n
(cid:107) 2 .
Proof
[Table]
(cid:88) i =
[Equation] [Equation] + n (cid:88) i = E (cid:107)∇ f i (
[Equation]
it − q − 1 (cid:88) s = η (cid:101) h si ( X it )) (cid:107)(28) ≤
[Equation] + n (cid:88) i = [Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] [Equation] (cid:101)
[Equation]
i ( X it )) −
[Equation]
i ( µ [Equation] [Equation] f i ( µ t ) − ∇ f ( µ t ) + n (cid:88)
j = ∇ f j ( µ t ) /n − n (cid:88) j = [Equation] f
[Equation]
[Equation]
jt − q [Equation] 1 (cid:88) s = η (cid:101)
h s j ( X jt )) /n
+ n (cid:88)
j = ∇ f j ( X jt − q − 1 (cid:88) s =
η (cid:101)
h s j ( X jt )) /n
(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)
Cauchy − Schwarz ≤ nσ 2 + n (cid:88) i = 4 E (cid:107)∇ f i ( µ t ) − h qi ( X it ) (cid:107) 2 + 4 n E (cid:107) n (cid:88) i
= ( ∇ f i ( µ t ) − h qi ( X it )) /n
(cid:107)+ 4 n E (cid:107) n (cid:88) i = h qi ( X it ) /n
(cid:107) 2 +n (cid:88) i
=
(cid:107)∇ f i ( µ t ) − ∇ f ( µ t ) (cid:107)
Cauchy − Schwarz, (31) ≤ nσ 2 + 4 nρ 2 + n (cid:88) i = 8 E (cid:107)∇ f i ( µ t ) − h qi ( X it ) (cid:107) 2 + 4 n E (cid:107) n (cid:88) i
= h qi ( X it ) /n
(cid:107)
Lemma H.2 ≤ nσ 2 + 4 nρ 2 + 16 L 2 E [Γ t ] + n (cid:88) i = L 2 η 2 E (cid:107) q − 1 (cid:88) s
= (cid:101) h si ( X it ) (cid:107) 2 + 4 n E (cid:107) n (cid:88) i = h qi ( X it ) /n (cid:107) 2 .
Next we use the above lemma to show the upper bound for (cid:80)
Hq =1 (cid:80) ni =1 E (cid:107) (cid:101)
h q i ( X it ) (cid:107) 2 :
Lemma H
[Equation]
For η ≤ 1 6 LH , we have that :
[Equation]
Proof.
Notice that if η ≤ 1 6 LH the Lemma H.3 gives us that :
[Equation]
For 0 ≤ q ≤ H [Equation] let
[Equation]
Observe that the inequality 33 can be rewritten as:
[Equation]
which is the same as
[Equation]
By unrolling the recursion we get that
[Equation]
Since,
[Equation]
we have that
[Equation]
Next we derive the upper bound for (cid:80)
Tt =
E [Γ t ] :
Lemma H.5.
For η ≤[-
-]
[Equation]
/λ
[Equation]
8 r 2 /λ 22 , we have that :
[Equation]
Proof.
By Lemma H.1 we get that:
[Equation]
Notice that for η ≤ {+
+}
[Equation]
√
[Equation] λ [Equation] 8 r 2 /λ 2 2 we can rewrite the above inequality as
[Equation]
since (cid:80) ∞
[Equation]
2 we get that:
[Equation]
Now, we are ready to prove the following theorem:
Theorem 4.2.
Let f be an non-convex, L-smooth function whose minimum x (cid:63) we are trying to ﬁnd via the SwarmSGD procedure given in Algorithm 1.
Let local functions of agents satisfy conditions (27), (28), (29), (30) and (31).
Let H be the number of local stochastic gradient steps performed by each agent upon interaction.
Deﬁne µ t = (cid:80) ni
=
X it /n , where X it is a value of model i after t interactions.
For learning rate η = n √ T and T ≥ 57600 n 4 H 2 max(1 , L 2 )( r 2 λ 22 + 1) 2 we have that:
[Equation]
Proof.
Let E t denote expectation conditioned on { X t 1 , X t 2 , ..., X tn } .
By L -smoothness we have that
[Equation] [Equation]
First we look at E t [ µ t +1 − µ t ] .
If agents i and j interact (which happens with probability 1 rn/ 2 ), we have that µ t +1 − µ t = − ηn (cid:101)
h i ( X it ) − ηn (cid:101) h
j ( X jt ) .
Hence we get:
[Equation]
Using the above inequality we have:
[Equation]
Here we used Cauchy-Schwarz inequality at the last step.
Next we look at E t (cid:107) µ t +1 − µ t (cid:107) 2 .
If agents i and j interact, (which happens with probability 1 rn/ 2 ).
We have that µ t +1 − µ t = − ηn (cid:101) h
i ( X it ) − ηn (cid:101) h
j ( X jt ) .
Hence we get that
[Equation]
So, we can rewrite (35) as:
[Equation]
Next, we remove conditioning:
[Equation]
[Table]
[Equation]
Next we choose η ≤ 19 L and η ≤ n 80 LH , so that 16 L 2 η 3 n ≤ η 5 n and 16 LHη 2 n≤ η 5 n .
This together with the above inequalities allows us to derive the following upper bound for E [ f ( µ t +1 )]
(we eliminate terms with positive multiplicative factor E (cid:107) (cid:80) ni
=1 h qi ( X it ) /n (cid:107) 2 ):
[Equation]
q
[Equation]
i =
[Equation]
We proceed by summing up the above inequality for 0 ≤ t ≤ T − 1 :
[Equation]
Next we use Lemma H.5:
[Equation]
X
[Equation]
By choosing
[Equation]
and
[Equation]
we can eliminate terms with the multiplicative factor (cid:80)
Hq =1 E (cid:107) [Equation]
ni =1 h qi ( X it ) /n
(cid:107) 2 in the inequality (36):
[Equation]
After rearranging terms and dividing by ηTHn we get that
[Equation]
8 r 2 /λ 22 ) + 512 3 H 3 η 3 ( σ 2 + 4 ρ 2 ) n (2 r/λ 2 + 8 r 2 /λ 22 ) . Next [Table]
[Equation]
Recall that η = n
[Table]
T to get:
[Equation]
Notice that all assumptions and upper bounds on η are satisﬁed if
[Equation]
which is true
[-=
-]
[Equation]
[-i ( X it ) (cid:107) 
-]
[-g i ( µ t ) (cid:107) 
-]
I A DDITIONAL E XPERIMENTAL R ESULTS
We validated our analysis, by applying the algorithm to training deep neural networks for image classiﬁcation and machine translation.
Target System and Implementation.
We run SwarmSGD on the CSCS Piz Daint supercomputer, which is composed of Cray XC50 nodes, each with a Xeon E5-2690v3 CPU and an NVIDIA Tesla
P100 GPU, using a state-of-the-art Aries interconnect.
Please see (Piz) for hardware details.
We implemented SwarmSGD in Pytorch and TensorFlow using NCCL/MPI respectively.
Basically, each node implements a computation thread, and a communication thread, each of which stores a copy of the model.
The “live” copy, which is being updated with gradients, is stored by the [Figure] computation thread.
Periodically, the threads synchronize their two models.
When interacting, the two nodes exchange model information via their communication threads.
Our implementation closely follows the non-blocking Swarm algorithm description.
We used SwarmSGD to train ResNets on the classic CIFAR-10/ImageNet datasets, and a Transformer Vaswani et al.
(2017) on the WMT17 dataset (English-Germa).
The code will be made available upon publication.
Hyperparameters.
The only additional hyperparameter is the total number of epochs we execute for.
Once we have ﬁxed the number of epochs, we do not alter the other training hyperparameters : in particular, the learning rate schedule, momentum and weight decay terms are identical to sequential SGD, for each individual model.
Practically, if sequential SGD trains ResNet18 in 90 epochs, decreasing the learning rate at 30 and 60 epochs, then SwarmSGD with 32 nodes and multiplierwould 90 ∗ 2 / 32 (cid:39) 5 .
6 epochs per node, decreasing the learning rate at 2 and 4 epochs.
Speciﬁcally, for the ImageNet experiments, we used the following hyper-parameters.
For ResNet and ResNet50, we ran for 240 total parallel epochs using 32 parallel nodes.
The ﬁrst communicated every 3 local steps, whereas the second communicated every 2 local steps.
We used the same hyperparameters (initial learning rate 0.1, annealed at 1/3 and 2/3 through training, and standard weightdecay and momentum parameters).
For the WMT17 experiments, we ran a standard Transformer-large model, and executed forglobal epochs at 16, 32, and 64 nodes.
We ran a version with multiplier 1 (i.e. 10 / NUM NODES epochs per model) and one with multiplier 1 .
5 (i.e. 15 / NUM NODES epochs per model) and registered the BLEU score for each.
Baselines.
We consider the following baselines:
• Data-parallel SGD: Here, we consider both the small-batch (strong scaling) version, which executes a global batch size of 256 on ImageNet/CIFAR experiments, and the largebatch (weak-scaling) baseline, which maximizes the batch per GPU.
For the latter version, the learning rate is tuned following Goyal et al.
• Local SGD: Stich (2018); Lin et al.
We follow the implementation of Lin et al.
(2018), communicating globally every 5 SGD steps (which was the highest setting which provided good accuracy on the WMT task).
• Previous decentralized proposals: We experimented also with D-PSGD Lian et al.
AD-PSGD Lian et al.
(2018), and SGP Assran et al.
Due to computational constraints, we did not always measure their end-to-end accuracy.
Our method matches the sequential / large-batch accuracy for the models we consider within 1% .
We note that the best performing alternative (AD-PSGD) is known to drop accuracy relative to the baselines, e.g. (Assran et al., 2018).
Results.
The accuracy results for ImageNet experiments are given in Table 1 and Figures 3(a) and 3(b).
As is standard, we follow Top-1 validation accuracy versus number of steps.
Communication cost.
We now look deeper into SwarmSGD’s performance.
For this, we examine in Figure 4 the average time per batch of different methods when executed on our testbed.
The base value on the y axis (0.4s) is exactly the average time per batch , which is the same across all methods.
Thus, the extra values on the y axis equate roughly to the communication cost of each algorithm.
The results suggest that the communication cost can be up to twice the batch cost (SGP and D-PSGD).
Moreover, this cost is increasing when considered relative to the number of workers (X axis), for all methods except SwarmSGD.
This reduced cost is justiﬁed simply because our method reduces communication frequency: it communicates less often, and therefore the average cost of communication at a step is lower.
We can therefore conclude that our method is scalable , in the sense that its communication cost remains constant relative to the total size of the system.
Figure 3(b) shows the convergence versus time for
ResNet18 on the ImageNet dataset, at 32 nodes, with 3 local steps per node, and ∼ 7 epochs per model.
Convergence versus Steps and Epochs.
Figure 8 shows and discusses the results of additional ablation studies with respect to the number of nodes/processes and number of local steps / total epochs on the CIFAR-10 dataset / ResNet20 model.
In brief, the results show that the method still preserves convergence even at very high node counts (256), and suggest a strong correlation between accuracy and the number of epochs executed per model.
The number of local steps executed also impacts accuracy, but to a much lesser degree.
[Figure]
[Figure]
Quantization.
Finally, we show convergence and speedup for a WideResNet-28 model with width factor 2, trained on the CIFAR-10 dataset.
We note that the epoch multiplier factor in this setup is 1 , i [Figure] e. Swarm (and its quantized variant) execute exactly the same number of epochs as the baseline.
Notice that the quantized variant provides approximately 10% speedup in this case, for a < 0 .
3% drop in Top-1 accuracy.
[Figure]