I NT SGD: A DAPTIVE F LOATLESS C OMPRESSION OF S TOCHASTIC G RADIENTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a family of adaptive integer compression operators for distributed
Stochastic Gradient Descent ( SGD ) that do not communicate a single ﬂoat.
This is achieved by multiplying ﬂoating-point vectors with a number known to every device and then rounding to integers.
In contrast to the prior work on integer compression for SwitchML by Sapio et al. (2021), our IntSGD method is provably convergent and computationally cheaper as it estimates the scaling of vectors adaptively.
Our theory shows that the iteration complexity of IntSGD matches that of SGD up to constant factors for both convex and non-convex, smooth and nonsmooth functions, with and without overparameterization.
Moreover, our algorithm can also be tailored for the popular all-reduce primitive and shows promising empirical performance.
I NTRODUCTION
Many recent breakthroughs in machine learning were made possible due to the introduction of large, sophisticated and high capacity supervised models whose training requires days or even weeks of computation (Hinton et al., 2015; He et al., 2016; Huang et al., 2017; Devlin et al., 2018).
However, it would not be possible to train them without corresponding advances in parallel and distributed algorithms capable of taking advantage of modern hardware.
Very large models are typically trained on vast collections of training data stored in a distributed fashion across a number of compute nodes that need to communicate throughout the training process.
In this scenario, reliance on efﬁcient communication protocols is of utmost importance.
Communication in distributed systems.
The training process of large models relies on fast synchronization of gradients computed in a parallel fashion.
Formally, to train a model, we want to solve the problem of parallel/distributed minimization of the average of n functions:
[Equation]
where we will compute the gradients of stochastic realizations
f i ( x ; ξ ) .
The two dominating protocols for gradient synchronization are all-reduce and all-gather aggregation, which may use either
Parameter Server or all-to-all communication under the hood.
The core difference between them lies in that all-gather communicates all vectors, whereas all-reduce only outputs their average.
As shown in previous works, current distributed deep learning algorithms predominantly use all-reduce as it scales much better than all-gather (Vogels et al., 2019; Agarwal et al., 2021).
A popular way to reduce the communication cost of both all-reduce and all-gather primitives is to use lossy compression of gradients (Ramesh et al., 2021).
To study the beneﬁt of lossy compression, large swaths of recent literature on distributed training attribute the cost of sending a single vector from a worker to the server to the number of bits needed to represent it.
Based on this abstraction, various elaborate vector compression techniques (see Table 1 in (Beznosikov et al., 2020; Xu et al., 2020; Safaryan et al., 2020)) and algorithms have been designed for higher and higher compression ratios.
However, in real systems, the efﬁciency of sending a vector is not fully characterized by the number of bits alone, because:
• First, many compressors with high compression ratio (e
g., natural compression (Horváth et al., 2019), quantization (Alistarh et al., 2017), top- k sparsiﬁcation, sign (Bernstein et al.,
2018)) are not compatible with the efﬁcient all-reduce primitive and require all-gather implementation.
Secondly, some compressors rely on expensive operations such as low-rank decomposition (Wang et al., 2018; Vogels et al., 2019) or bit-level operations (Horváth et al., 2019), whose computation overhead may outweigh the beneﬁts of reduced communication load.
• Thirdly, algorithms with biased compressors such as Top- k SGD , SignSGD , PowerSGD (Vogels et al., 2019), require the error-feedback ( EF-SGD ) mechanism (Stich et al., 2018;
Karimireddy et al., 2019) to ensure the convergence.
Alas, error feedback needs extra sequences that may not ﬁt the low memory budget of GPUs.
Moreover, to the best of our knowledge, no convergence guarantee has been established for EF-SGD on the non-smooth objectives with multiple workers.
SwitchML.
Another approach to combating long communication times is to improve the hardware itself.
The recently proposed SwitchML is an alternative to the NVIDIA Collective Communications Library (NCCL) on real-world hardware (Sapio et al., 2021).
The ﬁrst key component of SwitchML is the in-network aggregation (INA) by a programmable switch.
INA reduces the communication cost and latency because the execution can be paralleled and pipelined.
To be speciﬁc, it splits the vector to aggregate into chunks and processes them individually by the switch pipeline.
The advantages of INA over parameter server and all-reduce in terms of latency and communication cost have been theoretically and empirically justiﬁed by Sapio et al.
The second key component of SwitchML is stochastic gradient descent with integer rounding and aggregation.
Instead of reducing the data volume to exchange, the goal of integer rounding in SwitchML is to ﬁt the limited computation capability of the modern programmable switch, which only supports integer additions or logic operations.
To increase the rounding precision, the gradient g ki on device i multiplies a positive scaling factor α k known to every worker and then rounded to an integer number
I nt ( α k ◦ g ki ) .
As there is no additional scaling or decompression before aggregating the communicated vectors, their sums can be computed on the ﬂy.
Then, each worker can divide the aggregated gradient by nα k to update the model.
However, Sapio et al.
(2021) remark that the choice of the scaling factor α k requires special care.
In their presentation 1 , one of the authors notes: “A bad choice of scaling factor can reduce the performance.”
To this end, they propose a heuristic-based proﬁling step that is executed before the gradient aggregation and keeps the rounded integers small to ﬁt in 32 bits.
We refer to their algorithm including the proﬁling step as Heuristic IntSGD .
Unfortunately, no convergence guarantee for that algorithm has been established.
This is where our theory comes to the rescue.
By rigorously and exhaustively analyzing integer rounding based on scaling, we ﬁnd adaptive rules for the scaling factor α k that do not require the proﬁling employed by Sapio et al.
As we will show in the remainder of the paper, our algorithm is perfectly suited for both in-network aggregation (INA) of SwitchML and for other efﬁcient primitives such as all-reduce.
1. C ONTRIBUTIONS
We summarize the key differences of our algorithm and prior work in Table 1, and we also list our main contributions below.
• Adaptive IntSGD.
We develop a family of computationally cheap adaptive scaling factors for provably convergent IntSGD .
It is a better alternative to the Heuristic IntSGD in (Sapio et al., 2021) that requires expensive operations and does not ensure convergence.
• Rates.
We obtain the ﬁrst analysis of the integer rounding and aggregation for distributed machine learning.
For all of the proposed variants, we prove convergence rates of IntSGD that match those of full-precision SGD up to constant factors.
Our results are tight and apply to both convex and nonconvex problems.
Our analysis does not require any extra assumption compared to those typically invoked for SGD .
In contrast to other compression-based methods, IntSGD has the same rate as that of full-precision SGD even on non-smooth problems.
• IntDIANA
We observe empirically that IntSGD struggles when the devices have heterogeneous (non-identical) data—an issue it shares with vanilla SGD —and propose an alternative method, IntDI-
[Table]
ANA , that can provably alleviate this issue.
We also show that our tools are useful for extending the methodology beyond SGD methods, for example, to variance reduced methods (Johnson & Zhang, 2013; Allen-Zhu & Hazan, 2016; Kovalev et al., 2020; Gower et al., 2020) with integer rounding.
Please refer to Appendix A.2 for theoretical results and Appendix C.5 for the empirical veriﬁcation.
A DAPTIVE I NTEGER R OUNDING AND I NT SGD
By randomized integer rounding we mean the mapping I nt : R → Z deﬁned by
[Equation]
where [ t ] denotes the ﬂoor of t ∈ R , i.e., [ t ] =
k ∈ Z , where k is such that k ≤ t < k + 1 .
Note that
[Equation]
We extend this mapping to vectors x ∈ R d by applying in element-wise: I nt ( x ) i def
=
I nt ( x i ) .
2. A DAPTIVE INTEGER ROUNDING
Given a scaling vector α ∈ R d with nonzero entries, we further deﬁne the adaptive integer rounding operator Q : R d → R d by [Equation]
[Equation]
where a ◦ b def = ( a 1 b 1 , . .
. , a d b d ) ∈ R d denotes the Hadamard product of two vectors a = ( a 1 , . .
a d ) ∈ R d and b = ( b 1 , . . , b d ) ∈ R d .
As we show below, the adaptive integer rounding operator (2) has several properties which will be useful in our analysis.
In particular, the operator is unbiased, and its variance can be controlled by choice of a possibly random scaling vector
α ∈ R d
++ .
Lemma 1.
For any x ∈ R d and α ∈ R d ++ , we have
[Equation]
1: Params: Stepsizes η k , scaling vectors α k ∈ R d 2: Init: x 0 ∈ R d , x 1 = x 0 − η 0 1 n (cid:80) ni
=1 g 0 i 3: for k = 1 , 2 , . . .
do 4: for each device i = 1 , 2 , . . .
, n do 5:
Compute stochastic gradient g ki ( E (cid:2)
g ki | x k (cid:3) ∈
∂f i ( x k ) ) 6:
Maintain the moving average: r k = βr k − 1 + (1 − β ) (cid:107) x k − x k − 1 (cid:107){ r k is a scalar} 7:
Compute the adaptive scaling factor: α k = √ d √ 2 nr k /η 2 k + ε8:
Scale and round the local gradient Q ( g ki )
=
I nt ( α k ◦ g ki ) 9: end for 10:
Aggregate Q ( g ki ) by either all-reduce or in-network aggregation (INA) 11: for each device i = 1 , 2 , . .
. , n do 12:
Compute the (sub)gradient estimator: ˜ g k =
nα k (cid:80) ni
=1 Q ( g ki ) 13:
Update the model parameter x k +
= x
k − η k ˜
g
k 14: end for 15: end for
The expectations above are taken with respect to the randomness inherent in the rounding operator.
2. N EW ALGORITHM : I NT SGD
We are ready to present our algorithm, IntSGD .
At iteration k , each device i computes a stochastic (sub)gradient vector g ki , i.e., a vector satisfying
E (cid:2) g ki | x k (cid:3) ∈
∂f i ( x k ) .
Prior to communication, each worker i rescales its stochastic (sub)gradients g ki using the same vector α k ∈ R d ++ , and applies the randomized rounding operator I nt .
The resulting vectors I nt ( α k ◦ g ki ) are aggregated to obtain (cid:80) ni
=
I nt ( α k ◦ g ki ) , which is also an integer.
Each device subsequently performs division by n and inverse scaling to decode the message, obtaining the vector
[Equation]
Here α k is a random adaptive scaling factor calculated based on the historical information.
We left the design of α k to Section 4.
By combining (5) and (3), we observe that g k is a stochastic (sub)gradient of f at x k .
Finally, all devices perform in parallel an SGD -type step of the form x k +
= x
k − η k ˜ g k and the process is repeated.
Our IntSGD method is formally stated as Algorithmwith the suggested rule of α .
Relation to QSGD (Alistarh et al., 2017).
QSGD bears some similarity to the IntSGD :
Both of them scale g ki by a factor before the quantization (the scaling factor in QSGD is 1 / (cid:107) g ki (cid:107) for normalization).
However, some key difference makes the communication efﬁciency of IntSGD much better than that of QSGD .
It is worth noting that the normalization factors 1 / (cid:107) g ki (cid:107) in QSGD are different for various workers.
Then, the quantized values of various workers need to be gathered and decompressed before aggregation.
On the contrary, the scaling factor α k in our IntSGD is the same for all workers such that the sum of integers can be computed on the ﬂy.
Thus, IntSGD supports the efﬁcient allreduce primitive while QSGD does not.
As seen in the experimental results in Section 5 , this makes a big difference in empirical performance.
Moreover, the proof technique for IntSGD is also intrinsically different from that of QSGD .
Please see the next section for the details.
A NALYSIS OF I NT SGD
To establish convergence of IntSGD , we introduce the following assumption on the scaling vector α k
= ( α k, 1 , . .
. , α k,d ) (cid:62) ∈ R d
++ .
Assumption 1.
There exists β ∈ [0 , 1) and a sufﬁciently small ε > 0 such that d
[Equation]
is bounded above by η 2 k ε 2 + 2 n (1 − β ) k − 1 (cid:80)
[Equation]
While this assumption may look exotic, it captures precisely what we need to establish the convergence of IntSGD , and it holds for several practical choices of α k , including the one shown in Section 4 and more choices in Appendix A.1.
Challenges in IntSGD analysis.
Although the I nt operation is unbiased and has ﬁnite variance as shown in Lemma 1, we highlight that it is non-trivial to obtain the convergence analysis of IntSGD and the analysis is different from that of QSGD and similar methods.
Indeed, QSGD , Rank- k , and
NatSGD all use unbiased operators Q with variance satisfying E [ (cid:107)Q ( g ) − g (cid:107) 2 ] ≤ ω (cid:107) g (cid:107) 2 for some ω > 0 .
For them, the convergence theory is simply a plug-in of the analysis of Compressed SGD (Khirirat et al., 2018).
However, the integer rounding operator I nt does not satisfy this property, and the variance of the integer compressor will not decrease to zero when (cid:107) g (cid:107) 2 → 0 .
Moreover, to estimate α k adaptively, we use the values of past iterates, which makes its value itself random, so the analysis of Compressed SGD cannot apply.
As shown in our proofs, an extra trick is required: we reserve an additional term (cid:80) kt =
E [ (cid:107) x t +1 − x t (cid:107) 2 ] to control the variance of the rounding.
Furthermore, the analysis of moving-average estimation is particularly challenging since the value of α k is affected by all past model updates, starting from the very ﬁrst iteration.
3. N ON - SMOOTH ANALYSIS : GENERIC RESULT
Let us now show that IntSGD works well even on non-smooth functions.
Assumption 2.
Stochastic (sub)gradients g k 1 , . .
. , g kn sampled at iteration k satisfy the inequalities
[Equation]
where the former inequality corresponds to G -Lipschitzness of f and the latter to bounded variance of stochastic (sub)gradients.
Theorem 1.
Let functions f 1 , . .
. , f n be convex and Assumptions 1 and 2 be satisﬁed.
Then
[Equation]
where ˆ x k = 1 (cid:80) kt =
η t (cid:80) kt =
η t x t is a weighted average of iterates.
3. S MOOTH ANALYSIS : GENERIC RESULT
We now develop a theory for smooth objectives.
Assumption 3.
There exist constants L , σ ∗ ≥ 0 such that the stochastic gradients g k 1 , . . .
, g kn at iteration k satisfy E k
[ g ki ] = ∇ f i ( x k ) and
[Equation]
Assumption 3 is known as the expected smoothness assumption (Gower et al., 2019).
In its formulation, we divide the constant term σ 2 ∗ by n , which is justiﬁed by the following proposition.
Proposition 1 (Section 3.3 in (Gower et al., 2019)) .
Let f i ( x ) =
E ξ [ f i ( x ; ξ )] , g ki = ∇ f i ( x k ; ξ ki ) , and f i ( · ; ξ ) be convex and its gradient be L i -Lipschitz for any ξ .
Then, the second part of Assumption 3 is satisﬁed with σ 2 ∗ def = 2 n (cid:80) ni =1 E ξ (cid:2) (cid:107)∇ f i ( x ∗ ; ξ ) (cid:107) 2 (cid:3) and L def = 4 max i =1 ,...,n L i .
Gower et al.
(2019) state and prove this result in a more general form, so for the reader’s convenience, we provide a proof in the appendix.
Theorem 2.
Assume that f is convex and Assumption 3 holds.
If η k ≤ 12 L and ˆ x k =
[Equation]
t x t is a weighted average of iterates, then
[Equation]
Corollary 1 (Overparameterized regime) .
When the model is overparameterized (i.e., the losses can be minimized to optimality simultaneously: σ ∗ = 0 ), we can set ε = 0 and obtain O (cid:0) 1 k (cid:1) rate.
3. N ON - CONVEX ANALYSIS : GENERIC RESULT
We now develop a theory for non-convex objectives.
Assumption 4.
The gradient of f is L -Lipschitz and there exists f inf ∈ R such that f inf ≤ f ( x ) for all x .
Furthermore, for all i and k we have
[Equation] [Equation]
Our main result in the non-convex regime follows.
Theorem 3.
Let f be L -smooth and let Assumption 1 hold.
If η k ≤ 12 L for all k , then
[Equation]
where ˆ x k is sampled from { x 0 , . .
, x k } with probabilities proportional to η 0 , . . .
, η k .
E
XPLICIT COMPLEXITY RESULTS
Having developed generic complexity results for IntSGD in the non-smooth (Section 3.1), smooth (Section 3.2) and non-convex
(Section 3.3) regimes, we now derive explicit convergence rates.
Corollary 2.
For any sequence of scaling vectors α k satisfying Assumption 1, we recover the following complexities:
(i) if f 1 , . . .
, f n are convex, Assumption 2 is satisﬁed and η t = η =
[Equation] [Equation]
k
[Equation]
for
[Equation]
= 0 , . .
, k , then
[Equation]
(ii) if f is convex, Assumption 3 holds and η t = min (cid:110)
12 L
[Equation]
(cid:107) x
[Equation]
x ∗ (cid:107)√ n
[Equation]
k ( σ ∗ +
[Equation]
) (cid:111) , then
[Equation]
(iii) if f is non-convex, Assumption 4 holds and η t = min
[Equation]
then
[Equation]
Based on Corollary 2, our IntSGD has linear speed-up in case (ii) and (iii).
Comparison with error-feedback ( EF-SGD ).
Distributed SGD with biased compressors (like
PowerSGD , SignSGD , Top- k SGD ) requires the error-feedback modiﬁcation to converge.
In the non-convex and smooth case, EF-SGD leads to the O (cid:16) σ √ kn + (cid:0) Gk (cid:1) 2 / 3 (cid:17) rate in Koloskova et al.
(2019) when assuming the second moment of stochastic gradient is bounded by G 2 .
Compared to their result, our rate is never worse and does not require the second moment of stochastic gradient to be bounded, which is often violated in practice even for quadratic objectives and simplest neural networks.
The convergence guarantee of EF-SGD for the convex and non-smooth function (Assumption 2) is even weaker: Karimireddy et al.
(2019) show the O (cid:16) σ √ δk (cid:17) convergence rate of
EF-SGD only for the single-worker case ( n = 1 ), which is 1 √ δ -times worse than IntSGD ( δ could be fairly small, e.g., δ = 1 /d in Top- 1 compression).
To the best of our knowledge, there is no convergence guarantee of EF-SGD for the non-smooth function when there are multiple workers.
In contrast, our IntSGD has the same rate as SGD under the same set of assumptions.
D ESIGN OF S CALING F ACTORS 4. A DAPTIVE SCALING FACTOR WITH THE MOVING AVERAGE AND SAFEGUARD
We now present an effective rule of adaptive α k (presented in Algorithm 1) that satisﬁes Assumption 1 for the convergence rates listed in previous section.
In the appendix, we provide more options that also satisfy Assumption 1 and the proof still goes through.
For simplicity, we assume that the ﬁrst communication is exact, which allows us to estimate α k adaptively without worrying about α 0 .
Proposition 2.
Assumption 1 holds if we choose β ∈ [0 , 1) , ε ≥ 0 and α k = √ d √ {+2 nr k /η 
+}
k + ε 2 , where
[Equation]
Remark 1.
Here β ∈ [0 , 1) is a constant factor to control the moving average update of r k , which prevents the scaling factor α k from changing too rapidly.
ε 2 could be any sufﬁciently small number, which serves as a safeguard to avoid the potential “divide by zero” error.
We study the sensitivity of our IntSGD to β and ε in Appendix C.4.
4. C OMPRESSION EFFICIENCY
Let us now discuss the number of bits needed for the compressed vectors.
Although the main attraction of IntSGD is that it can perform efﬁcient in-network communication, we may also hope to gain from the smaller size of the updates.
Consider for simplicity the case where (cid:107) x k − x k − 1 (cid:107) ≈ (cid:107) η k g ki (cid:107) with some i .
The adaptive scheme with β = 0 , ε = 0 gives α k =
[Equation]
η
[Equation]
n (cid:107)
[Equation]
x k (cid:107) = √ d √ 2 n (cid:107) g ki (cid:107) , so that (cid:107) α k g ki (cid:107) ∞ = √
[Equation]
(cid:107) g ki (cid:107) ∞ cid:107) g ki (cid:107) ≤ √ d √ 2 n .
Since we only use signed integers, we need at most 1 + log 2 √ d √ 2 n bits for each coordinate.
For instance, for d ∼ 10 10 and n ∼ 100 , the upper bound is 1 + log 2 ( √ 5 · 10 7 ) <bits.
The situation becomes even better when (cid:107) g ki (cid:107) (cid:29) (cid:107) g ki (cid:107) ∞ , i.e., when the stochastic gradients are dense.
This property has been observed in certain empirical evaluations for deep neural networks; see for example the study in (Bernstein et al., 2018).
E XPERIMENTS 5. S ETUP
We empirically compare our IntSGD algorithm with several representative and strong baselines:
SGD , Heuristic IntSGD (Sapio et al., 2021), SGD , PowerSGD + Error-feedback (PowerSGD (EF))
(Vogels et al., 2019), NatSGD (Horváth et al., 2019), and QSGD (Alistarh et al., 2017).
The experiments are performed on 16 NVIDIA Tesla V100 GPUs located on 8 compute nodes of a cluster (
GPUs per node) following the PowerSGD paper.
The compute nodes in the cluster utilize InﬁniBand
HDR-100 Director Switch at 100Gbps speed for network connection.
The cluster also supports the
NVIDIA Collective Communications Library (NCCL).
We consider two tasks: image classiﬁcation by ResNet18 (He et al., 2016) on the CIFAR-10 dataset and language modeling by a 3-layer LSTM on the Wikitext-2 dataset.
The neural network architectures and hyperparameters are from some public PyTorch implementations 2 .
Our code is built on the codebase of PowerSGD 3 .
We also borrow their all-reduce-based implementations of SGD and PowerSGD .
It is worth noting that QSGD and NatSGD do not support all-reduce.
Thus, we implement their collective communications by all-gather.
The implementations for compression and decompression in QSGD and NatSGD are from the authors of NatSGD 4 .
For the sake of comparison, we also implement the all-gather-based SGD .
We report the results of 3 repetitions with varying seeds.
Apart from the IntSGD with randomized integer rounding ( IntSGD (Random) ) analyzed in our theory, we also consider the variant of IntSGD with deterministic integer rounding ( IntSGD (Determ.) ) which can use the PyTorch built-in function torch.round .
For all IntSGD variants, we clip the local stochastic gradients to ensure that each aggregated value ﬁts in either 8 bits or 32 bits.
For more details of the experimental setup, please refer to Appendix C.1.
5. I NT SGD VS . H EURISTIC I NT SGD
First, we compare our IntSGD with the most related algorithm Heuristic IntSGD (Sapio et al., 2021).
For both algorithms, we consider two potential communication data types: int8 and int32 .
Note that the rule of scaling factor in Heuristic IntSGD is α = 2 nb
[Figure]
1 n · 2 max_exp , where “nb” represents the number of bits to encode each coordinate and “max_exp” is the rounded exponent of the largest absolute value in the communicated package.
Although this scaling rule is straightforward and avoids overﬂows, it cannot guarantee convergence, even with int32 as the communication data type.
Indeed, the Heuristic IntSGD may fail to match the testing performance of full-precision SGD according to
Figure 1.
On the contrary, our IntSGD can perfectly match the performance of full-precision SGD on both image classiﬁcation and language modeling tasks, which is in accordance with our theory that IntSGD is provably convergent.
[Figure]
5. I NT SGD VS . OTHER BASELINES
We also compare our IntSGD algorithms to the other baselines including the all-gather-based SGD ,
QSGD , NatSGD and the all-reduce-based SGD , PowerSGD (EF) on the two tasks.
See the test performance and time breakdown in Table 2 and Table 3.
First, we can observe that the all-gather based SGD + compressors (e.g., QSGD , NatSGD ) are indeed faster than the all-gather based full-precision SGD , which shows the beneﬁt of lossy compressions.
However, they are even much slower than the all-reduce-based full-precision SGD .
Unfortunately,
QSGD and NatSGD ) does not support the more efﬁcient all-reduce primitive.
Similar observation can be seen in previous works (Vogels et al., 2019; Agarwal et al., 2021).
All of PowerSGD (EF) , IntSGD (Random) , and IntSGD (Determ.) are consistently faster than the allreduce-based full-precision SGD on both tasks.
Compared to IntSGD (Determ.) , IntSGD (Random) leads to slightly more computation overhead due to the randomized rounding.
However, IntSGD (Determ.) fails to match the testing performance of SGD on the language modeling task.
Compared to PowerSGD (EF) , our IntSGD variants are better on the task of training ResNet18 but inferior on
[Table]
the task of training a 3-layer LSTM.
Although IntSGD is not always better than PowerSGD (EF) , there are several scenarios where IntSGD is preferrable as explained in in Section 1 and Section 3.4.
In addition, as seen in Figure 3 of the Appendix C.
[Table]
PowerSGD (EF) converges much slower than
SGD and IntSGD in the ﬁrst 150 epochs of the ResNet training (which has non-smooth activations).
[Table]
C ONCLUSION
In this paper
[Table]
we propose the provably convergent and computationally cheap IntSGD algorithm for efﬁcient distributed machine learning.
The core component of IntSGD is the adaptively estimated scaling factor shared by all users [Table] which makes it compatible with the widely used communication primitive all-reduce and the recently proposed in-network aggregation (INA) (Sapio et al., 2021)
[Table]
The convergence rates of IntSGD match that of SGD up to constant factors on a broad spectrum of problems.
Experimental results on two deep learning tasks show its promising empirical performance.
Reproducibility statement.
Regarding the theoretical results: We describe the mathematical setting and algorithms in Section 1, 2, and Appendix A; Assumptions and the main theoretical results are presented in Section 3; We provide the complete proof for those results in Appendix B.
Regarding the experimental results: We report the number of repetitions, the computing infrastructure used, the range of hyper-parameters considered, and the evaluation metrics in Section 5 and Appendix C.1;
We attach our code in the supplementary material.
Appendix A O THER V ARIANTS OF I NT SGD A. O THER CHOICES OF SCALING FACTOR α k
In Section 4.1, we provide an effective scaling factor with the moving average and the safeguard.
However, there are other choices of scaling factor that also satisfy Assumption 1, and the convergence proof still goes through.
Proposition 3 (Adaptive α k ) .
If we choose
[Equation]
then Assumption 1 holds with ε = 0 and β = 0 .
One can also consider applying an integer quantization with individual values of α t for each coordinate or block, for instance, with an α t,l corresponding to the l -th layer in a neural network.
It is straightforward to see that this modiﬁcation leads to the error (cid:80)
[Equation]
, where B is the total number of blocks and d l is the dimension of the l -th block.
Proposition 4 (Adaptive block α k ) .
Assume we are given a partition of all coordinates into B ≤ d blocks with dimensions d 1 , . . .
, d B , and denote by ( x k ) l the l -th block of coordinates of x k .
Then
Assumption 1 holds with
[Equation]
There are two extreme cases in terms of how we can choose the blocks.
One extreme is to set
B = 1 , in which case we have a single scalar for the whole vector.
The other extreme is to use
B = d , which means that α k = η [Equation] 2 √ n | x k − x k − 1 | , where the division and absolute values are computed coordinate-wise.
[Equation]
a partitioning of R d into B blocks of sizes d 1 , . .
, d B such that R d = R d 1 × · · · × R d B 2: for k = 1 , 2 , . . .
do 3: for each device i = 1 , 2 , . . .
, n do 4:
Compute independent stochastic gradients g ki ( E k
[ g ki ] ∈
∂f i ( x k ) ) 5:
Maintain the exponetial moving average: r k,l = βr k − 1 ,l +(1 − β ) (cid:107) ( x k ) l − ( x k − 1 ) l (cid:107) 2 {for each block l = 1 , . . .
, B } 6:
Compute the adaptive scaling factors: α k,l = η k √ d l (cid:113) 2 nr k,l + η 2 k dld ε7:
Scale and round the local gradient ( Q ( g ki ))
l
= I nt
( α k,l ( g ki ) l ) 8: end for 9:
Aggregate Q ( g ki ) by either all-reduce or in-network aggregation (INA) 10: for each device i = 1 , 2 , . .
. , n do 11:
Compute the (sub)gradient estimator: (˜ g k ) l = {+1 nα k,l (cid:80) ni =
+}
( Q ( g ki ))
l 12: x k +1 =
x
k − η k ˜ g k 13: end for 14: end for
Compression efﬁciency of IntSGD with adaptive block quantization.
Our block-wise and coordinate-wise compression can further beneﬁt from reduced dimension factors in the upper bounds, leading to the estimate of log
√ d l 2 √ n bits for block with dimension d l .
However, for smaller blocks it is less likely to happen that (cid:107) ( x k ) l − ( x k − 1 ) l (cid:107)
≈ (cid:107) η
k
( g k i ) l (cid:107) , so the estimate should be taken
1: Params: Stepsizes η k , scaling vectors α k ∈ R d 2: Init: x 0 ∈ R d , x 1 = x 0 − η 0 1 n (cid:80) ni
=1 g 0 i , h 1 i = 0 , h 1 =3: for k = 1 , 2 , . . .
do 4: for each device i = 1 , 2 , . . .
, n do 5:
Compute stochastic gradient g ki ( E (cid:2)
g ki | x k (cid:3) ∈
∂f i ( x k ) ).
Compute the adaptive scaling factor: α k = η k √ d √ 2 n (cid:107) x k − x k − 1 (cid:107) 7:
Scale and round the local gradient Q ( g ki )
=
I nt ( α k ◦ ( g ki − h ki )) 8:
Update the local shift
h k
+1 i
= h ki + Q ( g ki ) 9: end for 10:
Aggregate Q ( g ki ) by either all-reduce or in-network aggregation (INA) 11: for each device i = 1 , 2 , . . . , n do 12:
Compute the (sub)gradient estimator: ˜ g k = h k
+ 1 nα k (cid:80) ni
=1 Q ( g ki ) 13:
Update the model parameter x k +
= x
k − η k ˜
g
k 14:
Update global shift
h k +
=
h k
+ 1 nα k (cid:80) ni
=1 Q ( g ki ) 15: end for 16: end for with a grain of salt.
We hypothesize that using ε as in Proposition 2 is required to make block compression robust.
Notice that if stochastic gradients have bounded coordinates, i.e., (cid:107) g ki (cid:107) ∞
≤
G ∞ for all i, k , then we would need at most 1 + log 2 √ dG ∞ ε bits to encode the integers.
Since any ε ≤ σ + √ nG does not change the rate in the non-smooth case (see Equation (9)), we get for free the upper bound of 1 + log 2 √ dG ∞ √ nG bits.
A. H ANDLING HETEROGENEOUS DATA
IntSGD can be equipped with the full gradient or variance-reduced gradient estimator to enjoy faster convergence than O (cid:16) 1 √ k (cid:17) shown in Corollary 2.
For example, if we plug σ ∗ =
(no variance) and ε ≤ √ n √ k
(sufﬁciently small safeguard) into item 2 of Corollary 2, the convergence rate of IntSGD is O (cid:0) 1 k (cid:1) .
However, when the data is heterogeneous (i.e., minimizing f ( x ) = 1 n (cid:80) ni
=
f i ( x ) will not make (cid:107)∇ f i ( x ∗ ) (cid:107) = 0 , ∀ i ∈ [ n ] ), the transmitted integer of IntSGD with σ ∗ = 0 , ε = 0 can be gigantically large, which leads to very inefﬁcient communications or even exception value error.
E.g., if we choose the adaptive α k and the full gradient g ki = ∇ f i ( x k ) , the largest integer to transmit from worker i to the master is (cid:107) α k ∇ f i ( x k ) (cid:107) ∞
≈ (cid:107)∇ f i ( x k ) (cid:107) ∞ (cid:107) x k − x k − 1 (cid:107) , where the denominator is 0 while the numerator is nonzero as the iterate converges to the optimum.
To alleviate this issue, one needs to compress gradient differences as is done for example by Mishchenko et al.
in their DIANA method.
By marrying IntSGD with the DIANA trick, we obtain IntDIANA (Algorithm 3).
For IntDIANA with adaptive α k , the largest transmitted integer from worker to the master is (cid:107) α k
( g ki − h ki ) (cid:107) ∞
≈ (cid:107) g ki − h ki (cid:107) ∞ (cid:107) x k − x k − 1 (cid:107) .
We will show that both the nominator and the denominator are inﬁnitesimal when x k converges to the optimum, such that the issue mentioned above can hopefully be solved.
Note that we can either use the full gradient g ki = ∇ f i ( x k ) or the L-SVRG estimator Kovalev et al. [Equation]
[Equation]
on the i -th worker.
For the variance-reduced method, we further assume that f i has a ﬁnite-sum structure, i.e.,
[Equation]
such that ∇ f i ( x ; ξ ) = ∇ f il ( x ) , E [ ∇ f i ( x ;
[Equation]
=
[Equation]
m (cid:80) ml (cid:48)
[Equation]
1 ∇ f il (cid:48) ( x ) and l is sampled from [ m ] uniformly at random by ξ .
Our main convergence theorem describing the behavior of IntDIANA follows:
Theorem 4.
Assume that f is µ -strongly convex ( µ ≥ 0 ) and f ( · ; ξ ) has L i -Lipschitz gradient for any ξ , L def = 4 max i L i .
If µ > 0 , the iterates of IntDIANA with adaptive α k =
[Equation]
• For IntDIANA with the GD estimator g ki = ∇ f i ( x k ) , we have θ def = max (cid:8) 1 − ηµ, 34 (cid:9) < 1 and Ψ k
def = (cid:107) x k − x ∗ (cid:107)
+ (cid:107) x
k − x k − 1 (cid:107) 2 + η 2 L 2 4 n 2 n (cid:80) i =1 (cid:107) h ki − ∇ f i ( x ∗ ) (cid:107) 2 , where η k = η ≤ 1 2( L + L 32 n ) ; • For IntDIANA with the L-SVRG estimator, we have θ def = max (cid:8) 1 − ηµ, 34 , 1 − 38 m (cid:9) <and Ψ k
def = (cid:107) x
k − x ∗ (cid:107)
+ (cid:107) x
k − x k − 1 (cid:107) 2 + 8 η 2 n 2 (cid:80) ni =1 (cid:80) ml
=1 (cid:107)∇ f il ( w ki ) − ∇ f il ( x ∗ ) (cid:107) 2 + η 2 L 2 4 n 2 (cid:80) ni
=1 (cid:107)
h
ki − ∇ f i ( x ∗ ) (cid:107) 2 , where p = 1 m , and η k = η ≤ [Equation] [Equation] L /n
[Equation]
If µ = 0 , the iterates of IntDIANA with adaptive α k =
[Equation]
satisfy
[Equation]
where ˆ x k = 1 k +1 (cid:80) ki
=
x i .
• IntDIANA with the GD estimator requires that η k = η ≤
[Equation]
n
[Equation]
, • IntDIANA with the L-SVRG estimator requires that η k = η ≤
[Equation] [Equation]
L +2 L /n ) .
The above theorem establishes linear convergence of two versions of IntDIANA in the strongly convex regime and sublinear convergence in the convex regime.
Compression efﬁciency of IntDIANA.
If µ > 0 , for IntDIANA with adaptive α k and either GD or L-SVRG estimator, both (cid:107) h ki − ∇ f i ( x ∗ ) (cid:107) 2 and (cid:107) x k − x k − 1 (cid:107) 2 converge to 0 linearly at the same rate, while g ki → ∇ f i ( x ∗ ) .
Thus, the largest integer to transmit is (cid:107)
[Equation]
is hopefully upper bounded.
B P ROOFS B. P ROOFS FOR I NT SGD
In the section, we provide the complete convergence proof of IntSGD .
B.1. P ROOFS FOR L EMMA
Proof.
Take y =
α ◦ x and let p y
=
y − [ y ] , where [ y ] is the coordinate-wise ﬂoor, and p y is the vector of probabilities in the deﬁnition of I nt ( y ) .
By deﬁnition it holds
[Equation]
Plugging back y
= α ◦ x , we obtain the ﬁrst claim.
Similarly,
[Equation]
After substituting y = α ◦
, it remains to mention
[Equation]
To obtain the last fact, notice that I nt ( y ) − [ y ] is a vector of Bernoulli random variables.
Since the variance of any Bernoulli variable is bounded by 14 , we have
[Equation]
The staring point of our analysis is the following recursion.
Let ρ k def = (cid:107) x k − x ∗ (cid:107) 2 , δ k def = f ( x k ) − f ( x ∗ ) and ζ k def = (cid:107) 1 n (cid:80) ni
=1 g ki (cid:107) 2 .
Lemma 2.
Assume that either i) functions f 1 , . . .
, f n are convex, or ii) f is convex and f 1 , . . .
, f n are differentiable.
Then
[Equation]
where A k def = 2 η 2 k E k (cid:2) ζ k (cid:3) and B k def = 12 n (cid:80) dj
=1 η 2 k α 2 k,j − (cid:107) x
k +1 − x k (cid:107) 2 are the SGD and quantization error terms, respectively.
B.1. P ROOF OF L EMMA
Proof.
The last term in the expression that we want to prove is needed to be later used to compensate quantization errors.
For this reason, let us save one (cid:107) x k +1 − x k (cid:107) 2 for later when expanding (cid:107) x
k +1 − x ∗ (cid:107) 2 .
Consider the IntSGD step x k +1 − x k = η k 1 n (cid:80) ni
=1 Q ( g ki ) , where Q ( g ki )
=
1 α k ◦
I nt ( α k ◦ g ki ) .
[Equation]
Now let us forget about the ﬁrst and last terms, which will directly go into the ﬁnal bound, and work with the other terms.
By our assumptions, we either have E k [ Q ( g ki )]
=
E k
[ g ki ] ∈
∂f i ( x k ) orn
[Equation]
ni =1 E k [ g ki ] =
∇ f ( x k ) , so we obtain by convexity
E k
[Equation]
Moreover, using the tower property of expectation
[Equation]
we can decompose the penultimate term in (10) as follows:
[Equation]
where in the last step we also used independence of the quantization errors ( Q ( g k 1 )
− g k 1 ) , . .
, ( Q ( g kn ) − g kn ) .
Next, we are going to deal with the quantization terms:
[Equation]
Dividing both sides by n 2 and plugging it into (10), we obtain the desired decomposition into SGD and quantization terms.
We now show how to control the quantization error by choosing the scaling vector α k in accordance with Assumption 1.
Lemma 3.
If the assumptions of Lemma 2 hold together with Assumption 1, then
[Equation]
B.1. P ROOF OF L EMMA
Proof.
Firstly
[Equation]
let us recur the bound in Lemma 2 from k to 0:
[Equation]
Note that in the bound we do not have α 0 ,j for any j as we assume that the ﬁrst communication is done without compression.
Assumption 1 implies for the quantization error
[Equation]
It is clear that the latter terms get canceled when we plug this bound back into the ﬁrst recursion.
B.1. P ROOF OF T HEOREM
Proof.
Most of the derivation has been already obtained in Lemma 3 and we only need to take care of the SGD terms.
To do that, we decompose the gradient error into expectation and variance:
[Equation]
n
[Equation]
Thus, we arrive at the following corollary of Lemma 3:
[Equation]
Furthermore, by convexity of f we have
[Equation]
Plugging it back, rearranging the terms and dropping E (cid:2) (cid:107) x k +1 − x ∗ (cid:107) 2 (cid:3) gives the result.
B.1. P ROOF OF P ROPOSITION
Proof.
Fix any i .
By Young’s inequality and independence of ξ k 1 , . . .
, ξ kn we have
[Equation]
[Table]
n
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Substituting the deﬁnition of σ 2 ∗ and applying Jensen’s inequality, we derive
[Equation]
By our assumption, f i ( · ; ξ ) is convex and has L i -Lipschitz gradient, so we can use Equation (2.1.7) in Theorem 2.1.5 in Nesterov (2013):
[Equation]
Taking the average over i = 1 , . . . , n and noticing (cid:80) ni =1 ∇ f i ( x ∗ )
=
yields
[Equation]
which is exactly our claim
[Equation]
B.1. P ROOF OF T HEOREM
Proof.
The proof is almost identical to that of Theorem
[Equation]
but now we directly use Assumptionand plug it in inside Lemma 3 to get
[Equation]
Rearranging this inequality yields
[Equation]
To ﬁnish the proof, it remains to upper bound f (ˆ x k ) using convexity the same way as it was done in
Equation (12).
B.1. P ROOF OF T HEOREM
Proof.
By L -smoothness of f we have
[Equation]
Similarly to Lemma
[Equation]
we get a decomposition into SGD and quantization errors:
[Equation]
We proceed with the two terms separately.
To begin with
[Equation]
we further decompose the SGD error into its expectation and variance:
[Equation]
Moving on, we plug it back into the upper bound E k [ f ( x k +1 )] .
Assuming η k ≤ 12 L , we get
[Equation]
Finally, reusing Equation (11) produces the bound
[Equation]
Notice that by Assumption 4 f inf ≤ f ( x k +1 ) , so we have
[Equation]
The left-hand side is equal to E (cid:2) (cid:107)∇ f (ˆ x k ) (cid:107) 2 (cid:3) by deﬁnition of ˆ x k , and we conclude the proof.
B.1. P ROOF OF C OROLLARY
Proof.
For the ﬁrst part, we have
[Equation]
The other complexities follow similarly.
B.1. P ROOF OF P ROPOSITION
Proof.
By deﬁnition of α k
[Equation]
B.1. P ROOF OF P ROPOSITION
Proof.
Indeed
[Equation]
we only need to plug in the values of α k,j : d (cid:88) j
=
[Equation]
B.1. P ROOF OF P ROPOSITION
Proof.
Since the l -th block has d l coordinates, we get d (cid:88)
[Equation]
B. P ROOFS FOR I NT DIANA
Assumption 5.
f il ( x ) has L il -Lipschitz gradient.
We deﬁne L def = 4 max i ∈ [ n ] max l ∈ [ m ] L il .
Proposition 5.
Suppose that Assumption 5 holds.
Then, we have the following for IntDIANA and any x ∈ R d :
[Equation] [Equation]
Proof.
Based on Assumption 5 and Theorem
[Equation] [Equation]
5 Nesterov (2013)
[Equation]
we have:
[Equation]
Thus
[Equation]
double averaging leads to:
[Equation]
Considering that 1 mn (cid:80) ni =1 (cid:80) ml
=1 ∇ f il ( x ∗ ) = 0 and deﬁning 4 max i ∈ [ n ] max l ∈ [ m ] L il leads to the claim in the proposition.
Lemma 4.
For IntDIANA (Algorithm 3) and g k def = 1 n (cid:80) ni =1 ( h ki + Q ( g ki )) , we have E k (cid:2)
g k (cid:3) = ∇ f ( x k ) and:
[Equation]
Proof.
By deﬁnition, g k = 1 n (cid:80)
[Equation]
, so
[Equation]
Thus, we have shown that g k is an unbiased estimate of ∇ f ( x k ) .
Let us proceed with the second moment of g k :
[Equation]
+ g ki (cid:19)(cid:13)(cid:13)(cid:13) (cid:13) 2 (cid:35) (
[Equation]
=
[Equation]
(cid:34)(cid:13)(cid:13)(cid:13)(cid:13)n n (cid:88) i =
(cid:18)α k ◦ I nt
( α k ◦ ( g ki − h ki )) − ( g ki − h ki ) (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) + E k (cid:34)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
i =
g ki (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35)
=nn
[Table]
i =
E
[Equation]
(cid:34)(cid:13)(cid:13)(cid:13)(cid:13) 1 α k ◦
I nt ( α k ◦ ( g ki − h ki )) − ( g ki − h ki ) (cid:13)(cid:13)(cid:13)(cid:13)
(cid:35)
+ E k (cid:34)(cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88)
i =
g ki (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) (4) ≤4 n
[Equation]
(cid:88)
[Equation]
=  α
[Equation]
k
[Equation]
(cid:34)(cid:13)(cid:13)(cid:13)(cid:13) n n (cid:88) i =
g ki (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) .
Lemma 5.
If L-SVRG estimator g ki = ∇ f il ( x k ; ξ ki ) − ∇ f il ( w ki ; ξ ki ) + u ki is used in IntDIANA , we have E k (cid:2) g ki (cid:3) =
∇ f i ( x k ) and
[Equation] [Equation]
where σ k 1 = 1 mn (cid:80) ni =1 (cid:80) ml
=1 (cid:107)∇ f il ( w ki ) − ∇ f il ( x ∗ ) (cid:107) 2 .
Proof.
Recall that E (cid:2) (cid:107)
X
− E
[ X ] (cid:107) 2 (cid:3) ≤ E (cid:2) (cid:107) X (cid:107) 2 (cid:3) for any random variable X .
For the L-SVRG estimator g ki = ∇ f il ( x k ; ξ ki ) − ∇ f il ( w ki ; ξ ki ) + u ki , we have:
[Equation]
n
[Equation]
i =
[Equation]
[Table]
[Equation]
where σ k 1 = 1 mn (cid:80) ni =
[Equation]
ml
=1 (cid:107)∇ f il ( w ki ) − f il ( x ∗ ) (cid:107) 2 .
Based on the update of control sequence in
L-SVRG , we have:
[Equation]
Lemma 6.
Deﬁne σ k 2 def = 1 n (cid:80) ni
=1 (cid:107)
h
ki − ∇ f i ( x ∗ ) (cid:107) 2 .
For IntDIANA algorithm, we have:
[Equation]
For the full gradient, we have 1 n (cid:80) ni
=1 E k (cid:2) (cid:107) g ki − ∇ f
[Equation]
( x ∗ ) (cid:107) 2 (cid:3) ≤ L 2 ( f ( x k ) − f ( x ∗ )) .
For the LSVRG estimator, we have 1 n (cid:80) ni
=1 E k (cid:2) (cid:107) g ki − ∇ f i ( x ∗ ) (cid:107) 2 (cid:3) ≤ 4 σ k 1 + 3 L ( f ( x k ) − f ( x ∗ )) .
Proof.
We deﬁne σ k 2 def = 1 n (cid:80) ni =1 (cid:107) h ki − ∇ f i ( x ∗ ) (cid:107) 2 .
Consider the step
h
k +1 i
= h ki + Q ( g ki ) , where Q ( g ki )
=
1 α k ◦
I nt ( α k ◦ ( g ki − h ki )) .
Note that E k (cid:2) (cid:104) g ki − h ki , g ki − 2 ∇ f i ( x ∗ )
+ h ki (cid:105) (cid:3) =
E k (cid:2) (cid:107) g ki − ∇ f i ( x ∗ ) (cid:107) 2 − (cid:107) h
ki − ∇ f i ( x ∗ ) (cid:107) 2 (cid:3) , which explains the last equality below:
[Equation]
For the full gradient g k i = ∇ f i ( x k ) , we have:
[Equation]
For the L-SVRG estimator
[Equation]
we have by Young’s inequality:
[Equation]
Lemma 7.
Suppose that Assumption 5 holds.
Besides, we assume that f ( · ) is µ -strongly convex ( µ ≥ 0 ).
For IntDIANA with adaptive α k =
[Equation]
n
[Equation]
and GD gradient estimator, we have:
[Equation]
For IntDIANA with adaptive α k and L-SVRG gradient estimator, we have:
[Equation]
Proof.
By µ -strong convexity
[Equation]
we have:
[Equation]
Applying Proposition 3 to the obtained bound results in the following recursion
[Equation]
With the GD estimator, the produced bound simpliﬁes to
[Equation]
Based on Lemma 6, the following is satisﬁed for IntDIANA with GD estimator and adaptive α k =
[Equation] [Equation]
L
[Equation]
In turn,
Lemma gives for
L-SVRG estimator
[Equation]
≤
[Equation]
2 L + L n
[Equation]
f ( x k ) − f ( x ∗ ) (cid:1) + 2 n σ k 1 , so we can derive that
E k
[Equation]
≤
[Equation]
Let us now combine Equation (16) and Lemma 6:
[Equation]
Lemma 8.
We deﬁne the Lyapunov function as Ψ k def
= (cid:107)
[Equation]
k
[Equation] [Equation] [Equation]
1 η
[Equation]
k σ
[Equation]
1 + c 2 η 2 k σ k 2 .
Assume that the conditions of Lemma 7 hold.
If µ > 0 , we have:
[Equation]
where θ def = max (cid:8) (1 − η
[Equation]
µ ) ,
[Equation]
, c 1 = 0 , c {+
+} = L [Equation]
and η k ≤ 1 2( L + L 32 n ) for IntDIANA with GD estimator.
Alternatively, for IntDIANA with L-SVRG estimator, we have θ def = max (cid:8) (1 − η k µ ) , 34 , (cid:0) 1 − 38 m (cid:1)(cid:9) and set c 1 = 8 mn , c
=
L 2 4 n , p = 1 m , and η k ≤ 1 2( L +2 L /n ) .
If µ = 0 , we have
[Equation]
where η k ≤ 1 4( L + L 32 n ) for the GD variant and η k ≤
[Equation]
L /n ) for the L-SVRG variant.
Proof.
We deﬁne the Lyapunov function as Ψ k def = (cid:107) x k − x ∗ (cid:107) 2 + (cid:107) x k − x k − 1 (cid:107) 2 + c 1 η 2 k σ k 1 + c 2 η 2 k σ k 2 .
For IntDIANA with GD estimator, we can set c 1 = 0 and derive the following inequality from Lemma 7 and η k +1 ≤ η k :
[Equation]
We ﬁrst consider µ > 0 case.
Let c 2 =
L 2 4 n , and η k ≤ 1 2( L + L 32 n ) .
We have E (cid:2)
Ψ k +1 (cid:3) ≤ max (cid:8) (1 − η k µ ) , 34 (cid:9) E (cid:2)
Ψ k (cid:3) .
For IntDIANA with L-SVRG estimator, we have the following based on Lemma 7:
[Equation]
x ∗
[Equation]
Let c 1 = 8 mn , c
=
L 2 4 n , p = 1 m , and η k ≤ 1 2( L +2 L /n ) .
Plugging these values into the recursion, we get E (cid:2)
Ψ k +1 (cid:3) ≤ max (cid:8) (1 − η k µ ) , 34 , (cid:0) 1 − 38 m (cid:1)(cid:9) E (cid:2)
Ψ k (cid:3) .
If µ = 0 , we instead let η k ≤ 1 4( L + L 32 n ) for the GD variant and η k ≤ 1 4( L +2 L /n )
for the L-SVRG variant to obtain from the same recursions:
[Equation]
C D ETAILS AND A DDITIONAL R ESULTS OF E XPERIMENTS C. M ORE DETAILS
Here we provide more details of our experimental setting.
We use the learning rate scaling technique (Goyal et al., 2017; Vogels et al., 2019) with 5 warm-up epochs.
As suggested in previous works
(Vogels et al., 2019; Alistarh et al., 2017; Horváth et al., 2019), we tune the initial single-worker learning rate on the full-precision SGD and then apply it to PowerSGD , QSGD , and NatSGD .
For the task of training ResNet18 on the CIFAR-10 dataset, we utilize momentum β = 0 .
9 and weight decay with factor 10 − 4 (except the Batchnorm parameters) for all algorithms.
All algorithms run for 300 epochs.
The learning rate decays by 10 times at epoch 150 and 250.
The initial learning rate is tuned in the range {0.05, 0.1, 0.2, 0.5} and we choose 0.1.
For the task of training a 3-layer LSTM, all algorithms run for 90 epochs.
We set the size of word embeddings to 650, the sequence length to 30, the number of hidden units per layer to 650, and the dropout rate to 0.4.
Besides, we tie the word embedding and softmax weights.
We tune the initial learning rate in the range of {0.6, 1.25, 2.5, 5} and we choose 1.25.
For both tasks, we report the results based on 3 repetitions with random seeds {0, 1, 2}.
To measure the time of computation, communication, and compression/decompression of the algorithms, we use the timer (a Python context manager) implemented in the PowerSGD code 5 .
For PowerSGD , we use rank = 2 in the task of training ResNet18 on the CIFAR-10 dataset and rank = 4 in the task of training LSTM on the Wikitext-2 dataset as suggested byVogels et al.
For QSGD , we use the gradient matrix of each layer as a bucket and set the number of quantization levels to be 64 (6-bit).
C. T OY EXPERIMENT ON TIMINGS
[Figure]
Figure 2 shows the different manners of PowerSGD and IntSGD to save the communication time based on the all-reduce compared to full-precision SGD : 1) IntSGD (8-bit) communicates the data with int8 data type but does not reduce the number of coordinates; 2) PowerSGD does not change the data type but breaks one communication round of a big number of coordinates into three communication rounds of much smaller numbers of coordinates.
C. C ONVERGENCE CURVES OF THE DEEP LEARNING TASKS
Please see Figure 3 and Figure 4.
C. S ENSITIVITY ANALYSIS OF HYPERPARAMETERS
We analyze the sensitivity of IntSGD to its hyperparameters β and ε .
As shown in Figure 5, the performance of IntSGD is quite stable across the choices of β ∈ { 0 . 0 , 0 . 3 , 0 . 6 , 0 . 9 } and ε ∈ { 10 − 4 , 10 − 6 , 10 − 8 } on the two considered tasks.
Overall, β = 0 .
9 and ε = 10 − 8 is a good default setting for our IntSGD algorithm.
[Figure] [Figure]
l
[Figure]
i
[Figure]
[Figure] [Figure]
¯ 94.
Test Accuracy (%) " 94.
10 ¡10 ¡10 ¡ 4 " 0 .0 .0 .0 .¯ 4.
Test Loss
C. L OGISTIC REGRESSION EXPERIMENT
Setup: We run the experiments on the (cid:96) 2 -regularized logistic regression problem with four datasets (a5a, mushrooms, w8a, real-sim) from the LibSVM repository 6 , where
[Equation]
and f
[Equation]
and x ∈ R d , λ 2 is chosen proportionally to 1 mn and A i,l ∈ R d , b i,l ∈ {− 1 , 1 } are the feature and label of l -th data point on the i -th worker.
The experiments are performed on a machine with 24 Intel(R)
Xeon(R)
Gold 6246 CPU @ 3.30GHz cores, where 12 cores are connected to a socket (there are two sockets in total).
All experiments use 12 cpu cores and each core is utilized as a worker.
The communications are implemented based on the MPI4PY library Dalcín et al.
The “optimum” x ∗ is obtained by running GD with the whole data using one cpu core until there are 5000 iterations or (cid:107)∇ f ( x ) (cid:107) 2 ≤ 10 − 30 .
[Table]
The whole dataset is split according to its original indices into n folds, and each fold is assigned to a local worker, i.e., the data is heterogeneous.
There are m = (cid:98) Nn (cid:99) data points on each worker.
For each dataset, we run each algorithm multiples times with 20 random seeds for each worker.
For the stochastic algorithms, we randomly sample 5% of the local data as a minibatch (i.e., batch size τ = (cid:98) m 20 (cid:99) ) to estimate the stochastic gradient g ki on each worker.
We set p = τm in VR-IntDIANA .
Apart from IntSGD with g ki = ∇ f i ( x k ) ( IntGD )
[Figure]
we also evaluate IntDIANA (Algorithm 3) with the
GD or L-SVRG estimator (called IntDIANA and VR-IntDIANA , respectively).
[Figure]
t e g e r i n n
[Figure]
As shown in Figure 6, IntSGD with g ki = ∇ f i ( x k ) ( IntGD ) suffers from low compression efﬁciency issue (very large integer in the aggregated vector (cid:80)
ni =1 Q ( g ki ) ) and IntDIANA can solve this issue and only requires less then 3 bits per coordinate in the communication.
IntDIANA with SVRG gradient estimator ( VR-IntDIANA ) further improves IntDIANA with GD estimator in terms of gradient oracles.