Shifted Compression Framework: Generalizations and Improvements
Abstract
Communication is one of the key bottlenecks in the distributed training of large-scale machine learning models, and lossy compression of exchanged information, such as stochastic gradients or models, is one of the most effective instruments to alleviate this issue.
Among the most studied compression techniques is the class of unbiased compression operators with variance bounded by a multiple of the square norm of the vector we wish to compress.
By design, this variance may remain high, and only diminishes if the input vector approaches zero.
However, unless the model being trained is overparameterized, there is no a-priori reason for the vectors we wish to compress to approach zero during the iterations of classical methods such as distributed compressed SGD , which has adverse effects on the convergence speed.
Due to this issue, several more elaborate and seemingly very different algorithms have been proposed recently, with the goal of circumventing this issue.
These methods are based on the idea of compressing the difference between the vector we would normally wish to compress and some auxiliary vector that changes throughout the iterative process.
In this work we take a step back, and develop a uniﬁed framework for studying such methods, both conceptually and theoretically.
Our framework incorporates methods compressing both gradients and models, using unbiased and biased compressors, and sheds light on the construction of the auxiliary vectors.
Furthermore, our general framework can lead to the improvement of several existing algorithms, and can produce new algorithms.
Finally, we performed several numerical experiments to illustrate and support our theoretical ﬁndings.
INTRODUCTION
We consider the distributed optimization problem
[Equation]
where n is the number of workers/clients and f i : R d → R is a smooth function representing the loss of the model parametrized by x ∈ R d for data stored on node i .
This formulation has become very popular in recent years due to the increasing need for training large-scale machine learning models (Goyal et al., 2018).
Communication bottleneck.
Compute nodes have to exchange information in a distributed learning process.
The size of the sent messages (usually gradients or model updates) can be very large, which creates a signiﬁcant bottleneck (Luo et al., 2018;
Peng et al., 2019; Sapio et al., 2021) to the whole training procedure.
One of the main practical solutions to this problem is lossy communication compression (Seide et al., 2014; Kone ˇ cný et al., 2016;
Alistarh et al., 2017).
It suggests applying a (possibly randomized) mapping C to a vector/matrix/tensor x before it is transmitted in order to produce a less accurate estimate C ( x ) : R d → R d and thus save bits sent per every communication round.
Compression operators.
The topic of gradient compression in distributed learning has been studied extensively over the last years from both practical (Xu et al., 2020) and theoretical (Beznosikov et al., 2020; Safaryan et al., 2021c;
Albasyoni et al., 2020) approaches.
Compression operators are typically divided into two large groups: unbiased and biased operators.
The ﬁrst group includes methods based on some sort of rounding or quantization : Random Dithering (Goodall, 1951; Roberts, 1962), Ternary quantization (Wen et al., 2017), Natural (Horváth et al., 2019a), and Integer (Mishchenko et al., 2022) compression.
Another popular example is random sparsiﬁcation – Rand-K (Wangni et al., 2018;
Stich et al., 2018; Koneˇcný and Richtárik, 2018), which preserves only a subset of the original vector coordi-
[Table]
nates.
These two approaches can also be combined (Basu et al., 2019) for even more aggressive compression.
There are also many other approaches based on low-rank approximation (Vogels et al., 2020;
Wang et al., 2018; Safaryan et al., 2021b), vector quantization (Gandikota et al., 2021), etc.
The second group of biased compressors mainly includes greedy sparsiﬁcation – Top-K (Alistarh et al., 2018;
Stich et al., 2018) and various sign-based quantization methods (Seide et al., 2014; Bernstein et al., 2018; Safaryan and Richtárik, 2021).
For a more complete review of compression operators, one can refer to the surveys by Xu et al.
(2020) and Beznosikov et al.
Safaryan et al.
(2021c).
Optimization algorithms.
Compression operators on their own are not sufﬁcient for building a distributed learning system because they always go along with optimization algorithms.
Distributed Compressed Gradient Descent ( DCGD ) (Khirirat et al., 2018) is one of the ﬁrst theoretically analyzed methods which considered arbitrary unbiased compressors.
The issue with DCGD is that it was proven to converge linearly only to a neighborhood of the optimal point with constant step-size.
DIANA (Mishchenko et al., 2019) ﬁxed this problem by compressing specially designed gradient differences.
Later DIANA was generalized (Condat and Richtárik, 2021), combined with variance reduction (Horváth et al., 2019b), accelerated (Li et al., 2020) in Nesterov’s sense (Nesterov, 1983) and by using smoothness matrices (Safaryan et al., 2021a) with a properly designed sparsiﬁcation technique.
On the other side are methods working with biased compressors, which require the use of the error-feedback ( EF ) mechanism (Seide et al., 2014; Alistarh et al., 2018; Stich and Karimireddy, 2020).
Such algorithms were often considered to be better in practice due to the smaller variance of biased updates (Beznosikov et al., 2020).
However, it was recently demonstrated that biased compressors can be incorporated into specially designed unbiased operators, and show superior to error-feedback results (Horváth and
Richtárik, 2021).
In addition, error-feedback was recently combined with the DIANA trick (Gorbunov et al., 2020), which led to the ﬁrst linearly converging method with EF . Later Condat et al.
(2022) proposed a uniﬁed framework for methods with biased and unbiased compressors.
Compressed iterates.
Most of the existing literature (including all methods described above) focuses on compression of the gradients, while in applications like Federated
Learning (McMahan et al., 2017; Koneˇcný et al., 2016; by:
Peter Kairouz and McMahan, 2021), it is vital to reduce the size of the broadcasted model parameters (Reisizadeh et al., 2020).
This demand gives rise to optimization algorithms with compressed iterates.
The ﬁrst attempt to analyze such methods was done by Khaled and Richtárik (2019) for Gradient Descent with Compressed iterates ( GDCI ) in a single node set up.
Later GDCI was combined with variancereduction for noise introduced by compression and generalized to a much more general setting of distributed ﬁxedpoint methods (Chraibi et al., 2019).
Summary of contributions.
The obtained results are summarized in Table 1, with the improvements over previous works highlighted.
The main contributions include:
Generalizations of existing methods.
We introduce the concept of a Shifted Compressor , which generalizes a common deﬁnition of compression operators used in distributed learning
This technique allows to study various strategies for updating the shifts using both biased and unbiased compressors to recover and improve such previously known methods as DCGD and DIANA .
Additionally, as a byproduct, a new algorithm is obtained: DCGD-STAR , which achieves linear convergence to the exact solution if we know the local gradients at the optimum.
Improved rates.
The notion of a shifted compressor allows us to revisit existing analysis of distributed methods with compressed iterates and improve guarantees in both cases: with and without variance-reduction.
Obtained results indicate that algorithms with model compression can have the same complexity as compressed gradient methods. 3.
New algorithm.
We present a novel distributed algorithm with compression, called Randomized DIANA , with linear convergence rate to the exact optimum.
It has a signiﬁcantly simpler analysis than the original DIANA method.
Via examination of its experimental performance we highlight the cases when it can outperform DIANA in practice.
GENERAL FRAMEWORK
In this section we introduce compression operators and the framework of shifted compressors.
2.1 STANDARD COMPRESSION
At ﬁrst recall some basic deﬁnitions.
Deﬁnition 1 (General contractive compressor) .
A (possibly) randomized mapping C : R
d
→ R d is a compression operator ( C ∈ B ( δ ) for brevity) if for some δ ∈ (0 , 1] and ∀ x ∈ R d [Equation]
where the expectation is taken w.r.t.
(possible) randomness of operator C .
One of the most known operators from this class is greedy sparsiﬁcation ( Top-K for K ∈ { 1 , . . .
, d } ):
[Equation]
where coordinates are ordered by their magnitudes so that | x (1) | ≤ | x (2) | ≤ · · · ≤ | x ( d ) | , and e 1 , . .
e d ∈ R
d are the standard unit basis vectors.
This compressor belongs to B ( K/d ) .
Deﬁnition 2 (Unbiased compressor) .
A randomized mapping Q : R d → R d is an unbiased compression operator ( Q ∈ U ( ω ) for brevity) if for some ω ≥ 0 and ∀ x ∈ R d ( a ) E Q ( x ) = x, (Unbiasedness) ( b ) E (cid:107)Q
[Equation]
The last inequality implies that
[Equation]
A notable example from this class is the random sparsiﬁcation ( Rand-K for K ∈ { 1 , . . .
, d } ) operator:
[Equation]
Rand
[Equation]
where S is a random subset of [ d ] := { 1 , . .
, d } sampled from the uniform distribution on the all subsets of [ d ] with cardinality K .
Rand-K belongs to U ( d/K − 1) .
Notice that property (a) from Deﬁnition 2 is “uniform” across all vectors x , while property (b) is not.
Namely, vector x = 0 is treated in a special way because E (cid:107)Q (0) − 0 (cid:107) 2 = 0 , which means that the compressed zero vector has zero variance .
In other words, zero is mapped to itself with probability 1.
2.2 COMPRESSION WITH SHIFT
We can generalize the class of unbiased compressors U ( ω ) to a class of operators with other (not only 0) “special” vectors.
Speciﬁcally, this class allows for shifts away from the origin, which is formalized in the following deﬁnition.
Deﬁnition 3 (Shifted compressor) .
A randomized mapping
Q h : R d → R d is a shifted compression operator ( Q h ∈
U ( ω ; h ) in short) if exists ω ≥ 0 such that ∀ x ∈ R d
[Equation]
Vector h ∈ R
d is called a shift .
Note that class of unbiased compressors U ( ω ) is equivalent to U ( ω ; 0) .
The next lemma shows that shifts add up and all shifted compression operators
Q h ∈ U ( ω ; h ) arise by a shift of some operator Q 0 from U ( ω ; 0) .
Lemma 1 (Shifting a shifted compressor) .
Let Q h ∈ U ( ω ; h ) and v ∈ R d .
Then the (possibly) randomized mapping Q deﬁned by
Q
[Equation]
satisﬁes Q ∈ U ( ω ; h + v ) .
The shifted compressor concept allows us to construct a shifted compressed gradient estimator Q h ∈ U ( ω ; h ) given by
[Equation]
which is the main focus of this work.
In particular, we are going to study different mechanisms for choosing this shift vector throughout the optimization process.
Note: The estimator (3) is clearly unbiased, as soon as the operator Q satisﬁes E Q ( x ) = x .
Estimator (3) uses operator Q from class of unbiased compressors U ( ω ) , which are usually easier to analyze but have higher empirical variance than their biased counterparts (Beznosikov et al., 2020).
In an attempt to kill two birds with one stone, we can incorporate the (possibly) biased compressor C ∈ B ( δ ) into h using a similar shift trick:
[Equation]
as g h ( x ) allows for virtually any shift vector.
This leads to the following estimator
[Equation]
2.3 THE META-ALGORITHM
Now we are ready to present the general distributed optimization algorithm for solving ( (cid:63) ) that employs shifted gradient estimators
[Equation]
1: Parameters: learning rate γ > 0 ; unbiased compressors Q 1 , . . .
, Q n ; initial iterate x 0 ∈ R d , initial local shifts h 01 , . . .
, h 0 n ∈ R d (stored on the n nodes) 2: Initialize: h 0 = 1 n (cid:80) ni
=1 h
i (stored on the master) 3: for k = 0 , 1 , 2 . . .
do 4:
Broadcast x k to all workers 5: for i = 1 , . . .
n do in parallel 6:
Compute local gradient: ∇ f i ( x k ) 7:
Compress: m ki
= Q i ( ∇ f i ( x k ) − h ki ) 8:
Update the local shift: h k
+
i 9:
Send m ki and/or (maybe)
h k +1 i to the master 10: end for 11:
Aggregate received messages: m k = 1 n (cid:80) ni
=
m ki 12:
Compute global estimator: g k = h k + m k 13:
Take gradient descent step: x k +
= x
k − γg k 14:
Update aggregated shift: h k +
= 1 n (cid:80) ni
=
h k
+
i 15: end for
In Algorithm 1, each worker i = 1 , . . .
, n queries the gradient oracle ∇ f i ( x k ) in iteration k .
Then, a compression operator is applied to the difference between the local gradient and shift, and the result is sent to the master (and also possibly the new shift).
The shift is updated on both the server and workers.
After receiving the messages m ki , a global gradient estimator g k is formed on the server, and a gradient step is performed.
Note that this method is not fully deﬁned because it requires a description of the mechanism for updating the shifts
h k
+1 i (highlighted in color) throughout the iteration process on both workers and master.
In the next section, we illustrate how the shifts can be chosen and updated.
CHOOSING THE SHIFTS
First, in Table 2, we show the generality of our approach by presenting some of the existing and new distributed methods that fall into our framework of DCGD-SHIFT with shift updates of the form (4).
The following assumptions are needed to analyze convergence and compare with previous results.
Assumption 1 (Strong convexity) .
Function f : R d → R is µ -strongly convex if
[Equation]
If µ = 0 , then the function is convex.
Assumption 2 (Smoothness) .
Function f : R d → R is
L -smooth if
[Equation]
Now, we can provide a general convergence guarantee for
Algorithm 1 with ﬁxed shifts
[Equation]
Theorem 1 ( DCGD with fixed SHIFT ) .
Assume each f
i is convex and L i -smooth, and f is L -smooth and µ -strongly convex.
Let Q i ∈ U
( ω i ) be independent unbiased compression operators.
If the step-size satisﬁes
[Equation]
then the iterates of Algorithm 1 with ﬁxed shifts
h ki ≡ h
i satisfy
[Equation]
This theorem establishes a linear convergence rate up to a certain oscillation radius, controlled by the average distance of shift vectors
h i to the optimal local gradients ∇ f i ( x (cid:63) ) multiplied by the step-size γ .
This means
that in the interpolation/ overparameterized regime ( ∇ f i ( x (cid:63) ) =
for all i ), method reaches exact solution with zero shifts h
i = 0 .
[Table]
In the following subsections, we study how the shifts can be formed to guarantee linear convergence to the exact optimum.
We start by introducing practically useless, but theoretically insightful DCGD-STAR , and then move onto implementable algorithms that learn the optimal shifts.
3.1 OPTIMAL SHIFTS
Assume, for the sake of argument, that we know the values ∇ f i ( x (cid:63) ) for every i ∈ [ n ] .
Then, we can construct optimally shifted compressed shift updates sequence using the form (4)
[Equation]
k +
[Equation]
This is enough to fully characterize the Algorithm 1 and obtain the following convergence guarantee:
Theorem 2 ( DCGD-STAR ) .
Assume each f
i is convex and
L i -smooth, and f is L -smooth and µ -strongly convex.
Let
Q i ∈ U ( ω i ) , C i ∈ U ( δ i ) be independent compression operators.
If the step-size satisﬁes
[Equation]
then the iterates of DCGD with optimally shifted compressed shift update (8) satisfy
[Equation]
This is the ﬁrst presented algorithm with linear convergence to the exact solution for the general not-overparameterized case .
Notice that for zero-identity operators C i ≡ 0 we obtain the simplest optimal shift h i = ∇ f i ( x (cid:63) ) and the term δ i in (9) should be interpreted as zero.
The issue with the described method is that, in general, we do not know the values h (cid:63) i : = ∇ f i ( x (cid:63) )
(unless the problem is overparametrized), which makes method impractical.
3.2 LEARNING THE OPTIMAL SHIFTS
We need to design the sequences { h k 1 } k ≥ 0 , . .
. , { h kn } k ≥ 0 in such a way that they all converge to the optimal shifts: h ki → ∇ f i ( x (cid:63) ) as k → ∞ .
However, at the same time, we do not want to send uncompressed vectors from workers to the master.
So, the challenge is not only learning the shifts, but doing so in a communication-efﬁcient way.
We present two different solutions to this problem in this work.
3.2.1 DIANA -like Trick
Our ﬁrst approach is based on the celebrated DIANA (Mishchenko et al., 2019; Horváth et al., 2019b) algorithm:
[Equation]
where α is a suitably chosen step-size.
For C i ≡ 0 , it takes the simpliﬁed form
[Equation]
This recursion resolves both of the raised issues earlier.
Firstly, this sequence of h ki indeed converges to the optimal shifts ∇ f i ( x (cid:63) ) , which is formalized in the Theorem 3 presented later.
Moreover, the shift on the master
[Equation]
[Equation]
is updated as follows:
[Equation]
which requires aggregation of the compressed vectors c ki
[Equation]
=
C i ( ∇ f i ( x k ) − h ki ) and m ki :
=
Q i
[Equation]
∇
[Equation]
x
[Equation]
from the workers.
In the case of update (11), it is not even needed to send anything in addition to the messages m ki required by default in Algorithm 1.
Furthermore, simpliﬁed recursion (11) can be interpreted as one step of Compressed Gradient Descent ( CGD ) with step-size α applied to such optimization problem:
[Equation]
which is in fact a 1-smooth and 1-strongly concave function.
In this way, h k +1 i keeps track of the latest local gradient and produces a better estimate than the previous shift h ki .
Now we present the convergence result for the Algorithmwith described before shift learning procedure.
Theorem 3 (Generalized DIANA ) .
Assume each f
i is convex and L i -smooth, and f is µ -strongly convex.
Let Q i ∈ U
( ω i ) ,
C i ∈ U ( δ i ) be independent compression operators.
If the step-sizes for all i satisfy
[Equation]
where L max := max i L i , M > 2 / ( nα ) and δ i should be interpreted as zero for C i ≡ 0 , then the iterates of DCGD with the DIANA -like shift update (10) satisfy
[Equation]
where the Lyapunov function V k is deﬁned by
[Equation]
Our result represents an improvement over the original DIANA in several ways.
Firstly, we use a much more general shift updates involving C i , which allow biased operators to be used for learning the optimal shifts.
Secondly, one can use different compressors Q i , which can be particularly beneﬁcial when different workers have various bandwidths/connection speeds to the master.
Thus, the slower workers can compress more, and therefore use operators with higher ω
i .
At the same, time the opposite makes sense for “faster” workers.
3.2.2 Randomized DIANA ( Rand-DIANA )
Recalling the original issue stated in Section 3.2 that we are dealing with: design
[Equation]
The simplest possible solution would be just to set h ki to ∇ f i ( x k ) because if x k → x (cid:63) in the optimization process, then ∇ f i ( x k ) converges to the optimal local shift.
However, this approach is not efﬁcient, as workers have to transfer full (uncompressed) vectors h k
i = ∇ f i ( x k ) .
Our alternative to the DIANA solution is to update a reference point w ki for calculating the shift h ki = ∇ f i ( w ki ) infrequently (with a small probability p i ∈ (0 , 1] ), so that h ki needs to be communicated very rarely:
[Equation]
This method has a remarkably simpler analysis than DIANA , but can solve the original problem of eliminating the variance introduced by gradient compression.
Next, we state the convergence result for DCGD with shifts updated in a randomized fashion (12).
We named it RandomizedDIANA ( Rand-DIANA in short) to acknowledge the original method (Mishchenko et al., 2019) to ﬁrst solve this problem.
Theorem 4 ( Rand-DIANA ) .
Assume that f i are convex, L ismooth for all i and f is µ -convex.
If the step-size satisﬁes
[Equation]
where M > 2 ω np m and p m :
= min i p i .
Then, the iterates of
DCGD with Randomized-DIANA shift update (12) satisfy
[Equation]
where the Lyapunov function V k is deﬁned by
[Equation]
Though appropriate choice of the parameters M = 4 ω np m and p i ≡ p = 1 ω
+1 for every i , we can obtain basically the same iteration complexity as the original DIANA (Horváth et al., 2019b) max
[Equation]
3.3 COMPRESSING THE ITERATES
In this section, we discuss how the shifted compression framework can be applied and leads to improved results for the case where the iterates/models themselves need to be compressed.
Let Q ∈ U ( ω ) .
Consider the following shifted by vector x/γ compressor
[Equation]
which clearly belongs to the class U ( ω ; x/γ ) .
Based on the fact that for γ (cid:54) =
compressor ¯ Q ( z ) :
= − 1 γ · Q ( − γz ) ∈
U ( ω ) we can transform ˆ Q to operator
[Equation]
which also belongs to U ( ω ; x/γ ) and is helpful for analysing algorithms with compressed iterates.
Distributed Gradient Descent with Compressed Iterates ( GDCI ) was ﬁrst analyzed by Khaled and Richtárik (2019) for single node and, in short, was relaxed and formulated in a convenient form by Chraibi et al.
[Equation]
This algorithm can be reformulated using the previously described shifted compressor ˜ Q ∈ U ( ω ; x k /γ )
[Equation]
which for the distributed case takes the form
[Equation]
The essence of this method is compression of the local workers’ iterates Q i (cid:0) x
k − γ ∇ f i ( x k ) (cid:1) , their aggregation on the master and convex combination with the previous model.
Next we present established linear convergence up to a neighborhood introduced due to variance of compression operator (similarly to DCGD with ﬁxed shifts Theorem 1).
Theorem 5 ( GDCI ) .
Assume each f
i is convex and L ismooth, and f is L -smooth and µ -strongly convex.
Let Q i ∈ U ( ω ) be independent compression operators.
If the stepsizes satisfy
[Equation]
then the iterates of the Distributed GDCI (13) satisfy
[Equation]
In the interpolation regime ( ∇ f i ( x
[Equation]
=
= x
[Equation]
− γ
[Equation]
i
[Equation]
) , for every i ) this result matches the complexity of DCGD with ﬁxed shifts (7)
[Equation]
and improves over the original rate of GDCI by Chraibi et al.
(2019) analyzed for ﬁxed point problems and specialized for gradient mappings:
[Equation]
Due to space limitations, the results for Distributed
Variance-Reduced Gradient Descent with Compressed
Iterates ( VR-GDCI ) , which eliminates the neighborhood in (14), along with detailed proofs of all stated theorems are presented in the Supplementary Material.
EXPERIMENTS
In this section, we present some of the experimental results obtained.
The remainder of the results (including real-world data and other models) are available in the Supplementary Materia.
To provide evidence that our theory translates into observable predictions, we focus on well-controlled settings that satisfy the assumptions in our work.
Consider a classical ridge-regression optimization problem
[Equation]
where λ = 1 /m and A ∈ R m × d , y ∈
R m are generated using the Scikit-learn library (Pedregosa et al., 2011) method sklearn.datasets.make_regression
with default parameters for m = 100 , d = 80 .
The obtained data is uniformly, evenly, and randomly distributed among 10 workers.
To compare selected algorithms, we evaluate the logarithm of a relative argument error log (cid:0) (cid:107) x k − x (cid:63) (cid:107) 2 / (cid:107) x 0 − x (cid:63) (cid:107) 2 (cid:1) on the vertical axis, while the horizontal axis presents the number of communicated bits needed to reach a certain error tolerance ε .
The starting point x 0 ∈ R d entries are sampled from the normal distribution N (0 , 10) .
In our simulations we thoroughly examine the Rand-DIANA method, which is presented for the ﬁrst time.
Extensive studies of the methods with compressed iterates can be found in the works by Khaled and Richtárik (2019); Chraibi et al.
4.1 RANDOMIZED-DIANA VS DIANA
In the ﬁrst set of experiments, we compare Rand-DIANA and DIANA with different compressors
Q
i
( C i ≡ 0 ) and varied operators’ parameters.
The results obtained are summarized in Figure 1.
The designation q :=
k/d is used for the share of non-zeroed coordinates of the
[Figure] [Figure]
Random sparsification ( Rand-K ) operator, and s corresponds to the number of levels for the Natural
Dithering ( ND )
(Horváth et al., 2019a) compressor.
The p parameter of Rand-DIANA was set at 1 / ( ω + 1) for every run.
The left plot in Figure 1 clearly shows that Rand-DIANA performs better than DIANA for every value of the Rand-K compressor parameter.
It is worth noting that DIANA performs better at higher q , while the opposite holds for RandDIANA .
From the right plot in Figure 1, one can see that DIANA with
ND can be superior to Rand-DIANA for the optimized parameter s (cid:63) .
Nevertheless, Rand-DIANA is highly preferable for very aggressive compression (e.g., s = 2 ).
In the next experimental setup, we more closely investigate the behavior of Rand-DIANA with respect to its parameters.
4.2 RANDOMIZED-DIANA STUDY
According to the formulation of Theorem 4, the constant
M has to be strictly greater than M (cid:48) :
= 2 ω/ ( np ) .
In the left plot of Figure 2, we show that the method becomes less stable and can even diverge for smaller values of M (set to
M (cid:48) · b ).
However, too high M (for b = 1 . 5 ) can lead to an overall (stable) slowdown.
We conclude that the condition imposed by theoretical analysis is indeed critical.
The right plot in Figure 2 examines how the parameter p affects the convergence in a high compression regime ( q = 0 . 1 ).
The method converges faster for smaller p and can diverge above a certain threshold, similarly to the previous study of M trade-off.
We did not conduct additional experiments to show the effect of combining unbiased compressors with biased counterparts, as the beneﬁts of such an approach have already been clearly demonstrated by Horváth and Richtárik (2021) for distributed training of deep neural networks.
Acknowledgements
We would like to thank the anonymous reviewers, Laurent
Condat and Konstantin Mishchenko for their helpful comments and suggestions to improve the manuscript.