M INIBATCH VS L OCAL SGD WITH S HUFFLING : T IGHT C ONVERGENCE B OUNDS AND B EYOND
Shashank Rajput rajput
Suvrit Sra
MIT EECS suvrit@mit.edu
A BSTRACT
In distributed learning local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods.
Most existing analyses of these methods assume independent and unbiased gradient estimates obtained via with-replacement sampling.
In contrast, we study shufﬂingbased variants: minibatch and local Random Reshufﬂing , which draw stochastic gradients without replacement and are thus closer to practice.
For smooth functions satisfying the Polyak-Łojasiewicz condition, we obtain convergence bounds (in the large epoch regime) which show that these shufﬂing-based variants converge faster than their with-replacement counterparts.
Moreover, we prove matching lower bounds showing that our convergence analysis is tight .
Finally, we propose an algorithmic modiﬁcation called synchronized shufﬂing that leads to convergence rates faster than our lower bounds in near-homogeneous settings
I NTRODUCTION
Distributed learning within the framework of federated learning (Kone ˇ cn ` y et al., 2016; McMahan et al., 2017) has witnessed increasing interest recently.
A key property of this framework is that models are trained locally using only private data on devices/machines distributed across a network, while parameter updates are aggregated and synchronized at a server.
1 Communication is often the key bottleneck for federated learning, which drives the search for algorithms that can train fast while requiring less communication—see Li et al.
(2020a)
; Kairouz et al.
(2021) for recent surveys.
A basic algorithm for federated learning is local stochastic gradient descent (SGD), also known as federated averaging.
The goal is to minimize the global objective that is an average of the local objectives.
In local SGD , we have M machines and a server.
After each round of communication, each of the M machines locally runs B steps of SGD on its local objective.
Every B iterations, the server aggregates the updated local iterates from the machines, averages them, and then synchronizes the machines with the average.
Convergence analysis of local SGD and its variants has drawn great interest recently (Dieuleveut & Patel, 2019; Haddadpour et al., 2019; Haddadpour & Mahdavi, 2019; Stich, 2019; Yu et al., 2019; Li et al., 2020b;c; Koloskova et al., 2020; Khaled et al., 2020;
Spiridonoff et al., 2020; Karimireddy et al., 2020; Stich & Karimireddy, 2020; Qu et al., 2020).
Of the many, the biggest motivation for our paper comes from the line of work by Woodworth et al.
(2020a;b; 2021).
In (Woodworth et al., 2020a;b), minibatch SGD is studied as a simple yet powerful baseline for this intermittent communication setting.
Instead of locally updating the iterates B times, minibatch SGD aggregates B gradients (evaluated at the last synced iterate) from each of the M machines, forms a minibatch of size MB , and then updates the shared iterate.
Given the same M and B , local SGD and minibatch SGD have the same number of gradient computations per round of communication, so it is worthwhile to understand which converges faster.
Woodworth et al.
(2020a;b) point out that many existing analyses on local SGD show inferior convergence rate compared to minibatch SGD.
Through their new upper and lower bounds, they identify regimes where local SGD can be faster than minibatch SGD.
While the theory of local and minibatch SGD has seen recent progress, there is still a gap between what is analyzed versus what is actually used.
Most theoretical results assume independent and unbiased gradient estimates obtained via with-replacement sampling of stochastic gradients (i.e., choosing training data indices uniformly at random).
In contrast, most practitioners use withoutreplacement sampling, where they shufﬂe indices randomly and access them sequentially.
Convergence analysis of without-replacement methods is challenging because gradients sampled within an epoch lack independence.
As a result, the standard theory based on independent gradient estimates does not apply to shufﬂing-based methods.
While shufﬂing-based methods are believed to be faster in practice (Bottou, 2009), broad theoretical understanding of such methods remains elusive, except for noteworthy recent progress mainly focusing on the analysis of SGD (G¨urb¨uzbalaban et al., 2019; Haochen & Sra, 2019;
Nagaraj et al., 2019; Nguyen et al., 2020; Safran & Shamir, 2020; 2021; Rajput et al., 2020; 2021; Ahn et al., 2020; Mishchenko et al., 2020; 2021; Tran et al., 2021).
These results indicate that in the large-epoch regime (where the number of epochs is greater than some threshold), without-replacement SGD converges faster than with-replacement SGD.
1. O UR CONTRIBUTIONS
We analyze convergence rates of without-replacement versions of local and minibatch SGD, where local component functions are reshufﬂed at every epoch.
We call the respective algorithms local
RR (Algorithm 1) and minibatch RR (Algorithm 2), and their with-replacement counterparts local
S GD and minibatch S GD .
Our key contributions are as follows:
• In Section 3, we present convergence bounds on minibatch and local RR for L -smooth functions satisfying the µ -Polyak-Łojasiewicz condition (Theorems 1 & 2).
Our theorems give highprobability bounds, a departure from the common in-expectation bounds in the literature.
We show that minibatch and local RR converge faster than minibatch and local S GD when the number of epochs is sufﬁciently large.
We also identify a regime where local RR converges as fast as minibatch RR: when synchronization happens frequently enough and local objectives are not too heterogeneous.
See also Appendix A for a detailed comparison with existing upper bounds.
• In Section 4, we prove that the upper bounds obtained in Section 3 are tight , in all factors except L and µ .
We present Theorems 3 & 4 and Proposition 5 which show lower bounds that match the upper bound up to a factor of L 2 /µ 2 .
Our lower bound on local RR indicates that if the synchronization interval B is too large, then local RR has no gain from parallel computation.
• In Section 5, we propose a simple modiﬁcation called synchronized shufﬂing that allows us to bypass the lower bounds in Section 4, at the cost of a slight increase in communication.
By having the server broadcast random permutations to local machines, we show that in near-homogeneous settings, the modiﬁed algorithms converge faster than the lower bounds (Theorems 6 & 7).
• In Appendix C, we present numerical experiments that corroborate our theoretical ﬁndings.
P ROBLEM SETUP
Notation.
For a natural number a ∈ N , let [ a ] := { 1 , 2 , . .
. , a } .
Let S a be the set of all permutations of [ a ] .
Since our indices start from 1 , we redeﬁne the modulo operation between a ∈ Z and b ∈ N as a mod b := a − (cid:98) a − 1 b (cid:99) b , to make a mod b ∈ [ b ] .
Optimization task.
Consider M machines, each with its objective F m ( x ) :
= 1 N (cid:80) Ni =1 f mi ( x ) , for m ∈
[ M ] .
The m -th machine has access only to the gradients of its own N local components f m 1 ( x ) , . . .
, f mN ( x ) .
In this setting, we wish to minimize the global objective function which is an average of the local objectives: F ( x ) :
= 1 M (cid:80) Mm =1 F m ( x ) = 1 MN (cid:80)
Mm =1 (cid:80) Ni =1 f mi ( x ) .
Further, we assume that each individual component function f mi is L -smooth, so that
[Equation]
and that the global objective F satisﬁes the µ -Polyak-Łojasiewicz (PŁ) condition.
[Equation]
where µ >
[Equation]
Algorithms.
Under the above setting, we analyze local RR (Algorithm 1) and minibatch RR (Algorithm 2) and characterize their worst-case convergence rates.
The algorithms are run over K epochs,
PŁ nonconvex strongly convex
In Algorithms 1 and 2, consider S YNC S HUF as F ALSE for now.
We will discuss S YNC S HUF in Section 5.
Input
Initialization m do do 10: do
Synchronize
Input
Initialization
(cid:46) i.e., K passes over the entire component functions.
At the beginning of epoch k , each machine m shufﬂes its local component functions { f mi } Ni =1 using a random permutation σ mk ∼ Unif( S N ) .
In local RR, each machine makes B local RR updates to its iterate by sequentially accessing its shufﬂed component functions, before the server aggregates iterates from all the machines and then synchronizes the machines with the average iterate.
In minibatch RR, instead of making B local updates, each machine collects B gradients evaluated at the last iterate, and the server aggregates them to make an update using these MB gradients.
Since these two algorithms use the same amount of communication and local gradients, minibatch RR is a simple yet powerful baseline for local RR.
Below, we collect our assumptions on the algorithm parameters used throughout the paper.
Assumption 1 (Algorithm parameters) .
We assume M ≥ 1 , N ≥ 2 , and K ≥ 1 .
Also, assume that
B divides N .
We restrict 1 ≤ B ≤ N 2 for minibatch RR because B = N makes the algorithm equal to GD.
We also assume 2 ≤ B ≤ N for local RR because B = 1 makes the two algorithms the same.
We choose a constant step-size scheme, i.e., η > 0 is kept constant over all updates.
We next state assumptions on intra- and inter-machine deviations used in this paper.
Assumption 2 (Intra-machine deviation) .
There exists ν ≥ 0 such that for all m ∈
[ M ] and i ∈ [ N ] ,
[Equation]
Assumption 2 requires that the difference between the gradient of each local component function f mi ( x ) and its corresponding local objective function F m ( x ) is uniformly bounded.
It models the variance of local components f mi within each machine.
While the uniform boundedness requirement may look strong, we use this assumption to prove high-probability upper bounds, which are stronger than the common in-expectation bounds.
See Appendix A for comparisons with other assumptions, and also Appendix D.7 for ways to avoid uniform boundedness over the entire R d .
The next two assumptions capture the deviation across different machines, i.e., the degree of heterogeneity, in two different levels of granularity: objective-wise and component-wise .
Assumption 3 (Objective-wise inter-machine deviation) .
There exist τ ≥ 0 and ρ ≥ 1 such that
[Equation]
Assumption 3 models the heterogeneity by bounding the mean of (cid:107)∇ F m (cid:107) by a constant plus a multiplicative factor times (cid:107)∇ F (cid:107) .
The assumption includes the homogeneous case (i.e., F 1 = · · · =
F M = F ) by τ = 0 and ρ = 1 .
Assumption 3 is weaker than many other heterogeneity assumptions in the literature (e.g., Karimireddy et al.
; see Appendix A for detailed comparisons.
Assumption 3 measures heterogeneity by only considering the local objectives F m , not the local components f mi .
We consider a more ﬁne-grained notion of heterogeneity in Assumption 4:
Assumption 4 (Component-wise inter-machine deviation) .
For all i ∈ [ N ] , let ¯ f i :=
1 M (cid:80) Mm =1 f mi .
There exist λ ≥ 0 such that for all m ∈
[ M ] and i ∈ [ N ] ,
[Equation]
Assumption 4 states that the gradients of the i -th components of local machines are “close” to each other.
The assumption subsumes the component-wise homogeneous setting, i.e., f 1 i = f
i
=
· · · = f Mi , by λ = 0 .
In distributed learning, this choice corresponds to the setting where each machine has the same training dataset.
Assumption 4 with λ > 0 is also relevant to the case where each device has a slightly perturbed (e.g., by data augmentation techniques) version of a certain dataset.
It is straightforward to check that Assumption 4 implies Assumption 3 with τ = λ and ρ = 1 .
We conclude this section by deﬁning the function classes we study in this paper.
Deﬁnition 1 (Function classes) .
We consider two classes of global objective functions F , also taking into account their local objectives F m and local components f mi .
We assume throughout that f mi are differentiable and F is bounded from below.
F obj ( L, µ, ν, τ, ρ ):= (cid:8) F | F is µ -PŁ; f mi are L -smooth; F , F m , f mi satisfy Assumptions 2 & 3 (cid:9) ,
F cmp ( L, µ, ν, λ ):= (cid:8) F | F is µ -PŁ; f mi are L -smooth; F , F m , f mi satisfy Assumptions 2 & 4 (cid:9) .
Notice that F obj ( L, µ, ν, τ, ρ ) ⊃ F cmp ( L, µ, ν, τ ) for any ρ ≥ 1 .
We only make the PŁ assumption on the global objective F , not on the local objectives F m nor on the local components f mi .
Using
L and µ , we deﬁne the condition number κ :=
L / µ ≥ 1 .
C ONVERGENCE ANALYSIS OF MINIBATCH AND LOCAL RR 3. U PPER BOUND FOR MINIBATCH RR
We ﬁrst begin with the convergence result for minibatch RR on F obj ( L, µ, ν, τ, ρ ) , which exhibits a faster large-epoch rate compared to the single-machine setting.
For upper bounds, we use ˜ O ( · ) to hide universal constants and logarithmic factors of 1 δ , M , N , K , and B .
Theorem 1 (Upper bound for minibatch RR) .
Suppose that minibatch RR has parameters satisfying
Assumption 1.
For any F ∈ F obj ( L, µ, ν, τ, ρ ) , consider running the algorithm using step-size η = B log( MNK 2 ) µNK for epochs K ≥ 6 κ log( MNK 2 ) .
Then, with probability at least 1 − δ ,
[Equation]
Proof.
The proof is in Appendix D.2.
The key challenge in the convergence analysis of our shufﬂing-based method stems from the indices sampled within an epoch being dependent on each other.
For example, if f m 1 is accessed already, then the index i
= 1 will not be used in later iterations of the epoch; this dependence signiﬁcantly complicates the analysis.
Our approach starts with realizing that for any permutation σ , (cid:80) Ni =
1 f mσ ( i ) =
NF m .
We decompose gradients ∇ f mσ mk ( j ) ( x k,i − 1 ) (see Line 10 of Algorithm 2) into ∇ f mσ mk ( j ) ( x k, 0 ) plus noise, then aggregate all updates over an epoch to get “one big step of GD plus noise”:
x k +1 , 0 = x k, 0 − ηN ∇ F ( x k, 0 ) + η 2 r k .
We bound the noise r k using Lemma 8 (Appendix D.6), which is our extension of the Hoeffding-Serﬂing inequality to the mean of M independent without-replacement sums of vectors; the lemma might be of independent interest too.
Lemma 8 shows that averaging accumulated gradients over M machines reduces variance by M , which leads to the reduction by a factor of M in the bound (3).
Theorem 1 shows that for large enough epochs K (cid:38) κ , minibatch RR converges at a rate of ˜ O
( L 2 ν 2 µ 3 MNK 2 ) , with high probability.
Compared to the large-epoch rate ˜ O
( L 2 ν 2 µ 3 NK 2 ) of singlemachine RR (e.g., Ahn et al.
, we see an additional factor M in the denominator, which highlights the advantage of multiple machines.
If we compare against the with-replacement counterpart, it is known that for strongly convex and smooth F , the optimal convergence rate of minibatch
S GD is Θ( ν 2 µMNK ) , 5 which is worse than our bound (3) if K (cid:38) κ 2 .
Also notable is that the convergence rate does not depend on the heterogeneity constants (i.e., τ and ρ from Assumption 3) of the local objective functions.
This observation that minibatch RR is “immune” to heterogeneity is consistent with minibatch S GD in the with-replacement setting (Woodworth et al., 2020b).
Epoch vs communication complexity.
One might wonder why (3) does not have the batch size B .
In (3), we wrote convergence rates in terms of epochs K , which captures the gradient computation complexity because the same number of gradients are evaluated in a single epoch regardless of B .
If we are interested in communication complexity instead, we can write (3) in terms of the number of communication rounds R :=
NKB and get a rate of ˜ O
( L 2 ν 2 N µ 3 MB 2 R 2 ) .
From these, we can also discuss the overall cost of the algorithm.
If the cost of a communication round is c c , and the cost of local gradient computations over an epoch is c e , then the total cost to obtain an (cid:15) -accurate solution is
[Equation]
omitting L and µ for simplicity.
The total cost shows that there is essentially no harm increasing the batch size B in minibatch RR, as we can get more accurate estimates of true gradients as B becomes larger.
In the next subsection, we will see that this is not the case in local RR.
What about K (cid:46) κ ?
We remark that all upper bounds in this paper hold only for the “large-epoch” regime, where K (cid:38) κ .
Such requirements are common in the literature of without-replacement
SGD (Haochen & Sra, 2019; Nagaraj et al., 2019; Rajput et al., 2020; Ahn et al., 2020), and there is a recent result (Safran & Shamir, 2021) suggesting that faster convergence of without-replacement
SGD may not be possible in the K (cid:46) κ regime.
We defer a more detailed discussion on this regime to Section 4, after Theorem 3.
3. U PPER BOUND FOR LOCAL RR
Next, we are interested in how fast local RR can converge, what is the optimal batch size B , and whether local RR can be as fast as minibatch RR.
Theorem 2 (Upper bound for local RR) .
Suppose that local RR has parameters satisfying Assumption 1.
For any F ∈ F obj ( L, µ, ν, τ, ρ ) , consider running the algorithm using step-size η = log( MNK 2 ) µNK for epochs K ≥ 7 ρκ log( MNK 2 ) .
Then, with probability at least 1 − δ ,
[Equation]
Proof.
The proof is in Appendix D.3.
We take the same “big GD step plus noise” approach as in
Theorem 1; however, due to local updates, bounding the noise is much more involved.
In the proof, we obtain the epoch update y k +1 ,
=
y k, 0 − ηN ∇ F ( x k, 0 )
+ η 2 r k, 1 + η 2 r k, 2 − η 3 r k, 3 , where r k,and r k, 3 contain errors introduced by local updates.
Noise from local updates accumulates over B iterations, which cannot be remedied by averaging over M machines.
They result in two additional terms in the rate (5), one from intra-machine variance and the other from heterogeneity.
3.2. D ISCUSSION OF T HEOREM
Let us compare our high-probability bound (5) with existing in-expectation bounds.
For strongly convex F , the corresponding last-iterate bound of local S GD is ˜ O
( Lν 2 µ 2 MNK + L
[Equation]
2 B
[Equation]
N
[Equation]
K 2 + L [Equation] B
[Equation] [Equation]
(Khaled et al., 2020; Spiridonoff et al., 2020;
Qu et al., 2020).
Notice that (5) is better than this with-replacement bound when K (cid:38) κ .
For average iterates , there are known bounds
˜ O
[Equation]
2 µMNK
[Equation]
Lν 2 B µ 2 N 2 K 2 + Lτ 2 B 2 µ 2 N 2 K 2 ) 6 (Koloskova et al., 2020;
Woodworth et al., 2020b) which are smaller than the last-iterate bound by a factor of κ .
It is unclear if averaging iterates could improve our rate, because most such analyses exploit Jensen’s inequality, which we cannot use for nonconvex F .
Dependence on τ and ρ .
Out of the two heterogeneity constants τ and ρ (Assumption 3), ρ does not appear in (5), and it only affects the epoch requirement K (cid:38) ρκ .
Consider the case τ = 0 and ρ > 1 , which is heterogeneous but in the “interpolation regime,” because ∇ F m ( x ) = 0 whenever ∇ F ( x ) = 0 .
In such a case, the rate (5) is equal to the homogeneous case.
Using B = Θ( N ) is no better than single-machine.
A close look at Theorem 2 reveals a rather surprising fact.
Even in the homogeneous case ( τ = 0 ), if we choose B = Θ( N ) , then local RR converges at the rate of ˜ O
( 1 NK 2 ) : the same rate as the single-machine RR!
In Section 4, we show that this observation is not due to a suboptimal analysis; the rate ˜ O ( 1 NK 2 ) is tight for B =
Θ( N ) .
Trade-off in the choice of B .
As done for Theorem 1, we can compute from (5) that the total cost of local RR for (cid:15) -accuracy is (omitting L and µ for simplicity)
[Equation]
Note that for local RR, there exists a trade-off between communication and epoch complexity in the choice of B .
If B is too small, this reduces the number of epochs required but increases communication costs.
On the other hand, if B is too large, this reduces communication rounds but errors that accumulate in local updates get severer, resulting in the need for more epochs.
Hence, the optimal choice of B must balance the two complexity measures.
The existence of this trade-off is indeed different from minibatch RR where larger B always reduces the total cost C minibatch ( (cid:15) ) .
When can local RR match minibatch RR?
Comparing the convergence rates (3) and (5), we can identify some regimes in which local RR converges as fast as minibatch RR.
In a nutshell, if machines are not too heterogeneous and communication happens frequently, then local RR can have the same upper bound as minibatch RR.
For example, if B is chosen to be a constant, M (cid:46) N , and τ (cid:46) ν (cid:112) N/M , then the ˜ O
( L 2 ν 2 µ 3 MNK 2 ) term in (5) becomes the dominating factor and hence matches (3).
Another example of such a regime is when B (cid:46) NM and τ (cid:46) ν (cid:112) M/N .
Note that this comparison assumes that the same values of B are chosen for both algorithms.
Also, such “frequent communication” regimes are favorable if the communication cost c c is small.
Can local RR ever beat minibatch RR?
The upper bounds (3) and (5) indicate that local RR is always no better than minibatch RR, at least for the function class F obj ( L, µ, ν, τ, ρ ) .
This is in fact consistent with Woodworth et al.
(2020a;b), because the authors identify a regime where local S GD performs better than minibatch S GD for convex objective functions, but fail to do so for strongly convex functions.
However, as was also pointed out in Woodworth et al.
(2020a), there is a simple extreme scenario in which local RR can be faster: when ν
≈ τ ≈ 0 and ρ
≈ 1 .
In this case, we have f mi ≈ F for all m and i , so local RR corresponds to NK steps of GD, whereas minibatch RR corresponds to NKB steps of GD.
Clearly, local RR will converge faster, exploiting the advantage of more updates.
Finding out other such regimes is an important future direction.
M ATCHING LOWER BOUNDS
In Section 3, we presented large-epoch upper bounds (i.e., for K (cid:38) κ ) for constant step-size minibatch and local RR.
In this section, we prove matching lower bounds to show that the upper bounds are tight, in all factors except L and µ .
We use Ω( · ) to hide universal constants in lower bounds.
4. L OWER BOUND FOR MINIBATCH RR
Theorem 3 (Lower bound for minibatch RR) .
Suppose that minibatch RR has parameters satisfying
Assumption 1.
Additionally, assume that N is a multiple of 2 .
Then, there exist large enough constants c 1 , c 2 > 0 such that the following holds:
For L and µ satisfying κ =
Lµ ≥ c 1 , there exists a function
F ∈ F cmp
( L, µ, ν, 0)
such that for any constant step-size η ,
[Equation]
Proof.
We prove Theorem 3 in
Appendix E.
The proof is an extension of Rajput et al.
; Safran & Shamir (2020; 2021) to minibatch RR.
We will sketch some key intuitions after Theorem 4.
First notice that the function F is from F cmp ( L, µ, ν, 0) , where all the machines are component-wise homogeneous.
As seen in Deﬁnition 1, F cmp ( L, µ, ν, 0) ⊂ F obj ( L, µ, ν, τ, ρ ) for any τ ≥ 0 and ρ ≥ 1 , so Theorem 3 provides a lower bound for F cmp ( · ) and F obj ( · ) , with arbitrary heterogeneity constants.
We assume that N is even because we construct functions g 1 and g 2 such that f mi := gif i ≤ N 2 , and f mi :=
g
if i > N 2 .
One can remove this assumption by using a zero function when
N is odd (see e.g., Safran & Shamir (2020)).
It is rather unsatisfactory that our theorem requires large enough constants c 1 and c 2 ; we believe a tighter analysis can relax this restriction.
Theorem 3 proves lower bounds for two different regimes: K (cid:38) κ and K (cid:46) κ .
In the large-epoch regime ( K (cid:38) κ ), we can observe that the lower bound Ω( ν 2 µMNK 2 ) matches the upper bound (3) in
Theorem 1, modulo a factor of κ 2 .
Tightening the κ 2 gap between upper and lower bounds is left for future work.
In the small-epoch regime ( K (cid:46) κ ), we observe that the lower bound Ω( ν 2 µMNK ) exactly matches the convergence rate of (with-replacement) minibatch S GD ; hence, the lower bound implies that minibatch RR has no hope for faster convergence than minibatch S GD , at least in the constant step-size and small-epoch regime.
This observation is in line with Safran & Shamir (2021).
Upper bounds for K (cid:46) κ ?
Even for single-machine RR ( M = 1 ), proving an upper bound that matches the small-epoch lower bound Ω( ν 2 µNK ) still remains a challenge.
Nagaraj et al.
(2019, Theorem 2) prove an upper bound for non-quadratic strongly convex functions that matches Ω( ν 2 µNK ) if
NK (cid:38) κ 2 ; however, they use sufﬁx averaging, so it is not directly comparable to Theorem 3 which considers last iterates.
Safran & Shamir (2021) prove upper bounds for quadratic strongly convex functions, but assume that their Hessian matrices commute.
For noncommutative cases, proving a small-epoch upper bound seems to require some form of matrix AM-GM inequalities, whose availability is an open problem (Recht & R´e, 2012; Lai & Lim, 2020; De Sa, 2020; Yun et al., 2021).
Remark 1 (Strong convexity in construction) .
We note that all lower bounds in this paper are constructed with strongly convex functions, a stronger assumption than PŁ functions (2).
Thus, our lower bounds are also applicable to strong convexity counterparts of F obj ( · ) and F cmp ( · ) .
4. L OWER BOUNDS FOR LOCAL RR
In this subsection, we present lower bounds for local RR.
We prove two bounds that correspond to homogeneous and heterogeneous cases.
By combining the two bounds, we get a lower bound that matches our upper bound (5) in Theorem 2 up to a factor of κ 2 .
Theorem 4 (Lower bound for local RR: homogeneous case) .
Suppose that local RR has parameters satisfying Assumption 1.
Additionally, assume that B is a multiple of 4 .
Then, there exist large enough constants c 3 , c 4 > 0 such that the following holds:
For L and µ satisfying κ =
Lµ ≥ c 3 , there exists a function
F ∈ F cmp
( L, µ, ν, 0)
such that for any constant step-size η ,
[Equation]
Proof.
The proof is in Appendix G. For the large-epoch lower bounds in Theorems 3 and 4, we use “skewed” quadratics f mi ( x ) =
( L 1 x ≤ 0 + µ 1 x> 0 ) x 2
+ z i νx , where z i =
+1 if i ≤ N
and z i = − 1 otherwise.
For x ≈ 0 , the imbalance results in a “drift” towards positive x , whose strength is approximately proportional to the absolute value of partial sums of random permutations over N+1 ’s
and N 2 − 1 ’s. By averaging the sums over M machines (minibatch RR), their absolute values shrink by 1 √ M ; in contrast, if each machine makes local updates (local RR), the magnitude of the drift cannot be reduced with M , because we average after local iterates already have taken B “big” steps.
The proof uses techniques from Rajput et al.
Proposition 5 (Lower bound for local RR: heterogeneous case) .
Suppose that local RR has parameters satisfying Assumption 1.
Additionally, assume that B is a multiple of 2 and κ =
Lµ ≥ 2 .
Then, there exists a function
F ∈ F obj
( L, µ, 0 , τ, 1) such that for any constant step-size η ,
[Equation]
Proof.
We note that Proposition 5 is almost identical to Theorem II of Karimireddy et al.
(2020); however, we provide a proof speciﬁc to our algorithm in Appendix I.
Theorem 4 constructs a component-wise homogeneous function from F cmp ( L, µ, ν, 0) and Proposition 5 constructs a heterogeneous function from F obj ( L, µ, 0 , τ, 1) .
Since F cmp ( L, µ, ν, 0) ∪
F obj ( L, µ, 0 , τ, 1) ⊂
F obj ( L, µ, ν, τ, ρ ) for any ρ ≥ 1 , combining (8) and (9) for the K ≥ max { c 4 κ, MBN } case gives a lower bound Ω(max {
[Equation]
} ) that matches the large-epoch upper bound (5) in Theorem 2, up to a factor of κ 2 .
When κN (cid:38) MB , c 4 κ becomes the dominating term in the max , in which case the threshold in (8) is Θ( κ ) .
Tightening the κ 2 gap as well as removing additional requirements such as κ ≥ c 3 and κN (cid:38) MB are left for future work.
Using B = Θ( N ) does not help, indeed.
In Section 3.2.1, we observed that if B = Θ( N ) , then even in the homogeneous case ( τ = 0 ), local RR converges at the rate of ˜ O
( 1 NK 2 ) .
This is the same rate as single-machine RR, meaning the efforts by M − 1 machines become meaningless.
Our lower bound (8) shows that ˜ O ( 1 NK 2 ) is in fact the best we can hope for (treating L and µ as constants).
In order to make the best use of M machines, B should be smaller than Θ( N ) , as suggested in
Section 3.2.1.
In an existing work, Mishchenko et al.
(2021) consider local RR with B =
N as a special case of a proximal algorithm.
In Theorem 8 of Mishchenko et al. (2021), the authors claim “the convergence bound improves with the number of devices involved” because the bound has a factor of M in the denominator.
However, at least under our assumption, this is not the case; if we apply our Assumption 3 to upper-bound their σ ∗ , the term “ Nσ 2 ∗ ” in the numerator grows linearly with M .
Hence, our bounds do not contradict Mishchenko et al.
; see Appendix A for details.
Remark 2 (Small-epoch bound is likely loose) .
We note that while we focused on deriving a matching large-epoch lower bound, we did not try hard to tighten the small-epoch lower bound.
Our small-epoch lower bound in (8) misses a term (such as ν 2 B µN 2 K 2 ) that corresponds to the error from local updates.
We leave investigations on small-epoch lower and upper bounds for future work.
S YNCHRONIZED SHUFFLING : HOW TO BYPASS LOWER BOUNDS
Recall from the total complexity of minibatch RR (4) that the total cost shrinks with a factor of 1 √ M .
Using M machines, we are only getting a √ M -factor speedup.
Ideally, we hope to see a linear speedup , i.e., cost inverse proportional to M .
Hence, Theorem 1 falls short of achieving this goal, and our lower bound in Theorem 3 conﬁrms that linear speedup is indeed impossible.
In this section, we show that the desired linear speedup is possible, at least in some special cases.
We consider the component-wise near-homogeneous case (i.e., Assumption 4 with small λ ) and discuss how a simple modiﬁcation to minibatch and local RR can let us “break” the lower bounds and achieve linear speedup.
This comes at a cost of broadcasting permutations: at the beginning of the k -th epoch, the server samples σ ∼ Unif( S N ) and π ∼ Unif( S M ) , and broadcasts them to the machines.
Then, local machines choose their permutations σ mk to be shifted versions of σ ,i.e., σ mk ( i ) :
= σ (cid:0)(cid:0) i + NM π ( m ) (cid:1) mod N (cid:1) .
We call this trick synchronized shufﬂing , denoted as
S YNC S HUF .
Please revisit Algorithms 1 and 2 for the precise descriptions of the modiﬁed algorithms local RR with S YNC S HUF and minibatch RR with S YNC S HUF , respectively.
The intuition why this should help is simple.
In the proof of RR, we aggregate the component gradients over an epoch (i.e., N iterations) to write it as a full gradient plus noise.
If we are in the component-wise homogeneous setting and permutations are synchronized, then instead of aggregating N component gradients on a single machine, we can aggregate NM component gradients on
M machines to get a full gradient.
This allows us to reduce the “noise” from without-replacement sampling.
We emphasize here that we do not necessarily set B = NM to get a full gradient every time; our analysis works for arbitrary B and M , as long as both divide N .
See Appendix B for a detailed illustration of S YNC S HUF ; also, see Appendix C for experiments showing its effectiveness.
The idea of synchronized shufﬂing is similar to approaches in distributed learning that shufﬂe and partition datasets and distribute them to local machines (see e.g., Lee et al.
; Meng et al.
In contrast, we do not communicate data, but communicate how to permute datasets stored in local machines.
Meng et al.
(2017, Theorem 3.3) provide an analysis for a distributed method similar to minibatch RR, but fail to show convergence to global minima in strongly convex cases.
We also note that an independent concurrent result (Szlendak et al., 2021) uses the same idea as
S YNC S HUF to build compressors for communication-efﬁcient distributed optimization.
5. U PPER BOUNDS FOR MINIBATCH AND LOCAL RR WITH S YNC S HUF
With S YNC S HUF , we can show that the M ’s appearing in the convergence rates ((3) and (5)) in
Theorems 1 and 2 can be replaced with M 2 , for a more stringent function class F cmp ( · ) that requires bounded component-wise inter-machine deviation (Assumption 4).
Theorem 6 (Upper bound for minibatch RR with S YNC S HUF ) .
Suppose that minibatch RR with
S YNC S HUF has parameters satisfying Assumption 1.
Additionally assume that M divides N .
For any F ∈ F cmp ( L, µ, ν, λ ) , consider running the algorithm using step-size η = B log( M 2 NK 2 ) µNK for epochs K ≥ 6 κ log( M 2 NK 2 ) .
Then, with probability at least 1 − δ ,
[Equation]
The proof of Theorem 6 is presented in Appendix D.4.
One can check that if the componentwise deviation constant λ satisﬁes λ (cid:46)
[Equation]
MN (i.e., near-homogeneous), then the rate (10) becomes ˜ O
( 1 M 2 NK 2 ) .
It is then easy to conﬁrm that M machines reduce total costs by 1 M —a linear speedup.
A similar speedup can be shown for local RR.
In Appendix D.5, we prove that
Theorem 7 (Upper bound for local RR with S YNC S HUF ) .
Suppose that local RR with S YNC S HUF has parameters satisfying Assumption 1.
Additionally assume that M divides N .
For any F ∈
F cmp ( L, µ, ν, λ ) , consider running the algorithm with step-size η = log( M 2 NK 2 ) µNK for epochs K ≥ 7 κ log( M 2 NK 2 ) .
Then, with probability at least 1 − δ ,
[Equation]
We can similarly check that if B (cid:46) NM 2 and λ (cid:46) ν √ MN , i.e., frequent communication and nearhomogeneity, then the ˜ O ( 1 M 2 NK 2 ) term dominates in (11), and hence gives a linear speedup that matches the best rate of minibatch RR with S YNC S HUF (10).
Nevertheless, we note again that for local RR, such a small B is favorable only when the communication cost c c is small (recall (6)).
C ONCLUSION
We studied convergence bounds for local RR and minibatch RR, which are the practical withoutreplacement versions of local and minibatch S GD studied in the theory literature.
For smooth functions satisfying the Polyak-Łojasiewicz condition, we showed large-epoch convergence bounds for minibatch and local RR that are faster than their with-replacement counterparts.
We also proved matching lower bounds showing that our convergence analysis is tight.
We also proposed a simple modiﬁcation called synchronized shufﬂing that leads to convergence rates faster than our lower bounds in near-homogeneous settings.
Immediate future research directions include extension to small-epoch regimes, as well as to general convex and nonconvex functions.
E THICS S TATEMENT
This paper develops theoretical guarantees for popular distributed stochastic optimization algorithms.
Therefore, the authors do not see any particular concerns related to its ethical aspects or future societal consequences.
R EPRODUCIBILITY S TATEMENT
This paper is a theoretical work, without any experimental results.
Deﬁnitions and assumptions are provided in Section 2.
Our theoretical contributions as well as some additionally required assumptions are clearly stated in Sections 3, 4, and 5.
Complete proofs of all the theorems are provided in the appendix.
C ONTENTS 1 Introduction
Our contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Problem setup
3 Convergence analysis of minibatch and local RR 3.
Upper bound for minibatch RR . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Upper bound for local RR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 Matching lower bounds 4.
Lower bound for minibatch RR . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Lower bounds for local RR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Synchronized shufﬂing: how to bypass lower bounds 5.
Upper bounds for minibatch and local RR with S YNC S HUF . . . . . . . . . . . .
Conclusion
A Comparisons with assumptions and rates in existing results
B More detailed illustration of synchronized shufﬂing
C Experimental results
C.1 S YNC S HUF improves convergence of minibatch/local RR . . . . . . . . . . . . .
C.2 Local RR becomes closer to single-machine RR as B → N . . . . . . . . . . . .
C.3 With- vs. without-replacement sampling . . . . . . . . . . . . . . . . . . . . . . .
D Proofs of upper bounds
D.1 Proof outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of upper
bound for minibatch RR (Theorem
D.3 Proof of upper bound for local RR (Theorem
D.4 Proof of upper bound for minibatch RR with S YNC S HUF (Theorem
D.
Proof of upper bound for local RR with S YNC S HUF (Theorem
D.6 A generalized vector-valued Hoeffding-Serﬂing inequality
D.7 How can we avoid uniform bounds over R d in our assumptions?
E Proof of lower bound for minibatch RR (Theorem 3)
E.1 Lower bound for η ≤
[Table] [Equation] [Equation]
E.2 Lower bound for η ≥ [Table]
and
[Table] [Equation]
E.3 Lower bound for η ≥ [Table] µNK and
[Table]
F Proofs of helper lemmas for Appendix E
F.
Proof of Lemma 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F
Proof of Lemma 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Proof of Lemma 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Proof of Lemma 14 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Proof of Lemma 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G Proof of lower bound for local RR: homogeneous case (Theorem 4)
G
[Table]
1 Lower
[Table]
for η
[Table]
[Table] [Table]
≥
[Table]
H Proofs of helper lemmas for Appendix G
H.1 Proof of Lemma 16 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.2 Proof of Lemma 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.3 Proof of Lemma 18 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.4 Proof of Lemma 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I
Proof of lower bound for local RR: heterogeneous case (Proposition 5)
I.
Lower bound for 1 8 µNK ≤ η ≤ 1 8 µB and 1 8 µB ≤ η ≤ 1 µ . . . . . . . . . . . . . . .
I.
Lower bound for η ≤ 1 8 µNK and η ≥ 1 µ . . . . . . . . . . . . . . . . . . . . . . .
A C OMPARISONS WITH ASSUMPTIONS AND RATES IN EXISTING RESULTS
In this section, we compare our assumptions and convergence bounds against other existing results mentioned in the main text.
Most existing results that study independent and unbiased gradient estimates state their assumptions in terms of the expectation over the randomness in the estimate; for such assumptions, we adapt them to our ﬁnite sum setting in order to make for easier comparison.
Heterogeneity assumptions.
We start by discussing our deﬁnition of objective-wise heterogeneity (Assumption 3), namely that there exist τ ≥ 0 and ρ ≥ 1 such that
[Equation]
Perhaps the most relevant to this assumption is the ( G, B ) -BGD assumption that appears in Karimireddy et al.
For all x ∈ R d ,
[Equation]
that thanks to Jensen’s inequality and a 2 + b 2 ≤ ( a + b ) 2 for a, b ≥ 0 , (13) implies (cid:32)
[Equation]
F ( x ) (cid:107) )
[Equation]
, and hence (12) with τ = G and ρ = B .
Therefore, our Assumption 3 is weaker than the ( G, B ) -
BGD assumption.
Several papers (Haddadpour & Mahdavi, 2019; Li et al., 2020b) use the same assumption (13), with G = 0 ; therefore, Assumption 3 also subsumes the heterogeneity assumption posed in these papers.
Note that G = 0 implies that, the minima for F are also the minima for F m , for every m , and hence G = 0 results in a weak form of heterogeneity.
Some papers (Yu et al., 2019; Li et al., 2020c; Qu et al., 2020) assume bounded local gradients: for all m ∈
[ M ] and x ∈ R d ,
[Equation]
and this in fact implies the ( G, 0) -BGD assumption (13).
To see why, from Jensen’s inequality
[Equation]
Therefore, (14) is a stronger assumption for objective-wise heterogeneity than Assumption 3.
In Theorem 3 of Woodworth et al. (2020b), the authors use the following assumption on heterogeneity: for all x ∈ R d ,
[Equation]
Noting that
[Equation]
F ( x ) (cid:107) 2 , we can see that (15) implies (13) and hence Assumption 2 (12), with τ = ¯ ζ and ρ = 1 .
Indeed, there are also some results that make weaker heterogeneity assumptions than ours (12), by requiring bounded deviation only at the global optimum x ∗ .
Given the global optimum x ∗ of F ,
Koloskova et al.
[Equation]
2020) deﬁne
[Equation]
and use this constant in their bounds.
Khaled et al.
(2020) also deﬁne a similar quantity that can capture heterogeneity, but does not provide a result on strongly convex functions in the heterogeneous setting.
While assuming bounded ¯ ζ 2 ∗ (16) is weaker than Assumption 3 in the sense that only a bound at x ∗ is required, we note that these assumptions cannot be applied easily in nonconvex settings; in fact, for nonconvex (but not necessarily PŁ) functions, Koloskova et al.
(2020) also use (13) as their heterogeneity assumption.
Intra-machine variance assumptions.
We next consider our notion of intra-machine deviation (Assumption 2), namely that there exists ν ≥ 0 such that for all m ∈
[ M ] and i ∈ [ N ] ,
[Equation]
[Equation]
In many existing results using independent and unbiased gradient estimates (Karimireddy et al., 2020; Yu et al., 2019; Li et al., 2020c; Qu et al., 2020; Woodworth et al., 2020a;b; 2021), the bounded variance assumption is adopted: For all m ∈ [ M ] and x ∈ R d ,
[Equation]
[Equation]
We note that the bounded local gradients assumption (14) also implies (18), by
N
N (cid:88) i =
[Equation]
Some other papers (Haddadpour et al., 2019; Haddadpour & Mahdavi, 2019; Spiridonoff et al., 2020) consider a generalized version of (18), namely that for all m ∈
[ M ] and x ∈ R d ,
[Equation]
for c, σ ≥ 0 .
There are other papers (Koloskova et al., 2020; Khaled et al., 2020) that use intramachine variance at the global minimum x ∗ in their bounds.
Koloskova et al.
(2020) deﬁne
[Equation]
for the global minimum x ∗ , and use it in their strong convexity and convexity bounds.
Khaled et al.
(2020) deﬁne
[Equation]
which is used in their bound on strongly convex functions in homogeneous cases.
Note that for homogeneous cases, ∇ F m ( x ∗ )
= ∇ F ( x ∗ )
= 0 , so ¯ σ 2 ∗ =
σ 2opt .
We note that in contrast to our discussion on inter-machine deviation (Assumption 3), the intramachine variance assumptions in the existing literature are weaker than our Assumption 2.
However, we utilize our stronger assumption to prove our high-probability upper bounds, which is a departure from in-expectation bounds in the literature.
Existing upper bounds on local S GD .
In the discussion after Theorem 2, we mentioned some recent upper bounds on (with-replacement) local S GD .
We make more detailed comparisons here.
For the reader’s convenience, we restate our theorem on local RR below.
Theorem 2 (Upper bound for local RR) .
Suppose that local RR has parameters satisfying Assumption 1.
For any F ∈ F obj ( L, µ, ν, τ, ρ ) , consider running the algorithm using step-size η = log( MNK 2 ) µNK for epochs K ≥ 7 ρκ log( MNK 2 ) .
Then, with probability at least 1 − δ ,
[Equation]
We start with last-iterate bounds in homogeneous cases.
Theorem 3 and Corollary 3 of Khaled et al.
(2020) consider local S GD on µ -strongly convex and L -smooth F .
Khaled et al.
(2020) allow variable synchronization intervals, where the maximum interval is upper-bounded by B .
In this setting, Corollary 3 of Khaled et al.
(2020) shows that local S GD after T total local update steps yield
[Equation]
where ¯ x i is the average over all M machines’ i -th local iterates, and σ opt is from (22).
Noting that
T corresponds to NK in local S GD and σ opt corresponds to ν in Assumption 2, (23) is comparable to an upper bound
[Equation]
(cid:19) .
Here, we do not compare F ( y 0 ) − F ∗ MNK 2 and L (cid:107) ¯ x 0 − x ∗ (cid:107) 2 Tbecause in Theorem 2, the term F ( y 0 )
− F ∗ MNK 2 can be made arbitrarily “small” (e.g., F ( y 0 ) − F ∗ ( MNK ) l for any l ∈ N ) by changing the log factor in η .
A similar homogeneous, strongly convex, and smooth setting is considered in Spiridonoff et al.
(2020), under a intra-machine variance assumption deﬁned in (20).
Theorem 1 of Spiridonoff et al.
(2020) proves general theorem statement for arbitrary synchronization intervals.
If we specialize to constant interval B , Corollary 1 of Spiridonoff et al.
(2020) gives
[Equation]
where β ≥ 2 κ 2 is a constant deﬁned to choose algorithm parameters such as the step-size.
Noting again that T corresponds to NK and σ in (20) is comparable to ν in Assumption 2, (25) also translates to (24).
The next last-iterate bound we compare against is Qu et al.
Theorem 1 of Qu et al.
(2020) uses bounded intra-machine variance assumption (18) and bounded gradient assumption (14).
Specializing Theorem 1 of Qu et al.
(2020) to full device participation and uniform weight ( p 1 = · · · = p N = 1 N ) case, their bound reads
[Equation]
Recalling that T corresponds to NK , σ to ν in Assumption 2, and G to the heterogeneity constant τ in Assumption 3 and also ν (due to (19)), (26) translates to
[Equation]
Comparing the local S GD last-iterate bounds (24) and (27) against our local RR bound (5) in Theorem 2, we can see that the last iterate of local RR satisﬁes a smaller upper bound as soon as K ≥ κ .
Admittedly, this is not a fully rigorous comparison given the differences in assumptions and types of bounds; nevertheless, we believe that the comparison at least provides some degree of evidence for faster convergence of local RR than local S GD .
It is also interesting to see that the “error from local updates” terms match in with- and without-replacement bounds.
Next, we review existing average-iterate bounds mentioned in the main text, which are better than the last-iterate bounds.
Koloskova et al.
(2020) present a unifying framework for analyzing distributed optimization algorithms over networks, which can specialize to local S GD .
For µ -strongly convex and L -smooth F , Theorem 2 of Koloskova et al.
(2020) shows that
[Equation]
where ˆ x is some weighted average of iterates and ¯ ζ 2 ∗ and ¯ σ 2 ∗ are deﬁned in (16) and (21), respectively.
Noting that T corresponds to NK , ¯ ζ ∗ to τ in Assumption 3, and ¯ σ ∗ to ν in Assumption 2 (although our assumptions are stronger), we can see that the bound (28) for large enough T can be translated into
[Equation]
Notice that (29) is smaller than the last-iterate bounds (24) and (27) by a factor of κ .
Similarly,
Theorem 3 of Woodworth et al.
(2020b) proves that for µ -strongly convex and L -smooth F ,
[Equation]
for some weighted average of iterates ˜ x .
Here, ¯ σ 2 ∗ , σ 2 , and ¯ ζ 2 are as deﬁned in (21), (18), and (15), respectively.
Substituting ν to its comparable constants ¯ σ ∗ and σ , and τ to ¯ ζ , we can similarly check that (30) can be converted to (29).
Comparison to Mishchenko et al.
In Mishchenko et al.
(2021), the authors study a proximal algorithm referred to as Proximal Random Reshufﬂing (ProxRR), and obtain a distributed optimization algorithm called FedRR as a special case.
If we set R ≡ 0 in FedRR, the algorithm then is equal to local RR with B = N , i.e., the one that synchronizes only after one entire epoch.
Assuming that all component functions f mi ( x ) are µ -strongly convex 9 and L -smooth, and
objectivewise homogeneity F 1 = · · · = F m = F , the authors obtain Theorem 8 (Mishchenko et al., 2021), which states that [Equation]
[Equation]
where y 0 and y K are the initialization and last iterate of the algorithm, and σ 2opt was deﬁned above in (22).
The term MNσ 2opt in (31) corresponds to the term “ Nσ 2 ∗ ” as per the notation in Mishchenko et al.
If we apply our Assumption 3 to bound σ 2opt , we get σ 2opt ≤ ν 2 , which reduces the last term in (31) to η 2 Lν 2 N µ .
If we substitute η = log( [Equation] to the bound (31), we get
[Equation]
which translates to the same convergence rate on F ( y K ) − F ∗ as single-machine RR.
For the heterogeneous setting, applying Lemma 3 of Mishchenko et al.
(2021) to Theorem 2, we can obtain
[Equation]
where ¯ σ 2 ∗ was deﬁned in (21).
Note that in Lemma 3 (Mishchenko et al., 2021), the function “ F m ” in the authors’ notation is equal to NF m in our notation.
Recall that the ( G, B ) -BGD assumption (13) is “comparable” to Assumption 3 (12).
If we apply (13) to the bound (33), we get
M (cid:80) Mm =1 (cid:107)∇ F m ( x ∗ ) (cid:107) 2 ≤ G 2 .
Similarly, if we apply Assumption 3, we get ¯ σ 2 ∗ ≤ ν 2 .
Substituting these upper bounds and η = log( MNK 2 ) µNK to (33) gives
[Equation]
and after translating this into a bound on function value, we get an upper bound ˜ O
( L 2 ν
[Equation]
µ [Equation] NK 2 + L
[Equation]
G
[Equation]
3 K 2 ) which in fact matches our upper bound (5) in Theorem 2 when we set B = N .
The two upper bounds obtained for homogeneous (32) and heterogeneous (34) settings indicate that, at least under assumptions on intra- and inter-machine deviation such as ours, the claimed advantage that “the convergence bound improves with the number of devices involved” (Mishchenko et al., 2021) is not achievable.
As our lower bound shows, one needs to choose B smaller than N in order to get the most out of parallelism.
That being said, since Mishchenko et al.
(2021) is free of uniform intra- and inter-machine deviation assumptions, there may still exist certain scenarios where multiple machines can speed up performance even with B = N .
B M ORE DETAILED ILLUSTRATION OF SYNCHRONIZED SHUFFLING
In this section, we provide a more detailed explanation on synchronized shufﬂing that we introduced in Section 5.
For the illustration, let us consider the component-wise homogeneous case.
Component-wise homogeneity means that all the machines have the same set of components: f 1 i = f 2 i = · ·
· = f Mi =: f i for i ∈ [ N ] .
Hence, we have F = F m for all m ∈
[ M ] and our goal is to minimize F = 1 N (cid:80)
Ni =
1 f i .
In the proof of Theorem 1 (presented in Appendix D.2; see Appendix D.1 for a sketch), we add the component gradients over an epoch and then use the following key identity: for any permutation σ ,
[Equation] [Equation]
We use this identity (35) to represent the per-epoch progress as “one big GD step plus noise.”
For the rest of the proof we bound the “noise” term, and the key to bounding it is to upper bound the norm of summations of the following form, for i ∈
[ N/B − 1] (see (41) and (43)):
[Equation]
To
[Table]
M M
[Table]
and we use concentration bounds (44) on the ﬁrst term of the RHS, which gives a high-probability upper bound on the RHS of the form ˜ O (cid:16) ν (cid:113) iBM (cid:17) + iB (cid:107)∇ F (cid:107) (45).
The key to proving fast convergence is to make the upper bound above as small as possible.
To make the bound (45) even smaller, we wish to be able to apply the identity (35) to the summation (36).
However, under the standard way of choosing permutations σ mk independently over machines, one cannot apply the identity because we do not sum over all j = 1 , . . .
, N .
This limitation motivates our proposed technique synchronized shufﬂing , a manipulation on the choices of σ mk that lets us prove even faster convergence.
Recall the deﬁnition of synchronized shufﬂing.
At the beginning of the k -th epoch, the server samples σ ∼ Unif( S N ) and π ∼ Unif( S M ) , and broadcasts them to the machines.
Then, local machines choose their permutations σ mk to be shifted versions of σ : σ mk ( i ) :
= σ (cid:0)(cid:0) i + NM π ( m ) (cid:1) mod N (cid:1) .
Now set N = 6 and M = 3 .
Assume for simplicity that the permutation π ∈ S 3 of machines satisﬁes π ( m ) = m for m ∈ [3] .
10 Suppose the server samples σ = ( σ (1) , σ (2) , σ (3) , σ (4) , σ (5) , σ (6)) and broadcasts it.
Under synchronized shufﬂing, the local machines choose
[Equation]
One can see that each permutation is a shifted version of σ , with an offset that is a multiple of
NM = 2 .
Now consider adding ∇ f σ mk ( i ) over m = 1 , 2 , 3 and j = 1 , 2 (in fact, any two consecutive j ’s will do).
By synchronized shufﬂing, we get a summation over all N = 6 component functions, which by (35) gives us the full gradient:
[Equation]
The point here is that the permutation identity (35) can be applied to the summations (36) to further reduce their norm bounds.
This is in contrast to sampling independent σ mk ’s where one cannot apply (35).
For this reason, synchronized shufﬂing signiﬁcantly reduces the noise that comes from without-replacement sampling, thus resulting in faster convergence rates in (near-)homogeneous cases.
C E XPERIMENTAL RESULTS
In this section, we present some simple numerical experiments that support our theoretical analysis.
We evaluate the performance of the algorithms considered in this paper on the “hard instance” constructed in our lower bounds (Theorems 3 and 4).
Our hard instance F ∈ F cmp ( L, µ, ν, 0) is a function in the component-wise homogeneous setting, where all the machines have the same set of local component functions: f 1 i = f 2 i = · · · = f
Mi =: f i .
In the proofs of Theorems 3 and 4, we construct the global objective F ( x ) =
1 N (cid:80) Ni =
1 f i ( x ) as the following:
[Equation]
With this set of component functions, the global objective F ( x ) =
( L 1 x ≤ 0 + µ 1 x> 0 ) x 2 2 is µ -strongly convex and L -smooth with a unique global minimizer at x = 0 .
We compare the performance of the algorithms on this problem instance, with L = 100 , µ = 1 , ν = 1 , N = 768 , and M = 16 , while varying the choice of B ∈ { 1 , 4 , 16 , 64 , 256 } and
K ∈ { 1 , 3 , 5 , 7 , 10 , 30 , 50 , 70 , 100 , 300 , 500 , 700 , 1000 } .
For each value of B and K , we run the algorithms for K epochs ( KN/B communication rounds for with-replacement algorithms) starting at x 0 = 0 and return the values of F evaluated at the last iterates.
Note that the algorithms are not deterministic, because the sampling/shufﬂing schemes are random.
In order to account for randomness, for each combination of ( algorithm , B, K ) we execute 20 independent runs of the algorithm and plot the mean, ﬁrst quartile, and third quartile of the ﬁnal objective values.
In the subsequent subsections, we compare the following seven algorithms with constant step-sizes.
• Minibatch RR, η = B log( MNK 2 ) µNK ; • Local RR, η = log( MNK 2 ) µNK ; • Minibatch RR with S YNC S HUF , η = B log( MNK 2 ) µNK ; • Local RR with S YNC S HUF , η = log( MNK 2 ) µNK ; • Single-machine RR with minibatch size B , η = log( NK 2 ) µNK ; • With-replacement minibatch S GD , η = B log( MNK 2 ) µNK ; • With-replacement local S GD , η = log( MNK 2 ) µNK .
C. S YNC S HUF IMPROVES CONVERGENCE OF MINIBATCH / LOCAL RR
In Figure 1, we compare minibatch RR and local RR, with and without S YNC S HUF .
Each plot in
Figure 1 shows how the methods’ performance changes with K , for a ﬁxed value B .
Each point on the curve is the mean of the ﬁnal objective function values over 20 independent runs of the corresponding algorithm with the speciﬁc B and K , and its error bar indicates the ﬁrst and third quartiles.
Recall from our Theorems 1, 2, 6, and 7 that the four methods satisfy the following convergence bounds, in homogeneous settings (i.e., τ = λ = 0 ):
• Minibatch RR: ˜ O (cid:16)
L 2 µν 2 MNK 2 (cid:17) , • Local RR: ˜ O (cid:16) L 2 µ 3 (cid:16) ν 2 MNK 2 + ν 2 B N 2 K 2 (cid:17)(cid:17) , • Minibatch RR with S YNC S HUF : ˜ O (cid:16)
L 2 µν 2 M 2 NK 2 (cid:17) , • Local RR with S YNC S HUF : ˜ O (cid:16)
L 2 µ 3 (cid:16) ν 2 M 2 NK 2 + ν 2 B N 2 K 2 (cid:17)(cid:17) .
In fact, if B = 1 , local RR is identical to minibatch RR.
Figure 1(a) conﬁrms that this is indeed true, and also that the versions with S YNC S HUF outperforms the ones without S YNC S HUF .
This corroborates the additional M factor speedup in our bounds.
In Figure 1(b) and 1(c), we can see that as B increases, the performance of local RR with S YNC S HUF degrades and becomes closer to local RR without S YNC S HUF .
This shows that the ˜ O (cid:16) L 2 µ 3 ν 2 B N 2 K 2 (cid:17) term starts to dominate.
Also, as we increase B further, in Figure 1(c) and 1(d) we see that local RR (without S YNC S HUF ) also starts to degrade and its gap between minibatch RR becomes larger.
Again, this means that the ˜ O (cid:16) L 2 µ
[Equation] B N 2 K 2 (cid:17) term becomes the dominant factor in the local RR bound.
The performance of minibatch RR, with and without S YNC S HUF , stays relatively independent of B .
One thing to note is that for large values of B , the small-epoch behavior of minibatch RR looks rather unstable.
The choice of step-size η = B log( MNK 2 ) µNK seems to cause overshooting when B is large and K is small.
We
[Figure]
note that this does not contradict our convergence analysis because our theorems only characterize the large-epoch behavior ( K above certain thresholds) of the algorithms.
Perhaps in the small-epoch regime, our choice of η is not necessarily optimal and a smaller η is needed to prevent overshooting.
C. L OCAL RR BECOMES CLOSER TO SINGLE - MACHINE RR AS B → N
Our next set of plots presented in Figure 2 provides a comparison of minibatch RR, local RR, and single-machine RR (i.e., minibatch RR with M = 1 ).
In Theorems 2 and 4, we showed that when
B = Θ( N ) , then the convergence of local RR becomes just as fast as the single-machine RR.
[Figure]
The large-epoch performance of local RR becomes similar to that of single-shufﬂe RR as B be-
Indeed, we can observe from Figure 2 that this is really the case.
As B increases, the curve of local
RR moves closer and closer to that of single-machine RR, especially in the large-epoch regime
[Figure]
C. W ITH - VS . WITHOUT - REPLACEMENT SAMPLING
Lastly
[Figure]
in Figure 3 we compare the with-replacement and without-replacement versions of minibatch/local SGD.
In all plots, we can see that the without-replacement versions outperform withreplacement ones, at least for our problem instance.
It is also intriguing to note that the two versions perform very similarly in the small-epoch regime (for K up to ∼ 10 ), but without-replacement starts
[Figure]
to outperform for larger K ’s. This observation supports our theoretical prediction from Theoremthat in the small-epoch regime, minibatch RR can at best perform as fast as minibatch S GD
[Figure]
D P ROOFS OF UPPER BOUNDS
In this section
[Figure]
we provide proofs of our upper bounds stated in Sections 3 and 5.
We start by describing a high-level proof outline that we use for all the proofs presented in this section (Appendix D.1).
In the subsequent subsections, we prove Theorems 1, 2, 6, and 7, in the order they appeared in the main text.
The next subsection (Appendix D.6) states and proves a key lemma that gives concen- tration bounds for the mean of multiple without-replacement sums of vectors.
This general-purpose lemma can be of independent interest and it can prove useful in various other settings.
Lastly, in
Appendix D.7 we discuss how we can modify the theorem statements to remove the requirement in
Assumptions 2, 3, and 4 that they must hold for the entire R d .
Notation.
Throughout this section, we use the product notation (cid:81) in a slightly unconventional manner.
For indices i ≤ j and square matrices
A i , A i +1 , . . .
A j − 1 , A j , we use (cid:81) il =
j A l to denote the matrix product
A j A j − 1 · · · A i +
A i .
If i > j , then (cid:81) il =
j A l = I .
D. P ROOF OUTLINE
The proofs of upper bounds follow a common structure, consisting of the following three steps:
writing one epoch as one step of GD plus noise; 2. getting a high-probability upper bound on the noise term using concentration inequalities; 3.
obtaining the convergence rate using the bounds on the noise term.
We ﬁrst unroll the update equations over an epoch, and write an epoch of the algorithms as one step of gradient descent plus noise:
[Equation]
Substituting the above to the deﬁnition of L -smoothness of F and arranging terms, we obtain
[Equation]
The next step is to get high-probability upper bounds on the term (cid:107) r k (cid:107) .
This is done by applying our concentration inequality lemma (Lemma 8) to partial without-replacement sums of component gradients.
As a result, we will get upper bounds on (cid:107) r k (cid:107) and (cid:107) r k (cid:107) 2 , for k = 1 , . .
, K , which hold with probability at least 1 − δ .
In the last part, we substitute the high-probability bounds to (37) and invoke the deﬁnition of PŁ functions to get a per-epoch progress bound.
We then unroll the per-epoch inequality for all epochs k = 1 , . . .
, K .
Arranging the terms in the resulting inequality gives our desired convergence bound that holds with probability at least 1 − δ .
D. P ROOF OF UPPER BOUND FOR MINIBATCH RR (T HEOREM 1)
One epoch as one step of GD plus noise.
To simplify the notation throughout the proof, we will prove the same convergence rate for a rescaled update rule and step-size:
[Equation]
38) for i ∈ [ N/B ] and k ∈ [ K ] , and η = log( MNK 2 ) µNK .
Note that the gradient term is scaled up by B and the step-size is scaled down by B .
We will prove the convergence rate for this equivalent algorithm.
We start the proof by unrolling the update equations over an epoch and expressing the progress as
[Equation]
To this end, we decompose the gradient ∇ f mσ mk ( j ) ( x k,i − 1 ) into the signal ∇ f mσ mk ( j ) ( x k, 0 ) and noise:
[Equation]
where ∇ 2 f ( x ) denotes the Hessian of f at x , whenever it exists.
We remark that the integral exists, due to the following reason.
Since we assumed that each f mσ mk ( j ) is differentiable and smooth, its gradient ∇ f mσ mk ( j ) is Lipschitz continuous, and hence absolutely continuous.
This means that ∇ f mσ mk ( j ) is differentiable almost everywhere (i.e., ∇ 2 f mσ mk ( j ) ( x ) exists a.e.), and the fundamental theorem of calculus for Lebesgue integral holds; hence the integral exists.
To simplify notation, we deﬁne the following for all i ∈ [ N/B ] :
[Equation]
so that we can write (38) as
[Equation] [Equation]
From L -smoothness of f mi ’s, it is straightforward to check that (cid:107) H i (cid:107) ≤ LB .
Unrolling (39) for i = 1 , . .
. , N/B , it turns out that we can write
[Equation]
Due to summation by parts, the following identity holds:
[Equation]
We apply this to the last term, by substituting a i = (cid:81)
i +1 j = N/B
( I − η H j ) and b i
= g
i : η
[Equation]
i
[Equation]
 
[Equation]
N
[Equation]
B (
[Equation]

 g i =
η
N/B (cid:88)
j = g
[Equation]
− η
N/B
[Equation]
(cid:88) i =   i +2 (cid:89) t =
N/B
( I − η H t ) − i +1 (cid:89) t =
N/B ( I − η H t ) 
 i (cid:88)
j = g j = ηN ∇ F ( x k, 0 ) − η
N/B − 1 (cid:88) i =   i +2 (cid:89) t =
N/B ( I − η H t ) 
 H i + i (cid:88)
[Equation]
With the “noise” r k deﬁned as above, we can write x k +1 ,
= x k, 0 − ηN ∇ F ( x k, 0 ) + η 2 r k , as desired.
Next, it follows from L -smoothness of F that
[Equation]
[Equation]
where the last inequality used (cid:107) a + b (cid:107) 2 ≤ 2 (cid:107) a (cid:107) 2 + 2 (cid:107) b (cid:107) 2 .
Bounding noise term using concentration.
It is left to bound (cid:107) r k (cid:107) .
We have
[Equation]

[Equation] +
[Equation]
g
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation]
N/B − 1 (cid:88) i
=
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) i +
(cid:89) t =
N/B ( I − η [Equation] t ) 
 H i + i (cid:88)
j = g j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation] [Equation] (cid:88) i
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
i (cid:88)
[Equation]
[Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] (41) where the last step used (cid:107) H
i (cid:107) ≤ LB for i ∈
[ N/B ] .
Recall from the theorem statement that
K ≥ 6 κ log( MNK 2 ) and η = log( MNK 2 ) µNK .
This means that
[Equation]
Now, we use Lemma 8 to bound the norm of
[Equation]
Note that for any epoch k , the permutations σ 1 k , . . .
, σ Mk are independent of the ﬁrst iterate x k,of the epoch, and hence independent of all ∇ f mi ( x k, 0 ) .
Therefore, we can apply Lemma 8 to the partial sum (43), with v mi ← ∇ f mi ( x k, 0 ) , n ← iB , and δ ← BδNK .
By Lemma 8, with probability at least 1 − BδNK , we have
[Equation]
iBM
[Equation]
Using this concentration bound, with probability at least 1 − BδNK we have
[Equation]
i (cid:88) j =
[Equation]
iBν
[Equation]
We can now substitute (42) and (45) to (41) to get
[Equation]
which holds with probability at least 1 − δK , due to the union bound over i = 1 , .
N/B − 1 .
The bound (46) holds for all k ∈
[ K ] with probability 1 − δ if we apply the union bound over k = 1 , . . .
, K .
Next, by ( a + b ) 2 ≤ 2 a
+
b
2 , we have
[Equation]
which also holds for all k ∈
[ K ] with probability at least 1 − δ .
Getting a high-probability convergence rate.
Given our high-probability bounds (46) and (47), we can substitute them to (40) and get
[Equation]
The second term in the RHS of (48) can be bounded using ab ≤ a 2 2 + b 2 2 :
[Equation]
2 Lν
[Equation]
N
[Equation] [Equation]
Putting this inequality to (48) and noting N − B ≤ N gives
[Equation]
Recall from K ≥ 6 κ log( MNK 2 ) and η = log( MNK 2 ) µNK that ηLN ≤ 16 .
Since the inequality − 78 z + 2 z 2 + 2 z 4 ≤ − 12 z holds on z ∈ [0 , 16 ]
[Equation]
we have
[Equation]
Applying this bound to (49) results in
[Equation]
We now recall that F is µ -PŁ, so (cid:107)∇ F ( x k, 0 ) (cid:107) 2 ≥ 2 µ ( F ( x k
[Equation]
) −
[Equation]
∗ ) :
[Equation] [Equation]
Recall that (50) holds for all k ∈
[ K ] , with probability 1 − δ .
Therefore, by unrolling the inequality,
F (
[Equation]
K
[Equation]
[Equation]
Lastly, substituting η = log( MNK 2 ) µNK gives
[Equation]
Getting an in-expectation bound from the high-probability bound.
We conclude this subsection by brieﬂy describing how we can obtain an in-expectation bound from the high-probability bound we just proved.
Recall that the bound we proved above holds under the event E that all the concentration bounds used throughout the proof hold.
The key to proving an in-expectation bound is to obtain an upper bound under its complement E c , i.e., conditioned on the event that at least one of our concentration bounds does not hold.
We do so by repeating the same proof without ever using the Hoeffding-Serﬂing bounds (Lemma 8).
Of course, this leads to a much looser bound, but we can choose δ to be small enough so that the desired bound ˜ O (cid:16) L 2 ν [Equation]
[Equation]
holds in expectation.
For the version without concentration bounds, the proof proceeds in the same way until it starts diverge at (44).
Instead of applying concentration inequalities, we loosely bound the quantity as the following:
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
iB [Equation] With this bound, the RHS of the upper bound (45) on (cid:107) (cid:80) ij =1 g j (cid:107) becomes iBν + iB (cid:107)∇ F ( x k, 0 ) (cid:107) .
This results in the bounds on (cid:107) r k (cid:107) and (cid:107) r k (cid:107) 2 (corresponding to (46) and (47)) that read [Equation]
[Equation]
The rest is substituting the bounds above to (40), and going through the same steps to obtain the ﬁnal bound.
The resulting bound that corresponds to (51) is
[Equation]
, which, by substituting η = log( MNK 2 )
[Equation]
, yields
[Equation]
To ﬁnish the proof of in-expectation bound, choose δ = 1 MN .
Recall that the probabilistic event E occurs when all our concentration bounds hold.
Conditioned on E , which occurs with probability at least 1 − 1 MN , the tighter bound (52) holds, with log 1 δ replaced by log( MN ) .
The complement event E c occurs with probability at most 1 MN , under which the looser bound (53) is true.
Thus, in expectation,
[Equation]
For the remaining high-probability upper bounds proved in the paper, we can similarly follow this process to obtain matching (up to log factors) in-expectation upper bounds.
D. P ROOF OF UPPER BOUND FOR LOCAL RR (T HEOREM 2)
One epoch as one step of GD plus noise.
The update rule of local RR can be written as the following.
For k ∈ [ K ] , i ∈ [ N ] , and m ∈
[ M ] ,
[Equation]
if B does not divide i,
[Equation]
if
[Equation]
i.
[Equation]
Recall that x mk, 0 ’s are the initial points of an epoch and they all the same regardless of the machine m .
We deﬁne y k, 0 :
= x 1 k, 0 .
For any i in the range of ( l − 1) B + 1 ≤ i ≤ lB for some l ∈ [ M ] , we will use y k,l − 1 as the “pivot” and decompose the gradients into the ones evaluated at y k,l − 1 plus noise terms.
[Equation]
where the integral H mi exists due to the reason discussed in Appendix D.2.
Also note from Lsmoothness of f mi ’s that (cid:107) H mi (cid:107) ≤ L .
Using the decomposition, one can unroll the updates (54) and write y k,l in terms of y k,l − 1 in the following way:
[Equation]
for l = 1 , . . . , N/B .
Next, we again decompose the gradient ∇ f mσ mk ( i ) ( y k,l − 1 ) , this time using y k,as the pivot:
[Equation]
This decomposition allows us to rewrite (55) in the following form
[Equation]
56) where
[Equation]
Unrolling (56) for l = 1 , . .
. , N/B then gives the progress over an epoch:
[Equation]
As done in the proof of Theorem 1 (Appendix D.2)
[Equation]
we will express (59) as one step of GD on F plus some noise.
Of course, the noise terms here will be more complicated to handle than they were in Theorem 1.
Due to summation by parts, the following identity holds:
[Equation]
We apply this to the last term of (59), by substituting a i = (cid:81)
l +1 j = N/B ( I − η S j ) and b l = t l : η
[Equation]
We also apply the summation by parts to the inner summation of t l ’s (57):
[Equation]
Substituting (61) to (60) gives
[Equation]
3 , where r k, 1 , r k, 2 , and r k, 3 are noise terms deﬁned as
[Equation]
Deﬁning r k :=
r k, 1 + r k, 2 − η r k, 3 , it follows from L -smoothness of F that
[Equation]
[Equation]
Bounding noise terms using concentration.
We next bound (cid:107) r k (cid:107) by bounding each (cid:107) r k, 1 (cid:107) , (cid:107) r k, 2 (cid:107) , and (cid:107) r k, 3 (cid:107) .
From this point on, we write g mt :=
∇ f mσ mk ( t ) ( y k, 0 ) to simplify notation.
[Equation]
i =( l − 1) B + (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:32) i +2 (cid:89) t = lB
( I − η H mt ) (cid:33)
H mi + i (cid:88) t =( l − 1) B + g mt (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
≤ L (1 + ηL ) B
M
N/B (cid:88) l =
M (cid:88) m
=
lB − 1 (cid:88) i =( l − 1) B + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) i (cid:88) t =( l − 1)
B + g mt (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) , (63) where we used (cid:107) H mi (cid:107) ≤ L .
Recall from the theorem statement that K ≥ 7 ρκ log( MNK 2 ) and η = log( MNK 2 ) µNK .
This means that
[Equation]
Also note from the deﬁnition of S l (58) that (cid:107) S l (cid:107) ≤ LB (1 + ηL ) B ≤ e 1 / 7 LB , which we use to get similar bounds for the next two terms r k, 2 and r k, 3 .
[Equation]
65) and we can bound
[Equation]
NK
[Equation] [Equation]
We similarly bound the norm of the last noise term r k, 3 : (cid:107) r k, 3 (cid:107) ≤ e 1 / 7 LB (1 + e 1 / 7 ηLB ) N/B
M ×
N/B − 1 (cid:88) l
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) l (cid:88)
j =
M (cid:88) m = jB − 1 (cid:88)
i =( j − 1) B +   i +2 (cid:89) t =
jB ( I − η H mt ) 

H mi + i (cid:88) t =( j − 1) B + g mt (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
≤ e 1 / 7 L 2 B (1 + e 1 / 7 ηLB ) N/B (1 + ηL ) B
M
N/B − 1 (cid:88)
l = l (cid:88) j =
M (cid:88) m = jB − 1 (cid:88)
i =( j − 1) B + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) i (cid:88) t =( j −
) B + g mt (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 11 L 2 B 7 M
N/B − 1 (cid:88)
l = l (cid:88) j =
M (cid:88) m = jB − 1 (cid:88)
i =( j − 1) B + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) i (cid:88) t =( j −
) B + g mt (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) , (67) where the last inequality used (64), (66), and e 2 / 7 exp( e 1 / 7 / 7) ≤ 11 / 7 .
Given the bounds (63), (65), and (67), we now use Lemma 8 to get high-probability bounds for the partial sums of g mt that appear in the bounds.
For any i satisfying ( j − 1) B +1 ≤ i ≤ jB − 1 , where j ∈ [ N/B ] , and for any m ∈ [ M ] , the following bound holds with probability at least 1 − δ 2 MNK :
[Equation]
From this, with probability at least 1 − δ 2 MNK we have
[Equation]
[Equation] Similarly, for l ∈ [ N/B − 1] , the following bound holds with probability at least 1 − Bδ 2 NK :
[Equation] [Equation]
which gives us
[Equation]
By applying the union bound, with probability at least 1 − δK , the bound (68) holds for all m ∈
[ M ] and i ∈ (cid:83) N/Bj =
[( j − 1) B + 1 : jB − 1] , and the bound (69) holds for all l ∈ [ N/B − 1] .
We now substitute the bounds (68) and (69) to (63), (65), and (67) to get upper bounds for (cid:107) r k, 1 (cid:107) , (cid:107) r k, 2 (cid:107) , and (cid:107) r k, 3 (cid:107) , respectively.
First,
[Equation]
where the last inequality used (cid:80) B − 1 i =1 √ i ≤ (cid:82) B 1 √ zdz = 23 ( B 3 / 2 − 1) and Assumption 3.
For the next noise term, we have e 1 / 7 (1 + e 1 / 7 ηLB ) N/B ≤ 7 / 5 , so
[Equation]
N/B − 1 (cid:88) l =
[Equation]
N
[Equation]


[Equation]
≤ 8 Lν
[Equation]
N
[Equation]
Lastly,
[Equation]
Recalling the deﬁnition r k :=
r k, 1 + r k, 2 − η r k, 3 , we get an upper bound for (cid:107) r k
(cid:107) from (70), (71), and (72):
[Equation]
Recall again that we have K ≥ 7 ρκ log( MNK 2 ) and η = log( MNK 2 ) µNK
[Equation]
≤ 1 / 7 .
Using this and N − B ≤ N , we can further simplify (73).
[Equation]
which holds with probability at least 1 − δK .
The bound (74) holds for all k ∈
[ K ] with probability 1 − δ if we apply the union bound over k = [Equation]
, K .
Next, by ( a + b + c ) 2 ≤ 3 a 2 + 3 b 2 + 3 c 2 , we have
[Equation]
which also holds for all k ∈
[ K ] with probability at least 1 − δ .
Getting a high-probability convergence rate.
Given our high-probability bounds (74) and (75) [Equation] we can substitute them to (62) and get
[Equation]
The following terms in the RHS of (76) can be bounded using ab ≤ a 2 2 + b 2 2 :
[Equation]
[Equation]
Substituting (77) and (78) to (76) results in
[Equation] [Equation]
Again, we have K ≥ 7 ρκ log( MNK 2 ) and η = log( MNK 2 ) µNK , so ηLρN ≤ 17 .
Since the inequality − 78 z + 125 z 2 + 14725 z 4 ≤ − 12 z holds on z ∈ [0 , 17 ] , we have
[Equation]
Substituting this inequality to (79), together with 4 + 3 ηLN ≤ 317 < 92 , yields
[Equation]
− 1) 2 .
We now recall that F is µ -PŁ, so (cid:107)∇ F ( y k, 0 ) (cid:107) 2 ≥ 2 µ ( F ( y k, 0 ) − F ∗ ) : F ( y
[Equation]
+1 ,
[Equation]
Recall that (80) holds for all k ∈
[ K ] , with probability 1 − δ .
Therefore, by unrolling the inequality,
[Equation]
[Equation]
Recall that Φ := 12 N ( B 3 / 2 − 1) 5 B + 8( N 3 / 2 − B 3 / 2 ) 3 M 1 /, hence Φ 2 ≤ 288 N
B 3 / 2 − 1)
[Equation]
N
[Equation]
/ 2 − B 3 /
[Equation] [Equation]
Substituting this inequality and also η = log( MNK 2 ) µNK gives
[Equation]
with probability at least 1 − δ .
This ﬁnishes the proof.
D. P ROOF OF UPPER BOUND FOR MINIBATCH RR WITH S YNC S HUF (T HEOREM 6)
The ﬁrst part (“One epoch as one step of GD plus noise”) of the proof is identical to that of Theorem 1.
We start from the second part.
Bounding noise term using concentration.
It is left to bound (cid:107) r k (cid:107) .
As seen in (41), we have (cid:107)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
j = g j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Recall from the theorem statement that K ≥ 6 κ log( MNK 2 ) and η = log( MNK 2 ) µNK .
This means that
[Equation]
NK
[Equation]
Next, we bound the norm of i (cid:88)
[Equation]
(84) exploiting our modiﬁcation S YNC S HUF as well as Lemma 8.
For each ∇ f mσ mk ( j ) ( x k, 0 ) , we ﬁrst add and subtract its corresponding ∇ ¯ f σ mk ( j ) ( x k, 0 ) , where ¯ f i := 1 M (cid:80)
Mm =1 f mi as deﬁned in Assumption 4.
This way, (84) can be decomposed into two sums (cid:80) ij =1 g j = 1 M ( p i + q i ) , where
[Equation]
Using this decomposition, we will derive high-probability bounds for (cid:107) p
i (cid:107) and (cid:107) q
i (cid:107) .
To simplify expressions to follow, we decompose iBM (i.e., the total number of component gradients that are summed up) into a multiple of N and the remainder.
Let
[Equation]
so that iBM is decomposed into Nα ( i ) and the remainder 0 ≤ β ( i ) <
N .
Using this new notation, we can write p i as
[Equation]
Here, recall that with S YNC S HUF , we deﬁned σ mk ( j ) :
= σ (( j + NM π ( m )) mod N ) .
With this choice of “shifted” permutations, one can notice that { σ mk ( j ) } M, NM m =1 ,j =1 =
[ N ] , meaning that adding ¯ f σ mk ( j ) for m ∈
[ M ] and j ∈ [ N/M ] results in the sum of all N ¯ f i ’s.
In fact, this happens if we sum over m ∈
[ M ] and any N/M consecutive j ’s. From this observation and F = 1 N (cid:80) Ni =
¯ f i , (85) can be written as
[Equation]
Assume for now that β ( i ) > 0 , i.e., Nα ( i ) < iBM .
The summation in the second term of RHS in (86) is a without-replacement sum (note that the indices j + Nm M do not overlap) of β ( i ) terms.
Hence, it is equal in distribution to (cid:80) β
( i ) j =1 ∇ ¯ f σ ( j ) ( x k, 0 ) .
Also, from Assumption 2, it can be easily checked that for any i ∈ [ N ]
[Equation]
These observations mean that we can apply Lemma 8 to get a concentration bound, with M ← 1 , v 1 i ← ∇ ¯ f i ( x k, 0 ) , n ← β ( i ) , and δ ← Bδ 2 NK .
By Lemma 8, with probability at least 1 − Bδ 2 NK (over the randomness in σ ), we have
[Equation]
Combining (86) and (87), we get the following upper bound on (cid:107) p i (cid:107) , which holds with probability at least 1 − Bδ 2 NK .
[Equation]
where the last inequality used β ( i ) <
N .
Also recall that we assumed β ( i ) > 0 in order to use
Lemma 8 and derive (88).
However, note that even with β ( i ) = 0 , the bound (88) trivially holds.
We next bound (cid:107) q i (cid:107) .
This time, we will apply Lemma 8 to the permutation π over the local machines.
To do this, we will condition on a ﬁxed instantiation of the permutation σ and derive a high-probability bound that holds with conditional probability at least 1 − Bδ 2 NK .
The conditional probability is at least 1 − Bδ 2 NK irrespective of the choice of σ , so we can conclude that the (unconditional) probability that our bound holds is also at least 1 − Bδ 2 NK .
Without loss of generality, choose the instantiation σ ( l )
= l for all l ∈ [ N ] .
With this σ , we have σ mk ( j ) :
=
( j + NM π ( m ))
mod N , so the vector q i reads
[Equation] [Equation]
Let us consider rewriting this summation as the sum over l ∈ [ N ] , where l appears in the subscript of the component functions.
One can check that
[Equation]
m
[Equation]
and π
[Equation]
m )
=
[Equation]
l
[Equation] j [Equation] M N mod M, where a | b denotes “ a divides b .”
From this, we can rewrite (89) as
[Equation] [Equation]
By the same reasoning above and below (86), we can see from (90) that for a given index l ∈ [ N ] , the cardinality of the set J l := { j ∈ [ iB ] : NM | ( l − j ) } is either α ( i ) or α ( i )
+ 1 .
From this, we notice that each q i,l is a without-replacement sum of α ( i ) or α ( i ) + 1 terms.
For now, suppose α ( i ) > 0 .
For each q i,l , we can apply Lemma 8 to it, and show that with probability (conditioned on the instantiation σ ( l ) = l )
at least 1 − Bδ 2 N 2 K , we have
[Equation]
Note that cases in the RHS are all bounded from above by λ (cid:113)
α ( i ) log 4 N 2 K Bδ .
Applying union bound on all l ∈ [ N ] , we get that with probability at least 1 − Bδ 2 NK , we have
[Equation]
Now consider the case α ( i ) = 0 .
Recall from the deﬁnition α ( i ) :=
(cid:4) iBMN (cid:5) that α ( i ) = 0 implies iBM < N .
In this case, q i,l = 0 for N − iBM indices l satisfying |J l | = 0 , and (cid:107) q i,l (cid:107) ≤ λ for the remaining iBM l ’s satisfying |J l | = 1 .
Summing up, (cid:107) q
i (cid:107) is bounded from above by λiBM , which is in fact less than the upper bound in (91).
Therefore, the bound (91) holds even for α ( i )
= 0 .
Recall that our goal was to ﬁnd a bound on the norm of (cid:80) ij =1 g j = 1 M ( p i + q i ) .
From the highprobability bounds obtained in (88) and (91), with probability at least 1 − BδNK ,
[Equation]
i (cid:88)
[Equation]
=
[Equation]
We can now substitute (83) and (92) to (82) to get
[Equation]
which holds with probability at least 1 − δK , due to the union bound over i = 1 , .
N/B − 1 .
The bound (93) holds for all k ∈
[ K ] with probability 1 − δ if we apply the union bound over k = 1 , . . .
, K .
Next, by ( a + b + c ) 2 ≤ 3 a 2 + 3 b 2 + 3 c 2 , we have
[Equation]
[Equation]
which also holds for all k ∈
[ K ] with probability at least 1 − δ .
Getting a high-probability convergence rate.
Given our high-probability bounds (93) and (94), we can substitute them to (40) and get
[Equation]
Two terms in the RHS of (95) can be bounded using ab ≤ a 2 2 + b 2 2 :
[Equation]
Putting inequalities (96) and (97) to (95) and noting N − B ≤ N gives
[Equation]
Recall from K ≥ 6 κ log( M 2 NK 2 ) and η = log( M 2 NK 2 ) µNK that ηLN ≤ 16 .
Since the inequality − 78 z + 2 z 2 + 3 z 4 ≤ − 12 z holds on z ∈ [0 , 16 ] , we have
[Equation]
Applying this bound to (98) results in
[Equation]
Bδ We now recall that F is µ -PŁ, so (cid:107)∇ F ( x k, 0 ) (cid:107) 2 ≥ 2 µ ( F ( x k, 0 ) − F ∗ ) :
[Equation]
Recall that (99) holds for all k ∈
[ K ] , with probability 1 − δ .
Therefore, by unrolling the inequality, and using (cid:80) K − 1 k =0 (1 − ηµN ) k ≤ 1 ηµN , we get
F ( x
[Equation]
Lastly, substituting η = log( M 2 NK 2 ) µNK to (100) gives
[Equation] [Equation]
D. P ROOF OF UPPER BOUND FOR LOCAL RR WITH S YNC S HUF (T HEOREM 7)
The ﬁrst part (“One epoch as one step of GD plus noise”) of the proof is identical to that of Theorem 2.
The ﬁrst part deﬁnes our “noise” r k as the sum of three terms r k :=
r k, 1 + r k, 2 − η r k, 3 .
We start from the second part.
Bounding noise terms using concentration.
We next bound (cid:107) r k (cid:107) by bounding each (cid:107) r k, 1 (cid:107) , (cid:107) r k, 2 (cid:107) , and (cid:107) r k, 3 (cid:107) .
We have already seen from (63), (64), (65), (66), and (67) in Appendix D. that
[Equation] [Equation]
As in the previous subsections, the key is to bound the norm of the partial sums of ∇ f mσ mk ( t ) ( y k, 0 ) using Lemma 8.
For the summations appearing in (101) and (103), we apply Lemma 8 in the same way as (68).
For any i satisfying ( j − 1) B + 1 ≤ i ≤ jB − 1 , where j ∈ [ N/B ] , and for any m ∈ [ M ] , the following bound holds with probability at least 1 − [Equation] 3 MNK :
[Equation]
For the summation that appear in (102), we use the techniques from Appendix D.4.
For each l ∈ [ N/B − 1] , we can similarly decompose (cid:80) Mm =1 (cid:80) lBt =1 ∇ f mσ mk ( t ) ( y k, 0 ) into p l + q l , where
[Equation]
l
[Equation]
As done in Appendix D.4, we can follow the same steps and show a high-probability bound, which is a slightly different version of (88): with probability at least 1 − Bδ 3 NK ,
[Equation]
[Equation]
Similarly, for (cid:107) q l (cid:107) we can show a slight modiﬁcation of (91): (cid:107) q l (cid:107) ≤
[Equation]
λ (cid:114)
[Equation]
Bδ
[Equation]
(106) which holds with probability at least 1 − Bδ 3 NK .
Combining (105) and (106), with probability at least 1 − 2 Bδ 3 NK , we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] m [Equation]
lB
[Equation]
t
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Equation]
By applying the union bound, with probability at least 1 − δK , the bound (104) holds for all m ∈
[ M ] and i ∈ (cid:83) N/Bj =
[( j − 1) B + 1 : jB − 1] , and the bound (107) holds for all l ∈ [ N/B − 1] .
We now substitute the bounds (104) and (107) to (101), (102), and (103) to get upper bounds for (cid:107) r k, 1 (cid:107) , (cid:107) r k, 2 (cid:107) , and (cid:107) r k, 3 (cid:107) , respectively.
For (cid:107) r k, 1 (cid:107) and (cid:107) r k, 3 (cid:107) , we can apply the same calculations as in (70) and (72), modulo the fact that Assumption 3 is now implied by Assumption 4, with constants τ = λ and ρ = 1 .
We obtain
[Equation]
For (cid:107) r k, 2 (cid:107) , we have
[Equation]
Recalling the deﬁnition r k :=
r k, 1 + r k, 2 − η r k, 3 , we get an upper bound for (cid:107) r k
(cid:107) from (108), (109), and (110): (cid:107) r k (cid:107) ≤ (cid:107) r k, 1 (cid:107) + (cid:107) r k, 2 (cid:107) + η (cid:107) r k, 3 (cid:107) ≤ Lν (cid:114)
[Equation]
δ
[Equation]
[Equation]
+ Lλ (cid:32)
[Equation]
[Equation]
+ L (cid:107)∇ F
[Equation]
[Equation]
[Equation]
Recall again that we have K ≥ 7 κ log( M 2 NK 2 ) and η = log( M 2 NK 2 ) µNK , so ηLN ≤ 1 / 7 .
Using this and N − B ≤ N , we can further simplify (111).
[Equation]
(cid:124)
[Equation]
(cid:125)
[Equation]
[Equation]
which holds with probability at least 1 − δK .
The bound (112) holds for all k ∈
[ K ] with probability 1 − δ if we apply the union bound over k = 1 , . . .
, K .
Next, by ( a
[Equation]
b
[Equation]
c
[Equation]
[Equation]
c 2 , we have
[Equation]
which also holds for all k ∈
[ K ] with probability at least 1 − δ
[Equation]
Getting a high-probability convergence rate.
Given our high-probability bounds (112) and (113)
[Equation]
we can substitute them to (62) and get
[Equation]
The following terms in the RHS of (114) can be bounded using ab ≤ a 2 2 + b 2 2 :
[Equation]
[Equation]
Substituting (115) and (116) to (114) results in F
[Equation]
Again, we have K ≥ 7 κ log( M 2 NK 2 ) and η = log( M 2 NK 2 ) µNK , so ηLN ≤ 17 .
Since the inequality − 78 z + 125 z 2 + 14725 z 4 ≤ − 12 z holds on z ∈ [0 , 17 ] , we have
[Equation]
Substituting this inequality to (117), together with 4 + 3 ηLN ≤ 317 < 92 , yields
[Equation]
We now recall that F is µ -PŁ, so (cid:107)∇ F ( y k, 0 ) (cid:107) 2 ≥ 2 µ ( F ( y k, 0 ) − F ∗ ) : [Equation]
[Equation]
Recall that (118) holds for all k ∈
[ K ] , with probability 1 − δ .
Therefore, by unrolling the inequality, and using (cid:80) K − 1 k =0 (1 − ηµN ) k ≤ 1 ηµN , we get [Equation]
[Equation]
Recall that Φ ν :=
12 N ( B 3 / 2 − 1) 5 B + 4 N 1 / 2 ( N − B ) M and Φ λ :=
2 N ( B − 1)+ 15 N 1 / 2 ( N 3 / 2 − B 3 / 2 ) 4 M 1 /, hence Φ 2 ν ≤ 288 N 2 ( B 3 / 2 − 1)25 B
[Equation]
[Equation]
N
[Equation]
Φ
[Equation]
[Equation]
Substituting these inequalities and also η = log( M 2 NK 2 ) µNK gives
[Equation]
with probability at least 1 − δ .
This ﬁnishes the proof.
D.
A GENERALIZED VECTOR - VALUED H OEFFDING -S ERFLING INEQUALITY
We extend the vector-valued Hoeffding-Serﬂing inequality proved in Schneider (2016) to account for the mean of multiple independent without-replacement sums
[Equation]
Lemma 8.
Suppose there are MN vectors { v mi } M,Nm =1 ,i =1 ∈ R d that satisfy (cid:107) v mi − ¯ v m (cid:107) ≤ ν for m ∈
[ M ] , where ¯ v m := 1 N (cid:80) Ni
=1 v mi .
Consider M independently and uniformly sampled permutations σ 1 , . . .
, σ M ∼ Unif( S N ) .
For any n ≤ N − 1 , with probability at least 1 − δ , we have
[Equation]
Proof.
The proof is an extension of Theorem 2 of Schneider (2016) which proves the M = 1 case for vectors in smooth separable Banach spaces.
We prove our extended concentration inequality for R d , but we note that the proof technique can be applied directly to general smooth separable Banach spaces, as done in Schneider (2016).
Below, we state a special case of Theorem 3 of Pinelis (1992) and Theorem 3.5 of Pinelis (1994), because this R d case serves our purpose.
Lemma 9 (Pinelis (1992; 1994)) .
Suppose that a sequence of random variables { x j } j ≥ 0 is a martingale taking values in R d , and (cid:80)
∞
j =1 ess sup (cid:107) x
j − x j − 1 (cid:107) 2 ≤ c 2 for some c > 0 .
Then, for λ > 0 ,
[Equation]
The proof of Lemma 8 proceeds by deﬁning a sequence of random variables { x j } , showing that it is a martingale, and applying Lemma 9 to prove our concentration bound.
For m ∈ [ M ] , deﬁne index functions k m : N ∪ { 0 } → [0 : n ] in the following way:
[Equation]
Using these index functions, we introduce the following sequence of random variables { x j } :
[Equation]
and show that this is a martingale, i.e.,
[Equation]
for all j ≥ 1 .
Notice ﬁrst that by deﬁnition of k m ’s we have x
Mn = x
Mn +1 =
x Mn +
= . .
, so (121) is trivially satisﬁed for all j >
Mn .
Next, for any j satisfying ( l − 1) n + 1 ≤ j ≤ ln where l ∈ [ M ] , we have
[Equation]
Now note that for any k ∈
[ N − 1] , we have
[Equation]
where the last equality used (cid:80) Ni =1 ( v lσ l ( i ) − ¯ v l ) = 0 .
Using applying this fact to (122) and noting 1 N − k l ( j ) − 1 N − k l ( j )+
= 1 ( N − k l ( j
[Equation]
( N − k l ( j [Equation] +1)
[Equation]
hence proving that { x j } j ≥ 0 is a martingale.
We now apply Lemma 9 to our { x j } .
For j such that
[Equation]
l −
[Equation]
n + 1 ≤ j ≤ ln , notice from (122) that
[Equation] [Equation]
which leads to
[Equation]
by the triangle inequality.
From this, we get the bound c 2 in the statement of Lemma 9: ∞
[Equation]
where the second inequality used the inequality that (cid:80) bk
=
a +1 1 k 2 ≤ b − a a [Equation] b +1)
(Serﬂing, 1974,
Lemma 2.1).
Now, applying Lemma 9 to { x j } with c 2 = 4 ν 2 Mn ( N − n ) 2 (cid:0) 1 − n − 1 N (cid:1) gives
[Equation]
Recall from the deﬁnition of { x j } that
[Equation]
Substituting λ =
Mn(cid:15)N − n to (123) gives
[Equation]
which ﬁnishes the proof.
D. H OW CAN WE AVOID UNIFORM BOUNDS OVER R d IN OUR ASSUMPTIONS ?
In Section 2, we introduced Assumptions 2, 3, and 4 on the intra- and inter-machine deviation.
The assumptions required that inequalities such as (cid:107)∇ f mi ( x ) − ∇ F m ( x ) (cid:107) ≤ ν hold for all x ∈ R d .
In this subsection, we discuss more on this strong requirement “entire R d .”
In fact, the entire- R d requirement is posed in our assumptions to simplify the exposition of the main results, and is not strictly necessary.
One can easily check from our proofs that the assumptions are only applied to the beginning iterates x k, 0 (for minibatch RR) or y k, 0 (for local RR) of epochs.
Hence, if these iterates lie in a bounded set, then the constants ν , τ , ρ , and λ may become much smaller, depending on problem instances.
Actually, if we explicitly assume that the iterates lie in a compact set S , 12 then Assumptions 2–4 are even guaranteed to hold for some constants; e.g., for
Assumption 2, we can choose
[Equation]
since the maximum always exists.
However, assuming that the iterates lie in a speciﬁc set S can be problematic because the distance that the iterates travel depend on the objective functions.
One cannot know a priori if all iterates will stay in a ﬁxed set S ; hence, explicitly assuming bounded iterates should be avoided.
Then, a natural question is whether we can prove bounded iterates under some reasonable conditions, instead of assuming it.
We point out that this can be done by applying the technique developed in
Ahn et al.
(2020, Theorem 1) to our upper bound theorems.
Using the technique, a modiﬁed version of our Theorem 1 can be written as follows:
Theorem 10 (Best-iterate version of Theorem 1) .
Suppose that minibatch RR has parameters satisfying Assumption 1.
Assume that all local component functions f mi are L -smooth, the global objective function F is µ -PŁ, and the set of global minima of F is nonempty and compact.
Consider running the algorithm using step-size η = B log( MNK 2 ) µNK and initialization x 1 , 0 := x 0 , for epochs
K ≥ 6 κ log( MNK 2 ) .
Then, with probability at least 1 − δ ,
[Equation]
where the constant ν < ∞ is deﬁned as
[Equation]
In the theorem, we used x K +1 , 0 to denote the last iterate of the algorithm x K, NB .
Theoremdiffers from Theorem 1 in three aspects: 1) it considers the best-iterate , not the last-iterate ; 2) it additionally assumes that the set of global minima of F is nonempty and compact, which always holds if F is strongly convex; and 3) it does not rely on Assumption 2, but instead “proves” it for the F ( x 0 ) -sublevel set of F (124).
Note that the constant ν (124) can be much smaller than the uniform bound required to make Assumption 2 hold for the entire R d .
For Theorems 2, 6, and 7, we can also apply similar techniques to prove best-iterate bounds with smaller intra- and inter-machine deviation constants ν , τ , ρ , and λ ; we omit the precise statements.
We conclude this subsection with the proof of Theorem 10.
Proof.
The proof follows that of Ahn et al.
(2020, Theorem 1).
Existence of ν .
We ﬁrst show the existence of ν < ∞ (124).
The global objective function F is µ -PŁ.
If we denote the set of global minima of F as X ∗ , the set X ∗ is nonempty and compact by assumption.
Then, by Karimi et al.
(2016, Theorem 2) µ -PŁ functions satisfy quadratic growth, i.e., denoting by x ∗ the closest global minimum in X ∗ to the point x ,
[Equation]
Deﬁne the sublevel set S :
= { x | F ( x ) ≤ F ( x 0 ) } .
Due to the quadratic growth property, we have
F ( x 0 ) − F ∗ ≥ F ( x ) − F ∗ ≥ 2 µ (cid:107) x − x ∗
[Equation]
2 for all x ∈ S .
This implies that
[Equation]
Since we assumed that X ∗ is compact, S is also bounded, and hence compact.
Now, for any m ∈ [ M ] and i ∈ [ N ] , (cid:107)∇ f mi ( x ) − ∇ F m ( x ) (cid:107) is a continuous function on a compact set S , so there must exist a constant ν mi < ∞ such that (cid:107)∇ f mi ( x ) − ∇ F m ( x ) (cid:107) ≤ ν mi for all x ∈ S .
Taking the maximum of ν mi over all m
and i gives ν.
Proving the best-iterate bound.
With the constant ν (124), if all the iterates { x k, 0 } k ∈ [ K +1] stay within the sublevel set S :
= { x | F ( x ) ≤ F ( x 0 ) } , one can consider Assumption 2 to be true with constant ν .
From this observation, we consider two cases:
All the iterates { x k, 0 } k ∈
[ K +1] stay in the sublevel set S . 2.
There exists an iterate x k, 0 / ∈ S .
In fact, the ﬁrst case can be proven by exactly the same steps as Theorem 1, described in Appendix D.2.
For the second case, suppose that there exists an iterate x k, 0 that escapes the sublevel set S .
Let k (cid:48) ∈ { 2 , . .
, K + 1 } be the ﬁrst such k .
Then, since x k (cid:48) − 1 , 0 is still in S , it follows from (50) in
Appendix D.2 that we have
[Equation]
However, the fact that x k (cid:48) , 0 / ∈ S and x k (cid:48) − 1 , 0 ∈ S implies
[Equation]
Combining the two bounds (125) and (126), we get
[Equation]
which implies
[Equation]
Substituting η = log( MNK 2 ) µNK 13 gives the desired bound and ﬁnishes the proof.
E P ROOF OF LOWER BOUND FOR MINIBATCH RR (T HEOREM 3)
For Theorem 3, we consider three step-size ranges and do case analysis for each of them.
We construct functions for each corresponding step-size regime such that the convergence of minibatch
RR is “slow” for the functions on their corresponding step-size regime.
The ﬁnal lower bound is the minimum among the lower bounds obtained for the three regimes.
More concretely, we will construct three one-dimensional functions F 1 ( x ) , F 2 ( x ) , and F 3 ( x ) satisfying L -smoothness (1), µ -PŁ condition (2), and Assumption 2 such that
• Minibatch RR on F 1 ( x ) with η ≤ B µNK and initialization x
= νµ results in
[Equation]
• Minibatch RR on F 2 ( x ) with η ≥ B µNK and η ≤ B 513 LN and initialization x 0 = 0 results in
[Equation]
x
[Equation]
, NB
[Equation] Note that the step-size range requires K ≥ 513 κ , hence this lower bound occurs only in the “large-epoch” regime, i.e., K (cid:38) κ .
• Minibatch RR on F 3 ( x ) with η ≥ B µNK and η ≥ B 513 LN and initialization x 0 = 0 results in E [ F 3 ( x [Equation] , NB )
[Equation]
Ω [Equation] Then, the three dimensional function F ([ x, y, z ] (cid:62) ) =
F 1 ( x ) + F 2 ( y ) + F 3 ( z ) will show bad convergence in any step-size regime.
Furthermore,
[Equation]
, that is, if F 1 , F 2 and F 3 are µ -strongly convex and L -smooth, then so is F .
Moreover, since the component functions in each coordinate are designed to satisfy Assumption 2 with ν , the resulting three dimensional function F also satisﬁes Assumption 2 with √ 3 ν .
Since the ﬁnal lower bound is the minimum among the lower bounds obtained in the step-size ranges, the lower bound becomes Ω (cid:16) [Equation] and [Equation]
if K < 513 κ (in which case the second step-size range does not exist).
In the subsequent subsections, we prove the lower bounds for F 1 , F 2 , and F 3 separately.
E. L OWER BOUND FOR η ≤ B µNK
Consider the case where every function at every machine is the same: for all i ∈
[ N ] and m ∈
[ M ] , f mi ( x ) :
= µx 2 2 .
Hence, F 1 ( x ) = µx 2 2 .
Let x k, 0 and x k, NB denote the iterates where the k -th epoch starts and ends respectively.
Then, {+x k +1 , 
+}
= x k, NB = (1 − ηµ ) NB x k, 0 .
Initializing at x 1 ,
= νµ and unrolling this for K epochs, we get
[Equation]
, since N ≥ 2 , K ≥ 1 , and B divides N .
Hence, F 1 ( x K, NB ) =
Ω( ν 2 µ ) .
E. L OWER BOUND FOR η ≥ B µNK AND η ≤ B 513 LN
For most part of this subsection, we consider iterates within a single epoch, and hence we will omit the subscripts denoting epochs.
Let x 0 denote the iterate at the beginning of the epoch, and x i denote the iterate after the i -th communication round in that epoch.
In our construction, each machine will have the same set of component functions, that is, there will be no inter-machine deviation.
We therefore omit the superscript m from the local component functions f mi .
The function we construct for the lower bound and its component functions are as follows:
[Equation]
f
[Equation]

 , where
[Equation]
and
[Equation]
Note that the function F 2 ( x ) =
( L 1 x ≤ 0 + µ 1 x> 0 ) x 2 2 is µ -strongly convex and L -smooth with minimizer at 0, and also satisﬁes Assumption 2.
Let σ m be a random permutation of N 2 +1 ’s and N 2 − 1 ’s. Then, machine m computes gradients on f − 1 and f +1 in the order given by σ m .
Let σ mj denote the j -th ordered element of σ m .
Then,
[Equation]
Hence, the last iterate of an epoch, x NB , is given by
[Equation]
Since (cid:80)
[Equation]
[Equation]
Thus, E [ x NB − x 0 ] = − η (cid:80) NB {+− 1 i =
+}
E [( L 1 x i ≤ 0 + µ 1 x i > 0 )
x i ] .
We want to prove that E [ x NB ] keeps increasing over an epoch, that is E [ x NB − x 0 ] > 0 when x 0 is close enough to the minimizer 0.
For this, we ﬁrst consider the case where the ﬁrst iterate x 0 of the epoch satisﬁes x 0 ≥ 0 .
The x 0 < 0 case will be considered later.
For the case x 0 ≥ 0 , we will show that whenever x 0 is small, the expected amount of update made in the ( i +1) -th iteration, E [( L 1 x i ≤ 0 + µ 1 x i > 0 ) x
i ] , is negative in the ﬁrst half of the epoch and not too big in the second half.
We use the following lemmas, proven in Appendices F.1 and F.2, respectively.
Lemma 11.
For x 0 ≥ 0 , 0 ≤ i ≤ (cid:98) N 2 B (cid:99) , η ≤
[Equation]
513 LN , and Lµ ≥ 7695 ,
[Equation]
Lemma 12.
For x 0 ≥ 0 , 0 ≤ i ≤ NB − 1 , and η ≤
[Equation]
The key intuition is that, for Lµ big enough, we can use the lemmas above in (127) to get
[Equation]
whenever | x 0 | is small.
Multiplying the above by K (for K epochs) will give us the required lower bound (up to factors of Lµ ).
We will make this approximate calculation precise in the rest of the proof.
Using the two lemmas above in (127)
[Equation]
we get that
[Equation]
Since iηL ≤ ηLNB ≤ 1513 , µ ≤ L 7695 , and N/B ≥ 2 , the following bound holds:
[Equation]
[-≤ (cid:18)(cid:22) N 2 B (cid:23) +(cid:19) 
-]
[Equation]
Also, note that (cid:98) N 2 B (cid:99) ≥ N 3 B whenever N/B ≥ 2 .
We have[-(cid:98) N 2 B (cid:99) (cid:88) i = √ i ≥ (cid:90) (cid:98) N 2 B (cid:99) √ tdt = (cid:18)(cid:22) N 2 B (cid:23)(cid:19) 3 /≥ (cid:18) N 3 B (cid:19) 
-]
[Equation]
Substituting the bounds (129), (130), and (131) into (128), and using µ ≤ L 7695 ,
[Equation]
For the other case x 0 < 0 , we have the following Lemma, which we prove in Appendix F.3:
Lemma 13.
If η ≤
B 513 NL and an epoch starts at x 0 < 0 , then
[Equation]
Further, if the ﬁrst epoch of the algorithm is initialized at 0 , then for any starting iterate x 0 of any following epoch, we have P ( x 0 ≥ 0) ≥ 1 / 2 .
Using (132) and Lemma 13 we get
[Equation]
Thus far, we have characterized the expected per-epoch update, starting from the initial iterate x 0 and iterating until the last iterate x NB of the epoch.
Now recall that we run the algorithm for K epochs.
Using x k,i to denote the i -th iterate of the k -th epoch, we get a lower bound on the expectation of the last iterate x k, N B if we initialize at x 1 , 0 = 0 :
[Equation]
(Since η ≥
B µNK )
Note that since Lµ ≥ 7695 and
K ≥ 513 Lµ (which is implied by
[Equation]
[Equation]
Hence, we get from η ≥
B µNK that
[Equation]
and by Jensen’s inequality, we ﬁnally have
[Equation]
L OWER BOUND FOR η ≥ B µNK AND η ≥ B 513 LN
Similar to earlier parts of the proof, here as well, each machine will have the same component functions, that is, there will be no inter-machine deviation.
The proof uses a similar construction as
Safran & Shamir (2020; 2021):
[Equation]

 , where
[Equation]
Hence, F 3 ( x ) =
Lx 2 2 , and has its minimizer at 0.
We ﬁrst compute the expected “progress” over a given epoch.
For simplicity, let us omit the subscript for epochs for now.
Let x 0 denote the iterate at the beginning of the epoch and x i denote the iterate after the i -th communication round in that epoch.
For a given epoch, let σ m be the permutation of N 2 +1 ’s
and N 2 − 1 ’s, sampled by machine m .
Then,
[Equation]
For the rest of this proof, x 2 i refers to the square of the i -th iterate.
Then,
[Equation]
where we used the fact that E [ σ mj ] = 0 .
Further, because σ m and σ m (cid:48) are independent and identically distributed for different m and m (cid:48) , we get that
E    
NB (cid:88) i = (1 − ηL )
NB − i
M (cid:88) m = iB
[Equation]
j =( i − 1)
[Equation]
+ σ mj   2 

=
[Equation]
M
[Equation]
m =
NB
[Equation]
i = (1 − ηL
[Equation]
NB − i iB (cid:88)
j =( i − 1) B + σ mj 
 2 
 = [Equation] m = E    
NB
[Equation]
i = (1 −
[Equation]
NB − i iB (cid:88)
j =( i − 1) B + σ mj 
 2 
 +
[Equation]
(cid:54) =
[Equation]
(cid:48)
E  
NB (cid:88) i = (1 − ηL )
NB − i iB (cid:88)
j =( i − 1) B + σ mj 
 E  
NB (cid:88) i = (1 − ηL )
NB − i iB (cid:88)
j =( i − 1) B + σ m (cid:48)
j   =
M E    
NB (cid:88) i = (
[Equation]
NB − i iB (cid:88)
j =( i − 1) B + σ 1 j   2   , where the last equality used the fact that E [ σ mj ] = 0 for all m ∈
[ M ] and i ∈ [ N ] , and that σ m are identically distributed.
Since we only consider the permutation σ 1 (i.e., the one for machine 1 ) from now on, we henceforth omit the superscript.
Substituting this to (133) gives
[Equation]
iB
[Equation]
 [Equation] [Equation] From (134), we have calculated the per-epoch expected update.
Recall that we run the algorithm for K epochs.
Using x k,i to denote the i -th iterate of the k -th epoch, we get a lower bound on the expectation of the last iterate x k, NB squared:
[Equation]
where the inequality used x 1 , 0 = 0 and (cid:80) K − 1 k =0 (1 − ηL ) 2 NkB ≥ (1 − ηL )
= 1 .
Next, we analyze the expectation term, i.e., Φ , deﬁned in (134).
[Equation]
Noting that σ 2 j = 1 and E [ σ j σ j (cid:48) ]
= − 1 N − 1 , we get Φ
= B
[Equation]
j = (1 − ηL ) 2 j −
N −(cid:88)
j (cid:54) =
j (cid:48) (1 − ηL ) NB −(cid:98)
j − 1 B (cid:99)− 1 (1 − ηL ) NB −(cid:98)
j (cid:48)− 1 B (cid:99)−= B
NB − 1 (cid:88) j = (1 − ηL ) 2 j −
N −   
N (cid:88)
j
= (1 − ηL )
NB −(cid:98)
j − 1 B (cid:99)−  − N (cid:88)
j
= (1 − ηL ) 2( NB −(cid:98)
j − 1 B (cid:99)− 1) 
 =
B NB − 1 (cid:88) j = (1 − ηL ) 2 j −
N −  B  NB − 1 (cid:88) j = (1 − ηL )
j   − B
NB − 1 (cid:88)
j
= (1 − ηL ) 2 j   =
B
N −  N B
NB − 1 (cid:88)
j = (1 − ηL ) 2 j −  
NB − 1 (cid:88)
j
= (1 − ηL )
j   2  
[Equation]
Note that the term in the parenthesis is exactly the right hand side of Equation (23) in Safran &
Shamir (2020) modulo n and α replaced with NB and ηL , respectively.
Hence, by Lemma 1 of
Safran & Shamir (2020), we have
[Equation]
for some universal constant c > 0 .
Using the fact that η ≥
B 513 LN , it is easy to check that the RHS of (137) is lower-bounded by c (cid:48)
ηL , where c (cid:48) > 0 is a universal constant.
Combining (135), (136), and (137) gives
[Equation]
Since 2 B divides N , we have B ≤ N/ 2 .
Since N ≥ 2 , we have N − B N − 1 ≥
N 2( N − 1) ≥ 12 .
Using this and the fact that η ≥
B µNK , we get
[Equation]
F P ROOFS OF HELPER LEMMAS FOR A PPENDIX E F. P ROOF OF L EMMA
First, if i = 0 then the lemma trivially holds, because x 0 ≥ 0 gives
[Equation]
The inequality holds because Lµ ≥ 7695 .
For the rest of the proof, we consider the case 1 ≤ i ≤ N 2 B .
By the law of total expectation we have
E [( L 1 x i ≤ 0 + µ 1 x i > 0 ) x i ] =
P  
M (cid:88) m = iB (cid:88)
j = σ mj >

E  
( L 1 x i ≤ 0 + µ 1 x i > 0 )
x i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
M (cid:88) m = iB (cid:88)
j = σ mj >
 + P  
M (cid:88) m = iB (cid:88)
j = σ mj ≤ 
E  
( L 1 x i ≤ 0 + µ 1 x i > 0 )
x i (cid:12)(cid:12) (cid:12)(cid:12) (cid:12)(cid:12)
M (cid:88) m = iB (cid:88)
j = σ mj ≤  ≤ P  
M (cid:88) m = iB (cid:88)
j = σ mj >

L E  
x i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
M (cid:88) m
[Equation]
(138) where the last inequality used the fact that ( L 1 t ≤ 0 + µ 1 t> 0 ) t ≤ Lt and ( L 1 t ≤ 0 + µ 1 t> 0 ) t ≤ µt for any t ∈ R .
Deﬁne E := (cid:80)
Mm =1 (cid:80) iBj =1 σ mj .
We handle each of the two expectations in (138) separately.
We ﬁrst bound E [ x i |E > 0] .
[Equation]
[Equation]
[Table]
[Equation]
Next, we use the following lemma to bound the conditional expectations that arise in (139).
This lemma is proven in Appendix F.4 and it may be of independent interest to readers.
Lemma 14.
For m ∈ [ M ] , let σ m be a random permutation of N 2 +1 ’s
and N 2 − 1 ’s. Then, for any i ≤ N 2 and k ≤ B 2 , we have
[Equation]
Furthermore, for any 0 ≤ i ≤ N and 0 ≤ k ≤ N satisfying i + k ≤ N , we have
[Equation]
Lastly, for any 0 ≤ i ≤ N 2 and 0 ≤ k ≤ B 2 satisfying i + k ≥ 1 , we have
[Equation]
Lemma 14 implies that 1 M E [ E|E > 0] ∈ (cid:104) 164 (cid:113) iBM , (cid:113) iBM (cid:105) and P ( E > 0) =
P ( E < 0) ≥ 1 / 6 .
From this, we get ην
[Equation] [Equation]
[Equation]
Also, since η ≤ BLN we have
[Equation]
which implies that
[Equation]
[Equation]
Substituting (140), (141), and (142) to (139), we obtain
[Equation]
Next, we have the following lemma that we can apply to E [ | x j − x 0 | ] .
Proof of Lemma 15 can be found in Appendix F.5.
Lemma 15.
For x 0 ≥ 0 , [Equation] and
[Equation]
Applying this lemma to (143), we get
[Equation]
where we got the last inequality by using the fact that iηL ≤ ηLNB ≤ 1513 , which follows from η ≤ [Equation]
513 LN .
So far, we have obtained an upper bound for E [ x i |E > 0] .
Recall that there is another conditional expectation in (138) that we want to bound, namely E [ x i |E ≤ 0] .
We bound it below, using the tools developed so far.
For i ≤ N 2 B ,
[Equation]
Lemma 14)
[Equation]
[Equation]
Using (144) and (145) in (138), we get that for i ≤ N 2 B :
[Equation]
From Lemma 14, note that 16 ≤ P ( E > 0) ≤ 56 and 16 ≤ P ( E ≤ 0) ≤ 56 .
We use these inequalities, along with iηL ≤ ηLNB ≤ 1513 and Lµ ≥ 7695 , to bound the terms appearing in (146) [Equation]
[Equation]
We also have
[Equation]
where we used the assumption Lµ ≥ 7695 .
Substituting (147) and (148) to (146), we get
[Equation]
as desired
[Equation]
F. P ROOF OF L EMMA
[Equation]
F. P ROOF OF L EMMA
We consider iterates within a single epoch, and hence we will omit the subscripts denoting epochs.
In our construction [Equation] each machine has the same set of component functions, that is, there will be no inter-machine deviation.
We therefore omit the superscript m from the local component functions.
Consider the function
[Equation]
, where
[Equation]
Hence, G 2 ( x ) =
Lx 2 2 .
We will prove the lemma by coupling iterates corresponding to F 2 and G 2 .
In particular, we will perform minibatch RR on F 2 and G 2 such that both start the given epoch at xand all the corresponding machines use the same random permutations.
Let x i,F be the iterate after the i -th round of communication for F 2 and x i,G be the iterate after the i -th round of communication for G 2 .
We use mathematical induction to prove that x i,F ≥ x i,G for all i = 0 , . .
NB .
After that, we will use this to prove our desired statement E [ x NB ,F | x 0 < 0] ≥ (1 − 7 ηLN 8 B ) x 0 .
Let σ m be a random permutation of N 2 +1 ’s and N 2 − 1 ’s.
Base case.
x 0 ,F ≥ x 0 ,G since both start the epoch at the same point x 0 .
Inductive case.
There can be three cases:
• Case 1: x i,F ≥ x i,G ≥ 0 .
Then,
[Equation]
[Equation]
η
[Equation]
• Case 2: 0 ≥ x i,F ≥ x i,G .
Then,
[Equation]
• Case 3: x i,F ≥ 0 ≥ x i,G .
Then,
[Equation]
Note that since η ≤ BLN , x i,F (1 − ηµ ) ≥ 0 and x i,G (1 − ηL ) ≤ 0 , which proves that x i +1 ,F − x i +1 ,G ≥ 0 .
Thus, we see that x i +1 ,F ≥ x i +1 ,G .
Further, by linearity of expectation and gradient, it is easy to check that
[Equation]
Using the result that x NB ,F ≥ x NB ,G which we proved above, we get E [ x NB ,F ] ≥ (1 − ηL ) NB x 0 for any initial iterate x 0 .
Speciﬁcally for x 0 < 0 , this implies E [ x NB ,F | x 0 < 0] ≥ (1 − ηL )
NB x 0 .
Further, since ηL ≤
B 513 N , we have (1 − ηL ) NB ≤ 1 − 7 ηLN 8 B .
This is because 1 − 7 zN 8 B − (1 − z ) NB is nonnegative on the interval (cid:104) 0 , 1 − (7 / 8) 1 N/B − 1 (cid:105) , and 1 − (7 / 8) 1 N/B − 1 ≥
B 513 N for all NB ≥ 2 .
To see why, note that (1 − 1 513( n − 1) )
n − 1 ≥ 7 8 for all n ≥ 2 , and this gives 1 − (7 / 8) 1 n − 1 ≥ 1 513( n − 1) , which then implies 1 − (7 / 8) 1 n − 1 ≥ 1 513 n for all n ≥ 2 .
Therefore, for x 0 < 0 , we have E [ x NB ,F | x 0 < 0] ≥ (1 − 7 ηLN 8 B ) x 0 .
For the last statement of the lemma, note that by symmetry of the function G 2 , if we initialize the
Algorithm 2 at 0 , then for any starting iterate of an epoch we have P ( x 0 ,G ≥ 0) ≥ 1 / 2 .
This combined with the fact that x i,F ≥ x i,G gives us that P ( x 0 ,F ≥ 0) ≥ 1 / 2 .
F. P ROOF OF L EMMA
For m = 1 , . . .
, M , let σ m be a random permutation of N 2 +1 ’s
and N 2 − 1 ’s.
Then, we ﬁrst show that for any i ≤ N/ 2 and k ≤ B/ 2 ,
[Equation]
To prove the lower bound, we will use Khintchine’s inequality along with Lemma 12 from Rajput et al.
Let us deﬁne random variables a m := | 1 M (cid:80) ij =1 σ mj | , x m := sign( (cid:80) ij =1 σ mj ) , b M := | (cid:80) i
+ k j
= i +1 σ Mj | , and y M :=
sign( (cid:80) i
+ k j
= i +1 σ Mj ) .
For x m , if the sum (cid:80) ij =1 σ mj = 0 then x m is + with probability 0 .
5 and − 1 with probability 0 .
Ties occurring in y M are also broken similarly.
We can note that x m ’s and y M are i.i.d.
Rademacher random variables, which allows us to apply
Khintchine’s inequality, Then, by Khintchine’s inequality,
[Equation]
m =
i (cid:88)
[Equation]
(cid:88)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12) (cid:12) (cid:12)
[Equation]
(cid:34)(cid:12)(cid:12)(cid:12) (cid:12)
[Equation]
(cid:88)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12) (cid:12)
[Equation]
By applying (cid:107) z (cid:107) 2 ≥ 1 √ d (cid:107) z (cid:107) 1 for z ∈ R d twice, we get
[Equation]
Next, noticing that a m ’s are i.i.d.,
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:88)
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] [Equation] (Lemma 12 from Rajput et al
[Equation]
Note that Lemma 12 from Rajput et al.
(2020) has the requirement that N ≥ 256 .
However, that requirement is for the entire lemma to hold, whereas we need only the ﬁrst inequality in the lemma.
For that, the requirement is simply N ≥ 8 .
Further, note that for N = 2 , 4 , and 6 it can be manually veriﬁed that the required inequalities in Lemma 12 of Rajput et al.
(2020) hold.
Hence, this lemma holds for all even N .
The upper bound comes from Jensen’s inequality:
E 
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
i + k (cid:88)
j = i + σ Mj (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤
M (cid:118)(cid:117)(cid:117)(cid:117)(cid:117)(cid:116) E    
M (cid:88) m = i (cid:88)
j = σ mj 
 2 
 + (cid:118)(cid:117)(cid:117)(cid:117)(cid:117)(cid:116) E     i + k (cid:88)
j = i + σ Mj 
 2 
 =
M (cid:118)(cid:117)(cid:117)(cid:117)(cid:117)(cid:116) M (cid:88) m =
E     i (cid:88)
j = σ mj   2 
 + (cid:118)(cid:117)(cid:117)(cid:117)(cid:117)(cid:116) E     i + k (cid:88)
j = i + σ Mj 
 2 
 (Since (cid:80) ij =1 σ mj for m = 1 , 2 . .
are mean 0 and independent.)
[Equation]
E [ σ Mj σ Ml ] ≤ 0 because σ Mj and σ Ml are negatively correlated.
Hence, we get
[Equation]
as desired.
Next, it is left to show that for 0 ≤ i ≤ N 2 and 0 ≤ k ≤ B 2 satisfying i + k ≥ 1 , we have P [Equation] j =
[Equation]
By symmetry, proving the equality is straightforward, and hence it is sufﬁcient prove that
[Equation]
For this, it in fact sufﬁces to show that
[Equation]
because (149) can be derived from (150).
We ﬁrst explain why (150) implies (149), and then show (150).
Suppose (150) is true.
Then,
• Case 1: If i = 0 , then (149) becomes P ( (cid:80) kj =
1 σ Mj = 0) ≤ 23 , which is true due to (150).
• Case 2: If M = 1 , then (149) becomes P ( (cid:80) i
+ k j =1 σ 1 j = 0) ≤ 23 , which is true due to (150).
• Case 3: If i ≥ 1 and M ≥ 2 , then we can consider two events that partition the probability space:
[Equation]
[Equation]
Conditioned on this event E
[Equation]
due to independence of machines and (150).
E 2 :=
{ (cid:80) Mm =2 (cid:80) ij =
σ
mj + M (cid:80) i + k j
= i +1 σ Mj (cid:54) = 0 } .
Conditioned on this event E 2 , let c := (cid:80)
M m =2 (cid:80) i j =1 σ m j + M (cid:80) i + k j
= i +1 σ M j .
Then, P  
M (cid:88) m = i (cid:88) j = σ mj + M i +1 (cid:88)
j = i {++ σ Mj = 
+}
| E  = P  
i (cid:88)
j = σ 1 j = − c   .
However, by symmetry, P (cid:16)(cid:80) ij =1 σ 1 j = − c (cid:17) =
P (cid:16)(cid:80) ij =1 σ 1 j = c (cid:17) ≤ 1 2 .
From these two events, we conclude that (149) must hold.
It is now left to prove (150).
It is clear that P ( (cid:80) ij =1 σ 1 j = 0)
=
for all odd i , so we assume that i is even.
Also note that P ( (cid:80) ij =1 σ 1 j = 0)
=
P ( (cid:80) Nj =
i +1 σ 1 j = 0)
=
P ( (cid:80) N − i j =1 σ 1 j = 0) , since σ 1 j ’s sum to zero.
Therefore, for the rest of the proof, we can focus on even i ’s in the range 2 ≤ i ≤ N 2 .
Note that P ( (cid:80) ij =1 σ mj = 0) is just the probability of having i 2 +1 ’s
and i 2 − 1 ’s in the ﬁrst i spots in a random shufﬂing of N 2 +1 ’s
and N 2 − 1 ’s. This is equivalent to choosing i 2 indices (for +1 ) out of the ﬁrst i , and then choosing N − i 2 indices out of the remaining N − i .
Thus,
[Equation]
Note that the term above is a decreasing function of i for i ≤ N/ 2 .
Hence, putting i = 2 to the RHS we get
[Equation]
where the inequality holds for N ≥ 4 .
F. P ROOF OF L EMMA
[Equation]
E
[Equation] MB [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
i
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (By Lemma 14)
[Equation]
Let h ( i ) :
= ην (cid:113) iMB + iηLx {+
+} + ηL (cid:80) i − 1 j =0 h
( j ) , starting with h (0)
= 0 .
Then using induction, it can be seen that E [ | x i − x 0 | ] ≤ h ( i ) .
Further, since h ( i ) is an increasing function of i , we get
[Equation]
iηL .
Since i ≤ NB and η ≤
B 513 LN , we get that
[Equation]
iMB + 513512 iηLx 0 .
G P ROOF OF LOWER BOUND FOR LOCAL RR: HOMOGENEOUS CASE (T HEOREM 4)
Recall that Theorem 4 gives the bound for local RR in the homogeneous setting, where all machines have the same local objectives.
Similar to Theorem 3, we consider three step-size ranges and do case analysis for each of them.
We construct functions for each corresponding step-size regime such that the convergence of local RR is “slow” for the functions on their corresponding step-size regime.
The ﬁnal lower bound is the minimum among the lower bounds obtained for the three regimes.
More concretely, we will construct three one-dimensional functions F 1 ( x ) , F 2 ( x ) , and F 3 ( x ) satisfying
L -smoothness (1), µ -PŁ condition (2), and Assumption 2 such that
• Local RR on F 1 ( x ) with η ≤ 1 µNK and initialization y
= νµ results in
[Equation]
[Equation]
• Local RR on F 2 ( x ) with η ≥ 1 µNK and η ≤ 1 1025 LN and initialization y
= 0 results in E
[ F
[Equation]
y K, [Equation] Note that the step-size range requires K ≥ 1025 κ , hence this lower bound occurs only in the “large-epoch” regime, i.e., K (cid:38) κ .
• Local RR on F 3 ( x ) with η ≥ 1 µNK and η ≥ 1 1025 LN and initialization y
= 0 results in
[Equation]
Then, the three dimensional function F ([ x, y, z ] (cid:62) ) =
F 1 ( x ) + F 2 ( y ) + F 3 ( z ) will show bad convergence in any step-size regime.
Furthermore,
[Equation]
, that is, if F 1 , F 2 and F 3 are µ -strongly convex and L -smooth, then so is F .
Moreover, since the component functions in each coordinate are designed to satisfy Assumption 2 with ν , the resulting three dimensional function F also satisﬁes Assumption 2 with √ 3 ν .
Since the ﬁnal lower bound is the minimum among the lower bounds obtained in the step-size ranges, the lower bound becomes Ω (cid:16)
[Equation]
if K ≥ 1025 κ and K ≥ MBN (this inequality is required to make sure ν 2 B µN 2 K 2 ≤ ν 2 µMNK ) and Ω (cid:16) ν 2 µMNK (cid:17) otherwise.
In the subsequent subsections, we prove the lower bounds for F 1 , F 2 , and F 3 separately.
G. L OWER BOUND FOR η ≤ 1 µNK
Consider the case where every function at every machine is the same: for all i ∈
[ N ] and m ∈
[ M ] , f mi ( x ) :
= µx 2 2 .
Hence, F 1 ( x ) = µx 2 2 .
Since all f mi ’s are the same, the local updates in all the machines are identical.
Hence, for this subsection we omit the superscript for local machines.
Let x k, 0 and x k,N denote the local iterates at the beginning and end of the k -th epoch.
Then,
[Equation]
Initializing at x 1 ,
= νµ and repeating this for K epochs, we get that after K epochs, the last iterate y K, NB = x K,N satisﬁes [Equation] ν
[Equation]
since N ≥ 2 , and K ≥ 1 .
Hence, F 1 ( y K, NB ) =
Ω( ν 2 µ ) .
G. L OWER BOUND FOR η ≥ 1 µNK AND η ≤ 1 1025 LN
For most part of this subsection, we consider iterates within a single epoch, and hence we will omit the subscripts denoting epochs.
Let x 0 denote the iterate at the beginning of the epoch (which is the same across all the machines), and x mi denote the i -th local iterate for machine m .
After every B local iterates, the server aggregates the local iterates x miB , computes their average y i :=
[Equation] x m iB , and synchronizes all the machines x m iB : = y i .
Let y
denote the iterate at the beginning of the epoch (which is the same across all the machines x m 0 = y 0 ), and x mi denote the i -th local iterate at machine m .
In our construction, each machine will have the same set of component functions, that is, there will be no inter-machine deviation.
We therefore omit the superscript m from the local component functions f mi .
The function we construct for the lower bound and its component functions are as follows:
[Equation]
, where
[Equation]
and
[Equation]
Note that the function F 2 ( x ) =
( L 1 x ≤ 0 + µ 1 x> 0 ) x 2 2 is µ -strongly convex and L -smooth with minimizer at 0, and also satisﬁes Assumption 2.
Let σ m be a random permutation of N 2 +1 ’s and N 2 − 1 ’s. Then, machine m computes gradients on f − 1 and f +1 in the order given by σ m .
Let σ mj denote the j -th ordered element of σ m .
Then,
[Equation]
Hence, the last iterate of an epoch, y NB , is given by
[Equation]
where, in the last line, we used the fact that (cid:80) Nj =1 σ mj = 0 .
Recall that in the construction, each machine has the same component functions.
Hence,
[Equation]
where the last equality holds because the iterates x miB + j are identically distributed across different m ∈
[ M ] .
Hence, we need to bound E
[( L 1 x 1 iB + j ≤ 0 + µ 1 x 1 iB + j > 0 ) x 1 iB + j ] .
As we did for Theorem 3, we want to prove that E [ y NB ] keeps increasing over an epoch, that is E
[ y NB − y 0 ] > 0 when y 0 is close enough to the minimizer 0.
For this, we ﬁrst consider the case where the ﬁrst iterate y 0 of the epoch satisﬁes y 0 ≥ 0 .
The y 0 < 0 case will be considered later.
For the case y 0 ≥ 0 , we will show that whenever y 0 is small, the expected amount of update made in the ( iB + j +1) -th iteration, E
[( L 1 x 1 iB + j ≤ 0 + µ 1 x 1 iB + j > 0 )
x 1 iB + j ] , is negative if i ≤ (cid:98) N 2 B (cid:99) and B 4 ≤ j ≤ B 2 , and not too big otherwise.
We use the following lemmas, proven in Appendices H.1 and H.2, respectively.
Lemma 16.
For y 0 ≥ 0 , 0 ≤ i ≤ (cid:98) N 2 B (cid:99) , B 4 ≤ j ≤ B 2 , η ≤ 1 1025 LN , and Lµ ≥ 153752 ,
[Equation]
Lemma 17.
For y 0 ≥ 0 , 0 ≤ i ≤ NB − 1 , 0 ≤ j ≤ B − 1 , and η ≤
[Equation]
Next, we apply the two lemmas above in (151).
For now, we consider the case N/B ≥ 2 .
The case
B = N will be handled separately at the end of this subsection.
For simplicity of notation, deﬁne
E iB + j :=
E [( L 1 x 1 iB + j ≤ 0 + µ 1 x 1 iB + j > 0 ) x 1 iB + j ] .
We will divide the summation in (151) into four groups; for one of them we can apply Lemma 16, and for the other three we apply Lemma 17.
[Equation]
[Table]
[Equation]
where the last inequality is true because the RHS of the inequality in Lemma 17 is nonnegative
[Equation]
First consider the terms in (152) that involve y 0 .
Since ( iB + j ) ηL ≤ ηLN ≤ 1 1025 , µ ≤ 2 L 15375 ,
N/B ≥
[Equation]
and B ≥
[Equation]
we have the following loose bound:
[Equation]
We next bound the terms in (152) that involve summation of square roots.
From N/B ≥ 2 , we have (cid:98) N 2 B (cid:99) ≥ N 3 B and (cid:98) N 2 B (cid:99) + 1 ≥ N 2 B , so
[Equation]
[Equation]
[Equation]
For the other sum, we have
[Equation]
[Equation]
Substituting the bounds (153), (154), and (155) into (152), and using µ ≤
L 40000 ,
[Equation]
For the other case y 0 < 0 , we have the following lemma, a local RR counterpart of Lemma 13.
In
Appendix H.3, we prove the following:
Lemma 18.
If η ≤ 1 1025 LN and an epoch starts at y 0 < 0 , then
[Equation]
Further, if the ﬁrst epoch of the algorithm is initialized at 0 , then for any starting iterate y 0 of any following epoch, we have P ( y 0 ≥ 0) ≥ 1 / 2 .
Using (156) and Lemma 18, we get
E
[Equation]
Thus far, we have characterized the expected per-epoch update, starting from the initial iterate yand iterating until the last iterate y NB of the epoch.
Now recall that we run the algorithm for K epochs.
Using y k,i to denote the i -th aggregated iterate of the k -th epoch, we get a lower bound on the expectation of the last iterate y k, NB if we initialize at y 1 , 0 = 0 :
[Equation]
[Equation]
Since η ≥ 1 µNK )
Note that since Lµ ≥ 40000 and K ≥ 1025 L µ (which is implied by
[Equation]
Hence, we get from η ≥ 1 µNK that
[Equation]
and by Jensen’s inequality, we ﬁnally have
[Equation]
Recall that, from the paragraph below Lemmas 16 and 17 to this point, we have assumed N/B ≥ 2 .
We handle the case B = N now.
In this case, notice that all the (cid:113) iBM terms that appear in (152) disappear, because we always have i = 0 .
Therefore, the proof goes through in the same why, modulo the fact that we do not have the terms that originate from the (cid:113) iBM terms in (152).
Therefore, we can show
[Equation]
. L OWER BOUND FOR η ≥ 1 µNK AND η ≥ 1 1025 LN
Similar to earlier parts of the proof, here as well, each machine will have the same component functions, that is, there will be no inter-machine deviation.
The proof uses a similar construction as
Safran & Shamir (2020; 2021):
[Equation]
 , where
[Equation]
Hence, F 3 ( x ) =
Lx 2 2 , and has its minimizer at 0.
We ﬁrst compute the expected “progress” over a given epoch.
For simplicity, let us omit the subscript for epochs for now.
Let y
denote the iterate at the beginning of the epoch (which is the same across all the machines x m 0 = y 0 ) and x mi denote the i -th local iterate for machine m .
After every B local iterates, the server aggregates the local iterates x miB , computes their average y i :
= 1 M (cid:80) Mm =1 x miB , and synchronizes all the machines x m iB : = y i .
For the epoch, let σ m be the permutation of N 2 +1 ’s and N 2 − 1 ’s sampled by machine m .
Upon receiving the aggregated iterate x miB =
y i , each machine m performs B local updates.
Unrolling the local update rules, the iterate after the B updates (and before synchronization) can be written as follows:
[Equation]
[Equation]
After synchronization, we get
[Equation]
Unrolling the equation above from y NB (the ﬁnal iterate of the epoch, after synchronization) to y(the starting iterate)
[Equation]
we get
[Equation]
Then, by squaring both sides and taking expectations,
[Equation]
where we used the fact that E [ σ mi ] = 0 .
Further
[Equation] because σ m and σ m (cid:48) are independent‘ and identically distributed for different m and m (cid:48) [Equation] we get that
[Equation]
where the last equality used the fact that E [ σ mj ] = 0 for all m ∈
[ M ] and i ∈ [ N ] , and that σ m are identically distributed.
Since we only consider the permutation σ 1 (i.e., the one for machine 1 ) from now on, we henceforth omit the superscript.
Substituting this to (157) gives
[Equation]
From (158), we have calculated the per-epoch expected update, because the ﬁnal iterate y NB is also the initial iterate of the next epoch.
Recall that we run the algorithm for K epochs.
We now use y k,i to denote the iterate after the i -th communication round in the k -th epoch.
Using (158), we get a lower bound on the expectation of the last iterate y K, NB squared:
[Equation]
where the inequality used that we initialize at y 1 , 0 = 0 and (cid:80) K − 1 k =0 (1 − ηL ) 2 Nk ≥ (1 − ηL )
= 1 .
Next, we bound the expectation term, i.e., Φ , deﬁned in (158).
Using Lemma 1 from Safran &
Shamir (2020) with n and α replaced with N and ηL respectively, we have
[Equation]
for some universal constant c > 0 .
Using the fact that η ≥ 1 1025 LN , it is easy to check that the RHS of (160) is lower-bounded by c (cid:48) ηL , where c (cid:48) > 0 is a universal constant.
Combining (159) and (160) gives
[Equation]
Also using the fact that η ≥ 1 µNK , we get
[Equation]
H P ROOFS OF HELPER LEMMAS FOR A PPENDIX G H. P ROOF OF L EMMA
The proof of Lemma 16 is similar to its minibatch RR counterpart, Lemma 11.
From the given 0 ≤ i ≤ NB − 1 , 0 ≤ j ≤ B − 1 , deﬁne k :=
iB + j , in order to simplify notation.
We also deﬁne
E
[Equation]
By the law of total expectation we have
[Equation]
where the last inequality used the fact that ( L 1 t ≤ 0 + µ 1 t> 0 ) t ≤ Lt and ( L 1 t ≤ 0 + µ 1 t> 0 ) t ≤ µt for any t ∈ R .
We handle each of the two expectations in (161) separately.
We ﬁrst bound E (cid:2) x 1 k (cid:12)(cid:12) E > 0 (cid:3) .
Recall from the deﬁnition of algorithm iterates that
[Equation]
[Equation]
Expanding (162) using the deﬁnition of ∇ f σ ml
+
’s, we obtain
E (cid:2) x 1 k (cid:12)(cid:12) E > 0 (cid:3)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table] [Equation] where the last inequality used the fact that ( L 1 t ≤ 0 + µ 1 t> 0 ) t ≤ Lt and ( L 1 t ≤ 0 + µ 1 t> 0 ) t ≤ µt for any t ∈ R ; and that for different m = 2 , . .
, M , the local iterates x ml are identically distributed conditioned on E > 0 .
Next, we use the fact that for any nonnegative random variable v and event ∆ , we have that E [ v | ∆] ≤ E [ v ] / P (∆) .
Hence,
[Equation]
−
[Equation]
E [ E|E
[Equation]
[Equation]
(cid:35)
[Equation]
where the last equality used the fact that for different m ∈
[ M ] , the local iterates x ml are identically distributed when they are not conditioned.
Next, we use Lemma 14 again to bound the conditional expectations that arise in (163).
We restate the lemma for the reader’s convenience.
Lemma 14.
For m ∈ [ M ] , let σ m be a random permutation of N 2 +1 ’s
and N 2 − 1 ’s. Then, for any i ≤ N 2 and k ≤ B 2 , we have
[Equation]
Furthermore, for any 0 ≤ i ≤ N and 0 ≤ k ≤ N satisfying i + k ≤ N , we have
[Equation]
Lastly, for any 0 ≤ i ≤ N 2 and 0 ≤ k ≤ B 2 satisfying i + k ≥ 1 , we have
[Equation]
Lemma 14 implies that E [ E|E > 0] ∈ (cid:104) 164 (cid:16)(cid:113) iBM + √ j (cid:17) , (cid:113) iBM + √ j (cid:105) and P ( E > 0) =
P ( E < 0) ≥ 1 / 6 .
From this, we get
[Equation]
Also, since η ≤ 1 LN we have
[Equation]
which implies that {+y 
+}
E
[Equation]
y
[Equation]
Substituting (164), (165), and (166) to (163), we obatin
E (cid:2)
[Equation]
k (cid:12)(cid:12)
[Equation]
Next, we have the following lemma that we can apply to E [ | x 1 l − y
| ] .
Proof of Lemma 19 can be found in Appendix H.4.
Lemma 19.
For y 0 ≥ 0 , 0 ≤ i ≤ NB − 1 , 0 ≤ j ≤ B − 1 , and η ≤ 1 1025 LN ,
[Equation]
Applying this lemma to (167) and arranging the bounds (recall that k :=
iB + j ), we get
[Equation] [Equation]
The terms in (168) can be bounded using (cid:80) c −
[Equation]
=
[Equation]
√
[Equation]
≤ [Equation] √ tdt : [Equation]
[Equation]
where the second last inequality used B 4 ≤ j (and hence B 1 / 2 ≤ 2 j 1 / 2 ), and the last inequality used 2 iB + 3 j ≤ 4( iB + j ) = 4 k and 4 iB + 2 j ≤ 4( iB + j )
= 4 k .
Substituting (169) to (167), we get
E (cid:2) x 1 k (cid:12)(cid:12) E > 0 (cid:3) (170)
[Equation] [Equation]
[Equation]
The last inequality here used kηL ≤ ηLN ≤ 1 1025 , which follows from η ≤ 1 1025 LN .
Thus far, we have obtained an upper bound for E (cid:2) x 1 k (cid:12)(cid:12) E > 0 (cid:3) .
Recall that there is another conditional expectation in (161) that we want to bound, namely E (cid:2) xk (cid:12)(cid:12) E ≤ 0 (cid:3) .
We bound it below, using the tools developed so far.
For i ≤ N 2 B and B 4 ≤ j ≤ B 2 ,
[Equation]
(Using Lemma 14)
[Equation]
[Equation]
[-x 1 k − y 
 | [Equation] ≤ 0 (cid:3)-]Using Lemma 19)
[Equation]
Using (171) and (172) in (161), we get that for i ≤ N 2 B and B 4 ≤ j ≤ B 2 :
[Equation]
From Lemma 14, note that 16 ≤ P ( E > 0) ≤ 56 and 16 ≤ P ( E ≤ 0) ≤ 56 .
We use these inequalities, along with kηL ≤ ηLN ≤ 1 1025 and Lµ ≥ 153752 , to bound the terms appearing in (173).
[Equation]
We also have
[Equation]
where we used the assumption Lµ ≥ 153752 .
Substituting (174) and (175) to (173) [Equation] we get
[Equation]
which ﬁnishes the proof.
H. P ROOF OF L EMMA
[Equation]
where the last inequality used Lemma 19.
H. P ROOF OF L EMMA
We consider iterates within a single epoch, and hence we omit the subscripts denoting epochs.
In our construction, each machine has the same set of component functions, that is, there will be no inter-machine deviation.
We therefore omit the superscript m from the local component functions.
Consider the function
[Equation]
 , where
[Equation]
Hence, G 2 ( x ) =
Lx 2 2 .
We prove the lemma by coupling iterates corresponding to F 2 and G 2 .
In particular, we perform local RR on F 2 and G 2 such that both start the given epoch at y 0 and all the corresponding machines use the same random permutations.
Let x miB + j,F and x miB + j,G denote the iterates (for ( iB + j ) -th iteration at machine m ) for F 2 and G 2 respectively.
We use mathematical induction to prove that x miB + j,F ≥ x miB + j,G for all i = 0 , . .
. , NB − 1 and j = 0 , . . .
, B and machines m = 1 , . . .
, M .
After that, we will use this to prove our desired statement E [ y NB ,F | y 0 < 0] =
E [ 1 M (cid:80) Mm =
[Equation]
x mN,F | y 0 < 0] ≥ (1 − 7 ηLN 8 B [Equation] y 0 .
Let σ m be a random permutation of N 2 +1 ’s and
N 2 − 1 ’s.
First we consider i = 0 and 0 ≤ j ≤ B .
Base case.
For the base case, we know that x m 0 ,F ≥ x m 0 ,G , since x m 0 ,F = x m 0 ,G = y 0 for all m .
Inductive case.
There can be three cases:
• Case 1: x miB + j,F ≥ x miB + j,G ≥ 0 .
Then,
[Equation]
j
+
[Equation]
+
[Equation]
[Equation]
[Equation]
• Case 2: 0 ≥ x miB + j,F ≥ x miB + j,G .
Then,
[Equation]
[Equation]
j
[Equation]
• Case 3: x iB + j,F ≥ 0 ≥ x iB + j,G .
Then,
x miB + j +
F −
x
miB + j +
G = x
[Equation]
j
[Equation]
x
[Equation]
j [Equation] [Equation] Note that since η ≤ 1 LN , we get that x miB + j,F (1 − ηµ ) ≥ 0 and x miB + j,G (1 − ηL ) ≤ 0 , which proves that x miB + j +1 ,F − x miB + j +1 ,G ≥ 0 .
Thus, we see that x miB + j +1 ,F ≥ x miB + j +1 ,G for all the three cases, which proves by mathematical induction that x miB + j,F ≥ x miB + j,G for all 0 ≤ j ≤ B and i = 0 .
Note that this implies that, the aggregated averages y 1 ,F := 1 M (cid:80)
Mm =1 x mB,F and y 1 ,G := 1 M (cid:80) Mm =1 x mB,G satisfy y 1 ,F ≥ y 1 ,G .
Hence, after synchronization is complete, we get that for i
= 1 and j = 0 , x miB + j,F ≥ x miB + j,G for all machines m .
This proves the base case for i = 1 .
Now, we can repeat the Inductive cases for 1 ≤ j ≤ B and i = 1 , and thereby prove that y 2 ,F ≥ y 2 ,G .
Continuing on this process, we get that x miB + j,F ≥ x miB + j,G for all 0 ≤ j ≤ B and 0 ≤ i ≤ NB − 1 , and consequently, y NB ,F ≥ y NB ,G .
Further, by linearity of expectation and gradient, it is easy to check that for any machine m ,
[Equation]
Using the result that y NB ,F ≥ y NB ,G which we proved above, we get E [ y NB ,F ] ≥ (1 − ηL ) N y 0 for any initial iterate y 0 .
Speciﬁcally for y 0 < 0 , this implies E [ y NB ,F | y 0 < 0] ≥ (1 − ηL ) N y 0 .
Further, since ηL ≤ 1 1025 N , we have (1 − ηL ) N ≤ 1 − 7 ηLN 8 .
This is because 1 − 7 zN 8 − (1 − z ) N is nonnegative on the interval (cid:104) 0 , 1 − (7 / 8) 1 N − 1 (cid:105) , and 1 − (7 / 8) 1 N − 1 ≥ 1 1025 N for all N ≥ 2 .
To see why, note that (1 − 1 1025( n − 1) )
n − 1 ≥ 78 for all n ≥ 2 , and this gives 1 − (7 / 8) 1 n − 1 ≥ 1 1025( n − 1) , which then implies 1 − (7 / 8) 1 n − 1 ≥ 1 1025 n for all n ≥ 2 .
Therefore, for y 0 < 0 , we have E [ y NB ,F | y 0 < 0] ≥ (1 − 7 ηLN 8 ) y 0 .
For the last statement of the lemma, note that by symmetry of the function G 2 , if we initialize
Algorithm 1 at 0 , then for any starting iterate of an epoch we have P ( y 0 ,G ≥ 0) ≥ 1 / 2 .
This combined with the fact that y i,F ≥ y i,G gives us that P ( y 0 ,F ≥ 0) ≥ 1 / 2 .
H. P ROOF OF L EMMA
[Equation]
0 ) x ml + η iB + j − 1 (cid:88) l = iB νσ
1 l +1 + ( L 1 x 1 l < 0 + µ 1 x 1 l ≥ 0 )
x 1 l (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) ≤ ην (cid:32)(cid:114) iB
[Equation] iB + j −
[Equation]
l = ( L 1 x 1 l < 0 + µ 1 x 1 l ≥ 0 ) x 1 l (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35)
(By Lemma 14) ≤ ην (cid:32)(cid:114) iB
[Equation] ≤ ην (cid:32)(cid:114) iB
[Equation]
Now deﬁne
[Equation]
l
[Equation]
[-Then using induction, it can be seen that E [ | x 1 k − y 
 | ] ≤-] h (
[Equation]
In terms of i and j , note that k corresponds to k = iB + j .
Then using induction, it can be seen that E [ | x 1 k {+− y 
+}
| ] ≤ h ( k ) .
Further, since h ( k ) is an increasing function of k , we get h
[Equation]
Since k ≤ N and η ≤ 1 1025 LN , we get that
[Equation]
as desired.
I P ROOF OF LOWER BOUND FOR LOCAL RR: HETEROGENEOUS CASE (P ROPOSITION 5)
Recall that Proposition 5 gives the bound for local RR in the heterogeneous setting, where different machines have different local objectives.
In this section, we construct examples where there is no intra-machine variation (i.e., f m 1 = f m 2 = · · · = f mN for all m ∈
[ M ] ), but there is certain level of heterogeneity among different machines.
Similar to the other two lower bounds, we consider four step-size ranges and do case analysis for each of them.
This time, we construct a single function F for these step-size regimes such that the convergence of local RR is “slow” for F .
The ﬁnal lower bound is the minimum among the lower bounds obtained for the four regimes.
More concretely, we will construct a one-dimensional function F ( x ) satisfying L -smoothness (1), µ -PŁ condition (2), and Assumption 3 such that
• Local RR on F ( x ) with η ≤ 1 8 µNK and initialization y
= τµ results in
[Equation]
(cid:18) [Equation] µ (cid:19) . • Local RR on F ( x ) with 1 8 µNK ≤ η ≤ 1 8 µB and initialization y
= 0 results in
E [ F ( y K
[Equation]
Ω
[Equation]
BµN 2 K(cid:19)
[Equation]
• Local RR on F ( x ) with 1 8 µB ≤ η ≤ 1 µ and initialization y
= 0 results in
E [ F ( y K
[Equation]
Ω
[Equation]
µ
[Equation]
• Local RR on F ( x ) with η ≥ 1 µ and initialization y
= τµ results in
E
[Equation]
NB
[Equation]
µ
[Equation]
In the subsequent subsections, we prove the lower bounds for F for the four step-size intervals.
I. L OWER BOUND FOR 1 8 µNK ≤ η ≤ 1 8 µB AND 1 8 µB ≤ η ≤ 1 µ
We ﬁrst consider the two intervals in the middle, because they are more interesting cases.
The global objective function F and its local objective functions are as follows.
[Equation]

 , where f 1 ( x ) := − τx, and f 2 ( x ) := µx 2 + τx
In this construction, M/ 2 machines will have the function f 1 as their N local component functions (and hence their local objective functions) and the other M/ 2 machines will have the function f 2 .
Then, B local RR updates in each machine corresponds to B updates using either f 1 or f 2 .
If we start from x miB =
y i , the B local updates on machine m result in
[Equation]
if machine m has f 1 ,
[Equation]
if machine m has f 2 .
Taking the average of the M machines, we get that
[Equation]
[Equation]
Since there are total NKB such communication rounds over K epochs, at the end of the run we have
[Equation]
where we used initialization y 0 = 0 .
Having deﬁned the function and calculated its last iterate (176), let us now handle the two step-size regimes separately.
We ﬁrst consider 1 8 µNK ≤ η ≤ 1 8 µB .
In this case, we exploit the fact that
[Equation]
when 0 ≤ η ≤ 1 8 µB .
To see why, consider substituting z := 2 ηµ .
Then h 1 ( z ) := 1 − Bz + B 2 z 2 − (1 − z ) B has h (cid:48)(cid:48) 1 ( z ) ≥ 0 on z ∈ [0 , 1] , h (cid:48) 1 (0)
= 0 , and h 1 (0) =
0 , implying that h 1 ( z ) ≥ 0 on z ∈ [0 , 1] .
On the other hand, let h 2 ( z ) :
= 1 − Bz + B 2 z 2− (1 − z ) B .
If B = 2 , then h 2 ≡ 0 .
If
B > 2 , then it can be checked that h 2 ( z ) ≤ 0 for small enough interval [0 , 14 B ] .
Using (177) on (176),
[Equation]
where we used ηµB ≤ 18 .
Now, substituting η ≥ 1 8 µNK to above, we obtain
[Equation]
Therefore, F ( y K, NB ) = Ω (cid:16) τ 2 B 2 µN 2 K 2 (cid:17) .
Next, consider 1 8 µB ≤ η ≤ 1 µ .
We take a close look at the term that appears in (176):
[Equation]
For this term, we would like to ﬁnd a lower bound which holds for all η ∈ [ 1 8 µB , 1 µ ] .
To this end, consider substituting z :
=
2 ηµ .
Then, the function
[Equation]
is increasing on z ∈ [ 14 B , 1] , and we have
[Equation]
Using B ≥ 2 , h 3 ( 14 B ) can be lower-bounded as
[Equation]
Next, for z ≥ 1 , the derivative of h 3 ( z ) :
= B − (cid:80) B − 1 j =
[Equation]
1 − z ) j = B −
[Equation]
{+z is h (cid:48) 
 ( z ) = 1 − (1 − z ) B −+} 1 (( B − 1) z
+1) z . Since B ≥ 2 is assumed to be even, it is easy to check that h (cid:48) 3 ( z ) ≥ 0 for z ≥ 1 , which means that h 3 keeps increasing on [1 , 2] .
Therefore, we conclude that
[Equation]
, for all z
[Equation]
, and hence
[Equation]
for all 1 8 µB ≤ η ≤ 1 µ .
Using y K, NB from (176), we get
[Equation]
[Equation]
µ. Here, we used the fact that (cid:80) NKB − 1 l = (cid:0)
[Equation]
l ≥
[Equation] [Equation]
B
[Equation]
=
[Equation]
Hence, we obtain F ( y K, NB ) =
Ω( τ 2 µ ) , ﬁnishing the proof.
I. L OWER BOUND FOR η ≤ 1 8 µNK AND η ≥ 1 µ
We now conclude with the “extreme” step-size regimes.
We consider the same function F as in the previous subsection, but with a different initialization y
= τµ .
For F , recall from (176) that
[Equation]
This time, we want to lower-bound the second term on the RHS of (179) with zero and focus on the ﬁrst term.
To this end, we revisit our discussion on h 3 (178).
It is easy to check that h 3 is in fact increasing on the entire [0 , ∞ ) , and h 3 (0) = 0 .
This shows B − (cid:80) B − 1 j =0 (1 − 2 ηµ ) j ≥ 0 for any η ≥ 0 .
Next, since B is even, 12 (cid:0) 1 + (1 − 2 ηµ ) B (cid:1) ≥ 0 for any η ≥ 0 .
This gives
[Equation]
[Equation]
[Equation]
First, consider the interval 0 ≤ η ≤ 1 8 µNK .
Recall from (177) that for this η ,
[Equation]
so y K, NB ≥
[Equation]
since NKB ≥ 1 .
Hence, F ( y K, NB ) =
Ω( τ 2 µ ) .
Finally, if η ≥ 1 µ , then we have (1 − 2 ηµ ) B ≥ 1 , so
[Equation]
As a result, F ( y K, NB ) =
Ω( τ 2 µ ) .