P ERMUTATION -B ASED SGD: I S R ANDOM O PTIMAL ?
Anonymous authors
Paper under double-blind review
A BSTRACT
A recent line of ground-breaking results for permutation-based SGD has corroborated a widely observed phenomenon: random permutations offer faster convergence than with-replacement sampling.
However, is random optimal?
We show that this depends heavily on what functions we are optimizing, and the convergence gap between optimal and random permutations can vary from exponential to nonexistent.
We ﬁrst show that for 1-dimensional strongly convex functions, with smooth second derivatives, there exist permutations that offer exponentially faster convergence compared to random.
However, for general strongly convex functions, random permutations are optimal.
Finally, we show that for quadratic, strongly-convex functions, there are easy-to-construct permutations that lead to accelerated convergence compared to random.
Our results suggest that a general convergence characterization of optimal permutations cannot capture the nuances of individual function classes, and can mistakenly indicate that one cannot do much better than random.
I NTRODUCTION
Finite sum optimization seeks to solve the following:
[Equation]
Stochastic gradient descent (SGD) approximately solves ﬁnite sum problems, by iteratively updating the optimization variables according to the following rule:
[Equation]
where α is the step size and σ t ∈ [ n ] = { 1 , 2 , . .
. , n } is the index of the function sampled at iteration t .
There exist various ways of sampling σ t , with the most common being with- and without-replacement sampling.
In the former, σ t is uniformly chosen at random from [ n ] , and for the latter, σ t represents the t -th element of a random permutation of [ n ] .
We henceforth refer to these two SGD variants as vanilla and permutation-based, respectively.
Although permutation-based SGD has been widely observed to perform better in practice (Bottou, 2009; Recht & R´e, 2012; 2013), the vanilla version has attracted the vast majority of theoretical analysis.
This is because of the fact that at each iteration, in expectation the update is a scaled version of the true gradient, allowing for simple performance analyses of the algorithm, e.g. , see (Bubeck et al., 2015).
Permutation-based SGD has resisted a tight analysis for a long time.
However, a recent line of breakthrough results provides the ﬁrst tight convergence guarantees for several classes of convex functions F (Nagaraj et al., 2019; Safran & Shamir, 2019; Rajput et al., 2020; Mishchenko et al., 2020; Ahn et al., 2020; Nguyen et al., 2020).
These recent studies mainly focus on two variants of permutation-based SGD where (1) a new random permutation is sampled at each epoch (also known as R ANDOM R ESHUFFLE )
(Nagaraj et al., 2019; Safran & Shamir, 2019; Rajput et al., 2020), and (2) a random permutation is sampled once and is reused throughout all SGD epochs (S INGLE S HUFFLE )
(Safran & Shamir, 2019; Mishchenko et al., 2020;
Ahn et al., 2020).
Perhaps interestingly, R ANDOM R ESHUFFLE and S INGLE S HUFFLE exhibit different convergence rates and a performance gap that varies across different function classes.
In particular, when run for K epochs, the convergence rate for strongly convex functions is (cid:101) O
(1 /nK 2 ) for both R ANDOM
R ESHUFFLE and S INGLE S HUFFLE (Nagaraj et al., 2019; Ahn et al., 2020; Mishchenko et al., 2020).
However, when run speciﬁcally on strongly convex quadratics, R ANDOM R ESHUFFLE experiences an acceleration of rates, whereas S INGLE S HUFFLE does not (Safran & Shamir, 2019; Rajput et al., 2020; Ahn et al., 2020; Mishchenko et al., 2020).
All the above rates have been coupled by matching lower bounds, at least up to constants and sometimes log factors (Safran & Shamir, 2019; Rajput et al., 2020).
Algorithm
Input: Initialization x 10 , step size α , epochs K 1: σ = a random permutation of [ n ] 2: for k = 1 , . .
, K do 3: if IGD then 4: σ k = (1 , 2 , . . . , n ) 5: else if S INGLE S HUFFLE then 6: σ k = σ 7: else if R ANDOM R ESHUFFLE then 8: σ k = a random permutation of [ n ] 9: end if 10: if F LIP F LOP and k is even then 11: σ k = reverse of σ k −12: end if 13: for i = 1 , . . .
, n do 14: x ki := x ki − 1 − α ∇ f σ ki ( x ki − 1 )
 
Epoch k 15: end for 16: x k
+ := x kn 17: end for
Plain with F LIP F LOP
RR Ω (cid:18) n 2 K 2 + nK(cid:19) (cid:101) O (cid:18) n 2 K 2 + nK(cid:19)
[Table]
(RR), S INGLE S HUFFLE (SS) and I NCREMENTAL
G RADIENT D ESCENT (IGD) on strongly convex quadratics:
Plain vs. F LIP F LOP .
Lower bounds for the “plain” versions are taken from (Safran & Shamir, 2019).
When n (cid:29) K , that is when the training set is much larger than the number of epochs, which arguably is the case in practice, the convergence rates of R ANDOM R ESHUFFLE ,S INGLE S HUFFLE , and I NCREMENTAL G RADIENT
D ESCENT are Ω( 1 nK 3 ) , Ω( 1 nK 2 ) , and Ω( 1 K 2 ) respectively.
On the other hand, by combining these methods with F LIP F LOP the convergence rates become faster, i.e. , (cid:101) O
( 1 nK 5 ) , (cid:101) O
( 1 nK 4 ) , and (cid:101) O
( 1 K 3 ) , respectively.
From the above we observe that reshufﬂing at the beginning of every epoch may not always help.
But then there are cases where R ANDOM
R ESHUFFLE is faster than S INGLE
S HUFFLE , implying that certain ways of generating permutations are more suited for certain subfamilies of functions.
The goal of our paper is to take a ﬁrst step into exploring the relationship between convergence rates and the particular choice of permutations.
We are particularly interested in understanding if random permutations are as good as optimal, or if SGD can experience faster rates with carefully crafted permutations.
As we see in the following, the answer the above is not straightforward, and depends heavily on the function class at hand.
Our Contributions: We deﬁne as permutation-based SGD to be any variant of the iterates in (2), where a permutation of the n functions, at the start of each epoch, can be generated deterministically, randomly, or with a combination of the two.
For example, S INGLE S HUFFLE , R ANDOM R ESHUFFLE , and incremental gradient descent (IGD), are all permutation-based SGD variants (see
Algorithm 1).
We ﬁrst want to understand—even in the absence of computational constraints in picking the optimal permutations—what is the fastest rate one can get for permutation-based
SGD?
In other words, are there permutations that are better than random in the eyes of SGD?
Perhaps surprisingly, we show that there exist permutations that may offer up to exponentially faster convergence than random permutations, but for a limited set of functions.
Speciﬁcally, we show this for 1-dimensional functions (Theorem 1).
However, such exponential improvement is no longer possible in higher dimensions (Theorem 2), or for general strongly convex objectives (Theorem 3), where random is optimal.
The above highlight that an analysis of how permutations affect convergence rates needs to be nuanced enough to account for the structure of functions at hand.
Otherwise, in lieu of further assumptions, random permutations may just appear to be as good as optimal.
In this work, we further identify a subfamily of convex functions, where there exist easy-togenerate permutations that lead accelerated convergence.
We speciﬁcally introduce a new technique,
F LIP F LOP , which can be used in conjunction with existing permutation-based methods, e.g. , R ANDOM R ESHUFFLE , S INGLE S HUFFLE , or I NCREMENTAL G RADIENT D ESCENT , to provably improve their convergence on quadratic functions (Theorems 4, 5, and 6).
The way that F LIP F LOP works is rather simple: every even epoch uses the ﬂipped (or reversed) version of the previous epoch’s permutation.
The intuition behind why F LIP F LOP leads to faster convergence is as follows.
Towards the end of an epoch, the contribution of earlier gradients gets attenuated.
To counter this, we ﬂip the permutation for the next epoch so that every function’s contribution is diluted (approximately) equally over the course of two consecutive epochs.
F LIP F LOP demonstrates that ﬁnding better permutations for speciﬁc classes of functions might be computationally easy.
We summarize F LIP F LOP ’s convergence rates in Table 1 and report the results of numerical veriﬁcation in Section 6.2.
Note that in this work, we focus on the dependence of the error on the number of iterations, and in particular, the number of epochs.
However, we acknowledge that its dependence on other parameters like the condition number is also very important.
We leave such analysis for future work.
Notation: We use lowercase for scalars ( a ), lower boldface for vectors ( a ), and upper boldface for matrices ( A ).
R ELATED W ORK
G¨urb¨uzbalaban et al.
(2019a;b) provided the ﬁrst theoretical results establishing that R ANDOM
R ESHUFFLE and I NCREMENTAL G RADIENT D ESCENT (and hence S INGLE S HUFFLE ) were indeed faster than vanilla SGD, as they offered an asymptotic rate of O (cid:0) 1 /K 2 (cid:1) for strongly convex functions, which beats the convergence rate of O (1 /nK ) for vanilla SGD when K = Ω( n ) .
Shamir (2016) used techniques from online learning and transductive learning theory to prove an optimal convergence rate of (cid:101) O
(1 /n ) for the ﬁrst epoch of R ANDOM R ESHUFFLE (and hence S INGLE S HUFFLE ).
Later,
Haochen & Sra (2019) also established a non-asymptotic convergence rate of (cid:101) O (cid:0) 1 n 2 K 2 + 1 K 3 (cid:1) , when the objective function is quadratic, or has smooth Hessian.
Nagaraj et al.
(2019) used a very interesting iterate coupling based approach to give a new upper bound on the error rate of R ANDOM R ESHUFFLE , thus proving for the ﬁrst time that for general strongly convex smooth functions, it converges faster than vanilla SGD in all regimes of n and K .
This was followed by (Safran & Shamir, 2019), where the authors were able to establish the ﬁrst lower bounds, in terms of both n and K , for R ANDOM R ESHUFFLE .
However, there was a gap in these upper and lower bounds.
The gap in the convergence rates was closed by Rajput et al.
(2020), who showed that the upper bound given by Nagaraj et al.
and the lower bound given by Safran & Shamir (2019) were both tight up to logarithmic terms.
For S INGLE S HUFFLE , Mishchenko et al.
and Ahn et al.
(2020) showed an upper bound of (cid:101) O (cid:0) 1 nK 2 (cid:1) , which matched the lower bound given earlier by (Safran & Shamir, 2019), up to logarithmic terms.
Ahn et al.
(2020) and Mishchenko et al.
(2020) also proved tight upper bounds for R ANDOM
R ESHUFFLE , with a simpler analysis and using more relaxed assumptions than (Nagaraj et al., 2019) and (Rajput et al., 2020).
In particular, the results by Ahn et al.
(2020) work under the PŁ condition and do not require individual component convexity.
I NCREMENTAL G RADIENT D ESCENT on strongly convex functions has also been studied well in literature (Nedi´c & Bertsekas, 2001; Bertsekas, 2011; G¨urb¨uzbalaban et al., 2019a).
More recently,
Nguyen et al.
(2020) provide a uniﬁed analysis for all permutation-based algorithms.
The dependence of their convergence rates on the number of epochs K is also optimal for I NCREMENTAL G RADIENT
D ESCENT , S INGLE S HUFFLE and R ANDOM R ESHUFFLE .
There has also been some recent work on the analysis of R ANDOM R ESHUFFLE on non-strongly convex functions and non-convex functions.
Speciﬁcally, Nguyen et al.
(2020) and Mishchenko et al.
(2020) show that even there, R ANDOM R ESHUFFLE outperforms SGD under certain conditions.
Mishchenko et al.
(2020) show that R ANDOM R ESHUFFLE and S INGLE S HUFFLE beat vanilla SGD on non-strongly convex functions after Ω( n ) epochs, and that R ANDOM R ESHUFFLE is faster than vanilla SGD on non-convex objectives if the desired error is O (1 / √ n ) .
Speeding up convergence by combining without replacement sampling with other techniques like variance reduction (Shamir, 2016; Ying et al., 2020) and momentum (Tran et al., 2020) has also received some attention.
In this work, we solely focus on the power of “good permutations” to achieve fast convergence.
P RELIMINARIES
We will use combinations of the following assumptions:
Assumption 1 (Component convexity) .
f i ( x ) ’s are convex.
Assumption 2 (Component smoothness) .
[Equation]
are L -smooth
[Equation]
Note that Assumption 2 immediately implies that F also has L -Lipschitz gradients:
[Equation]
Assumption 3 (Objective strong convexity) .
F is µ -strongly convex, i.e.,
[Equation]
Note that Assumption 3 implies
[Equation]
We denote the condition number by κ , which is deﬁned as κ = Lµ .
It can be seen easily that κ ≥always.
Let x ∗ denote the minimizer of Eq.
(1), that is, x ∗ = arg min x F ( x ) .
We will study permutation-based algorithms in the constant step size regime, that is, the step size is chosen at the beginning of the algorithm, and then remains ﬁxed throughout.
We denote the iterate after the i -th iteration of the k -th epoch by x ki .
Hence, the initialization point is x 10 .
Similarly, the permutation of (1 , . .
. , n ) used in the k -th epoch is denoted by σ k , and its i -th ordered element is denoted by σ ki .
Note that if the ambient space is 1-dimensional, then we represent the iterates and the minimizer using non-bold characters, i.e. x ki and x ∗ , to remain consistent with the notation.
In the following, due to lack of space, we only provide sketches of the full proofs, when possible.
The full proofs of the lemmas and theorems are provided in the Appendix.
E XPONENTIAL C ONVERGENCE IN 1-D IMENSION
In this section, we show that there exist permutations for Hessian-smooth 1 -dimensional functions that lead to exponentially faster convergence compared to random.
Assumption 4 (Component Hessian-smoothness) .
f i ( x ) ’s have L H -smooth second derivatives, that is,
[Equation]
We also deﬁne the following instance dependent constants: G ∗ := max i (cid:107)∇ f i ( x ∗ ) (cid:107) , D = max
[Equation]
x 10 − x ∗ (cid:107) , G
[Equation]
2 L (cid:111) , and G =
G ∗ + 2 DL .
Theorem 1.
Let Assumptions 1,2,3 and 4 hold.
Let D and G be as deﬁned above.
If α = µ 8 n ( L 2 + L H G ) , then there exists a sequence of permutations σ 1 , σ 2 , . .
, σ K such that using those permutations from any initialization point x 10 gives the error
[Equation]
where C = µ 2 16( L 2 + L H G ) .
[Figure]
− −    x kopt x k | x kopt − x k | proof sketch.)
Assume that the minimizer is at the origin.
The proof of Theorem 1 ﬁrst shows that there exists an initialization and a sequence of permutations, such that using those, we get to the exact minimizer.
Let the sequence of iterates for this run be x kopt .
Consider a parallel run, which uses the same sequence of permutations, but an arbitrary initialization point.
Let this sequence be x k .
The ﬁgure shows how x kopt converges to the exact optima, and the distance between x kopt and x k decreases exponentially, leading to an exponential convergence for x k .
An important thing to notice in the theorem statement is that the sequence of permutations σ 1 , σ 2 , . . .
, σ K only depends on the function, and not on the initialization point x 10 .
This implies that for any such function, there exists a sequence of permutations σ 1 , σ 2 , . . .
, σ K , which gives exponentially fast convergence, unconditionally of the initialization.
Note that the convergence rate is slower than Gradient
Descent, for which the constant ‘ C ’ would be larger.
However, here we are purely interested in the convergence rates of the best permutations and their (asymptotic) dependence on K .
Proof sketch The core idea is to establish that there exists an initialization point x 10 (close to the minimizer x ∗ ), and a sequence of permutations such that that starting from x 1 0 and using that sequence of permutation leads us exactly to the minimizer.
Once we have proved this, we show that if two parallel runs of the optimization process are initialized from two different iterates, and they are coupled so that they use the exact same permutations, then they approach each other at an exponential rate.
Thus, if we use the same permutation from any initialization point, it will converge to the minimizer at an exponential rate.
See Figure 1 for a graphical depiction of this sketch.
We note that the ﬁgure is not the result of an actual run, but only serves to explain the proof sketch.
L OWER B OUNDS FOR P ERMUTATION - BASED SGD
The result in the previous section leads us to wonder if exponentially fast convergence can be also achieved in higher dimensions.
Unfortunately, for higher dimensions, there exist strongly convex quadratic functions for which there does not exist any sequence of permutations that lead to an exponential convergence rate.
This is formalized in the following theorem.
Theorem 2.
For any n ≥ 4 ( n must be even), there exists a 2 n + 1 -dimensional strongly convex function F which is the mean of n convex quadratic functions, such that for every permutation-based algorithm with any step size,
[Equation]
This theorem shows that we cannot hope to develop constant step size algorithms that exhibit exponentially fast convergence in multiple dimensions.
Proof sketch
In this sketch, we explain a simpler version of the theorem, which works for n =and 2-Dimensions.
Consider F ( x, y )
=
12 f 1 ( x, y )
+ 12 f 2 ( x, y ) such that
[Equation]
Hence, F = 1 4 ( x 2 + y 2 ) , and has minimizer at the origin.
Each epoch has two possible permutations, σ = (1 , 2) or σ = (2 , 1) .
Working out the details manually, it can be seen that regardless of the permutation, x k + + y k + > x k
+ y k 0 , that is, the sum of the co-ordinates keeps increasing.
This can be used to get a bound on the error term (cid:107)
[ x K
[Equation]
Next, we show that even in 1-Dimension, individual function convexity might be necessary to obtain faster rates than R ANDOM R ESHUFFLE .
Theorem 3.
There exists a 1-Dimensional strongly convex function F which is the mean of two quadratic functions f 1 and f 2 , such that one of the functions is non-convex.
Then, every permutationbased algorithm with constant step size α ≤ 1 L gives an error of at least
[Equation]
Proof sketch The idea behind the sketch is to have one of the two component functions as strongly concave.
This gives it the advantage that the farther away from its maximum the iterate is, the more it pushes the iterate away.
Hence, it essentially results in increasing the deviation in each epoch.
This leads to a slow convergence rate.
In the setting where the individual f i may be non-convex, Nguyen et al.
(2020) and Ahn et al.
(2020) show that S INGLE S HUFFLE , R ANDOM R ESHUFFLE , and I NCREMENTAL
G RADIENT D
ESCENT achieve the error rate of (cid:101) O
( 1 K 2 ) , for a ﬁxed n .
In particular, their results only need that the component functions be smooth and hence their results apply to the function F from Theorem 3.
The theorem above essentially shows that when n = 2 , this is the best possible error rate, for any permutationbased algorithm - deterministic or random.
Hence, at least for n = 2 , the three algorithms are optimal when the component functions can possibly be non-convex.
However, note that here we are only considering the dependence of the convergence rate on K .
It is possible that these are not optimal, if we further take into account the dependence of the convergence rate on the combination of both n and
K .
Indeed, if we consider the dependence on n as well, I NCREMENTAL G RADIENT D ESCENT has a convergence rate of Ω(1 /K 2 ) (Safran & Shamir, 2019), whereas the other two have a convergence rate of (cid:101) O
(1 /nK 2 ) (Ahn et al., 2020).
F LIPPING P ERMUTATIONS FOR F ASTER C ONVERGENCE IN Q UADRATICS
In this section, we introduce a new algorithm F LIP F LOP , that can improve the convergence rate of
S INGLE S HUFFLE ,R ANDOM R ESHUFFLE , and I NCREMENTAL G RADIENT D ESCENT on strongly convex quadratic functions.
The following theorem gives the convergence rate of F LIP F LOP WITH S INGLE S HUFFLE :
Assumption 5.
f i ( x ) ’s are quadratic.
Theorem 4.
If Assumptions 1, 2, 3, and 5 hold, then running F LIP F LOP WITH S INGLE S HUFFLE for K epochs, where K ≥ 80 κ 3 / 2 log( nK ) max (cid:110) 1 , √ κn (cid:111) is an even integer, with step size α = 10log( nK ) µnK gives the error
[Equation]
For comparison, Safran & Shamir (2019) give the following lower bound on the convergence rate of vanilla S INGLE S HUFFLE :
[Equation]
Note that both the terms in Eq.
(4) are smaller than the term in Eq.
In particular, when n (cid:29) Kand n is ﬁxed as we vary K , the RHS of Eq.
(5) decays as (cid:101) O
( 1 K 2 ) , whereas the RHS of Eq.
(4) decays as (cid:101) O
( 1 K 4 ) .
Otherwise, when K 2 (cid:29) n and K is ﬁxed as we vary n , the RHS of Eq.
(5) decays as (cid:101) O
( 1 n ) , whereas the RHS of Eq.
(4) decays as (cid:101) O
( 1 n 2 ) .
Hence, in both the cases, F LIP F LOP WITH
S INGLE S HUFFLE outperforms S INGLE S HUFFLE .
The next theorem shows that F LIP F LOP improves the convergence rate of R ANDOM R ESHUFFLE :
Theorem 5.
If Assumptions 1, 2, 3, and 5 hold, then running F LIP F LOP WITH R ANDOM R ESHUFFLE for K epochs, where K ≥ 55 κ log( nK ) max (cid:8) 1 , (cid:112) nκ (cid:9) is an even integer, with step size α = 10log nK µnK gives the error
[Equation]
[Figure]
We set n = 800 , so that n (cid:29) K and hence the higher order terms of K dominate the convergence rates.
Note that both the axes are in logarithmic scale.
For comparison, Safran & Shamir (2019) give the following lower bound on the convergence rate of vanilla R ANDOM R ESHUFFLE :
[Equation]
Hence, we see that in the regime when n (cid:29) K , which happens when the number of components in the ﬁnite sum of F is much larger than the number of epochs, F LIP F LOP WITH R ANDOM R ESHUFFLE is much faster than vanilla R ANDOM R ESHUFFLE .
Note that the theorems above do not contradict Theorem 2, because for a ﬁxed n , both the theorems above give a convergence rate of (cid:101)
O
(1 /K 2 ) .
We also note here that the theorems above need the number of epochs to be much larger than κ , in which range Gradient Descent performs better than with- or without- replacement SGD, and hence
GD should be preferred over SGD in that case.
However, we think that this requirement on epochs is a limitation of our analysis, rather than that of the algorithms.
Finally, the next theorem shows that F LIP F LOP improves the convergence rate of I NCREMENTAL
G RADIENT D ESCENT .
Theorem 6.
If Assumptions 1, 2, 3, and 5 hold, then running F LIP F LOP WITH I NCREMENTAL GD for K epochs, where K ≥ 36 κ log( nK ) is an even integer, with step size α = 6log nK µnK gives the error
[Equation]
For comparison, Safran & Shamir (2019) give the following lower bound on the convergence rate of vanilla I NCREMENTAL G RADIENT D ESCENT :
[Equation]
In the next subsection, we give a short sketch of the proof of these theorems.
6. P ROOF SKETCH
In the proof sketch, we consider scalar quadratic functions.
The same intuition carries over to multi-dimensional quadratics, but requires a more involved analysis.
Let f i ( x ) :
= a i x 2 2 + b i x + c .
Assume that F ( x ) := 1 n (cid:80) n i =
1 f i ( x ) has minimizer at 0.
This assumption is valid because it can be achieved by a simple translation of the origin (see (Safran & Shamir, 2019) for a more detailed explanation).
This implies that (cid:80) ni
=1 b i = 0 .
For the sake of this sketch, assume x 10 = 0 , that is, we are starting at the minimizer itself.
Further, without loss of generality, assume that σ = (1 , 2 , . . . , n ) .
Then, for the last iteration of the ﬁrst epoch,
[Equation] [Equation]
Applying this to all iterations of the ﬁrst epoch, we get
[Equation]
Substituting x 10 = 0 , we get
[Equation]
Note that the sum above is not weighted uniformly: b 1 is multiplied by (cid:81) nj
=2 (1 − αa j ) , whereas b n is multiplied by 1.
Because (1 − αa j ) < 1 , we see that b 1 ’s weight is much smaller than b n .
If the weights were all 1, then we would get x 0 n = − α (cid:80) ni
=1 b i = 0 , i.e., we would not move away from the minimizer.
Since we want to stay close to the minimizer, we want the weights of all the b i to be roughly equal.
The idea behind F LIP F LOP is to add something like − α (cid:80) n i =1 b i (cid:81) i j =1 (1 − αa j ) in the next epoch, to counteract the bias in Eq.
To achieve this, we simply take the permutation that the algorithm used in the previous epoch and ﬂip it for the next epoch.
Roughly speaking, in the next epoch b 1 will get multiplied by 1 whereas b n will get multiplied by (cid:81) n − 1 j =1 (1 − αa j ) .
Thus over two epochs, both get scaled approximately the same.
To see more concretely, we look at the ﬁrst order approximation of Eq.
[Equation]
where in the last step above we used the fact that (cid:80) ni
=1 b i = 0 .
Now, let us see what happens in the second epoch if we use F LIP F LOP .
Doing a recursion analogous to how we got Eq.
(6), but reversing the order of functions, we get:
[Equation]
Carefully doing a change of variables in the equation above, we get:
[Equation]
Note that the product in the second term in the equation above is almost complementary to the product in Eq.
This is because we ﬂipped the order in the second epoch.
This will play an important part in cancelling out much of the bias in Eq.
Continuing on from Eq.
(9), we again do a linear approximation similar to before and substitute Eq.
(8) (and use the fact that x 20 = x 1 n ):
[Equation]
n
[Equation]
We assume that α is small and hence we only focus on the quadratic terms:
[Equation]
Now, since (cid:80) ni =1 b i = 0 , we get
[Equation]
Now, comparing the coefﬁcients of the α 2 terms in Eq.
(8) and Eq.
(10), we see that the former has
O ( n 2 ) terms whereas the latter has only n terms.
This correction of error is exactly what eventually manifests into the faster convergence rate of F LIP F LOP .
The main reason that the analysis for multidimensional quadratics is not as simple as the 1-dimensional analysis done above, is because unlike scalar multiplication, matrix multiplication is not commutative, and the AM-GM inequality is not true in higher dimensions (Lai & Lim, 2020; De Sa, 2020).
One way to bypass this inequality is by using the following inequality for small enough α :
[Equation]
Ahn et al.
(2020) proved a stochastic version of this (see Lemma 6 in their paper).
We prove a deterministic version in Lemma 3 (in the Appendix).
6. N UMERICAL V ERIFICATION
We verify the theorems numerically by running R ANDOM R ESHUFFLE , S INGLE S HUFFLE and their
F LIP F LOP versions on the task of mean computation.
We randomly sample n = 800 points from a 100 -dimensional sphere.
Let the points be x i for i = 1 , . . .
, n .
Then, their mean is the solution to the following quadratic problem : arg min
[Equation]
F
[Equation]
) =
[Equation]
i (cid:107) 2 .
We solve this problem by using the given algorithms.
The results are reported in Figure 2.
The results are plotted in a log – log graph, so that we get to see the dependence of error on the power of K .
Note that since the points are sampled randomly, I NCREMENTAL G RADIENT D
ESCENT essentially becomes S INGLE S HUFFLE .
Hence, to verify Theorem 6, we need ‘hard’ instances of I NCREMENTAL
G RADIENT D ESCENT , and in particular we use the ones used in Theorem 3 in (Safran & Shamir, 2019).
These results are also reported in a log – log graph in Figure 2.
We also tried F LIP F LOP in the training of deep neural networks, but unfortunately we did not see a big speedup there.
We ran experiments on logistic regression for 1-Dimensional artiﬁcial data, the results for which are in Appendix H.
C ONCLUSION AND F UTURE W ORK
In this paper, we explore the theoretical limits of permutation-based SGD for solving ﬁnite sum optimization problems.
We focus on the power of good, carefully designed permutations and whether they can lead to a much better convergence rate than random.
We prove that for 1-dimensional, strongly convex functions, indeed good sequences of permutations exist, which lead to a convergence rate which is exponentially faster than random permutations.
We also show that unfortunately, this is not true for higher dimensions, and that for general strongly convex functions, random permutations might be optimal.
However, we think that for some subfamilies of strongly convex functions, good permutations might exist and may be easy to generate.
Towards that end, we introduce a very simple technique, F LIP F LOP , to generate permutations that lead to faster convergence on strongly convex quadratics.
This is a black box technique, that is, it does not look at the optimization problem to come up with the permutations; and can be implemented easily.
This serves as an example that for other classes of functions, there can exist other techniques for coming up with good permutations.
Finally, note that we only consider constant step sizes in this work for both upper and lower bounds.
Exploring regimes in which the step size changes, e.g. , diminishing step sizes, is a very interesting open problem, which we leave for future work.
We think that the upper and lower bounds in this paper give some important insights and can help in the development of better algorithms or heuristics.
We strongly believe that under nice distributional assumptions on the component functions, there can exist good heuristics to generate good permutations, and this should also be investigated in future work.
E THICS S TATEMENT
This paper explores the theoretical limits of convergence rates of variants of popular SGD type algorithms.
Thus, the authors do not think there are any ethical concerns regarding the paper.
R EPRODUCIBILITY S TATEMENT
The proof for all the theoretical claims are provided in the Appendix.
The assumptions and notations are speciﬁed in the Preliminaries section (Section 3).
The code for all the experiments is provided in the supplementary materials zip ﬁle.
A D ISCUSSION AND F UTURE W ORK
Being the ﬁrst paper (to the best of our knowledge) to theoretically analyze the optimality of random permutations, we limited our scope to a speciﬁc, but common theoretical setting - strongly convex functions with constant step size. We think that future work can generalize the results of this paper to settings of non-convexity, variable step sizes, as well as techniques like variance reduction, momentum, etc. A. L OWER BOUNDS FOR VARIABLE STEP SIZES
All the existing lower bounds (to the best of our knowledge) work in the constant step size regime (Safran & Shamir (2019); Rajput et al.
; Safran & Shamir (2021)).
Thus, generalizing the lower bounds to variable step size algorithms would be a very important direction for future research.
However, the case when step sizes are not constant can be tricky to prove lower bounds, since the step size could potentially depend on the permutation, and the current iterate.
A more reasonable setting to prove lower bounds could be the case when the step sizes follow a schedule over epochs, similar to what happens in practice.
A. F ASTER PERMUTATIONS FOR NON - QUADRATIC OBJECTIVES
The analysis of FlipFlop leverages the fact that the Hessians of quadratic functions are constant.
We think that the analysis of FlipFlop might be extended to strongly convex functions or even PŁ functions (which are non-convex in general), under some assumptions on the Lipschitz continuity of the Hessians, similar to how Haochen & Sra (2019) extended their analysis of quadratic functions to more general classes.
A key take-away from FlipFlop is that we had to understand how permutation based SGD works speciﬁcally for quadratic functions, that is, we did a white-box analysis. In general, we feel that depending on the speciﬁc class of non-convex functions (say deep neural networks), practitioners would have to think about permutation-based SGD in a white-box fashion, to come up with better heuristics for shufﬂing. In a concurrent work, (https://openreview.net/pdf?id=7gWSJrP3opB), it is shown that by greedily sorting stale gradients, a permutation order can be found which converges faster for some deep learning tasks. Hence, there do exist better permutations than random, even for deep learning tasks.
A shufﬂing scheme similar to FlipFlop has been used in random coordinate descent for faster practical convergence (see page 231 in Nocedal & Wright (2006)). This should be further investigated empirically and theoretically in future work.
Even though the current analysis of FlipFlop does not directly go through for random coordinate descent, we think the analysis can be adapted to work. B P ROOF OF T HEOREM In certain places in the proof, we would need that α ≤ 14 nL . To see how this is satisﬁed, note that we have assumed that α ≤
[Equation]
in the theorem statement. Using the inequality µ ≤ L in α ≤
[Equation]
gives that α ≤
[Equation]
In this proof, we assume that the minimizer of F is at 0 to make the analysis simpler.
This assumption can be satisﬁed by simply translating the origin to the minimizer (Safran & Shamir, 2019).
There are three main components in the proof:
Say that an epoch starts off at the minimizer.
We prove that there exists at least one pair of permutations such that if we do two separate runs of the epoch, the ﬁrst using the ﬁrst permutation and the second using the second, then at the end of that epoch, the iterates corresponding to the two runs end up on opposite sides of the minimizer.
There exists a sequence of permutations and a point in the neighborhood of the minimizer, such that intializing at that point and using these sequence of permutations, we converge exactly to the minimizer.
Starting from any other point, we can couple the iterates with the iterates which were shown in the previous component, to get that these two sequences of iterates come close to each other exponentially fast.
We prove the ﬁrst and second components in the Subsections B.1 and B.2; and conclude the proof in
Subsection B.3 where we also prove the third component.
B. P ERMUTATIONS IN ONE EPOCH
In this subsection, we prove that if x 0 , x 1 , . . .
, x n are the iterates in an epoch such that x 0 = 0 , then there exists a permutation of functions such that x n ≥ 0 .
By the same logic, we show that there exists a permutation of functions such that x n ≤ 0 .
These will give us control over movement of iterates across epochs.
Order the gradients at the minimizer, ∇ f i (0) , in decreasing order.
WLOG assume that it is just ∇ f 1 , ∇ f 2 , . .
∇ f n .
We claim that this permutation leads to x n ≥ 0 .
We will need the following intermediate result.
Let y i , y i − 1 be such that y i
=
y i − 1 − α ∇ f i
( y i − 1 ) .
Assume α ≤ 1 /L
and y i − 1 ≥ x
i − 1 .
Then,
[Equation]
(11) that is, y i − 1 ≥ x
i − 1 = ⇒
y
i ≥ x i .
Because 0 is the minimizer, we know that (cid:80) ni
=1 ∇ f i (0) = 0 .
Also, recall that x 0 = 0 .
There can be two cases:
∀ i ∈ [1 , n ] : x i < 0 .
This cannot be true because if ∀ i : 1 ≤ i ≤ n − 1 : x i < 0 , then
[Equation]
where we used the fact that ∇ f i ( x ) ≤ ∇ f i ( y ) if x ≤ y and f i is convex.
Thus, ∃ i ∈ [1 , n ] : x i ≥ 0 .
Now, consider the sequence y i , y i +1 , . .
, y n such that y i
=and for j ≥ i + 1 , y j =
y j − 1 − α ∇ f j ( y j − 1 ) .
Then because α ≤ 1 /L and x i ≥ y i = 0 , we get
that
x j ≥ y j for all j ≥ i (
Using Ineq.
Hence, there is an i ≥ 1 such that x
i ≥ y i = 0 , and further x j ≥ y j for all j ≥ i .
Next, we repeat the process above for y i , . . .
, y n .
That is, there can be two cases: 1.
∀ j ∈
[ i + 1 , n ] : y j < 0 .
This cannot be true because if ∀ j : i + 1 ≤ j ≤ n − 1 : y j < 0 , then
[Equation]
Thus, ∃ j ∈ [ i + 1 , n ] : y j ≥ 0 .
Now, consider the sequence z j , z j +1 , . .
, z n such that z j = 0 and for k ≥ j + 1 , z k
=
z k − 1 − α ∇ f k ( z k − 1 ) .
Then because α ≤ 1 /L and y j ≥ z j = 0 , we get that y k ≥ z k for all
k ≥ j (Using Ineq.
Hence, there exists an integer j > i > 0 such that y j ≥ 0 .
We have already proved that x j ≥ y j .
Thus, we have that x j ≥ 0 .
We can continue repeating this process (apply the same two cases above for z j , z j +1 , . . .
, z n , and so on), to get that x n ≥ 0 .
We deﬁne p to be this non-negative value of x n .
Note that the following lemma gives us that the gradients are bounded by G
Lemma 1.
Deﬁne G ∗ := max i (cid:107)∇ f i ( x ∗ ) (cid:107) and D = max (cid:110) (cid:107) x 10 − x ∗ (cid:107) , G ∗ 2 L (cid:111) .
If Assumptions 2 and 3 hold, and α < 1 8 κnL , then for any permutation-based algorithm (deterministic or random), we have ∀ i, j, k : (cid:107) x ki − x ∗ (cid:107) ≤ 2 D, and (cid:107)∇ f j ( x ki ) (cid:107) ≤ G ∗ + 2 DL.
Because the gradients are bounded by G , we get that p ≤ nαG .
Similarly, we can show that the reverse permutation leads to x n ≤ 0 .
We deﬁne q to be this nonpositive value of x n .
Because we have assumed that the gradients are bounded by G , we get that q ≥ − nαG .
B. E XACT CONVERGENCE TO THE MINIMIZER
In this section, we show that there exists a point such that if we initialize there and follow a speciﬁc permutation sequence, we land exactly at the minimizer.
In particular, we will show the following: There exists a point in [4 q, 4 p ] such that if we initialize there and follow a speciﬁc permutation sequence, then we land exactly at the minimizer.
We show this recursively.
We will prove that there exists a point m K ∈
[4 q, 4 p ] such that if the last epoch begins there, that is x K
= m K , then we land at the minimizer at the end of the last epoch, that is, x Kn = 0 .
Then, we will show that there exists a point m K − 1 ∈
[4 q, 4 p ] such that if the x K − 1= m K − 1 , then x K
= x K − 1 n = m K .
Repeating this for K − 2 , . .
, 1 , we get that there exists a point m 0 ∈
[4 q, 4 p ] such that if we initialize the ﬁrst epoch there, that is x 10 = m 0 , then there is a permutation sequence such that ultimately x Kn = 0 .
We prove that any point m j ∈ [4 q, 4 p ] can be reached at the end of an epoch by beginning the epoch at some point m j − 1 ∈
[4 q, 4 p ] , that is if x j − 1= m j − 1 , then x j − 1 n = m j .
• Case: m j ∈
[ p, 4 p ] .
In this case, we show that m j − 1 ∈
0 , 4 p ] .
We have proved in the previous subsection that there exists a permutation σ such that if x j − [-
 =-]1= 0 then x j − 1 n = p .
Next, we have the following helpful lemma that we will also use later.
Lemma 2.
Let x 0 , x 1 , . . .
, x n be a sequence of iterates in an epoch and y 0 , y 1 , . . .
, y n be another sequence of iterates in an epoch such that both use the same permutation of functions.
If α ≤ µ 2 n ( L 2 + L H G ) , then
[Equation]
If we set x 0 = 4 p, y 0 = 0 in Lemma 2 and we follow the permutation σ , then we get that
[Equation]
(Since using σ and y 0 = 0 results in y n = p .)
[Equation]
where we used the fact that α ≤ 14 nL is the last step.
Thus, if x j − 1= 4 p and we follow the permutation σ , then x j −
[Equation] p .
Next, note that
[Equation]
Looking above, we see that x j − 1is
a continuous function of x j − 1; x j − 1is a continuous function of x j − ; and so on.
Thus, using the fact that composition of continuous functions is continuous, we get that x j − n is also a continuous function of
x
{+j − 
+}
We have shown that if x j − = then x j − 1 n = p and if x j − 1= 4 p
, then
x j − 1 n ≥ 4 p
Thus, using the fact that that x j − 1 n is a continuous function of x j − {+
 ,+} we get that for any point m
j ∈
[ p 4 p ] , there is at least one point m j − 1 ∈
[0 , 4 p ] , such that x j −
= m j − 1 leads to x j −
n = m j .
• Case: m j ∈ [4 q, q ] .
We can apply the same logic as above to show that there is at least one point m j − 1 ∈
[4 q, 0] , such that x j −
[Equation]
• Case: m j ∈ [ q, p ] .
WLOG assume that | q | < | p | .
Let σ q be the permutation such that if x j − 1=
and the epoch uses this permutation, then we end up at x j − 1 n = q .
If we set x 0 = 4 p, y 0 = 0 in Lemma 2 and we follow the permutation σ q , then we get that
[Equation]
(Since using σ q and y [-
-]0 = 0 results in y n = q .)
[Equation]
, where we used the fact that α ≤ 14 nL is the last step.
Thus, if x j − 1= 4 p and we follow the permutation σ q , then x j − 1 n ≥ 2 p .
Thus, using similar argument of continuity as the ﬁrst case, we know that there is a point m j − 1 ∈
[0 , 4 p ] , such that x j − 1= m j − 1 leads to x j − 1 n = m j when we use the permutation σ q .
B. S AME SEQUENCE PERMUTATIONS GET CLOSER
In the previous subsection, we have shown that there exists a point m 0 ∈ [4 q, 4 p ] and a sequence of permutations σ 1 , σ 2 , . . .
, σ K such that if x 10 = m 0 and epoch j uses permutation σ j , then x
Kn = 0 .
In this subsection, we will show that if x 10 is initialized at any other point such that | x 10 | ≤ D , then using the same permutations σ 1 , σ 2 , . . .
, σ K gives us that | x Kn | ≤ ( D + nαG ) e − K .
For this we will repeatedly apply Lemma 2 on all the K epochs.
Assume that x 10 = ν 0 with | ν 0 | ≤ D .
Let y ji be the sequence of iterates such that y 10 = m 0 and uses the permutation sequence σ 1 , σ 2 , . . .
, σ K .
Then, we know that y
Kn = 0 .
Let x ji be the sequence of iterates such that x 10 = νand uses the same permutation sequence σ 1 , σ 2 , . . .
, σ K .
Then, using Lemma 2 gives us that | y 1 n − x 1 n | ≤ | ν 0 − m 0 | (1 − µαn 2 ) .
Thus, we get that | y 20 − x 20 | ≤ | ν 0 − m 0 | (1 − µαn 2 ) .
Again applying Lemma 2 gives us that | y 2 n − x 2 n | ≤ | ν 0 − m 0 | (1 − µαn 2 ) 2 .
Therefore, after applying it K times, we get
[Equation]
Note that | x 10 | = | ν 0 | ≤ D , and | y 10 | = | m 0 | , with m 0 ∈ [4 q, 4 p ] .
We showed earlier in Subsection
B.1 that | p | ≤ nαG and | q | ≤ nαG .
Therefore,
[Equation]
Further, we know that y K n = 0 .
Thus,
[Equation]
Substituting the value of α completes the proof.
Next we prove the lemmas used in this proof.
B. P ROOF OF L EMMA
We restate the lemma below.
Lemma.
Deﬁne G ∗ :=
max i (cid:107)∇ f i ( x ∗ ) (cid:107) and D = max (cid:110) (cid:107) x 10 − x ∗ (cid:107) , G ∗ 2 L (cid:111) .
If Assumptions 2 andhold, and α < 1 8 κnL , then for any permutation-based algorithm (deterministic or random), we have [Equation]
i, j, k : (cid:107) x ki
[Equation]
and
[Equation]
This lemma says that for any permutation-based algorithm, the domain of iterates and the norm of the gradient stays bounded during the optimization.
This means that we can assume bounds on norm of iterates and gradients, which is not true in general for unconstrained SGD.
This makes analyzing such algorithms much easier, and hence this lemma can be of independent interest for proving future results for permutation-based algorithms.
Remark 1.
This lemma does not hold in general for vanilla SGD where sampling is done with replacement.
Consider the example with two functions f 1 ( x ) = x 2 − x , and f 2 ( x ) = x ; and
F ( x ) = f 1 ( x ) + f 2 ( x ) .
This satisﬁes Assumptions 2 and 3, but one may choose f 2 consecutively for arbitrary many iterations, which will lead the iterates a proportional distance away from the minimizer.
This kind of situation can never happen for permutation-based SGD because we see every function exactly once in every epoch and hence no particular function can attract the iterates too much towards its minimizer, and by the end of the epochs most of the noise gets cancelled out.
Note that the requirement α < 1 8 κnL is stricter than the usual requirement α = O (cid:0) 1 nL (cid:1) , but we believe the lemma should also hold in that regime.
For the current paper, however, this stronger requirement sufﬁces.
Proof.
To prove this lemma, we show two facts: If for some epoch k , (cid:107) x k 0 − x ∗ (cid:107) ≤ D , then a) ∀
i : (cid:107) x ki − x ∗ (cid:107) ≤ 2 D and b) (cid:107) x k + − x ∗ (cid:107) =
(cid:107) x
kn − x ∗ (cid:107) ≤ D .
To see how a) and b) are sufﬁcient to prove the lemma, assume that they are true.
Then, since the ﬁrst epoch begins inside the bounded region (cid:107) x 10 − x ∗ (cid:107) ≤ D , we see using b) that every subsequent epoch begins inside the same bounded region, that is (cid:107) x k 0 − x ∗ (cid:107) ≤ D as well.
Hence using a) we get that during these epochs, the iterates satisfy (cid:107) x
ki − x ∗ (cid:107) ≤ 2 D , which is the ﬁrst part of the lemma.
Further, this bound together with the gradient Lipshitzness directly gives the upper bound G ∗ + 2 DL on the gradients.
Thus, all we need to do to prove this lemma is to prove a) and b), which we do next.
We will prove a) and b) for D = max {(cid:107) x 10 − x ∗ (cid:107) , 2 κnαG ∗ 1 − 4 κnαL } .
Once we do this, using α < 1 8 κnL will give us the exact statement of the lemma.
Let | x k 0 − x ∗ | ≤ D for some epoch k .
Then, we try to ﬁnd the minimum number of iterations i needed so that | x ki − x ∗ | ≥ 2 D .
Within this region, the gradient is bounded by G ∗ + 2 DL .
Thus, the minimum number of iterations needed are
[Equation]
However,
[Equation]
(Using the fact that D ≥ 2 κnαG
[Equation] [Equation]
Thus, the minimum number iterations needed to go outside the bound | x k i − x ∗ | ≥ 2 D is more than the length of the epoch.
This implies that within the epoch, (cid:107) x ki − x ∗ (cid:107) ≤ 2 D , which proves a).
We prove b) next:
[Equation]
[Equation] Note that (cid:80) ni =0 ∇ f σ ki ( x k 0 ) is just the sum of all component gradients at x k 0 , that is (cid:80)
ni
=0 ∇ f σ ki ( x k 0 ) = n ∇ F ( x k 0 ) .
Using this, we get
[Equation]
Triangle inequality.)
[Equation]
where we used gradient Lipschitzness (Assumption 2) in the last step.
To bound the ﬁrst term above
[Equation]
we use the standard analysis of gradient descent on smooth, strongly convex functions as follows
[Equation]
where in the last step we used that α ≤ µnL 2 since α ≤ 1 8 κnL .
Substituting this inequality in Ineq.
(12), we get
[Equation]
We have already proven a) that says that the iterates x ki satisfy (cid:107) x ki − x ∗ (cid:107) ≤ 2 D .
Using gradient
Lipschitzness, this implies that the gradient norms stay bounded by G ∗ +2 DL .
Hence, (cid:13)(cid:13) x k 0 − x ki (cid:13)(cid:13) ≤ αi ( G ∗ + 2 DL ) .
Using this,
[Equation]
where we used the fact that D ≥ 2 κnαG
B. P ROOF OF L EMMA
Without loss of generality, let σ = (1 , 2 , 3 , . .
. , n ) .
This is only done for ease of notation.
The analysis goes through for any other permutation σ too [Equation]
First we show the lower bound.
WLOG assume y 0 > x 0 .
Because α < 1 /L , we have that ∀ i, y i >
x i by induction (see the equations below).
Then,
[Equation]
Next we show the upper bound y
[Equation]
[-y 
-]
Using strong convexity)
We use the fact that the function is twice differentiable:
[Equation]
In the above
[Equation]
we used the convention that (cid:82) b a f ( x ) dx is the same as − (cid:82) a b f ( x ) dx if a > b . Now
[Equation]
we can use the Hessian Lipschitzness to bound the term as follows
[Equation]
{+−x i − 1 +( y 0 − x 0 ) ∇ 
 f i ( t ) d t (cid:33) ≤ (1 − nαµ+} )
( y 0 − x 0 ) + α n (cid:88) i =
(cid:32)(cid:90) yx
L H Gαn d t − (cid:90) y
{+i −x i − 
+}
+( y 0 − x 0 ) ∇ 2 f i ( t ) d t (cid:33) =
(1 − nαµ )
( y 0 − x 0 ) + L H Gα 2 n 2 ( x 0 − y 0 ) − α n (cid:88) i = (cid:90) y
{+i −x i − 
 +( y 0 − x 0 ) ∇ 2 f i ( t ) d t ≤ (1 − nαµ ) ( y 0 − x 0 ) + L H Gα 2 n 2 ( x 0 − y 0 ) + α n (cid:88) i = L (( y 0 − x 0 ) − ( y i − x i ))+}
[Equation]
Using Ineq.
[Equation]
[Equation]
Thus, if we have α ≤ µ 2 n ( L 2 + L H G ) , then
[Equation]
C P ROOF OF T HEOREM
To prove this theorem, we consider three step-size ranges and do a case by case analysis for each of them.
We construct functions for each range such that the convergence of any permutation-based algorithm is “slow” for the functions on their corresponding step-size regime.
The ﬁnal lower bound is the minimum among the lower bounds obtained for the three regimes.
In this proof, we will use different notation from the rest of the paper because we work with scalars in the proof, and hence the superscript will denote the scalar power, and not the epoch number.
We will use x k,i to denote the i -th iterate of the the k -th epoch.
We will construct three functions F 1 ( x ) , F 2 ( y ) , and F 3 ( z ) , each the means of n component functions, such that
• Any permutation-based algorithm on F 1 ( x ) with α ∈ [ {+
+}
[Equation]
n
[Equation]
1) KL , 3 nL ] and initialization x 1 , 0 = 0 results in
[Equation]
F 1 will be an n -dimensional function, that is x ∈ R n .
This function will have minimizer at 0 ∈ R n .
NOTE:
This is the ‘key’ step-size range, and the proof sketch explained in the main paper corresponds to this function’s construction.
• Any permutation-based algorithm on F 2 ( y ) with α ∈ [ 3 nL , 1 L ] and initialization y 1 , {+
+} =results in (cid:107) y K,n
[Equation]
F 2 will be an n -dimensional function, that is y ∈ R n .
This function will have minimizer at 0 ∈ R n .
The construction for this is also inspired by the construction for F 1 , but this is constructed signiﬁcantly differently due to the different step-size range.
• Any permutation-based algorithm on F 3 ( z ) with α / ∈ [ 1 2( n − 1) KL , 1 L ] and initialization z 1 , 0 = 1 results in
[Equation]
F 3 will be an 1 -dimensional function, that is z ∈ R .
This function will have minimizer at 0 .
Then, the 2 n + 1 -dimensional function F ([ x (cid:62) , y (cid:62) , z ] (cid:62) )
=
F 1 ( x ) + F 2 ( y ) + F 3 ( z ) will show bad convergence in any step-size regime.
This function will have minimizer at 0 ∈ R 2 n +1 .
Furthermore,
[Equation]
that is, F 1 , F 2 , F 3 , and F are all n − 1 n L -strongly convex and 2 L -smooth.
In the subsequent subsections, we prove the lower bounds for F 1 , F 2 , and F 3 separately. NOTE:
In Appendix C.4, we have discussed how the lower bound is actually invariant to initialization.
C. L
[Table] BOUND FOR [Table] , α ∈ [
[Table] [Equation] [Table] We will work in n -dimensions ( n is even) and represent a vector in the space as z = [ x 1 , y 1 , . . .
, x n/ 2 , y n/ 2 ] .
These x i and y i are not related to the vectors used by F 2 and F 3 later, we only use x i and y i to make the proof for this part easier to understand.
We start off by deﬁning the n component functions: For i ∈ [ n/ 2] , deﬁne
[Equation]
, and
[Equation]
Thus, F ( z ) :
= 1 n (cid:16)(cid:80) n/ 2 i =
[Equation]
f i + [Equation] [Equation] g
[Equation]
=
[Equation]
This function has minimizer at z ∗ = 0 .
Let z k,j denote z at the j -th iteration in k -th epoch.
We initialize at z 1 , 0 = 0 .
For the majority of this proof, we will work inside a given epoch, so we will skip the subscript denoting the epoch.
We denote the j -th iterate within the epoch as z j and the coordinates as z j = [ x j, 1 , y j, 1 , . . .
, x j,n/ 2 , y j,n/ 2 ]
In the current epoch, let σ be the permutation of { f 1 , . . .
, f n/ 2 , g 1 , . .
. , g n/ 2 } used.
For any i ∈ [ n/ 2] , let p and q be indices such that σ p = f
i and σ q = g i .
Let us consider the case
that p <
q (the case when p > q will be analogous).
Then, it can be seen that
[Equation]
Hence,
[Equation]
In the other case, when p > q , we will get the same inequality.
Let x K,n,i and y K,n,i denote the value of x n,i and y n,i at the K -th epoch.
Then, recalling that z was initialized to 0 , we use the inequality above to get
[Equation]
Since this inequality is valid for all i , we get that
[Equation]
Note that if α ≤ 3 nL and n ≥ 4 , then (1 − αL ) n − 1 ≥ 18 .
Using this in (16), we get
[Equation]
n (cid:107) 2 ≥ n
[Equation]
[Equation]
We consider two cases:
Case A , αL ≤ 1 2( n +2) :
It can be veriﬁed that
[Equation]
−
[Equation]
[Equation]
αL
[Equation]
1 is an increasing function of α when αL ≤ 1 2( n +2) .
Noting that we are working in the range when α ≥ 1 2( n [Equation] 1) KL , then
[Equation]
e − 1 / 2 (cid:17)= Ω (cid:18) n 3 K 2 L(cid:19) .
2. Case B
αL >
1 2( n +2) :
In this case, (cid:107) z K,n (cid:107) 2 ≥ n 128 L  (cid:16) 1 2( n +2) (cid:17)   (cid:16) 1 − (1 − αL ) ( n − 1) K (cid:17)≥ n 128 L  (cid:16) 1 2( n +2) (cid:17)   (cid:16) 1 − e − αL ( n − 1) K (cid:17)≥ n 128 L  (cid:16) 1 2( n +2) (cid:17)   (cid:16) 1 − e − 3( n − 1) K 2( n +2)
(cid:17)= Ω (cid:18)n 3 L(cid:19) .
C. L OWER BOUND FOR F 2 , α ∈ [ 3 nL , 1 L ]
We will work in n -dimensions and represent a vector in the space as y =
[ y 1 , . . .
, y n ] .
We start off by deﬁning the n component functions: For i ∈ [ n ] , deﬁne f i ( y ) :
= − y i + (cid:88) j (cid:54) = i (cid:32) Ly 2 j 2 + y j n −(cid:33)
Thus, F ( y ) : = 1 n (cid:80) ni =1 f i = (cid:0) n − 1 n (cid:1)
L 2 (cid:107) y (cid:107) 2 .
This function has minimizer at y ∗ = 0 .
Let y k,j denote y at the j -th iteration in k -th epoch.
We initialize at y 1 , 0 = 0 .
For the majority of this proof, we will work inside a given epoch.
We denote the j -th iterate within the epoch as y j and the coordinates as y j =
[ y j, 1 , . . .
, y j,n ]
In the current epoch, let σ be the permutation of { f 1 , . . .
, f n } used.
Let i be the index such that σ n =
i , that is, i is the last element of the permutation σ .
Then at the end of the epoch,
[Equation]
For some j ∈ [ n ] , let s be the integer
such that σ s =
j , that is j is the s -th element in the permutation σ .
Then for any j and any epoch,
[Equation]
Then,
[Equation]
Note that the above is independent of σ , and hence applicable for all epochs.
Applying it recursively and noting that we initialized y 1 , 0 = 0 , we get
[Equation]
Note that y 0 ,i is just y n,i from the previous epoch.
Hence we can substitute the inequality above for y 0 ,i in (17).
Thus,
[Equation]
This gives us that (cid:107) y k,n [Equation] 2 = Ω (cid:0) 1 n 2 L 2 (cid:1) for any k .
C. L OWER BOUND FOR F 3 , α / ∈ [ 1 2( n − 1) KL , 1 L ]
Consider the function F ( z ) = 1 n (cid:80) ni
=
f i ( z ) , where f i ( z ) =
Lz 2 for all i .
Note that the gradient of any function at z is just − 2 Lz .
Hence, regardless of the permutation, if we start the shufﬂing based gradient descent method at z 1 , 0 = 1 , we get that
[Equation]
In the case when α ≤ 1 2( n − 1) KL , we see that
[Equation]
≥
[Equation]
= Ω(1) , for n, K ≥ 2 .
Finally, in the case when α ≥ 1 L , we see that
[Equation]
C. D ISCUSSION ABOUT INITIALIZATION
The lower bound partitions the step size in three ranges -
• In the step size ranges α ∈
[ 1 2( n − 1) KL , 3 nL ] and α ∈ [ 3 nL , 1 L ] , the initializations are
done
at the minimizer and it
is shown
that any permutation-based algorithm will still move away from the minimizer.
The choice of initializing at the minimizer was solely for the convenience of analysis and
calculations and the proof should work for any other initialization as well.
Furthermore, the effect of initializing at any arbitrary (non-zero) point will decay exponentially fast with epochs anyway.
To see how, note that every epoch can be treated as n steps of full gradient descent and some noise, and hence the full gradient descent part will essentially keep decreasing the effect of initialization exponentially, and what we would be left with is the noise in each epoch.
Thus, it was more convenient for us to just assume initialization at the minimizer and only focus on the noise in each epoch.
• The step size range α / ∈ [ 1 2( n − 1) KL , 1 L ] can be divided into two parts, α ∈ [0 , 1 2( n − 1) KL ] and α ∈ [ 1 L , ∞ ) .
{+For the range α ∈ [0 , 
+}
( n − 1) KL ] , we essentially show that the step size is too small to make any meaningful progress towards the minimizer.
Hence, instead of initializing at 1 , initializing at any other arbitrary (non-zero) point will also give the same slow convergence rate.
For the range α ∈ [ 1 L , ∞ ) , we show that the optimization algorithm will in fact diverge since the step size is too large.
Hence, even here, any other arbitrary (non-zero) point of initialize will also give divergence.
D P ROOF OF T HEOREM
Deﬁne f 1 ( x ) :
=
L 2 x 2 − x and f 2 ( x, y ) :
= − L 4 x 2 + x .
Thus, F ( x, y ) =
L 8 x 2 .
This function has minimizer at x ∗ = 0 .
For this proof, we will use the convention that x i,j is the iterate after the i -th iteration of the j -th epoch.
Further, the number in the superscript will be the scalar power.
For example x 2 i,j = x i,j · x i,j .
Initialize at x 0 , 0 = 1 L .
Then at epoch k , there are two possible permutations: σ = (1 , 2) and σ = (2 , 1) .
In the ﬁrst case , σ = (1 , 2) , we get that after the ﬁrst iteration of the epoch, x 1 ,k
= x 0 ,k − α ∇ f 1 ( x 0 ,k , y 0 ,k ) =
(1 −
[Equation]
) x 0 ,k +
[Equation]
Continuing on, in the second iteration, we get x 2 ,k
= x 1 ,k − α ∇ f 2 ( x 1 ,k , y 1 ,k )
[Equation]
Note that x 0 ,k +
= x 2 ,k .
Thus, x 0 ,k +1 = (cid:0) 1 + 12 αL (cid:1) (1 − αL ) x 0 ,k + 12 α 2 L .
Similarly, for the other possible permutation, σ = (2 , 1) , we get [-(cid:0) 
-]{+x 0 ,k +
+}
= (cid:0) 1 + 12 αL (cid:1) (1 − αL ) x 0 ,k + α 2 L .
Thus, regardless of what permutation we use, we get that
[Equation]
Hence, recalling that we initialized at x 0 , 0 = 1 L , we get
[Equation]
Now, if α ≥ 1 KL , then
[Equation]
and hence, x 2 n,K = Ω( 1 K 2 L 2 ) .
Otherwise, if α ≤ 1 KL , then continuing on from (18),
[Equation]
= Ω(1 /L ) , and hence in this case, x 2 n,K = Ω( 1 L 2 ) .
E P ROOF OF T HEOREM
Let F ( x ) :
= 1 n (cid:80) ni
=
f i ( x ) be such that its minimizer it at the origin.
This can be assumed without loss of generality because we can shift the coordinates appropriately, similar to (Safran & Shamir, 2019).
Since the f i are convex quadratics, we can write them as f i ( x ) = 12 x (cid:62) [Equation]
i x + c i , where A i are symmetric, positive-semideﬁnite matrices.
We can omit the constants
c i because they do not affect the minimizer or the gradients.
Because we assume that the minimizer of F ( x ) is at the origin, we get that
[Equation]
Let σ = ( σ 1 , σ 2 , . . .
, σ n ) be the random permutation of (1 , 2 , . .
. , n ) generated at the beginning of the algorithm.
Then for k ∈ (1 , 2 , . . . , K/ 2) , epoch 2 k − 1 sees the n functions in the following sequence:
[Equation]
whereas epoch 2 k sees the n functions in the reverse sequence:
[Equation]
We deﬁne S i := α A σ i and t i = α b σ i for convenience of notation.
We start off by computing the progress made during an even indexed epoch.
Since the even epochs use the reverse permutation, we get
[Equation]
f
[Equation]
is used at the last iteration of even epochs.)
[Equation]
We recursively apply the same procedure as above to the whole epoch to get the following
[Equation] [Equation]
where the product of matrices { M i } is deﬁned as (cid:81) mi = l M i
=
M l M l +1 . . .
M m if m ≥ l andotherwise.
Similar to Eq.
(20), we can compute the progress made during an odd indexed epoch.
Recall that the only difference will be that the odd indexed epochs see the permutations in the order ( σ 1 , σ 2 , . . .
, σ n ) instead of ( σ n , σ n − 1 , . . . ,
σ 1 ) .
After doing the computation, we get the following equation:
[Equation]
Combining the results above, we can get the total progress made after the pair of epoch 2 k − 1 and 2 k :
[Equation]
In the sum above, the ﬁrst term will have an exponential decay, hence we need to control the next two terms.
We denote the sum of the terms as z (see the deﬁnition below) and we will control its norm later in this proof.
[Equation]
To see where the iterates end up after K epochs, we simply set 2 k = K in Eq.
and then keep applying the equation recursively to preceding epochs.
Then, we get
[Equation]
( I − S n − i +1 ) (cid:33) z
+ z ...
= (cid:32)(cid:32) n (cid:89) i =
( I − S i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S n − i +1 ) (cid:33)(cid:33) K/x 10 +
K 2 − 1 (cid:88) k
= (cid:32)(cid:32) n (cid:89) i
=
( I − S i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S n − i +1 ) (cid:33)(cid:33)
k z .
Taking squared norms and expectations on both sides, we get
E [ (cid:107)
[Equation]
Kn (cid:107) [Equation] ] =
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:32)(cid:32) n (cid:89) i
= ( I − S i ) (cid:33) (cid:32) n (cid:89)
i =
( I − S n − i +1 )
(cid:33)(cid:33) K/x+
K 2 − 1 (cid:88) k
= (cid:32)(cid:32) n (cid:89) i
= ( I − S i ) (cid:33) (cid:32) n (cid:89) i = ( I − S n − i +1 ) (cid:33)(cid:33) k z (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 ≤ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:32)(cid:32) n (cid:89) i
= ( I − S i ) (cid:33) (cid:32) n (cid:89) i = ( I − S n − i +1 ) (cid:33)(cid:33) K/x 10 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K 2 − 1 (cid:88) k
= (cid:32)(cid:32) n (cid:89) i
=
( I − S i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S n − i +1 ) (cid:33)(cid:33)
k z (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 (Since ( a + b ) 2 ≤ 2 a
+
b
2 ) ≤ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:32)(cid:32) n (cid:89)
i =
( I − S i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S n − i +1 ) (cid:33)(cid:33) K/x 10 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 + 2 E     (cid:107) z (cid:107)
K 2 − 1 (cid:88) k
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:89) i
=
( I − S i ) n (cid:89) i
= ( I − S n − i +1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k 
 2   .
We assumed that the functions f i have L -Lipschitz gradients (Assumption 2).
This translates to
A i having maximum eigenvalue less than L .
Hence, if α ≤ 1 /L , we get that I − α A
i is positive semi-deﬁnite with maximum eigenvalue bounded by 1.
Hence, (cid:107) I − S
i (cid:107) ≤ 1 .
Using this and the fact that for matrices M 1 and M 2 , (cid:107) M 1 M 2 (cid:107) ≤ (cid:107) M 1 (cid:107)(cid:107) M 2 (cid:107) , we get that
[Equation]

[Table]
We handle the two terms above separately.
For the ﬁrst term, we have the following bound:
Lemma 3.
If α ≤ 18 κL min (cid:110) 2 , √ κn (cid:111) , then
[Equation]
Note that K ≥ 80 κ 3 / 2 log( nK ) max (cid:110) 1 , √ κn [Equation] = ⇒
[Equation]
18 κL min (cid:110) 2 , √ κn (cid:111) .
We also have the following lemma that bounds the expected squared norm of z .
Lemma 4.
If α ≤ 1 L , then
[Equation]
where G ∗ = max i (cid:107) b i (cid:107) , and the expectation is taken over the randomness of z .
Using these lemmas, we get that
[Equation]
Substituting α = 10log nK µnK gives us the result.
E. P ROOF OF L EMMA
We deﬁne ( (cid:101) S 1 , . . .
, (cid:101) S n ) :
= ( S 1 , . . .
, S n ) and ( (cid:101) S n +1 , . . .
, (cid:101) S 2 n ) :
= ( S n , . . .
, S 1 ) .
Then, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:89) i =
( I − S i ) n (cid:89) i
= ( I − S n − i +1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 n (cid:89) i
=
( I − (cid:101) S i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
I − 2 n (cid:88) i
= (cid:101) S i + (cid:88)
i<j (cid:101) S i (cid:101) S j − . . . (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
I − 2 n (cid:88) i
= (cid:101) S i + (cid:88)
i<j (cid:101) S
i (cid:101) S j (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88) i< j<k (cid:101) S i [Table]
Note that [Table]
j
[Table]
[Table] S [Table] S j + n
[Table] S 2 i =(cid:32) n (cid:88)
[Table]
S i (cid:33) (cid:32) n (cid:88)
[Table]
S i (cid:33) − n (cid:88) i =
[Table]
S 2 i .
Substituting this and noting that (cid:80) 2 ni =1 (cid:101) S i = 2 (cid:80) ni =1 S i , we get [Table] n [Table]
I [Table] + [Table]
Let us denote T : = (cid:80) ni
=1 S i . Then, we know by Assumptions 2 and 3 that T has eigenvalues in [ nαµ, nαL ] . Then, as long as α ≤ 14 nL , we get that [Table]
= (cid:13)(cid:13)
I − 2 T + 2 T 2 (cid:13)(cid:13) ≤ 1 − 3 2 nαµ.
Substituting this, we get (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:89)
i =
( I − S i ) n (cid:89) i = ( I − S n − i +1 ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:18) 1 − 3 2 nαµ (cid:19)
+ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i =
S 2 i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + (cid:88) i<j<k
(cid:13)(cid:13)(cid:13) (cid:101) S i (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:101) S j (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:101)
S k
(cid:13)(cid:13)(cid:13) + . . . .
By Assumption 2, we know that (cid:107) A i (cid:107) ≤ L .
Hence, (cid:107) (cid:101)
S
i (cid:107) ≤ αL . Hence, (cid:13)(cid:13)(cid:13)(cid:13) (cid:13) n (cid:89) i
=
( I − S i ) n (cid:89) i
= ( I − S n − i +1 ) (cid:13)(cid:13)(cid:13)(cid:13)
(cid:13) ≤ (cid:18)
[Equation]
(Since α ≤ 1 / 4 nL .)
Finally, as long as α ≤ 14 κL and α ≤
[Equation]
E. P ROOF OF L EMMA
We start off by computing the ﬁrst order expansion of z .
We have the following lemma for this:
Lemma 5.
[Equation]
=
[Equation]
(cid:33) , where ( (cid:101) S 1 , . . .
, (cid:101) S n ) :
= ( S 1 , . . .
, S n ) and ( (cid:101) S n +1 , . . .
, (cid:101) S 2 n ) :
= ( S n , . . .
, S 1 ) .
The proof of this lemma is quite algebraic and hence has been pushed to the end, in Appendix E.4.
The strategy is to bound (cid:107) S i t
i (cid:107) , (cid:107) (cid:80) 2 n − j i =1 t i (cid:107) , and (cid:107) (cid:80) n − j i =1 t n +1 − i (cid:107) .
Hence, we apply Lemma 5 and use triangle inequality: [Equation]
1 (cid:88)
j
[Equation]
n
[Equation]
j − 1 (cid:88) k
[Equation]
(cid:32)
[Equation]
(cid:89) p
[Equation]
( I [Equation] (cid:101) S p ) (cid:33) (cid:101) S [Equation]
(cid:101)
[Equation]
j (cid:32)
[Equation]
n
[Equation]
j (cid:88) i = t i (cid:33) + n [Equation] (cid:88)
[Equation]
[Equation]
(cid:88)
[Equation]
(cid:89) p
[Equation]
( I
[Equation]
p ) (cid:33)
[Equation]
(cid:88)
[Equation]
(cid:33)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) [Table] [Equation] 
[Table]
E
[Equation]
 n (cid:88) i = (cid:107) [Equation] (cid:107)(cid:107)
[Equation]
i (cid:107) + 2 n
[Equation] 1 (cid:88) j = n + j
[Equation] 1 (cid:88) l = (cid:32) l − 1 (cid:89) p
= (cid:107) I − (cid:101) S p (cid:107) (cid:33) (cid:107) (cid:101) S l (cid:107)(cid:107) (cid:101) S j (cid:107) (cid:32) 2 n − j (cid:88) i
= (cid:107) t i (cid:107) (cid:33) + n − 1 (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= (cid:107) I − S p (cid:107) (cid:33) (cid:107) S l (cid:107)(cid:107) [Equation]
[-= (cid:107) t n +
 − i (cid:107) (cid:33)  2 -]  .
Now, we recall that (cid:107) S
i (cid:107) ≤ αL and (cid:107) t i (cid:107) ≤ αG .
Because α ≤ 1 /L , we also get that (cid:107) I − S
i (cid:107) ≤ 1 .
Using these,
[Equation] t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ nα 2 Ln − 1 (cid:88)
j = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n − j (cid:88) i
=
t n +1 − i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 = n 2 α 4 L 2 E     G + 2 L 2 n − 1 (cid:88)
j = n + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 n − j (cid:88) i = t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ L n − 1 (cid:88)
j = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n − j (cid:88) i
=
t n +1 − i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 ≤ 2 n 2 α 4 L  G 2 + L 2 E    2 n − 1 (cid:88)
j = n + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 n − j (cid:88) i = t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + n − 1 (cid:88) j = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n − j (cid:88) i
=
t n +1 − i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2     .
(Since ( a + b ) 2 ≤ 2 a
+
b
Using Hoeffding-Serﬂing inequality for bounded random vectors (Schneider, 2016, Theorem 2), we get the following lemma
Lemma 6.
For all j, l ∈ [1 , n ] we have that
[Equation]
[Equation]
where G ∗ = max i (cid:107) b i (cid:107) , and the expectation is taken over the randomness of t i .
Writing out the expansion of (cid:16) 2 (cid:80) 2 n − 1 j = n +1 (cid:13)(cid:13)(cid:13)(cid:80) 2 n − j i =
1 t i (cid:13)(cid:13)(cid:13) + (cid:80) n − 1 j =1 (cid:13)(cid:13)(cid:13)(cid:80) n − j i =1 t n +1 − i (cid:13)(cid:13)(cid:13)(cid:17) 2 and using the lemma above on the individual terms, we get [Equation] E
[Equation]
P ROOF OF L EMMA
This proof is similar to the one in (Ahn et al., 2020).
Deﬁne G ∗ := max i (cid:107) b i (cid:107) .
We use the following theorem adapted to our setting
Theorem 7.
[(Schneider, 2016, Theorem 2)]
With probability at least 1 − δn , (cid:13) (cid:13) (cid:13) (cid:13)(cid:13) j (cid:88) i =
t i (cid:13) (cid:13) (cid:13) (cid:13)(cid:13) ≤ αG ∗ (cid:115) 8 j (cid:18) 1 −
[Equation]
−n (cid:19) log 2 n δ .
Then taking a union bound over j = 1 , . .
. , n , we get that with probability at least 1 − δ , ∀ j ∈ [1 , n ] : (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) j (cid:88) i =
t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ αG ∗ (cid:115)
[Equation]
(cid:18)
[Equation]
(cid:114)
[Equation]
Then, for the complementary event (which happens with probability δ ), we use the fact that (cid:107) t i (cid:107) =
(cid:107) α b σ i (cid:107) ≤ αG ∗ to get the following: ∀ j ∈ [1 , n ] : (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) j (cid:88) i
=
t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ j (cid:88) i
= (cid:107) t i (cid:107) ≤ αG ∗ j.
Now, choose δ = 1 /n .
Then, we get that
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) j (cid:88)
i =
t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ (cid:18)
[Equation]
[Equation]
n.
Similarly, we can also get
E (cid:34)(cid:13) (cid:13) (cid:13)(cid:13)(cid:13) j (cid:88)
i =
t
[Equation]
(cid:13) (cid:13) (cid:13)(cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) l (cid:88)
[Equation]
E. P ROOF OF L EMMA
As in the lemma’s statement, deﬁne ( (cid:101) S 1 , . . .
, (cid:101) S 2 n ) as ( (cid:101) S 1 , . . .
, (cid:101) S n ) =
( S 1 , . . .
, S n ) and ( (cid:101) S n +1 , . . .
, (cid:101) S 2 n ) =
( S n , . . .
, S 1 ) .
As a reminder, the term z is deﬁned as follows:
[Equation]
First, we analyze the ﬁrst term in z .
Towards that end, we start by expanding (cid:16)(cid:81)
[Equation]
[Equation]
Similarly
[Equation]
we expand the term (cid:16)(cid:81) 2 n − i − 1 j =
[Equation]
−
[Equation]
S j ) (cid:17) and then recursively keep doing it to get the following:
[Equation]
−
[Equation]

[Equation]
Note that the term (cid:81) j − 1 l =1 ( I − (cid:101) S l ) above is similar to the RHS of Eq.
Hence, we repeat the process again on this term to get the following 
[Equation]
Using this in the ﬁrst term (cid:80) ni =1 (cid:16)(cid:81) nj =
( I − S j ) (cid:17) (cid:16)(cid:81) n − i j =
( I − S n +1 − j ) (cid:17) t
i (in Eq.
, we get n (cid:88) i =   n (cid:89) j
= ( I − S j ) 
   n − i (cid:89)
j = ( I − S n +1 − j )   t i = n (cid:88) i =   I − 2 n − i (cid:88) j = (cid:101) S j + 2 n − i (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101) S j 
 t i = n (cid:88) i
=
t i − n (cid:88)
i = 2 n − i (cid:88)
j = (cid:101) S
j t i + n (cid:88) i = 2 n − i (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S
j t i .
Now, we use the fact that (cid:80) ni =1 b i =
(Eq.
(19)) to get that
(cid:80) ni
=1 t i = 0 .
Then, n (cid:88) i =   n (cid:89)
j
=
( I − S j ) 
   n − i (cid:89)
j = ( I − S n +1 − j )

 t i = − n (cid:88) i = 2 n − i (cid:88)
j = (cid:101) S
j t i + n (cid:88) i = 2 n − i (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S
j t
i
For convenience, we deﬁne (cid:102) M j :=
(cid:80) j − 1 l =1 (cid:16)(cid:81) l − 1 p =
( I − (cid:101) S p ) (cid:17) (cid:101) S l (cid:101) S j .
Then, n (cid:88)
i =   n (cid:89)
j
=
( I − S j ) 
   n − i (cid:89)
j = ( I − S n +1 − j )

 t i = − n (cid:88) i = 2 n − i (cid:88)
j = (cid:101) S
j t i + n (cid:88) i = 2 n − i (cid:88)
j = (cid:102)
M j t i = − n (cid:88) i = 2 n − i (cid:88)
j = (cid:101) S
j t i + n (cid:88) i = n (cid:88)
j = (cid:102)
M j t i + n (cid:88) i = 2 n − i (cid:88)
j = n + (cid:102)
M j t i = − n (cid:88) i = 2 n − i (cid:88)
j = (cid:101) S
j t i + n (cid:88)
j = (cid:102)
M j n (cid:88)
i =
t i + n (cid:88) i = 2 n − i (cid:88)
j = n + (cid:102)
M j t i = − n (cid:88) i = 2 n − i (cid:88)
j = (cid:101) S
j t i + n (cid:88) i = 2 n − i (cid:88)
j = n + (cid:102)
M j t i .
(Since (cid:80) ni =1 t i = 0 )
Note that [Table] ni = [Table]
n − i j [Table] 1 (cid:102) M j t i = (cid:80) 2 n − 1 j = n +1 (cid:102)
M j (cid:16)(cid:80) 2 n − j i =
1 t i (cid:17) .
Hence, [Table]   [Table]
j [Table] [Table] S
[Table] ) 
  [Table] j = [Table]
[Table] S [Table]
)   t i = − n [Table] i = 2 n − i (cid:88)
j = (cid:101) S
j t i + 2 n − 1 (cid:88)
j = n + (cid:102)
M j (cid:32) 2 n − j (cid:88) i
= t i (cid:33) = − n (cid:88) i = n (cid:88)
j = (cid:101) S
j t i − n (cid:88) i = 2 n − i (cid:88)
j = n + (cid:101) S
j t i + 2 n − 1 (cid:88)
j = n + (cid:102)
M j (cid:32) 2 n − j (cid:88) i
= t i (cid:33) = − n (cid:88)
j = (cid:101) S j n (cid:88)
i =
t i − n (cid:88)
i = 2 n − i (cid:88)
j = n + (cid:101) S
j t i + 2 n − 1 (cid:88)
j = n + (cid:102)
M j (cid:32) 2 n − j (cid:88) i
= t i (cid:33) = − n (cid:88) i = 2 n − i (cid:88)
j = n + (cid:101) S
j t i + 2 n − 1 (cid:88)
j = n + (cid:102)
M j (cid:32) 2 n − j (cid:88) i
= t i (cid:33) (Since (cid:80) ni =1 t i = 0 ) = − n (cid:88) i = 2 n − i (cid:88) j = n + (cid:101) S
j t i + 2 n − 1 (cid:88)
j = n + j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p = ( I [Equation] [Equation]
Next we analyze the second term in z .
For this, we start by expanding (cid:81) n − i j =
( I − S j ) in a similar way as Eq.
[Equation]
Using this, we get n (cid:88) i =   n − i (cid:89)
j = ( I − S j ) 
 t n +1 − i = n (cid:88) i =   I − n − i (cid:88)
j =
S j + n − i (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − S p ) (cid:33)
S l S j 
 t n +1 − i = n (cid:88) i =
t n +1 −
i − n (cid:88) i
= n − i (cid:88)
j =
S j t n +1 − i + n (cid:88)
i
= n − i (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − S p ) (cid:33)
S l S j t n +1 − i = − n (cid:88) i
= n − i (cid:88)
j =
S j t n +1 − i + n (cid:88)
i
= n − i (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − S p ) (cid:33)
S l S j t n +1 −
i , where we used the fact that (cid:80) n i =1 t i = {+
+} in the last equality.
For convenience, we deﬁne M j := (cid:80) j − 1 l =1 (cid:16)(cid:81) l − 1 p =1 ( I − S p ) (cid:17) S l S j .
Then, n (cid:88)
i =   n − i (cid:89) j
= ( I − S j ) 
 t n +1 − i = − n (cid:88) i = n − i (cid:88)
j =
S j t n +1 − i + n (cid:88)
i
= n − i (cid:88)
j =
M j t n +1 − i .
Since (cid:80) ni =1 (cid:80) n − i j =1 M j t n +1 − i = (cid:80) n − 1 j =1 M j (cid:16)(cid:80) n − j i =1 t n +1 − i (cid:17) , we get n (cid:88)
[Equation]
[Equation]
Finally, substituting Eq.
(25) and (27) in the deﬁnition of z (Eq.
, we get z = − n (cid:88)
[Equation]
[Equation]
−
[Equation]
+
[Equation]
t
[Equation]
+ n
[Equation]
j = j −
[Equation]
(cid:88) l = (cid:32) l [Equation] 1 (cid:89) p = ( I − S p ) (cid:33)
S l S j (cid:32) n − j (cid:88) i =
t n +1 − i (cid:33) = − n (cid:88) i =   n (cid:88) j =
S j 
 t i + n (cid:88) i = S i t i + 2 n − 1 (cid:88) j = n + j − 1 (cid:88) l = (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101) S j (cid:32) 2 n − j (cid:88) i = t i (cid:33) + n − 1 (cid:88)
j = j − 1 (cid:88) l = (cid:32) l − 1 (cid:89) p = ( I − S p ) (cid:33) S l S j (cid:32) n − j (cid:88) i =
t n +1 − i (cid:33) = −   n (cid:88) j = S j   n (cid:88) i = t i + n (cid:88) i =
S i t i + 2 n − 1 (cid:88) j = n + j − 1 (cid:88) l = (cid:32) l − 1 (cid:89) p = ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j (cid:32) 2 n − j (cid:88) i = t i (cid:33) + n − 1 (cid:88) j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − S p ) (cid:33) S l S j (cid:32) n − j (cid:88) i = t n +1 − i (cid:33) = n (cid:88) i = S i t i + 2 n − 1 (cid:88)
j = n + j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101) S j (cid:32) 2 n − j (cid:88) i = t i (cid:33) + n − 1 (cid:88)
j = j − 1 (cid:88) l = (cid:32) l − 1 (cid:89) p = ( I − S p ) (cid:33) S l S j (cid:32) n − j (cid:88) i =
t n +1 − i (cid:33) , where we used the fact that (cid:80) ni =1 t i = 0 in the last equality.
F P ROOF OF T HEOREM
The proof is similar to that of Theorem 4, except for that here we leverage the independence of random permutations in every other epoch.
The setup is also the same as Theorem 4, but we explain it again here nevertheless, for the completeness of this proof.
Let F ( x ) :
= 1 n (cid:80) ni
=
f i ( x ) be such that its minimizer it at the origin.
This can be assumed without loss of generality because we can shift the coordinates appropriately, similar to (Safran & Shamir, 2019).
Since the f i are convex quadratics, we can write them as f i ( x ) = 12 x (cid:62) A
i x − b (cid:62) i x + c i , where A i are symmetric, positive-semideﬁnite matrices.
We can omit the constants
c i because they do not affect the minimizer or the gradients.
Because we assume that the minimizer of F ( x ) is at the origin, we get that
[Equation]
Let σ k =
( σ k 1 , σ k 2 , . . .
, σ kn ) be the random permutation of (1 , 2 , . . . , n ) sampled in epoch 2 k − 1 .
Then epoch 2 k − 1 sees the n functions in the reverse sequence:
[Equation]
whereas epoch 2 k sees the n functions in the reverse sequence:
[Equation]
We deﬁne S ki := α A σ ki and t ki = α b σ ki for convenience of notation.
We start off by computing the progress made duting an even indexed epoch.
Since the even epochs use the reverse permutation of σ k , we get
[Equation]
used at the last iteration of epoch 2 k
[Equation]
We recursively apply the same procedure as above to the whole epoch to get the following
[Equation]
where the product of matrices { M i } is deﬁned as (cid:81) mi = l M i
=
M l M l +1 . . .
M m if m ≥ l andotherwise.
Similar to Eq.
(29), we can compute the progress made during an odd indexed epoch.
Recall that the only difference will be that the odd indexed epochs see the permutations in the order ( σ k 1 , σ k 2 , . . .
σ kn ) instead of ( σ kn , σ kn − 1 , . . .
, σ k 1 ) .
After doing the computation, we get the following equation:
[Equation]
Combining the results above, we can get the total progress made after the pair of epoch 2 k − 1 and 2 k : [Equation]
[Equation]
In the sum above, the ﬁrst term will have an exponential decay, hence we need to control the next two terms.
Similar to Theorem 4, we denote the sum of the terms as z k :
[Equation]
To see where the iterates end up after K epochs, we simply set 2 k = K in Eq.
and then keep applying the equation recursively to preceding epochs.
Then, we get
[-K − 
-]
[Equation]
i =
[Equation]
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 )
(cid:33)(cid:33)
z K 2 − k .
Taking squared norms and expectations on both sides, we get
E [ (cid:107) x
Kn (cid:107) 2 ] =
E  
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) K
[Equation]
(cid:89) k
[Equation]
(cid:32) n (cid:89) i
[Equation]
( I −
[Equation]
− k i ) (cid:33) (cid:32)
[Equation]
(cid:89)
[Equation]
[Equation]
) (cid:33)
[Equation]
+
[Equation]
(cid:32)
[Equation]
(cid:32)
[Equation]
(cid:89) [Equation] [Equation]
[Equation]
(cid:32) [Equation] (cid:89) i = ( [Equation]
− [Equation]
) (cid:33)(cid:33)
[Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 ≤ 2 E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)  
K 2 − 1 (cid:89) k
= (cid:32) n (cid:89) i
=
( I − S K 2 − k i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − k n − i +1 ) (cid:33)  x 10 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where we used the fact that ( a + b ) 2 ≤ 2 a
+ 2 b 2 .
Next, we expand the second term above to get
[Table]
− 1 (cid:89) l = (cid:32) n (cid:89)
i =
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (cid:33) 
z K 2 − k (cid:48)
(cid:43)  .
We handle each of the three terms separately.
Let
N k := (cid:16)(cid:81) n i =
( I − S K 2 − k i ) (cid:17) (cid:16)(cid:81) {+n i =
+}
[-=
-]
( I − S K 2 − k n − i +1 ) (cid:17) .
The the ﬁrst term can be written as:
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K [Equation] (cid:89) k
= (cid:32) n (cid:89) i
=
( I − S K 2 − k i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − k n − i +1 ) (cid:33)  x 10 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 =
E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
K 2 − 1 (cid:89) k
=
N k 
 x 10 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 = E 
[Equation] [Equation] (cid:62)  
K 2 − [Equation] (cid:89) k
=
N k 
 (cid:62) 
[Equation]
where, in the last line, we used the fact that the permutations in every epoch are independent of the permutations in other epochs.
Next, we have the following lemma that bounds the spectral norm of E (cid:2) N (cid:62) k N k (cid:3) for any k ∈ [0 , K 2 − 1] :
Lemma 7.
For any 0 ≤ α ≤
[Equation]
Note that K ≥ 55 κ log( nK ) max (cid:8) 1 , (cid:112) nκ (cid:9) =
⇒ α ≤ 3 16 nL min { 1 , nκ } , and thus this lemma.
Using it, we get
E 
[Equation]
n (cid:89) i
= ( I − S K 2 − k n − i +1 )
(cid:33)  x 10 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 =
E   ( x 1 0 ) (cid:62)  
K 2 − 1 (cid:89) k
=
N k 
 (cid:62)
E (cid:2) N (cid:62) 0 N 0 (cid:3)  
K 2 − 1 (cid:89) k
=
N k 
 x 1  ≤ (1 − nαµ )
E   ( x 10 ) (cid:62)  
K 2 − 1 (cid:89) k
=
N k 
 (cid:62)  
K 2 − 1 (cid:89) k
=
N k   x 
[Equation]
Next we look at the second term in Ineq.
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where in the last step we used that (cid:107) I − S ki (cid:107) ≤ 1 for all i, k .
To see why this is true, recall that
S ki = α A σ ki .
Further by Assumption 2, (cid:107) A σ ki (cid:107) ≤ L and hence as long as α ≤ 1 /L , we have that (cid:107) I − S ki (cid:107) ≤ 1 .
Next, note that for any k we can apply Lemma 4 on E [ (cid:107)
z K 2 − k (cid:107) 2 ] .
Hence, we get the following bound on the second term:
[Equation]
Finally, we focus on the third term in Ineq.
We have the following lemma that gives an upper bound for it:
Lemma 8.
Let α ≤ 12 nL and n > 6 .
Then for 0 ≤ k (cid:48) <
k ≤ K 2 − 1 ,
[Equation]
(cid:89) i =
( I − S K 2 − l n − i +1 ) (cid:33)(cid:33) z K 2 − k ,   k (cid:48) − 1 (cid:89) l = (cid:32) n (cid:89) i
= ( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i = ( I − S K 2 − l n − i +1 ) (cid:33)  z K 2 − k (cid:48)
(cid:43)  ≤ 1000 n 2 α 4 L 2 ( G ∗ ) 2 + 2000 n 6 α 7 L 5 ( G ∗ ) 2 log n.
Using Lemma 8, and inequalities (33) and (34) in Ineq.
(32), we get
E [ (cid:107) x
Kn {+(cid:107) 2 ] ≤ 2(1 − nαµ ) K/ 2 (cid:107) x 10 (cid:107) 
+}
+ 2 n 2 Kα 4 L 2 ( G ∗ ) 2 + 170 n 5 Kα 6 L 4 ( G ∗ ) 2 log n + 1000 n 2 K 2 α 4 L 2 ( G ∗ ) 2 + 2000 n 6 K 2 α 7 L 5 ( G ∗ ) 2 log n ≤ 2(1 − nαµ ) K/ 2 (cid:107) x 10 (cid:107) 2 + 1002 n 2 Kα 4 L 2 ( G ∗ ) 2 + 2170 n 6 K 2 α 7 L 4 ( G ∗ ) 2 log n.
Substituting α = 10log nK µnK gives us the desired result.
F. P ROOF OF L EMMA
Recall that we have deﬁned N k :=
(cid:16)(cid:81) ni
=
( I − S K 2 − k i ) (cid:17) (cid:16)(cid:81) ni
=
( I − S K 2 − k n − i +1 ) (cid:17) .
Hence,
N (cid:62) k
= (cid:32)(cid:32) n (cid:89) i
=
( I − S K 2 − k i ) (cid:33) (cid:32) n (cid:89) i = ( I − S K 2 − k n − i +1 ) (cid:33)(cid:33) (cid:62) [Equation]
where we used the fact that S K 2 − k
i are symmetric.
Hence N k is symmetric.
Then,
[Equation]
Next, note that (cid:107) N k (cid:107) ≤ 1 as long as α ≤ 1 nL .
This combined with the fact that N k is symmetric gives us that x (cid:62) N k N k x ≤ x (cid:62) N k x .
Hence, we get
[Equation]
E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
To complete the proof, we apply the following lemma from Ahn et al.
Lemma 9 (Lemma 6, Ahn et al.
For any 0 ≤ α ≤ 3 16 nL min { 1 , nκ } and k ∈ [ K ] , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E   (cid:32) n (cid:89) i = [Equation] I − [Equation]
K 2 − k i )
[Equation]
n [Equation] k
[Equation]
≤ 1 −
[Equation]
Remark 2.
To avoid confusion, we remark here that, the term ‘ S k ’ in the paper Ahn et al.
(2020) is the same as the term ‘ (cid:81) ni
=
( I − S ki ) ’ in our paper, and hence the original lemma statement in their paper looks different from what is written above.
F. P ROOF OF L EMMA
We begin by decomposing the product into product of independent terms, similar to proof of Lemma 8 in Ahn et al.
However, after that we diverge from their proof since we use F LIP F LOP speciﬁc analysis.
[Equation]
− S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (cid:33)  (cid:62) [Table] [Table]
Since k (cid:48) <
k , we get that (cid:16) z K 2 − k (cid:17) (cid:62) , (cid:16)(cid:81)
k
+1 l = k (cid:48) (cid:16)(cid:81) ni
=
( I − S K 2 − l i ) (cid:17) (cid:16)(cid:81) ni
=
( I − S K 2 − l n − i +1 )
(cid:17)(cid:17) (cid:62) and 
[Table]
[Table] [Table]
S
[Table]
+
[Table]
(cid:33) 
[Table]
 [Table] [Table] =
[Table]
[Table] S [Table] ) (cid:33)
[Table]
are independent.
Hence, we can write the expectation as product of expectations:
E [Table] l = (cid:32) [Table]
i = ( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I
[Equation]
S
[Equation]
2 − l n − i
[Equation]
1 ) (cid:33)(cid:33)
z
[Equation]
,  
[Equation]
(cid:48)
[Equation]
1 (cid:89) l
[Equation]
(cid:32) n (cid:89) i
[Equation]
( I
[Equation]
S K 2 − l i ) (cid:33) (cid:32) n (cid:89)
i
[Equation]
( I [Equation] S K 2 − l n − i [Equation]
1 ) (cid:33)
[Equation]
[Equation]
(cid:48) (cid:43)  =
E (cid:20)(cid:16)
[Equation]
[Equation]
(cid:17) (cid:62) (cid:21) E   (cid:32)
[Equation]
1 (cid:89) l =
[Equation]
(cid:48) (cid:32) n (cid:89) i =
( I − S K
[Equation]
l i ) (cid:33) (cid:32) n (cid:89) i =
( I − S K 2 − l n − i +1 )
(cid:33)(cid:33) (cid:62) 

E  
[Equation]

[Equation]
(cid:48) (cid:89) l = (cid:32) n (cid:89) i
= ( I
[Equation]
K
[Equation]
l i ) (cid:33) (cid:32) n (cid:89) i = ( I − S K [Equation] l n − i
[Equation]
1 ) (cid:33)  (cid:62)   k (cid:48)
[Equation]
1 (cid:89) l
= (cid:32) n (cid:89) i = ( I − S K [Equation] l i ) (cid:33) (cid:32) n
[Table] i =
( I − S K 2 − l n
[Table]
i +1 ) (cid:33)  z K 2 − k (cid:48)   .
Applying the Cauchy-Schwarz inequality on the decomposition above, we get
E (cid:34)(cid:42)(cid:32) k − 1 (cid:89) l
= (cid:32) n (cid:89) i
=
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (cid:33)(cid:33)
z K 2 − k ,   k (cid:48)
[Equation] (cid:89) l = (cid:32) n (cid:89) i = ( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i = ( I − S K 2 − l n − i +1 ) (cid:33) 
z K 2 − k (cid:48) (cid:43)  ≤ (cid:13)(cid:13)(cid:13) E (cid:104) z [Equation]
k (cid:105)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E (cid:34)
k +1 (cid:89) l
= k (cid:48) (cid:32) n (cid:89) i
=
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 )
(cid:33)(cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E     k (cid:48) (cid:89) l
= (cid:32) n (cid:89) i
=
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (cid:33) 
 (cid:62)   k (cid:48) − 1 (cid:89) l
= (cid:32) n (cid:89) i
=
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (cid:33) 
z K 2 − k (cid:48) 
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13) E (cid:104) z K 2 − k (cid:105)(cid:13)(cid:13)(cid:13) E (cid:34) k
+1 (cid:89) l
= k (cid:48) (cid:32) n (cid:89)
i =
(cid:107) I − S K 2 − l
i
cid:107) (cid:33) (cid:32) n (cid:89) i
{+= (cid:107) I − S K 2 − l n − i +
+}
(cid:107) (cid:33)(cid:35) (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
E     k (cid:48) (cid:89) l
= (cid:32) n (cid:89) i
=
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S
[Table]
[Table]
where in the last step we used that (cid:107) I − S ki (cid:107) ≤ 1 for all i, k .
To see why this is true, recall that
S ki = α A σ ki .
Further by Assumption 2, (cid:107) A σ ki (cid:107) ≤ L and hence as long as α ≤ 1 /L , we have (cid:107)
I − S ki (cid:107) ≤ 1 .
For the two terms in the product above, we have the following lemma:
Lemma 10.
If n > 6 and α ≤ 1 nL , then
[Equation]
2 n log n.
Lemma 11.
If n > 6 and α ≤ 12 nL , then (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E    
[Equation]
(cid:48) (cid:89) l = (cid:32) n (cid:89)
[Equation] = ( I [Equation] [Equation]
l i ) (cid:33) (cid:32) [Equation] (cid:89) i = ( I − S K 2 − l n − i +1 )
(cid:33)  (cid:62)  
[Equation]
(cid:89) l = [Equation] (cid:89) i = ( I −
[Equation]
K 2 − l i ) (cid:33) (cid:32) n (cid:89) i =
( I − S K 2 − l n − i +1 )
(cid:33)  z K 2 − k (cid:48) 
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation] nα [Equation] [Equation]
n
[Equation]
(cid:112) 2 n log n. Finally, using these lemma we get E (cid:34)(cid:42)(cid:32)
[Equation]
(cid:89) l = (cid:32) n (cid:89) i =
( I
[Equation]
S
[Equation]
[Equation]
l i ) (cid:33) (cid:32) n (cid:89) i = ( I − S K 2 − l n − i +1 ) (cid:33)(cid:33) z K
[Equation]
k ,   k (cid:48) − 1 (cid:89) l = (cid:32) n (cid:89) i =
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (cid:33) 
z K 2 − k (cid:48)
{+(cid:43)  ≤ (cid:16) 28 nα 2 LG ∗ + 9 α 5 L 4 n 4 G ∗ (cid:112) 2 n log n (cid:17) · (cid:16) 32 nα 2 LG ∗ + 24 α 5 L 4 n 4 G ∗ (cid:112) 2 n log n (cid:17) ≤ 896 n 2 α 4 L 2 ( G ∗ ) 
+}
+ 960 α 7 L 5 n 5 ( G ∗ ) 2 (cid:112) 2 n log n + 432 α 10 L 8 n 9 ( G ∗ ) 2 log n ≤ 1000 n 2 α 4 L 2 ( G ∗ ) 2 + 2000 α 7 L 5 n 6 ( G ∗ ) 2 log n.
F. P ROOF OF L EMMA
Since we are dealing with just a single epoch, we will skip the superscript.
Using Lemma 5, we get (cid:107) E
[ z ] (cid:107) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E (cid:34) n (cid:88) i =
S i t i (cid:35) + E   2 n − 1 (cid:88)
j = n + j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j (cid:32) 2 n − j (cid:88) i
=
t i (cid:33)  + E   n − 1 (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − S p ) (cid:33)
S l S j (cid:32)
[Table]
[Equation]
Deﬁne G ∗ :=
max i (cid:107) b i (cid:107) .
Then (cid:107) S i (cid:107)(cid:107) t i (cid:107)
=
(cid:107) α A σ ki (cid:107)(cid:107) α b σ ki (cid:107) ≤ α 2 LG ∗ and hence n (cid:88) i =
E [ (cid:107) S
i (cid:107)(cid:107) t i (cid:107) ] ≤ nα 2 LG ∗ . [Equation]
Next we bound the other two terms.
Using Eq.
(26), we get that for any l < j ,
E [Equation] p [Equation]
[Equation]
n
[Equation] =
E (cid:34)(cid:32)
I − [Equation] p [Equation]
S
[Equation]
+ l [Equation] 1 (cid:88) p = p − 1 (cid:88) q
= (cid:32) q − 1 (cid:89) r
= ( I − S q ) (cid:33)
S r S p (cid:33) S
[Equation] (cid:32) n − j (cid:88) i = t n
[Equation]
1 − i (cid:33)(cid:35) = n
[Equation]
(cid:88) i =
[Equation]
(cid:32)
[Equation]
(cid:88)
i
[Equation]
[Equation]
(cid:33)(cid:35) + E
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
(cid:32)
[Equation]
(cid:89)
[Equation]
[Equation]
(cid:33)
[Equation]
(cid:32) n − j (cid:88) i =
t n + [Equation]
i (cid:33)(cid:35) =
(cid:88)
i
[Equation]
(cid:88)
[Equation]
E (cid:34)(cid:32) q −
[Equation]
(cid:89) r
= ( I [Equation] q ) (cid:33)
[Equation] (cid:32) n −
[Equation]
(cid:88) i =
t n +1 −
i
(cid:33)(cid:35) = (cid:88) [Equation]
>
[Equation] +
[Equation]
=
[Equation]
E (cid:34)(cid:32) q − 1 (cid:89) r
= ( I − S q ) (cid:33)
S
[Equation]
S p S l S j (cid:32) n − j (cid:88) i =
t n +1 − i
(cid:33)(cid:35) .
Since (cid:80) ni =1 t i = 0 , and we use uniform random permutations, E
[ t i | S l , S j ] = (cid:80) t i (cid:54) =
t l , t i (cid:54) =
t j t i n − 2 = − t j − t l n − 2 .
Similarly, E [ t i | S l , S j , S p ] = − t j − t l − t p n −.
Hence, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E (cid:34)(cid:32) l − 1 (cid:89) p
= ( I − S p ) (cid:33)
S l S j (cid:32) n − j (cid:88) i =
t n +1 −
i
(cid:33)(cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:88) i>j,l (cid:107)
E
[ S l S j E [ t i | S l , S j ]] (cid:107) +
(cid:88) p<l,j<i (cid:107) E
[ S p S l S j E [ t i | S l , S j , S p ]] (cid:107) + l − 1 (cid:88) p
=
p − 1 (cid:88) q =
E (cid:34)(cid:32) q − 1 (cid:89) r
= (cid:107) I − S q (cid:107) (cid:33) (cid:107) S r
cid:107)(cid:107) S p (cid:107)(cid:107)
[Equation]
E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:35) (Since (cid:107) S
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
(cid:107) [Equation] αG
[Equation] [Equation]
n
[Equation]
where we used Lemma 6 and the assumption that n > 6 in the last step.
The third term in Ineq.
(35) can be handled similarly.
For any l, j :
[Equation]
S l (cid:101) S j (cid:32) 2 n − j (cid:88) i
=
t i (cid:33)(cid:35) .
Now, it is easy to see that i (cid:54) =
j and i (cid:54) =
2 n − j + 1 .
Then, if i = l or i = 2 n − l + 1 we use for that case the fact that (cid:107) E
[ (cid:101) S l (cid:101) S
j t
i ] (cid:107) ≤ α 3 L 2 G ∗ .
For all other i , we can again use that
E [ t i | (cid:101) S l , (cid:101) S j ] = − t l − t j n − 2 if l ≤ n or E [ t i | (cid:101) S l , (cid:101) S j ] = − t 2 n − l
+
− t j n −otherwise.
Similarly, we can bound (cid:13)(cid:13)(cid:13) E [Table]
S p [Table]
S l [Table]
S j t i (cid:105)(cid:13)(cid:13)(cid:13) .
Proceeding in a way similar to how we derived Ineq.
(37), we get (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E (cid:34)(cid:32) [Table] p
[Equation]
( I
[Equation]
(cid:101)
[Equation] [Table] (cid:33) (cid:101) S
[Equation]
(cid:101) [Table] j (cid:32)
[Equation] n [Equation] (cid:88) i
=
t i (cid:33)(cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 2 n − j (cid:88) i = (cid:13)(cid:13)(cid:13) E
[-j t i (cid:105)(cid:13)(cid:13)(cid:13) + l − 1 (cid:88) p = 
 n − j (cid:88) i =-][Table]
S
[Table]
S
[Table]
t
[Table]
[Equation]
≤ 5 α 3 L 2 G ∗ + 5 nα 4 L 3 G ∗ + l − 1 (cid:88) p = p − 1 (cid:88) q
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E (cid:34)(cid:32)
q − 1 (cid:89) r = (cid:107) I − (cid:101) S q (cid:107) (cid:33) (cid:107) (cid:101) S r (cid:107)(cid:107) (cid:101) S p (cid:107)(cid:107) (cid:101) S l (cid:107)(cid:107) (cid:101) S j (cid:107) (cid:32) 2 n − j (cid:88)
i =
t i (cid:33)(cid:35)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 5 α 3 L 2 G ∗ + 5 nα 4 L 3 G ∗ + α 5 L 4 n 2 G ∗ (cid:112) 18 n log n.
Substituting Ineq.
(36), (37) and (38) into (35), we get (cid:107) E
[ z ] (cid:107) ≤ nα 2 LG ∗ + 10 n 2 α 3 L 2 G ∗ + 10 n 3 α 4 L 3 G ∗ + 2 α 5 L 4 n 4 G ∗ (cid:112) 18 n log n + 4 n 2 α 3 L 2 G ∗ + 3 n 3 α 4 L 3 G ∗ + α 5 L 4 n 4 G ∗ (cid:112) 18 n log n ≤ 28 nα 2 LG ∗ + 9 α 5 L 4 n 4 G ∗ (cid:112) 2 n log n, where we used the assumption that α ≤ 1 nL in the last step.
F. P ROOF OF L EMMA
Deﬁne the matrix M as
M :=
[Equation]
− 1 (cid:89) l = (cid:32) n (cid:89)
i =
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S K 2 − l n − i +1 ) (
cid:33)  (cid:62)   k (cid:48) − 1 (cid:89) l = (cid:32) n (cid:89) i
[-=
-]=
( I − S K 2 − l i ) (cid:33) (cid:32) n (cid:89) i
[-=
-]= ( I − S K 2 − l n − i +1 ) (cid:33)  .
Since M is independent of (cid:16)(cid:81) ni
{+=
+}
( I − S K 2 − k (cid:48) i ) (cid:17) (cid:16)(cid:81) ni
{+=
+}
( I − S K 2 − k (cid:48)
n − i +1 ) (cid:17) and z K 2 − k (cid:48) , using the tower rule, we get
E     k
[Equation]
[Equation]
We will now drop the superscript K 2 − k (cid:48) for convenience.
Hence, we need to control the following term:
[Equation]
(cid:32) n (cid:89) i
= ( I − S n − i +1 )
(cid:33)(cid:33) (cid:62)
E [ M ] z  
We deﬁne ( (cid:101) S 1 , . . .
, (cid:101) S 2 n ) as ( (cid:101) S 1 , . . .
, (cid:101) S n ) =
( S 1 , . . .
, S n ) and ( (cid:101) S n +1 , . . .
, (cid:101) S 2 n ) =
( S n , . . .
, S 1 ) .
Then, we use Eq.
(24) to get (cid:32)(cid:32) n (cid:89) i
= ( I − S i ) (cid:33) (cid:32) n (cid:89)
i =
( I − S n − i +1 )
(cid:33)(cid:33) =
2 n (cid:89) i =
( I − (cid:101) S i )
= I − 2 n (cid:88)
j = (cid:101) S j + 2 n (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j .
Note that I − (cid:80) 2 nj =1 (cid:101)
S
j is a constant matrix.
Since (cid:101) S j = α A σ j , we have that (cid:107) (cid:101) S j (cid:107) ≤ αL by
Assumption 2.
Hence, α ≤ 12 nL then (cid:107) I − (cid:80) 2 nj =1 (cid:101) S j (cid:107) ≤ 1 .
Further, α ≤ 1 /L implies that (cid:107) I − S
i (cid:107) ≤ 1 , which implies (cid:107) M (cid:107) ≤ 1 .
Hence, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E   (cid:32)(cid:32) n (cid:89) i =
( I − S i ) (cid:33) (cid:32) n (cid:89) i
= ( I − S n − i +1 ) (cid:33)(cid:33) (cid:62)
E [ M ] z   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
E     I − 2 n (cid:88)
j = (cid:101) S j 
 (cid:62)
E [ M ] z   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E     2 n (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j 
 (cid:62)
E [ M ] z   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) I − 2 n (cid:88)
j = (cid:101) S j 
 (cid:62)
E [ M ] E [ z ] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E     2 n (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j 
 (cid:62)
E [ M ] z   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:107) E [ z ] (cid:107) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) E     2 n (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j 
 (cid:62)
E [ M ] z   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) .
We can apply Lemma 10 to bound (cid:107) E [ z ] (cid:107) .
So, we focus on the other term.
Using Lemma 5, (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E    
[Table]
(cid:33) (cid:101) [Table] 

[Table]
E [ M ]
[Table]

[Table]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
E    
[Table]
[Table]
(cid:33) [Table]


[Table]
E [ M ]
[Table]
(cid:33)  
 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E
[Table] 
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
(cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ (cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13) E    
[Table]
[Table]
t
[Table]
(cid:33)  
[Table]
(cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) E    [Table] M ] [Table]
t
[Table]
(cid:33)    (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Table] + E     2 n (cid:88)
j = j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= (cid:107) I − (cid:101) S p (cid:107) (cid:33) (cid:107) (cid:101) S l (cid:107)(cid:107) (cid:101) S j (cid:107) 
 E [ (cid:107) M (cid:107) ]   n − 1 (cid:88) j
= j − 1 (cid:88) l
= (cid:32) l − 1 (cid:89) p
= (cid:107) I − S p (cid:107) (cid:33) (cid:107) S l (cid:107)(cid:107) S j (cid:107) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
n − j (cid:88) i =
t n +1 −
i
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
Now, we use
[Table]
M
[Table]
[Table]
−
[Table]
[Table]
(cid:102)
[Table]
and
[Table]
αG
[Table] : (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) E     2 n (cid:88) j = j − 1 (cid:88) l
[Table]
(cid:32) [Table] − 1 (cid:89) p
= ( I − (cid:101) S p ) (cid:33) (cid:101) S l (cid:101)
S j 
 (cid:62)
E [ M ] z   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:13) ≤ 4 n 3 α 4 L 3 G ∗ + 4 n 4 α 4 L 4 E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 n − j (cid:88) i
= t i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:35) + n 4 α 4 L 4 E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n − j (cid:88) i =
t n +1 − i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:35) . Using Lemma 6, we get
[Equation]
E 
[Equation]
p
[Equation]
 
[Equation]
  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
G
[Equation] [Equation]
G
[Equation]
(cid:112)
[Equation]
log n.
Putting everything together, (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:13) E   
[Table]
[Equation]
G P ROOF OF T HEOREM
Proof.
We start off by deﬁning the error term
[Equation]
where k ∈ [ K/ 2] .
This captures the difference between true gradients that the algorithms observes, and the gradients that a full step of gradient descent would have seen.
For two consecutive epochs of F LIP F LOP WITH I NCREMENTAL GD [Equation] we have the following inequality
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the ﬁrst inequality is due to Theorem 2.1.11 in Nesterov (2004) and the second one is simply ( a + b ) 2 ≤
a
+ 2 b 2 .
What remains to be done is to bound the two terms with r k dependence.
Firstly, we give a bound on the norm of r k :
[Equation]
Next, we will use the smoothness assumption and bounded gradients property (Lemma 1).
[Equation]
Hence,
[Equation]
For the r k term
[Equation]
we need a more careful bound.
Since the Hessian is constant for quadratic functions
[Equation]
we use H i to denote the Hessian matrix of function f i ( · ) .
We start off by using the deﬁnition of r k : r k =
[Equation]
where we used the fact that for a quadratic function f with Hessian H , we have that ∇ f ( x ) −∇ f
( y ) =
H ( x − y ) .
After that, we express x 2 k − 1 i − 1 − x 2 k − 1and x 2 k i − 1 − x 2 k − 1as sum of gradient descent steps: r k = n (cid:88) i = H i   i − 1 (cid:88) j = − α ∇ f j ( x 2 k − 1 j − 1 ) 
 + n (cid:88) i =
H n − i +   n (cid:88)
{+j = − α ∇ f j ( x 2 k − 1 j − 1 ) + i − 1 (cid:88) j = − α ∇ f n − j +
 ( x 2 kj − 1 ) +}
 = − α n (cid:88) i =
H i   i − 1 (cid:88)
j = ∇ f j ( x 2 k − 1 j − 1 ) 
 − α n (cid:88) i =
H n − i +   i − 1 (cid:88)
j = ∇ f n − j +
( x 2 kj − 1 ) 
 − α n (cid:88) i =
H n − i +   n (cid:88)
j = ∇ f j ( x 2 k − 1 j − 1 ) 
 = − α n (cid:88) i =
H i   i − 1 (cid:88)
j = ∇ f j ( x 2 k − 1) 
 − α n (cid:88) i =
H n − i +   i − 1 (cid:88)
j = ∇ f n − j +
( x 2 k 0 ) 
 − α n (cid:88) i =
H n − i +   n (cid:88)
j = ∇ f j ( x 2 k − 1) 
 − α n (cid:88) i =
H i   i − 1 (cid:88)
j = ∇ f j ( x 2 k − 1 j − 1 ) − f j
( x 2 k − 1)   − α n (cid:88) i = {+H n − i +   i − 1 (cid:88) j = ∇ f n − j +
+}
{+( x 2 kj − 1 ) − ∇ f n − j +
 ( x 2 k 0 )   − α n (cid:88) i =+} H n − i +   n (cid:88)
j = ∇ f j ( x 2 k − 1 j − 1 ) − ∇ f j ( x 2 k − 1) 
 = − 2 α n (cid:88) i =
H i   n (cid:88)
j = ∇ f j ( x 2 k − 1)  
+ α n (cid:88) i = H i ∇ f
i x 2 k − 1) − α n (cid:88) i = H i   i − 1 (cid:88)
j = ∇ f j[-+
-] ( x 2 k − 1 j − 1 ) − f j[-+
-]
( x 2 k − 1) 
 − α n (cid:88) i =
H n − i +   i − 1 (cid:88)
j = ∇ f n − j {++
+}
( x 2 kj − 1 ) − ∇ f n {+− j +
 ( x 2 k 0 ) +}

− α n (cid:88) i = H i   n (cid:88)
j = ∇ f
[Equation]
x
[Equation]
k − 1 j − 1 ) − ∇ f
[Equation]
x
[Equation]
− 1)

 Next, we use the fact that (cid:80) nj =1 ∇ f j ( x ) = n ∇ F ( x ) .
We will also again use the fact that for a quadratic function f with Hessian H , we have that ∇ f ( x ) − ∇ f ( y ) =
H ( x − y ) :
r k = − 2 α n (cid:88) i =
H
[Equation]
n
[Equation]
k − 1)) + α n (cid:88) i = H i (
[Equation]
f
[Equation]
( x
[Equation]
− 1) − ∇ f i ( x ∗ ))
+ α n (cid:88) i =
H i
[Equation]
f i ( x ∗ ) − α n (cid:88) i =
H i   i − 1 (cid:88)
j = ∇ f j ( x 2 k [Equation] 1 j − 1 )
[Equation]
f j ( x 2 k − 1) 
 − α n (cid:88) i =
H n − i +   i − 1 (cid:88)
j = ∇ f n [-− j +
-][Equation]
( x [Equation] [Equation] f n[-− j +
-]
[Equation]
1 ( x 2 k 0 ) 
 − α n (cid:88) i =
H i   n (cid:88)
j =
[Equation]
f
[Equation]
( x [Equation] [Equation] − 1 )
[Equation]
f
[Equation]
( x 2 k − 1) 
 =
[Equation]
(cid:32) n (cid:88) i =
H i (cid:33)
[Equation]
( x 2 k − 1− x ∗ )
+ α n (cid:88) i =
H 2 i ( x 2 k − 1− x ∗ ) + α n (cid:88) i =
H i ∇ f i ( x ∗ ) − α n (cid:88) i =
H i   i − 1 (cid:88)
j = ∇ f j ( x 2 k − 1 j − 1 ) − f j
( x 2 k − 1) 
 − α n (cid:88) i =
H n − i +   i − 1 (cid:88)
{+j = ∇ f n − j +
 ( x 2 kj − 1 ) − ∇ f n − j +
 ( x 2 k 0 ) +}  − α n (cid:88) i =
H i   n (cid:88) j = ∇ f j ( x [Equation] k −
[Equation] j − 1 ) − ∇ f j ( x [Equation]
k − 1) 
 =
a k + b k , where the random variables a k , b k as a k := − 2 α (cid:32) n (cid:88) i =
H i (cid:33)( x 2 k − 1− x ∗ ) + α n (cid:88) i = H 2 i ( x 2 k − 1− x ∗ ) + α n (cid:88) i = H i ∇ f i ( x ∗ ) , and b k := − α n (cid:88) i = H i   i − 1 (cid:88)
j = ∇ f j ( x 2 k − 1 j − 1 ) − f j
( x 2 k − 1) 
 − α n (cid:88) i = H n − i +   i − 1 (cid:88)
{+j = ∇ f n − j +
 ( x 2 kj − 1 ) − ∇ f n − j +
 ( x 2 k 0 ) +}
 − α n (cid:88) i = H i   n (cid:88)
[Equation]
Again, using smoothness assumption and bounded gradients property (Lemma 1) we get,
[Equation]
Next, we decompose the inner product of x 2 k − 1− x ∗ and E (cid:2) r k (cid:3) in Eq.
[Equation]
[Equation] [Equation] [Equation]
[Equation]
For the ﬁrst term in (42), − 2 α (cid:10) x 2 k − 1− x ∗ , a k (cid:11) = 4 α(cid:42) x 2 k − 1− x ∗ , (cid:32) n (cid:88) i =
H i (cid:33)( x 2 k − 1− x ∗ ) (cid:43) − 2 α(cid:42) x 2 k − 1− x ∗ , n (cid:88) i = H 2 i ( x 2 k − 1− x ∗ ) (cid:43) − 2 α(cid:42) x 2 k − 1− x ∗ , n (cid:88) i = H i ∇ f i ( x ∗ ) (cid:43) ≤ 4 α(cid:42) x 2 k − 1− x ∗ , (cid:32) n (cid:88) i =
H i (cid:33)( x 2 k − 1− x ∗ ) (cid:43) − 2 α(cid:42) x 2 k − 1− x ∗ , n (cid:88) i = {+H i ∇ f i ( x ∗ ) (cid:43) = 
+}
α 2 n 2 (cid:107)∇ F ( x 2 k − 1) (cid:107) 2 − 2 α(cid:42) x 2 k − 1− x ∗ , n (cid:88) i = H i ∇ f i ( x ∗ ) (cid:43) {+≤ 
+}
{+α 2 n 2 (cid:107)∇ F ( x 2 k − 1) (cid:107) 
 + 
 α 2 nLG+}
[Equation]
For the second term in (42), we use Cauchy-Schwarz and Ineq.
[Equation]
[Equation]
Substituting (43) and (44) back to (42), we get
[Equation]
Substituting (40) (45) back to (39), we ﬁnally get a recursion bound for one epoch:
[Equation]
−
[Equation]
Next, we use the fact that 2 ab ≤ λa 2 + b 2 /λ (for any λ > 0 ) on the term 2 α 2 nLG (1 + 3 αLn 2 ) (cid:107) x 2 k − 1− x ∗ (cid:107) to get that
[Equation]
αLn
[Equation]
[Equation]
Substituting this back we get, (cid:107) x 2 k n − x ∗
[Equation]
[Equation]
[-1 α 3 nL 2 G 2 (1 + 3 αLn 2 ) 2 + nαµ (cid:107) x 2 k − 1− x ∗ (cid:107) 2 + 
 α 4 n 4 G 2 L≤ (1 − 2 nαµ + nαµ ) (cid:107) x 2 k − 1− x-]Since µ ≤ L
[Equation]
Now, substituting the values of α and the bound on K , we get that 4 nα 1 L + µ − 12 α 2 n 2 ≥ 0 and hence,
[Equation]
Now, iterating this for K/ 2 epoch pairs, we get
[Equation]
+ 4 Kα 4 n 4 G 2 L≤ e − nαµK/ 2 (cid:107) x 1 0 − x ∗ (cid:107) 2 + K 2 µ − 1 α 3 nL 2 G 2 (1 + 3 αLn 2 ) {+
 + 
 Kα 4 n 4 G 2 L≤ e − nαµK/ 2 (cid:107) x 10 − x ∗ (cid:107) 
 + Kµ − 1 α 3 nL 2 G 2 + 9 Kµ − 1 α 5 n 5 L 4 G 2 + 4 Kα 4 n 4 G 2 L(Since+}
(1 + a ) 2 ≤ 2 + 2 a 2 )
Substituting α = 6log nK µnK gives us the desired result.
H A DDITIONAL E XPERIMENTS
[Figure]
The ﬁgures show the median and inter-quartile range after 10 runs of each algorithm, with random initializations and random permutation seeds (note that IGD exhibits extremely small variance). We set n = 800 , so that n (cid:29) K and hence the higher order terms of K dominate the convergence rates. Note that both the axes are in logarithmic scale.
Although our theoretical guarantees for F LIP F LOP only hold for quadratic objectives, we conjecture that F LIP F LOP might be able to improve the convergence performance for other classes of functions, whose Hessians are smooth near their minimizers.
To see this, we also ran some experiments on 1-dimensional logistic regression.
As we can see in Figure 3, the convergence rates are very similar to those on quadratic functions.
The data was synthetically generated such that the objective function becomes strongly convex and well conditioned near the minimizer.
Note that logistic loss is not strongly convex on linearly separable data.
Therefore, to make the loss strongly convex, we ensured that the data was not linearly separable.
Essentially, the dataset was the following: all the datapoints were z = ± 1 , and their labels were y =
1 z> 0 with probability 3 / 4 and y = 1 z< 0 with probability 1 / 4 .
Framing this as an optimization problem, we have
[Equation]
where h ( xz ) = 1 / (1 + e − xz ) .
Note that x = − log 3 is the minimizer of this function, which is helpful because we can use it to compute the exact error.
Similar to the experiment on quadratic functions, n was set to 800 and step size was set in the same regime as in Theorems 4, 5, and 6.