A G ENERAL A NALYSIS OF E XAMPLE -S ELECTION FOR S TOCHASTIC G RADIENT D ESCENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Training example order in SGD has long been known to affect convergence rate.
Recent results show that accelerated rates are possible in a variety of cases for permutation-based sample orders, in which each example from the training set is used once before any example is reused.
In this paper, we develop a broad condition on the sequence of examples used by SGD that is sufﬁcient to prove tight convergence rates in both strongly convex and non-convex settings.
We show that our approach sufﬁces to recover, and in some cases improve upon, previous state-of-the-art analyses for four known example-selection schemes: (1) shufﬂe once, (2) random reshufﬂing, (3) random reshufﬂing with data echoing and (4)
Markov Chain Gradient Descent.
Motivated by our theory, we propose two new example-selection approaches.
First, using quasi-Monte-Carlo methods, we achieve unprecedented accelerated convergence rates for learning with data augmentation.
Second, we greedily choose a ﬁxed scan-order to minimize the metric used in our condition and show that we can obtain more accurate solutions from the same number of epochs of SGD.
We conclude by empirically demonstrating the utility of our approach for both convex linear-model and deep learning tasks
I NTRODUCTION
To minimize a differentiable function f : R d → R , stochastic gradient descent (SGD) iteratively updates a parameter vector w ∈ R d starting at some w 0 by running
[Equation] [Equation]
where α t is the step size at iteration t , x t is a data example (often a minibatch of data) chosen by some process—typically by subsampling a training dataset—for SGD to use at iteration t , and ∇ f ( w t ; x t ) is an example gradient , which we hope will be a good approximation for the gradient of the objective ∇ f ( w t ) .
In the standard setup, the x ’s are drawn from a dataset D of size n , and f ( w ) = 1 n (cid:80) x ∈D f
( w ; x ) , which is often referred to as Empirical Risk Minimization (ERM).
The order in which the example sequence x 0 , x 1 , . . .
are chosen is known to affect the convergence of SGD.
For instance, compare so-called “random reshufﬂing” to with-replacement sampling: optimizing a strongly convex objective with T total iterations, random reshufﬂing samples the x t from D without replacement and achieves an accelerated convergence rate of 1 / T 2 , while with-replacement sampling yields a convergence rate of 1 / T (Bottou, 2012; Recht & R´e, 2012; G¨urb¨uzbalaban et al., 2021).
Similar accelerated rates have been shown in other settings and for other example orders, such as shufﬂing the dataset once (Nguyen et al., 2020; Ahn et al., 2020; Mishchenko et al., 2020).
However, these analyses have mostly focused only on speciﬁc example-selection schemes, and study only the case of ERM-type ﬁnite-sum objectives.
This does not help us understand how new example orders (e.g., the data echoing method of Choi et al.
(2019)) affect the convergence of SGD.
This paper develops a general condition on the example gradients themselves that is sufﬁcient to provide a convergence rate for SGD.
Intuitively, our main result is: the convergence rate of SGD depends on how fast the averages of consecutive example gradients ∇ f ( w ; x t ) converge to the full objective gradient ∇ f ( w ) .
With-replacement sampling causes the average of m consecutive gradient examples starting at any timestep τ , 1 m (cid:80) τ
+ m − 1 t = τ ∇ f ( w ; x t ) , to converge to ∇ f ( w ) at a rate of O ( 1 / m ) in terms of the norm squared: we show SGD with any example sequence that fulﬁlls this condition will converge at the same asymptotic rate as with-replacement sampling.
Alternatively, if that average converges at the faster ˜ O
( 1 / m 2 ) rate typical of Quasi-Monte-Carlo (QMC)
(Caﬂisch, 1998), then we show SGD enjoys the accelerated rate that random reshufﬂing gets.
Our contributions are as follows:
• We propose a new condition on the example gradients— average gradient error —and provide convergence analysis using this general condition for both non-convex and strongly-convex problems.
We justify the validity of this condition on synthetic experiments (Section 3).
• We show that many commonly used example orderings—shufﬂe once, random reshufﬂing, data echoing, Markov Chain Gradient Descent—can be analyzed as special cases under our theoretical results, which match or improve upon their existing rates in the literature (Section 4).
• We propose two new algorithms: (1) QMC-based data augmentation that transforms examples via a low-discrepancy sequence, improving generalization; and (2) a greedy algorithm that sorts the examples before each epoch based on our new average gradient error metric (Section 5).
• Empirically, we evaluate our two algorithms on several image classiﬁcation benchmarks including MNIST, CIFAR10/100 and ImageNet.
We show with QMC-based data augmentation, a higher validation accuracy can be achieved without hyperparameter tuning—this suggests that QMC may be a good default driver to use with data augmentation for deep learning in general.
Meanwhile, the greedy algorithm converges faster both in terms of iteration and wall-clock time (Section 6).
R ELATED WORK
Example ordering in stochastic optimization.
Traditional example ordering in SGD is carried out in a with-replacement fashion, which is used to ensure unbiased estimation of the full gradient (Robbins & Monro, 1951; Bach & Moulines, 2011; Zhang, 2004;
Bottou et al., 2018; Drori & Shamir, 2020).
Signiﬁcant attention has been paid to importance sampling with respect to various measures, such as Lipschitz constants (Schmidt et al., 2017; Needell et al., 2014), example gradient norms and bounds (Zhao & Zhang, 2015; Alain et al., 2015;
Papa et al., 2015; Lee et al., 2019), individual losses (Kawaguchi & Lu, 2020; Loshchilov & Hutter, 2015), and data heterogeneity (Lu et al., 2021).
Without-replacement sampling, however, is more common in practice and empirically allows faster convergence (Bottou, 2012).
Among the most popular without-replacement approaches are shufﬂe once (SO)
(Bertsekas, 2011; G¨urb¨uzbalaban et al., 2019) and random reshufﬂing (RR) (Ying et al., 2017).
In theory, Recht & R´e (2012) undertook the ﬁrst investigation on convergence of RR via the noncommutative arithmetic-geometric mean conjecture, to which subsequent works provide counter examples (Yun et al., 2021; De Sa, 2020).
HaoChen & Sra (2019) performed an epoch-wise acceleration analysis on RR while G ¨ urb ¨ uzbalaban et al.
(2021) considered its convergence over inﬁnite epochs.
Safran & Shamir (2020) analyzed the lower bounds for both SO and RR methods, and their results are further polished by Mishchenko et al.
(2020) via the Bregman divergence bound.
Aside from manual ordering, another line of research focuses on perturbed example ordering from data echoing (Choi et al., 2019; Agarwal et al., 2020).
Quasi-Monte Carlo.
Quasi-Monte Carlo (QMC) is a variant of Monte Carlo (MC) methods that uses a low-discrepancy sequence instead of a pseudorandom sequence.
QMC has been successfully applied in a wide variety of domains including computer graphics (Keller, 1995), ﬁnance (Joy et al., 1996), and computational biology (Cieslak et al., 2008).
In machine learning, using QMC in place of MC can signiﬁcantly improve many techniques including variational inference (Buchholz et al., 2018; Liu & Owen, 2021), feature mapping (Yang et al., 2014; Avron et al., 2016), normalizing ﬂows (Wenzel et al., 2018), deep learning based PDE (Chen et al., 2019), and time series analysis (Philipson et al., 2020).
For stochastic optimization, early works like Homem-de Mello (2008) and
Pennanen (2005) established the asymptotic convergence of a QMC sequence in terms of the training set size, while Jank (2005) proposed replacing MC with QMC in computing the E-step of the EM algorithm.
Similar to our motivation, Buchholz et al.
(2018) analyzed the convergence of SGD when samples are drawn using QMC.
Their approach differs signiﬁcantly from ours in that their method draws an independent unbiased length- b QMC sequence for each minibatch, while our examples come from contiguous subsequences of a length- T QMC sequence that is used across all iterations.
E XAMPLE - GRADIENT AVERAGES AND SGD CONVERGENCE
In this section, we describe our setup, deﬁne the average gradient error condition we are proposing, and state our main result.
Our objective is to minimize a continuously differentiable function f : R d
→ R using examples x t from some set X retrieved at each iteration t .
We make the usual assumption that both the loss gradient and the example gradients are L -Lipschitz continuous.
Assumption 1 ( L -Smoothness) .
For some L < ∞ , for any u, v ∈ R d and any example x ∈ X , (cid:107)∇ f ( u ; x ) − ∇ f ( v ; x ) (cid:107) ≤ L · (cid:107) u − v (cid:107) and (cid:107)∇ f ( u ) − ∇ f ( v ) (cid:107) ≤ L · (cid:107) u − v (cid:107) .
We propose a new condition on the average gradient error .
Informally, this condition bounds how averages of consecutive example gradients approximate the objective gradient.
Formally,
Assumption 2.
In the context of Equation (1), we say that the example sequence x 0 , x 1 , . . .
is ( γ, C, Φ) -concentrating for γ ∈ [1 , 2] , C > 0 , Φ ≥ 0 , if for any timestep τ ≥ 0 and any m > 0 ,
[Equation]
where w τ is the weight parameter vector arrived at after τ SGD update steps in Equation (1).
The constants C and Φ here may depend on the total number of iterations T : speciﬁcally, they may absorb logarithmic factors like log( τ + m ) 2 s typical in a QMC error bound, where s is the dimension of the sample space.
In addition, when the { x t } ’s are random, we will show that this assumption holds with high probability.
Furthermore, to show this assumption holds, it sufﬁces to show that the inequality holds for any w : our requirement that it holds only for the speciﬁc w τ arrived at by SGD is weaker.
We can develop intuition about Assumption 2 by considering familiar cases:
• For general with-replacement sampling, the sum in (2) is a sum of independent random variables; so, a concentration argument would yield Assumption 2 with γ = 1 with high probability.
• For general without-replacement sampling from a dataset of size n , any whole-epoch subsequences from the sum in (2) will cancel to zero, so we expect that sum to have magnitude O ( n ) independent of m , and thus Assumption 2 should hold with γ = 2 (Propositions 1 to 4).
• For low-discrepancy sampling, we would expect Assumption 2 to hold with γ = 2 , along with a multiplicative log( τ + m ) 2 s term: this is the classic error rate for QMC (Proposition 6).
In Section 4, we will make this intuition rigorous with high probability under the assumption that the error of the example gradients is bounded.
This intuition illustrates both how Assumption 2 can cover previously analyzed settings and how it can generalize to cases not previously studied (e.g.,
QMC data augmentation).
Also observe that Assumption 2 is easily adapted to minibatch SGD: if it holds for the single-example case, it should also hold with modiﬁed constants for minibatches of size b (consisting of averages of b consecutive example gradients) by substituting m (cid:55)→ mb .
Since this is straightforward, for simplicity of presentation we focus our theoretical analysis on the batch-sizecase.
Our analysis is based on the following key lemma, which bounds the evolution of SGD over an “analysis phase” of m steps.
Lemma 1.
Suppose our setup satisﬁes Assumptions 1 and 2 and that we use a constant step size α .
For all timesteps τ ≥ 0 , let m > 0 be some integer such that 3 αm 1 − γ / 2 Φ ≤ αm ≤ 16 L .
Then the objective at timestep τ + m is bounded by
[Equation]
By applying this lemma inductively and choosing the step size appropriately, we can derive convergence rates for SGD in a variety of settings.
Note that while here for simplicity we state results for a ﬁxed step size, Lemma 1 can also be used to get essentially the same rates for diminishing step size schemes, which we analyze in Appendices A.2 and A.3.
In what follows, we let f ∗ be the global minimum of f , let ∆ := f ( w 0 ) − f ∗ , and use ˜ O to hide logarithmic terms in the problem parameters such as C , Φ , L , ∆ , and (cid:15) , while treating γ as a constant.
Theorem 1 (Non-convex case) .
Suppose that our setup satisﬁes Assumptions 1 and 2, and let (cid:15) >be any target error.
Using SGD (1) with a constant step size α = 16 L (cid:6) (4 C/(cid:15)
+ 3Φ) 2 / γ (cid:7) − 1 , the number of steps T needed to achieve min t =0 , ··· ,T − 1 (cid:107)∇ f ( w t ) (cid:107) 2 ≤ (cid:15) 2 is at most
[Equation]
[Figure]
Observe that when γ = 1 , this gives the standard (cid:15) − 4 rate expected for non-convex SGD, and for γ = 2 , this gives us the accelerated (cid:15) − 3 rate of shufﬂing methods.
We can get an even faster rate if f satisﬁes the µ -Polyak-Łojasiewicz (PL) condition (cid:107)∇ f ( w ) (cid:107) 2 ≥ 2 µ ( f ( w ) − f ∗ ) , which generalizes strong convexity and has been applied before in the study of sample orders (Mishchenko et al., 2020;
Ahn et al., 2020): in particular the following theorem holds for µ -strongly convex functions.
Theorem 2.
Suppose that f satisﬁes the µ -PL condition and our setup satisﬁes Assumptions 1 and 2.
Let (cid:15) > 0 be any target error, and κ = L / µ be the condition number of the problem.
Using SGD (1) with a constant step size α = 16 L (cid:6) (8 C 2 / ( µ(cid:15) 2 ) + 9Φ 2 ) 1 / γ (cid:7)
[Equation]
1 , the number of steps T needed to guarantee f ( w T ) − f ∗ ≤ (cid:15) 2 is at most
[Equation]
Observe that when γ = 1 , this recovers the ordinary T = κ(cid:15) − 2 rate we usually get for strongly convex SGD, and when γ = 2 we get a faster rate of κ(cid:15) − 1 .
These theorems together show that
our
Assumption 2 is sufﬁcient to show the convergence of SGD, and the convergence-rate parameter γ of the assumption translates to affect the convergence rate of SGD.
This validates our intuition that faster convergence of averages of consecutive example gradients to the full gradient ∇ f ( w ) leads to faster convergence of SGD.
Synthetic experiments.
We quickly validate these results on a synthetic 10 -dimensional strongly-convex problem.
The ﬁrst setting we consider is the expected risk minimization of
R ( w ) =
E [( x T w − y ) 2 ] , where x ∼ N (0 , I d ) and y | x ∼ N ( x T w ∗ , 1) for some optimal value w ∗ .
We run SGD in an online fashion—at iteration t we draw samples ( x t , y t ) from the underlying distribution to compute ∇ R ( w t ; x t ) used in the update.
We compare drawing these samples independently at random against drawing using a QMC sequence (Sobol in [0 , 1] d ) via an inverse transform, using for both cases the same diminishing step size scheme selected to minimize the expected risk for the random-sampling case—the optimal step size scheme for vanilla SGD.
The online plot of
Fig.
1(a) shows that the convergence rate is strictly superior with QMC, which achieves a O ( 1 / t 2 ) rate compared to the O ( 1 / t ) rate of random sampling, which is what Theorem 2 predicts.
We also evaluate the ofﬂine setting, where we draw n independent examples from the same distribution to form a training set, and minimize the empirical risk R n ( w ) = 1 n (cid:80) ni
=1 ( x T i w − y i ) 2 .
This corresponds to a least squares problem with optimal solution w ∗ n .
We run SGD in an epoch-wise manner for K epochs, in which we compare sampling from the training set uniformly with replacement (IID Uniform), random reshufﬂing (RR), shufﬂe once (SO), and sampling using one QMC sequence in [0 , 1] of length T = nK followed by a mapping to example indices (Sobol).
In the ofﬂine plot of Fig. 1(a), we see that the low-discrepancy methods all yield an accelerated rate compared to sampling with replacement, which again validates our theory.
Additional details on the experiment setup, hyperparameters, and diminishing step size derivation can be found in Appendix A.1.3.
With the same synthetic setup, we also veriﬁed our bound in Assumption 2 by measuring the average gradient error over a sequence of examples.
The ﬁxed point w τ is arbitrarily set to the origin.
For the online setting (left of Fig. 1(b)), we use the same set of examples as in the SGD experiments above starting at t = 0 .
Similarly, in the ofﬂine setting, we go through the examples epoch-wise.
As we can see, the sample orderings given by QMC, RR and SO (ofﬂine only) indeed give us an accelerated rate of decrease in the average gradient error as we increase m ( γ ≈ 2 ), justifying our main assumption.
A NALYSIS OF EXISTING SCAN ORDERS
In this section, we illustrate the power of our approach by proving convergence rates for exampleselection methods proposed and analyzed in previous literature.
For each method, we show Assumption 2 holds with high probability, and then by applying Theorems 1 and 2 to these results, we show how existing rates for these methods can be recovered and in some ways improved upon.
To the best of our knowledge, these are the ﬁrst high-probability results for shufﬂe once and random reshufﬂe for general non-convex optimization.
When using a ﬁnite training set of examples, we let n denote its size, let the examples be indexed as x (0) , x (1) , . . .
, x ( n − 1) (to avoid confusion with x t , the example used by SGD at step t ), and let T = nK denote the total number of iterations after K epochs.
We will also require the following standard assumption that bounds the error of a single example gradient.
Assumption 3.
For all examples x ∈ X and points w ∈ R d , there exists A, B ≥ 0 such that the gradient errors satisfy (cid:107)∇ [Equation] [Equation] x
[Equation]
Shufﬂe once (SO).
In the shufﬂe-once variant of SGD, a single permutation σ of { 0 , . . . , n − 1 } is chosen uniformly at random at the start, and the examples are used repeatedly in that order: explicitly, x t = x ( σ ( t mod n )) .
One way to analyze shufﬂe-once is to prove Assumption 2 for permutation-based methods generally.
Proposition 1.
Let Assumptions 1 and 3 hold, and suppose that we are using a permutation-based method of sampling, that is, any method such that x kn , x kn +1 , . . .
, x kn + n − 1 is a permutation of x (0) , x (1) , . . .
, x ( n − 1) for all epochs k ≥ 0 .
Any such method satisﬁes Assumption 2 with γ = 2 ,
C 2 = n 2 A 2 and Φ 2 = n 2 B 2 .
This immediately lets us recover previous rates up to constant factors, but we can do better.
In the case where we are learning over a bounded region, we can prove a stronger result for shufﬂe once.
Proposition 2.
Suppose that we are using the shufﬂe once variant of SGD to learn over a region
B ∈ R d of radius at most R , such that the iterates w t are guaranteed to remain within this region.
Assume that for all w ∈ B and all examples x in the training set of size n , Assumption 1 ( LSmoothness) and Assumption 3 hold.
Then with probability at least 1 − p , Assumption 2 holds with γ = 2 , C 2 =
˜ O ( dA 2 ( n + B 2 )) , and Φ 2 =
˜ O ( ndB 2 ) .
In comparison to previous results and to our rate implied by Proposition 1, this improves the dependence from n to √ nd , which is a signiﬁcant improvement over the best rates for shufﬂe once available in the literature when the dimension is small relative the training set size.
In particular, if we set B = 0 and consider small (cid:15) , our rate in the non-convex case becomes
[Equation]
, which implies (cid:15) 2 ≤
[Equation]
This rate matches that for shufﬂe once achieved in Nguyen et al.
(2020, Corollary 1) in terms of (cid:15) , up to logarithmic factors.
In the µ -PL (or strongly convex) case for B = 0 , we again obtain a rate matching that of Nguyen et al.
(2020), Ahn et al.
(2020), and Mishchenko et al.
[Equation]
which implies (cid:15)
[Equation]
Random reshufﬂing (RR).
Random reshufﬂing is similar to shufﬂe once, except that a new ordering is chosen at each epoch: sampling the dataset without replacement.
Concretely, if σ k denotes the permutation used by random reshufﬂing at the k th epoch, then x t = x ( σ (cid:98) t/n (cid:99) ( t mod n )) .
Proposition 3.
Suppose that we are using the random reshufﬂing variant of SGD.
Assume that for all w ∈ R d and all examples, Assumption 1 ( L -Smoothness) and Assumption 3 hold.
For some p ∈ (0 , 1) , set the constant step size to satisfy α ≤ (cid:0) max (cid:8) 1460 BnL · log (cid:0) 4 e 2 T/p (cid:1) , 2 nL (cid:9)(cid:1) − 1 .
Then with probability at least 1 − p , Assumption 2 holds with γ = 2 , C 2 =
˜ O ( nA 2 ) and Φ 2 =
˜ O ( nB 2 ) .
Setting B = 0 , our rate in the non-convex case now becomes
[Equation]
which implies
[Equation]
Here we match the best rate obtained by Mishchenko et al.
(2020, Corollary 3) in the small (cid:15) setting.
In the µ -PL (or strongly-convex) case, we get
[Equation]
which implies
[Equation]
As in shufﬂe once, here our rate for random reshufﬂing matches that obtained by Nguyen et al.
(2020) and Ahn et al. (2020), as well as Mishchenko et al.
(2020) albeit with a slightly worse dependency on κ .
It’s worth noting that for simple quadratics, RR and SO are only faster than with-replacement
SGD when K (cid:38) 1 / µ (Safran & Shamir, 2021).
Random reshufﬂing with data echoing.
Data echoing is a technique that can be easily implemented in a machine learning training pipeline to increase throughput and improve performance.
It was ﬁrst introduced and tested empirically by Choi et al.
and
analyzed by Agarwal et al.
The idea is to perform multiple SGD updates on each example x i (or minibatch) before proceeding to the next.
By “echoing” examples we allow more time for upstream data loading and preprocessing, and consequently decrease downstream GPU idle time for gradient computation.
For simplicity, we also use a ﬁxed number of echos c as in Agarwal et al.
Concretely, a c -echoed version of a sample order ˆ x is given by x t
= ˆ x (cid:98) t/c (cid:99) .
Data echoing can essentially be applied to any example-ordering scheme, and here we provide one analysis under random reshufﬂing.
The justiﬁcation for Assumption 2 under random reshufﬂing with data echoing follows essentially without modiﬁcation from the c = 1 version in the previous subsection.
Proposition 4.
Suppose that we are using the random reshufﬂing variant of SGD, where each example is echoed c times using the same step size in RR.
Under the same assumptions as in Proposition 3, with probability at least 1 − p , Assumption 2 holds with γ = 2 , C 2 =
˜ O ( cnA 2 ) and Φ 2 =
˜ O
( cnB 2 ) .
It immediately follows that data echoing should get the same convergence rates we showed in
Section 4 with A 2 and B 2 multiplied by c .
Although Agarwal et al.
(2020) also provided an analysis for data echoing, they require that the examples are sampled independently, rather than the random reshufﬂing setting that is more commonly-used: as a result, their analysis did not achieve the accelerated (cid:15) − 3 rate that shufﬂed methods enjoy.
Another advantage of our analysis is that the proof follows exactly from that of vanilla random reshufﬂing, from which the constant c simply propagates.
Markov chain gradient descent (MCGD).
To illustrate the versatility of Assumption 2, we show how it can be satisﬁed by a problem where the objective is not a ﬁnite sum and where γ (cid:54)
= 2 .
Consider f ( w ) =
E ξ ∼ Ξ [ f ( w ; ξ )] with some underlying distribution Ξ .
Running SGD then requires that at each iteration t , we draw ∇ f ( w t ; ξ t ) where ξ t ∼ Ξ ; however, sampling from Ξ can be intractable.
The method of Markov Chain Gradient Descent addresses this problem by sampling the ξ t from the trajectory of a single Markov chain with stationary distribution Ξ (Sun et al., 2018).
The intuition is that although at early iterations the ξ ’s have not converged to their true distribution, the iterates visited by SGD are also far from the optimum, thus larger approximation error in the early ξ ’s is rather harmless.
As we continue iterating, the Markov chain will mix as SGD converges.
We show that the convergence of MCGD can be bounded in terms of the mixing time of that Markov chain.
Proposition 5.
Suppose that we use samples x t from a Markov chain with mixing time t mix .
Assume that for all w ∈ R and all examples x t , Assumption 3 holds.
Then with probability at least 1 − p ,
Assumption 2 holds with γ = 1 , C 2 =
˜ O
( A 2 t 2 mix ) , and Φ 2 =
˜ O
( B 2 t 2 mix ) .
It follows that for non-convex optimization in the B = 0 case, our convergence rate is given by
[Equation]
, which implies (cid:15)
[Equation]
(cid:17) .
Sun et al.
(2018, Theorem 2) use a diminishing step size O ( 1 / t q ) to obtain T
= O ( (cid:15)
− 2 / 1 − q ) , where q ∈ ( 1 / 2 , 1) .
In contrast, our rate is faster and holds with high probability instead of in expectation.
N EW EXAMPLE - SELECTION METHODS FOR FASTER CONVERGENCE
The analysis in Section 4 focused on recovering the convergence rates for SGD with known exampleordering algorithms.
In this section, we propose two new example-selection approaches that allow faster convergence: QMC-based data augmentation and greedily minimizing the metric in (2). Input: number
- tion: g i ← 0 , i , n − } , g ← , initial initial σ 0 .
1: for t = 0 , · · · , T/n − 1 do 2:
Initialize : g i ← 0 , ∀ i ∈ { 0 , · · · , n − 1 } ; g ← 0 ; I ← ∅ .
3: for n − 1 do 4:
Update the model parameters: w tn
+ i +1 ← w tn
+ i − α ∇ f ( w tn
+ i ; x σ t ( i )) ) .
Update the buffers: g σ t ( i ) ← Π ∇ f ( w tn + i ; x ( σ t ( i )) )
; g ← g + g σ t ( i ) .
6: end for 7: for i = 0 , · · · n − 1 do 8: σ t +1 ( i ) ← arg min i ∈{ 0 , ··· ,n − 1 }\I (cid:20)(cid:13)(cid:13)(cid:13)(cid:13) (cid:80) j ∈I∪{ i } ( g j − g/n ) (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:21) ; I ← I ∪ { σ t +1 ( i ) } .
9: end for 10: end for 11: return w T
QMC-based data augmentation.
In many scenarios where only limited examples are given, we want to augment the dataset for better generalization.
More formally, given a transform function
A that takes example x and a random variable ζ uniformly distributed in [0 , 1] s as input, where s denotes the dimension of augmentation space, the augmented objective for a dataset D of size n is f ( w ) = 1 n (cid:80)
[Equation]
The rationale is that by performing some reasonable random transformation on a given example, we assume the output would be another example that is identically distributed, and the expected value models an inﬁnitely-large training set consisting of such transformed examples.
For example, in an image classiﬁcation task, we could set s = 1 and have A ( x, ζ ) output the image x rotated by an angle of 20 ◦ (2 ζ − 1) , modeling that a slight rotation of an image should preserve its label.
QMC can approximate this expectation with a low-discrepancy sequence of ζ t drawn from the s -dimensional unit cube [0 , 1] s .
Examples of such sequences include the Halton and Sobol sequences (Drmota & Tichy, 2006).
QMC is especially favorable for data augmentation because s is usually small in most data augmentation methods.
We propose to use QMC for data augmentation together with random reshufﬂing.
Concretely, if ζ 0 , ζ 1 , . . . is our low-discrepancy sequence and σ k denotes the permutation used by random reshufﬂing in the k th epoch, then we propose to use the example x t =
A ( x (
[Equation]
(cid:98) t/n
[Equation]
t
[Equation]
n ))
, ζ (cid:98) t/
[Equation]
(cid:99)
+ [Equation] t/n (cid:99) ( t mod n ) ) . That is, when we sample example i in epoch k , we use the ( k + i ) th element of the low-discrepancy sequence.
(This is not the only reasonable way of combining QMC and RR: it is just one way we found to work well.)
In theory we would expect the approximation error here to decay at the rate ˜ O
( 1 / m 2 ) , instead of the O ( 1 / m ) of the random sampling that is standard for data augmentation.
To prove this rigorously, we make two additional assumptions which are commonly used in analyzing QMC sequences (Aistleitner & Dick, 2014).
Assumption 4 (Bounded gradient variation) .
There exists a constant V > 0 such that for any ﬁxed w ∈ R d and x ∈ X , the example gradients as a function of ζ under QMC data augmentation, F ( ζ ) =
∇ f ( w ; A ( x, ζ )) , has Hardy-Krause variation (Tezuka, 2000) at most V , that is V HK ( F ) ≤ V .
Assumption 5.
The QMC sequence { ζ t } t ≥ 0 has low star-discrepancy (Owen, 2003): for all m > 0 ,
[Equation]
where [0 , a ) = { x ∈ [0 , 1] s | 0 ≤ x j < a j , j = 1 , . . .
, d } for some constant C QMC .
Proposition 6.
Suppose that we are using the random reshufﬂing variant of SGD with QMC data augmentation as described.
Assume that for all w ∈ R d and all examples, Assumptions 1 , 3, 4 and 5 hold for Equation 3.
For some p ∈ (0 , 1) , set the step size to be a constant such that α ≤ (cid:0) max { 1460 BnL · log(4 e 2 T/p (cid:1) , 2 nL } ) − 1 .
Then with probability at least 1 − p , Assumptionholds with γ = 2 , C 2 = ˜ O
( n 2 V 2 C 2 QMC log( T ) 2 s + nA 2 ) and Φ 2 = ˜
O ( nB 2 ) .
Comparing it with Proposition 3, this QMC variant enjoys the same ˜ O ( 1 / T 2 ) rate we get for vanilla random reshufﬂing: to our knowledge, this is the ﬁrst accelerated rate for learning with data augmen-
[Figure]
tation.
In addition to the theoretical advance, we demonstrate in Section 6 that our QMC variant can achieve better validation performance in practice in multiple applications.
Better example ordering via greedy selection
Taking a closer look at Assumption 2, the magnitude of the left-hand side plays a crucial role in the convergence.
In the ERM setting, this motivates us to select a permuted example order that minimizes this expression, following which
SGD would converge with minimized average gradient error.
However, naively constructing such a sequence is tedious as iterating over all the τ, m > 0 and all permutations can be computationally intensive.
In light of this, we apply several approximation techniques into the construction and formulate it into Algorithm 1.
The ﬁrst technique is to use stale gradients , i.e., using the gradients computed at each epoch to estimate the sequence used in the next epoch (line 5 in Algorithm 1).
The intuition is that based on the smoothness of loss function (Assumption 1), we would expect the stale gradients to approximate the current gradients with tolerant approximation error as long as the step size is reasonably small.
The second technique is random projection (line 9): we search a lower-dimensional space, which allows faster construction and reduces memory use (see similar strategies of using smoothness and dimension reduction techniques in (Caﬂisch, 1998)).
Because this is a selection-by-permutation method, under Assumption 3 our greedy selection method will trivially satisfy Assumption 2 with γ = 2 , C = nA , and Φ = nB (see Proposition 1).
E XPERIMENTS
[Table]
In this section we evaluate our new algorithms on several deep learning benchmarks.
First, we compare QMC-based data augmentation against
IID-uniform augmentation on CIFAR10/100 and
ImageNet datasets.
Second, we compare greedy ordering (Algorithm 1) with RR and SO, and show how to further accelerate it with randomly projected sorting.
Other details on the experimental setup can be found in Appendix A.1.1.
Each experiment is repeated 10 times with consistent seeds among the algorithms.
QMC-based data augmentation.
We start by training ResNet20 on CIFAR10 and CIFAR100, where discrete and continuous data augmentations are applied, respectively.
Speciﬁcally, CIFAR uses random crop and random horizontal ﬂip while CIFAR100 uses an additional random rotation of 15 degrees.
To apply the QMC-based data augmentation, we ﬁrst generate a Sobol sequence of appropriate dimension using the qmcpy package (Choi et al., 2020+), and then replace the pseudorandom sequence used in the original random augmentation pipelines with that Sobol sequence.
We run the baseline IID-uniform method with ﬁnetuned hyperparameters (weight decay 1e-4), which reproduces the result from He et al.
(2016) with an error rate 8.4%.
Then we run QMC-base augmentation with the same hyperparameter (untuned) and ﬁnetuned counterparts (we perform grid search on weight decay in the range of { 1e-4, 2e-4, 3e-4, 4e-4 } ).
From Figure 2 we observe the QMCbased augmentation consistently outperforms the baseline methods, even without hyperparameter tuning.
Comparing Figure 2 with He et al.
, we observe the QMC-based augmentation allows
ResNet20 to reach comparable validation accuracy as ResNet44 while requiring only 40% as many parameters (0.27M vs 0.66M).
We run a t -test on these results (in Appendix A.1.2) to show the validation accuracy from the two augmentation methods are different statistically signiﬁcantly (p-value
[Figure]
[Figure] p =
7e-7 on CIFAR10 and p =0.036 on CIFAR100).
Importantly, this improvement comes essentially for free, as generating low discrepancy sequences in low dimension has very little overhead.
We also evaluate our method on ﬁne-tuning ResNet18 on ImageNet.
We apply different augmentation methods on a pre-trained model and train for 5 additional epochs with step size 1e-4.
We report their
Top1 accuracies in Table 6, which shows that our QMC method improves the validation accuracy by 0 .
3% compared to the same number of epochs of ﬁne-tuning using random sampling.
Better example ordering via greedy selection.
In this section we evaluate Algorithm 1 on two benchmarks: Logistic Regression on MNIST and ResNet20 on CIFAR10.
As discussed, sorting the stale gradients naively could incur substantial overhead on memory and computation.
To mitigate this, we adopt two methods: random projection and QR decomposition.
The former is mainly to reduce storage: we obtain a gradient computed at some time in an epoch and project it into a lower-dimensional space before storing it.
Classic ways of projection include Gaussian projection or random sparsiﬁcation: we adopt the latter as it does not require storing the projection matrix, which minimizes the storage cost.
After we obtain all the stale gradients, we concatenate them into a matrix and perform QR decomposition before sorting, which allows us to sort in a low-dimensional space while preserving the order of gradients since the inner products between any two tensors will remain the same Gander (1980).
We set the target dimension to be of 10% size of the original space.
In the spirit of evaluating the applicability of Algorithm 1, we do not perform hyperparameter tuning in this section but reuse the ones tuned in the literature on Random Reshufﬂing.
We plot the results in Figure 3.
In Figure 3(a) we observe greedy algorithms can consistently converge faster than RR and SO epoch-wise with optimizing the same loss function.
When QR is used without projection, the algorithm is able to reach higher validation accuracy but converges slower with respect to the wall-clock time.
On the other hand, when we use random sparsiﬁcation additionally, the algorithm converges faster with respect to both epoch and wall-clock time without compromising the validation accuracy.
For CIFAR10, we observe the greedy method can converge faster when optimizing the same loss as other baselines, and achieves higher validation accuracy when ﬁne-tuned.
C ONCLUSION
We present a uniﬁed analysis on example orderings used in SGD, which generalizes several widelyused orderings in the literature.
We propose a greedy algorithm that allows faster convergence via constructing a better example order with approximate sorting techniques, as well as QMC-based augmentation that achieves higher validation accuracy on multiple benchmarks. One potential future direction is designing example orderings to more efﬁciently minimize the average gradient errors.
T ABLE OF CONTENTS
A.1 Experiment details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.1.1 Additional details for Section 6 . . . . . . . . . . . . . . . . . . . . . . .
A.1.2 t -test for data augmentation results . . . . . . . . . . . . . . . . . . . . . .
A.1.3 Small experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.1.4 Optimal step size derivation . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Convergence analysis: diminishing step size . . . . . . . . . . . . . . . . . . . . .
A.2.1 Non-convex case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2.2 Strongly-convex case . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Proof for Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Convergence analysis: constant step size . . . . . . . . . . . . . . . . . . . . . . .
A.4.1 Non-convex case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4.2 Strongly-convex case . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Justiﬁcations for Assumption 2 under various example orderings . . . . . . . . . .
A.
5.1 Arbitrary permutation
A.
5.2 Shufﬂe once . .
A.
3 Random reshufﬂing
A.5.
4 Random reshufﬂing with data echoing
A.
[Table]
5 Markov chain gradient descent
A.5.
6 QMC-based data augmentation with random reshufﬂing
A.
6 Miscellaneous lemmas
A. E XPERIMENT DETAILS A.1. A DDITIONAL DETAILS FOR S ECTION
In Section 6, all the training scripts are implemented via PyTorch1.6 and run on a single machine conﬁgured with an 2.6GHz 4-core Intel (R) Xeon(R) CPU, 16GB memory and NVIDIA GeForce
GTX 1080Ti with CUDA 10.1.
In the example ordering comparison, we use the same seed among different algorithms in the same run so as to guarantee every algorithm works with the same loss function.
In the ImageNet experiment, the standard step size schedule for ImageNet training is starting at 0.1 and decaying by 10 every 30 epochs.
The pre-trained model is the trained model at epoch 90.
Naturally, our learning rate should be 1e-4 by the same schedule.
Other hyperparameters are adopted by the open source implementation: https://github.com/pytorch/examples/tree/ master/imagenet .
In the data augmentation comparison, obviously other augmentation techniques can be used.
The strategies we used are taken from open source implementation https://github.com/ akamaster/pytorch_resnet_cifar10 and https://github.com/weiaicunzai/ pytorch-cifar100 that can reproduce the validation accuracy in He et al.
(2016), so that our comparison can be consistent with the correct benchmarks.
Below we include the convergence plot for the experiment on ImageNet with ResNet18, where each algorithm is repeated three times with seeds uniformly selected from [0, 1000].
A.1. t - TEST FOR DATA AUGMENTATION RESULTS
We now perform t-test on the validation accuracy on two data augmentation results, to show the improvement from QMC is statistically signiﬁcant.
We include validation accuracy at epochfor CIFAR10 and best accuracy for CIFAR100 in Table A.1.2.
We compute the p-value between
IID-Uniform and tuned QMC, and found the p-value on CIFAR10 and CIFAR100 to be 7e-7 and 0.036, respectively.
Since they are both smaller than 0.05, we reject the null hypothesis which concludes they are of the same mean.
A.1. S MALL EXPERIMENTS
For both experiments in Section 3, we generated w ∗ from a standard Normal distribution.
The minibatch size is 1 , and the total number of iterations is 10 7 , with n = 10 4 over 1000 epochs for the ofﬂine setting.
For all variants, we use the theoretical step size optimized for SGD with replacement (corresponding to IID Uniform) on this particular problem, given by
[Equation]
which are derived in Appendix A.1.4 below.
To obtain low-discrepancy samples from the Gaussian distribution, we use the inverse transform method.
First we obtain the QMC sequences (Sobol in our [Table] case) ζ t ∈ [0 , 1] s where s is the appropriate dimension, and then use that in the inverse CDF function of a Gaussian distribution to obtain the corresponding Gaussian sample.
In addition to using synthetic data, we also performed an ofﬂine version of the experiment Figure 1(b) on a real dataset, a6a from the LIBSVM repository (Chang & Lin, 2011).
The dataset contains n =11220 examples with d =124 features (including bias), and all labels are binary.
We use logistic regression with (cid:96) 2 regularization (with λ =1e-4) as the empirical risk:
[Equation] To measur the left hand side of Assumption 2, we again ﬁx w τ to be at the origin, and we take the number of epochs to be 1000. The results are in Figure 5 for one run with an arbitrarily-set seed, and we draw similar conclusions as observed with synthetic data (see Figure 1(b) in the main paper). [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
Offline (a6a)
IID Uniform
Sobol
RR SO A.1. O PTIMAL STEP SIZE DERIVATION In this section, we derive the optimal step-size sequence used in our synthetic toy example from
Section 3.
Recall our setup
[Equation]
where w ∗ is the true model parameters that we are trying to recover, and (cid:15) ∼ N (0 , 1) is some intrinsic error.
The goal is to minimize the expected risk
[Equation]
By using the tower rule of expectation conditioning on x , this objective can be written as
[Equation]
The full gradient and the example gradient given a pair ( x, y ) drawn from the above distribution are
[Equation]
Deﬁne the gradient error to be
[Equation]
Let us ﬁrst analyze the expected gradient error conditioned on any randomness in w that might arise from the sampling history:
[Equation]
The last term is simply E [ (cid:15) 2 x T x ]
=
d since (cid:15) is independent of x .
The middle term cancels to 0 again by independence and zero-mean of (cid:15) .
For the ﬁrst term,
[Equation]
Together, we have
[Equation]
Now let us derive an suboptimality gap recursion for the SGD update step using the step size α t : [Equation] t
[Equation]
where ∆ g t is the expression in Eq.
(4) with w t as input.
The expected suboptimality gap conditioned on all randomness up to timestep t is then
[Equation]
Observe that under iid uniform sampling [Equation] E [∆ gt ] =
[Equation]
and using Eq.
(5) gives us
[Equation]
Taking an expectation over the entire history to remove conditioning and let ρ t := E (cid:104)
[Equation]
Note that the RHS is convex in α t , we can differentiate to minimize the expected suboptimality at every iteration ρ t +1 =
E (cid:104) (cid:107) w t +1 − w ∗ (cid:107) 2 (cid:105) :
[Equation]
This also gives us an expression for ρ t in terms of α t :
[Equation]
which combined with Eq
(6) gives us
[Equation]
Finally, the optimal step-size sequence can be implemented via the following recursion
[Equation]
It is easy to verify that this is indeed a decreasing sequence.
If we initialize at w 0 = 0 , then our step sizes are given by
[Equation]
A. C ONVERGENCE ANALYSIS : DIMINISHING STEP SIZE
In the main text, our convergence results focused on the constant step size regime for simplicity of presentation and ease of comparison to prior works, as they often lack diminishing step size results for permutation-based SGD variants.
Here we formally present the convergence rate under our main
Assumption 2 using a diminishing step size sequence.
In Theorems 3 and 4 that we present below, the rate is of the same order as what we obtained under a constant step size (Theorems 1 and 2) up to a factor of log( T ) that typically arises when using a diminishing step size.
Our analysis for the diminishing step size setting is accomplished by breaking the total number of iterations into phases.
Suppose we want to run a total of T iterations of SGD updates.
We will break the analysis into I number of phases.
In each phase i = 0 , . . .
I − 1 , we run K ( i ) m
( i ) number of iterations with constant step size α ( i ) .
The step size is decayed at the beginning of each phase.
For this, we deﬁne the triplets ( α ( i ) , m ( i ) , K ( i ) ) , where (cid:8) m ( i ) (cid:9) is an increasing sequence.
Recall that γ and Φ are constants in Assumption 2.
In the non-convex case, the inner interval length is chosen to be
[Equation]
where as for the strongly-convex setting it is
[Equation]
For both function classes the diminishing step-size is set to
[Equation]
A.2. D IMINISHING STEP SIZE : NON - CONVEX CASE
Theorem 3 (Diminishing step size: non-convex case) .
Suppose that our setup satisﬁes Assumptionsand 2, and let (cid:15) > 0 be any target error.
After T iterations of SGD (Eq.
(1) ) with a diminishing step-size (Eq.
, we can obtain
[Equation]
This implies the number of gradient evaluations to achieve min t (cid:107)∇ f ( w t ) (cid:107) 2 ≤ (cid:15) 2 is
[Equation]
Proof.
To apply Lemma 1 to the iterations within a particular phase i , we need
[Equation]
where we have temporarily dropped the index ( i ) for convenience.
Lemma 1 gives us the following bound on the objective value between any length- m number of iterations within that phase: for τ ∈ { 0 , m − 1 , 2 m − 1 , . . .
, ( K − 1) m − 1 } ,
[Equation]
since the step size is constant throughout one phase.
Summing over K such intervals from τ =followed by a telescope,
[Equation]
We now restore our phase index,
[Equation]
where we have chosen the largest possible α ( i ) = 1 / 6 Lm ( i ) for all i .
Summing over I phases and letting ∆ :
= f ( w 0 ) − f ∗ ,
[Equation]
where the left hand side can be further lower bounded with
I − 1 (cid:88)
[Equation]
=
[Equation]
using the inclusion that (cid:8) 0 , m ( i ) − 1 , . .
, K ( i ) m
( i ) − 1 (cid:9) I − 1 i =
⊂ { 0 , . .
, T − 1 } .
Now let us choose
[Equation]
Here the choice of m ( i ) guarantees that
[Equation]
Next, observe that for the last term in the previous sum,
[Equation]
Combining with the above,
[Equation]
Furthermore, using our choice of α ( i )
I
[Equation] [Equation]
Re-arranging Eq.
[Equation]
[Equation]
The total number of iterations is given by
[Equation]
Solving for I and using lg to denote log 2 , we obtain
I
[Equation]
Moreover
[Equation]
we can also upper bound T using similar arguments,
[Equation]
which gives
[Equation]
We are left to bound
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
which yields our convergence rate in the nonconvex setting.
A.2. D IMINISHING STEP SIZE : STRONGLY - CONVEX CASE
Theorem 4 (Diminishing step size: strongly-convex case) .
Suppose f is µ -Polyak-Łojasiewicz (PL), and that our setup satisﬁes Assumptions 1 and 2 Let (cid:15) > 0 be any target error, and κ = L / µ be the condition number of the problem.
After T iterations of SGD (Eq.
(1) ) with a diminishing step-size (Eq.
, we can obtain
[Equation]
This implies the number of gradient evaluations to achieve f ( w T ) − f ∗ ≤ (cid:15) 2 is
[Equation]
γ
[Equation]
Proof.
We will begin with the same analysis technique as used in the non-convex case.
To apply
Lemma 1 to the iterations within a particular phase i , we need
[Equation]
dropping the index ( i ) for convenience and will re-introduce it later when appropriate.
Lemmagives us the following bound on the objective value between any length- m number of iterations within that phase: for τ ∈ { 0 , m − 1 , . . .
, ( K − 1) m − 1 } ,
[Equation]
since the step size is constant throughout one phase.
Strong-convexity (or the Polyak-Łojasiewicz (PL) inequality) of f implies (cid:107)∇ f ( w ) (cid:107) 2 ≥ 2 µ ( f ( w ) − f ∗ ) for all w ∈ R d .
Using this while subtracting f ∗ on both sides leads to
[Equation]
Applying this recursively K times gives
[Equation]
where we have used
[Equation]
We now restore the phase index.
Letting T i :=
(cid:80) i − 1 i (cid:48)
=
m ( i (cid:48) )
K ( i (cid:48) ) be the total number of iterations passed after i phases so
that T = T I
[Equation]
and ∆ t := f
[Equation]
w t )
− f ∗
[Equation]
Now let us choose
[Equation]
Here the choice of m ( i ) guarantees that
[Equation]
Using the constant K across all phases
[Equation]
our recursion can be simpliﬁed to
[Equation]
Since this holds for any i = 0 , . . .
[Equation]
I −
[Equation]
applying recursion over I phases yields,
[Equation]
where we deﬁned ρ := 1 / 12 κ .
Observe that for our choice of m ( i ) and K ,
[Equation]
which implies
[Equation]
and so
[Equation]
note that
[Equation]
This gives us
[Equation]
Substituting this into Eq.
[Equation]
For the last term,
[Equation]
Substituting these into Eq.
[Equation]
and using Eq.
(11) to bound (1 − ρ ) KI gives us the convergence rate for strongly-convex functions using a diminishing step size:
[Equation]
A. P ROOF FOR L EMMA
We now prove the lemma that bounds the evolution of SGD over a length- m analysis phase.
This lemma is the key to our convergence analyses in both the diminishing and constant step size regimes.
Before we proceed, we ﬁrst state and prove the following bound on the gradient error when scaled by a nonincreasing sequence.
Lemma 2.
Suppose Assumption 2 holds.
If { ρ t } is a deterministic, nonincreasing, and nonnegative sequence, then
[Equation]
Proof.
Let β t = ρ t − ρ t +1 for t ∈ { τ, τ + 1 , . .
, τ + m − 2 } , and let β τ + m − 1 =
ρ t + m − 1 .
Observe that these are all nonnegative, and
[Equation]
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) k
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
[Equation]
(cid:88) k
[Equation]
(cid:88) t =
[Equation] [Equation]
[Equation] [Equation]
[Equation]
)) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
[Table]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) τ + m − 1 (cid:88) t
=
τ
[Equation]
t (
[Equation]
f
[Equation]
t
[Equation]
−
[Equation]
f ( w
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
ρ 2 τ
[Equation]
+ m
[Equation]
(cid:88)
[Equation]
k
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) t
[Equation]
f ( w
[Equation]
x t ) − ∇ f ( w τ ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
=
ρ 2 τ τ + m −
[Equation]
(cid:88) k
= τ
[Equation]
k ρ
[Equation]
( k − τ
[Equation]
1) 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Applying Assumption 2 on the squared norm,
[Equation]
which is what we wanted to show.
We now recall the statement for Lemma 1.
In the main paper, we state the lemma with constant step size.
In the following proof, we prove it holds for non-decreasing step size, where constant step size will hold naturally as a special case.
For this, we deﬁne η := (cid:80) τ
+ m − 1 t = τ α t .
Lemma 1.
Suppose our setup satisﬁes Assumptions 1 and 2 and that we use a constant step size α For all timesteps τ ≥ 0 , let m > 0 be some integer such that 3 α τ m 1 − γ / 2 Φ ≤ η ≤ 16 L .
Then the objective at timestep τ + m is bounded by
[Equation]
Proof.
Let η and g denote
[Equation]
This means that w τ + m =
w τ − ηg.
From Assumption 1 ( L -Smoothness),
[Equation]
where the last inequality follows because we assumed ηL ≤ 1 / 6 ( < 1) .
Next, observe that
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
To bound the second term,
[Equation]
Combining the above,
[Equation]
Applying Lemma 2 on the ﬁrst term in Eq.
(13) gives
[Equation]
[Table] [Table]
[Equation]
where the second term in the last inequality follows from Lemma 2.
Continuing,
[Equation]
This implies that
[Equation]
And because by our assumption, α 2 τ m 2 − γ Φ 2 ≤ η 2 9 , it follows that
[Equation]
Since we also assumed η ≤ 16 L , it follows that 3 η 2 L 2 ≤ 112 , and so
[Equation]
which gives
[Equation]
So, putting this all together,
[Equation]
This is what we wanted to show
[Equation]
A. C ONVERGENCE ANALYSIS : CONSTANT STEP SIZE A.4. C ONSTANT STEP SIZE : NON - CONVEX CASE
Theorem 1 (Non-convex case) .
Suppose that our setup satisﬁes Assumptions 1 and 2, and let (cid:15) >be any target error.
Using SGD (1) with a constant step size α = 16 L (cid:6) (4 C/(cid:15)
+ 3Φ) 2 / γ (cid:7) − 1 , the number of steps T needed to achieve min t =0 , ··· ,T − 1 (cid:107)∇ f ( w t
[Equation]
2 ≤ (cid:15) 2 is at most
[Equation]
Proof.
To satisfy the requirements of Lemma 1 in the constant step size, constant m case, we need
[Equation]
This breaks apart into ﬁrst a constraint on only m :
[Equation]
(14) and then a constraint on α in terms of m : α ≤ 6 Lm.
In the nonconvex setting, we invoke Lemma 1 followed by summing up over K phases and telescoping,
[Equation]
which gives a rate of
K
[Equation]
If we want to minimize the right side, observe that we will always want to set α as large as possible,
i
[Equation]
e. set α = 1 / 6 Lm .
This gives
[Equation]
Now, for this to all be less than (cid:15) 2 , it sufﬁces for
[Equation]
2 and 8 m
[Equation]
The former occurs when
[Equation]
while the latter happens when
[Equation]
If we let ∆ =
f ( w 0 ) − f ∗ , using Eqs.
(14) and (15) and taking the minimum m and K required, we can bound the number of iterations as
[Equation]
So,
[Equation]
where we suppose that γ acts as a constant.
A.4. C ONSTANT STEP SIZE : STRONGLY - CONVEX CASE
Theorem 2.
Suppose that f satisﬁes the µ -PL condition and our setup satisﬁes Assumptions 1 and 2.
Let (cid:15) > 0 be any target error, and κ = L / µ be the condition number of the problem.
Using SGD (1) with a constant step size α = 16 L (cid:6) (8 C 2 / ( µ(cid:15) 2 ) + 9Φ 2 ) 1 / γ (cid:7) − 1 , the number of steps T needed to guarantee f ( w T ) − f ∗ ≤ (cid:15) 2 is at most
[Equation]
Proof.
The strongly convex setting has the same constraints on m and α as the non-convex setting.
Recall that µ -strong convexity or µ -PL of f implies that for all x , (cid:107)∇ f ( x ) (cid:107) 2 ≥ 2 µ ( f ( x ) − f ∗ ) .
Applying this to the result of Lemma 1 gives
[Equation]
which is equivalent to
[Equation]
Applying this recursively over K intervals of length m starting from τ = 0 ,
[Equation]
As in the non-convex case, it best advantages us to set α as large as possible.
Setting α = 1 / 6 Lm gives
[Equation]
For this to be less than (cid:15) 2 , it sufﬁces for each of these two terms to be less than (cid:15) 2 / 2 .
In the latter case, we would need
[Equation]
[Equation]
The former case holds when
[Equation]
where κ = L / µ .
Applying a ceiling, using the minimum requirement on K and m from Eq.
(16) above and Eq.
(14) required in Lemma 1, we can lower bound the total number of iterations as
[Equation]
Ignoring logarithmic terms gives
[Equation]
A. J USTIFICATIONS FOR A SSUMPTION 2 UNDER VARIOUS EXAMPLE ORDERINGS A.5. P ROOF FOR P ROPOSITION 1 (A RBITRARY PERMUTATION )
Proposition 1.
Let Assumptions 1 and 3 hold, and suppose that we are using a permutation-based method of sampling, that is, any method such that x kn , x kn +1 , . . .
, x kn + n − 1 is a permutation of x (0) , x (1) , . . .
, x ( n − 1) for all epochs k ≥ 0 .
Any such method satisﬁes Assumption 2 with γ = 2 ,
C 2 = n 2 A 2 and Φ 2 = n 2 B 2 .
Proof.
This result follows trivially from Assumption 3.
Consider an arbitrary sum of gradient errors going from τ to τ + m − 1 :
[Equation]
where σ ( t ) is the index into the training set given by the permutation σ used at time step t .
Since the interval { τ, τ + 1 , . . .
, τ + m − 1 } is arbitrary, this m can potentially be greater than n , the epoch size.
We can split this interval up as follows.
Let τ 1 be the ﬁrst epoch boundary in the interval, such that all t going from τ to τ 1 − 1 are within the same epoch as w τ , or else τ 1 = τ
+ m if there is no epoch boundary in the interval.
Let τ 2 be the last epoch boundary in the interval, such that all t going from τ 2 to τ + m − 1 are within a later epoch than w τ (it may be the case that τ 1 = τ 2 ).
Then the sum
[Equation]
must be zero, since this interval goes over full epochs.
This leaves us with
[Equation]
t
[Equation]
[Table]
[Equation]
where we have used the fact that the intervals { τ, . .
, τ 1 } and { τ 2 , τ + m − 1 } are of length at most (cid:98) n/ 2 (cid:99) (if such interval is of length greater than (cid:98) n/ 2 (cid:99) , then bounding its norm is equivalent to bounding the sums of the remaining terms, for which there are at most (cid:98) n/ 2 (cid:99) of them).
Therefore
[Equation]
[Equation]
and so Assumption 2 is satisﬁed with γ = 2 , C 2 = n 2 A 2 , and Φ 2 = n 2 B 2 .
A.5. P ROOF FOR P ROPOSITION 2 (S HUFFLE ONCE )
Proposition 2.
Suppose that we are using the shufﬂe once variant of SGD to learn over a region
B ∈ R d of radius at most R , such that the iterates w t are guaranteed to remain within this region.
Assume that for all w ∈ B and all examples x in the training set of size n , Assumption 1 ( LSmoothness) and Assumption 3 hold.
Then with probability at least 1 − p , Assumption 2 holds with γ = 2 , C 2 =
˜ O ( dA 2 ( n + B 2 )) , and Φ 2 =
˜ O ( ndB 2 ) .
Proof.
We begin by invoking Lemma 6 (Permuted vector Hoeffding inequality) with
[Equation]
where we use x i,j to denote the j -th example after time step i , for all i ∈ { 1 , . .
m } , j ∈ { 1 , . . . , n } , with n ≥ m > 0 .
Note that here we are using a ﬁxed w ∈ B that does not depend on the permutation σ used in shufﬂe once.
Clearly, (cid:80) nj =1 X
i,j = 0 for all i due to the periodicity in shufﬂe once, and (cid:107) X
i,j (cid:107) ≤ 1 by our assumption.
Therefore the requirements of Lemma 6 are satisﬁed, which gives the following high probability bound for some (cid:15) > 0 on any sequence drawn without replacement:
[Equation]
Using the periodicity property of shufﬂe once, i.e. if σ is the drawn permutation at the beginning of training then σ ( t + n ) =
σ ( t ) , which allows us to arbitrarily shift the starting time step in the above:
[Equation]
Consider an arbitrary interval of length m out of the permutation.
As all of the n ( n − 1) / 2 intervals are equivalent to an interval of size at most (cid:98) n / 2 (cid:99) , by the sum to 0 property, we only need to consider intervals with m ≤ n / 2 .
This gives us
P
[Equation]
By a union bound across all intervals we have
P 
[Equation]
Unfortunately, we cannot take a union bound over R n or B (or even just over { w 0 , . . .
, w T } ), since the set may contain iterates visited by shufﬂe once that depend on σ , and thus Lemma 6 may not hold.
Instead, suppose that we cover the whole region B with balls of radius δ .
We will be able to do this with (1 + 2 R / δ ) d balls by Lemma 7 ( (cid:15) -net lemma).
So, if the centers of the balls form a set W , then
[Equation]
Note that the centers we use to cover the region is completely independent of the running algorithm, therefore the union bound over Lemma 6 can be applied.
Next, consider some w τ that is not necessarily the center of a ball.
The function
[Equation]
is Ln -Lipschitz continuous, because each of the components is L -Lipschitz, and we can sum up only at most n / 2 of them.
Let ˆ w ∈ W be such that (cid:107) w τ − ˆ w (cid:107) ≤ δ .
Adding and subtracting ∇ f ( ˆ w ; σ ( i ) ) and ∇ f ( ˆ w ) , applying triangle inequality followed by Lipschitz continuity, we have
[Equation]
[Table]
Adding and
∇ τ from
) and bound using and Lipschitz continuity again,
[Equation]
and consequently
[Equation]
Note that this inequality will fail to hold with probability at most
[Equation]
If we want this to fail with probability less than some p , it sufﬁces to set
[Equation]
R δ (cid:19)(cid:19) .
This gives
[Equation]
If we set δ such that L 2 nδ 2 =
A 2 , then we can bound this with
[Equation]
which gives
[Equation]
m
[Equation]
t
[Equation]
Hiding logarithmic terms involving n, p, R, L, A , the shufﬂe once setup satisﬁes the requirement of
Assumption 2 that with probability at least 1 − p ,
[Equation]
, and so the parameters are γ = 2 , C 2 =
˜ O ( dA 2 ( n + B 2 )) , and Φ 2 =
˜ O ( ndB 2 ) .
A.5. P ROOF FOR P ROPOSITION 3 (R ANDOM R ESHUFFLING )
We ﬁrst introduce a lemma that bounds the distance between iterates obtained from SGD with random permutation and that obtained from deterministic gradient descent, over one epoch.
Lemma 3.
Consider a single epoch of SGD with constant step size α > 0 , where the examples come from a random permutation over the training set.
Let Assumption 1 ( L -Smoothness) and the bounded gradient error Assumption 3 be satisﬁed.
Without loss of generality, assume that the epoch starts at time t = 0 at w 0 .
Let the sequence u t be deﬁned by u 0 = w 0 and
[Equation]
while
[Equation]
for some x t chosen from the permutation.
With probability at least (1 − δ ) it will hold that for all
T ∈ { 1 , . . . , n } , if we set
[Equation]
min
[Equation] [Equation]
[Equation]
then
[Equation]
Proof.
Given our setup, observe that we can write the difference between these sequences as
[Equation]
such that summing this up and using w 0 = u 0 gives
[Equation]
T − u T = − α
T − 1 (cid:88) t
= ( ∇ f ( w t ; x t ) − ∇ f ( u t ; x t )) − α
T
[Equation]
(cid:88) t = ( ∇ f ( u t ; x t ) − ∇ f ( u t )) , and so (cid:107) w T
[Equation]
T
[Equation]
t =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
[Equation]
T
[Equation] [Equation]
Recall the update rule of u t , with Assumption 1 ( L -Smoothness), we obtain (cid:107)∇ f ( u t +1 ) − ∇ f ( u t ) (cid:107) ≤ αL (cid:107)∇ f ( u t ) (cid:107) .
By the reverse triangle inequality, this implies
[Equation]
, which further implies for any T ∈ { 1 , · · · , n } and t ∈ { 0 , · · · , T − 1 } , (cid:107)∇ f ( u t ) (cid:107) ≤ (1 − αL ) − n (cid:107)∇ f ( u T ) (cid:107)
=
(cid:18) 1 + αL 1 − αL (cid:19) n (cid:107)∇ f ( u T ) (cid:107) ≤ exp (cid:18) αLn 1 − αL (cid:19) (cid:107)∇ f ( u T ) (cid:107) ≤ exp (2 αLn ) (cid:107)∇ f ( u T ) (cid:107) ≤ e (cid:107)∇ f ( u T ) (cid:107) , while for any t ∈ { T + 1 , · · · , n } ,
[Equation]
where we apply the condition that α ≤ 1 / (2 nL ) .
For a given T
[Equation]
consider the following vector set
[Equation]
it can be easily veriﬁed that they sum to zero.
Now we apply Lemma 6 (Permuted vector Hoeffding inequality) , for any γ ≥ 0 ,
[Equation]
as T ≤ n .
Now, this holds for just one T .
By a union bound,
[Equation]
where Q T = (cid:112)
A 2 + B 2 e 2 (cid:107)∇ f ( u T ) (cid:107) 2 .
Now, if we set γ such that
[Equation]
then we get that
P (cid:32)
[Equation]
In this case, we will have that with probability at least (1 − δ ) , (cid:107) w T − u T (cid:107) ≤ αL
T −
[Equation]
t = (cid:107) w
[Equation]
− u t (cid:107) + αγQ T .
If we let ρ 0 = 0 and ρ T = αL
T − 1 (cid:88) t
= ρ t + αγQ T , then (cid:107) w T − u T (cid:107) ≤ ρ T .
Here, if T > 0 , ρ T +1 − ρ T = αLρ T + αγ ( Q T +1 − Q T ) , on the other hand, obviously ρ 1 = αγQ 1 , ρ T = αγ
T − 1 (cid:88) k
= (1 + αL ) T − k − 1 ( Q k +1 − Q k ) =
αγ (cid:32) T (cid:88) k
= (1 + αL ) T − k Q k −
T − 1 (cid:88) k
= (1 + αL ) T − k − 1 Q k (cid:33) ≤ αγ (cid:32) (1 + αL )
T − 1 (cid:88) k
= (1 + αL ) T − k − 1 Q k −
T − 1 (cid:88) k
= (1 + αL ) T − k − 1 Q k + Q T (cid:33) ≤ αγ (cid:32) αLe 1 / 2 T − 1 (cid:88) k
=
Q k +
[Equation]
T (cid:33) ≤
[Equation]
[Equation] [Equation]
T =
[Equation] [Equation] [Equation]
≤
[Equation] [Equation]
Be
[Equation]
f
[Equation] [Equation]
Put it back we obtain
[Equation]
which gives
[Equation]
If we require
[Equation]
Squaring and substituting the value of γ gives
[Equation]
with probability at least (1 − δ ) .
We now provide justiﬁcations to Assumption 2 for the random reshufﬂing scheme.
Proposition 3.
Suppose that we are using the random reshufﬂing variant of SGD.
Assume that for all w ∈ R d and all examples, Assumption 1 ( L -Smoothness) and Assumption 3 hold.
For some p ∈ (0 , 1) , set the constant step size to satisfy α ≤ (cid:0) max (cid:8) 1460 BnL · log (cid:0) 4 e 2 T/p (cid:1) , 2 nL (cid:9)(cid:1) − 1 .
Then with probability at least 1 − p , Assumption 2 holds with γ = 2 , C 2 =
˜ O ( nA 2 ) and Φ 2 =
˜ O ( nB 2 ) .
Proof.
For some γ > 0 (but different from the γ of Lemma 3, consider the event that for some speciﬁc epoch k , for some w τ ∈ { w 0 , w 1 , . . .
, w n ( k − 1) } ∪ { u 0 , u 1 , . .
. , u n } , where the u i are the u from Lemma 3 for epoch k , and for some τ and m k such that n ( k − 1) ≤ τ < τ + m k ≤ nk ,
[Equation]
Since all the x σ ( t ) are from the k -th epoch, but w τ is independent of any randomness in the k -th epoch (as it is either a point visited in a previous epoch, or a value from the u sequence which depends only on the position at the start of the k -th epoch and not on any k -th epoch randomness), it follows that we can apply Lemma 6 (Permuted vector Hoeffding inequality) on either this sum or, alternatively, the terms from epoch k but not in the sum (the terms left out) to get that
[Equation]
As m k ≤ n , it follows that
[Equation]
Now
[Equation]
by a union bound the probability that there exists some τ , w τ
[Equation]
and m k such that the average gradient error is large is bounded by
[Equation]
where T is the total number of iterations across all epochs (and we assume that we ﬁnish all epochs so n divides T ). This follows from the fact that there are at most T such τ that we could take on, at most T values of w τ that can be adopted for each, and at most n values m k can take on.
If we set γ such that
[Equation]
then with probability at least (1 − p / 2 ) , it will hold that for all epochs, w τ , τ , and m k , [Equation]
[Equation]
Additionally, set the δ in Lemma 3 to be pn / 2 T .
By a union bound on the result of Lemma 3, across all T / n epochs, with probability at least (1 − p / 2 ) , it must follow that
[Equation]
for the corresponding u t sequence for all epochs.
Therefore, both of these inequalities Eqs.
(18) and (19) hold together with probability at least (1 − p ) .
Now, consider an arbitrary sum of gradient errors going from τ to τ + m − 1 .
Note since the interval here is arbitrary, this m can be different from m k and potentially greater than n .
We can split this interval up as follows.
Let τ 1 be the ﬁrst epoch boundary in the interval, such that all t going from τ to τ 1 − 1 are within the same epoch as w τ , or else τ 1 = τ
+ m if there is no epoch boundary in the interval.
Let τ 2 be the last epoch boundary in the interval, such that all t going from τ 2 to τ + m −are within a later epoch than w τ (it may be the case that τ 1 = τ 2 ).
It follows that
[Equation]
Observe that since the second of these sums must go over some number of full epochs, its value must be 0 .
Therefore,
[Equation]
Observe that the term in the ﬁrst sum must be nL / 2 -Lipschitz continuous in w , because it can be written as the sum of at most (cid:98) n / 2 (cid:99) terms each of which is L -Lipschitz (either as the actual terms of the sum, or else the terms left out of the sum).
So, add and subtract ∇ f
( u τ ; x σ ( t ) ) and ∇ f ( u τ ) , (cid:13)(cid:13)
[Equation]
τ + m − 1 (cid:88)
[Equation]
[Table]
[Table]
[Table]
[Table]
[Equation]
Squaring both sides for simplicity, we can bound this with
[Equation]
where we have also used Eq.
(17) for γ .
If we apply our requirement that αLn ≤ 1 /
[Table]
we get
[Equation]
It follows that for random reshufﬂing with probability 1 − p , if we set
[Equation]
random reshufﬂing satisﬁes the requirement of Assumption 2 with γ = 2 , C 2 =
˜ O ( nA 2 )
[Equation]
and Φ 2 = ˜
O
( nB 2 ) , where ˜ O ( · ) hides logarithmic terms in n, T, p .
A.5. P ROOF FOR P ROPOSITION 4 (R ANDOM RESHUFFLING WITH DATA ECHOING )
The proof for Proposition 4 is nearly a repeat of the random reshufﬂing proof in Proposition 3.
The trick is to re-deﬁne an epoch when data echoing is used.
Instead of referring to an epoch as a random permutation of the n examples in the training set as in vanilla random reshufﬂing, here we deﬁne the cn samples as an epoch, where each example σ ( i ) is repeated c times.
For instance, let c = 3 , then the k -th epoch with permutation σ k is given by the sequence .
[Equation] [Equation]
[Table]
As a result of this redeﬁnition, Lemma 3 needs to be modiﬁed such that wherever n appears, we now need cn .
We omit the proof for the following lemma as it is a trivial adaptation from that of Lemma 3.
Lemma 4. Consider a single epoch of SGD with constant step size α > 0 , where the examples come from a random permutation over the training set, each echoed c times.
Without loss of generality, assume that the epoch starts at time t = 0 at w 0 .
Let the sequence u t be deﬁned by u 0 = w 0 and u t +1 =
u t − α ∇ f ( u t ) , while
[Equation]
for some x t chosen from the permutation.
Under the same assumptions as Lemma 3, with probability at least (1 − δ ) it will hold that for all T ∈ { 1 , . . .
, cn } ,
[Equation]
The justiﬁcations for Assumption 2 for the random reshufﬂing with data echoing scheme can also be obtained from Appendix A.5.3 similarly.
At appropriate places one should invoke Lemma 4 instead of Lemma 3, and replace n with cn .
We omit the proof as it is again a trivial modiﬁcation.
A.5. P ROOF FOR P ROPOSITION 5 (M ARKOV C HAIN G RADIENT D ESCENT (MCGD))
To justify Assumption 2 for MCGD, we will ﬁrst need the following lemma.
Lemma 5.
Let F be any vector-valued measurable function, and let x 0 , x 1 , . .
. be a sequence of samples from a Markov chain with mixing time t mix and stationary distribution π starting from an arbitrary initial distribution.
If the function is constrained such that (cid:107) F ( x ) (cid:107) ≤ 1 for all x , and if we also have E X ∼ π
[ F ( X )]
= 0 , then for any δ ∈ (0 , 1) ,
[Equation]
≤ δ
[Equation]
Proof.
Consider the Doob martingale
[Equation]
, where F k contains all randomness up to timestep k , i.e. x 0 , x 1 , . . .
, x k , and so (as usual for a Doob martingale) the martingale property is trivially satisﬁed using repeated conditioning:
[Equation]
Observe that the sum we want is W m = E (cid:104)(cid:80) m − 1 i =
F ( x i ) | F m (cid:105) =
(cid:80) m − 1 i =
F ( x i ) , and that this sum has increments
[Equation]
Now, observe that since the mixing time of the Markov chain is t mix , for any i ≥ k , if µ denotes the distribution of x i conditioned on F k , then using results from Levin & Peres (2017, Section 4.5)
[Equation]
It follows from this and the fact that F is bounded that
[Equation]
Therefore,
[Equation]
and similarly for the last term.
It follows that
[Equation]
Therefore
[Equation]
by the vector Azuma’s inequality of Hayes (2005, Theorem 1.8)
[Equation]
for any a > 0 ,
[Equation]
On the other hand
[Equation]
by the same reasoning as before,
[Equation]
≤
[Equation]
So,
P
[Equation]
Now, setting a such that
[Equation]
and noting that this makes a ≥ 2 , we get
[Equation]
which is what we wanted to show.
The justiﬁcation for Assumption 2 then follows straightforwardly.
Proposition 5.
Suppose that we use samples x t from a Markov chain with mixing time t mix .
Assume that for all w ∈ R and all examples x t , Assumption 3 holds.
Then with probability at least 1 − p ,
Assumption 2 holds with γ = 1 , C 2 =
˜ O
( A 2 t 2 mix ) , and Φ 2 =
˜ O
( B 2 t 2 mix ) .
Proof.
Observe that we need Lemma 5 to hold for all subintervals of examples, of which there are only at most T 2 .
So, setting δ to be p / T 2 in Lemma 5, we can show that for all τ, m , the probability
[Table]
is at least 1 − p .
Equivalently,
[Equation]
It follows that MCGD satisﬁes the requirements of Assumption 2 with γ = 1 ,
[Equation]
The big- ˜ O expressions in the proposition statement immediately follow.
A.5. J USTIFICATION FOR QMC- BASED DATA AUGMENTATION WITH RANDOM RESHUFFLING
Before we begin with the proof, let us ﬁrst present the introductory material necessary on quasi-Monte
Carlo methods.
Similar to Monte Carlo integration, QMC is also used for numerical integration but using low-discrepancy sequences instead of pseudorandom number sequences.
Concretely, the problem is to approximate the integral of a function f over some s -dimensional hypercube [0 , 1] s ,
[Equation]
using the average of the function evaluated at a sequence of points x 1 , . . . , x m , [Equation] The approximation error rate is deﬁned as (cid:15) =
| I ( f ) − I m ( f ) | , and it is well-known that in the case of Monte Carlo integration where the x i ’s are drawn uniformly at random from [0 , 1] s , the error rate is (cid:15) 2 =
O ( 1 / m ) .
If the sequence of x i ’s has low star-discrepancy , which is deﬁned as
[Equation]
where U =
(cid:81) sj =1 [0 , b i ] for b i ∈ [0 , 1) , and the volume is measured using the s -dimensional Lebesgue measure.
Intuitively, the smaller this quantity is the more evenly the sequence of points covers the space.
Some popular low-discrepancy sequences include the Halton sequence, Sobol sequence, van der Corput sequence, etc., which are essentially deterministic sequences that are cleverly constructed to mimic random numbers but in fact have low star-discrepancy.
For instance, the Halton sequence satisﬁes D ∗ m =
O ( (log m ) s / m ) .
To ensure fast convergence of I m to I as we increase m , in addition to using low discrepency sequences we also need f to be relatively well-behaved.
For this, the Hardy-Krause variation V HK is often used, for which we refer the reader to Drmota & Tichy (2006, Deﬁnition 1.13) for its detailed characterization.
Most importantly, if f has ﬁnite V HK on [0 , 1] s , then the Koksma-Hlawka inequality guarantees that
[Equation]
{+This implies that using quasi-Monte Carlo with, for instance, the Halton sequence, to integrate a function f with bounded Hardy-Krause variation, the error rate would be (cid:15) 
+}
= O ( (log m ) 2 s / m 2 ) .
In comparison to the Monte Carlo error rate, the QMC error rate can be much faster when the dimensionality s is relatively small.
For an in-depth exposition of the related materials we recommend
Owen (2003) and Drmota & Tichy (2006).
We are now ready to restate our proposition from Section 5, in which we describe our QMC data augmentation setup.
Proposition 6.
Suppose that we are using the random reshufﬂing variant of SGD with QMC data augmentation as described.
Assume that for all w ∈ R d and all examples, Assumptions 1 , 3, 4 and 5 hold for Equation 3.
For some p ∈ (0 , 1) , set the step size to be a constant such that α ≤ (cid:0) max { 1460 BnL · log(4 e 2 T/p (cid:1) , 2 nL } ) − 1 .
Then with probability at least 1 − p , Assumptionholds with γ = 2 , C 2 = ˜ O
( n 2 V 2 C 2 QMC log( T ) 2 s + nA 2 ) and Φ 2 = ˜
O ( nB 2 ) .
Proof.
Recall that the example used in the t -th iteration is being transformed as
[Equation]
We start from the average gradient error term, that is,
[Equation]
where the ﬁrst step follows from the deﬁnition of f in the data augmentation setup (Eq. (3)).
In the second step, the ﬁrst norm relates to the QMC approximation error while the second norm relates to the RR analysis.
From the analysis in Section A.5.
we know the second norm can be bounded asymptotically (and probabilistically) by ˜ O
( nA 2 ) + ˜ O
nB 2 ) (cid:107)∇ f ( w τ ) (cid:107) 2 /m 2 .
Now we analyze the ﬁrst term.
Since we use a contiguous QMC subsequence on each individual example, for the period of [ τ, τ + m − 1] , we deﬁne τ j and m j as the starting point and length of example x ( j ) being chosen during this period, such that (cid:80) nj =1 m j = m .
With this notation, we can now rewrite the ﬁrst norm as
[Equation]
m
[Equation]
[Table]
[Equation]
Now we can use the Koksma–Hlawka inequality (Aistleitner & Dick, 2014) to bound these norms, which gives us
[Equation]
Putting it together
[Equation]
we have shown that random reshufﬂing with QMC data augmentation satisﬁes
Assumption 2 with γ = 2 , C 2 =
˜ O (cid:0) n 2 C 2 QMC V 2 log( T ) 2 s + nA 2 (cid:1) [Equation] and Φ 2 = ˜
O (cid:0) nB 2 (cid:1) .
A. M ISCELLANEOUS LEMMAS
We now collect some technical lemmas used in our analyses.
The following lemma is a Hoeffding-type concentration bound on the sums of random permutations of vectors.
This lemma is particularly useful for simplifying the logic of the proofs of our shufﬂing propositions above, as it frees us from having to use Doob-martingale-type arguments throughout.
Lemma 6 (Permuted vector Hoeffding inequality) .
Let n ≥ m > 0 be some integers, and let X i,j for i ∈ { 1 , . .
, m } and j ∈ { 1 , . . .
, n } be vectors in R d .
We also require for all i, j , (cid:107) X
i,j (cid:107) ≤ 1 , and that for all i ,
[Equation]
Then, if σ : { 1 , . . . , n } → { 1 , . .
. , n } is a random permutation and that the individual X i,j ’s do not depend on σ ,
[Equation]
Note that this lemma trivially also applies to random subsamples of vector sums, by just letting
X i,j = Y j for the desired vector sequence Y j .
Proof.
This proof is adapted from the proof of Theorem 4.3 in Bercu et al.
Their approach is more sophisticated and tends to a Bernstein-type inequality, but is specialized to only scalars.
Consider sampling the elements of the permutation one at a time, and let F j be the ﬁltration containing the random variables σ (1) , σ (2) , . . .
, σ ( j ) but not σ ( j + 1) , . . .
, σ n .
For k ≤ n , deﬁne the process
[Equation]
Observe that this must be a (vector) martingale process, as it is a Doob martingale.
Explicitly, we can write it as
[Equation]
where here we used the fact that the X i,j ’s sum to 0 over j = [ n ] .
Thus, the increments of this process will have
[Equation]
where we have repeatedly applied the martingale property.
Now to bound all of these, ﬁrst by our assumption
[Equation]
Also,
[Equation]
Next, again using the sum to 0 assumption, we have
[Equation]
and ﬁnally, by a combination of these bounds,
[Equation] [Equation]
It follows that the increment satisﬁes
[Equation]
with probability 1 .
We can now apply the vector Azuma’s inequality of Hayes (2005, Theorem 1.8).
Applying this to W m gives the result that, for any a ,
[Equation]
This proves the lemma.
The next lemma we state is a standard result showing that we can cover a region of radius R with a bounded number of balls of radius (cid:15) .
Lemma 7 ( (cid:15) -net lemma) .
For any region D of radius R in d -dimensional space, and any (cid:15) > 0 , there exists a subset S ⊆ D such that S is of size at most
[Equation]
and for every point x ∈ D , there exists a ˆ x ∈ S such that (cid:107) x − ˆ x (cid:107) ≤ (cid:15) .
Proof.
Consider the following procedure.
As long as there are points in D that are not within a distance (cid:15) of an existing point in S , choose one such point arbitrarily and add it to S .
Observe that with this construction, any two points in S must be at a distance greater than (cid:15) from each other.
This means that if we center a ball with radius (cid:15)/ 2 at each of the points in S , these balls will be disjoint.
The total volume of these balls will be |S| · V 1 · ( (cid:15)/ 2) d , where V 1 is the volume of the unit ball in d -dimensional space.
However, the centers of all these balls must lie in D , and so the balls themselves must all lie within a slightly larger region of radius R + (cid:15)/ 2 .
This region will have volume
V 1 · ( R + (cid:15)/ 2) d .
This shows that our process must eventually stop, implying that S must exist, and gives us a bound on the size of S as
[Equation]
and the proof is complete.