Efficiency Ordering of Stochastic Gradient Descent
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the stochastic gradient descent (SGD) algorithm driven by a generalstochastic sequence, including i.i.d noise and random walk on an arbitrary graph,among others; and analyze it in the asymptotic sense.
Specifically, we employ thenotion of ‘efficiency ordering’, a well-analyzed tool for comparing the performanceof Markov Chain Monte Carlo (MCMC) samplers, for SGD algorithms in theform of Loewner ordering of covariance matrices associated with the scaled iterateerrors in the long term.
Using this ordering, we show that input sequences that aremore efficient for MCMC sampling also lead to smaller covariance of the errorsfor SGD algorithms in the limit.
This also suggests that an arbitrarily weighted
MSE of SGD iterates in the limit becomes smaller when driven by more efficientchains.
Our finding is of particular interest in applications such as decentralizedoptimization and swarm learning, where SGD is implemented in a random walkfashion on the underlying communication graph for cost issues and/or data privacy.
We demonstrate how certain non-Markovian processes, for which typical mixingtime based non-asymptotic bounds are intractable, can outperform their Markoviancounterparts in the sense of efficiency ordering for SGD.
We also show the utilityof our method by applying it to gradient descent with shuffling and mini-batchgradient descent, reaffirming key results from existing literature under a unifiedframework.
Introduction
Stochastic gradient descent (SGD) is widely used in machine learning, signal processing and otherengineering fields to solve the optimization problem
[Equation]
where Θ ⊂ R d is some closed and convex set and F ( · , i ) : R d → R for i ∈ [ n ] ≜ { 1 , · · · , n } aresmooth functions on Θ , not necessarily convex, such that their summation f : R d → R exhibits aminimizer θ ∗ ∈ Θ satisfying ∇ f ( θ ∗ ) =0 .
The update rule of the iterative SGD scheme is of the form
[Equation]
where γ t is the step size that can be constant or diminishing as t →∞ , Proj Θ is a projection operatoronto the constraint set Θ , and { X t } t ≥ 0 is some sequence taking values in [ n ] .
This sequence isoften generated in a stochastic manner, and samples can be drawn from temporally independentand identically distributed (i.i.d) random variables that are either uniformly distributed over [ n ][53, 48, 11], or leverage importance sampling techniques for variance reduction [46, 9, 24].
{ X t } t ≥ 0can also be constructed by repeatedly shuffling over all possible states without repetition, 1 leading tofaster convergence than stochastic counterparts drawing i.i.d samples from [ n ] [57, 3, 29, 66].
Random Walk Stochastic Gradient Descent (RWSGD): Some applications observe restrictedaccess to the state space, such as decentralized optimization [58, 64, 40], where communicationoccurs between nodes in a network to collaboratively solve the optimization problem (1).
For instance,disease classification in confidential clinical swarm learning [62] considers peer-to-peer networksdue to the highly private nature of medical data.
In such a setting, the random sequence { X t } t ≥ 0is usually realized as a Markov chain on a general graph G ( V , E ) that only samples local gradientsof the nodes in V ≜ [ n ] and traverses the network via edges connecting them without divulgingthe update history or its own gradient.
The randomness of the communication path ensures that thecompromised node can not easily leak the data of its neighbors [40].
Apart from the privacy concern, such dynamics are also employed in swarm learning/optimizationin robotics [14] and wireless sensor networks [38] due to their low communication cost and asynchronous nature.
The need for data privacy and demand for communication-efficient algorithms fordecentralized optimization has spurred the study of RWSGD algorithms in recent years [51, 31, 60],with the underlying Markov chain in the form of Metropolis-Hasting random walk (MHRW) [42].
Common analytical approach - Finite time bounds based on mixing time: Most of the existingworks analyzing iteration (2) provide so-called finite-time upper bounds on expected error in eitherthe objective function E [ f (˜ θ t ) − f ( θ ∗ )] , where ˜ θ t is some weighted average of the iterates, or itsgradient E [ ||∇ f ( θ t ) || 2 2 ] ; and are used to infer the convergence rate of the iterate sequence [51, 22, 60].
For diminishing step sizes γ t = t − α with α ∈ (0 . 5 , 1) , 2 the upper bound on E [ ∥∇ f ( θ t ) ∥ 22 ] reads as
[Equation]
, (3) and a similar form for E [ f (˜ θ t ) − f ( θ ∗ )] as well [60, 21, 6].
Here, β ∈ (0 , 1) is the second largesteigenvalue modulus (SLEM) of the underlying Markov chain’s transition matrix and is related toits mixing time property, since smaller SLEM leads to faster mixing of the Markov chain [15, 39].
On the other hand, M > 0 is usually a quantity proportional to the local gradients evaluated at theminimizer, or their upper bound.
Both the gradient information and the mixing time play a key rolein quantifying the convergence rate derived from this upper bound, and the mixing time is especiallyimportant since it hints that convergence rate of the SGD algorithm can potentially be acceleratedusing faster mixing Markov chains for the input driving sequence.
It has also been noted that theinherent correlation of the underlying random walk has to be addressed in any analysis concerning
Markov-chain-driven gradient descent [60].
The mixing time technique, by capturing the rate atwhich the chain converges to its stationary distribution [15, 39], is one way of doing so.
Alternative approach - Asymptotic analysis and efficiency ordering:
In addition to the aforementioned mixing time, another widely used metric for characterizing the second order propertiesof
Markov chains is the asymptotic variance (AV).
For any scalar valued function g : [ n ] → R ,the estimator ˆ µ t ( g ) ≜ 1 t (cid:80) ti =1 g
( X i ) , associated with an irreducible Markov chain { X t } t ≥ 0 withstationary distribution π , is the average of the samples of g ( · ) obtained along the chain’s sample pathup to time t > 0 .
The AV of the Markov chain, denoted by σ 2 X ( g ) , is then defined as the the limitingvariance of the estimator; that is,
[Equation]
For all functions g ( · ) satisfying E π ( g 2 ) < ∞ , the AV is associated with the Central Limit Theorem(CLT) for any Markovian kernel on a finite state space, as the variance of the normally distributedestimates in the limit [54, 32, 15].
More formally, we have
[Equation]
A smaller AV means that fewer samples are required post mixing of the chain 3 in order to obtain adesired accuracy - in some sense quantifying the chain’s efficiency .
Both the AV and the mixing time of a Markov chain are very strongly related concepts 4 .
In fact, the
AV has an upper bound in terms of the SLEM, which decreases as the SLEM gets smaller (chain mixes faster)
However, an ordering of the SLEM between two Markov chains does not implyan ordering of their AV, as we shall demonstrate later in Section 4 for a special case.
Both of thesesecond-order properties therefore lead to different notions of optimality; and the comparison of twochains based on their AV leads to the concept of efficiency ordering [44], where we say that a chain ismore efficient than the other if it has a smaller AV, uniformly over all functions g : [ n ] → R . [Figure]
As mentioned earlier, the common intuition asserted by finite time bounds such as (3) is that Markov chains with smaller SLEM lead to fasterconvergence of the SGD iteration (2) to the minimizer [60, 6].
We put this logic to test by simulating the RWSGD algorithm with three different reversible Markov chains (w.r.t.
uniformstationary distribution) as the stochastic inputsthe MHRW, a modification of MHRW, which isalso shown in Appendix I [1] to be more efficientthan MHRW, and the so-called ’fastest mixing Markov chain’ (FMMC) as defined in [13] asthe Markov chain obtained by minimizing the SLEM over the entire class of reversible chainsfor a given graph topology.
We employ RWSGDto minimize a quadratic objective function fortwo underlying graphs.
The exact details of thesetup are deferred to Appendix I [1], and ournumerical results in Figure 1 show that eventhough the FMMC is theoretically guaranteed to have the smallest SLEM ( β i for i ∈ { 1 , 2 , 3 } ) ofthe three reversible chains simulated, it is the worst performing one with largest mean square error(MSE).
Although MHRW and Modified-MHRW share the same SLEM in the lower plot of Figure1, they still have performance differences.
This is contradictory to the intuition derived from (3),and could be attributed to the finite time results providing upper bounds for all times t > 0 , whichmay therefore not necessarily be tight.
On the other hand, the performance of the chains seem to beordered according to their AV ( σ 2 i for i ∈{ 1 , 2 , 3 } ) evaluated for a test function.
This lends credenceto developing techniques based on AV, for judging the performance of different stochastic inputs for SGD, as possible alternatives to using SLEM as the sole performance metric.
The asymptotic variance also appears in the CLT for stochastic approximation (SA) algorithms [8, 18],though this time not directly as the variance in the limit, but as a component of the limiting covariancematrix of the scaled iterate errors.
Recent works [18, 45] point out that the covariance matrix itself isof special interest, and typically contains more information than the non-asymptotic MSE bounds[45].
In the sense of SGD algorithms, we will show in Section 3 that it embeds explicit informationof the exact vector-valued gradient evaluated at the optimizer as well as the entire spectrum of thetransition matrix; as opposed to the upper bound M of the gradients and only the second largesteigenvalue modulus commonly found in mixing time based non-asymptotic bounds.
It has beensuggested [17, 20], and also proved for the special case of linear SA [18], that the covariance matrixemerging out of the CLT dominates as a leading term of the finite-time MSE bounds.
This alsoholds true for finite-time bounds on weighted MSE for any preferred weight; the weighted MSEbeing utilized in fields such as wireless MIMO [61] and process optimization [27].
Overall, whilefinite-time bounds have enjoyed great success in the literature, the potential for performance gainsout of the asymptotic analysis of SGD algorithms have remained largely unexplored.
Contributions: We employ asymptotic analysis to propose a general framework that offers seamlessconnection between AV in the MCMC literature with efficiency ordering and covariance matrix in the
SGD algorithms.
Our framework can be used to design different random walk variants and also tosystematically compare the existing sampling methods in the SGD iteration (2) with diminishing stepsize, not just limited to random walks.
In particular, we show that any two random walks followingan efficiency ordering have their covariance matrices Loewner ordered, including non-Markovianstochastic processes versus its Markovian counterpart, which defies any mixing-time (SLEM) basedanalysis.
Such ordering can be harnessed into improving the accuracy of SGD iterates, which impliesa reduction in the weighted MSE with arbitrary weights.
Moreover, via a specific augmentationof the state space, we are able to analyze SGD for both single and random shuffling and show the efficiency of shuffling over i.i.d sampling for a set of objective functions that may not satisfy‘Polyak-Łojasiewicz inequality’.
We further extend such comparison to mini-batch SGD algorithms.
Lastly, we present numerical results where the efficiency ordering via asymptotic analysis tends tohold over all time periods and input sequences with higher efficiency have smaller errors in SGD.
Modeling setup
Basic notations: We use lower case, bold faced letters to denote vectors ( v ∈ R d ), and use uppercase, bold faced letters to denote matrices ( M “ ∈ R d × d ).
|| · || 2 denotes the l 2 norm for vectors or2 -norm for matrices.
We use ∇ f ( · ) as Jacobian matrix of vector-valued function f ( · ) , and ∇ 2 g ( · ) as
Hessian matrix of scalar-valued function g ( · ) .
We let ∇ g ( θ, X ) be the partial derivative of scalarvalued function g ( θ, X ) with respect to θ .
Loewner ordering of matrices is denote by ‘ ≤ L ’ suchthat A ≤ L B ⇐⇒ x T ( A − B ) x ≤ 0 for any x ∈ R d .
The term Tr ( A ) denotes the trace ofmatrix A , and let 1 {·} be the indicator function.
We write N (0 , V ) to represent a multivariate Gaussian distribution with zero mean and covariance matrix V .
For a connected and undirectedgraph G ( V , E ) with node set V and edge set E , we use N ( i ) for the set of neighbors of node i ∈ Vand d ≜ [ d 1 , d 2 , · · · , d n ] T for the degree vector where d i = | N ( i ) | .
SGD algorithm with arbitrary input sequence: We consider random walks { X t } t ≥ 0 for whichthe limit π i ≜ lim t →∞ 1 t (cid:80) tk =1
X k = i } exists almost surely and is positive for all i ∈ [ n ] , withπ =
[ π i ] i ∈ [ n ] denoting the limiting or stationary distribution.
This is trivially satisfied via strong lawof large numbers [23] when X t for each t > 0 are i.i.d random variables with distribution π over [ n ] ,and via the ergodic theorem [15] when { X t } t ≥ 0 is an irreducible, aperiodic and positive recurrent(ergodic)
Markov chain.
Note however that this way of defining the stationary distribution π allowsfor the input sequence { X t } t ≥ 0 to be more general, possibly being non-Markov on [ n ] .
Then, we canuse π to rewrite the objective in (1) as
[Equation]
where function G ( θ, i ) ≜ 1 nπ i F ( θ, i ) for any θ ∈ Θ , i ∈ [ n ] .
The generalized update rule then becomes
[Equation]
This change of notation allows us to consider input sequences having possibly non-uniform stationarydistributions, and is a version of importance sampling for RWSGD schemes, as in [6].
For example,the iteration (7) with the input sequence generated from a MHRW with uniform target distributionπ = 1 /n will reduce down to (2) with G ( θ, i )
=
F ( θ, i ) for all θ ∈ Θ , i ∈
[ n ] .
If the input sequenceis instead a simple random walk on a connected graph G ( V , E ) with V =
[ n ] , we have π ∝ d , and
G ( θ, i ) =
1 T d nd i F ( θ, i ) for all θ ∈ Θ , i ∈ V .
Asymptotic covariance matrix.
We now quickly review the multivariate CLT for Markov chains,since it is a natural way to introduce the asymptotic covariance matrix, used heavily throughoutthe paper.
For any finite, irreducible Markov chain { X t } t ≥ 0 with stationary distribution π , itsestimator is defined as ˆ µ t ( g ) ≜ 1 t (cid:80) tk =1 g ( X t ) for any vector-valued function g : [ n ] → R d .
Then, the ergodic theorem [15, 16] states that for any initial distribution and any g ( · ) such that
E π ( g ) =
(cid:80) i ∈ [ n ] g ( i ) π
i < ∞ , we have ˆ µ t ( g )
a.s. −−−→ t →∞ E π ( g ) .
Similarly to the asymptotic varianceσ 2 X ( g ) for a scalar-valued function g ( · ) , we can also define the asymptotic covariance matrix Σ X ( g )for vector-valued function g ( · ) ,
[Equation]
(8) where ∆ t ≜ (cid:80) ts =1 ( g ( X s ) − E π ( g )) .
The associated multivariate CLT is then given as follows.
Theorem 2.1 (Chapter 1 [16]) .
For any function g : [ n ] → R d that satisfies E π ( g 2 ) < ∞ , we have
[Equation]
In the next section, we will show how the the asymptotic covariance matrix Σ X ( · ) also appears aspart of the CLT result for SGD algorithms.
Efficiency Ordering of SGD Algorithms
In this section, we present our main result concerning the performance comparison of different SGDalgorithms to solve (1).
We first begin by stating our assumptions on the objective function and thestochastic input sequence, providing a CLT result for SGD algorithms, and analyzing the covariancematrix arising therein.
We then introduce the notion of efficiency ordering of Markov chains in thecontext of MCMC sampling, and form the connection with covariance matrices as our main result in
Theorem 3.6.
For the rest of this section we assume that the functions F ( · , i ) (possibly non-convex), the summandsof the objective function in (1), and the input process { X t } t ≥ 0 for the SGD iteration (7) satisfy:
(A1)
The step size is given by γ t = t − α for α ∈ (1 / 2 , 1] ;
(A2)
There exists a unique minimizer θ ∗ in the interior of the compact set Θ with ∇ f ( θ ∗ ) = 0 ,and matrix ∇ 2 f ( θ ∗ ) (resp.
∇ 2 f ( θ ∗ ) − I / 2 ) is positive definite for a ∈ (1 / 2 , 1) (resp.
a =1 );
(A3) Gradients are bounded in the compact set Θ , that is, sup θ
∈ Θ sup i ∈ [ n ] ||∇ F ( θ, i ) || 2 < ∞ ;
(A4)
For every z ∈ [ n ] , θ ∈ R d , the solution ˜ F ( θ, z ) ∈ R d of the Poisson equation ˜ F ( θ, z ) −E [ ˜ F ( θ, X t +1 ) | X t =
z ] = ∇ F ( θ, z ) −∇ f ( θ ) exists, and sup θ ∈ Θ ,z ∈ [ n ] ∥ ˜ F ( θ, z ) ∥ 2 < ∞ ; (A5)
The functions F ( θ, i ) are L -smooth for all i ∈ [ n ] , that is, ∀ θ 1 , θ 2 ∈ Θ , ∀ i ∈ [ n ] , we have ∥∇ F ( θ 1 , i ) − ∇ F ( θ 2 , i )
[Equation]
We then have the following CLT result for SGD algorithms.
Lemma 3.1.
For iterates { θ t } t ≥ 0 of the SGD algorithm (7) satisfying (A1)–(A5), we have
[Equation]
a.s.
[Equation] , and ( θ [Equation] where covariance matrix V X is the unique solution to the Lyapunov equation Σ X + KV X
+ V X K T
=0 when α ∈ (0 . 5 , 1) (resp.
Σ X + (cid:0) K
+ I 2 (cid:1) V X + V X
(cid:0) K
+ I 2 (cid:1) T = 0 )
when α = 1 ).
Here, Σ X ≜Σ X ( ∇ F ( θ ∗ , · )) is the asymptotic covariance matrix 6 as in (8) , and K ≜ ∇ 2 f ( θ ∗ ) .
Additionally, for the averaged iterates { ¯ θ t } t ≥ 0 where ¯ θ t ≜ 1 t (cid:80) t − 1 i =
θ t , we have
[Equation]
10) where V ′ X
=
K − 1 Σ X ( K − 1 ) T with the same matrices K and Σ X as in the non-averaged case.
Remark 3.2.
Lemma 3.1 is itself a special case of the more general CLT result for SA algorithmsprovided in Appendix A [1], and as proved in Appendix B [1].
Remark 3.3.
While (A2) may appear to be too strict at first, it can be relaxed to the setting ofthe objective function f ( · ) having multiple minimizers, by leveraging more general CLT resultsfrom SA literature, such as Theorem 2.1 in [25].
However, this comes at a cost of cumbersomenotation, requiring conditioning of iterates converging to one of the minimizers, potentially makingthe mathematical parts harder to follow.
We also show in Appendix C [1] that (A2) is no stricter thanthe Polyak-Łojasiewicz inequality – a popularly adopted weak assumption in recent SGD literaturestudying non-convex objective functions [33, 41, 63, 66].
Remark 3.4.
Assumptions (A3) and (A5) are widely seen in the RWSGD literature [51, 31, 60],while (A4) is automatically satisfied for any ergodic Markov chain (see [43, 18] for details), acommon assumption for the stochastic noise sequence [31, 22, 6].
The compactness in (A3) can alsobe relaxed, given assumptions on the objective function in [34], such that the estimator θ t generatedby Markov-driven sequences can still be ‘locked in’ a compact set after a sufficiently long time.
Lemma 3.1 implicitly indicates that the asymptotic convergence rate (in distribution) for θ t − θ ∗(resp.
¯ θ t − θ ∗ ) is O ( √ γ t ) (resp.
O (1 / √ t ) ).
While this does not necessarily translate to O ( √ γ t )convergence rate for E [ ∥ θ t − θ ∗ ∥ 2 ] ( O (1 / √ t ) for E [ ∥ ¯ θ t − θ ∗ ∥ 2 ] ), it has been suggested [17, 20], and is in fact true for cases such as quadratic objective functions since they satisfy the linear stochasticapproximation in [18], which is of the form
[Equation] [Equation]
for which the connection between finite-time MSE and covariance matrix V X has been established[18].
This is also true for arbitrarily weighted MSE, which can be obtained as a weighted sum ofdiagonal entries of the covariance matrices V X and V ′ X .
In addition to the apparent connection to MSE, the covariance matrix plays a wider role in SGDperformance.
Given any vector of weights w ∈ R d , from Lemma 3.1 we also have that the weightedsum of errors w T ( θ t − θ ∗ ) converges to zero almost surely, and that w T ( θ t − θ ∗ ) / √ γ t Dist −−−→ t →∞
N (0 , w T V X w ) .
This means that, for sufficiently large t , we can estimate
[Equation]
such that, for instance, the 95% confidence interval for w T θ t is approximately w T θ ∗ ± 2 (cid:112) γ t w T V X w .
In other words, smaller w T V X w leads to narrower confidence interval and higher accuracy.
Theform w T V X w for any vector w ∈ R d naturally implies that Loewner ordering should come into playwhen concerning the performance of SGD algorithms.
To proceed, we first employ the widely used notion of efficiency ordering of Markov chains.
Theefficiency of different chains is compared by ordering them using their respective AV as follows.
Definition 3.5 ( Efficiency Ordering [44]) .
For two random walks { X t } t ≥ 0 and { Y t } t ≥ 0 with thesame stationary distribution π , we say { X t } t ≥ 0 is more efficient than { Y t } t ≥ 0 , which we write as
X ≥ E Y , if and only if σ 2 X ( g ) ≤ σ 2 Y ( g ) for any g : [ n ] → R .
We are now ready to state our main result.
We first extend the efficiency ordering of Markov chainsby proving the equivalence of comparing their scalar-valued AVs, to comparing their asymptoticcovariance matrices via Loewner ordering.
We then use this extension to show that more efficientinputs { X t } t ≥ 0 (as in Definition 3.5) to the SGD algorithm lead to performance improvements in theform of smaller covariance matrices in the Loewner ordering sense.
Theorem 3.6.
Consider two SGD iterations with random walks { X t } t ≥ 0 and { Y t } t ≥ 0 as inputsequences, with the same stationary distribution π , satisfying (A1)–(A5).
Then,
(i) X ≥ E Y
if and only if Σ X ≤ L Σ Y ;
(ii)
If X ≥ E Y , then V X ≤ L V Y ( V ′ X ≤ L V ′ Y for the case of averaged iterates ) ;
where Σ X , V X , and V ′ X (resp.
Σ Y , V Y , and V ′ Y ) are the covariance matrices from Lemma 3.1,corresponding to { X t } t ≥ 0 (resp.
{ Y t } t ≥ 0 ) as the stochastic input sequence.
Theorem 3.6 enables us to provide a sense of efficiency ordering of SGD algorithms which aredriven by different stochastic inputs.
Since this is achieved via Loewner ordering, it also leads tosmaller confidence intervals in the long run as mentioned earlier, as well as potentially smaller MSE 7depending on the objective function.
Remark 3.7.
In addition to the CLT result for SGD algorithms with diminishing step size describedin Lemma 3.1, we include in Appendix E [1] similar results for constant step sizes and quadraticobjective functions, where the statement of Theorem 3.6 still holds.
Applications: Towards More Efficient SGD
In this section, we present some SGD variants and compare them in terms of efficiency orderingof SGD.
Specifically, we first show that a certain class of non-Markov random walks can providea better input sequence than its Markovian counterpart.
We then analyze shuffling-based gradient descent and compare it to the SGD with i.i.d input in terms of efficiency ordering for SGD algorithm.
We also extend our approach to a more general mini-batch version.
High-Order Efficient Random Walk for SGD: The simple random walk (SRW) is a popular
Markov chain that has been extensively studied in the literature [55, 52, 26].
Several recent workshave focused on the non-backtracking random walk (NBRW) on a connected undirected graph
G ( V , E ) in the MCMC literature, which is an extension of SRW with the same limiting distributionπ =
d / 1 T d [47, 5, 37, 35, 7].
Intuitively speaking, NBRW is a random walk that selects one of itsneighbors uniformly at random except the one it just came/transitioned from.
Specifically, the NBRW{ Y t } t ≥ 0 is a second-order non-reversible Markov chain (i.e., it is non-Markov on V =
[ n ] ) with itstransition probability given by
[Equation]
From the nature of same limiting distribution, NBRW can be used as the input for SGD iterations(7) with the same re-weighted local functions G ( θ ∗ , i ) as that of SRW for all i ∈ [ n ] whenever theapplications call for random-walk type of inputs.
Let Σ Y ( ∇ G ( θ ∗ , · )) be the asymptotic covariancematrix of this NBRW { Y t } t ≥ 0 , as defined in (8).
One of the main results in [37] concerns theefficiency ordering of NBRW and SRW.
They show that NBRW has a smaller AV, or equivalently,from our Theorem 3.6 (i), a smaller asymptotic covariance in terms of Loewner ordering.
Our nextresult forms the necessary connection between the asymptotic covariance matrix arising in the CLTresult and Σ Y ( ∇ G ( θ ∗ , · )) .
Proposition 4.1.
Consider the SGD iteration (7) with two input sequences SRW { X t } t ≥ 0 and NBRW{ Y t } t ≥ 0 respectively.
Then, both the respective estimators θ Xt , θ Yt a.s. −−−→ t →∞ θ ∗ , and V Y ≤ L V X , thatis, NBRW is more efficient than SRW in the SGD algorithm.
By augmenting the state space, we can represent NBRW as a Markov chain Z t =
( Y t − 1 , Y t ) ∈ V ×V ,as was done in [47, 37].
This transformation then allows us to build CLT for an SGD iteration with{ Z t } t ≥ 0 as the input.
The subtlety here is to prove that the asymptotic covariance matrix arisingout of the CLT with respected to the augmented process { Z t } t ≥ 0 is indeed equal to Σ Y ( ∇ G ( θ ∗ , · )) .
This is shown by cultivating the relationship between the stationary distribution of { Z t } t ≥ 0 on theaugmented state space V × V and { Y t } t ≥ 0 on the node space V , as provided in [47].
Thus, our Theorem 3.6 together with the existing works on efficiency ordering of NBRW versus SRWin the MCMC literature [47, 37] enable us to show that NBRW is a more efficient input sequencethan SRW for the SGD iteration (7).
Interestingly, it has been shown that non-backtracking walksmix faster when the underlying graph is d − regular [5].
In this case, a faster convergence rate is alsosuggested by mixing time based non-asymptotic bounds prevalent in RWSGD literature.
However,no such results concerning mixing time and SLEM exists for NBRW on a general graph.
Thus, in theform of Proposition 4.1, we demonstrate the utility of our approach in settings where mixing timebased comparisons are unavailable.
Shuffling versus i.i.d Input Sequence: Shuffling-based methods have been widely used in machinelearning applications [10].
They work by repeatedly passing over the entire state space [ n ] withoutrepetition, each complete pass forming an epoch .
Random shuffling and single shuffling are twoversions therein and differ in the order in which they pass over [ n ] .
Random shuffling, as thename suggests, makes the pass in a randomly chosen order in each epoch, while single shufflingmaintains the same predetermined order (often randomly chosen once at the beginning) for allepochs.
Shuffling-based methods are known to show better empirical performance than i.i.d input[12], although intense theoretical analysis for shuffling-based gradient descent has only emergedin recent years [59, 30, 57, 3, 29].
In what follows, we use our results from Section 3 to compareshuffling-based gradient descent to SGD with i.i.d input.
To do so, we first analyze the asymptoticcovariance matrix for shuffling-based methods.
Lemma 4.2.
Let the input process { X t } t ≥ 0 be single or random shuffling.
Then, for any vectorvalued function g : [ n ] → R d , Σ X ( g ) = 0 , where Σ X ( g ) is defined in (8) .
For i.i.d input sequence with distribution ˆ π , the asymptotic covariance from Lemma 3.1 reduces to
[Equation]
following its definition in (8), and thus, trivially, Σ X ( ∇ G ( θ ∗ , · )) ≥ L 0 .
Lemma 4.2 shows thatshuffling-based methods are more efficient than i.i.d input sequence due to a smaller asymptoticcovariance matrix in Loewner ordering.
Next, we show that they also outperform i.i.d input whenused for driving the input sequence of SGD algorithms.
Proposition 4.3.
Consider the SGD iteration (7) with stochastic inputs single/random shuffling{ X t } t ≥ 0 and i.i.d sampling { Y t } t ≥ 0 , we have θ Xt , θ Yt a.s. −−−→ t →∞ θ ∗ and V X = 0 ≤ L V Y .
Though it may seem so at first, Proposition 4.3 is not a simple application of Theorem 3.6, especially for random shuffling because it is hard to check if random shuffling, formulated as atime-inhomogeneous Markov chain, indeed satisfies (A4).
To overcome this difficulty, in Appendix
H [1] we come up with a non-trivial augmentation to a much higher dimensional state space ( [ n ] n +1 )to make random shuffling a time-homogeneous periodic Markov chain in order to show that bothsingle shuffling and random shuffling satisfy (A4) and thus apply Theorem 3.6.
The case of shuffling versus i.i.d inputs is an example of a setting where the sequence with larger
SLEM is more efficient than one with smaller SLEM 8 as an input sequence to the SGD iteration (7).
For quadratic objective functions that satisfy the linear SA iteration in [18], it also attains a fasterconvergence speed in terms of MSE than i.i.d inputs to SGD algorithms.
Although some recentworks provide more informative finite-time error bounds on the MSE of the objective function forshuffling-based methods, by studying a special case of the matrix norm AM-GM inequality andproving faster convergence rate than i.i.d inputs [50, 3, 29], our result is not a subset of theirs.
To beprecise, we show in Appendix C [1] that our assumption (A2) on the objective function is no lessgeneral than their most general setting based on the Polyak-Łojasiewicz inequality.
Mini-Batch Gradient Descent with Shuffling: Mini-batch gradient descent is another populargradient descent variant and is widely used in the machine learning tools [19, 2, 49] to accelerate thelearning process when compared to SGD.
Instead of sampling a single element, mini-batch gradientdescent samples multiple elements from [ n ] in each iteration that form a batch. To incorporate the notion of mini-batches in our SGD framework, we provide a reformulation of thegeneral SGD iteration based on a similar formulation in [28] for the general analysis of SGD withi.i.d inputs. Consider a stochastic process { B t } t ≥ 0 as the driving sequence, which randomly samplesbatches of size S (without replacement) from the state space [ n ] , that is B t ⊂ [ n ] and | B t | = S for allt ≥ 0 .
Here we assume [ n ] mod S = 0 for simplicity. B t will therefore refer to the batch chosen atany time t > 0 .
We assume that B t for all t > 0 are i.i.d random variables drawn from a distribution
P , such that P ( B ) > 0 is the probability with which a batch B ⊂ [ n ] is picked. We associate withany batch B , v ( B ) ≜ (cid:2)(cid:80) i ∈ B e i (cid:3) / (cid:0) NS (cid:1) P ( B ) , where e i is the i ’th vector of the canonical basis of R d .
We then denote F ( θ ) ≜
[ F ( θ, 1) , · · · , F ( θ, n )] T , and ∇ F ( θ ) ≜ [ ∇ F ( θ, 1) , · · · , ∇ F ( θ, n )] T for allθ ∈ Θ . With this notation, we can rewrite the general update rule for mini-batch SGD as [Equation] Note that this way of defining the mini-batch based random input ensures that E P [ F ( θ ) T v ( · )]
= f ( θ )for all θ ∈ Θ , maintaining the same objective function irrespective of the distribution from whichbatches are sampled. With X t = B t for all t ≥ 0 , and ∇ G ( θ t , X t +1 ) = ∇ F ( θ t ) T v ( B t +1 ) , the iteration (14) can still bewritten in the form of (7) with i.i.d input sequence { X t } t ≥ 0 .
We can thus apply the CLT for SGDalgorithms to the mini-batch SGD with i.i.d input, and in a similar fashion as (13) derive the explicitform of the asymptotic covariance matrix of (14), that is, [Equation] [Equation] In practice, mini-batch gradient descent with shuffling is more widely used than i.i.d sampling [2], inwhich B t is generated by shuffling-based method instead of independent drawn from a distribution.
At the beginning of each epoch, Mini-batch gradient descent with random shuffling shuffles the whole
The reformulation paper.
dataset [ n ] and split it into small batches.
On the other hand, mini-batch gradient descent with singleshuffling only shuffles the dataset [ n ] once before dividing it into batches, sticking to a predeterminedsequence of batches for all epochs of the training process.
As pointed out by [65], there is still agap between practical implementation and theoretical analysis for mini-batch gradient descent withshuffling.
Nevertheless, by extrapolating the analysis from Proposition 4.3, we are able to analyze theefficiency ordering of shuffling and i.i.d sampling in the mini-batch version, as stated next.
Proposition 4.4.
Consider the mini-batch gradient descent (14) with stochastic inputs single/randomshuffling { X t } t ≥ 0 and i.i.d sampling { Y t } t ≥ 0 , we have θ Xt , θ Yt a.s. −−−→ t →∞ θ ∗ and V X = 0 ≤ L V Y .
Proposition 4.4 generalizes Proposition 4.3 (special case with mini-batch of size S = 1 ) in that thesame efficiency ordering between shuffling and i.i.d input holds true even with mini-batches.
Numerical Experiments
In this section, we empirically validate our theoretical analysis.
We compare the efficiency orderingof the SGD algorithm for various stochastic inputs on two objective functions in (16), f ( θ ) beingstrongly convex and g ( θ ) being nonconvex.
[Equation]
For l 2 -regularized logistic regression ˜ f ( θ ) , we choose the dataset CIFAR-10 [36] where n is thetotal number of data points.
Here, x i ∈ R 108 is the vector flattened from the cropped image iwith shape (6 , 6 , 3) , and y i ∈ R is the label.
For sum-of-non-convex functions ˆ f ( θ ) , which isbased on the experiment setup in [28, 4], we random generate a diagonal matrix D i ∈
R 10 × 10 andensures (cid:80) ni
=1 D i = [-
-]0 .
Vectors a 1 , · · · , a n ∈ R 10 are {+randomly generated and (cid:80) ni =
 a i a Ti shouldbe invertible.+}
For both experiments, we assign a data point to each node i on the general graph‘Dolphins’ ( 62 nodes) [56].
We set the step size in the SGD algorithm to be 1 /t 0 .
9 , and use MSE 372 E [ ∥ θ t − θ ∗ ∥ 22 ] as our performance metric to measure the relative performance of different inputs.
Wealso employ the scaled MSE E [ ∥ θ t − θ ∗ ∥ 22 ] /γ t to empirically show its relationship to the CLT result(9).
More simulation results are deferred to Appendix I [1].
In Figure 2 we compare NBRW and SRW as input sequences on the graph ‘Dolphins’ for twoobjective functions in (16).
We also compare uniform sampling, random shuffling and single shuffling,assuming that they can access any node on the graph in each iteration.
We can see in Figure 2a and2c that NBRW always falls below SRW throughout all time periods, which indicates that NBRWtends to have smaller MSE than SRW.
Single and random shuffling are both better than uniformsampling in terms of smaller MSE.
The oscillation of single shuffling comes from a predeterminedfixed data sampling sequence, while random shuffling changes the permutation whenever traversingall nodes.
Figure 2b shows that the scaled MSEs of NBRW, SRW and uniform sampling approachsome constants after some time, which is consistent with the CLT result (9).
The curves of NBRWare still below that of SRW, showing that the input with smaller scaled MSE tends to have higherefficiency, which supports Proposition 4.1.
We can see from Figure 2d that NBRW, SRW have notyet entered the regime when the covariance matrix becomes the main factor (the curve is increasing)while uniform sampling and both shuffling methods are just entering this regime. The curves of singleand random shuffling in Figure 2b and 2d fall below that of uniform sampling and still decreasebecause eventually their covariance matrices will be zero matrix, as indicated in Proposition 4.3.
[Figure]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section __.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs are
included in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they werechosen)?
[Yes] The details of the simulation setup are provided in the supplementary
material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The citations can be
found in Section 5.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]