Efficiency Ordering of Stochastic Gradient Descent
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the stochastic gradient descent (SGD) algorithm driven by a generalstochastic sequence, including i.i.d noise and random walk on an arbitrary graph,among others; and analyze it in the asymptotic sense.
Specifically, we employ thenotion of ‘efficiency ordering’, a well-analyzed tool for comparing the performanceof Markov Chain Monte Carlo (MCMC) samplers, for SGD algorithms in theform of Loewner ordering of covariance matrices associated with the scaled iterateerrors in the long term.
Using this ordering, we show that input sequences that aremore efficient for MCMC sampling also lead to smaller covariance of the errorsfor SGD algorithms in the limit.
This also suggests that an arbitrarily weighted
MSE of SGD iterates in the limit becomes smaller when driven by more efficientchains.
Our finding is of particular interest in applications such as decentralizedoptimization and swarm learning, where SGD is implemented in a random walkfashion on the underlying communication graph for cost issues and/or data privacy.
We demonstrate how certain non-Markovian processes, for which typical mixingtime based non-asymptotic bounds are intractable, can outperform their Markoviancounterparts in the sense of efficiency ordering for SGD.
We show the utilityof our method by applying it to gradient descent with shuffling and mini-batchgradient descent, reaffirming key results from existing literature under a unifiedframework.
Empirically, we also observe efficiency ordering for variants of SGDsuch as accelerated SGD and Adam, open up the possibility of extending our notionof efficiency ordering to a broader family of stochastic optimization algorithms.
Introduction
Stochastic gradient descent (SGD) is widely used in machine learning, signal processing and otherengineering fields to solve the optimization problem
[Equation]
where Θ ⊂ R d is some closed and convex set, and F ( · , i ) : R d → R for i ∈ [ n ] ≜ { 1 , · · · , n } aresmooth functions on Θ , not necessarily convex, such that their summation f : R d → R exhibits aminimizer θ ∗ ∈ Θ satisfying ∇ f ( θ ∗ )=0 .
The update rule of the iterative SGD scheme is of the form
[Equation]
where γ t is the step size that can be constant or diminishing as t →∞ , Proj Θ is a projection operatoronto the constraint set Θ , and { X t } t ≥ 0 is some sequence taking values in [ n ] .
This sequence isoften generated in a stochastic manner, and samples can be drawn from temporally independentand identically distributed (i.i.d) random variables that are either uniformly distributed over [ n ][60, 54, 14], or leverage importance sampling techniques for variance reduction [52, 12, 27].
{ X t } t ≥ 0 can also be constructed by repeatedly shuffling over all possible states without repetition, 1 leading tofaster convergence than stochastic counterparts drawing i.i.d samples from [ n ] [64, 3, 32, 72].
Random Walk Stochastic Gradient Descent (RWSGD): Some applications observe restrictedaccess to the state space, such as decentralized optimization [65, 71, 46], where communicationoccurs between nodes in a network to collaboratively solve the optimization problem (1).
For instance,disease classification in confidential clinical swarm learning [69] considers peer-to-peer networksdue to the highly private nature of medical data.
In such a setting, the random sequence { X t } t ≥ 0is usually realized as a Markov chain on a general graph G ( V , E ) that only samples local gradientsof the nodes in V ≜ [ n ] and traverses the network via edges connecting them without divulgingthe update history or its own gradient.
The randomness of the communication path ensures that thecompromised node can not easily leak the data of its neighbors [46].
Apart from the privacy concern, such dynamics are also employed in swarm learning/optimizationin robotics [17] and wireless sensor networks [43] due to their low communication cost and asynchronous nature.
The need for data privacy and demand for communication-efficient algorithms fordecentralized optimization has spurred the study of RWSGD algorithms in recent years [57, 34, 67],with the underlying Markov chain in the form of Metropolis-Hasting random walk (MHRW) [48].
Common analytical approach - Finite time bounds based on mixing time: Most of the existingworks analyzing iteration (2) provide so-called finite-time upper bounds on expected error in eitherthe objective function E [ f (˜ θ t ) − f ( θ ∗ )] , where ˜ θ t is some weighted average of the iterates, or itsgradient E [ ||∇ f ( θ t ) || 22 ] ; and are used to infer the convergence rate of the iterate sequence [57, 25, 67].
For diminishing step sizes γ t = t − α with α ∈ (0 . 5 , 1) , 2 the upper bound on E [ ∥∇ f ( θ t ) ∥ 22 ] reads as
[Equation]
and a similar form for E [ f (˜ θ t ) − f ( θ ∗ )] as well [67, 24, 7].
Here, β ∈ (0 , 1) is the second largesteigenvalue modulus (SLEM) of the underlying Markov chain’s transition matrix and is related toits mixing time property, since smaller SLEM leads to faster mixing of the Markov chain [18, 44].
On the other hand, M > 0 is usually a quantity proportional to the local gradients evaluated at theminimizer, or their upper bound.
Both the gradient information and the mixing time play a key rolein quantifying the convergence rate derived from this upper bound, and the mixing time is especiallyimportant since it hints that convergence rate of the SGD algorithm can potentially be acceleratedusing faster mixing Markov chains for the input driving sequence.
It has also been noted that theinherent correlation of the underlying random walk has to be addressed in any analysis concerning
Markov-chain-driven gradient descent [67].
The mixing time technique, by capturing the rate atwhich the chain converges to its stationary distribution [18, 44], is one way of doing so.
Alternative approach - Asymptotic analysis and efficiency ordering:
In addition to the aforementioned mixing time, another widely used metric for characterizing the second order propertiesof
Markov chains is the asymptotic variance (AV).
For any scalar valued function g : [ n ] → R ,the estimator ˆ µ t ( g ) ≜ 1 t (cid:80) ti =1 g
( X i ) , associated with an irreducible Markov chain { X t } t ≥ 0 withstationary distribution π , is the average of the samples of g ( · ) obtained along the chain’s sample pathup to time t > 0 .
The AV of the Markov chain, denoted by σ 2 X ( g ) , is then defined as the the limitingvariance of the estimator; that is,
[Equation]
For all functions g ( · ) satisfying E π ( g 2 ) < ∞ , the AV is associated with the Central Limit Theorem(CLT) for any Markovian kernel on a finite state space, as the variance of the normally distributedestimates in the limit [61, 35, 18].
More formally, we have
[Equation]
A smaller AV means that fewer samples are required post mixing of the chain 3 in order to obtain adesired accuracy - in some sense quantifying the chain’s efficiency .
Both the AV and the mixing time of a Markov chain are very strongly related concepts 4 .
In fact, the
AV has an upper bound in terms of the SLEM, which decreases as the SLEM gets smaller (chainmixes faster)
However, an ordering of the SLEM between two Markov chains does not implyan ordering of their AV, as we shall demonstrate later in Section 4 for a special case.
Both of thesesecond-order properties therefore lead to different notions of optimality; and the comparison of twochains based on their AV leads to the concept of efficiency ordering [50], where we say that a chain ismore efficient than the other if it has a smaller AV, uniformly over all functions g : [ n ] → R .
As mentioned earlier, the common intuition assertedby finite time bounds such as (3) is that Markovchains with smaller SLEM lead to faster convergence of the SGD iteration (2) to the minimizer[67, 7].
We put this logic to test by simulating the RWSGD algorithm with three different reversible Markov chains (w.r.t.
uniform stationary distribution) as the stochastic inputs - the MHRW, a modification of MHRW, which is also shown in Appendix I [1] to be more efficient than MHRW, and the socalled ’fastest mixing Markov chain’ (FMMC) asdefined in [16] as the Markov chain obtained byminimizing the SLEM over the entire class of reversible chains for a given graph topology.
Weemploy RWSGD to minimize a quadratic objectivefunction for two underlying graphs.
The exact details of the setup are deferred to Appendix I [1], andour numerical results in Figure 1 show that eventhough the FMMC is theoretically guaranteed tohave the smallest SLEM ( β i for i ∈{ 1 , 2 , 3 } ) of thethree reversible chains simulated, it is the worst performing one with largest mean square error (MSE).
Although MHRW and Modified-MHRW share thesame SLEM in the lower plot of Figure 1, they still have performance differences.
This is contradictory to the intuition derived from (3), and could be attributed to the finite time results providing upperbounds for all times t > 0 , which may therefore not necessarily be tight.
On the other hand, theperformance of the chains seem to be ordered according to their AV ( σ 2 i for i ∈{ 1 , 2 , 3 } ) evaluatedfor a test function.
This lends credence to developing techniques based on AV, for judging theperformance of different stochastic inputs for SGD, as possible alternatives to using SLEM as thesole performance metric.
The asymptotic variance also appears in the CLT for stochastic approximation (SA) algorithms[11, 21], though this time not directly as the variance in the limit, but as a component of the limitingcovariance matrix of the scaled iterate errors.
Recent works [21, 51] point out that the covariancematrix itself is of special interest, and typically contains more information than the non-asymptotic MSE bounds [51].
In the sense of SGD algorithms, we will show in Section 3 that it embeds explicitinformation of the exact vector-valued gradient evaluated at the optimizer as well as the entirespectrum of the transition matrix; as opposed to the upper bound M of the gradients and only thesecond largest eigenvalue modulus commonly found in mixing time based non-asymptotic bounds.
Ithas been suggested [20, 23], and also proved for the special case of linear SA [21], that the covariancematrix emerging out of the CLT dominates as a leading term of the finite-time MSE bounds.
Thisalso holds true for finite-time bounds on weighted MSE for any preferred weight; the weighted MSEbeing utilized in fields such as wireless MIMO [68] and process optimization [30].
Overall, whilefinite-time bounds have enjoyed great success in the literature, the potential for performance gainsout of the asymptotic analysis of SGD algorithms have remained largely unexplored.
Contributions: We employ asymptotic analysis to propose a general framework that offers seamlessconnection between AV in the MCMC literature with efficiency ordering and covariance matrix in the
SGD algorithms.
Our framework can be used to design different random walk variants and also tosystematically compare the existing sampling methods in the SGD iteration (2) with diminishing step size, not just limited to random walks.
In particular, we show that any two random walks followingan efficiency ordering have their covariance matrices Loewner ordered, including non-Markovianstochastic processes versus its Markovian counterpart, which defies any mixing-time (SLEM) basedanalysis.
Such ordering can be harnessed into improving the accuracy of SGD iterates, which impliesa reduction in the weighted MSE with arbitrary weights.
Moreover, via a specific augmentationof the state space, we are able to analyze SGD for both single and random shuffling and showthe efficiency of shuffling over i.i.d sampling for a set of objective functions that may not satisfy‘Polyak-Łojasiewicz inequality’.
We further extend such comparison to mini-batch SGD algorithms.
Lastly, we present numerical results where the efficiency ordering via asymptotic analysis tends tohold over all time periods and input sequences with higher efficiency have smaller errors in SGD.
Modeling setup
Basic notations: We use lower case, bold-faced letters to denote vectors ( v ∈ R d ), and use uppercase, bold-faced letters to denote matrices ( M ∈ R d × d ).
∥ · ∥ 2 denotes the l 2 norm for vectors or2 -norm for matrices.
We use ∇ f ( · ) as Jacobian matrix of vector-valued function f ( · ) , and ∇ 2 g ( · ) as
Hessian matrix of scalar-valued function g ( · ) .
We let ∇ θ g ( θ, X ) be the gradient of the scalar-valuedfunction g ( θ, X ) with respect to θ and omit the subscript θ for simplicity.
Loewner ordering ofmatrices is denote by ‘ ≤ L ’ such that A ≤ L B ⇐⇒ x T ( A − B ) x ≤ 0 for any x ∈ R d .
The term Tr ( A ) denotes the trace of matrix A , and let 1 {·} be the indicator function.
We write N (0 , V )to represent a multivariate Gaussian distribution with zero mean and covariance matrix V .
For aconnected and undirected graph G ( V , E ) with node set V and edge set E , we use N ( i ) for the set ofneighbors of node i ∈ V and d ≜ [ d 1 , d 2 , · · · , d n ] T for the degree vector where d i = | N ( i ) | .
SGD algorithm with arbitrary input sequence: We consider random walks { X t } t ≥ 0 for whichthe limit π i ≜ lim t →∞ 1 t (cid:80) tk =1
X k = i } exists almost surely and is positive for all i ∈ [ n ] , withπ =
[ π i ] i ∈ [ n ] denoting the limiting or stationary distribution.
This is trivially satisfied via strong lawof large numbers [26] when X t for each t > 0 are i.i.d random variables with distribution π over [ n ] ,and via the ergodic theorem [18] when { X t } t ≥ 0 is an irreducible, aperiodic and positive recurrent(ergodic)
Markov chain.
Note however that this way of defining the stationary distribution π allowsfor the input sequence { X t } t ≥ 0 to be more general, possibly being non-Markov on [ n ] .
Then, we canuse π to rewrite the objective in (1) as
[Equation]
where function G ( θ, i ) ≜ 1 nπ i F ( θ, i ) for any θ ∈ Θ , i ∈ [ n ] .
The generalized update rule then becomes
[Equation]
This change of notation allows us to consider input sequences having possibly non-uniform stationarydistributions, and is a version of importance sampling for RWSGD schemes, as in [7].
For example,the iteration (7) with the input sequence generated from a MHRW with uniform target distributionπ = 1 /n will reduce down to (2) with G ( θ, i )
=
F ( θ, i ) for all θ ∈ Θ , i ∈
[ n ] .
If the input sequenceis instead a simple random walk on a connected graph G ( V , E ) with V =
[ n ] , we have π ∝ d , and
G ( θ, i ) =
1 T d nd i F ( θ, i ) for all θ ∈ Θ , i ∈ V .
Asymptotic covariance matrix.
We now quickly review the multivariate CLT for Markov chains,since it is a natural way to introduce the asymptotic covariance matrix, used heavily throughoutthe paper.
For any finite, irreducible Markov chain { X t } t ≥ 0 with stationary distribution π , itsestimator is defined as ˆ µ t ( g ) ≜ 1 t (cid:80) tk =1 g ( X k ) for any vector-valued function g : [ n ] → R d .
Then, the ergodic theorem [18, 19] states that for any initial distribution and any g ( · ) such that
E π ( g ) =
(cid:80) i ∈ [ n ] g ( i ) π
i < ∞ , we have ˆ µ t ( g )
a.s. −−−→ t →∞ E π ( g ) .
Similarly to the asymptotic varianceσ 2 X ( g ) for a scalar-valued function g ( · ) , we can also define the asymptotic covariance matrix Σ X ( g )for vector-valued function g ( · ) , [Equation]
[Equation]
where ∆ t ≜ (cid:80) ts =1 ( g ( X s ) − E π ( g ))
The associated multivariate CLT is then given as follows.
Theorem 2.1 (Chapter 1 [19]) .
For any function g : [ n ] → R d that satisfies E π ( g 2 ) < ∞ , we have
[Equation]
In the next section, we will show how the the asymptotic covariance matrix Σ X ( · ) also appears aspart of the CLT result for SGD algorithms.
Efficiency Ordering of SGD Algorithms
In this section, we present our main result concerning the performance comparison of different SGDalgorithms to solve (1).
We first begin by stating our assumptions on the objective function and thestochastic input sequence, providing a CLT result for SGD algorithms, and analyzing the covariancematrix arising therein.
We then introduce the notion of efficiency ordering of Markov chains in thecontext of MCMC sampling, and form the connection with covariance matrices as our main result in
Theorem 3.6.
For the rest of this section we assume that the functions F ( · , i ) (possibly non-convex), the summandsof the objective function in (1), and the input process { X t } t ≥ 0 for the SGD iteration (7) satisfy:
(A1)
The step size is given by γ t = t − α for α ∈ (1 / 2 , 1] ;
(A2)
There exists a unique minimizer θ ∗ in the interior of the compact set Θ with ∇ f ( θ ∗ ) = 0 ,and matrix ∇ 2 f ( θ ∗ ) (resp.
∇ 2 f ( θ ∗ ) − I / 2 ) is positive definite for a ∈ (1 / 2 , 1) (resp.
a =1 );(A3)
Gradients are bounded in the compact set Θ , that is, sup θ
∈ Θ sup i ∈ [ n ] ||∇ F ( θ, i ) || 2 < ∞ ;
(A4)
For every z ∈ [ n ] , θ ∈ R d , the solution ˜ F ( θ, z ) ∈ R d of the Poisson equation ˜ F ( θ, z ) − E [Equation]
θ,
[Equation]
z [Equation] θ, z ) −∇ f ( θ ) exists, and sup θ ∈ Θ ,z ∈ [ n ] ∥ ˜ F ( θ, z ) ∥ 2 < ∞ ;(A5)
The functions F ( θ, i ) are L -smooth for all i ∈ [ n ] , that is, ∀ θ 1 , θ 2 ∈ Θ , ∀ i ∈ [ n ] , we have
[Equation]
We then have the following CLT result for SGD algorithms.
Lemma 3.1.
For iterates { θ t } t ≥ 0 of the SGD algorithm (7) satisfying (A1)–(A5), we have
[Equation]
where covariance matrix V X is the unique solution to the Lyapunov equation Σ X + KV X
+ V X K T
=0 when α ∈ (0 . 5 , 1) (resp.
Σ X + (cid:0) K
+ I 2 (cid:1) V X + V X
(cid:0) K
+ I 2 (cid:1) T = 0 )
when α = 1 ).
Here, Σ X ≜Σ X ( ∇ G ( θ ∗ , · )) is the asymptotic covariance matrix 6 as in (8) , and K ≜ ∇ 2 f ( θ ∗ ) .
Additionally, for the averaged iterates { ¯ θ t } t ≥ 0 where ¯ θ t ≜ 1 t (cid:80) t − 1 i =
θ t , we have
[Equation]
[Equation] t [Equation] θ ∗ , and √ t (¯ θ t − θ ∗ ) Dist −−−→ t →∞ N (0 , V ′ X ) , (10) where V ′ X
=
K − 1 Σ X ( K − 1 ) T with the same matrices K and Σ X as in the non-averaged case.
Remark 3.2.
Lemma 3.1 is itself a special case of the more general CLT result for SA algorithmsprovided in Appendix A [1], and as proved in Appendix B [1].
Remark 3.3.
While (A2) may appear to be too strict at first, it can be relaxed to the setting ofthe objective function f ( · ) having multiple minimizers, by leveraging more general CLT resultsfrom SA literature, such as Theorem 2.1 in [28].
However, this comes at a cost of cumbersomenotation, requiring conditioning of iterates converging to one of the minimizers, potentially making the mathematical parts harder to follow.
We also show in Appendix C [1] that (A2) is no stricter thanthe Polyak-Łojasiewicz inequality – a popularly adopted weak assumption in recent SGD literature studying non-convex objective functions [36, 47, 70, 72].
Remark 3.4.
Assumptions (A3) and (A5) are widely seen in the RWSGD literature [57, 34, 67],while (A4) is automatically satisfied for any ergodic Markov chain (see [49, 21] for details), acommon assumption for the stochastic noise sequence [34, 25, 7].
The compactness in (A3) can alsobe relaxed, given assumptions on the objective function in [37], such that the estimator θ t generatedby Markov-driven sequences can still be ‘locked in’ a compact set after a sufficiently long time.
Lemma 3.1 implicitly indicates that the asymptotic convergence rate (in distribution) for θ t − θ ∗(resp.
¯ θ t − θ ∗ ) is O ( √ γ t ) (resp.
O (1 / √ t ) ).
While this does not necessarily translate to O ( √ γ t )convergence rate for E [ ∥ θ t − θ ∗ ∥ 2 ] ( O (1 / √ t ) for E [ ∥ ¯ θ t − θ ∗ ∥ 2 ] ), it has been suggested [20, 23], andis in fact true for cases such as quadratic objective functions since they satisfy the linear stochasticapproximation in [21], which is of the form
[Equation] [Equation]
) for which the connection between finite-time MSE and covariance matrix V X has been established[21].
This is also true for arbitrarily weighted MSE, which can be obtained as a weighted sum ofdiagonal entries of the covariance matrices V X and V ′ X .
In addition to the apparent connection to MSE, the covariance matrix plays a wider role in SGDperformance.
Given any vector of weights w ∈ R d , from Lemma 3.1 we also have that the weightedsum of errors w T ( θ t − θ ∗ ) converges to zero almost surely, and that w T ( θ t − θ ∗ ) / √ γ t Dist −−−→ t →∞
N (0 , w T V X w ) .
This means that, for sufficiently large t , we can estimate
[Equation]
such that, for instance, the 95% confidence interval for w T θ t is approximately w T θ ∗ ± 2 (cid:112) γ t w T V X w .
In other words, smaller w T V X w leads to narrower confidence interval and higher accuracy.
Theform w T V X w for any vector w ∈ R d naturally implies that Loewner ordering should come into playwhen concerning the performance of SGD algorithms.
To proceed, we first employ the widely used notion of efficiency ordering of Markov chains.
Theefficiency of different chains is compared by ordering them using their respective AV as follows.
Definition 3.5 ( Efficiency Ordering [50]) .
For two random walks { X t } t ≥ 0 and { Y t } t ≥ 0 with thesame stationary distribution π , we say { X t } t ≥ 0 is more efficient than { Y t } t ≥ 0 , which we write as
X ≥ E Y , if and only if σ 2 X ( g ) ≤ σ 2 Y ( g ) for any g : [ n ] → R .
We are now ready to state our main result.
We first extend the efficiency ordering of Markov chainsby proving the equivalence of comparing their scalar-valued AVs, to comparing their asymptoticcovariance matrices via Loewner ordering.
We then use this extension to show that more efficientinputs { X t } t ≥ 0 (as in Definition 3.5) to the SGD algorithm lead to performance improvements in theform of smaller covariance matrices in the Loewner ordering sense.
Theorem 3.6.
Consider the SGD iteration (7) with two random walks { X t } t ≥ 0 and { Y t } t ≥ 0 as inputsequences, with the same stationary distribution π , satisfying (A1)–(A5).
Then,
(i) X ≥ E Y
if and only if Σ X ( g ) ≤ L Σ Y ( g ) for any vector-valued function g ;
(ii) If Σ X ( ∇ G ( θ ∗ , · ))
≤ L Σ Y ( ∇ G ( θ ∗ , · )) , then V X ≤ L V Y ( V ′ X ≤ L V ′ Y for the case of averagediterates ) ;
where function ∇ G ( θ ∗ , · ) : [ n ] → R d is defined in the SGD iteration (7) , V X and V ′ X (resp.
V Yand V ′ Y ) are the covariance matrices from Lemma 3.1, corresponding to { X t } t ≥ 0 (resp.
{ Y t } t ≥ 0 )as the stochastic input sequence.
Theorem 3.6 enables us to provide a sense of efficiency ordering of SGD algorithms which aredriven by different stochastic inputs.
Since this is achieved via Loewner ordering, it also leads tosmaller confidence intervals in the long run as mentioned earlier, as well as potentially smaller MSE 7depending on the objective function.
Remark 3.7.
In addition to the CLT result for SGD algorithms with diminishing step size describedin Lemma 3.1, we include in Appendix E [1] similar results for constant step sizes and quadraticobjective functions, where the statement of Theorem 3.6 still holds.
Applications: Towards More Efficient SGD
In this section, we present some SGD variants and compare them in terms of efficiency ordering of SGD.
Specifically, we first show that a certain class of non-Markov random walks can provide a betterinput sequence than its Markovian counterpart.
We then analyze shuffling-based gradient descent andcompare it to the SGD with i.i.d input in terms of efficiency ordering for SGD algorithm.
We alsoextend our approach to a more general mini-batch version, the discussion for which is deferred to
Appendix H.3 [1].
High-Order Efficient Random Walk for SGD: The simple random walk (SRW) is a popular
Markov chain that has been extensively studied in the literature [62, 58, 29].
Several recent workshave focused on the non-backtracking random walk (NBRW) on a connected undirected graph
G ( V , E ) in the MCMC literature, which is an extension of SRW with the same limiting distributionπ =
d / 1 T d [53, 5, 41, 38, 10].
Intuitively speaking, NBRW is a random walk that selects one of itsneighbors uniformly at random except the one it just came/transitioned from.
Specifically, the NBRW{ Y t } t ≥ 0 is a second-order non-reversible Markov chain (i.e., it is non-Markov on V =
[ n ] ) with itstransition probability given by
[Equation]
Since the limiting distributions of NBRW and SRW are the same, NBRW can be used as the input for SGD iterations (7) with the same re-weighted local functions G ( θ ∗ , i ) as that of SRW for all i ∈ [ n ]whenever the applications call for random-walk type of inputs.
Let Σ Y ( ∇ G ( θ ∗ , · )) be the asymptoticcovariance matrix of this NBRW { Y t } t ≥ 0 , as defined in (8).
One of the main results in [41] concernsthe efficiency ordering of NBRW and SRW.
They show that NBRW has a smaller AV, or equivalently,from our Theorem 3.6 (i), a smaller asymptotic covariance in terms of Loewner ordering.
Our nextresult forms the necessary connection between the asymptotic covariance matrix arising in the CLTresult and Σ Y ( ∇ G ( θ ∗ , · )) .
Proposition 4.1.
Consider the SGD iteration (7) with two input sequences SRW { X t } t ≥ 0 and NBRW{ Y t } t ≥ 0 respectively.
Then, both the respective estimators θ Xt , θ Yt a.s. −−−→ t →∞ θ ∗ , and V Y ≤ L V X , thatis, NBRW is more efficient than SRW in the SGD algorithm.
By augmenting the state space, we can represent NBRW as a Markov chain Z t =
( Y t − 1 , Y t ) ∈ V ×V ,as was done in [53, 41].
This transformation then allows us to build CLT for an SGD iteration with{ Z t } t ≥ 0 as the input.
The subtlety here is to prove that the asymptotic covariance matrix arisingout of the CLT with respected to the augmented process { Z t } t ≥ 0 is indeed equal to Σ Y ( ∇ G ( θ ∗ , · )) .
This is shown by cultivating the relationship between the stationary distribution of { Z t } t ≥ 0 on theaugmented state space V × V and { Y t } t ≥ 0 on the node space V , as provided in [53].
Thus, our Theorem 3.6 together with the existing works on efficiency ordering of NBRW versus SRWin the MCMC literature [53, 41] enable us to show that NBRW is a more efficient input sequencethan SRW for the SGD iteration (7).
Interestingly, it has been shown that non-backtracking walksmix faster when the underlying graph is d − regular [5].
In this case, a faster convergence rate is alsosuggested by mixing time based non-asymptotic bounds prevalent in RWSGD literature.
However,no such results concerning mixing time and SLEM exists for NBRW on a general graph.
Thus, in theform of Proposition 4.1, we demonstrate the utility of our approach in settings where mixing timebased comparisons are unavailable.
Shuffling versus i.i.d Input Sequence: Shuffling-based methods have been widely used in machinelearning applications [13].
They work by repeatedly passing over the entire state space [ n ] withoutrepetition, each complete pass forming an epoch .
Random shuffling and single shuffling are twoversions therein and differ in the order in which they pass over [ n ] .
Random shuffling, as thename suggests, makes the pass in a randomly chosen order in each epoch, while single shufflingmaintains the same predetermined order (often randomly chosen once at the beginning) for all epochs.
Shuffling-based methods are known to show better empirical performance than i.i.d input[15], although intense theoretical analysis for shuffling-based gradient descent has only emergedin recent years [66, 33, 64, 3, 32].
In what follows, we use our results from Section 3 to compareshuffling-based gradient descent to SGD with i.i.d input.
To do so, we first analyze the asymptoticcovariance matrix for shuffling-based methods.
Lemma 4.2.
Let the input process { X t } t ≥ 0 be single or random shuffling.
Then, for any vectorvalued function g : [ n ] → R d , Σ X ( g ) = 0 , where Σ X ( g ) is defined in (8) .
For i.i.d input sequence with distribution ˆ π , the asymptotic covariance from Lemma 3.1 reduces to
[Equation]
following its definition in (8), and thus, trivially, Σ X ( ∇ G ( θ ∗ , · )) ≥ L 0 .
Lemma 4.2 shows thatshuffling-based methods are more efficient than i.i.d input sequence due to a smaller asymptoticcovariance matrix in Loewner ordering.
Next, we show that they also outperform i.i.d input whenused for driving the input sequence of SGD algorithms.
Proposition 4.3.
Consider the SGD iteration (7) with stochastic inputs single/random shuffling{ X t } t ≥ 0 and i.i.d sampling { Y t } t ≥ 0 , we have θ Xt , θ Yt [Equation] [Equation] →∞
[Equation] ∗ and V X = 0 ≤ L V Y . Though it may seem so at first, Proposition 4.3 is not a simple application of Theorem 3.6, especially for random shuffling because it is hard to check if random shuffling, formulated as atime-inhomogeneous Markov chain, indeed satisfies (A4).
To overcome this difficulty, in Appendix
H [1] we come up with a non-trivial augmentation to a much higher dimensional state space ( [ n ] n +1 )to make random shuffling a time-homogeneous periodic Markov chain in order to show that bothsingle shuffling and random shuffling satisfy (A4) and thus apply Theorem 3.6.
The case of shuffling versus i.i.d inputs is an example of a setting where the sequence with larger
SLEM is more efficient than one with smaller SLEM 8 as an input sequence to the SGD iteration (7).
For quadratic objective functions that satisfy the linear SA iteration in [21], it also attains a fasterconvergence speed in terms of MSE than i.i.d inputs to SGD algorithms.
Although some recentworks provide more informative finite-time error bounds on the MSE of the objective function forshuffling-based methods, by studying a special case of the matrix norm AM-GM inequality andproving faster convergence rate than i.i.d inputs [56, 3, 32], our result is not a subset of theirs.
To beprecise, we show in Appendix C [1] that our assumption (A2) on the objective function is no lessgeneral than their most general setting based on the Polyak-Łojasiewicz inequality.
Remark 4.4. Mini-batch gradient descent is another popular gradient descent variant and is widelyused in the machine learning tools [22, 2, 55] to accelerate the learning process when compared to
SGD.
In Appendix H.3 [1] we show how our framework can be applied to study min-batch based
SGD algorithms, and include the asymptotic analysis on mini-batch gradient descent with shuffling.
Besides mini-batch gradient descent, there are other SGD variants, e.g., momentum SGD, Nesterovaccelerated SGD and ADAM, that have been studied in the SGD literature for non-asymptotic analysis[39, 59, 6].
However, asymptotic analysis on the SGD variants is not well studied in the literature,with only very recent results on the CLT for i.i.d input sequences [42, 8, 9, 45].
Asymptotic analysisand CLT for variants more general Markovian input sequences, which would be a prerequisite for
Theorem 3.6, remains an open problem.
We defer the discussion on related works to Appendix I.3[1], where we also empirically test the SGD variants and find that the efficiency ordering result stillholds for these SGD variants - opening up the possibility for further theoretical analysis.
Numerical Experiments
In this section, we empirically validate our theoretical analysis.
We select two convex objectivefunctions as follows.
[Equation]
For l 2 -regularized logistic regression ˜ f ( θ ) , we choose the dataset CIFAR-10 [40] where n is thetotal number of data points.
Here, x i ∈ R 108 is the vector flattened from the cropped image i with shape (6 , 6 , 3) , and y i ∈ R is the label.
For sum-of-non-convex functions ˆ f ( θ ) , which is based onthe experiment setup in [31, 4], we generate random vectors a i , b and matrices D i which ensure theinvertibility of matrix (cid:80) ni
=1 a i a Ti and (cid:80) ni
=1 D i = {+
+}
(details are [-randomly generated and (cid:80) ni =
 a i a Ti shouldbe invertible.-]deferred to Appendix I.2 [1]).
Forboth experiments, we assign a data point to each node i on the general graph ‘Dolphins’ ( 62 nodes)[63].
We set the step size in the SGD algorithm as 1 /t 0 .
9 , and use MSE E [ ∥ θ t − θ ∗ ∥ 22 ] to measurethe relative performance of different inputs.
We also employ the scaled MSE E [ ∥ θ t − θ ∗ ∥ 22 ] /γ t toempirically show its relationship to the CLT result (9).
Due to space constraints, additional simulationresults which support our efficiency ordering result in the setting of large graphs, and non-convexfunctions are deferred to Appendix I.3 in [1]. Therein, via numerical simulations, we also observethe efficiency ordering for other SGD variants such as Nesterov accelerated SGD and ADAM whencomparing their iterations under efficiency ordered noise sequences.
In Figure 2 we compare NBRW and SRW as input sequences on the graph ‘Dolphins’ for twoobjective functions in (14).
We also compare uniform sampling, random shuffling and single shuffling,assuming that they can access any node on the graph in each iteration.
We can see in Figure 2a and 2cthat NBRW always falls below SRW throughout all time periods, which indicates that NBRW tendsto have smaller MSE than SRW.
Single and random shuffling are both better than uniform samplingin terms of smaller MSE.
The oscillation of single shuffling comes from a predetermined fixed datasampling sequence, while random shuffling changes the permutation whenever traversing all nodes.
Such oscillation is not visible in Figure 2c and Figure 2d because it is small on the current y-axisscale.
The curves of single and random shuffling in Figure 2b and 2d fall below that of uniformsampling and still decrease in the linear rate because eventually their covariance matrices will be zeromatrix, as indicated in Proposition 4.3.
Figure 2b shows that the scaled MSEs of NBRW, SRW anduniform sampling approach some constants after some time, which is consistent with the CLT result(9).
The curves of NBRW are still below that of SRW, showing that the input with smaller scaled MSE tends to have higher efficiency, which supports Proposition 4.1.
We can see from Figure 2dthat the curves NBRW, SRW are still increasing andd they have not yet entered the regime where thecovariance matrix becomes the main factor. On the other hand, uniform sampling and both shufflingmethods are just entering this regime.
[Figure]
Conclusion
We have introduced the notion of efficiency ordering of SGD algorithms, and shown that processeswith smaller asymptotic covariance are always more efficient as input sequences for SGD algorithms.
Furthermore, we numerically observe that this sense of efficiency ordering is also seen SGD variantssuch as Nesterov accelerated SGD and Adam.
Since the asymptotic analysis of such SGD variantsis not well-established under general stochastic inputs, establishing theoretical results on efficiencyordering remain an open problem.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:•
Did you include the license to the code and datasets?
[Yes] See Section __.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs are
included in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The details of the simulation setup are provided in the supplementary
material.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The citations can be
found in Section 5.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/ A]
(b) Did you describe any potential participant risks , with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]