Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime
Anonymous Author(s)
AffiliationAddress email
Abstract
Stochastic gradient descent (SGD) has achieved great success due to its superiorperformance in both optimization and generalization.
Most of existing generalization analyses are made for single-pass SGD, which is a less practical variantcompared to the commonly-used multi-pass SGD.
Besides, theoretical analyses formulti-pass SGD often concern a worst-case instance in a class of problems, whichmay be pessimistic to explain the superior generalization ability for some particularproblem instance.
The goal of this paper is to provide an instance-dependent andalgorithm-dependent excess risk bound of multi-pass SGD for least squares inthe interpolation regime, which is expressed as a function of the iteration number,stepsize, and data covariance.
We show that the excess risk of SGD can be exactlydecomposed into the excess risk of GD and a positive fluctuation error, suggestingthat SGD always performs worse, instance-wisely, than GD, in generalization.
Onthe other hand, we show that although SGD needs more iterations than GD toachieve the same level of excess risk, it saves the number of stochastic gradientevaluations, and therefore is preferable in terms of computational time.
Introduction
Stochastic gradient descent (SGD) is one of the workhorses in modern machine learning due to itsefficiency and scalability in training and good ability in generalization to unseen test data.
From the optimization perspective, the efficiency of SGD is well understood.
For example, to achieve the same level of optimization error, SGD saves the number of gradient computations compared to itsdeterministic counterpart, i e., batched gradient descent (GD) [7, 8] and therefore saves the totalamount of running time.
However, the generalization ability (e g., excess risk bounds) of SGD is farless clear, especially from theoretical perspective.
Single-pass SGD, a less practical SGD variant where each training data is used only once, has beenextensively studied in theory.
In particular, a series of works establishes excess risk bounds ofsingle-pass SGD for learning general smooth and convex objectives [31, 23, 25] as well as learningleast squares [3, 10, 17, 18, 26, 13, 40, 36].
In practice, though, one often runs SGD with multiplepasses over the training data and outputs the final iterate, which is referred to as multi-pass SGD (orsimply SGD in the rest of this paper when there is no confusion).
Compared to single-pass SGD thathas limited number of optimization steps, multi-pass SGD allows the algorithm to perform arbitrarynumber of optimization steps, which is more powerful in optimizing the empirical risk and thus leadsto smaller bias error [27].
Despite the extensive application of multi-pass SGD in practice, there are only a few theoreticaltechniques being developed to study the generalization of multi-pass SGD.
One is based on theuniform stability [12, 16], which is defined as the change of the model outputs under a small changein the training data.
However, the stability based generalization bound is a worst-case guarantee, which is relatively crude and does not show difference between GD and SGD (See, e.g., Chen et al.[9] showed GD and SGD have the same stability parameter in the convex smooth setting).
On thecontrary, one easily observes a generalization difference between SGD and GD even in learning thesimplest least square problem (see Figure 1).
In addition, Lin and Rosasco [21], Pillaud-Vivien et al.[27], Mücke et al.
[24] explored the risk bounds for multi-pass SGD using the operator methodsthat are originally developed for analyzing single-pass SGD.
Their bounds are sharp in the minimaxsense for a class of least square problems that satisfy certain source condition (which restricts thenorm of the optimal parameter) and capacity condition (or effective dimension, which restricts thespectrum of the data covariance matrix). Still, their bounds are not problem-dependent, and could bepessimistic for benign least square instances.
In this paper, our goal is to establish an algorithm-dependent and problem-dependent excess riskbound of multi-pass SGD for least squares.
Our focus is the interpolation regime where the trainingdata can be perfectly fitted by a linear interpolator (which holds almost surely when the number ofparameter d exceeds the number of training data n ).
We assume the data has a sub-Gaussian tail [4].
Our main contributions are summarized as follows:
• We show that for any iteration number and stepsize, the excess risk of SGD can be exactly decomposed into the excess risk of GD (with the same stepsize and iteration number) and the
so-called fluctuation error , which is attributed to the accumulative variance of stochastic gradientsin all iterations.
This suggests that GD (with optimally tuned hyperparameters) always achieves smaller excess risk than SGD for least square problems.
• We further establish problem-dependent bounds for the excess risk of GD and the fluctuation error, stated as a function of the eigenspectrum of the data covariance, iteration number, training sample size, and stepsize.
Compared to the bounds proved in prior works [21, 27, 24], our bounds holdunder a milder assumption on the data covariance and ground-truth model. Moreover, our boundscan be applied to a wider range of iteration numbers t , i.e., for any t > 0 , in contrast to the priorresults that will explode when t → ∞ .•
We develop a new suite of proof techniques for analyzing the excess risk of multi-pass SGD.
The key to our analysis is describing the error covariance based on the tensor operators definedby the second-order and fourth-order moments of the empirical data distribution (i.e., samplingwith replacement from the training dataset), rather than the operators used in the single-pass SGDanalysis that are defined based on the (population) data distribution [17, 40] (i.e., sampling fromthe data distribution), together with a sharp characterization on the properties of the operators.
Based on the excess risk upper bounds for SGD and GD, we make the following complexitycomparison between SGD and GD: to achieve the same order of excess risk, while SGD may needmore iterations than GD, it can have fewer stochastic gradient evaluations than GD.
For example,consider the case that the data covariance matrix has a polynomially decaying spectrum with ratei − (1+ r ) , where r > 0 is an absolute constant.
In order to achieve the same order of excess risk, wehave the following comparison in terms of iteration complexity and gradient complexity 1 :
• Iteration Complexity: SGD needs to take (cid:101) O ( n max { 0 . 5 , r r +1 } ) more iterations than GD, with optimallytuned iteration number and stepsize.
• Gradient Complexity: SGD needs (cid:101) O ( n max { 0 . 5 ,
1 r +1 } ) less stochastic gradient evaluations than GD.
Notation.
For n > 0 , we use poly( n ) to define some positive high-degree polynomial functions ofn .
For two positive-value functions f ( x ) and g ( x ) we write f ( x ) ≲ g ( x ) if f ( x ) ≤ cg ( x ) for someconstant c > 0 , we write f ( x ) ≳
g ( x ) if g ( x ) ≲ f ( x ) , and f ( x ) ≂ g ( x ) if both f ( x ) ≲ g ( x ) andg ( x ) ≲ f ( x ) hold.
We use (cid:101) O ( · ) to hide some polylogarithmic factors in the standard
big-
O notation.
For two matrices A and B , we denote ⟨ A , B ⟩ = (cid:80) i,j A ij B ij and A ⊗ B as their Kronecker product.
Related Work
Optimization.
Regarding optimization efficiency, the benefit of SGD is well understood [7, 8, 22,5, 33, 34].
For example, for strongly convex losses (can be relaxed with certain growth conditions),
[Figure]
GD has less iteration complexity, but SGD enjoys less gradient complexity [7, 8].
More recently, itis shown that SGD can converge at an exponential rate in the interpolating regime [22, 5, 33, 34],therefore SGD can match the iteration complexity of GD.
Nevertheless, all the above results areregrading the optimization performance; our focus in this paper is to study the generalizationperformance of SGD (and GD).
Risk Bounds for Multi-Pass SGD.
The risk bounds of multi-pass SGD are also studied from theoperator perspective [29, 21, 27, 24].
The work by Rosasco and Villa [29] focused on cyclic SGD ,i.e., SGD with multiple passes but fixed sequence on the training data.
Their results are limitedto small stepsizes ( γ = O (1 /n ) ), while ours allow constant stepsize.
Similar to Lin and Rosasco[21], Pillaud-Vivien et al.
[27], Mücke et al.
[24], we decompose the population risk of SGD iteratesinto a risk term caused by batch GD iterates and a fluctuation error term between SGD and GDiterates.
But our methods of bounding the fluctuation error are different (see more in Section 5).
Moreover, our results are based on different assumptions:
Lin and Rosasco [21], Pillaud-Vivien et al.[27], Mücke et al.
[24] assumed strong finiteness on the optimal parameter, and their results onlyapply to data covariance with a specific type of spectrum (nearly polynomially decaying ones); incontrast, our results assume a Gaussian prior on the optimal parameter (which might not admit afinite norm), and our results cover more general data covariance (including those with polynomiallydecaying spectrum).
Lei et al.
[20] studied risk bounds for multi-pass SGD with general convex loss.
When applied to least square problems, their bounds are cruder than ours.
Uniform Stability.
Another approach for characterizing the generalization of multi-pass SGD isthrough uniform stability [16, 9, 19, 39, 6].
There are mainly two differences between this andour approach.
First, we directly bound the excess risk of SGD; but the uniform stability can onlybound the generalization error, there needs an additional triangle inequality to relate excess riskwith generalization error plus optimization error (plus approximation error) — this inequality caneasily be loose (consider the algorithmic regularization effects).
Secondly, the uniform stabilitybound is also crude.
For example, in the non-strongly convex setting, the uniform stability bound for
SGD/GD linearly scales with the total optimization length (i.e., sum of stepsizes), which grows as t[16, 9, 19, 39, 6] (this is minimaxly unavoidable according to Zhang et al.
[39], Bassily et al.
Notably, Bassily et al.
[6] extended the uniform stability approach to the non-convex and smoothsetting.
We left such an extension of our method as a future work.
Problem Setup
Let x be a feature vector in a Hilbert space H (its dimension is denoted by d , which is possiblyinfinite) and y ∈ R be its response, and assume that they jointly follow an unknown populationdistribution D .
In linear regression problems, the population risk of a parameter w is defined by
[Equation]
and the excess risk is defined by
[Equation]
where H :=
E D [ xx ⊤ ]
[Equation]
(3.1) where w ∗ = arg min w L D ( w ) denotes the global minimizer of the population risk.
In the statisticallearning setting, the population distribution D is unknown, and one is provided with a set of n trainingsamples, S =
( x i , y i ) ni
=1 , that are drawn independently at random from the population distribution.
We also use X :=
( x 1 , . . .
, x n ) ⊤ and y :
= ( y 1 , . .
. , y n ) ⊤ to denote the concatenated features andlabels, respectively.
The linear regression problems aim to find a parameter based on the training set S that affords a small excess risk.
Multi-Pass SGD.
We are interested in solving the linear regression problem using multi-passstochastic gradient descent (SGD) 2 with a constant learning rate.
The algorithm generates a sequenceof iterates ( w t ) t ≥ 1 according to the following update rule: the initial iterate is w 0 =
(which can beassumed without loss of generality); then at each iteration, an example ( x i t , y i t ) is drawn from Suniformly at random, and the iterate is updated by
[Equation]
where η > 0 is a constant stepsize (i.e., learning rate).
GD.
Another popular algorithm is gradient descent (GD).
For the clarity of notations, we use ( (cid:98) w t ) t ≥ 1to denote the GD iterates, which follow the following updates:
[Equation]
where η > 0 is a constant stepsize.
Definitions and Assumptions.
The eigenvalues of the population data covariance H is denoted by( λ i )
i ≥ 1 , sorted in non-increasing order.
Given the training data ( X , y ) , we define ϵ = y −
Xw
∗the collection of model noise, A = XX ⊤ as the gram matrix, and Σ = n − 1 X ⊤ X as the empiricalcovariance.
Then the minimum-norm solution is defined by
[Equation]
It is clear that in the interpolation regime, with appropriate stepsizes, both SGD and GD algorithmsconverge to (cid:98) w [14, 4].
The assumptions required by our theorems are summarized in below.
Assumption 3.1 For the linear regression problem:
A The components of H − 1 / 2 x are independent and 1 -subGaussian.
B The response y is generated by y := ⟨ w ∗ , x ⟩ + ξ , where w ∗ is the ground truth weight vector andξ is a noise independent of x .
Furthermore, the additive noise satisfies E [ ξ ] = 0 , E [ ξ 2 ] ≤ σ 2 .
C
The ground truth w ∗ follows a Gaussian prior N ( 0 , ω 2 · I ) , where ω 2 is a constant.
D The minimum-norm solution (cid:98) w linearly interpolates all training data,
i
e., y i = (cid:98) w ⊤ x i for i ∈ [ n ] .
149 Assumptions 3.1A and B are standard for analyzing overparameterized linear regression problem[4, 32].
Assumption 3.1C is also widely adopted in analyzing least square problems (see, e.g., Aliet al.
[2], Dobriban et al.
[11], Xu and Hsu [37]).
Finally, Assumption 3.1D holds almost surely whend > n , i.e., the number of parameter exceeds the number of data.
In the following, the presented risk bounds will hold (i) with high-probability with respect to therandomness of sampling feature vectors X , and (ii) in expectation with respect to the randomness ofmulti-pass SGD algorithm, the randomness of sampling additive noise ϵ and the randomness of thetrue parameter w ∗ as a prior.
For these purpose, we will use E SGD , E w ∗ to refer to taking expectationwith respect to the SGD algorithm and the prior distribution of w ∗ , respectively.
Main Results
Our first theorem shows that, under the same stepsize and number of iterates, SGD always generalizesworse than GD.
Theorem 4.1 (Risk decomposition) Suppose that Assumption 3.1D holds.
Then the excess risk of
SGD can be decomposed by
[Equation]
Moreover, the fluctuation error is always non-negative.
A Risk Comparison.
Theorem 4.1 shows that, in the interpolation regime, SGD affords a strictlylarger excess risk than GD, given the same hyperparameters (stepsize η and number of iteratest ).
Therefore, despite of a possibly higher computational cost, the optimally tuned GD dominatesthe optimally tuned SGD in terms of the generalization performance.
This observation is verifiedempirically by experiments in Figure 1.
Theorem 4.1 relates the risk of SGD iterates to that of GD iterates.
This idea has appeared in earlierliterature [21, 27, 24].
However, their decomposition is obtained via Young’s inequality (see, e.g., Eq.(13) in Appendix A of Mücke et al.
[24]), and is therefore stated as an upper bound on the SGD risk.
Our next theorem is to characterize the fluctuation error of SGD (with respect to GD).
Theorem 4.2 (Fluctuation error bound) Suppose that Assumptions 3.1A, B and D all hold.
Thenfor every n ≥ 1 , t ≥ 1 and η ≤ c/ tr( H ) for some absolute constant c , with probability at least1 − 1 / poly( n ) , it holds that [Equation] k † [Equation]
i
[Equation]
Σ [Equation] , where k † ≥ 0 is an arbitrary index (can be infinity).
We first explain the factor min (cid:8) ∥ (cid:98) w ∥ 22 , tη · ∥ (cid:98) w ∥ 2 Σ (cid:9) in our bound.
First of all, when the interpolator(cid:98) w has a small ℓ 2 -norm, the quantity is automatically small.
Furthermore, ∥ (cid:98) w ∥ 2 Σ ≲ ω 2 ≲ 1 easilyholds under mild assumptions on w ∗ , e.g., Assumption 3.1C. Then, for finite t one can bound thefactor with min (cid:8) ∥ (cid:98) w ∥ 22 , tη · ∥ (cid:98) w ∥ 2 Σ (cid:9) ≲ ω 2 ηt .
More interestingly, for SGD with constant stepsizeand infinite optimization steps ( t → ∞ ), our risk bound can still vanish, while all risk bounds in priorworks [21, 27, 24] are vacuous.
To see this, one can consider a sequence of k † , e.g., k † = √ t , then itis clear that log( t )
k
† ( t ) /t, (cid:80) i>k † ( t ) λ i → 0 when t → ∞ , so the fluctuation error vanishes.
To complement the above results, we provide the following finite-time risk bound for GD.
Nonetheless,we emphasize that any risk bound for GD can be plugged into Theorems 4.1 and 4.2 to obtain a riskbound for SGD.
Theorem 4.3 (GD risk) Suppose that Assumptions 3.1A, B and C all hold.
Then for every n ≥ 1 ,t ≥ 1 and η < 1 / ∥ H ∥ 2 , with probability at least 1 − 1 / poly( n ) , it holds that [Equation] λ [Equation]
n 2 · (cid:88) i ≤ k
[Equation]
λ i + (cid:88)
i>k ∗ λ i (cid:33) + σ [Equation] · (cid:32) k
[Equation] n + n (cid:101) λ(cid:88) i>k ∗ λ 2 i (cid:33) , where k ∗ : = min { k : nλ k +1 ≤ nηt + (cid:80) i>
k λ i } and (cid:101) λ :
= nηt + (cid:80) i>k ∗ λ i .
The bound presented in Theorem 4.3 is comparable to that for ridge regression established by Tsiglerand Bartlett [32] and will be much better than the bound of single-pass SGD when the signal-to-noiseratio is large [41, Theorem 5.1], e.g., ω 2 ≫ σ 2 .
In fact, Theorem 4.3 is proved via a reduction toridge regression results.
In particular, the quantity n/ ( ηt ) for GD is an analogy to the regularizationparameter λ for ridge regression [38, 28, 35, 2].
As a final remark, the assumption that w ∗ follows a
Gaussian prior is the main concealing in Theorem 4.3 (which is not required by Tsigler and Bartlett[32] for ridge regression).
The Gaussian prior on w ∗ is known to allow a connection between early stopped GD with ridge regression [2].
We conjecture that this assumption is not necessary andpotentially removable.
Combining Theorems 4.1, 4.2 and 4.3, we obtain the following excess risk bound for multi-pass SGDin the interpolating least square problems:
Corollary 4.4 Suppose that Assumptions 3.1A, B, C and D all hold.
Then with probability at least1 − 1 / poly( n ) , it holds that
[Equation]
where k † is and arbitrary index, k ∗ :
= min { k : nλ k +1 ≤ nηt + (cid:80) i>
k λ i } and (cid:101) λ :
= nηt + (cid:80) i>k ∗ λ i .
Comparison with Existing Results.
We now discuss relationships between our bound and existingones for multi-pass SGD [21, 27, 24].
First, we highlight that our bound is problem-dependent in thesense that the bound is stated as a function of the spectrum of data covariance; in contrast, existingpapers only provide a minimax analysis for multi-pass SGD.
Secondly, we rely on a different setof assumptions from the aforementioned papers.
In particular, Pillaud-Vivien et al.
[27] requiresa source condition on the data covariance, and Lin and Rosasco [21], Mücke et al.
[24] require aneffective dimension (defined by the data covariance) to be small, but our results are more generalregarding the data covariance.
Moreover, we assume w ∗ follows a Gaussian prior (Assumption 3.1C),but existing works require a source condition on w ∗ , which are not directly comparable.
Bartlett et al.
[4] showed that OLS generalizes in the so called benign overfitting regime.
Since whent → ∞ , the SGD solution (with constant stepsize) converges to OLS, it would be interesting tocompare the SGD solution with OLS in such a regime.
We will do so with the following Corollary4.5.
Corollary 4.5 Suppose that Assumptions 3.1A, B, C and D all hold.
Assume the spectrum of Hsatisfies λ i
= i − 1 log
( i + 1) − β for some absolute constant β > 1 , then with probability at least1 − 1 / poly( n ) , there exists a choice of t and η such that
[Equation]
Besides, for any fixed stepsize η , we have
[Equation]
As a sanity check, our bound for t → ∞ matches the upper and lower bounds on the excess riskof OLS (which can be obtained by setting λ = 0 in Theorem 1 and Lemmas 2 & 3 in Tsigler and
Bartlett [32]).
Moreover, Corollary 4.5 suggests that the excess risk achieved by multipass SGD isalways no worse than that of OLS, and could be strictly smaller than that of OLS when β > 0 .Thisdemonstrates
the benefit of multi-pass SGD over OLS.
The following corollary characterizes the risk of multi-pass SGD for data covariance with a polynomially decaying spectrum.
Corollary 4.6 Suppose that Assumptions 3.1A, B, C and D all hold.
Assume the spectrum of Hdecays polynomially, i.e., λ i
= i − 1 − r for some absolute constant r > 0 , then with probability atleast 1 − 1 / poly( n ) , it holds that
[Equation]
Risk
S t e p s
GD, i = i 1 log 2 ( i +10)
SGD, i = i 1 log
( i +10)
GD, i = i
SGD, i = i(a) Iteration Complexity vs. Risk 0.2 0.4 0.
Risk
G r a d i e n t
C o m p u t a t i o n
GD, i = i 1 log 2 ( i +10)
SGD, i = i 1 log
( i +10)
GD, i = i
SGD, i = i
[Figure]
Corollary 4.6 provides concrete excess risk bounds for SGD and GD, based on which we can make acomparison between SGD and GD in terms of their iteration and gradient complexities.
For simplicity,in the following discussion, we assume that ω 2 ≂ σ 2 ≂ 1 .
Then choosing tη ≂ n minimizes theupper bound for GD risk and yields the O ( n − r/ ( r +1) ) rate.
Here GD can employ a constant stepsize.
Similarly, SGD can match the GD’s rate, O ( n − r/ ( r +1) ) , by setting tη ≂ n and
[Equation]
The above implies that SGD (fixed stepsize, last iterate) can only cooperate with small stepsize.
Iteration Complexity.
We first compare GD and SGD in terms of the iteration complexity.
To reachthe optimal rate, GD can employ a constant stepsize and set the number of iterates to be t ≂ n .
However, in order to shelve the fluctuation error, the stepsize of SGD cannot be large, as required by(4.1).
More precisely, in order to match the optimal rate, SGD needs to use a small stepsize, η ≂ n/t ,with a large number of iterates,
[Equation]
It can be seen that the iteration complexity of SGD is much worse than that of GD.
This result isempirically verified by Figure 2 (a).
Gradient Complexity.
We next compare GD and SGD in terms of the gradient complexity.
Recallthat for each iterate, GD computes n gradients but SGD only computes 1 gradient.
Therefore, to reachthe optimal rate, the total number of gradient computed by GD needs to be Θ( n 2 ) , but that computedby SGD is only (cid:101) O ( n max { (2 r +1) /
( r +1) , 1 . 5 } ) .
Thus, the gradient complexity of SGD is better than thatof GD by a factor of (cid:101) O
( n min { 0 . 5 , 1 / ( r +1) } ) .
This result is empirically verified by Figure 2 (b).
Overview of the Proof Technique
In this section, we will provide an overview of our proof technique and sketch the proof of Theorems4.1 and 4.2.
The remaining proof is deferred to Appendix.
Our proof technique is inspired by the operator methods for analyzing single-pass SGD [3, 10, 17,18, 26, 13, 40, 36].
In particular, they track an error matrix , ( w t − w ∗ ) ⊗ ( w t − w ∗ ) that keepsricher information than the error norm ∥ w t − w ∗ ∥ 22 .
For single-pass SGD where each data is usedonly once, the resulted iterates enjoy a simple dependence on history that allows an easy calculationof the expected error matrix (with respect to the randomness of data generation).
However formulti-pass SGD, a data might be used multiple times, which prevents us from tracking the expectederror matrix directly.
Instead, a trackable analogy to the error matrix is the empirical error matrix ,( w t − (cid:98) w ) ⊗ ( w t − (cid:98) w ) where (cid:98) w is the minimum norm interpolator.
More precisely, note that
[Equation]
Therefore the expected (over the algorithm’s randomness) empirical error matrix updates as follows:let E t :=
E SGD (cid:2) ( w t − (cid:98) w )( w t − (cid:98) w ) ⊤ (cid:3) , then E t +1 =
E
i t (cid:2) ( I − η x i t x ⊤ i t )
E t ( I − η x i t x ⊤ i t ) (cid:3) .
Let Σ := 1 n X ⊤ X be the empirical covariance matrix.
We then follow the operator method [40] todefine the following operators on symmetric matrices (e.g., J ):
[Equation]
Based on these operators, we can obtain a close form update rule for E t :
[Equation]
Here the first term Θ 1 :
= ( I − η Σ )
[Equation]
η Σ
[Equation]
w
[Equation]
−
[Equation]
w )
[Equation]
(cid:98) w t − (cid:98) w ) ⊤ is exactly the errormatrix caused by GD iterates (with stepsize η and iteration number t ), and the second term Θ 2 is afluctuation matrix that captures the deviation of w t with respect to a corresponding GD iterate (cid:98) w t .
We remark that the expected error matrix E t contains all information of w t .
Risk Decomposition (Theorem 4.1).
The following fact is clear from the update rule (5.1).
Fact 5.1 The GD iterates satisfy (cid:98) w t +1 − (cid:98) w
=
( I − η Σ )( (cid:98) w t − (cid:98) w ) and E SGD [ w t − (cid:98) w ] = (cid:98) w t − (cid:98) w .
Based on Fact 5.1 and (5.2), we have
[Equation]
[Equation]
where Θ 1 and Θ 2 are defined in (5.2) and the last equality is due to Θ
= ( (cid:98) w t − (cid:98) w )( w t − (cid:98)
[Equation]
⊤ .
Also note that
[Equation]
Combining these two inequalities proves Theorem 4.1:
[Equation]
Fluctuation
[Equation]
Finally, the fluctuation error is non-negative because both G and M − (cid:102) M are PSD mappings.
Bounding the Fluctuation Error (Theorem 4.2).
There are several challenges in the analysis offluctuation error: (1) it is difficult to characterize the matrix ( M − (cid:102) M ) ◦ E k since the matrix
E kis unknown; (2) the operator G involves an exponential decaying term with respect to the empiricalcovariance matrix Σ , which does not commute with the population covariance matrix H .
To address the first problem, we note that the operators (cid:102) M , G , M are PSD mapping and enjoycommutative property and then obtain the following result:
[Equation]
Now, the input of the operator M ◦ G t − 1 − k will not be an unknown matrix but a fixed one (i.e., H ),and the remaining effort will be focusing on characterizing M ◦ G k ◦ H .
Applying the definitions of
M and G implies
[Equation]
Then our idea is to first prove an uniform upper bound on the quantity
x ⊤ i
( I − η Σ ) k H ( I − η Σ )
k x ifor all i ∈ [ n ] (e.g., denoted as U ( k, η, n ) ), then it can be naturally obtained that
[Equation]
then we will only need to characterize the inner product ⟨ E k , Σ ⟩ in (5.4), which can be understoodas the optimization error at the k -th iteration.
In order to precisely characterize U ( k, η, n ) , we encounter the second problem that the populationcovariance H and empirical covariance Σ are not commute, thus the exponential decaying term( I − η Σ ) k will not be able to fully decrease H since some components of H may lie in the smalleigenvalue directions of Σ .
Therefore, we consider the following decomposition
[Equation]
Then for Θ 1 , it can be seen that the decaying term ( I − η Σ ) k is commute with Σ thus can successfullymake it decrease.
For Θ 2 , we will view the difference H − Σ as the component of H that cannot beeffectively decreased by ( I − η Σ ) k , which will be small as n increases.
More specifically, we canget the following upper bound on Θ 1 .
Lemma 5.
If the stepsize satisfies γ ≤ c/ tr( H ) for some small absolute constant c , then withprobability at least 1 − 1 / poly( n ) , it holds that Θ 1 ≲ tr( H ) · log( n ) · min (cid:8)
[Equation]
2 (cid:9) .
For Θ 2 , we will rewrite x i as e ⊤ i X where e
i ∈ R n and X ∈ R n × d , then
[Equation]
Then since X and Σ have the same column eigenspectrum, we can fully unleash the decayingpower of the term ( I − η Σ ) k on X .
Further note the that the row space of X is uniform distributed(corresponding to the index of training data), which is independent of e i .
This implies that we canadopt standard concentration arguments with covering on n fixed vectors { e i } ni =1 to prove a sharphigh probability upper bound (compared to the naive worst-case upper bound).
Consequently, westate the upper bound on Θ 2 in the following lemma.
Lemma 5.
For every i ∈ [ n ] and k ∗ ∈ [ d ] , it holds with probability at least 1 − 1 / poly( n ) thatΘ
[Equation]
Conclusion and Discussion
In this paper, we establish an instance-dependent excess risk bound of multi-pass SGD for interpolating least square problems.
The key takeaways include: (1) the excess risk of SGD is alwaysworse than that of GD, given the same setup of stepsize and iteration number; (2) in order to achievethe same level of excess risk, SGD requires more iterations than GD; and (3) however, the gradientcomplexity of SGD can be better than that of GD.
The proposed technique for analyzing multi-pass
SGD could be of broader interest.
Several interesting problems are left for future exploration:
A problem-dependent excess risk lower bound could be useful to help understand the sharpness ofour excess risk upper bound for multi-pass SGD and establish a clear separation between SGD and
GD in terms of iteration and gradient complexity.
The challenge here is mainly from the fact thatthe empirical covariance matrix Σ does not commute with the population covariance matrix H .
Inparticular, one needs to develop an even sharper characterization on the quantity M ◦ G k ◦ H (see(5.5)); more precisely, a sharp lower bound on x ⊤ i
( I − η Σ ) k H ( I − η Σ )
k x i is required.
SGD with decaying stepsizes could potentially improve the generalization performance of SGD witha constant stepsize.
In this regard, most of our analysis can be extended to SGD/GD with decayingstepsizes.
For example, Theorem 4.1 directly holds even for varying stepsizes, and Theorem 4.3also holds under a small modification, i.e., changing tη to (cid:80) t − 1 k =0 η k .
However, our bounds on thefluctuation error may be more subtle to adapt.
We conjecture that our analysis idea can still be applied,but the detailed calculations will depend on the particular stepsize scheduler of interests.
Multi-pass SGD without replacement is a more practical SGD variant than the multi-pass SGD withreplacement studied in this work.
The key difference is that, the former does not pass training dataindependently (since each data must be used for equal times).
In terms of optimization complexity, ithas already been demonstrated in theory that multi-pass SGD without replacement (e.g., SGD withsingle shuffle or random shuffle) outperforms multi-pass SGD with replacement [15, 30, 1].
However,in terms of generalization, whether or not multi-pass SGD without replacement can outperformmulti-pass SGD with replacement is still an open problem, as there lacks a sharp excess risk analysisfor multi-pass SGD without replacement.
The techniques presented in this paper can shed light onthis direction.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
[No] The reported mean behaviors in our experiments are
sufficient to support our theoretical results
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No] Resources are not a concern for our
purpose of numerically verifying our theoretical results
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[-B.
-]
[-Proof of Theorem 4.
-]
[-In this part we seek to bound (cid:80) t − 1 k =
-]
[-E 
-]
[-Lemma C.
-]
[-4 Proof of Theorem 4.
-]
spent on participant compensation?
[N/A]