Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime
Abstract
Stochastic gradient descent (SGD) has achieved great success due to its superior performance in both optimization and generalization.
Most of existing generalization analyses are made for single-pass SGD, which is a less practical variant compared to the commonly-used multi-pass SGD.
Besides, theoretical analyses for multi-pass SGD often concern a worst-case instance in a class of problems, which may be pessimistic to explain the superior generalization ability for some particular problem instance.
The goal of this paper is to provide an instance-dependent and algorithm-dependent excess risk bound of multi-pass SGD for least squares in the interpolation regime, which is expressed as a function of the iteration number, stepsize, and data covariance.
We show that the excess risk of SGD can be exactly decomposed into the excess risk of GD and a positive fluctuation error, suggesting that SGD always performs worse, instance-wisely, than GD, in generalization.
On the other hand, we show that although SGD needs more iterations than GD to achieve the same level of excess risk, it saves the number of stochastic gradient evaluations, and therefore is preferable in terms of computational time.
Introduction
Stochastic gradient descent (SGD) is one of the workhorses in modern machine learning due to its efficiency and scalability in training and good ability in generalization to unseen test data.
From the optimization perspective, the efficiency of SGD is well understood.
For example, to achieve the same level of optimization error, SGD saves the number of gradient computations compared to its deterministic counterpart, i e., batched gradient descent (GD) [7, 8] and therefore saves the total amount of running time.
However, the generalization ability (e.g., excess risk bounds) of SGD is far less clear, especially from theoretical perspective.
Single-pass SGD, a less practical SGD variant where each training data is used only once, has been extensively studied in theory.
In particular, a series of works establishes excess risk bounds of single-pass SGD for learning general smooth and convex objectives [32, 24, 26] as well as learning least squares [3, 10, 18, 19, 27, 13, 42, 38].
In practice, though, one often runs SGD with multiple passes over the training data and outputs the final iterate, which is referred to as multi-pass SGD (or simply SGD in the rest of this paper when there is no confusion).
Compared to single-pass SGD that has limited number of optimization steps, multi-pass SGD allows the algorithm to perform arbitrary number of optimization steps, which is more powerful in optimizing the empirical risk and thus leads to smaller bias error [28].
Despite the extensive application of multi-pass SGD in practice, there are only a few theoretical techniques being developed to study the generalization of multi-pass SGD.
One is based on the uniform stability [12, 16], which is defined as the change of the model outputs under a small change in the training data.
However, the stability based generalization bound is a worst-case guarantee, which is relatively crude and does not show difference between GD and SGD (See, e.g., Chen et al.
[9] showed GD and SGD have the same stability parameter in the convex smooth setting).
On the contrary, one easily observes a generalization difference between SGD and GD even in learning the simplest least square problem (see Figure 1).
In addition, Lin and Rosasco [22], Pillaud-Vivien et al.
[28], Mücke et al.
[25] explored the risk bounds for multi-pass SGD using the operator methods that are originally developed for analyzing single-pass SGD.
Their bounds are sharp in the minimax sense for a class of least square problems that satisfy certain source condition (which restricts the norm of the optimal parameter) and capacity condition (or effective dimension, which restricts the spectrum of the data covariance matrix).
Still, their bounds are uniform for a class of problem instances, and cannot point-wisely adapt to each problem instance (e.g., a least square problem with a particular data covariance matrix).
In particular their reults are pessimistic for the benign-overfitting [4] least square instances (see Theorem 4.2 and related discussions).
In this paper, our goal is to establish an algorithm-dependent and problem-dependent excess risk bound of multi-pass SGD for least squares.
Our focus is the interpolation regime where the training data can be perfectly fitted by a linear interpolator (which holds almost surely when the number of parameter d exceeds the number of training data n ).
We assume the data has a sub-Gaussian tail [4].
Our main contributions are summarized as follows:
• We show that for any iteration number and stepsize, the excess risk of SGD can be exactly decomposed into the excess risk of GD (with the same stepsize and iteration number) and the so-called fluctuation error , which is attributed to the accumulative variance of stochastic gradients in all iterations.
This suggests that GD (with optimally tuned hyperparameters) always achieves a smaller excess risk than SGD for least square problems.
• We further establish problem-dependent bounds for the excess risk of GD and the fluctuation error, stated as a function of the eigenspectrum of the data covariance, iteration number, training sample size, and stepsize.
Compared to the bounds proved in prior works [22, 28, 25], our bounds allow a wider range of iteration numbers t , and correctly vanishes when t → ∞ in the benign overfitting regime [4].
In contrast, the prior results do not allow t → ∞ .
• We develop a new suite of proof techniques for analyzing the excess risk of multi-pass SGD.
The key idea is considering the error in its matrix form and how it is updated based on the tensor operators defined by the second-order and fourth-order moments of the empirical data distribution (i.e., sampling with replacement from the training dataset), rather than the operators used in the single-pass SGD analysis that are defined based on the population data distribution [18, 42], together with a sharp characterization on the properties of the operators.
Based on the excess risk upper bounds for SGD and GD we make the following complexity comparison between SGD and GD: to achieve the same order of excess risk, while SGD may need more iterations than GD, it can have fewer stochastic gradient evaluations than GD.
For example, consider the case that the data covariance matrix has a polynomially decaying spectrum with rate i − (1+ r ) , where r > 0 is an absolute constant.
In order to achieve the same order of excess risk, we have the following comparison in terms of iteration complexity and gradient complexity 2 :
• Iteration Complexity: SGD needs to take (cid:101) O ( n max { 0 . 5 , rr +1 } ) more iterations than GD, with optimally tuned iteration number and stepsize.
Gradient Complexity: SGD needs (cid:101) O
( n max { 0 . 5 , 1 r +1 } ) less stochastic gradient evaluations than GD.
Notation.
For n > 0 , we use poly( n ) to define some positive high-degree polynomial functions of n .
For two positive-value functions f ( x ) and g ( x ) we write f ( x ) ≲ g ( x ) if f ( x ) ≤ cg ( x ) for some constant c > 0 , we write f ( x ) ≳
g ( x ) if g ( x ) ≲ f ( x ) , and f ( x ) ≂ g ( x ) if both f ( x ) ≲ g ( x ) and g ( x ) ≲ f ( x ) hold.
We use (cid:101) O ( · ) to hide some polylogarithmic factors in the standard
big-
O notation.
For two matrices A and B , we denote ⟨ A , B ⟩ = (cid:80) i,j A ij B ij and A ⊗ B as their Kronecker product.
[Figure]
Related Work
Optimization.
Regarding optimization efficiency, the benefit of SGD is well understood [7, 8, 23, 5, 34, 35, 26].
For example, for strongly convex losses (can be relaxed with certain growth conditions), GD has less iteration complexity, but SGD enjoys less gradient complexity [7, 8].
More recently, it is shown that SGD can converge at an exponential rate in the interpolating regime [26, 23, 5, 34, 35], therefore SGD can match the iteration complexity of GD.
Nevertheless, all the above results are regrading the optimization performance; our focus in this paper is to study the generalization performance of SGD (and GD).
Risk Bounds for Multi-Pass SGD.
The risk bounds of multi-pass SGD are also studied from the operator perspective [30, 22, 28, 25].
The work by Rosasco and Villa [30] focused on cyclic SGD , i.e., SGD with multiple passes but fixed sequence on the training data.
Their results are limited to small stepsizes ( γ = O (1 /n ) ), while ours allow constant stepsize.
Similar to Lin and Rosasco [22], Pillaud-Vivien et al.
[28], Mücke et al.
[25], we decompose the population risk of SGD iterates into a risk term caused by batch GD iterates and a fluctuation error term between SGD and GD iterates.
But our methods of bounding the fluctuation error are different (see more in Section 5).
Moreover, our results are based on different assumptions:
Lin and Rosasco [22], Pillaud-Vivien et al.
[28], Mücke et al.
[25] assumed finiteness on the optimal parameter, and their results only apply to data covariance with a specific type of spectrum (nearly polynomially decaying ones); in contrast, our results assume a Gaussian prior on the optimal parameter (which might not admit a finite norm), and our results are stated as a function of the entire eigenspectrum of the data covariance, thus cover more general data covariance (including those with polynomially decaying spectrum).
Lei et al.
[21] studied risk bounds for multi-pass SGD with general convex loss.
When applied to least square problems, their bounds are cruder than ours.
Uniform Stability.
Another approach for characterizing the generalization of multi-pass SGD is through uniform stability [16, 9, 20, 41, 6].
There are mainly two differences between this and our approach.
First, we directly bound the excess risk of SGD; but the uniform stability can only bound the generalization error, there needs an additional triangle inequality to relate excess risk with generalization error plus optimization error (plus approximation error) — this inequality can easily be loose (consider the algorithmic regularization effects).
Secondly, the uniform stability bound is also crude.
For example, in the non-strongly convex setting, the uniform stability bound for
SGD/GD linearly scales with the total optimization length (i.e., sum of stepsizes), which grows as t [16, 9, 20, 41, 6] (this is minimaxly unavoidable according to Zhang et al.
[41], Bassily et al.
Notably, Bassily et al.
[6] extended the uniform stability approach to the non-convex and smooth setting.
We left such an extension of our method as a future work.
Problem Setup
Let x be a feature vector in a Hilbert space H (its dimension is denoted by d , which is possibly infinite) and y ∈ R be its response, and assume that they jointly follow an unknown population distribution D .
In linear regression problems, the population risk of a parameter w is defined by
[Equation]
and the excess risk is defined by
[Equation]
where H :=
E D [ xx ⊤ ] , (3.1) where w ∗ =
arg min w L D ( w ) denotes the global minimizer of the population risk.
Additionally, following Zou et al.
[42, 43], we assume that the data covariance matrix H is positive definite.
In the statistical learning setting, the population distribution D is unknown, and one is provided with a set of n training samples,
S =
( x i , y i ) ni
=1 , that are drawn independently at random from the population distribution.
We also use X :=
( x 1 , . . .
, x n ) ⊤ and y :
= ( y 1 , . .
. , y n ) ⊤ to denote the concatenated features and labels, respectively.
The linear regression problems aim to find a parameter based on the training set S that affords a small excess risk.
Multi-Pass SGD.
We are interested in solving the linear regression problem using multi-pass stochastic gradient descent (SGD) 3 with a constant learning rate.
The algorithm generates a sequence of iterates ( w t ) t ≥ 1 according to the following update rule: the initial iterate is w 0 =
(which can be assumed without loss of generality); then at each iteration, an example ( x i t , y i t ) is drawn from S uniformly at random, and the iterate is updated by
[Equation]
where η > 0 is a constant stepsize (i.e., learning rate).
GD.
Another popular algorithm is gradient descent (GD).
For the clarity of notations, we use ( (cid:98) w t ) t ≥to denote the GD iterates, which follow the following updates:
[Equation]
where η > 0 is a constant stepsize.
Definitions and Assumptions.
The eigenvalues of the population data covariance H is denoted by ( λ i )
i ≥ 1 , sorted in non-increasing order.
Given the training data ( X , y ) , we define ϵ = y −
Xw ∗ the collection of model noise, A = XX ⊤ as the Gram matrix, and Σ = n − 1 X ⊤ X as the empirical covariance.
Then the minimum-norm solution is defined by
[Equation]
It is clear that in the interpolation regime, with appropriate stepsizes, both SGD and GD algorithms converge to (cid:98) w [14, 4].
The assumptions required by our theorems are summarized in below.
Assumption 3.1 For the linear regression problem:
A The components of H − 1 / 2 x are independent and 1 -subGaussian.
B
The response y is generated by y := ⟨ w ∗ , x ⟩ + ξ , where w ∗ is the ground truth weight vector and ξ is a noise independent of x .
Furthermore, the additive noise satisfies E [ ξ ] = 0 , E [ ξ 2 ] ≤ σ 2 .
C
The ground truth w ∗ follows a Gaussian prior N ( 0 , ω 2 · I ) , where ω 2 is a constant.
D The minimum-norm solution (cid:98) w linearly interpolates all training data,
i
e., y i = (cid:98) w ⊤ x i for i ∈ [ n ] .
Assumptions 3.1A and B are standard for analyzing overparameterized linear regression problem in the benign overfitting regime [4, 33].
Note that Assumption 3.1A is widely made in the analysis of high-dimensional least squares estimations [11, 37, 17].
However, Assumption 3.1A is not standard for analyzing SGD [3, 22, 28, 18, 42].
We conjecture Assumption 3.1A can be relaxed and leave this as a future work.
Moreover, Assumption 3.1C is also widely adopted in analyzing least square problems (see, e.g., Ali et al.
[2], Dobriban et al.
[11], Xu and Hsu [39]).
There are also many different conditions being made on the ground truth w ∗ in existing works [28] to study the generalization of SGD (e.g., ∥ H 1 / 2 − r w ∗ ∥ 2 → ∞ for r ≥ 0 ).
However, they are not directly comparable to
Assumption 3.1C. Finally, Assumption 3.1D holds almost surely when d > n , i.e., the number of parameter exceeds the number of data.
In the following, the presented risk bounds will hold (i) with high-probability with respect to the randomness of sampling feature vectors X , and (ii) in expectation with respect to the randomness of multi-pass SGD algorithm, the randomness of sampling additive noise ϵ and the randomness of the true parameter w ∗ as a prior.
For these purpose, we will use E i t , E SGD , E w ∗ to refer to taking expectation with respect to the randomness of sampling data (from the training set) at the t -th iteration, the randomness of the entire SGD algorithm (i.e., sampling data at each iteration, i 1 , . . .
, i t , . . . )
and the prior distribution of w ∗ , respectively.
Main Results
Our first theorem shows that, under the same stepsize and number of iterates, SGD always generalizes worse than GD.
Theorem 4.1 (Risk decomposition) Suppose that Assumption 3.1D holds.
Then the excess risk of
SGD can be decomposed by
[Equation]
Moreover, the fluctuation error is always non-negative.
A Risk Comparison.
Theorem 4.1 shows that, in the interpolation regime, SGD affords a strictly larger excess risk than GD, given the same hyperparameters (stepsize η and number of iterates t ).
Therefore, despite of a possibly higher computational cost, the optimally tuned GD dominates the optimally tuned SGD in terms of the generalization performance.
This observation is verified empirically by experiments in Figure 1.
Theorem 4.1 relates the risk of SGD iterates to that of GD iterates.
This idea has appeared in earlier literature [22, 28, 25].
However, their decomposition is obtained via Young’s inequality (see, e.g., Eq.
(13) in Appendix A of Mücke et al.
[25]), and is therefore stated as an upper bound on the SGD risk.
Our next theorem is to characterize the fluctuation error of SGD (with respect to GD).
Theorem 4.2 (Fluctuation error bound) Suppose that Assumptions 3.1A, B and D all hold.
Then for every n ≥ 1 , t ≥ 1 and η ≤ c/ tr( H ) for some absolute constant c , with probability at least 1 − 1 / poly( n ) , it holds that
FluctuationError
[Equation]
where k † ≥ 0 is an arbitrary index (can be infinity).
We first explain the factor min (cid:8) ∥ (cid:98) w ∥ 22 , tη · ∥ (cid:98) w ∥ 2 Σ (cid:9) in our bound.
First of all, when the interpolator (cid:98) w has a small ℓ 2 -norm, the quantity is automatically small.
Furthermore, ∥ (cid:98) w ∥ 2 Σ ≲ ω 2 ≲ 1 easily holds under mild assumptions on w ∗ , e.g., Assumption 3.1C. Then, for finite t one can bound the factor with min (cid:8) ∥ (cid:98) w ∥ 22 , tη · ∥ (cid:98) w ∥ 2 Σ (cid:9) ≲ ω 2 ηt .
More interestingly, for SGD with constant stepsize and infinite optimization steps ( t → ∞ ), our risk bound can still vanish, while all risk bounds in prior works [22, 28, 25] are vacuous.
To see this, one can consider a sequence of k † , e.g., k † = √ t , then it is clear that log( t ) k † ( t ) /t, (cid:80) i>k † ( t )
λ i → 0 when t → ∞ , so the fluctuation error vanishes.
To complement the above results, we provide the following risk bound for GD.
We emphasize that any risk bound for GD can be plugged into Theorems 4.1 and 4.2 to obtain a risk bound for SGD.
Theorem 4.3 (GD risk) Suppose that Assumptions 3.1A, B and C all hold.
Then for every n ≥ 1 , t ≥ 1 and η < 1 / ∥ H ∥ 2 , with probability at least 1 − 1 / poly( n ) , it holds that
[Equation]
where k
[Equation]
:= min { k : nλ
[Equation]
≤ nηt
[Equation]
} and
[Equation]
The bound presented in Theorem 4.3 is comparable to that for ridge regression established by Tsigler and Bartlett [33] and will be much better than the bound of single-pass SGD when the signal-to-noise ratio is large [43, Theorem 5.1], e.g., ω 2 ≫ σ 2 .
In fact, Theorem 4.3 is proved via a reduction to ridge regression results.
In particular, the quantity n/ ( ηt ) for GD is an analogy to the regularization parameter λ for ridge regression [40, 29, 36, 2].
As a final remark, the assumption that w ∗ follows a
Gaussian prior is the main concealing in Theorem 4.3 (which is not required by Tsigler and Bartlett [33]).
The Gaussian prior on w ∗ is known to allow a connection between early stopped GD with ridge regression [2].
We conjecture that this assumption is not necessary and potentially removable.
Combining Theorems 4.1, 4.2 and 4.3, we obtain the following risk bound for multi-pass SGD:
Corollary 4.4 Suppose that Assumptions 3.1A, B, C and D all hold.
Then with probability at least 1 − 1 / poly( n ) , it holds that
[Equation]
where k † is an arbitrary index, k ∗ :
= min { k : nλ k +1 ≤ nηt + (cid:80) i>
k λ i } and (cid:101) λ :
= nηt + (cid:80) i>k ∗ λ i .
Comparison with Existing Results.
We now discuss relationships between our bound and existing ones for multi-pass SGD [22, 28, 25].
First, we highlight that our bound is problem-dependent in the sense that the bound is stated as a function of the spectrum of data covariance; in contrast, existing papers only provide a minimax analysis for multi-pass SGD.
Secondly, we rely on a different set of assumptions from the aforementioned papers.
In particular, Pillaud-Vivien et al.
[28] requires a source condition on the data covariance (e.g. ∥ H 1 / 2 − r w ∗ ∥ 2 < ∞ for some constant r ≥ 0 ), and
Lin and Rosasco [22], Mücke et al.
[25] require an effective dimension to be small, but our results are more general regarding the data covariance.
Moreover, we assume w ∗ follows a Gaussian prior (Assumption 3.1C), which is also not directly comparable to the source condition in existing works.
Bartlett et al.
[4] showed that OLS generalizes in the so called benign overfitting regime.
Since when t → ∞ , SGD (with constant stepsize) converges to OLS, it would be interesting to compare the SGD solution with OLS in such a regime.
We will do so with the following Corollary 4.5.
Corollary 4.5 Suppose that Assumptions 3.1A, B, C and D all hold.
Assume the spectrum of H satisfies λ
i =
i − 1 log( i + 1) − β for some absolute constant β > 1 , then with probability at least 1 − 1 / poly( n ) , there exists a choice of t and η such that
[Equation]
Besides, for any fixed stepsize η , we have
[Equation]
As a sanity check, our bound for t → ∞ matches the upper and lower bounds on the excess risk of OLS (which can be obtained by setting λ = 0 in Theorem 1 and Lemmas 2 & 3 in Tsigler and
Bartlett [33]).
Moreover, Corollary 4.5 suggests that the excess risk achieved by multipass SGD is always no worse than that of OLS, and could be strictly smaller than that of OLS when β > 0 .This demonstrates the benefit of multi-pass SGD over OLS.
The following corollary characterizes the risk of multi-pass SGD for data covariance with a polynomially decaying spectrum.
Corollary 4.6 Suppose that Assumptions 3.1A, B, C and D all hold.
Assume the spectrum of H decays polynomially, i.e., λ i
= i − 1 − r for some absolute constant r > 0 , then with probability at least 1 − 1 / poly( n ) , it holds that
[Equation]
Risk
S t e p s
GD, i = i 1 log 2 ( i +10)
SGD, i = i 1 log
( i +10)
GD, i = i
SGD, i = i(a) Iteration Complexity vs. Risk 0.2 0.4 0.
Risk
G r a d i e n t
C o m p u t a t i o n
GD, i = i 1 log 2 ( i +10)
SGD, i = i 1 log
( i +10)
GD, i = i
SGD, i = i
[Figure]
[Equation]
Corollary 4.6 provides concrete excess risk bounds for SGD and GD, based on which we can make a comparison between SGD and GD in terms of their iteration and gradient complexities.
For simplicity, in the following discussion, we assume that ω 2 ≂ σ 2 ≂ 1 .
Then choosing tη ≂ n minimizes the upper bound for GD risk and yields the O ( n − r/ ( r +1) ) rate.
Here GD can employ a constant stepsize.
Similarly, SGD can match the GD’s rate, O ( n − r/ ( r
[Equation]
1) ) , by setting tη ≂ n and
[Equation]
The above implies that SGD (fixed stepsize, last iterate) can only cooperate with small stepsize.
Iteration Complexity.
We first compare GD and SGD in terms of the iteration complexity.
To reach the optimal rate, GD can employ a constant stepsize and set the number of iterates to be t ≂ n .
However, in order to shelve the fluctuation error, the stepsize of SGD cannot be large, as required by (4.1).
More precisely, in order to match the optimal rate, SGD needs to use a small stepsize, η ≂ n/t , with a large number of iterates,
[Equation]
It can be seen that the iteration complexity of SGD is much worse than that of GD.
This result is empirically verified by Figure 2 (a).
Gradient Complexity.
We next compare GD and SGD in terms of the gradient complexity.
Recall that for each iterate, GD computes n gradients but SGD only computes 1 gradient.
Therefore, to reach the optimal rate, the total number of gradient computed by GD needs to be Θ( n 2 ) , but that computed by SGD is only (cid:101) O
( n max { (2 r +1) /
( r +1) , 1 . 5 } ) .
Thus, the gradient complexity of SGD is better than that of GD by a factor of (cid:101) O
( n min { 0 . 5 , 1 / ( r +1) } ) .
This result is empirically verified by Figure 2 (b).
Overview of the Proof Technique
In this section, we will provide an overview of our proof technique and sketch the proof of Theorems 4.1 and 4.2.
The remaining proof is deferred to Appendix.
Our proof technique is inspired by the operator methods for analyzing single-pass SGD [3, 10, 18, 19, 27, 13, 42, 38].
In particular, they track an error matrix , ( w t − w ∗ ) ⊗ ( w t − w ∗ ) that keeps richer information than the error norm ∥ w t − w ∗ ∥ 22 .
For single-pass SGD where each data is used only once, the resulted iterates enjoy a simple dependence on history that allows an easy calculation of the expected error matrix (with respect to the randomness of data generation).
However for multi-pass SGD, a data might be used multiple times, which prevents us from tracking the expected error matrix directly.
Instead, a trackable analogy to the error matrix is the empirical error matrix , ( w t − (cid:98) w ) ⊗ ( w t − (cid:98) w ) where (cid:98) w is the minimum norm interpolator.
More precisely, note that
[Equation]
Therefore the expected (over the algorithm’s randomness) empirical error matrix updates as follows: let E t :=
E SGD (cid:2) ( w t − (cid:98) w )( w t − (cid:98) w ) ⊤ (cid:3) , then E t +1 =
E
i t (cid:2) ( I − η x i t x ⊤ i t )
E t ( I − η x i t x ⊤ i t ) (cid:3)
Let Σ := 1 n X ⊤ X be the empirical covariance matrix.
We then follow the operator method [42] to define the following operators on symmetric matrices (e.g., J ):
[Equation]
Based on these operators, we can obtain a close form update rule for E t :
[Equation]
(cid:125)
[Equation]
Here the first term Θ 1 :
= ( I − η Σ ) t
E
( I − η Σ ) t = ( (cid:98) w t − (cid:98) w )
( (cid:98) w t − (cid:98) w ) ⊤ is exactly the error matrix caused by GD iterates (with stepsize η and iteration number t ), and the second term Θ 2 is a fluctuation matrix that captures the deviation of w t with respect to a corresponding GD iterate (cid:98) w t .
We remark that the expected error matrix E t contains all information of w t .
Risk Decomposition (Theorem 4.1).
The following fact is clear from the update rule (5.1).
Fact 5.1 The GD iterates satisfy (cid:98) w t +1 − (cid:98) w
=
( I − η Σ )( (cid:98) w t − (cid:98) w ) and E SGD [ w t − (cid:98) w ] = (cid:98) w t − (cid:98) w .
Based on Fact 5.1 and (5.2), we have
[Equation]
where Θ 1 and Θ 2 are defined in (5.2) and the last equality is due to Θ
= ( (cid:98) w t − (cid:98) w )( (cid:98)
[Equation]
t −
[Equation]
w ) ⊤ .
Also note that
[Equation]
Combining these two inequalities proves Theorem 4.1:
[Equation]
Finally, the fluctuation error is non-negative because both G and M − (cid:102) M are PSD mappings.
Bounding the Fluctuation Error (Theorem 4.2).
There are several challenges in the analysis of fluctuation error: (1) it is difficult to characterize the matrix ( M − (cid:102) M ) ◦ E k since the matrix E k is unknown; (2) the operator G involves an exponential decaying term with respect to the empirical covariance matrix Σ , which does not commute with the population covariance matrix H .
To address the first problem, we note that the operators (cid:102) M , G , M are PSD mappings and enjoy commutative property and then obtain the following result:
[Equation]
Now, the input of the operator M ◦ G t − 1 − k will not be an unknown matrix but a fixed one (i.e., H ), and the remaining effort will be focusing on characterizing M ◦ G k ◦ H .
Applying the definitions of
M and G implies
[Equation]
Then our idea is to first prove an uniform upper bound on the quantity
x ⊤ i
( I − η Σ ) k H ( I − η Σ )
k x i for all i ∈ [ n ] (e.g., denoted as U ( k, η, n ) ), then it can be naturally obtained that
[Equation]
then we will only need to characterize the inner product ⟨ E k , Σ ⟩ in (5.4), which can be understood as the optimization error at the k -th iteration.
In order to precisely characterize U ( k, η, n ) , we encounter the second problem that the population covariance H and empirical covariance Σ are not commute, thus the exponential decaying term ( I − η Σ ) k will not be able to fully decrease H since some components of H may lie in the small eigenvalue directions of Σ .
Therefore, we consider the following decomposition x ⊤ i
[Equation]
Then for Θ 1 , it can be seen that the decaying term ( I − η Σ ) k is commute with Σ thus can successfully make it decrease.
For Θ 2 , we will view the difference H − Σ as the component of H that cannot be effectively decreased by ( I − η Σ ) k , which will be small as n increases.
More specifically, we can get the following upper bound on Θ 1 .
Lemma 5.
If the stepsize satisfies γ ≤ c/ tr( H ) for some small absolute constant c , then with probability at least 1 − 1 / poly( n ) , it holds that Θ 1 ≲ tr( H ) · log( n ) · min (cid:8)
[Equation]
For Θ 2 , we will rewrite x i as e ⊤ i X where e
i ∈ R n and X ∈ R n × d , then
[Equation]
Then since X and Σ have the same column eigenspectrum, we can fully unleash the decaying power of the term ( I − η Σ ) k on X .
Further note the that the row space of X is uniform distributed (corresponding to the index of training data), which is independent of e i .
This implies that we can adopt standard concentration arguments with covering on n fixed vectors { e i } ni =1 to prove a sharp high probability upper bound (compared to the naive worst-case upper bound).
Consequently, we state the upper bound on Θ 2 in the following lemma.
Lemma 5.
For every i ∈ [ n ] and k ∗ ∈ [ d ] , it holds with probability at least 1 − 1 / poly( n )
that
[Equation]
Conclusion and Discussion
In this paper, we establish an instance-dependent excess risk bound of multi-pass SGD for interpolating least square problems.
The key takeaways include: (1) the excess risk of SGD is always worse than that of GD, given the same setup of stepsize and iteration number; (2) in order to achieve the same level of excess risk, SGD requires more iterations than GD; and (3) however, the gradient complexity of SGD can be better than that of GD.
The proposed technique for analyzing multi-pass
SGD could be of broader interest.
The code and data for our experiments can be found on Github 4 .
Several interesting problems are left for future exploration:
A problem-dependent excess risk lower bound could be useful to help understand the sharpness of our excess risk upper bound for multi-pass SGD and establish a clear separation between SGD and
GD in terms of iteration and gradient complexity.
The challenge here is mainly from the fact that the empirical covariance matrix Σ does not commute with the population covariance matrix H .
In particular, one needs to develop an even sharper characterization on the quantity M ◦ G k ◦ H (see (5.5)); more precisely, a sharp lower bound on x ⊤ i
( I − η Σ ) k H ( I − η Σ )
k x i is required.
SGD with decaying stepsizes could potentially improve the generalization performance of SGD with a constant stepsize.
In this regard, most of our analysis can be extended to SGD/GD with decaying stepsizes.
For example, Theorem 4.1 directly holds even for varying stepsizes, and Theorem 4.
also holds under a small modification, i.e., changing tη to (cid:80) t − 1 k =0 η k .
However, our bounds on the fluctuation error may be more subtle to adapt.
We conjecture that our analysis idea can still be applied, but the detailed calculations will depend on the particular stepsize scheduler of interests.
Multi-pass SGD without replacement is a more practical SGD variant than the multi-pass SGD with replacement studied in this work.
The key difference is that, the former does not pass training data independently (since each data must be used for equal times).
In terms of optimization complexity, it has already been demonstrated in theory that multi-pass SGD without replacement (e.g., SGD with single shuffle or random shuffle) outperforms multi-pass SGD with replacement [15, 31, 1].
However, in terms of generalization, whether or not multi-pass SGD without replacement can outperform multi-pass SGD with replacement is still an open problem, as there lacks a sharp excess risk analysis for multi-pass SGD without replacement.
The techniques presented in this paper can shed light on this direction.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers and area chairs for their helpful comments.
This work was done when DZ was a Ph.D. student at UCLA.
DZ is partially supported by Bloomberg data science Ph.D. fellowship and his startup funding in the institute of data science, the University of Hong Kong.
JW and VB are supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR00112190130.
QG is partially supported by the National Science Foundation award IIS-1906169 and IIS-2008981.
SK acknowledges funding from the Office of Naval
Research under award N00014-22-1-2377 and the National Science Foundation Grant under award CCF-1703574.
The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.