L EARNING C URVES FOR S TOCHASTIC G RADIENT D ESCENT ON S TRUCTURED F EATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
The generalization performance of a machine learning algorithm such as a neural network depends in an intricate way on the structure of the data distribution.
To analyze the inﬂuence of data structure on test loss dynamics, we study an exactly solveable model of stochastic gradient descent (SGD) which predicts test loss when training on features with arbitrary covariance structure.
We solve the theory exactly for both Gaussian features and arbitrary features and we show that the simpler
Gaussian model accurately predicts test loss of nonlinear random-feature models and deep neural networks trained with SGD on real datasets such as MNIST and
CIFAR-10.
We show that the optimal batch size at a ﬁxed compute budget is typically small and depends on the feature correlation structure, demonstrating the computational beneﬁts of SGD with small batch sizes.
Lastly, we extend our theory to the more usual setting of stochastic gradient descent on a ﬁxed subsampled training set, showing that both training and test error can be accurately predicted in our framework on real data.
I NTRODUCTION
Understanding the dynamics of SGD on realistic learning problems is fundamental to learning theory.
Due to the challenge of modeling the structure of realistic data, theoretical studies of generalization often attempt to derive data-agnostic generalization bounds or study the typical performance of the algorithm on high-dimensional, factorized data distributions (Engel & Van den Broeck, 2001).
Realistic datasets, however, often lie on low dimensional structures embedded in high dimensional ambient spaces (Pope et al., 2021).
For example, MNIST and CIFAR-10 lie on surfaces with intrinsic dimension of ⇠ 14 and ⇠ 35 respectively (Spigler et al., 2020).
To understand the average-case performance of SGD in more realistic learning problems and its dependence on data, model and choice of hyperparameters, incorporating structural information about the learning problem is necessary.
In this paper, we calculate the average case performance of SGD on models of the form f ( x ) = w · ( x ) for nonlinear feature map trained with MSE loss.
We express test loss dynamics in terms of the induced second and fourth moments of . Under a regularity condition on the fourth moments, we show that the test error can be accurately predicted in terms of second moments alone.
We demonstrate the accuracy of our theory on random feature models and wide neural networks trained on MNIST and CIFAR-10 and accurately predict test loss scalings on these datasets.
We explore in detail the effect of minibatch size, m , on learning dynamics.
By varying m , we can interpolate our theory between single sample SGD ( m = 1 ) and gradient descent on the population loss ( m ! 1 ).
To explore the computational advantages SGD compared to standard full batch gradient descent we analyze the loss achieved at a ﬁxed compute budget C = tm for different minibatch size m and number of steps t , trading off the number of parameter update steps for denoising through averaging.
We show that generally, the optimal batch size is small, with the precise optimum dependent on the learning rate and structure of the features.
Overall, our theory shows how learning rate, minibatch size and data structure interact with the structure of the learning problem to determine generalization dynamics.
It provides a predictive account of training dynamics in wide neural networks.
1. O UR C ONTRIBUTIONS
The novel contributions of this work are described below. • We calculate the exact expected test error for SGD on MSE loss for arbitrary feature structure in terms of second and fourth moments as we discuss in Section 4.2. We identify how the structured gradient noise induced by sampling alters the loss curve compared to vanilla GD.
• For Gaussian features (or those with regular fourth moments), we compute the test error in Section 4.1.
This theory is shown to be accurate in experiments with random feature models and wide networks in the kernel regime trained on MNIST and CIFAR-10.
• We show that for ﬁxed compute/sample budgets and structured features with power law spectral decays, optimal batch sizes are small.
We study how optimal batch size depends on the structure of the feature correlations and learning rate.
• We extend our exact theory to study multi-pass SGD on a ﬁxed ﬁnite training set.
Both test and training error can be accurately predicted for random feature models on MNIST.
R ELATED W ORK
The analysis of stochastic gradient descent has a long history dating back to seminal works of Polyak & Juditsky (1992) and Ruppert (1988), who analyzed time-averaged iterates in a noisy problem.
Many more works have examined a similar setting, identifying how averaged and accelerated versions of SGD perform asymptotically when the target function is noisy (not a deterministic function of the input)
(Flammarion & Bach, 2015; 2017; Shapiro, 1989; Robbins & Monro, 1951; Chung, 1954;
Duchi & Ruan, 2021; Yu et al., 2020; Anastasiou et al., 2019; Gurbuzbalaban et al., 2021).
Recent studies have also analyzed the asymptotics of noise-free MSE problems with arbitrary feature structure to see what stochasticity arises from sampling.
Prior works have found exponential loss curves problems as an upper bound (Jain et al., 2018) or as typical case behavior for SGD on unstructured data (Werfel et al., 2004).
A series of more recent works have considered the overparameterized (possibly inﬁnite dimension) setting for SGD, deriving power law test loss curves emerge with exponents which are better than the O ( t  1 ) rates which arise in the noisy problem (Berthier et al., 2020; Pillaud-Vivien et al., 2018; Dieuleveut et al., 2016;
Varre et al., 2021; Dieuleveut & Bach, 2016; Ying & Pontil, 2008; Fischer & Steinwart, 2020; Zou et al., 2021).
These works provide bounds of the form O ( t   ) for exponents  which depend on the task and feature distribution.
Several works have analyzed average case online learning in shallow and two-layer neural networks.
Classical works often analyzed unstructured data (Heskes & Kappen, 1991; Biehl & Riegler, 1994;
Mace & Coolen, 1998; Saad & Solla, 1999; LeCun et al., 1991; Goldt et al., 2019), but recently the hidden manifold model enabled characterization of learning dynamics in continuous time when trained on structured data, providing an equivalence with a Gaussian covariates model (Goldt et al., 2020; 2021).
In the continuous time limit considered in these works, SGD converges to gradient ﬂow on the population loss, where ﬂuctuations due to sampling disappear and order parameters obey deterministic dynamics.
Other recent works, however, have provided dynamical mean ﬁeld frameworks which allow for ﬂuctuations due to random sampling of data during a continuous time limit of SGD, though only on simple generative data models (Mignacco et al., 2020; 2021).
Studies of fully trained linear (in trainable parameters) models also reveal striking dependence on data and feature structure.
Analysis for models trained on MSE (Bartlett et al., 2020; Tsigler &
Bartlett, 2020; Bordelon et al., 2020; Canatar et al., 2020), hinge loss (Chatterji & Long, 2021; Cao et al., 2021; Cao & Gu, 2019) and general convex loss functions (Loureiro et al., 2021) have now been performed, demonstrating the importance of data structure for ofﬂine generalization.
Other works have studied the computational advantages of SGD at different batch sizes m . Ma et al. (2018) study the tradeoff between taking many steps of SGD at small m and taking a small number of steps at large m . After a critical m , they observe a saturation effect where increasing m provides diminishing returns.
Zhang et al. (2019) explore how this critical batch size depends on SGD and momentum hyperparameters in a noisy quadratic model. Since they stipulate constant gradient noise induced by sampling, their analysis results in steady state error rather than convergence at late times, which may not reﬂect the true noise structure induced by sampling.  P ROBLEM D EFINITION AND S ETUP
We study stochastic gradient descent on a linear model with parameters w and feature map ( x )
R N (with N possibly inﬁnite).
Some interesting examples of linear models are random feature models, where ( x ) =
( G x ) for random matrix G and point-wise nonlinearity  (Rahimi & Recht, 2008; Mei & Montanari, 2020).
Another interesting linearized setting is wide neural networks with neural tangent kernel (NTK) parameterization (Jacot et al., 2020; Lee et al., 2020).
Here the features are parameter gradients of the neural network function (
[Equation]
at initialization.
We will study both of these special cases in experiments.
We optimize the set of parameters w by SGD to minimize a population loss of the form
[Equation]
where x are input data vectors associated with a probability distribution p ( x ) , ( x ) is a nonlinear feature map and y ( x ) is a target function which we can evaluate on training samples.
We assume that the target function is square integrable ⌦ y ( [Equation] ↵ x < 1 over p ( x ) .
Our aim is to elucidate how this population loss evolves during stochastic gradient descent on w .
We derive a formula in terms of the eigendecomposition of the feature correlation matrix and the target function ⌃
[Equation]
where h y ?
( x ) ( x ) i = 0 .
We justify this decomposition of y ( x ) in the Appendix A using an eigendecomposition and show that it is general for target functions and features with ﬁnite variance.
During learning, parameters w are updated to estimate a target function y which, as discussed above, can generally be expressed as a linear combination of features y = w ⇤ · [Equation] y ? .
At each time step t , the weights are updated by taking a stochastic gradient step on a fresh mini-batch of m examples
[Equation]
where each of the vectors t,µ are sampled independently and y t,µ = w ⇤ · t,µ .
The learning rate ⌘ controls the gradient descent step size while the batch size m gives a empirical estimate of the gradient at timestep t .
At each timestep, the test-loss, or generalization error, has the form
[Equation]
which quantiﬁes exactly the test error of the vector w t .
Note, however, that L t is a random variable since w t depends on the precise history of sampled feature vectors D t
= { t,µ } .
Our theory, which generalizes the recursive method of (Werfel et al., 2004) allows us to compute the expected test loss by averaging over all possible sequences to obtain h L t i D t .
Our calculated learning curves are not limited to the one-pass setting, but rather can accommodate sampling minibatches from a ﬁnite training set with replacement and testing on a separate test set which we address in Section 4.4.
In summary, we will develop a theory that predicts the expected test loss h
L t i D t averaged over training sample sequences D t in terms of the quantities {  k , v k , ⌦ y ?
( x )↵ x } .
This will reveal how the structure in the data and the learning problem inﬂuence test error dynamics during SGD.
This theory is a quite general analysis of linear models on square loss, analyzing the performance of linearized models on arbitrary data distributions, feature maps , and target functions y ( x ) .
A NALYTIC F ORMULAE FOR L EARNING C URVES 4. L EARNABLE AND N OISE F REE P ROBLEMS
Before studying the general case, we ﬁrst analyze the setting where the target function is learnable , meaning that there exist weights w ⇤ such that y ( x ) = w ⇤ · ( x ) .
For many cases of interest, this is a reasonable assumption, especially when applying our theory to real datasets by ﬁtting an atomic measure on P points 1 P
P µ  ( x  x µ ) .
We will further assume that the induced feature distribution is Gaussian so that all moments of can be written in terms of the covariance ⌃ .
We will remove these assumptions in later sections.
Theorem 4.1.
Suppose the features follow a Gaussian distribution ⇠ N (0 , ⌃ ) and the target function is learnable in these features y = w ⇤ · .
After t steps of SGD with minibatch size m and learning rate ⌘ , the expected (over possible sample sequences D t ) test loss h
L t i D t has the form
[Equation]
where  is a vector containing the eigenvalues of ⌃ and v 2 is a vector containing elements ( v 2 ) k
= v 2 k = ( u k · w ⇤ ) 2 for eigenvectors
u k of ⌃ .
The function diag ( · ) constructs a diagonal matrix with the argument vector placed along the diagonal.
Proof.
See Appendix B for the full derivation.
We will provide a brief sketch of the proof here.
The strategy of the proof relies on the fact that h L t i =
Tr ⌃ C t where C t =
[Equation]
We derive the following recursion relation for this error matrix
[Equation]
The loss only depends on c k,t = u > k C t u k .
Solving the recurrence, c t =
A t v 2 and using h L t i = P k  k u > k C t u k = P k c k
[Equation]
t  k =  > A t v 2 , we obtain the desired result.
Below we provide some immediate interpretations of this result.
• The matrix A contains two components; a matrix ( I  ⌘ diag (  )) 2 which represents the timeevolution of the loss under average gradient updates .
The remaining matrix ⌘ 2 m
[Equation]
arises due to ﬂuctuations in the gradients, a consequence of the stochastic sampling process.
• The test loss obtained when training directly on the population loss can be obtained by taking the minibatch size m !
In this case, A !
( I  ⌘ diag (  )) 2 and one obtains the population loss L popt =
P k v 2 k  k (1  ⌘ k ) 2 t .
This population loss can also be obtained by considering small learning rates, i.e. the ⌘ ! 0 limit,
where A = ( I  ⌘ diag (  )) {+
+} + O ( ⌘ 2 ) .
• For general  and ⌘ 2 /m > 0 , A is non-diagonal, indicating that the components { u 1 , ..., u k } are not learned independently as t increases like for L popt , but rather interact during learning due to non-trivial coupling across eigenmodes at large ⌘ 2 /m .
This is unlike ofﬂine theory for learning in feature spaces (kernel regression), (Bordelon et al., 2020; Canatar et al., 2020), this observation of mixing across covariance eigenspaces agrees with a recent analysis of SGD, which introduced recursively deﬁned “mixing terms” that couple each mode’s evolution (Varre et al., 2021).
• Though increasing m always improves generalization at ﬁxed time t (proof given in Appendix D), learning with a ﬁxed compute budget (number of gradient evaluations)
C
= tm , can favor smaller batch sizes.
We provide an example of this in the next sections and Figure 1 (d)-(f).
• The lower bound h L t i   > v
[Equation]
i t can be used to ﬁnd necessary stability conditions on m, ⌘ .
This bound implies that h L t i will diverge if m < ⌘ 2  ⌘ |  | 2 .
The learning rate must be sufﬁciently small and the batch size sufﬁciently large to guarantee convergence.
This stability condition depends on the features through |  | 2 =
P k  2 k .
One can derive heuristic optimal batch sizes and optimal learning rates through this lower bound.
See Figure 2 and Appendix C.
4.1. S PECIAL C ASE 1: U NSTRUCTURED I SOTROPIC F EATURES
This special case was previously analyzed by Werfel et al.
(2004) which takes ⌃
=
I 2 R N ⇥ N and m = 1 .
We extend their result for arbitrary m , giving the following learning curve
[Equation]
where the second expression has optimal ⌘ .
First, we note the strong dependence on the ambient dimension N : as N  m , learning happens at a rate h
L t i ⇠ e  tm/N .
Increasing the minibatch size m improves the exponential rate by reducing the gradient noise variance.
Second, we note that this feature model has the same rate of convergence for every learnable target function y .
At small m , [Figure] the convergence at any learning rate ⌘ is much slower than the convergence of the m !
1 limit,
L pop = (1  ⌘ ) 2 t ||
w ⇤ || 2 which does not suffer from a dimensionality dependence due to gradient noise.
Lastly, for a ﬁxed compute budget C = tm , the optimal batch size is m ⇤ = 1 ; see Figure 1 (d)
This can be shown by differentiating ⌦
L C/m ↵ with respect to m (see Appendix E).
In Figure 1 (a) we show theoretical and simulated learning curves for this model for varying values of N at the optimal learning rate and in Figure 1 (d), we show the loss as a function of minibatch size for a ﬁxed compute budget C = tm = 100 .
While ﬁxed C represents ﬁxed sample complexity, we stress that it may not represent wall-clock run time when data parallelism is available (Shallue et al., 2018).
4.1. S PECIAL C ASE 2: P OWER L AWS AND E FFECTIVE D IMENSIONALITY
Realistic datasets such as natural images or audio tend to exhibit nontrivial correlation structure, which often results in power-law spectra when the data is projected into a feature space, such as a randomly intialized neural network (Spigler et al., 2020; Canatar et al., 2020; Bahri et al., 2021).
In the ⌘ 2 m ⌧ 1 limit, if the feature spectra and task specra follow power laws,  k ⇠ k  b and  k v 2 k ⇠ k  a with a, b > 1 , then Theorem 4.1 implies that generalization error also falls with a power law: h L t i ⇠ Ct   ,  = a  1 b where C is a constant.
See Appendix G for a derivation with saddle point integration.
Notably, these predicted exponents we recovered as a special case of our theory agree with prior work on SGD with power law spectra, which give exponents in terms of the feature correlation structure (Berthier et al., 2020; Dieuleveut et al., 2016; Velikanov & Yarotsky, 2021; Varre et al., 2021).
Further, our power law scaling appears to accurately match the qualitative behavior of wide neural networks trained on realistic data (Hestness et al., 2017; Bahri et al., 2021), which we study in Section 5. [Figure]
We show an example of such a power law scaling with synthetic features in Figure 1 (b).
Since the total variance approaches a ﬁnite value as N ! 1 , the learning curves are relatively insensitive to N and are rather sensitive to the eigenspectrum through terms like |  | 2 and 1 >  , etc.
In Figure 1 (c), we see that the scaling of the loss is more similar to the power law setting than the isotropic features setting in a random features model of MNIST, agreeing excellently with our theory.
For this model, we ﬁnd that there can exist optimal batch sizes when the compute budget C = tm is ﬁxed (Figure 1 (e) and (f)).
In Appendix C.1, we heuristically argue that the optimal batch size for power law features should scale as, m ⇤ ⇡ 1 (2 b  1) .
Figure 2 shows a test of this result and related experiments.
We provide further evidence of the existence of power law structure on realistic data in Figure(a)-(c), where we provide spectra and test loss learning curves for MNIST and CIFAR-10 on ReLU random features.
The eigenvalues  k ⇠ k  b and the task power tail sums
[Equation]
both follow power laws, generating power law test loss curves.
These learning curves are contrasted with isotropically distributed data in R 784 passed through the same ReLU random feature model and we see that structured data distributions allow much faster learning than the unstructured data.
Again, our theory predicts experimental curves accurately across variations in learning rate, batch size and noise (Figure 3).
4. A RBITRARY I NDUCED F EATURE D ISTRIBUTIONS : T HE G ENERAL S OLUTION
The result in the previous section was proven exactly in the case of Gaussian vectors (see Appendix
B).
For arbitrary (possibly non-Gaussian) distributions, we obtain a slightly more involved result (see
Appendix F).
Theorem 4.2.
Let ( x ) 2 R N be an arbitrary feature map with covariance matrix ⌃ =
P k  k u k u > k .
After diagonalizing the features  k ( x ) = u > k ( x ) , introduce the fourth moment tensor  4 ijkl
=
h
i  j
k
l i .
The expected loss is exactly h L t i =
P k  k c k
(  ,  , v , t ) .
We provide an exact formula for c k in the Appendix F We see that the test loss dynamics depends only on the second and fourth moments of the features through quantities  k and  ijk ` respectively.
We recover the Gaussian result as a special case when  ijkl is a simple weighted sum of these three products of Kronecker tensors  Gaussijkl
[Equation]
ij  kl
+  i
j  il  jk .
As an alternative to the above closed form expression for h L t i , a recursive formula which tracks N mixing coefﬁcients [Figure] has also recently been utilized to analyze the test loss dynamics for arbitrary distributions (Varre et al., 2021).
Next we show that a regularity condition, similar to those assumed in other recent works (Jain et al., 2018; Berthier et al., 2020; Varre et al., 2021), on the fourth moment structure of the features allows derivation of an upper bound which is qualitatively similar to the Gaussian theory.
Theorem 4.3.
If the fourth moments satisfy
[Equation]
for any positive-semideﬁnite G , then
[Equation]
We provide this proof in Appendix F.1.
We note that the assumed bound on the fourth moments is tight for Gaussian features with ↵ = 1 , recovering our previous theory.
Thus, if this condition on the fourth moments is satisﬁed, then the loss for the non-Gaussian features is upper bounded by the
Gaussian test loss theory with the batch size effectively altered ˜ m = m/ ↵ .
The question remains whether the Gaussian approximation will provide an accurate model on realistic data .
We do not provide a proof of this conjecture, but verify its accuracy in empirical experiments on MNIST and CIFAR-10 as shown in Figure 3.
In Appendix Figure F.1, we show that the fourth moment matrix for a ReLU random feature model and its projection along the eigenbasis of the feature covariance is accurately approximated by the equivalent Gaussian model.
4. U NLEARNABLE OR N OISE C ORRUPTED P ROBLEMS
In general, the target function y ( x ) may depend on features which cannot be expressed as linear combinations of features ( x ) , y ( x ) =
w ⇤ · ( x )+ y ?
( x ) .
Let ⌦ y ?
( x )↵ x =  2 .
Note that y ?
does [Figure] not have to be a deterministic function of x , but can also be a stochastic process which is uncorrelated with ( x ) .
Theorem 4.4.
For a target function with unlearnable variance
[Equation]
on Gaussian
[Equation] the expected test loss has the form
[Equation]
which has an asymptotic, irreducible error h
L 1 i = 1 m ⌘ [Equation]  [Equation]
[Equation]
( I [Equation] See Appendix H for the proof.
The convergence to the asymptotic error takes the form h L t  L 1 i =  >
A t
[Equation]
.
We note that this quantity is not necessarily monotonic in t and can exhibit local maxima for sufﬁciently large  2 , as in Figure 3 (f).
4. T EST /T RAIN S PLITS
Rather than interpreting our theory as a description of the average test loss during SGD in a one-pass setting, where data points are sampled from the a distribution at each step of SGD, our theory can be suitably modiﬁed to accommodate multiple random passes over a ﬁnite training set.
To accomplish this, one must ﬁrst recognize that the training and test distributions are different.
Theorem 4.5.
Let ˆ p ( x ) = 1 M
P µ  ( x  x µ ) be the empirical distribution on the M training data points and let ˆ ⌃ =
⌦ ( x ) ( x ) > ↵ x ⇠ ˆ p ( x ) =
P k ˆ  k u k u > k be the induced feature correlation matrix on this training set.
Further, let p ( x ) be the test distribution ⌃ its corresponding feature correlation.
Then we have
[Equation]
We provide the proof of this theorem in Appendix I.
The interpretation of this result is that it provides the expected training and test loss if, at each step of SGD, m points from the training set { x 1 , ..., x M } are sampled uniformly with replacement and used to calculate a stochastic gradient.
Note that while ⌃ can be full rank, the rank of ˆ ⌃ has rank upper bounded by M , the number of training samples.
The recurrence for C t can again be more easily solved under a Gaussian approximation which we employ in Figure 4.
Since learning will only occur along the M dimensional subspace spanned by the data, the test error will have an irreducible component at large time, as evidenced in Figure 4.
While the training errors continue to go to zero, the test errors saturate at a M -dependent ﬁnal loss.
This result can also allow one to predict errors on other test distributions.
C OMPARING N EURAL N ETWORK F EATURE M APS
We can utilize our theory to compare how wide neural networks of different depths generalize when trained with SGD on a real dataset.
With a certain initialization of the model parameters, inﬁnite width [Figure] networks behave as linear functions of their parameters f ( x , ✓ ) ⇡ f ( x , ✓ 0 )
+ r ✓ f ( x , ✓ 0 )
· ( ✓  ✓ 0 ) (Lee et al., 2020).
To predict test loss dynamics with our theory, it therefore sufﬁces to characterize the geometry of the gradient features ( x ) =
r ✓ f ( x , ✓ ) .
In Figure 5, we show the Neural Tangent
Kernel (NTK) eigenspectra and task-power spectra for fully connected neural networks of varying depth, calculated with the Neural Tangents API (Novak et al., 2020).
We compute the kernel on a subset of 10 , 000 randomly sampled MNIST images and estimate the power law exponents for the kernel and task spectra  k and v 2 k .
We ﬁnd that, accross architectures, the task spectra v 2 k are highly similar, but that the kernel eigenvalues  k decay more slowly for deeper models, corresponding to a smaller exponent b .
As a consequence, deeper neural network models train more quickly during stochastic gradient descent as we show in Figure 5 (c).
After ﬁtting power laws to the spectra  k ⇠ k  b and the task power v 2 k ⇠ k  a , we compared the true test loss dynamics (color) for a width-500 neural network model with the predicted power-law scalings  = a  1 b from the ﬁt exponents a, b .
The predicted scalings from NTK regression accurately describe trained networks at ﬁnite width.
On CIFAR-10, we compare the scalings of the CNN model and a standard MLP and ﬁnd that the CNN obtains a better exponent due to its faster decaying tail sum
[Equation]
C ONCLUSION
Studying a simple model of SGD, we were able to uncover how the feature geometry governs the dynamics of the test loss.
We derived average learning curves h L t
i for both Gaussian and general features and showed conditions under which the Gaussian approximation is accurate.
The proposed model allowed us to explore the role of the data distribution and neural network architecture on the learning curves, and choice of hyperparameters on realistic learning problems. While our theory accurately describes networks in the lazy training regime, average case learning curves in the feature learning regime would be interesting future extension.
Further extensions of this work could be used to calculate the expected loss throughout curriculum learning where the data distribution evolves over time as well as alternative optimization strategies such as SGD with momentum.