Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
Anonymous Author(s)
AfﬁliationAddress email
Abstract
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expectedperformance improvements under different model and data scaling regimes.
Currently, theoretical understanding of the learning curves that characterize how theprediction error depends on the number of samples is restricted to either largesample asymptotics ( m → ∞ ) or, for certain simple data distributions, to thehigh-dimensional asymptotics in which the number of samples scales linearly withthe dimension ( m ∝ d ).
There is a wide gulf between these two regimes, includingall higher-order scaling relations m ∝ d r , which are the subject of the presentpaper.
We focus on the problem of kernel ridge regression for dot-product kernelsand present precise formulas for the test error, bias, and variance, for data drawnuniformly from the sphere in the r th-order asymptotic scaling regime m → ∞ withm/d r held constant.
We observe a peak in the learning curve whenever m ≈ d r /r !
for any integer r , leading to multiple sample-wise descent and nontrivial behaviorat multiple scales.
Introduction
Modern machine learning has entered an era in which scaling is arguably the most critical ingredientto improve performance.
Recent breakthroughs such as GPT-3 [16] and PaLM [9] have demonstratedthat performance of various learning algorithms improves in a predictable manner as the amount ofdata and computational resources used in training increases.
The functional relationships betweenperformance and resources are loosely referred to as learning curves.
While extrapolation of empiricallearning curves is widely used to make predictions about how a model might perform when extraresources become available, a rigorous theoretical understanding is lacking.
A fundamental obstaclein developing a detailed theoretical model of such learning curves is that they depend on many movingparts, e.g. the data distribution, the network architecture, the training algorithm, among others.
Inaddition, even in the simplest possible settings, the learning curves can exhibit non-trivial
structurethat naive scaling laws fail to model, e.g. the well-known double-descent phenomenon [7, 3].
In the past couple years, a large amount of effort from the community has improved our theoreticalunderstanding of such phenomena and in some cases precise characterizations of learning curveshave been obtained (see e.g. [14, 1, 21]).
These results have helped clarify several puzzling empiricalobservations, such as the origin of the double-descent peak [2, 19, 10] and linear trends betweenin- and out-of-distribution generalization performance [27, 28, 20], among many others.
However,the precise predictions from many of these analyses have been possible only in the linear highdimensional scaling regime in which the number of training samples m scales linearly with thedimension d , i.e. m ∝ d .
In these asymptotics, the model’s effective capacity is limited to linearfunctions of the features.
In contrast, many state-of-the-art models operate in a regime where the
[Figure]
amount of data is much larger than the data dimensionality; for example, large text corpora cancontain trillions of tokens, whereas the effective input dimensionality of language models is at mostmillions .
Therefore, going beyond the linear scaling regime ( m ∝ d ) to higher-order scaling regimes
( m ∝ d r ) is essential in improving our understanding of modern machine learning systems, and is the focus of the current paper.
Several works have investigated the behavior of the learning curves for nonlinear scalings in thedot-product kernel or random features setting, but they have done so only in the noncritical regimem (cid:54)∝ d r [13, 22, 18].
Extending these results to include the critical regime m ∝ d r is nontrivial, bothfrom the technical perspective and also from the phenomenological perspective, as we shall see thecritical behavior induces nonmonotonicity and multiple sample-wise descent.
In this work, we obtain precise formulas for the sample-wise learning curves in the kernel ridgeregression setting for a family of dot-product kernels for spherical input data in the polynomial scalingregimes m ∝ d r for all r ∈ N ∗ .
This family of kernels includes the neural network Gaussian Process(NNGP) kernels and Neural Tangent Kernels (NTK) associated with multi-layer fully-connectednetworks or convolutional networks.
Both kernels serve as important starting points towards a deeperunderstanding of neural networks as they often capture the ﬁrst order learning dynamics of neuralnetworks in certain scaling limits [15, 17, 4].
1.1 Contributions
Our primary contributions are to establish the following, for data drawn uniformly from the sphere:
The empirical spectral density of the Gram matrix induced by degree- r spherical harmonicsconverges to a Marchenko-Pastur distribution when ( d r /r !)
/m converges to a positiveconstant as d → ∞ (Theorem 1);2.
A precise closed-form formula for the sample-wise learning curves for dot-product kernelregression when m ∝ d r for all r ∈ N ∗ as d → ∞ (Theorem 2);3.
Empirically, the theoretical predictions agree with ﬁnite-size simulations surprisingly welleven in the strong ﬁnite-size correction regime (Fig.
An extension of the above results to convolutional kernels (Section 5).
Notation and Setup
Let X = S d − 1 denote the input space, where S d − 1 is the unit sphere in R d and X is equipped withthe normalized uniform measure σ .
We use ∆ d to represent any quantity (a scalar, vector or a matrix)with ||| ∆ d ||| → 0 as d → ∞ (in probability if ∆ d is stochastic), where ||| · ||| can be the absolutevalue of a scalar, the norm of a vector or the operator norm of a matrix.
Let X ∈ R m ×
d be the training inputs where the i -th row of X is x (cid:62) i .
We assume { x i } i ∈ [ m ] issampled uniformly, iid from X .
The label function f : S d − 1 → R will be deﬁned in Section 4.
Let
K = K ( d ) be a dot-product kernel deﬁned on S d − 1 × S d − 1 , i.e., K ( x , x (cid:48) )
=
h ( x (cid:62) x (cid:48) )
for somefunction h ∈ [ − 1 , 1] → R .
We assume h has the following decomposition
[Equation] [Equation]
=
[Equation]
with
[Equation]
(1) where P k is the k -th order Legendre polynomials in d dimensions.
For simplicity, we assumeˆ h
= (ˆ h k ) k ≥ 1 is a sequence that is independent of d and ˆ h k (cid:54) =
for all k ≤ k 0 where k 0 is sufﬁcientlylarge.
As such, we can decompose the kernel function using sperical harmonics,
[Equation]
[Equation]
where Y k,l is the l -th spherical harmonic of degree k , N ( d, k )
=
d k /k !
+ O ( d k − 1 ) is the total numberof degree k spherical harmonics in d dimensions, σ 2 k
= ˆ h 2 k /N
( d, k ) is the eigenvalue of Y kl , and
Y k
( x ) is the column vector
[ Y k,l ( x )] (cid:62) l ∈ [ N ( d,k )] .
We also denote by Y k
( X ) the m × N ( d, k ) matrixwhose
i -th row is Y k
( x i ) (cid:62) .
Structure of the Empirical Kernel and Marchenko-Pastur Distribution
The structure of the empirical kernel matrix K ( X , X ) plays a critical role in characterizing thesample-wise test error for the kernel ridge regressor associated to K .
We assume the training set sizescales polynomially, i.e. m ∼ d r for some positive integer r ∈ N ∗ .
Decompose this kernel into low-,critical- and high-frequency modes as follows,
[Equation] [Equation]
The low- and high-frequency parts have simple structures since N ( d, k ) /m
either diverges to inﬁnity or converges to zero with rate as least d ± 1 , yielding concentration that results in signiﬁcantsimpliﬁcation.
To be precise, for high-frequency modes k > r , Y k
( X ) is a “fat" matrix and
Y k
( X )
Y k
( X ) (cid:62) /N
( d, k )
=
I m + ∆ d where ∆ d vanishes as d → ∞ [22].
Thus, the high-frequencyparts behave like a regularizer in the following sense,(cid:88)
[Equation]
On the other hand, when k < r , Y k ( X ) is a m × N ( d, k ) “tall" matrix with N ( d, k ) /m =
O ( d k /m )
=
O ( d − ( r − k ) )
→ 0 .
Similarly, Mei et al.
[22] show that Y k ( X ) (cid:62) Y k
( X ) /m =
I N ( d,k ) + ∆ d , implying that when restricted to the subspace spanned by low-frequency functions{ Y kl } k<r , the regressor associated to the empirical kernel K ( X , X ) acts like a pure multiplicativescaling.
It remains to understand the critical-frequency mode
Y r
( X )
(cid:62) Y r
( X ) .
It turns out that if
N ( d, k ) /m
→ α ∈ (0 , ∞ ) , then the empirical spectral measure of the random matrix
Y r ( X ) (cid:62) Y r
( X ) /m converges to the Marchenko-Pastur distribution µ α , whose density is givenby
[Equation]
, where α
[Equation]
D e n s i t y r=2, d=420, m=17681 , =
[Figure]
Empirical Spectrum   3 0.
D e n s i t y r=3, d=60, m=18880 , =0.
[Figure]
Spectrum
D e n s i t y r=4, d=24, m
[Figure]
where δ 0 ( t )
=
if t (cid:54) = 0 else 1 .
See Fig.
2 for visualizations of µ α .
The r = 1 case is obviousas Y 1 ( X ) =
c d X for some normalizing constant c d and it is clear 1 m Y 1 ( X ) (cid:62) Y 1 ( X ) =
c 2 d m X (cid:62) Xconverges to the Marchenko-Pastur distribution µ α if d/m → α ∈ (0 , ∞ ) as d → ∞ [26].
Our ﬁrstresult show that this result continues to hold for all degrees.
Theorem 1.
For ﬁxed r ∈ N and α ∈ (0 , ∞ ) , if N ( d, r ) /m → α ∈ (0 , ∞ ) as d → ∞ , then theempirical spectral distribution of 1 m Y r ( X ) (cid:62) Y r ( X ) converges in distribution to the MarchenkoPastur distribution µ M P ( α ) .
In the top panel of Fig. 2, we generate the empirical spectra 1 of 1 m Y r
( X )
(cid:62) Y r
( X ) for various valuesof r , d , and α .
The Marchenko-Pastur distribution µ α perfectly captures the empirical measures ofthe random matrices 1 m Y r
( X )
(cid:62) Y r
( X ) for all r considered.
We sketch the main steps of the proof ofthe theorem below; see Sec.
B for the whole proof.
Sketch of Proof.
From Bai and Zhou [5, Theorem 1.1], it sufﬁces to prove concentration of thefollowing quadratic forms: for every sequence of N ( d, k ) × N ( d, k ) matrices { A d } with operatornorm (cid:107)
A d (cid:107) op ≤ 1 , the variance
[Equation]
∞
[Equation] [Equation]
For the purpose of illustration, we assume A ≡ A d is a diagonal matrix.
Then we only need to show
N ( d, [Equation] (cid:88)
[Equation]
By hypercontractivity of spherical harmonics [6],
[Equation]
[Equation]
where C k is some absolute constant.
Since | A ll | ≤ (cid:107) A (cid:107) op ≤ 1 , we can drop any o ( N ( d, l ) 2 ) pairsof ( l, l (cid:48) ) in Eq.
We show that for the remaining pairs ( l, l (cid:48) ) , the eigenfunctions are asymptoticallyuncorrelated in the sense
[Equation]
which implies Eq
[Equation]
Generalization Error of Dot-Product Kernel Regression
In this section, we establish the average generalization error for the kernel regression in the asymptoticregime N ( d, r ) /m
→ α , for some α ∈ (0 , ∞ ) and r ≥ 1 ﬁxed.
We assume the label functionf ∈ L 2 ( S d − 1 ) is given by
[Equation]
where ˆ f kl are the “Fourier" coefﬁcients and ˆ f k
= [ f kl ] (cid:62) l ∈ N ( d,k ) .
We need to make a technicalassumption that for k (cid:48) , k ≥ r
[Equation]
i.e. ˆ f k is centered with isotropic covariance and { ˆ f k } k ≥ r are mutually uncorrelated.
Note that weallow ˆ f kl to be deterministic for k < r .
We let F =
( ˆ F k ) ≥ 1 be a ﬁxed sequence with (cid:80) k ≥ 1 ˆ F 2 k < ∞ ,where ˆ F 2 k = (cid:80) l ∈ [ N ( d,k )]
ˆ f 2 kl for k < r .
For convenience, set ˆ F 2 >j = (cid:80) k>
j ˆ F 2 k (similarly for ˆ F 2 ≥ j ,ˆ F 2 ≤ j , etc.)
and use f to denote the random vector { ˆ f kl } kl .
Given training inputs X and observedlabels Y = f
( X ) + (cid:15) , where (cid:15) ∼ N ( 0 , σ 2 (cid:15) I m ) is the noise, the prediction using kernel function Kis given by
[Equation]
Here λ ≥ 0 is the regularization.
As such, the mean test error over the random labels is given by
Err
[Equation]
X
[Equation]
λ, F , ˆ h ) =
E f Err
[Equation]
X
[Equation]
λ, f , ˆ h ) where Err( X ; λ, f , ˆ h ) =
E x
[Equation]
To state our results, we need to introduce two functions χ B and χ V which are related to the bias andvariance in the generalization error,
[Equation]
Both χ B and χ V have closed-form representations; see Sec C.5.
Deﬁne the effective regularizationassociated to the r -th order scaling to be
[Equation] [Equation]
Finally, we deﬁne the bias and variance associated to the r -th order scaling to be
[Equation]
The following is our main result, which characterizes the test error in the asymptotic regime m ∝ d r .
Theorem 2.
Let α ∈ (0 , ∞ ) and r ≥ 1 be ﬁxed.
Assume N ( d, r ) /m
→ α as d → ∞ .
Then theaverage test error is given by
[Equation]
where ∆ d → 0 in probability.
4.1 Interpretations
We provide some high-level interpretations of the bias term B r and variance term V r .
The Bias.
From Eq.
(16), the regressor learns all low-frequency modes ( k < r ) but none of thehigh-frequency modes ( k > r ) as the bias B r contains no low-frequency modes (i.e. k < r ) butall high-frequency modes
ˆ F 2 >r .
Importantly, the regressor is progressively learning the criticalfrequency mode Y r as the training size m = 1 α N ( d, r ) increases, i.e. from α = ∞ to α =
+since χ B ( α, ξ r (ˆ h , λ, α ))
→ 1 if α → ∞ and χ B ( α, ξ r (ˆ h , λ, α ))
→ 0 if α → 0 + .
See Fig.3 for theillustration.
N(d, 1)
N(d, 2) N(d, 3) N(d, 4) m: # Training Samples (log scale)
F
F
F 2 3 F
T e s t M S E ( li n e a r s c a l e )
Bias-Variance Decomposition B 1 + V 1 B
V 1 B 2 + V 2 B
V
B 3 + V 3 B
V 3 B 4 + V 4 B
V
The Variance.
From Eq.
(17), the variance termχ V treats all high-frequency modes ˆ F 2 >r the sameas the noise term (cid:15) .
Moreover, χ V → 0 as α → 0or ∞ and is peaked at α = 1 .
The height of thepeak depends on the effective regularization ξ r andit diverges to inﬁnity with rate ξ 12 r as ξ r → ∞ .
Indeed, when α = 1 and ξ − 1 / 2 ≤ t ≤ ξ − 1 , we have
1+ µ α ( t ) ∝
− 1 / 2 which implies χ V (1 , ξ ) ≥
[Equation] [Equation]
Finally, Eq.
(18) not only gives precise generalization formula (up to a vanishing term ∆ d ) when m ≈
N ( d, r ) ∼ d r but also when d r − 1+ δ (cid:46) m (cid:46) d r − δ(i.e.
when “ α = ∞ ") and when d r + δ (cid:46) m (cid:46)m r +1 − δ (i.e. when “ α = 0 + ") for any δ ∈ (0 , 1 / 2) .
Indeed, in the non-critical scaling regime d r − 1+ δ (cid:46)m (cid:46) d r − δ , α
= N ( d, k ) /m
→ ∞ as d → ∞ and
[Equation]
As such, the regressor learns all low-frequency modes but none of the critical- and high-frequencymodes ( k ≥ r ), which is consistent with the result in [13, 22].
A similar argument also shows
B r ( α =
+ )
+ V r
( α = 0 + )
=
ˆ F 2 >r , namely, the regressor also learns the r -frequency mode.
Thisobservation implies that we can glue together Eq.
(18) for r ≥ 1 and remove all duplicate terms togenerate a sample-wise learning curve (LC):
[Equation]
where N ( d, ≤ r ) = (cid:80) rk =1 N ( d, k ) .
The “ − ( r − 1) ˆ F 2 r " term in the above equation is due to thefact that ˆ F 2 r is over-counted ( r − 1) many times (one in each B k for k = 1 , ..., ( r − 1) .)
It is worthmentioning that using α = N ( d, ≤ r ) /m rather than α = N ( d, r ) /m in the variance V r capturesthe ﬁnite-size correction more accurately.
See Eq.
Corollary 1.
If, for 1 ≤ r ∈ N , (1) N ( d, r ) /m
→ α for some α ∈ (0 , ∞ ) , or (2) d r − 1+ δ (cid:46) m
(cid:46)d r − δ for some δ ∈ (0 , 1 / 2) , then
[Equation]
where ∆ d → 0 in probability as d → ∞ .
Recall that for each r , the variance term V r could diverge to inﬁnity as ξ r → ∞ at α = 1 .
Thus wemight expect a peak in the learning curve for each r , yielding the multiple-descent phenomenon, asshown in Fig.1.
However, such phenomena can disappear by making the heights of the peaks smallvia choosing ξ r small.
We will discuss this point in the experimental section.
4.2 Proof Sketch
The proof of this theorem is quite involved; see Sec.
C. For simplicity, we assume the observed labelsare noiseless, i.e. σ 2 (cid:15) = 0 .
An ingredient is to understand the structure of the operator
[Equation]
The high-level strategy is as follows.
We decompose the function into low-, critical- and highfrequency parts f = f <r + f r + f >r .
As such, the test error is roughly
Err( X ) ≈
Err <
r
( X ) + Err r ( X ) + Err >r ( X ) where Err r ( X ) =
E f E x | T K f r ( x ) − f r ( x )
| 2 , and similarly for Err <
r
( X ) and Err >r ( X ) .
The next step is to estimate each part separately.
Low-frequencies.
Using the fact that the low-frequency parts of the kernel function K is almost anisometric operator on the column space of Y <r ( X ) , one can show that E x | T K ( f <r )( x ) − f <r ( x ) | 2 =∆ d → 0 in
Critical-frequency.
Up to a vanishing term, one can remove all non-critical frequencies in thekernel function K in T K in the sense of making the following substitutions
[Equation]
Thus, with γ = ( λ + ˆ h 2 >r ) , T K f r ( x ) − f r ( x ) =
Y r
( x ) (cid:62) M r ( X ) f r + ∆ d , where
[Equation]
[Equation]
Taking expectation with respect to x (using orthogonality of Y r ( x ) ) and then with respect to f r ,
[Equation]
Applying the Sherman–Morrison–Woodbury formula and then Theorem 1,
[Equation]
High-frequencies.
The cross term E f E
[Equation]
f
[Equation]
x
[Equation]
f
[Equation]
d and thus
[Equation]
The second term is equal to ˆ F 2 > .
The calculation of the ﬁrst term is similar to that of the criticalfrequency above (namely, we remove all high-/low-frequency components in K .)
Convolutional Kernels5.1 One hidden layer
Our analysis can be extended to analyzing NNGP kernel and NT kernel for one-layer convolution [24,25, 31].
In this case, we assume the input space is X
= S pd 0 − 1 , where d 0 is the dimension of apatch, p is the number of patches, and d = pd 0 is the total dimensions of the inputs.
The measureassociated to X is the product of the uniform measure on S d 0 − 1 .
We assume that both the ﬁlter sizeand stride of the convolution are equal to d 0 .
As such, after the ﬁrst convolutional layer, the inputis reduced to a vector of dimension p .
We then apply a non-linearity and a dense layer to map thisp -dimensional vector to a scalar.
The NNGP and NT kernel have the following general form.
Letx =
( x i )
i ∈
[ p ] ∈ X , where x i ∈ S
d 0 − 1 is the i -th patch
[Equation]
Denote Y ( i ) kl ( x ) =
Y kl ( x i ) and Y k ( x ) =
[ Y ( i ) kl ( x ) (cid:62) ] (cid:62) l ∈
[ N ( d 0 ,k )] ,i ∈ [ p ] .
Then Y k ( x ) is the degree kspherical harmonics associated to this kernel, which span a space of dimension pN ( d 0 , k ) .
Theorem 3.
Let r ∈ N ∗ and α ∈ (0 , ∞ ) be ﬁxed.
If pN ( d 0 , r ) /m → α ∈ (0 , ∞ ) as d 0 → ∞ andthe rows of X are sampled uniformly, iid from S pd 0 − 1 , then the empirical spectral distribution
of1 m Y r
( X )
(cid:62) Y r
( X ) tends to the Marchenko-Pastur distribution µ α as d 0 → ∞ .
The assumptions on the label function are similar to that of dot-product kernel, e.g.
[Equation]
otherwise f k is deterministic with (cid:107) f k (cid:107)
= ˆ F 2 k .
Theorem 4.
Let α ∈ (0 , ∞ ) and r ≥ 1 be ﬁxed.
Assume pN
d 0 , r ) /m
→ α as d 0 → ∞ .
Then theaverage test error is given by
[Equation]
where ∆ d 0 → 0 in probability as d 0 → ∞ .
Corollary 2.
If, for 1 ≤ r ∈ N , (1) pN ( d 0 , r ) /m
→ α for some α ∈ (0 , ∞ ) , or (2) pd r − 1+ δ(cid:46)m (cid:46) pd r − δfor some δ ∈ (0 , 1 / 2) , then
[Equation]
[Equation]
where ∆ d 0 → 0 in probability as d 0 → ∞ .
5.2 Deep Convolutional Kernels
The eigenstructure of general CNN kernels are much more complicated as they depend on both thefrequencies (i.e. the order of the polynomials) and the topologies of the networks [30].
To rigorouslydescribe the eigenstructure, a heavy dose of notation must be introduced, which is beyond the scopeof the paper.
Nevertheless, the approach developed here is readily extended to cover general CNNkernels.
We brieﬂy describe the main ideas.
Following [30], we assume the input space is still X
= S pd 0 − 1 , where p is the number of patches.
Forsimplicity, we assume the network has L convolutional layers and in each layer, the ﬁlter size and thestride are all equal to d 0 .
Thus the spatial dimension of the input is reduced to 1 after L convolutionallayers.
We then add a non-linearality and a dense layer to generate the logits.
The kernel has thefollowing form
[Equation]
Unlike dot-product kernels in which k is a scalar and the eigenvalues depend only on | k | (i.e. thefrequencies), σ 2 k , l depends on both | k | and the spatial structure of the vector k in a rather complicatedmanner.
Nevertheless, as d 0 → ∞ , σ 2 k , l ∼ d − j k= d − j k /L for some L ≤ j k ∈ N .
We can thencategorize the eigenvectors according to the decay order of σ 2 k , l .
Unlike the case of dot-productkernels or the one-hidden layer CNN kernels, in which eigenvectors with same-order eigenvaluesare in the same eigenspace (i.e. the eigenvalues are the same), multiple-layer CNN kernels can havemultiple eigenspaces with the same-order eigenvalues.
Although this results in extra challenges (seebelow), our overall approach carries over.
Consider the critical scaling regime m ∼ d r , for r = j/Lfor some L ≤ j ∈ N .
Likewise, we can decompose the kernel into low-, critical- and high-frequencyparts according to j k < r , j k = r and j k > r , resp.
Following similar assumptions on the labels andeigenvalues, the bias and the variance can be essentially reduced to computing
[Equation]
[Equation]
33) where R r and R ≤ r are diagonal matrices whose entries are determined by the eigenvalues of thecritical-frequency modes.
In the dot-product kernels or one-hidden layer CNN kernels setting,
R r / R ≤ r is a scaled identity matrix and simple, closed-form expressions for the above traces straightforwardly follow from the Marcenko-Pastur distribution.
However, for a general diagonal matrix
R r with bounded limiting spectra, R r does not commute with Y (cid:62) r ( X ) Y r ( X ) , and a more detailedrandom matrix analysis is needed.
See the supplementary material for more details.
Experiments
We provide experiments to show that our learning curves (Eq.
(21)) accurately capture empiricalsample-wise learning curve even when the ambient dimensions remains small.
Even though ourtheoretical results require averaging the test error over random labels (aka, mean test error), ourexperimental results suggest this is unnecessary, i.e. the learning curve Eq.
(21) can capture the testerror accurately for any given draw of label function.
Experimental setup.
We generate a polynomial kernel function h ( t )
= (cid:80) 7 k
=1 ˆ h 2 k P k ( t ) , where P kis the degree- k Legendre polynomial in d dimensions.
The kernel function can be efﬁciently computedvia K ( x , x (cid:48) )
=
h ( x (cid:62) x (cid:48) ) .
We choose the label function to be f ( x ) = (cid:80) 7 k =
ˆ F k y k
( x ) , wherey k ( x ) = (cid:80)
j ∈ [ d ] w k,j (cid:81) j
+ k − 1 i
=
j x i , and the coefﬁcients w k,j are randomly sampled from a Gaussianand then normalized so that E x | y k ( x ) | 2 = 1 for each k .
Therefore E x | f ( x ) | 2 = (cid:80) 7 k =
ˆ
F 2 k .
Forsimplicity, we also set σ 2 (cid:15) =
(i.e. noiseless) and λ =
(i.e. ridgeless).
Note that when m (cid:46) d r , theregressor still contains “effective noise" ˆ F 2 >r from un-learnable high-frequency modes and “effectiveregularization" ˆ h 2 >r .
Finally, in our experiments, we choose ˆ F 2 k = k − 2 and ˆ h 2 k
=
Gap − ( k − 1) , where
N(d, 1)
N(d, 2) N(d, 3) N(d, 4) m: # Training Samples
F
F 2 2 F 2 3 F 21.
T e s t M S E
Theory Prediction Simulation (a) Dot Product Kernel pN(d, 1) pN(d, 2)
pN(d, 3) m: # Training Samples
F
F 2 2 F 2 3 F 21.
T e s t M S E
Theory Prediction Simulation
[Figure]
we will vary the value of the spectral gap:
Gap = ˆ h 2 k / ˆ h 2 k
+1 .
Under this setup, the predicted learningcurve LC( m ) =
LC( m ; Gap) depends only on the spectral gap of the kernel.
To simulate higher-order scaling ( r ≥ 3 ), the dimension d has to be very small as we need to invert asequence of matrices of size ranging from m = 1 to m ∝ d r /r ! .
Due to the constraints in computeand memory, the largest m we can have is typically m max ≈ 25 , 000 for one single GPU and d in ourexperiments is typically around d = 24 .
As such we are in a regime with strong ﬁnite-size corrections.
Finally, all experiments are run in a single A-100 using Google Cloud Colab Notebooks.
Learning Curves Accurately Capture Simulations.
In Fig.
4, we generate the empirical samplewise learning curve by applying kernel regression Eq.
(13) with training set X .
We vary the trainingset size m densely in [1 , m max ] and for each m we sample 20 independent X to get the errorbarplot for the test error.
The closed-form learning curve is obtained from Eq.
(21) and the calculationis done in Sec.
C.5.
Even in the low-dimensional regime with d = 24 for dot-product kernel ( d 0 = 20and p = 6 for one-hidden layer CNN kernel), the predicted learning curve captures the empiricallearning curve surprisingly well, which has a highly non-trivial multiple-descent behavior.
It is worthmentioning that, from the simulation, the deviation of the test error from its mean is relatively largewhen m is small but vanishes quickly as m becomes larger.
This suggests Theorem 2 and Corollary 1should hold in a pointwise fashion, i.e. without averaging the test error over random labels.
Small Spectral Gap Eliminates Multiple-descents.
In Fig.
1, we plot both the predicted learningcurves and simulations when Gap ranging in [2 , 8 , 32 , 128] .
For 1 ≤ r ≤ 6 , we have ξ r =
Gap − ( r − 1) / (cid:80) 7 k = r
Gap − ( k − 1) and ξ r ≈ Gap when Gap is large.
Recall that the variance term V rpeaks at α = 1 and the peak scales like ξ 1 / 2 r ≈ Gap 1 / 2 .
When Gap is large, e.g. Gap = 32 , 128 ,the variance is also large near α = 1 , the multiple-descent phenomena are more prominent.
On theother hand, when Gap is small, e.g. Gap = 8 , 2 , such phenomena disappear and learning curvesbecome monotonic.
Conclusion
In this work, we establish precise asymptotic formulas for the sample-wise learning curves in thekernel ridge regression setting for a family of dot-product kernels in the polynomial scaling regimesm ∝ d r for all r ∈ N ∗ .
We demonstrate that these formulas can capture empirical learning curvessurprisingly well even in the regime where strong ﬁnite-size corrections would be expected.
Werigorously prove that the learning curves can be non-monotonic near m ∝ d r /r !
for each r ∈ N ∗ .
There are a couple limitations of our approach which could be improved in future work.
The ﬁrst oneis the strong assumption on the distribution of the input data, namely, the uniform distribution onthe spherical type of data.
In addition, the learning curves are obtained only in the kernel regressionsetting and extending the results to the feature learning setting would be a meaningful future direction.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 7(c)
Did you discuss any potential negative societal impacts of your work?
[No] Puretheoretical work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] We did not
provide code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec. 6(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 64.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[TODO]
(b) Did you mention the license of the assets?
[TODO]
(c) Did you include any new assets either in the supplemental material or as a URL?
[TODO]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[TODO]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[TODO]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[TODO]
(b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[TODO]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[TODO]