T HE E IGENLEARNING F RAMEWORK : A C ONSERVATION L AW P ERSPECTIVE ON K ERNEL R EGRESSION AND W IDE N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We derive a simple unified framework giving closed-form estimates for the test risk and other generalization metrics of kernel ridge regression (KRR).
Relative to prior work, our derivations are greatly simplified and our final expressions are more readily interpreted.
These improvements are enabled by our use of a sharp conservation law which limits the ability of KRR to learn any orthonormal basis of functions.
Test risk and other objects of interest are expressed transparently in terms of our conserved quantity evaluated in the kernel eigenbasis.
We use our improved framework to: i) provide a theoretical explanation for the “deep bootstrap” of Nakkiran et al.
(2020), ii) generalize a previous result regarding the hardness of the classic parity problem, iii) fashion a theoretical tool for the study of adversarial robustness, and iv) draw a tight analogy between KRR and a wellstudied system in statistical physics.
I NTRODUCTION
Kernel ridge regression (KRR) is a popular, tractable learning algorithm that has seen a surge of attention due to equivalences to infinite-width neural networks (NNs)
(Lee et al., 2018;
Jacot et al., 2018).
In this paper, we derive a simple theory of the generalization of KRR that yields estimators for many quantities of interest, including test risk and the covariance of the predicted function.
Our framework is consistent with other recent works, such as those of Canatar et al.
(2021) and Jacot et al.
(2020), but is significantly simpler to derive and much more interpretable, facilitating its application to new problems.
This improvement is made possible by our utilization of a conservation law latent in KRR which limits any kernel’s ability to learn any complete basis of target functions.
Our conserved quantity, which we call “learnability,” is the inner product of the target and predicted functions and, as we show, can be interpreted as a measure of how well the target function can be learned by a particular kernel given n training examples.
We prove that the total learnability, summed over a complete basis of target functions (such as the kernel eigenbasis), is no greater than the number of training samples, with equality at zero ridge parameter.
This formal conservation law makes concrete the recent folklore intuition that, given n samples, KRR and wide NNs can reliably learn at most n orthogonal functions (Mei & Montanari, 2019; Bordelon et al., 2020).
The conservation of this quantity suggests that it will prove useful for understanding the generalization of KRR.
This intuition is borne out by our subsequent analysis: we derive a set of simple, closed-form estimates for test risk and other objects of interest and find that all of them can be transparently expressed in terms of eigenmode learnabilities.
Our expressions are more compact and readily interpretable than those of prior work and constitute a major simplification.
Our derivation of these estimators is significantly simpler and more accessible than those of prior work, which relied on the heavy mathematical machinery of replica calculations and random matrix theory to obtain comparable results.
By contrast, our approach requires only basic linear algebra, leveraging our conservation law at a critical juncture to bypass the need for advanced techniques.
We use our improved framework to shed light on several topics of interest:
i)
We provide a compelling theoretical explanation for the “deep bootstrap” phenomenon of Nakkiran et al.
and identify two regimes of NN fitting occurring at early and late training times.
ii)
We generalize a previous result regarding the hardness of the parity problem for rotationinvariant kernels.
Our technique is simple and illustrates the power of our framework.
iii)
We craft an estimator for predicted function smoothness , a new tool for the theoretical study of adversarial robustness.
iv)
We draw a tight analogy between our framework and the free Fermi gas, a well-studied statistical physics system, and thereby transfer insights into the free Fermi gas over to KRR.
We structure these applications as a series of vignettes.
The paper is organized as follows.
We give preliminaries in Section 2.
We define our conserved quantity and state its basic properties in Section 3.
We characterize the generalization of KRR in terms of this quantity in Section 4.
We check these results experimentally in Section 5.
Sectionconsists of a series of short vignettes discussing topics (i)-(iv).
We conclude in Section 7.
1. R ELATED WORK
The present line of work has its origins with early studies of the generalization of Gaussian process regression (Opper, 1997; Sollich, 1999), with Sollich (2001) deriving an estimator giving the expected test risk of KRR in terms of the eigenvalues of the kernel operator and the eigendecomposition of the target function.
We refer to this result as the “omniscient risk estimator 1 ,” as it assumes full knowledge of the data distribution and target function.
Bordelon et al.
(2020) and Canatar et al.
(2021) brought these ideas into a modern context, deriving the omniscient risk estimator with a replica calculation and connecting it to the “neural tangent kernel” (NTK) theory of wide neural networks (Jacot et al., 2018), with (Loureiro et al., 2021) extending the result to arbitrary convex losses.
Sollich & Halees (2002); Caponnetto & De Vito (2007); Spigler et al.
; Cui et al.
(2021); Mallinar et al.
(2022) study the asymptotic consistency and convergence rates of KRR in a similar vein.
Jacot et al.
Wei et al.
(2022) used random matrix theory to derive a risk estimator requiring only training data.
In parallel with work on KRR, Dobriban & Wager (2018);
Wu & Xu (2020); Richards et al. ; Hastie et al.
and Bartlett et al.
(2021) developed equivalent results in the context of linear regression using tools from random matrix theory.
In the present paper, we provide a clarifying new language for the discussion of this rich body of work in terms of a conserved quantity, provide simplified derivations of the main results of this line of work, and break new ground with applications to new problems of interest.
We compare selected works with ours and provide a dictionary between respective notations in Appendix A.
All prior works in this line — and indeed most works in machine learning theory more broadly — rely on approximations, asymptotics, or bounds to make any claims about generalization.
The conservation law is unique in that it gives a sharp equality even at finite dataset size.
This makes it a particularly robust starting point for the development of our framework (which does later make approximations).
In addition to those listed above, many works have investigated the spectral bias of neural networks in terms of both stopping time (Rahaman et al., 2019; Xu et al., 2019b;a; Xu, 2018; Cao et al., 2019;
Su & Yang, 2019) and the number of samples (Valle-Perez et al., 2018; Yang & Salman, 2019;
Arora et al., 2019).
Our investigation into the deep bootstrap ties together these threads of work: we find that the interplay of these two sources of spectral bias is responsible for the deep bootstrap phenomenology.
P RELIMINARIES AND NOTATION
We study a standard supervised learning setting in which n training samples D ≡ { x i } ni =1 are drawn i.i.d. from a distribution p over R d .
We wish to learn a (scalar) target function f given noisy evaluations y ≡ ( y i ) ni
=1 with y i =
f ( x i ) + η i , with η i ∼ N (0 , ϵ 2 ) .
As it simplifies later analysis, we assume this N (0 , ϵ 2 ) label noise is also applied to test targets 2 .
Our results are easily generalized to vector-valued functions as in Canatar et al.
For scalar functions g h , we define ⟨ g, h ⟩ ≡ E x ∼ p
[ g ( x ) h ( x )] and
|| g || 2 ≡ ⟨ g, g ⟩ .
We shall study the KRR predicted function ˆ f given by
[Equation]
where, for a positive-semidefinite kernel K , we have constructed the row vector
[ k x D ] i
= K ( x, x i ) and the empirical kernel matrix [ K DD ] ij =
K ( x i , x j ) (which we trust to be nonsingular), δ is a ridge parameter and I n is the identity matrix.
We wish to minimize test mean squared error (MSE)
E ( D ) ( f ) =
||
f − ˆ f ||
+ ϵ 2 and its expectation over training sets E ( f ) =
E D (cid:2) E ( D ) ( f ) (cid:3) (where the expectation over D also averages over noise values).
We emphasize that, here and in our discussion of learnability, ˆ f is understood to be the KRR predictor given by Equation 1 from training targets generated with target function f .
In the classical bias-variance decomposition of test risk, we have bias B ( f ) =
||
f − E D [ ˆ f ]
||
+ ϵ 2 and variance V ( f ) =
E ( f ) − B ( f ) .
We also define train MSE as
E tr ( f ) = 1 n (cid:80) ni
{+=
 ( f ( x i ) − ˆ f ( x i )) 2 .+}
2. T HE KERNEL EIGENSYSTEM
By Mercer’s theorem Mercer (1909), the kernel admits the decomposition K ( x, x ′ ) =
[Equation] i λ i ϕ
i ( x ) ϕ
i ( x ′ ) , with eigenvalues λ i ≥ 0 and a basis of eigenfunctions ϕ
i satisfying ⟨ ϕ i , ϕ j ⟩ = δ ij .
We assume eigenvalues are indexed in descending order.
As the eigenfunctions form a complete basis, we are free to decompose f and ˆ f as
[Equation]
where v ≡ ( v i ) i and ˆv ≡ (ˆ v i ) i are vectors of eigencoefficients
L EARNABILITY AND ITS CONSERVATION LAW
Here we define learnability, our conserved quantity.
Learnability is a measure of ˆ f defined similarly to test MSE, but it is linear instead of quadratic.
For any function f such that ||
f ||
= 1 , let
L ( [Equation] [Equation] ≡ ⟨ f, ˆ f ⟩ and L
[Equation] where, as with MSE, ˆ f is given by Equation 1.
We refer to L ( D ) ( f ) the D -learnability of function f with respect to the kernel and n and refer to L ( f ) as the learnability .
Up to normalization, this quantity is akin to the cosine similarity between f and ˆ f .
We shall show that, for KRR, learnability gives a useful indication of how well a function (particularly a kernel eigenfunction) is learned.
Results in this section are rigorous and exact; see Appendix G for proofs.
We begin by stating several basic properties of learnability to build intuition for the quantity.
Proposition 3.1.
The following properties of L ( D ) , L , { ϕ i } , and any f such that
||
f || = 1 hold:
(a) L ( ϕ i ) , L ( D ) ( ϕ i ) ∈ [0 , 1] .
(b) When n = 0 , L ( D ) ( f ) =
L ( f ) = 0 .
(c) Let D + be D ∪ x , where x ∈ X, x / ∈ D is a new data point.
Then L ( D + ) ( ϕ i ) ≥ L ( D ) ( ϕ i ) .
(d) ∂∂λ i L ( D ) ( ϕ i ) ≥ 0 , ∂∂λ i L ( D ) ( ϕ j ) ≤ 0 , and ∂∂δ L ( D ) ( ϕ i ) ≤ 0 .
(e) E ( f ) ≥ B ( f ) ≥ (1 − L ( f )) 2 .
Properties (a-c) together give an intuitive picture of the learning process: the learnability of each eigenfunction monotonically increases from zero as the training set grows, attaining its maximum of one in the ridgeless, maximal-data limit.
Properties (d) shows that the kernel eigenmodes are in competition — increasing one eigenvalue while fixing all others can only improve the learnability of the corresponding eigenfunction, but can only harm the learnabilities of all others — and that regularization only harms eigenfunction learnability.
Property (e) gives a lower bound on MSE in terms of learnability and will be useful when we discuss the parity problem.
We now state the conservation law obeyed by learnability.
This rule follows from the view of KRR as a projection of f onto the n -dimensional subspace of the RKHS defined by the n samples and is closely related to the “dimension bound” for linear learning rules given by Hsu (2021).
Theorem 3.2 (Conservation of learnability) .
For any complete basis of orthogonal functions F , when ridge parameter δ = 0 , (cid:88)
[Equation]
and when δ > 0 , (cid:88)
[Equation]
[Equation]
This result states that, summed over any complete basis of target functions, total learnability is at most the number of training examples, with equality at zero ridge 3 .
This theorem is a stronger version of the classic “no-free-lunch” theorem for learning algorithms, which states that, averaged over all target functions, all models perform at chance level (Wolpert, 1996). While deep and compelling, this classic result is rarely informative in practice because the set of all possible target functions is prohibitively large to make a nonvacuous statement.
By contrast, Theorem 3.2 requires only an average over a basis of target functions and, as we shall see, it is directly informative in understanding the generalization of KRR.
T HEORY
Eigenmode learnabilities are interpretable quantities, obeying a conservation law and several intuitive properties.
These features suggest they may prove a wise choice of variables in a theory of
KRR generalization.
Here we show that this is indeed the case: we derive a suite of estimators for various metrics of KRR generalization, all of which can be expressed entirely in terms of modewise learnabilities and thereby inheriting their interpretability.
Because they characterize KRR learning via eigenmode learnabilities, we call these equations the eigenlearning equations .
We sketch our method here and relegate the derivation to Appendix H. Our derivation leverages our conservation law to avoid the need for either a replica calculation or random matrix theoretic tools and is consequently more accessible and extensible than derivations of prior works. Results in this section are nonrigorous.
Like comparable works, our derivations use an experimentally-validated “universality” assumption that the kernel features may be replaced by independent Gaussian features with the same statistics without changing downstream generalization metrics.
We begin by observing that ˆv depends linearly on v , and we can thus construct a “learning transfer matrix” T ( D ) such that
ˆv = T ( D ) v .
T ( D ) is equivalent to the “KRR reconstruction operator” of Jacot et al.
and, viewing KRR as linear regression in eigenfeature space, is essentially the “hat” matrix of linear regression.
We then study E (cid:2) T ( D ) (cid:3) , leveraging our universality assumption to show that it is diagonal with diagonal elements of the form λ i /
( λ i + κ ) , where κ is a modeindependent constant.
We show the mode-independence of κ with an eigenmode-removal argument reminiscent of the cavity method of statistical physics (Del Ferraro et al., 2014).
We use Theorem 3.2 to determine κ .
Differentiating this result with respect to kernel eigenvalues, we obtain the covariance of T ( D ) and thus of ˆv , which permits evaluation of various test metrics 4 .
4. T HE EIGENLEARNING EQUATIONS
Let the effective regularization κ be the unique positive solution to
[Equation]
We calculate various test and train metrics to be (eigenmode learnability)
[Equation]
(overfitting coefficient)
[Equation]
bias
[Equation]
(variance of test MSE)
[Equation]
predictor
[Equation]
(covariance of predictor)
[Equation]
The learnability of an arbitrary normalized function can be computed as L ( f ) =
(cid:80)
i
L i v i .
The mean of the target function evaluated at input x can be obtained as E
[ ˆ f ( x )] = (cid:80) i E [ v i ] ϕ
i ( x ) , with the covariance obtained similarly.
4. I NTERPRETATION OF THE EIGENLEARNING EQUATIONS
Remarkably, all test metrics, and indeed all second-order statistics of ˆ f , can be expressed solely in terms of modewise learnabilities, with no additional reference to eigenvalues required.
This strongly suggests that we have identified the “correct” choice of variables for the problem.
We now interpret these equations through the lens of learnability.
Equation 6 gives a constant κ which decreases monotonically as n increases.
Equation 7 states that the learnability of eigenmode i is 0 when κ ≫ λ i and approaches 1 when κ ≪ λ
i , in line with the observation of Jacot et al.
that a mode is well-learned when κ ≪ λ i .
Inserting Equationinto 6 and setting δ = 0 , we recover our conservation law.
Equation 9 is the omniscient risk estimator for test MSE.
Modes with learnability equal to one are fully learned and do not contribute to the risk.
Noise acts the same as target weight placed in modes with learnability zero.
Equation 8 defines E 0 , the MSE when trained on pure-noise targets ( v i =and ϵ 2 = 1 ).
It is strictly greater than one and can be interpreted as the factor by which pure noise is overfit 5 .
The denominator explodes when the n units of learnability are fully allocated to the first n modes, not distributed among a greater number ( L i ≤ n = 1 , L i>n = 0 ).
In this sense, overfitting of noise is “overconfidence” on the part of the kernel that the target function lies in the top- n subspace, and “hedging” via a wider distribution of learnability (or sacrificing a portion of the learnability budget to the ridge parameter) lowers E 0 and fixes this problem.
This overconfidence occurs when the kernel eigenvalues drop sharply around index n and is the cause of double-descent peaks (Belkin et al., 2019), which other works have also found can be avoided with an appropriate ridge parameter
Canatar et al.
; Nakkiran et al.
Equations 10 and 11 show that, remarkably, the bias and variance can be expressed solely in terms of E ( f ) and E 0 .
Since learnabilities strictly increase as n grows, the bias strictly decreases while the variance can be nonmonotone, as also noted by Canatar et al.
Equation 12 states that train error is related to test error by the target-independent proportionality constant δ 2 /κ 2 .
Finally,
Equations 13 and 14 give the mean and covariance of the predicted eigencoefficients.
Different eigencoefficients are uncorrelated, and the variance of ˆv i is proportional to L 2 i but surprisingly independent of v i .
E XPERIMENTS
Here we describe experiments confirming our main results.
The targets are eigenfunctions on three synthetic domains — the unit circle discretized into M points, the (Boolean) hypercube {± 1 } d , and the d -sphere — as well as two-class subsets of MNIST and CIFAR-10.
Unless otherwise stated, experiments use a fully-connected four-hidden-layer ReLU architecture, and finite networks have width 500.
The kernel eigenvalues on each synthetic domain group into degenerate sets which we index by k ∈ Z + .
Eigenvalues and eigencoefficients for image datasets are approximated numerically from a large sample of training data.
Full experimental details can be found in Appendix
B.
Figure 1 illustrates Theorem 3.2 in a toy setting.
Modewise D -learnabilities indeed sum to n .
Figure 2 compares theoretical predictions for learnability and MSE with experiments on real and synthetic data, finding good agreement in all cases.
Appendix C repeats one of these experiments with network widths varying from ∞ to 20 , finding good agreement with theory even at narrow width.
V IGNETTES 6. E XPLAINING THE DEEP BOOTSTRAP
The deep bootstrap (DB) is a phenomenon observed by Nakkiran et al.
(2020) in which the performance of a neural network stopped after a given number of training steps is relatively insensitive to the size of the training set, unless the training set is so small that it has been interpolated.
The DB has been studied theoretically on kernel gradient flow (KGF), which describes the training of wide neural networks, by Ghosh et al.
(2022) in the toy case in which the data lies on a high-dimensional
L ( f )
A
Unit Circle k =
k
=2 k = k
=
B 8d
Hypercube k = k =1 k =3 k
=
C 7-sphere k
=
k
=1 k =2 k = k =
k =
D Image Datasets CIFAR10 deer/horse CIFAR10 bird/truck
MNIST 3/8 MNIST 0/ [-
 [Figure]-]22222 8 n
E ( f )
E k =
k
=2 k = k = [-
 [Figure] [Figure]-]22222 8 n
F k =
k
=1 k =3 k
= [-
 [Figure] 2-]22222 8 n
G k =
k
=1 k =2 k =3 k
=4 k {+= 2 
+}
[Figure]
10 n H
MNIST
[Figure]
[Figure]
[Figure]
sphere.
Here we give a convincing general explanation of this phenomenon using our framework and identify two regimes of NN fitting in the process.
Ali et al.
(2019) proved that KRR with finite ridge generalizes remarkably similarly to KGF with finite stopping time (a theoretical result confirmed empirically by Lee et al.
(2020) for NTKs on image datasets).
When considering a standard supervised learning setting, the effective training time corresponding to a ridge δ is τ eff ≡ δ − 1 n (see Appendix D for a discussion of this scaling).
As a proxy for KGF, we shall study KRR as τ eff increases from 0 to ∞ .
In Equation 9, the ridge parameter affects MSE solely through the value of κ .
Define κ 0 ≡ κ | δ =0 , the minimum effective regularization at a given dataset size.
In Appendix D, we show that, for powerlaw eigenspectra (as are commonly found in practice), there are two regimes of fitting:
Regularization-limited regime :
τ eff ≪ κ − 1and κ ≈ τ − 1 eff .
The generalization gap is small: E ( f ) / E tr ( f ) ≈ 1 .
Regularization dominates generalization, and adding training samples does not affect generalization 6 . 2.
Data-limited regime : τ eff ≫ κ −
1and κ ≈ κ 0 .
The generalization gap is large: E ( f ) / E tr ( f ) ≫ 1 .
Data is interpolated, and decreasing regularization (i.e. increasing training time) does not affect generalization.
We suggest that Nakkiran et al.
(2020) observe overlapping error curves for different n at early times because, at these times, the model is in the regularization-limited regime.
We now present an experiment confirming our interpretation.
In Figure 3, we reproduce an experiment of Nakkiran et al.
(2020) illustrating the DB using ResNets trained on CIFAR-10, juxtaposing it with our proposed model for this phenomenon — KRR with varying τ eff — trained on binarized
MNIST.
The match is excellent: in particular, both plots share the DB phenomenon that error curves for all n overlap at early times, with test and train error peeling off the master curve at roughly the same time.
We find that τ eff ≈ κ − 1 0 is indeed when the transition between regimes occurs for KRR, matching our theoretical prediction.
This experiment strongly suggests that both neural network and KRR fitting can be thought of as a transition between regularization-limited and data-limited regimes.
See Appendix D for experimental details.
6. T HE HARDNESS OF THE PARITY PROBLEM FOR ROTATION - INVARIANT KERNELS
The parity problem stands as a classic example of a function which is easy to write down but hard for common algorithms to learn.
The parity problem was shown to be exponentially hard for Gaus- [Figure] sian kernel methods by Bengio et al.
Here we generalize this result to KRR with arbitrary rotation-invariant kernels.
Our analysis is made trivial by the use of our framework and is a good illustration of the power of working in terms of learnabilities.
The problem domain is the hypercube X
= {− 1 , +1 } d , over which we define the subset-parity functions ϕ S ( x ) = ( − 1) (cid:80) i ∈
S
[ x i =1] , where S ⊆ { 1 , ..., d } ≡ [ d ] .
The objective is to learn ϕ
[ d ] .
For any rotation-invariant kernel (such as the NTK of a fully-connected neural network), { ϕ S } S are the eigenfunctions over this domain, with degenerate eigenvalues { λ k } dk =0 depending only on k = | S | .
Yang & Salman (2019) proved that, for any fully-connected kernel, the even and odd eigenvalues each obey a particular ordering in k .
Letting d be odd for simplicity, this result and
Equation 7 imply that L 1 ≥ L 3 ≥ ... ≥ L d .
Counting level degeneracies, this is a hierarchy of 2 d −learnabilities of which L d is the smallest.
The conservation law of 3.2 then implies that L d ≤ n 2 d − 1 , which, using Proposition 3.1(e), implies that
[Equation]
Obtaining an MSE below a desired threshold ϵ thus requires at least n min = 2 d − 1 (1 − ϵ 1 / 2 ) samples, a sample complexity exponential in d .
The parity problem is thus hard for all rotation-invariant kernels 7 .
6. M EAN - SQUARED GRADIENT AND ADVERSARIAL ROBUSTNESS
While the omniscient risk estimate is the most important of the eigenlearning equations, we have also obtained estimators for arbitrary covariances of ˆ f .
Here we point out a first use for these covariances: studying the smoothness of ˆ f and thus its adversarial robustness.
We fashion an estimator for function smoothness, confirm its accuracy with KRR experiments, and identify a discrepancy between intuition and experiment ripe for further exploration.
Consider the mean squared gradient (MSG) of ˆ f defined by G ( ˆ f ) ≡ E x (cid:104) |∇
x ˆ f ( x ) | 2 (cid:105)
=
||∇ ˆ f || 22 .
This quantity is a measure of function smoothness.
Eigendecomposition yields that
[Equation]
with g [Equation] ≡ E [Equation]
[Equation]
ϕ
[Equation]
ϕ
[Equation] x [Equation] The expectation E [ ˆv i ˆv j ] =
E [ ˆv i ]
E [ ˆv j ] + Cov [ ˆv i , ˆv j ] is given by the eigenlearning equations, and the structure constants g ij , which encode information about the domain, can be computed analytically for simple domains.
On the d -sphere, for which the ϕ i = ϕ
kℓ are spherical harmonics, these are g ( kℓ ) , ( k ′ ℓ ′ )
=
k ( k + d − 2) δ kk ′
δ ℓℓ ′ .
2222222n 0.
G ( ˆ f ) / G ( f ) d =4 d = d =64 d = d =
Figure 4 shows the MSG of the function learned by KRR with a polynomial kernel trained on k = 1 modes on spheres of increasing dimension d , normalized by the MSG of the groundtruth target function.
See Appendix E for experimental details and additional experiments in this vein.
True MSG matches predicted MSG well in all settings, particularly at large n and d .
The study of adversarial robustness currently suffers from a lack of theoretically tractable toy models, and we suggest our expression for
MSG can help fill this gap.
To illustrate this, we describe an insight that can be drawn from
Figure 4.
Vulnerability to gradient-based adversarial attacks can be viewed essentially as a phenomenon of surprisingly large gradients with respect to the input.
If such vulnerability is an inevitable consequence of high dimension, a common heuristic belief (e.g. Gilmer et al.
, one might expect that G ( ˆ f ) is generally much larger than G ( f ) at high dimension.
Surprisingly, we see no such effect, a discrepancy which can be investigated further using our framework.
6. A QUANTUM MECHANICAL ANALOGY AND UNIVERSAL LEARNABILITY CURVES
Here we describe a remarkably tight analogy between our picture of KRR generalization and the statistics of the free Fermi gas , a canonical model in statistical physics.
This allows certain insights into the free Fermi gas to be ported over to KRR.
This correspondence is also of fundamental interest: KRR and the free Fermi gas are both paradigmatic systems in their respective fields, and it is remarkable that their statistics are in fact the same.
We defer the details of this correspondence to Appendix F and focus here on the takeaways. The free Fermi gas is defined by a scalar µ and a set of states with energies { ε i } i , each of which may be occupied or not.
We find that − ln κ is analogous to µ , the state energies are analogous to kernel eigenvalues, and the states’ occupation probabilies are precisely analogous to the eigenmodes’ learnabilities.
In all prior work and in our work thus far, the constant κ has thus far been defined only as the solution to an implicit equation. Having an explicit equation would be advantageous.
Leveraging methods for the study of the free Fermi gas, we find the following explicit formula for κ : [Equation] where [Equation]
The second takeaway is the identification of a universal behavior in the learning dynamics of KRR.
In systems obeying Fermi-Dirac statistics, a plot of p i vs. ε i takes a characteristic sigmoidal shape.
As shown in Figure 10, we robustly see this sigmoidal shape in plots of L i vs. ln
λ i .
As the number of samples and even the task are varied, this sigmoidal shape remains universal, merely translating horizontally (in particular, moving left as samples are added).
This sigmoidal shape is thus a signature of KRR and could in principle be used to, for example, determine if an unknown kernel method resembles KRR.
C ONCLUSIONS
We have developed an interpretable unified framework for understanding the generalization of KRR centered on a previously-unexploited conserved quantity.
We then used this improved framework to break new theoretical ground in a variety of subjects including the parity problem, the deep bootstrap, and adversarial robustness and developed a tight analogy between KRR and canonical physical system allowing the transfer of insights.
We have covered much territory, and each of these subjects is ripe for further
A N OTATION DICTIONARY AND COMPARISON WITH RELATED WORKS
Tables 1, 2 and 3 provide a dictionary between the notations of our paper and the nearest related works.
Rows should be read in comparison with the top row and interpreted as “when the present paper writes X, the other paper would write Y for the same quantity.”
Comparing expressions for test
MSE in Table 2 and predicted function covariance in Table 3 makes it very clear that our learnability framework permits simpler and much more interpretable expression of these results than previously given.
[Table]
Canatar et al
[Table] [Table]
Jacot et al
[Table]
[Table]
[Table]
λn
[Table]
[Equation]
Canatar et al.
(2021) κn
[Equation]
Jacot et al.
ϑ
[Equation]
Cui et al
[Equation]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Table]
Sollich (2001) derived the omniscient risk estimate in the context of GP regression and was, to our knowledge, the first to obtain the result.
As its notation does not map cleanly onto the variables we use, we do not include this work in the above tables.
Other analogous works include Wu &
Xu (2020), Richards et al. (2021), Hastie et al. (see Definition 1) and Bartlett et al.
(2021) (see Theorem 4.13) [Table]
which derive comparable equations for the test MSE of linear ridge regression (LRR), which, taking the dual view of KRR as LRR in the kernel embedding space
[Table]
is essentially the same problem. All comparable works involve solving a self-consistent equation for an effective regularization parameter, which we call κ in our work.
It is worth pointing out that this literature is largely divided into two parallel camps: the works in the tables above, together with Loureiro et al.
(2021) and Cohen et al.
(who study GP regression using field theoretic tools), study the generalization of KRR, largely using approaches from statistical physics, while Dobriban & Wager (2018); Wu & Xu (2020); Richards et al. ; Hastie et al.
; Bartlett et al.
(2021) study LRR using random matrix theory.
Despite studying virtually the same problem, these camps have largely developed unaware of each other, and we hope the present paper helps unify them.
The KRR works all rely on approximations, particularly the spectral “universality” approximation that the eigenfunctions are random and structureless and the design matrix can thus be replaced by a random Gaussian matrix.
The works of Bordelon et al.
Canatar et al.
(2021), as well as ours, make additional approximations valid for reasonable eigenspectra and large n , while Jacot et al.
(2020) does not make approximations and provides bounds instead (though these bounds diverge at zero ridge, highlighting the difficulty of studying interpolating methods).
On the other side, the LRR works do not make a unversality approximation, but instead assume in their setting that the feature vectors are high-dimensional and have (sub)Gaussian moments (or some similar condition), which ultimately amounts to the same condition, now enforced in the setting instead of assumed as an approximation.
These works are typically mathematically rigorous.
Relative to the LRR works, one contribution of the KRR works is the empirical observation that this universality approximation is generally accurate in practice 11 , even for NTKs at initialization (for NTKs after training, the story appears mixed; see Wei et al.
for evidence for universality and Ba et al.
(2022) for evidence against it).
While our work is generally consistent with other works in the KRR set, we note one discrepancy regarding the covariance of the predicted function.
Our expression, Equation 14, agrees with that of Canatar et al.
However, the expression of Jacot et al.
(2020), given in Theorem 2, contains an extra O ( n − 1 ) term (blue in Table 3) dependent upon the target eigencoefficient of the mode in question.
This term is small enough that it can be removed without affecting the bounds the authors prove.
It contributes negligibly when summing over all eigenmodes to compute e.g. test error or mean squared gradient, but contributes to leading order when interrogating the variance of a particular mode.
A sanity-check calculation that the sum of variance over all modes ought to equal
V ( f ) provides evidence that our equations, which lack this term, are correct.
Lastly, we note that Cohen et al.
(2021) study Gaussian process regression in the spirit of Sollich (1999) using field theoretic tools.
Though they do not explicitly discuss the omniscient risk estimate,
Canatar et al.
(2021) note that their “equivalence kernel + corrections” learning curve can be viewed as a perturbative expansion of the omniscient risk estimate.
B E XPERIMENTAL METHODS B. S YNTHETIC DOMAINS
Our experiments use both real image datasets and synthetic target functions on the following three domains:
Discretized Unit Circle.
We discretize the unit circle into M points, X
= { (cos(2 πj/M ) , sin(2 πj/M ) } M j =1 .
Unless otherwise stated, we use M = 256 .
The eigenfunctions on this domain are ϕ
( θ ) = 1 , ϕ k ( θ ) =
√ 2 cos( kθ ) , and ϕ ′
k ( θ ) = √ 2 sin( kθ ) , for k ≥ 1 .
Hypercube.
We use the vertices of the d -dimensional hypercube X = {− 1 , 1 } d , giving M = 2 d .
As stated in Section 6.2, the eigenfunctions on this domain are the subset-parity functions with eigenvalues determined by the number of sensitive bits k . 3.
Hypersphere.
To demonstrate that our results extend to continuous domains, we perform experiments on the
d -sphere S d ≡ { x ∈ R d +1 | x 2 = 1 } .
The eigenfunctions on this domain are the hyperspherical harmonics (see, e.g., Frye & Efthimiou (2012);
Bordelon et al.
which group into degenerate sets indexed by k ∈ N .
The corresponding eigenvalues decrease exponentially with k , and so when summing over all eigenmodes when computing predictions, we simply truncate the sum at k max = 70 .
Eigenvalues and multiplicities for these synthetic domains are shown in Figure 5.
[Figure]
B. I MAGE DATASET EIGENINFORMATION
Experiments with binary image classification tasks used scalar targets of ± 1 .
To obtain the eigeninformation necessary for theoretical predictions, we select a large training set with M ∼ O (10 4 ) samples, computing the eigensystem of the data-data kernel matrix to obtain M eigenvalues { λ i } i and target vector eigencoefficients v .
We then compute theoretical learnability and MSE as normal.
The resulting test MSE predictions obtained this way correspond to an average over the M points in the sample, n of which are in fact part of the training set (compare with the discrete and continuous settings of Appendix H.1, and thus they are too low when n/M is not negligible.
We wish to predict purely the off-training-set error E OTS .
We do so by noting that M E = n E tr + ( M − n ) E OTS and solving for E OTS (doing the same thing for for learnability).
Another solution to this problem is to change the experimental design so train and test data are sampled from the same large pool (as done by Bordelon et al.
(2020) and Canatar et al.
, but this trick allows us a more standard setup in which train and test data are disjoint.
Experimental test metrics for image datasets are computed with a random subset of M ′ ∼ O (1 k ) images from the test set.
Eigenvalues and multiplicities for four two-class image datasets are shown in Figure 6.
Kernel eigenvalues as computed from
training points as described in Appendix B
Spectra for CIFAR10 tasks roughly follow power laws with exponent − 1 , while spectra for MNIST tasks follow power laws with slightly steeper descent
) Eigencoefficients as computed from 10 4 training points.
Tasks with higher observed learnability (Figure 2) place more weight in higher (i.e., lower-index) eigenmodes and less in lower ones.
B. R UNTIMES
For the dataset sizes we consider in this paper, exact NTK regression is typically quite fast, running in seconds, while the training time of finite networks varies from seconds to minutes and depends on width, depth, training set size, and eigenmode.
In particular, as described by Rahaman et al.
(2019), lower eigenmodes take longer to train (especially when aiming for near-zero training MSE as we do here).
B. H YPERPARAMETERS
We conduct all our experiments using JAX (Bradbury et al., 2018), performing exact NTK regression with the neural tangents library (Novak et al., 2019) built atop it.
Unless otherwise stated, all experiments used four-hidden-layer ReLU networks initialized with NTK parameterization (SohlDickstein et al., 2020) with σ w = 1 . 4 , σ b = .
The tanh networks used in generating Figureinstead used σ w = 1 . 5 .
Experiments on the unit circle always used a learning rate of . 5 , while experiments on the hypercube, hypersphere, and image datasets used a learning rate of . 5 or .depending on the experiment.
While higher learning rates led to faster convergence, they tended to match theory more poorly, in line with the large learning rate regimes described by Lewkowycz
• The toy experiment of Figure 1 used only a single trial per eigenmode by design.
• Other experiments on synthetic domains used 30 trials.
• The experiments on image datasets in Figure 2 used 15 trials.
B. I NITIALIZING THE NETWORK FUNCTION AT ZERO
Naively, when training an infinitely-wide network, the NTK only describes the mean learned function, and the true learned function will include an NNGP-kernel-dependent fluctuation term reflecting the random initialization (Lee et al., 2019).
However, by storing a copy of the parameters at t = 0 and redefining ˆ f t ( x ) :
= ˆ f t ( x ) − ˆ f 0 ( x ) throughout optimization and at test time, this term becomes zero.
We use this trick in our experiments with finite networks.
C V ARYING WIDTH EXPERIMENT
While kernel regression describes only infinitely-wide neural networks, it is natural to wonder whether our equations are nonetheless informative outside this regime.
Figure 7 compares predicted learnability and MSE with experiment for hypercube eigenmodes with networks with varying widths.
Our predictions remain informative down to width 50 , and lower-frequency modes are better learned at all widths, suggesting that kernel eigenanalysis has a role to play even in the study of feature learning
[Figure]
[Figure]
D E XPLAINING THE DEEP BOOTSTRAP IN KRR
In this appendix, we use the eigenlearning equations to arrive at an explanation of the deep bootstrap phenomenon of Nakkiran et al.
As explained in the main text, we study KRR as a proxy for
KGF.
We begin by finding the relationship between the ridge parameter and the effective training time (we assume for simplicity that the learning rate of KGF is one).
Ali et al.
(2019) obtained their result regarding the similarity of KRR and KGF 13 under the identifcation [ time ] =
[ ridge ] − 1 .
They assume a different ridge scaling than ours, and, in our notation, we identify τ eff = δ − 1 n .
Their KGF scaling is correct for modeling a standard supervised learning setup, as can be verified by taking the
KGF limit of ordinary minibatch SGD.
To study the effect of this “early stopping” on generalization, we must find how finite τ eff affects the constant κ .
Recalling Equation 6 defining κ ,
[Equation]
we can easily see that κ ≤ ( n − 1 (cid:80) i λ i + τ − 1 eff ) and κ ≥ τ − 1 eff ) .
Squeezing κ with these bounds, we find that, when n ≫ τ − 1 eff (cid:80) i λ i , then κ
≈ τ − 1 eff .
For any τ eff , there thus exists some n above which additional data does not further lower κ and permit the learning of additional eigenmodes.
Let us call this the regularization-limited regime , and its counterpart (in which τ eff is sufficiently large and regularization is effectively zero) the data-limited regime .
We can find the crossover regularization provided knowledge of the eigenspectrum.
Let us assume that λ i ∼ i − α for some α > 1 .
Mallinar et al.
(2022) show that κ 0 ≡ κ | δ =
→ (cid:0) α − 1 π csc( α − 1 π ) (cid:1)
α n − α as n → ∞ .
Extending their analysis to finite ridge, we find that, at large n , n = n ( κ/κ 0 ) − 1 /α
+ ( δκ ) − 1 , or equivalently
[Equation]
Inspection of Equation 18 reveals that when τ eff ≪ κ − 1 0 , then κ ≈ τ − 1 eff , and when τ eff ≫ κ − 1 0 , then κ ≈ κ 0 .
We thus expect a crossover from the regularization-limited to the data-limited regime when τ eff ≈ κ 0 .
In Figure 8, we plot the theoretical omniscient risk estimate using powerlaw eigenvalues ( λ i ∼ i − α ) and powerlaw eigencoefficients v 2 i ∼
i − α ) with α = 2 for various n .
The close resemblance of these curves to the experimental KRR curves of Figure 3 confirm that our powerlaw toy model is good.
If desired, Figures 3(A,B) and Figure 8 can be viewed as the distillation of the DB phenomenon in three stages of increasing artificiality.
It is worth noting that increasing n serves to both decrease κ and decrease E 0 .
For powerlaw spectra, as n increases with fixed τ eff , E 0 tends to its lower bound of 1 .
It approaches 1 when the regularization-limited regime begins.
Similar ideas regarding the scaling of KRR are discussed by Cui et al.
The notion of regimes limited by either dataset size or training time is explored using the language of scaling laws by
Kaplan et al.
(2020) and Bahri et al.
We expect that this analysis could be extended to proper KGF using the analysis of Bordelon & Pehlevan (2021).
D. D EEP BOOTSTRAP EXPERIMENTS
In replicating the deep bootstrap phenomenon for neural networks, we use the same experimental setup as Nakkiran et al.
(2020), a ResNet-18 architecture trained on CIFAR-10 with data augmentation (random horizontal flips and random crops).
We optimize cross-entropy loss using SGD, with batchsize 128, momentum 0.9, and initial learning rate 0.1 with cosine decay.
The test/train curves are the mean of 4 training runs; the curves have been gaussian-smoothed to remove high-frequency noise artifacts.
We perform kernel ridge regression (KRR) using the NTK of a fully-connected network with four hidden layers of width 500.
We binarize MNIST into two classes: { 0 , 1 , 2 , 3 , 4 } and { 5 , 6 , 7 , 8 , 9 } .
Although the test curves shift slightly depending on the binarization scheme, the theory curves all fall within one standard deviation of the empirical curves for all binarizations we tried.
The KRR test/train curves are the mean of 10 training runs.
[Figure]
E A DDITIONAL MEAN - SQUARED GRADIENT EXPERIMENTS
In the MSG experiment of 4, we run KRR with a polynomial kernel on data sampled from hyperspheres of varying dimension. The polynomial kernel is K ( x, x ′ )
{+= 1+ x ⊤ x ′ +( x ⊤ x ′ ) 
+}
+( x ⊤ x ′ ) 3 . We perform this experiment using PyTorch (Paszke et al., 2019) and compute ∇ ˆ f numerically. In the MSG experiment of 9, we train finite-width FCNs on eigenmodes on the (discretized) unit circle. 22222n 0.
G ( ˆ f ) / G ( f ) k
=k
=k =k
=
F Q UANTUM MECHANICAL ANALOGY F. E XPLICIT EQUATION FOR κ
Here we elaborate further the physical analogy laid out in Section 6.4.
In our system of noninteracting fermions, we have thus far identified [Equation]
[Equation] [Equation] What is the value of κ ?
Observe that [Equation]
(22) gives the expected occupation number in the grand canonical ensemble where the total number of fermions is allowed to fluctuate. By the equivalence of thermodynamic ensembles, we expect to get the same answer for ⟨ n i ⟩ in the canonical ensemble where the total number of fermions does not fluctuate and is exactly n . This suggests that we should attempt to compute ⟨ n i ⟩ in the canonical ensemble.
By comparing the answer to the grand canonical expression, we can solve for µ and hence for κ . We assume a total number of orbitals M ≫ n (which may be infinite). Note that the equivalence of ensembles holds only in the thermodynamic limit n ≫ 1 , so we must take this limit at the end.
In the canonical ensemble, each microstate is labeled by of a list of indices 1 ≤ j 1 < ... < j n ≤ M corresponding to the occupied orbitals. Direct computation of the canonical partition function shows that [Equation]
where m n is the so-called “elementary symmetric polynomial of order n ”: [Equation] We also define the same with index i disallowed: [Equation]
We find that
[Equation] Comparing Equations 22 and 26, solving for µ , and using Equation 20, we find that [Equation] for any i , where λ
= ( λ i ) Mi =1 .
We are free to choose i ≫ n so that λ i is negligible, yielding
[Equation]
We have arrived at an explicit expression for κ .
This expression “solves” Equation 6, the implicit equation defining κ , in the thermodynamic limit.
Numerics easily confirm a close match with the implicit solution. While many works have defined similar constants implicitly, this is the first place to our knowledge that the explicit solution appears. The meaning of κ is elucidated by analogy to µ in the canonical ensemble. In the canonical ensemble, all orbitals are “fighting” for the supply of n particles, and µ represents their “average pull” on each unit.
Similarly, in KRR, we can view eigenmodes as competing for the kernel’s supply of n units of learnability, with κ encoding their average pull on this supply.
This analogy deepens our picture of KRR as the competition among eigenmodes
F.
In the grand canonical ensemble, the total occupation N = (cid:80) i n i concentrates about its mean n and has fluctuations given by Var[ N ] = (cid:80) i ⟨ n i ⟩ (1 − ⟨ n i ⟩ ) (since each site constitutes a Bernoulli variable). Comparing with equation 8, we find that [Equation]
Smaller particle number fluctuations in the analogous physical system thus correspond to larger E 0 and greater overfitting of noise.
This provides a physical interpretation of the “overfitting as overconfidence” notion of Section 4.2.
The grand canonical partition function of our free Fermi gas is [Equation] The analogous quantity in KRR is
[Equation] [Equation]
It holds that
[Equation]
[Equation]
[-By analogy, our partition function for KRR is Z KRR = (cid:81) i (
 + λ i κ ) ,-] and indeed we find that
[Equation]
[Figure] Unit Circle [Figure] Hypercube
C 7-sphere
[Figure]
F. U NIVERSAL LEARNABILITY CURVES
Figure 10 shows that curves of learnability vs. eigenvalue collapse onto a single sigmoidal curve upon rescaling.
We note that similar collapse of data from different experiments upon rescaling occurs in many important statistical physics systems including superconductors (Kardar, 2007) and turbulent flows (Goldenfeld, 2006).
It is broadly worth noting that informative constants determined by self-consistency conditions are a hallmark of statistical mechanics.
It is thus sensible that such a constant would be emerge from the replica calculation of Canatar et al.
G P ROOFS : LEARNABILITY AND ITS CONSERVATION LAW
In this appendix, we provide proofs of the formal claims of Section 3.
We make use of our learning transfer matrix formalism (discussed in Appendix H.2) for these proofs as it improves clarity and compactness, but all our proofs can also be straightforwardly carried through without it.
The learning transfer matrix is defined as
[Equation]
51) with Φ ij = ϕ i ( x j ) and T ≡ E D (cid:2) T ( D ) (cid:3) .
It holds that ˆv
=
T ( D ) v .
It is easy to see (by making v a one-hot vector) that L ( D ) ( ϕ i ) =
T ( D ) ii and L ( ϕ i )
= T ii .
Property (a): L ( ϕ i ) , L ( D ) ( ϕ i ) ∈ [0 , 1] .
Letting e i be the one-hot unit vector with a one at index i , we observe that
[Equation]
where in the third line we have used the fact that e i is a unit vector and Λ is diagonal, in the fourth line we have defined z = λ 1 / 2 i Φ ⊤ e i and M =
Φ ⊤ ΛΦ + δ I n − zz ⊤ , and in the fifth line we have used the fact that M is positive semidefinite.
Given this, L ( ϕ i ) ∈ [0 , 1] by averaging.
Property (b): L ( D ) ( f ) = L ( f ) = 0 .
Proof.
When n = 0 , the predicted function ˆ f is uniformly zero, and so we have L ( D ) ( f ) =
L ( f ) = 0 .
Remark.
As a corollary, it is easy to show that, when M is finite, n = M , and δ = 0 , then
L ( D ) ( f ) =
L ( f ) = 1 .
Property (c): Let D + be D ∪ x , where x ∈ X, x / ∈ D is a new data point.
Then L ( D + ) ( ϕ i ) ≥
L ( D ) ( ϕ i ) .
We first use the Moore-Penrose pseudoinverse, which we denote by ( · ) + , to cast T ( D ) into the dual form
[Equation]
This follows from the property of pseudoinverses that A ( A ⊤ A + δ I ) + A ⊤ = ( AA ⊤ )( AA ⊤ + δ I ) + for any matrix A .
We now augment our system with one extra data point, getting
[Equation]
where ξ is an M -element column vector.
Equations 38 and 39 yield that
[Equation]
The rightmost expressions of Equations 40 and 41 both contain a factor of the form B ( B + δ I ) + , where A is a symmetric positive semidefinite matrix.
An operator of this form is a projector onto the row space of B when δ = 0 and a variant of this projector with “shrinkage” when δ > 0 .
Comparing these equations, we find that the projectors are the same except that, in Equation 41, there is one additional dimension in the row-space and thus one new basis vector in the projector (provided ξ is orthonormal to the other columns of Φ ; otherwise there are zero additional dimensions).
In the case δ = 0 , this new basis vector cannot decrease e ⊤ i T ( D + ) e i , and thus L ( D + ) ( ϕ i ) ≥
L ( D ) ( ϕ i ) in the ridgeless case.
In the case δ > 0 , a singular value decomposition of the projector confirms that the addition still cannot decrease e ⊤ i T ( D + ) e i .
This shows the desired property.
It follows as a corollary that increasing n → n + 1 cannot decrease L ( f ) .
Property (d): ∂∂λ i L ( D ) ( ϕ i ) ≥ 0 , ∂∂λ i L ( D ) ( ϕ j ) ≤ 0 , and ∂∂δ L ( D ) ( ϕ i ) ≤ 0 .
Proof.
Differentiating T ( D ) jj with respect to a particular λ
i , we find that
[Equation]
[Equation]
where ϕ ⊤ i is the i th row of Φ and K =
Φ ⊤ ΛΦ + δ I n .
Specializing to the case i
=
j , we note that ϕ ⊤
i K − 1 ϕ
i ≥ 0 because K is positive definite, and λ
i ϕ
i K − 1 ϕ ⊤ i ≤ 1 because λ i ϕ i
ϕ ⊤ i is one of the positive semidefinite summands in K =
(cid:80)
k λ k ϕ k ϕ ⊤ k
+ δ I n .
The first clause of the property follows.
To prove the second clause, we instead specialize to the case i ̸
= j , which yields that
[Equation]
(43) which is manifestly nonpositive because λ j > 0 .
The second clause follows.
Differentiating Equation 51 w.r.t.
δ yields that ∂∂δ T ( D ) = − ΛΦK − 2 Φ ⊤ .
We then observe that
[Equation]
which must be nonpositive because λ i > 0 and ΦK − 2 Φ ⊤ is manifestly positive definite.
The desired property follows.
Property (e): E ( f ) ≥ B ( f ) ≥ (1 − L ( f )) 2 .
Noting that || v
||
=
||
f
||
= 1 , expected MSE is given by
[Equation]
It is apparent that E ( f ) ≥ B ( f ) .
Projecting any vector onto an arbitrary unit vector can only decrease its magnitude, and so
[Equation]
G. P ROOF OF T HEOREM 3.2 (C ONSERVATION OF LEARNABILITY )
First, we note that, for any orthogonal basis F on X , (cid:88)
[Equation]
(47) where V is an orthogonal set of vectors spanning R M .
This is equivalent to Tr (cid:2) T ( D ) (cid:3) .
This trace is given by
[Equation]
When δ = 0 , this trace simplifies to Tr[ I n ] = n .
When δ > 0 , it is strictly less than n .
This proves the theorem.
Remark.
Theorem 3.2 is a consequence of the fact that T ( D ) is simply a projector onto an ndimensional space spanned by the embeddings of the n samples.
H D ERIVATIONS : THE EIGENLEARNING EQUATIONS
In this appendix, we derive the eigenlearning equations for the test risk and covariance of the predicted function.
Throughout this derivation, we shall prioritize clarity and interpretability over mathematical rigor, giving a derivation which can be understood without advanced mathematical tools and thereby filling a gap in the literature.
For a formal derivation using random matrix theory, see
Jacot et al.
(2020), and for a derivation using a replica calculation of a similar level of rigor as we use here, see Canatar et al.
In the interest of clarity, we begin with a brief summary of the appendix.
We begin by discussing the problem setting (Section H.1).
We then define the learning transfer matrix formalism (H.2) and use it to set the ridge parameter and noise to zero (H.3).
After stating our main approximation (H.4), we find the expectation of the learning transfer matrix (H.5, H.6), using our conservation law to fix κ (H.7, H.8).
Taking well-placed derivatives, we bootstrap the expectation of the learning transfer matrix to its second order statistics (H.9) and add back the ridge and noise (H.10).
We conclude with various useful bounds on κ (H.11).
H. T HE DATA DISTRIBUTION
We shall, in this derivation, consider a slightly more general setting than discussed in the main text.
In the main text, we supposed the data were drawn i.i.d.
from a continuous distribution p over R d .
Here, in addition to this case, we shall also consider a setting in which the data are sampled without replacement from a discrete set X ⊂ R d with |X| =
M .
We will refer to this as the “discrete setting” and the setting with continuous distribution as the “continuous setting.”
As discussed in Appendix B, the discrete setting matches several of our experiments (namely those on the discretized unit circle and on the vertices of the hypercube).
This discrete setting clearly converges to the continuous setting as M → ∞ : when M ≫ n 2 , the probability of sampling the same point twice is negligible (and so we can drop the “without replacement” and sample with replacement as in the continuous setting) and, by distributing X throughout R d with point density proportional to p , we approach sampling from a continuous distribution.
Alternatively, we could imagine X consists of M i.i.d.
samples from p , so that, when we later sample n points from X , they are themselves i.i.d.
samples from p as M → ∞ .
It is worth noting that the data distribution in e.g. a computer vision task is in fact discrete because pixel values are discretized, and thus it is reasonable to work with a discrete measure.
Kernel eigenmodes in the discrete setting are defined as
[Equation]
Note that, in this case, the number of eigenmodes is M , the same number as the cardinality of X .
In the continuous setting, the number of eigenmodes is infinite (though there may be only finitely many with nonzero eigenvalues), but, like Bordelon et al.
, we will find it useful to assume that we need only consider a finite (but very large) number M .
This serves merely permit us to work with finite matrices (to which the standard tools of linear algebra apply) and to thereby save us the trouble of dealing with infinite and semi-infinite matrices, which require greater care.
So long as
M is sufficiently large, we do not lose anything in discarding the exceedingly small eigenvalue tail λ i>M , as is typical in the study of kernel methods and as our final results will confirm.
Our subsequent derivation will generally apply to both the discrete and continuous settings.
H. T HE LEARNING TRANSFER MATRIX
We begin by translating the KRR predictor into the kernel eigenbasis.
Because K ( x, x ′ ) =
(cid:80)
Mi =
λ i ϕ
i ( x ) ϕ i ( x ′ )
, we can decompose the empirical kernel matrix as K DD =
Φ ⊤ ΛΦ , where Λ ≡ diag( λ 1 , ..., λ M ) and Φ is the M × n “design matrix” given by Φ ij ≡ ϕ i ( x j ) .
The predicted function coefficients ˆv are given by
[Equation]
49) where we have used the orthonormality of the eigenfunctions and defined [ ϕ i ] j = ϕ
i ( x j ) to be the i -th row of Φ .
Stacking these coefficients into a matrix equation, we find
[Equation]
where the learning transfer matrix
[Equation]
is an M × M matrix, independent of f , that fully describes the model’s learning behavior on a training set D 14 .
The learning transfer matrix is the same as the “reconstruction operator” of Jacot et al.
(2020) viewed as a finite matrix instead of a linear operator.
Full understanding of the statistics of T ( D ) will give us the statistics of ˆv and thus of ˆ f , and so our main objective will be to find the mean and the covariance of T ( D ) .
H. S ETTING THE RIDGE AND NOISE TO ZERO
Our setting includes both a nonzero ridge parameter and nonzero noise.
However, it is by this point modern folklore that many small eigenvalues together act as an effective ridge parameter equal to their sum and that power in zero-eigenvalue modes is effectively noise (see e.g. Canatar et al.
(2021) for a discussion of these).
Inverting these equivalences, we should expect to be able to convert δ into a small increase to many eigenvalues and convert ϵ 2 to power in a zero-eigenvalue mode, and thereby permit ourselves to consider neither ridge nor noise in our derivation and add them back at the end.
Here we explain our method for doing so.
The ridge parameter can be viewed as a uniform increase in all eigenvalues.
We first observe that
M − 1 Φ ⊤ Φ = I n in the discrete case.
Letting T ( D ) ( Λ ; δ ) denote the learning transfer matrix with eigenvalue matrix Λ and ridge parameter δ , it follows from Equation 51 and this fact that
[Equation]
In the continuous case, M − 1 Φ ⊤ Φ → I n as M → ∞ (since the columns of Φ are uncorrelated), and since M is very large in the continuous setting, we are again free to use Equation 52.
As for the noise, we simply set ϵ 2 = 0 for now and, once we have our final equations, add power ϵto a hypothetical zero-eigenvalue mode.
H. A SSUMPTION : THE UNIVERSALITY OF Φ
We wish to take averages over Φ in finding the statistics of T ( D ) .
The distribution of Φ is in fact highly structured (reflecting the eigenstructure of the kernel), and we know only that E
[ Φ ij Φ ij ′ ] = δ jj ′ .
We neglect this structure, making the “universality” assumption that we may take Φ to be sampled from a simple Gaussian measure without substantially changing the statistics of T ( D ) .
We henceforth assume Φ ij iid ∼ N (0 , 1) .
This universality assumption is also made in comparable works (implicitly by Bordelon et al.
Canatar et al.
(2021) and explicitly by Jacot et al.
See Appendix A for further references to relevant literature.
The validity of this approximation is ultimately justified by the close match of our theory with experiment.
H. V ANISHING O FF -D IAGONALS OF E (cid:2) T ( D ) (cid:3)
We next observe that
[Equation]
(53) where U is any orthogonal M × M matrix.
Defining U ( m ) as the matrix such that U ( m ) ab ≡ δ ab (1 − 2 δ am ) , noting that U ( m ) ΛU ( m ) = Λ , and plugging U ( m ) in as U in Equation 53, we find that
[Equation]
By choosing m = a , we conclude that E (cid:104) T ( D ) ab (cid:105)
=
if a ̸ = b .
H.
F IXING THE
F ORM OF E (cid:104) T ( D ) ii (cid:105) H. . I SOLATING THE DESIRED ELEMENT
We now isolate a particular diagonal element of the mean learning transfer matrix.
To do so, we write E (cid:104) T ( D ) ii (cid:105) in terms of λ
i (the i th eigenvalue), Λ ( i ) ( Λ with its i th row and column removed), ϕ ⊤ i (the i th row of Φ ), and Φ ( i ) ( Φ with its i th row removed).
Using the Sherman-Morrison matrix inversion formula, we find that
[Equation]
Inserting this into the expectation of T ( D ) , we find that
[Equation]
where κ ( Φ ( i ) , ϕ i ) ≡ κ ( Φ )
i
[Equation]
is a nonnegative scalar
[Equation]
H. C ONCENTRATION AND MODE - INDEPENDENCE OF κ ( Φ ) i
Important quantities in statistical mechanics systems are typically self-averaging (i.e. concentrating about their expectation) in the thermodynamic limit. Self-averaging quantities are the focus of random matrix theory [Equation] and generalization metrics in machine learning also tend to be self-averaging under most circumstances (e
[Equation] g. resampling the data and rerunning a training procedure will typically yield a similar generalization error). Here we argue that κ ( Φ ) i is self-averaging in the thermodynamic limit and can be replaced by its expectation. This could be shown rigorously by means of random matrix theory.
Here we opt to simply observe that, if κ ( Φ ) i were not self-averaging, then for modes i such that
λ i ∼ κ ( Φ ) i , T ( D ) ii and thus L ( D ) ( ϕ i ) would not be self-averaging either.
However, because L ( D ) ( ϕ i ) is a generalization metric like MSE, we should in general expect that it is self-averaging at large n .
Our experimental results (Figure 2) confirm that fluctuations in L ( D ) ( ϕ i ) are indeed small in practice, especially at large n .
We thus replace κ ( Φ ) i with its expectation κ i ≡ E Φ (cid:104) κ ( Φ ) i (cid:105) .
We next argue that κ i is approximately independent of i , so we can replace it with a modeindependent constant κ .
This, too, could be argued rigorously by means of random matrix theory. We opt instead for an eigenmode-removal argument inspired by the cavity method of statistical physics (Del Ferraro et al., 2014). Observe that, in the thermodynamic limit, the addition or removal of a single eigenmode should have a negligible effect on any observable and thus on κ i .
We shall here show that, by inserting one eigenmode and removing another, we can transform κ i into κ j for any i and j , implying that κ i ≈ κ j . Assume that the addition or removal of a single eigenmode negligibly affects κ i .
Concretely, assume that κ i
≈ κ
+ i , where κ + i is κ i computed with the addition of one extra eigenmode of arbitrary eigenvalue.
We choose the additional eigenmode to have
eigenvalue
λ i , and we insert it at index i , effectively reinserting the missing mode i into Φ ( i ) and Λ ( i ) .
To clarify the random variables in play, we shall adopt a more explicit notation, writing out Φ ( i ) in terms of its row vectors as Φ ( i ) =
[ ϕ 1 , ..., ϕ
i − 1 , ϕ i +1 , ..., ϕ M ] ⊤ .
Using this notation, we find upon adding the new eigenmode that
[Equation]
where Λ is the original eigenvalue matrix and ˜ ϕ
⊤ i is the design matrix row corresponding to the new mode.
We can also perform the same manipulation with κ j ( j ̸ = i ), this time adding an additional eigenvalue λ j at index j , yielding that
[Equation]
[ ϕ 1 , ..., ϕ j − 1 , ϕ j +1 , ..., ϕ M ] ⊤ (cid:1) − 1 ϕ j (cid:105) −
[Equation]
61) ≈ κ + j ≡ E { ϕ k } Mk =1 , ˜ ϕ
j [Equation] [Equation] We now compare Equations 59 and 62.
Each is an expectation over M +1 vectors from the isotropic measure .
T he statistics of these M + 1 vectors are symmetric under exchange, so we are free to relabel them.
Equation 59 is identical to Equation 62 upon relabeling ϕ
i → ϕ j , ˜ ϕ
i → ϕ i , and ϕ j → ˜ ϕ j , so they are equivalent, and κ + i = κ + j .
This in turn implies that κ i ≈ κ j .
In light of this, we now replace all κ i with a mode-independent (but as-yet-unknown) constant κ and conclude that
[Equation]
In summary, we have argued that κ i is not significantly changed by the addition or removal of single eigenmodes, and two such changes permit us to transform κ i into κ j , so they are therefore approximately equal.
Our argument here is similar to the cavity method of statistical physics (Del Ferraro et al., 2014), which essentially compares the behavior of a weakly-interacting system with and without a single element.
The cavity method is often used as a simpler and more intuitive alternative to the replica method, a role it reprises here (contrast our approach with the replica approach of Canatar et al.
H. D ETERMINING κ
We can determine the value of κ by observing that, using the ridgeless case of Theorem 3.2,
[Equation]
This is a much more straightforward method of fixing this constant than used in comparable works.
The ability to use the ridgeless version of Theorem 3.2 is the main motivation for setting δ =
at the start of the derivation.
H. D IFFERENTIATING W . R . T . Λ TO O BTAIN THE C OVARIANCE OF T ( D )
Here we obtain expressions for the covariance of T ( D ) and thereby the covariance of ˆv .
Remarkably, we shall need no further approximations beyond those already made in approximating E (cid:2) T ( D ) (cid:3) , which lends credence to our thesis that understanding modewise learnabilities is sufficient for understanding more interesting statistics of ˆ f .
We begin with a calculation that will later be of use: differentiating both sides of the constraint on κ with respect to a particular eigenvalue, we find that
[Equation]
yielding that
[Equation]
where
[Equation]
We now factor T ( D ) into two matrices as
[Equation]
where Z ≡ Φ (cid:0)
[Equation]
Φ
[Equation]
[Equation] Unlike T ( D ) , the matrix Z has the advantage of being symmetric and containing only one factor of Λ .
We will find the second-order statistics of Z , which will trivially give these statistics for T ( D ) .
From Equation 63, we find that the expectation of Z is
[Equation]
We also define a modified Z -matrix Z ( U ) ≡ Φ (cid:0) Φ ⊤ U ⊤ ΛUΦ (cid:1) − 1 Φ ⊤ , where U is an orthogonal M × M matrix.
Because the measure over which Φ is averaged is rotation-invariant, we can equivalently average over ˜ Φ ≡ UΦ with the same measure, giving
[Equation]
It is similarly the case that
[Equation]
Our aim will be to calculate expectations of the form E Φ [ Z ij Z kℓ ] .
With a clever choice of U , a symmetry argument quickly shows that most choices of the four indices make this expression zero.
We define U ( m ) ab ≡ δ ab (1 − 2 δ am ) and observe that, because Λ is diagonal, ( U ( m ) )
⊤ ΛU ( m ) =
Λ and thus Z ( U ( m ) )
=
Z .
Equation 70 then yields that
[Equation]
from which it follows that E Φ [ Z ij Z kℓ ] = 0 if any index is repeated an odd number of times.
In light of the fact that Z ij =
Z ji , there are only three distinct nontrivial cases to consider:
E Φ [ Z ii Z ii ] , 2.
E Φ [ Z ij Z ij ] with i ̸
= j , and 3.
E Φ [ Z ii Z jj ] with i ̸
= j .
Cases 1 and 2.
We now consider differentiating Z with respect to a particular element of the matrix Λ .
This yields
[Equation]
where as before ϕ i is the i -th row of Φ .
This gives us the useful expression that
[Equation]
We now set ℓ = i, k
= j and evaluate this expression using Equation 68, concluding that
[Equation]
and thus
Cov
[Equation]
Z
[Equation]
We did not require that i ̸
= j , and so Equation 74 holds for Case 1 as well as Case 2.
Case 3.
We now aim to calculate E [ Z ii Z jj ] for i ̸
=
j .
We might hope to use Equation 73 in calculating E [ Z ii Z jj ] , but this approach is stymied by the fact that we would need to take a derivative with respect to Λ ij , but we only have an approximation for Z for diagonal Λ .
We can circumvent this by means of Z ( U ) .
From the definition of Z ( U ) , we find that
[Equation]
Differentiating with respect to both U ij and U ji with opposite signs ensures that the derivative is taken within the manifold of orthogonal matrices.
Now, using Equation 69, we find that
[Equation]
Taking the expectation of Equation [Equation] plugging in Equation 74 for E (cid:2)
Z 2 ij (cid:3) , comparing to
[Equation]
and performing some algebra, we conclude that
[Equation]
and thus that Z ii , Z jj are anticorrelated with covariance
[Equation]
Cases 1-3 can be summarized as
[Equation]
Using the fact that T ( D ) ij = λ i Z ij , defining L i ≡ λ i ( λ i + κ ) − 1 , and noting that q = ( (cid:80) i L i (1 − L i )) − 1 , we find that
[Equation]
Noting that E ( f ) =
E (cid:2) | v − ˆv | 2 (cid:3) , recalling that ˆv =
T ( D ) v , and using Equation 81 to evaluate a sum over eigenmodes
, we find that expected MSE is given by
[Equation]
Taking a sum over indices of v , we find that the covariance of the predicted function can be written simply in terms of MSE as
[Equation]
H.
A DDING BACK THE RIDGE AND NOISE
We have thus far assumed δ = 0 .
We can now add the ridge parameter back using Equation 52.
To add a ridge parameter δ , we need merely replace λ i → λ i + δM and then change T ( D ) ij → λ i ( λ i + δM ) − 1 T ( D ) ij .
This yields that
[Equation]
where κ ≥ 0 satisfies (cid:88)
[Equation]
Taking either δ = 0 or M → ∞ , we find that
[Equation]
and the mean and covariance of T ( D ) are again given by Equations 63 and 81.
To summarize this simplification: in the continuous setting ( M → ∞ ), we recover the results of prior work.
In the discrete setting with zero ridge, we find that these expressions apply unmodified.
In the discrete setting with positive ridge, we find that these expressions contain corrections with perturbative parameter δM .
In the main text, we report the expressions with δM = 0 , and our experiments obey this condition.
Modifying v so as to place power ϵ 2 into an eigenmode with zero eigenvalue, Equation 82 for MSE becomes
[Equation]
H.
P ROPERTIES OF κ
In experimental settings, κ is in general easy to find numerically, but for theoretical study, we anticipate it being useful to have some analytical bounds on κ in order to, for example, prove that certain eigenmodes are or are not asymptotically learned for particular spectra.
To that end, the following lemma gives some properties of κ .
Lemma H.1.
For κ ≥ 0 solving (cid:80) Mi
=1 λ
i λ i
+ κ + δκ = n , with positive eigenvalues { λ i } Mi =1 ordered from greatest to least, the following properties hold:
(a) κ = ∞ when n = 0 , and κ = 0 when n → M and δ = 0 .
(b) κ is strictly decreasing with n .
(c) κ ≤ 1 n − ℓ (cid:16) δ + (cid:80) Mi = ℓ +1 λ i (cid:17) for all ℓ ∈ { 0 , ..., n − 1 } .
(d) κ ≥ λ ℓ (cid:0) ℓn − 1 (cid:1) for all ℓ ∈ { n, ..., M } .
Proof of property (a) : Because (cid:80) Mi
=1 λ
i λ i
+ κ + δκ is strictly decreasing with κ for κ ≥ 0 , there can only be one solution for a given n .
The first statement follows by inspection, and the second follows by inspection and our assumption that all eigenvalues are strictly positive.
Proof of property (b) : Differentiating the constraint on κ with respect to n yields (cid:34) M (cid:88) i =
[Equation]
i
[Equation]
1 , which implies that
[Equation]
Proof of property (c) : We observe that n = (cid:80) Mi
=1 λ
i λ i + κ + δκ ≤ ℓ + (cid:80) Mi
=
ℓ +1 λ
i κ + δκ .
The desired property follows.
Proof of property (d) : We set δ = 0 and consider replacing λ
i with λ ℓ if i ≤ ℓ and 0 if i > ℓ .
Noting that this does not increase any term in the sum, we find that n = (cid:80) Mi
=1 λ
i λ i + κ ≥ (cid:80) ℓi =1 λ ℓ λ ℓ + κ = ℓλ ℓ λ ℓ + κ .
The desired property in the ridgeless case follows.
A positive ridge parameter only increases κ , so the property holds in general.
We note that a positive ridge parameter can be incorporated into the bound, giving
[Equation]
We also note that, as observed by Jacot et al.
(2020) and Spigler et al.
(2020), the asymptotic scaling of κ can be fixed if the kernel eigenvalues follow a power law spectrum.
Specifically, if λ i ∼ i − α for some α > 1 , then Jacot et al.
(2020) 15 show that
[Equation]
[Equation]