N EURAL T ANGENT K ERNEL E IGENVALUES A CCURATELY P REDICT G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent work has shown that, in many cases, inﬁnitely-wide deep neural networks are equivalent to kernel regression using the network’s “neural tangent kernel.”
With the aim of shedding light on the learning behavior of wide neural networks, in this work we derive a new theory predicting the generalization of kernel regression.
Our theory accurately predicts not only test mean-squared-error but all ﬁrstand second-order statistics of the network’s learned function.
Furthermore, using a measure quantifying the “learnability” of a given target function, we prove a new “no-free-lunch” theorem characterizing a fundamental tradeoff in the inductive bias of wide neural networks: improving a network’s generalization for a given target function must worsen its generalization for orthogonal functions.
We further demonstrate the utility of our theory by analytically predicting two surprising phenomena — worse-than-chance generalization on hard-to-learn functions and nonmonotonic error curves in the small data regime — which we subsequently observe in experiments.
Though our theory is derived for networks in the inﬁnitewidth limit, we experimentally ﬁnd that its predictions are accurate for wide ﬁnite networks.
I NTRODUCTION
Deep neural networks have proven extraordinarily useful for a wide array of learning problems, but theoretical understanding of their high performance is notably lacking.
One longstanding mystery is the fact that the functions learned by neural networks typically generalize quite well to new data.
In light of neural networks’ extreme overparameterization and high expressivity (Zhang et al., 2017), conventional statistical intuition incorrectly predicts poor performance (Belkin et al., 2019; James et al., 2013), leading to a lack of useful generalization bounds and a need for new concepts to augment classical wisdom.
As is common when facing great mysteries, the ﬁrst challenge lies in framing a precise question.
To clarify our aims, we pose the following problem:
Can one efﬁciently predict, from ﬁrst principles, how well a given network architecture will generalize when learning a given function, provided a given number of training examples?
In this work, we provide an afﬁrmative answer for neural networks in the inﬁnite-width limit.
To do so, we build on two recent developments in deep learning theory.
The ﬁrst is the theory of wide networks, which has shown that, as hidden-layer widths tend to inﬁnity, commonly-used neural networks take remarkably simple analytical forms (Daniely et al., 2016; Lee et al., 2018; Novak et al., 2019a; Arora et al., 2019).
In particular, a wide network trained by gradient descent with mean-squared-error (MSE) loss is equivalent to a classical model called kernel regression , where the kernel is the network’s so-called “neural tangent kernel” (NTK)
(Jacot et al., 2018; Lee et al., 2019).
This line of work suggests that, by studying these simple inﬁnite-width expressions, we might gain insight into the behavior of real, ﬁnite networks.
The second development takes this approach, studying the generalization performance of kernel regression and showing agreement with ﬁnite networks.
In a pioneering study, Bordelon et al.
(2020) derived an approximation for the generalization MSE of kernel regression and showed that, using the network NTK as the kernel, their expressions accurately predict the MSE of neural networks learning arbitrary functions.
Their results reveal a simple picture of neural network generalization: as samples are added to the training set, the network generalizes well on a larger and larger subspace of input functions.
The natural basis for this subspace of learnable functions is the eigenbasis of the NTK, and its eigenfunctions are learned in descending order of their eigenvalues.
We note a substantial body of related work studying the inductive bias of neural networks and kernel regression (Valle-Perez et al., 2018;
Yang & Salman, 2019; Bietti & Mairal, 2019; Spigler et al., 2020; OrtizJim´enez
et al., 2021;
Cohen et al., 2021; Canatar et al., 2021).
In this paper, we extend these results signiﬁcantly, deriving a new predictive theory for the generalization of kernel regression and empirically conﬁrming its agreement with wide neural networks.
We begin in Section 2.2 by formulating a ﬁgure of merit we call the “learnability” of a target function, a measure which we prove obeys several desirable properties that MSE does not.
In Section 2.4, we use learnability to prove a strong new “no-free-lunch” theorem describing a fundamental tradeoff in the inductive bias of a kernel towards all functions in an orthogonal basis.
This shows that not only are higher NTK eigenmodes easier to learn, but these eigenmodes are in a zero-sum competition with one another for the ability to be learned at a given training set size.
We further prove that, for any kernel, this tradeoff necessarily leaves some functions with worse-than-chance generalization.
In Sections 2.5 and 2.6, we derive expressions for not just expected test MSE 1 but for all ﬁrst- and second-order statistics of the learned function.
Our expression for learnability 2 is extremely simple, involving just one dataset-size-dependent parameter C that acts as an eigenvalue threshold above which
eigenmodes are well-learned.
From the form of this expression, we conclude that for any supervised neural network learning problem, eigenmodes’ eigenvalues and learnabilities lie on the same universal curve.
Furthermore, in Section 2.7 we expand MSE for small dataset size and make the counterintuitive prediction that, for many target functions, MSE increases with training set size in this small-data regime.
In Section 3, we experimentally verify all our conclusions on three input spaces for both exact NTK regression and wide ﬁnite networks.
We ﬁnd that our analytical expressions give excellent agreement with the true generalization behavior of deep ReLU networks in the NTK regime.
We verify our nofree-lunch result with ﬁnite networks, and we experimentally observe our theoretical predictions of both worse-than-chance generalization and nonmonotonic MSE.
Finally, we compare our theory to ﬁnite nets with depth 4 and widths as low as 20 and ﬁnd that our theory holds with remarkable accuracy even at these narrow widths, suggesting that it is not merely applicable in the canonical
NTK regime but in fact approximately predicts generalization performance in more practical neural networks.
T HEORY 2. A REVIEW OF KERNEL REGRESSION
Consider the task of learning an m -element function f : X → R m given a set of n unique training points
D = { x i } ni
=1 ⊆ X and their corresponding function values f ( D ) ∈ R n × m .
To simplify our analysis, we will let the domain X be discrete with size M ≡ |X| and assume the n training points are uniformly sampled from X .
This choice of discrete domain will permit us to use matrices and vectors instead of operators and functions in our derivations.
By taking M → ∞ , our results easily extend to problems with continuous domain: for example, if M → ∞ and we allow the points in X to approach a density equal to some desired measure over R d , we recover the standard setting where the data are sampled nonuniformly from R d .
We will use ˆ f to denote the function learned by a neural network trained on this dataset.
Remarkably, for an inﬁnite-width neural network optimized via gradient descent to zero training MSE loss, this learned function is given by
[Equation]
where K : X × X → R is the network’s “neural tangent kernel” (NTK) (Jacot et al., 2018; Lee et al., 2019), K ( D , D ) is the “kernel matrix” deﬁned by K ( D , D ) ij =
K ( x i , x j ) , and K ( x, D ) is a row vector with components K ( x, D ) i
= K ( x, x i ) .
We give a brief introduction to the NTK in
Appendix C. Due to its similarity to the normal equation of linear regression, Equation 1 is often called “kernel regression.”
Equation 1 holds exactly in the inﬁnite-width limit of fully-connected networks (Lee et al., 2019), convolutional networks (Arora et al., 2019), transformers (Hron et al., 2020), and more (Yang, 2019).
Moreover, several empirical studies have shown it to be a good approximation for networks of even modest width (Lee et al., 2019; 2020).
Our approach will be to study the generalization behavior of Equation 1, conjecture that our results also apply to ﬁnite networks, and ﬁnally provide strong support for our conjecture with experiments.
Examining Equation 1, one ﬁnds that the m indices of f can each be treated separately: the learned ˆ f is equivalent to simply performing kernel regression with each of the m indices as a scalar target function and then vectorizing the results. For simplicity, then, we hereafter assume m = 1 . The extension to m > 1 is straightforward.
2. F IGURES OF MERIT OF ˆ f
We will study three measures of the quality of the learned function ˆ f .
All three will be deﬁned in terms of the inner product over X : for two functions g, h : X → R , their inner product is (cid:104)
[Equation]
The ﬁrst measure of quality is mean-squared error (MSE).
For a particular dataset D , MSE is given by E ( D ) ( f ) ≡ (cid:104)
f − ˆ f, f − ˆ f (cid:105) .
Of more interest will be the expected MSE over all datasets of size n , given by E ( f ) ≡ E D (cid:2) E ( D ) ( f ) (cid:3) .
We note that the inner product is taken over all X , including D , even though ˆ f ( x ) = f
( x ) for x ∈ D for kernel regression.
In maximizing the similarity of ˆ f to f , we typically wish to minimize its similarity to all functions orthogonal to f .
The second measure examines the coefﬁcient in ˆ f of one such orthogonal function to f .
Letting g : X → R be a function such that (cid:104) f, g (cid:105)
=
0, we consider the mean and variance of the quantity (cid:104)
ˆ f, g (cid:105) .
We will derive accurate predictions for this metric of generalization.
Lastly, we introduce a ﬁgure of merit quantifying the alignment of f and ˆ f , which we call “learnability.”
It is given by
[Equation]
where L ( D ) ( f ) is the dataset-dependent learnability of the function f (“ D -learnability”) and L ( f ) is its expectation over random data (“learnability”).
Though at ﬁrst glance these two seem like odd ﬁgures of merit, we will soon show that they have many desirable properties when ˆ f is given by Equation 1: unlike MSE, both are bounded in [0 , 1] (Lemma 1d), always change monotonically as new data points are added (Lemma 1e), are invariant to rescalings of f , and obey a simple conservation law (Theorem 1).
Furthermore, expanding the inner product in the deﬁnition of E and noting that E ( f ) ≥ (cid:104) f, f (cid:105) (1 − L ( f )) 2 , one can see that low MSE is impossible without high learnability.
We will ultimately derive an accurate approximation for learnability that is substantially simpler than any known approximation for MSE.
2. T HE KERNEL EIGENSYSTEM
By deﬁnition, any kernel function is symmetric and positive-semideﬁnite (Shawe-Taylor et al., 2004).
This implies that we can ﬁnd a set of orthonormal eigenfunctions { φ i } Mi and nonnega- tive eigenvalues { λ i } Mi =1 that satisfy
[Equation]
For simplicity (and to ensure that K ( D , D ) is invertible), we will assume that K is in fact positive deﬁnite and λ i > 0 , an assumption that will hold in most cases of interest.
We will now translate Equation 1 to this eigenbasis.
First we decompose f and ˆ f into weighted sums of the eigenfunctions as
[Equation]
where v and ˆv are vectors of coefﬁcients.
Using this notation, MSE is E ( D ) =
( v − ˆv ) 2 and
D -learnability is L ( D ) = v T ˆv / | v | 2 .
Noting that K ( x 1 , x 2 ) =
(cid:80) Mi
=1 λ
i φ i ( x 1 ) φ i ( x 2 ) , we can decompose the kernel matrix as
K ( D , D ) =
Φ T ( D ) ΛΦ ( D ) , where Φ ( D ) ij =
φ i ( x j ) is the M × n “design matrix” and Λ =
diag
( λ 1 , ..., λ M ) is a diagonal matrix of eigenvalues.
The learned coefﬁcients ˆv are then given by ˆ v i = (cid:104) φ
i , ˆ f (cid:105) =
λ i φ i ( D ) K
( D , D ) − 1 Φ T ( D ) v .
Stacking these coefﬁcients into a matrix equation, we ﬁnd that
[Equation]
where T ( D ) ≡ ΛΦ ( D ) (cid:0) Φ T ( D ) ΛΦ ( D ) (cid:1) − 1 Φ T ( D ) is an M × M matrix, independent of f , that fully describes the model’s learning behavior on a training set D .
We call this fundamental quantity the “learning transfer matrix.
If we can determine the statistical properties of this matrix, we will understand the learning behavior of our model.
We also deﬁne the mean learning transfer matrix T ≡ E D (cid:2) T ( D ) (cid:3) .
D -learnability and learnability are then respectively given by L ( D ) ( f ) = v T T ( D ) v / | v | 2 and L ( f ) = v T Tv / | v | 2 .
2. E XACT RESULTS
The following lemma gives basic properties of the quantities deﬁned above.
Lemma 1.
The following properties of T ( D ) , T , L ( D ) , and L hold:
(a) L ( D ) ( φ i ) =
T ( D ) ii , and L ( φ i )
= T ii .
(b) When n = 0 , T = T ( D ) = 0 and L ( f )
=
L ( D ) ( f ) = 0 .
(c) When n = M , T = T ( D ) =
I M and L ( f )
=
L ( D ) ( f ) = 1 .
(d) All eigenvalues of T ( D ) are in { 0 , 1 } , all eigenvalues of T are in [0 , 1] , and so L ( f ) , L ( D ) ( f ) ∈ [0 , 1] .
(e) Let D + be D ∪ x , where x ∈ X, x / ∈ D is a new data point.
Then L ( D + ) ( f ) ≥ L ( D ) ( f ) .
(f) For any i ∈ { 1 , ..., M } , ddλ i T ( D ) ii ≥ 0 , hence ddλ i L ( D ) ( φ i ) ≥ 0 .
(g)
For any i, j ∈ {
1 , ..., M } , i (cid:54) =
j , ddλ i T ( D ) jj ≤ 0 , hence ddλ i L ( D ) ( φ j ) ≤ 0 .
Property (a) in Lemma 1 formalizes the relationship between the transfer matrix and learnability.
Properties (b-e) together give an intuitive picture of the learning process: the learning transfer matrix monotonically interpolates between zero and I M as the training set grows — adding data never harms the learnability of any function.
Properties (f-g) show that the kernel eigenmodes are in competition: increasing one eigenvalue can only improve the learnability of the corresponding eigenfunction, but can only decrease the learnabilities of all others.
We prove Lemma 1 in Appendix
D.
We now present our ﬁrst major result.
Theorem 1 (“No-free-lunch” theorem for kernel regression) .
For any complete basis of orthogonal functions F ,
[Equation]
The proof, which hinges on the intermediate result that Tr (cid:2) T ( D ) (cid:3) = n , is given in Appendix E.
We note that this result is stronger than the ordinary “no-free-lunch” theorem for learning algorithms, which requires averaging over all target functions instead of merely an orthogonal basis (Wolpert, 1996).
To understand the signiﬁcance of this result, consider that one might naively hope to design a neural kernel that achieves generally high performance for all target functions f .
Theoremstates that this is impossible: averaged over a complete basis of functions, all kernels achieve the same learnability .
This exact result implies that, because there exist no universally high-performing kernels, we must instead aim to choose a kernel whose high-eigenvalue modes align well with the function to be learned .
To our knowledge, this is the ﬁrst exact result quantifying such a tradeoff in kernel regression or deep learning.
Illustrating a consequence of this tradeoff, the following theorem states that, for any kernel, there will always be functions poorly-aligned with the kernel’s inductive bias on which the model generalizes as bad or worse than it would by simply predicting zero on all unseen test points.
Theorem 2 (Negative generalization) .
There is always at least one eigenfunction φ i for which
L ( φ i ) ≤
L naive ( φ i ) and E ( φ i ) ≥
E naive ( φ i ) , where L naive and E naive are the learnability and mean MSE given by a naive, non-generalizing model with predictions given by
[Equation]
This theorem follows from Theorem 1.
We give a full proof in Appendix E.
D ERIVING A CLOSED - FORM EXPRESSION FOR T
We will now derive an approximation for T and the second moments of T ( D ) , ultimately yielding simple yet accurate expressions for L , E , and (cid:104) ˆ f, g (cid:105) .
We sketch our method and state our results here and provide a full derivation in Appendix F.
We begin by noting that the expectation in T ≡ E D (cid:2) T ( D ) (cid:3) is essentially an expectation over a combinatorially large set of possible design matrices Φ ( D ) , each of which has orthogonal columns.
We replace this with an average over all M × n matrices Φ satisfying Φ T Φ =
M 1 n , which we write
[Equation]
We can then readily prove by symmetry that the off-diagonal elements of T vanish, leaving the question of how each diagonal element depends on the set of eigenvalues.
To probe this, we consider adding an ( M + 1) -th
“test eigenmode” to the problem, augmenting the principal quantities as
[Equation] [Equation]
Using the Sherman-Morrison formula, we ﬁnd that the new mode’s learnability is given by
[Equation]
where C ( Φ , φ ) ≡ (cid:104) φ T (cid:0) Φ T ΛΦ (cid:1) − 1 φ (cid:105) − 1 is a nonnegative dataset-dependent constant.
For large n and realistic eigenspectra, C ( Φ , φ ) is distributed tightly around its mean, and so we can replace it with a constant, which we call C .
We emphasize that C is the same for every eigenmode.
Using
Theorem 1 to obtain a constraint on C , we reach our ultimate approximation that
[Equation]
where C ≥ 0 satisﬁes
[Equation]
From Lemma 1, we immediately obtain the following major result:
Lemma 2.
Under the above approximations, the learnability of a function f is given by
[Equation]
with C given by Equation 11.
A function is thus more learnable the more weight it places in high eigenvalue modes.
We show in
Section 3 that Equation 12 is in excellent agreement with experiments using both kernel regression and trained ﬁnite networks.
The learnability of each eigenmode depends critically on the value of C .
We now present several properties characterizing how C depends on n .
Lemma 3.
For C satisfying the constraint in Equation 11, with { λ i } Mi =1 ordered from greatest to least, the following properties hold:
(a) C = ∞ when n = 0 , and C = 0 when n = M .
(b) C is strictly decreasing with n .
(c) C ≤ 1 n − (cid:96) (cid:80) Mi
= (cid:96) +
λ
i for all (cid:96) ∈ { 0 , ..., n − 1 } .
(d) C ≥ λ (cid:96) (cid:0) (cid:96)n − 1 (cid:1) for all (cid:96) ∈ { n, ..., M }
Properties (a-b), paired with Equation 11, paint a surprisingly simple picture of the learning process: as the training set grows, C gradually decreases, and eigenmodes are learned high-to-low as C passes each eigenvalue.
Properties (c-d) provide bounds on C , which, in addition to then yielding bounds on learnability, can be used to furnish an initial guess when numerically solving for C .
2. S ECOND - ORDER STATISTICS OF T
We now have an expression for the mean of T ( D ) , but many quantities of interest, including MSE, depend on second-order ﬂuctuations about that mean.
In Appendix G, we show how, by taking a derivative with respect to Λ , we can obtain expressions for the second-order statistics of T ( D ) and, as a result, for MSE.
Our main second-order result is that
[Equation]
Using Equation 13, we can study the admixtures of particular spurious modes in the learned function ˆ f .
For example, if f = φ i and i (cid:54) =
j , then we ﬁnd that
[Equation] [Equation]
We note that E (cid:104) (cid:104) ˆ f, φ j (cid:105) 2 (cid:105) increases with λ j , reinforcing our broad conclusion that the model is biased towards high-eigenvalue modes.
Finally, by noting that E ( f ) =
E D (cid:2) | ( T ( D ) − I M ) v | 2 (cid:3) , we can recover the expression of Bordelon et al.
(2020) for MSE:
[Equation]
Our experiments corroborate existing evidence that this is an excellent approximation for MSE.
2. N ONMONOTONIC MSE CURVES
Expanding E for small n , we ﬁnd that
[Equation]
The second equation implies that d E ii dn (cid:12)(cid:12) n =0 > 0 for all modes i such that
λ
i < ( (cid:80) j λ 2 j ) / (2 (cid:80) j λ j ) , which suggests that, for such low-eigenvalue modes, MSE increases as the training set size grows from zero.
In practice, such low modes are commonplace; in fact, for a bounded kernel on a continuous input space (for which M → ∞ ), arbitrarily low modes are inevitable due to the constraint that (cid:80)
i λ i is bounded 7 .
We therefore expect that there quite often exist functions for which a given neural network yields an MSE nonmonotonic with n .
Figure A1 shows that experiments conﬁrm this surprising ﬁrst-principles prediction.
We emphasize that this is a different, more generic phenomenon than that noted by Canatar et al.
(2021), which required that f was either noisy or placed weight in unlearnable zero-eigenvalue modes.
E XPERIMENTS
Here we describe experiments conﬁrming our theoretical predictions for exact NTK regression and wide neural networks.
Unless otherwise stated, all experiments used a fully-connected (FC) fourhidden-layer (4L) ReLU architecture with width 500.
Because this model is FC, it has a rotationinvariant NTK (Lee et al., 2019).
These experiments used three distinct input spaces X .
For each, the eigenmodes of X can be grouped into degenerate subsets indexed by k ∈ N , with higher k corresponding to faster variation in space.
In all cases, we ﬁnd that as k increases, eigenvalues decrease, in concordance with the widespread belief that neural nets have a “spectral bias” towards slowly-varying functions (Rahaman et al., 2019; Canatar et al., 2021; Cao et al., 2019).
We now describe these three input spaces.
For full experimental details, see Appendix H.
Discretized Unit Circle.
The simplest input space we consider is the discretization of the unit circle into M points, X
= { (cos(2 πj/M ) , sin(2 πj/M ) } Mj =1 .
The eigenfunctions on this domain are φ 0 ( θ ) = 1 , φ k ( θ ) = √ 2 cos( kθ ) , and φ (cid:48) k
( θ ) = √ 2 sin( kθ ) , for k ≥ 1 .
Hypercube.
The next input space we consider is the set of verticies of the d -dimensional hypercube,
X = {− 1 , 1 } d , giving M = 2 d .
The eigenfunctions on this domain are the subset-parity functions
φ s ( x ) =
( − 1) s T x , where s ∈ { 0 , 1 } d is a vector indicating the elements of x to which the output is sensitive (Yang & Salman, 2019).
Here we deﬁne k = (cid:80)
i s i .
Hypersphere.
To illustrate that our results extend easily to continuous input spaces, we consider the d -sphere S d = { x ∈ R d +1 | x 2 = d + 1 } .
The eigenfunctions on this domain are the hyperspherical harmonics (see, e.g., Frye & Efthimiou (2012);
Bordelon et al.
which group into degenerate sets indexed by k ∈ N .
The corresponding eigenvalues decrease exponentially with k , and so when summing over all eigenmodes to compute C and q , we simply truncate the sum at k max = 70 .
Figure I5 shows the 4L ReLU NTK eigenvalues on each domain, all of which decrease almost monotonically with increasing k .
In particular, the eigenvalues on the unit circle roughly follow a power-law decay like the NTK spectra of common image datasets (e.g. Lee et al.
We now describe our experiments.
Predicting learnability L ( D ) .
We use both wide ﬁnite nets and exact NTK regression to learn several eigenmodes on all three domains and compare true learnabilities with our theoretical predictions.
Our theory predicts true generalization behavior quite well (Figure 1).
We further show
[Figure]
Unit
[Figure]
k =0 k
=2 k = k = 22222 8 n
B 8d
Hypercube k =
k
=1 k = k = 22222 8 n
C 7-sphere k =
k
=1 k =
k =3 k
=4 k
=
[Figure]
that the k = 8 mode on the 8 d hypercube has worse-than-chance generalization, an inevitable result of its low eigenvalue and Theorem 2.
We note that ﬁnite networks and NTK regression give very similar results, supporting the validity of approximating networks with the NTK.
Universal form for learnability.
We next ﬁx n and plot learnability vs. eigenvalue for eight modes over each domain.
In each case we ﬁnd L ( φ ) ≈ λ/ ( λ + C ) , and by rescaling the eigenvalues (a symmetry which leaves Equation 1 invariant), we see that for any neural network learning problem with any training set size n , eigenmode learnability always lies on one universal curve (Figure 2).
[Figure] [Figure]
l n
[Figure]
A n =
[Figure]
B n = No free lunch for neural networks. We then experimentally conﬁrm that our no-free-lunch result applies to ﬁnite networks as well as kernel regression (Figure 3). We use both models to learn a function on the discretized unit circle with M = 10 and sum the resulting D -learnabilities of each eigenmode.
Unlike in our other experiments, we only sample D once for each n ; even without the beneﬁt of averaging over datasets, we ﬁnd that total D -learnability is always conserved.
Nonmonotonic MSE curves.
We next conﬁrm our ﬁrst-principles prediction of nonmonotonic MSE curves at small n .
We plot curves for four eigenmodes on each domain, three of which Equationpredicts will have positive d E /dn | n =0 .
MSE is indeed increasing as predicted (Figure A1).
Agreement with narrow networks.
Finally, we repeat the experiment of Figure 1B for varying network width.
We ﬁnd that our theory gives accurate predictions of learnability and MSE even for networks of depth 4 and width as small as 20 (Figures B2 and B3).
This surprising agreement suggests our theory is promising starting point for understanding the generalization of practical neural networks.
C ONCLUSION
We have presented a ﬁrst-principles theory of generalization for kernel regression that predicts a variety of measures of generalization performance.
We then empirically demonstrated its predictive accuracy for both exact NTK regression and wide ﬁnite networks in the NTK regime.
This theory offers new insight into these models’ inductive bias and provides a general framework for understanding their learning behavior, opening the door to the principled study of many other deep learning mysteries.
A N ONMONOTONIC MSE CURVES AT SMALL TRAINING SET SIZE
[Figure]
B E XPERIMENTAL RESULTS FOR NARROW NETWORKS
[Figure]
n
[Figure]
( ) ( k )
A width A width A width A width k =
k
=
k =3 k
=
B width 500 B width 500 B width 500 B width
C width 200 C width 200 C width 200 C width22222n 0.
( ) ( k )
D width 100 D width 100 D width 100 D width22222n
E width 50 E width 50 E width 50 E width22222n
F width 20 F width 20 F width 20 F width
C R EVIEW OF THE NTK
In the main text, we assume prior familiarity with the NTK, using Equation 1 as the starting point of our derivations.
Here we provide a deﬁnition and very brief introduction to the NTK for unfamiliar readers.
For derivations and full discussions, see Jacot et al.
(2018) and Lee et al.
Consider a feedforward neural network representing a function ˆ f θ : X → R , where θ is a parameter vector.
Further consider one training example x with target value y and one test point x (cid:48) and suppose we perform one step of gradient descent with a small learning rate η with respect to the MSE loss (cid:96) θ ≡ ( ˆ f θ ( x ) − y ) 2 .
This gives the parameter update
[Equation]
We now wish to know how this parameter update changes ˆ f θ ( x (cid:48) ) .
To do so, we linearize about θ , ﬁnding that
[Equation]
where we have deﬁned K ( x, x (cid:48) )
≡ ∇ θ ˆ f θ ( x ) · ∇ θ ˆ f θ ( x (cid:48) ) .
This quantity is the NTK.
Remarkably, as network width 8 goes to inﬁnity, the O ( δθ 2 ) corrections become negligible, and K ( x, x (cid:48) ) is the same after any random initialization 9 and at any time during training.
This dramatically simpliﬁes the analysis of network training, allowing one to prove that after inﬁnite time training on MSE loss for an arbitrary dataset, the network’s learned function is given by Equation 1.
See, for example,
Equations 14-16 of Lee et al.
D P ROOF OF L EMMA
Property (a) : L ( D ) ( φ i ) =
T ( D ) ii , and L ( φ i )
= T ii .
Proof.
Using the fact that (cid:104) φ i , φ i (cid:105) = 1 , we see that L ( D ) ( φ i ) =
(cid:104)
ˆ φ i , φ i (cid:105) =
e Ti T ( D ) e i
= T ( D ) ii , where e i is a one-hot M -vector with the one at index i .
The second clause of the property follows by averaging.
Property (b) : When n = 0 , T = T ( D ) = 0 and L ( f )
=
L ( D ) ( f ) = 0 .
Proof.
When n = 0 , Φ ( D ) has no columns, and thus T ( D ) = 0 .
The other clauses follow from
Property (a) and averaging.
Property (c) : When n = M , T = T ( D ) =
I M and L ( f )
=
L ( D ) ( f ) = 1 .
Proof.
When n = M , Φ ( D ) is a full-rank M × M matrix.
Inspection of the formula for T ( D ) then shows that T ( D ) =
I M .
The other clauses follow from Property (a) and averaging.
Property (d) : All eigenvalues of T ( D ) are in { 0 , 1 } , all eigenvalues of T are in [0 , 1] , and so
L ( f ) , L ( D ) ( f ) ∈ [0 , 1] .
Proof.
From the deﬁnition of T ( D ) , it is easy to see that T ( D ) =
(cid:0) T ( D ) (cid:1) 2 .
T ( D ) is thus idempotent, with all eigenvalues in { 0 , 1 } .
The fact that all eigenvalues of T are in [0 , 1] follows by averaging.
The stated properties of L ( f ) , L ( D ) ( f ) follow from the fact that, for any compatible vector z and matrix A , it holds that z T A
z z T z is bounded by the maximum and minimum eigenvalues of A .
Property (e) : Let D + be D∪ x , where x ∈ X, x / ∈ D is a new data point.
Then L ( D + ) ( f ) ≥ L ( D ) ( f ) .
Proof.
To begin, we use the Moore-Penrose pseudoinverse, which we denote by ( · ) + , to cast T ( D ) into a more transparent form:
[Equation]
(19) where we have suppressed the D in Φ ( D ) .
This follows from the property of pseudoinverses that
A
[Equation]
A T A ) + A T =
( AA T )( AA T )
+ for any matrix A .
We now augment our system with one extra data point, getting
[Equation]
(20) where ξ is an M -element column vector orthogonal to the others of Φ .
We now convert the pseudoinverse into an inverse with a limit, getting
[Equation]
We now use the Sherman-Morrison matrix inversion formula to ﬁnd that
[Equation]
Because both ξξ T and the inverted matrix in Equation 22 are positive semideﬁnite, we conclude that, for any M -vector z , it will hold that z T T ( D + )
z ≥ z T T ( D ) z .
The desired property follows.
Property (f) :
For any i ∈ { 1 , ..., M } , ddλ i T ( D ) ii ≥ 0 , hence ddλ i L ( D ) ( φ i ) ≥ 0 .
Proof.
Differentiating T ( D ) jj with respect to a particular λ
i , we ﬁnd that
[Equation]
(23) where φ T i is the i -th row of Φ and K =
Φ T ΛΦ .
Specializing to the case i =
j , we note that φ Ti K − 1 φ i ≥ 0 because K is positive semideﬁnite, and
λ j φ i K − 1 φ Ti ≤ 1 because λ j φ i φ Ti is one of the positive semideﬁnite summands in K =
(cid:80) k
λ k φ k φ
Tk .
The desired property follows.
Property (g) :
For any i, j ∈ { 1 , ..., M } , i (cid:54) =
j , ddλ i T ( D ) jj ≤ 0 , hence ddλ i L ( D ) ( φ j ) ≤ 0 .
Proof.
Differentiating as in the proof of Property (f) and using the fact that i (cid:54)
=
j , we see that
[Equation]
24) which is manifestly nonpositive because λ j > 0 .
The desired property follows.
E P ROOFS OF T HEOREMS 1 AND
Here we prove Theorem 1, our “no-free-lunch” result, and Theorem 2, which states the existence of eigenfunctions that generalize worse than chance.
We begin with the former.
Proof of Theorem 1 .
First, we note that, for any orthogonal basis F on X ,
[Equation] [Equation]
where V is an orthogonal set of vectors spanning R M .
This is equivalent to Tr (cid:2) T ( D ) (cid:3) .
This trace is given by
[Equation]
which proves the desired theorem.
Proof of Theorem 2 .
First, we note that the naive, nongeneralizing model described in the theorem will always have a learnability of L naive ( f ) =
E D (cid:104) n (cid:104) f,f (cid:105) M (cid:104) f,f (cid:105) (cid:105) =
n M .
As per Theorem 1, this is exactly the mean learnability of all eigenfunctions, so either all have precisely this learnability or else one has lower learnability.
This proves the clause of the theorem speciﬁc to learnability.
To prove the clause speciﬁc to MSE, we now note that, as deﬁned by Equation 1, kernel regression will always perfectly memorize the training data, so for both the naive model and kernel regression,
MSE is given by
[Equation]
The ﬁrst term is the same for any model, the second term is nonnegative but zero for the naive model, and the third term is equivalent to − 2( L ( f ) − nM ) , which must be nonnegative for at least one eigenfunction.
Kernel regression therefore gives worse MSE than the naive model unless, as for the naive model, ˆ f ( x ) = 0 for x / ∈ D .
F A PPROXIMATING T
Here we provide details of the derivations of our approximation for T and of Lemma 3.
Whenever possible, we leave it implicit that all expressions for T in this appendix will be approximations and use the symbol for equality for simplicity.
We begin by taking the approximation of Equation 8 that
[Equation]
(28) where the expectation is taken over all M × n matrices Φ satisfying Φ T Φ =
M 1 n with the uniform measure.
It turns out that we can equivalently average over all all M × n matrices Φ with a meanzero i.i.d.
Gaussian measure over each element, without changing T .
To see this, note that this equation is symmetric under right-multiplication of Φ by arbitrary n × n invertible matrices R :
[Equation]
Letting R always be the ( Φ -dependent) matrix that orthogonalizes the columns of Φ (i.e.
Φ T R T RΦ =
M I n ), one can see that, no matter the distribution of Φ in Equation 28, it holds that E Φ (cid:104) ΛΦ (cid:0) Φ T ΛΦ (cid:1) − 1 Φ T (cid:105) =
E Φ (cid:104) ΛΦ
o (cid:0) Φ To ΛΦ o (cid:1) − 1 Φ To (cid:105) , where Φ o is the orthogonalized version of Φ .
The expectation can thus equivalently be taken over any distribution that induces the same distribution over the orthogonalized Φ o ; in particular, the uniform measure on orthogonal Φ can be interchanged with the standard Gaussian measure.
Moving forward, we will exploit this equivalence and assume that each element of Φ is sampled i.i.d.
from N (0 , 1) .
This assumption will largely remain in the background, but will be useful later.
We now evaluate T , starting with the off-diagonal elements.
We observe that
[Equation]
where U is any orthogonal M × M matrix.
Deﬁning U ( m ) as the matrix such that U ( m ) ab ≡ δ ab (1 − 2 δ am ) , noting that U ( m ) ΛU ( m ) = Λ , and plugging U ( m ) in as U in Equation 30, we ﬁnd that
[Equation]
By choosing m = a , we conclude that T ab =
if a (cid:54) = b .
To evaluate the diagonal elements of T , we consider augmenting our eigensystem with a new “test eigenmode” with eigenvalue λ , as described in the text and formalized in Equation 9.
To proceed, we make the core assumption that T ii ≈ T + ii when n, M (cid:29) 1 11 .
This assumption, combined with the symmetry T + ii =
T + jj whenever λ i =
λ
j , implies that T ii
=
T +( M +1)( M +1) | λ
= λ i .
Therefore, to evaluate T , it sufﬁces to evaluate T +( M +1)( M +1) as a function of λ .
We now manipulate the expression for T + to isolate T +( M +1)
( M +1) .
Using the Sherman-Morrison matrix inversion formula, we obtain
[Equation]
and inserting this into the full expression for T + gives
[Equation]
where C ( Φ , φ ) ≡ (cid:104) φ T (cid:0) Φ T ΛΦ (cid:1) − 1 φ (cid:105) − 1 is a nonnegative scalar.
We now argue that, for realistic values of M , n , and { λ i } i , we can simply replace C ( Φ , φ ) by its mean.
We ﬁrst note that, because λ/ ( λ + C ) is convex, we can use Jensen’s inequality to ﬁnd that
[Equation]
where we have deﬁned ¯ C ≡ E Φ , φ (cid:2) C ( Φ , φ ) (cid:3) .
Some algebra also yields the fact that
[Equation]
for C ( Φ , φ ) ≥ 0 , which, after taking an expectation, gives us the upper bound that
[Equation]
Armed with the bounds of Equation 34 and Equation 36, we can replace C ( Φ , φ ) by ¯ C in the last line of Equation 33 if we can show that Var Φ , φ (cid:2) C ( Φ , φ ) (cid:3) (cid:28) ¯ C 2 .
This will be nontrivial to show in general, but we can easily see that it holds in a caricature of a realistic problem.
In practice, we often ﬁnd that the NTK spectrum is dominated by a few large eigenvalues with a long tail of small eigenvalues (Figure I5).
To model this sort of spectrum, let p ∈ N with p (cid:28) n (cid:28) M , and suppose
[Equation]
with λ 0 (cid:29) λ (cid:48) /
( M − p ) .
The kernel matrix is given by
[Equation]
where φ i is the i -th row of Φ and we have made an approximation using the central limit theorem assuming that Φ is random with entries sampled i.i.d.
from N (0 , 1) .
This kernel matrix has p large,
O ( λ 0 ) eigenvalues and n − p remaining eigenvalues of order λ (cid:48) .
Its inverse therefore has p small eigenvalues and n − p similar large ones of order 1 /λ (cid:48) .
Letting { ξ i } ni =1 be these inverted eigenvalues, we have that
[Equation]
where { x i } ni =1 are the components of φ on each of the kernel eigendirections.
For typical { x i } ni = sampled i.i.d.
from N (0 , 1) , the denominator approaches ( n − p ) λ (cid:48) by the central limit theorem, and thus we reach our desired conclusion that C ( Φ , φ )
≈ 1 / (( n − p ) λ (cid:48) ) is approximately deterministic.
More precisely, we ﬁnd
[Equation]
from which we see that
[Equation]
which approaches zero when n (cid:29) p , as desired.
This argument used a somewhat unrealistic kernel eigenspectrum, but numerical experiments show that C ( Φ , φ ) is approximately deterministic even for real eigenspectra (Figure F4).
We leave the analysis of C ( Φ , φ ) for realistic kernels for future work.
[Figure]
Replacing C ( Φ , φ ) by a constant C yields our ﬁnal approximation for T , given in Equation 11.
We note that, though this constant C is roughly the mean of C ( Φ , φ ) , we circumvent deriving an explicit expression for this mean by instead using Theorem 1 to ﬁx C .
F. P ROOF OF L EMMA
Here we provide a proof of the Lemma 3, which states several properties of the constant C deﬁned by Equation 11.
As stated in the Lemma, these properties assume { λ i } Mi =1 ordered from greatest to least.
Property (a) : C = ∞
when n = 0 , and C = 0 when n = M .
Proof.
Because (cid:80) Mi =1 λ
i λ i + C is strictly decreasing with C for C ≥ 0 , there can only be one solution for a given n .
The ﬁrst statement follows by inspection, and the second follows by inspection given our assumption that all eigenvalues are strictly positive.
Property (b) : C is strictly decreasing with n .
Proof .
Differentiating the constraint on C with respect to n yields
[Equation]
dC
[Equation]
(42) giving dC
[Equation]
Property (c) :
C ≤ 1 n − (cid:96) (cid:80) Mi = (cid:96) +
λ
i for all (cid:96) ∈ { 0 , ..., n − 1 } .
Proof.
n = (cid:80) Mi
=1 λ
i λ i + C ≤ (cid:96) + (cid:80) Mi
= (cid:96) +
λ
i C .
The desired property follows.
Property (d) :
C ≥ λ (cid:96) (cid:0) (cid:96)n − 1 (cid:1) for all (cid:96) ∈ { n, ..., M } .
Proof.
We consider replacing λ
i with λ (cid:96) if i ≤ (cid:96) and 0 if i > (cid:96) .
Noting that this does not increase any term in the sum, we ﬁnd that n = (cid:80) Mi
=1 λ
i λ i + C ≥ (cid:80) (cid:96)i =1 λ (cid:96) λ
(cid:96) + C = (cid:96)λ (cid:96) λ
(cid:96) + C .
The desired property follows.
G S ECOND - ORDER STATISTICS OF T ( D ) G. P
RELIMINARIES
Here we derive expressions for the second-order statistics of T ( D ) .
These derivations make no further approximations beyond those already made in approximating T .
We begin a calculation that will later be of use.
Differentiating both sides of the constraint on C in
Equation 11 with respect to a particular eigenvalue, we ﬁnd that
[Equation]
yielding that
[Equation]
≡
[Equation]
We now factor T ( D ) into two matrices as
[Equation]
Unlike T ( D ) , the matrix Z has the advantage of being symmetric and containing only one factor of Λ .
Our approach will be to study the second-order statistics of Z , which will trivially give these statistics for T ( D ) .
From Equation 11, we can approximate the mean of Z as
[Equation]
We also deﬁne a modiﬁed matrix Z ( U ) ≡ Φ (cid:0) Φ T U T ΛUΦ (cid:1) − 1 Φ T , where U is an orthogonal M ×
M matrix.
Because the measure over which Φ is averaged is rotation-invariant, we can equivalently average over ˜ Φ ≡ UΦ with the same measure, giving
[Equation]
It is similarly the case that
[Equation] [Equation]
Our aim will be to calculate expectations of the form E Φ [ Z ij Z k(cid:96) ] .
With a clever choice of U , we can now see that most choices of the four indices will make this expression zero.
We deﬁne
U ( m ) ab ≡ δ ab (1 − 2 δ am ) and observe that, because Λ is diagonal, ( U ( m ) )
T ΛU ( m ) =
Λ and thus
Z ( U ( m ) )
=
Z .
Equation 49 then yields that
[Equation]
from which it follows that E Φ [ Z ij Z k(cid:96) ]
=
if any index is repeated an odd number of times.
In light of the fact that Z ij =
Z ji , there are only three distinct nontrivial cases to consider:
E Φ [ Z ii Z ii ] , 2.
E Φ [ Z ij Z ij ] with i (cid:54) =
j , and 3.
E Φ [ Z ii Z jj ] with i (cid:54) =
j .
We stress that we are not using the Einstein convention of summation over repeated indices.
G. C ASE STUDIES
Cases 1 and 2.
We now consider differentiating Z with respect to a particular element of the matrix Λ .
This yields
[Equation]
where φ i is the i -th row of Φ .
This gives us the useful expression that
[Equation]
We now set (cid:96) =
i and evaluate this expression using Equation 47, concluding that
[Equation]
We did not require that i (cid:54) =
j , and so Equation 53 holds for Case 1 as well as Case 2.
Case 3.
We now aim to calculate E Φ [ Z ii Z jj ] with i (cid:54) =
j .
We might hope to use Equationin calculating E Φ [ Z ii Z jj ] , but this approach is stymied by the fact that we would need to take a derivative with respect to Λ ij we only have an approximation for Z for diagonal Λ .
We can circumvent this by means of Z ( U ) .
From the deﬁnition of Z ( U ) , we ﬁnd that
[Equation]
Differentiating with respect to both U ij and U ji with opposite signs ensures that the derivative is taken within the manifold of orthogonal matrices.
Now, using Equation 48, we ﬁnd that
[Equation]
Taking the expectation of Equations
[Equation]
plugging in Equation 53 for the squared off-diagonal element, comparing to
[Equation]
and performing a bit of algebra
[Equation]
we conclude that
[Equation] [Equation]
and that Z ii , Z jj are anticorrelated with covariance
[Equation]
With the use of Kronecker deltas, we can combine Equations 54 and 58 into one expression covering all cases.
As can be veriﬁed by case-by-case evaluation, one such expression is
[Equation]
Using the fact that T ( D ) ij = λ i Z ij , we obtain the elementwise covariances of T ( D ) given in Equation 13.
G. D ERIVING AN EXPRESSION FOR MSE
Mean MSE is given by E ( f )
=
E (cid:2) ( ˆv − v ) 2 (cid:3) = v T E (cid:2)
( T ( D ) − I M ) T ( T ( D ) − I M ) (cid:3) v = v T Ev , where v and ˆv are the eigenfunction coefﬁcients of the true and learned functions and we have deﬁned E ≡ E (cid:2)
( T ( D ) − I M ) T ( T ( D ) − I M ) (cid:3) .
We can derive Equation 15 as follows:
[Equation]
Though it is not apparent at a glance
[Equation]
this expression for MSE is in fact equivalent to the main result of Bordelon et al.
(2020) with a ridge parameter of zero.
To obtain their expression from ours, substitute n → p , C → t/p , q → p 2 /t − p 3 γ/t 3 , and v i → λ 1 / 2 i w i .
Given that our derivations are quite different — they provide one using the method of characteristics and one using the replica trick from statistical mechanics that both directly yield ﬁnal approximate expressions, while ours ﬁrst derives an approximation with one unknown parameter C and then later determines C with the no-free-lunch theorem — it is interesting that we nonetheless recover their results.
We note that our results for other second-order statistics besides MSE did not appear in this or any other prior work.
G. S ECOND - ORDER STATISTICS CAN BE WRITTEN STRICTLY IN TERMS OF LEARNABILITY
As a ﬁnal theoretical note, we observe that our expressions for transfer matrix covariances and MSE can all be written strictly in terms of the modewise learnabilities
L i ≡ λ
i λ i + C .
The fact that this is possible supports our claim that modewise learnabilities are quantities of fundamental theoretical importance to kernel regression.
We begin by noting that
[Equation]
and
[Equation]
With these expressions in hand, we can quickly see that
[Equation]
and
[Equation]
These expressions allow one to more clearly see the effects of the model’s inductive bias.
For example, in light of the fact that L i ∈ [0 , 1] , Equation 62 shows that Var (cid:104) T ( D ) ij (cid:105) is larger the higher the learnability of the “learned mode” i and the lower the learnability of the “target mode” j , meaning that low-learnability modes tend to be mistaken for high-learnability modes.
Since the model tends to err towards high-learnability modes, we thus expect that the learned function ˆ f will tend to be spectrally biased, placing outsize weight in more learnable modes.
One recurring quantity in these expressions is L i (1 − L i ) , which is zero for L i = 0 , 1 and maximal at L i = .
Intuitively speaking, it describes the rate at which a mode is “currently being learned” with increasing n , as can be seen from the fact that
[Equation]
This equation states that a new unit of learnability (i.e. a new data point) is divvied up among all the eigenmodes in proportion to L i (1 − L i ) .
H E XPERIMENTAL DETAILS
We conduct all our experiments using JAX, performing exact NTK regression with the Neural Tangents library (Novak et al., 2019b) built atop it.
For the dataset sizes we consider in this paper, exact
NTK regression is typically quite fast, running in seconds, while the training time of ﬁnite networks varies from seconds to minutes and depends on width, depth, training set size, and eigenmode.
In particular, as described by Rahaman et al.
(2019), lower eigenmodes take longer to train (especially when aiming for near-zero training MSE as we do here).
Unless otherwise stated, all experiments used four-hidden-layer ReLU networks initialized with
NTK parameterization (Sohl-Dickstein et al., 2020) with σ w = 1 . 4 , σ b = .
The tanh networks used in generating Figure 3 instead used σ w = 1 . 5 .
Experiments on the unit circle always used a learning rate of . 5 , while experiments on the hypercube and hypersphere used a learning rate of .or .
1 depending on the experiment.
While higher learning rates led to faster convergence, they often also gave qualitatively different generalization behavior, in line with the large learning rate regimes described by Lewkowycz et al.
Means and 1 σ error bars always reﬂect statistics fromrandom dataset draws and initializations (for ﬁnite nets), except for the nonmonotonic MSE curves of Figure A1, which used 100 trials.
We emphasize that the “no free lunch” experiment of Figureused only a single trial.
I NTK EIGENVALUES
DOMAINS
[Figure]