G ENERALIZATION THROUGH THE L ENS OF L EAVEONE - OUT E RROR
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the tremendous empirical success of deep learning models to solve various learning tasks, our theoretical understanding of their generalization ability is very limited.
Classical generalization bounds based on tools such as the VC dimension or Rademacher complexity, are so far unsuitable for deep models and it is doubtful that these techniques can yield tight bounds even in the most idealistic settings (Nagarajan & Kolter, 2019).
In this work, we instead revisit the concept of leave-one-out error to measure the generalization ability of deep models in the so-called kernel regime.
While popular in statistics, it has been largely overlooked in the context of deep learning.
By building upon the recently established connection between neural networks and kernel learning, we leverage the closed-form expression for the leave-one-out error, giving us access to an efﬁcient proxy for the test error.
We show both theoretically and empirically that the leaveone-out error is capable of capturing various phenomena in generalization theory, such as double descent, random labels or transfer learning.
Our work therefore demonstrates that the leave-one-out error provides a tractable way to estimate the generalization ability of deep neural networks in the kernel regime, opening the door to potential, new research directions in the ﬁeld of generalization.
I NTRODUCTION
Neural networks have achieved astonishing performance across many learning tasks such as in computer vision (He et al., 2016), natural language processing (Devlin et al., 2019) and graph learning (Kipf & Welling, 2017) among many others.
Despite the large overparametrized nature of these models, they have shown a surprising ability to generalize well to unseen data.
The theoretical understanding of this generalization ability has been an active area of research where contributions have been made using diverse analytical tools (Arora et al., 2018; Bartlett et al., 2019; 2017; Neyshabur et al., 2015; 2018).
Yet, the theoretical bounds derived by these approaches typically suffer from one or several of the following limitations: i) they are known to be loose or even vacuous (Jiang et al., 2019), ii) they only apply to randomized networks (Dziugaite & Roy, 2018; 2017), and iii) they rely on the concept of uniform convergence which was shown to be non-robust against adversarial perturbations (Nagarajan & Kolter, 2019).
In this work, we revisit the concept of leave-one-out error (LOO) and demonstrate its surprising ability to predict generalization for deep neural networks in the so-called kernel regime.
This object is an important statistical estimator of the generalization performance of an algorithm that is theoretically motivated by a connection to the concept of uniform stability (Pontil, 2002).
It has also recently been advocated by Nagarajan & Kolter (2019) as a potential substitute for uniform convergence bounds.
Despite being popular in classical statistics (Cawley & Talbot, 2003; Vehtari et al., 2016; Fukunaga & Hummels, 1989; Zhang, 2003), LOO has largely been overlooked in the context of deep learning, likely due to its high computational cost.
However, recent advances from the neural tangent kernel perspective (Jacot et al., 2018) render the LOO error all of a sudden tractable thanks to the availability of a closed-form expression due to Stone (1974, Eq. 3.13).
While the role of the LOO error as an estimator of the generalization performance is debated in the statistics community (Zhang & Yang, 2015; Bengio & Grandvalet, 2004; Breiman, 1996; Kearns & Ron, 1999), the recent work by Patil et al.
(2021) established a consistency result in the case of ridge regression, ensuring that LOO converges to the generalization error in the large sample limit, under mild assumptions on the data distribution.
Inspired by these recent advances, in this work, we investigate the use of LOO as a generalization measure for deep neural networks in the kernel regime.
We ﬁnd that LOO is a surprisingly rich descriptor of the generalization error, capturing a wide range of phenomena such as random label ﬁtting, double descent and transfer learning.
Speciﬁcally, we make the following contributions
: • We extend the LOO expression for the multi-class setting to the case of zero regularization and derive a new closed-form formula for the resulting LOO accuracy.
• We investigate both the LOO loss and accuracy in the context of deep learning through the lens of the NTK, and demonstrate empirically that they both capture the generalization ability of networks in the kernel regime in a variety of settings.
• We showcase the utility of LOO for practical networks by accurately predicting their transfer learning performance.
• We build on the mathematically convenient form of the LOO loss to derive some novel insights into double descent and the role of regularization.
Our work is structured as follows.
We give an overview of related works in Section 2.
In Sectionwe introduce the setting along with the LOO error and showcase our extensions to the multi-class case and LOO accuracy.
Then, in Section 4 we analyze the predictive power of LOO in various settings and present our theoretical results on double descent.
We discuss our ﬁndings and future work in Section 5.
R ELATED W ORK
Originally introduced by Lachenbruch (1967); Mosteller & Tukey (1968), the leave-one-out error is a standard tool in the ﬁeld of statistics that has been used to study and improve a variety of models, ranging from support vector machines (Weston, 1999), Fisher discriminant analysis (Cawley & Talbot, 2003), linear regression (Hastie et al., 2020; Patil et al., 2021) to non-parametric density estimation (Fukunaga & Hummels, 1989).
Various theoretical perspectives have justiﬁed the use of the LOO error, including for instance the framework of uniform stability (Bousquet & Elisseeff, 2002) or generalization bounds for KNN models (Devroye & Wagner, 1979).
Elisseeff et al.
(2003) focused on the stability of a learning algorithm to demonstrate a formal link between LOO error and generalization.
The vanilla deﬁnition of the LOO requires access to a large number of trained models which is typically computationally expensive.
Crucially, Stone (1974) derived an elegant closed-form formula for kernels, completely removing the need for training multiple models.
In the context of deep learning, the LOO error remains largely unexplored.
Shekkizhar & Ortega (2020) propose to ﬁt polytope interpolators to a given neural network and estimate its resulting LOO error.
Alaa & van der Schaar (2020) rely on a Jackknife estimator, applied in a LOO fashion to obtain better conﬁdence intervals for Bayesian neural networks.
Recent works have established a connection between kernel regression and deep learning.
Surprisingly, it is shown that an inﬁnitely-wide, fully-connected network behaves like a kernel both at initialization (Neal, 1996; Lee et al., 2018; de G. Matthews et al., 2018) as well as during gradient ﬂow training (Jacot et al., 2018).
Follow-up works have extended this result to the non-asymptotic setting (Arora et al., 2019a; Lee et al., 2019), proving that wide enough networks trained with small learning rates are in the so-called kernel regime, i.e. essentially evolving like their corresponding tangent kernel.
Moreover, the analysis has also been adapted to various architectures (Arora et al., 2019a;
Huang et al., 2020; Du et al., 2019; Hron et al., 2020; Yang, 2020) as well as discrete gradient descent (Lee et al., 2019).
The direct connection to the ﬁeld of kernel regression enables the usage of the closed-form expression for LOO error in the context of deep learning, which to the best of our knowledge has not been explored yet.
S ETTING AND B ACKGROUND
In the following, we establish the notations required to describe the problem of interest and the setting we consider.
We study the standard learning setting, where we are given a dataset of inputtarget pairs S
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } where each ( x i , y i ) i.i.d.
∼ D for i = 1 , . .
. , n is distributed according to some probability measure D .
We refer to x i ∈
X ⊂ R d as the input and to y ∈ Y ⊂ R K as the target.
We will often use matrix notations to obtain more compact formulations, thus summarizing all inputs and targets as matrices X ∈ R n × d and Y ∈ R n × K .
We will mainly be concerned with the task of multiclass classiﬁcation, where targets y i are encoded as one-hot vectors.
We consider a function space F and an associated loss function L f : X × Y −→ R that measures how well a given function f ∈ F performs at predicting the targets.
More concretely, we deﬁne the regularized empirical error as
[Equation]
where λ > 0 is a regularization parameter and Ω : F −→ R is a functional.
For any vector v ∈ R K , let v ∗ =
argmax k ≤ K v k . Using this notation, we let a f ( x , y ) :
= 1 { f ∗ ( x )=
y ∗ } .
We then deﬁne the empirical accuracy ˆ A S as the average number of instances for which f ∈ F provides the correct prediction, i.e.
[Equation]
Finally, we introduce a learning algorithm Q F : ( X × Y ) n −→ F , that given a function class F and a training set S ∈ ( X × Y ) n , chooses a function f ∈ F .
We will exclusively be concerned with learning through empirical risk minimization, i.e.
[Equation]
and use the short-cut ˆ f S :
= ˆ f λ =0 S .
In practice however, the most important measures are given by the generalization loss and accuracy of the model, deﬁned as
[Equation]
A central goal in machine learning is to control the difference between the empirical error ˆ L S ( f ) and the generalization error L ( f ) .
In this work, we mainly focus on the mean squared loss,
[Equation]
and also treat classiﬁcation as a regression task, as commonly done in the literature (Lee et al., 2018;
Chen et al., 2020; Arora et al., 2019a; Hui & Belkin, 2021; Bachmann et al., 2021).
Finally, we consider the function spaces induced by kernels and neural networks, presented in the following.
3. K ERNEL L EARNING
A kernel K : X × X −→ R is a symmetric, positive semi-deﬁnite function, i.e. K( x , x (cid:48) )
=
K( x (cid:48) , x ) ∀ x , x (cid:48) ∈ X and for any set { x 1 , . . .
, x n } ⊂ X , the matrix K ∈ R n × n with entries
K ij = K
( x i , x j ) is positive semi-deﬁnite.
It can be shown that a kernel induces a reproducing kernel Hilbert space , a function space equipped with an inner product (cid:104)· , ·(cid:105) H , containing elements
[Equation]
where cl is the closure operator.
Although H is an inﬁnite dimensional space, it turns out that the minimizer ˆ f λ S can be obtained in closed form for mean squared loss, given by
[Equation]
where the regularization functional is induced by the inner product of the space, Ω( f )
=
||
f || 2 H ,
K ∈ R n × n has entries K ij =
K( x i , x j ) and K x ∈ R n has entries ( K x ) i
= K( x , x i ) .
A † denotes the pseudo-inverse of A ∈ R n × n .
We will analyze both the regularized ( ˆ f λ S ) and unregularized model ( ˆ f S ) and their associated LOO errors in the following.
For a more in-depth discussion of kernels, we refer the reader to Hofmann et al.
; Paulsen & Raghupathi (2016).
3. N EURAL N ETWORKS AND A SSOCIATED K ERNELS
Another function class that has seen a rise in popularity is given by the family of fully-connected neural networks of depth L ∈ N ,
[Equation]
, for W
[Equation] ∈ [Equation] , for d l ∈ N , d 0 = d , d L = K and σ : R −→ R is an element-wise non-linearity.
We denote by θ ∈ R m the concatenation of all parameters (cid:0) W (1) , . . .
, W ( L ) (cid:1) , where m {+= (cid:80) L − 1 i =
 d i d i ++} is the number of parameters of the model.
The parameters are initialized randomly according to W ( l ) ij ∼ N (0 , σ 2 w ) for σ w > 0 and then adjusted to optimize the empirical error ˆ L S , usually through an iterative procedure using a variant of gradient descent.
Usually, the entire vector θ is optimized via gradient descent, leading to a highly non-convex problem which does not admit a closed-form solution ˆ f , in contrast to kernel learning.
Next, we will review settings under which the output of a deep neural network can be approximated using a kernel formulation.
Random Features.
Instead of training all the parameters θ , we restrict the function space by only optimizing over the top layer W ( L ) and freezing the other variables W (1) , . .
, W ( L − 1) to their value at initialization.
This restriction makes the model linear in its parameters and induces a ﬁnitedimensional feature map
[Equation]
Originally introduced by Rahimi & Recht (2008) to enable more efﬁcient kernel computations, the random feature model has recently seen a strong spike in popularity and has been the topic of many theoretical works (Jacot et al., 2020; Mei & Montanari, 2021).
Inﬁnite Width.
Another, only recently established correspondence emerges in the inﬁnite-width scenario.
By choosing σ w = 1 √ d l for each layer, Lee et al.
(2018) showed that at initialization, the network exhibits a Gaussian process behaviour in the inﬁnite-width limit.
More precisely, it holds that f k i.i.d.
∼ GP (cid:0) 0 , Σ ( L ) (cid:1) , for k = 1 , . . .
, K , where Σ ( L ) : R d × R d −→ R is a kernel, coined NNGP, obeying the recursive relation
[Equation]
with base Σ (1) ( x , x (cid:48) )
= 1 √ d x T x (cid:48) and Σ ( l − 1) |
x , x (cid:48) ∈ R 2 × 2 is Σ ( l − 1) evaluated on the set { x , x (cid:48) } .
To reason about networks trained with gradient descent, Jacot et al.
(2018) extended this framework and proved that the empirical neural tangent kernel converges to a constant kernel Θ .
The so-called neural tangent kernel Θ : R d × R d −→ R is also available through a recursive relation involving the NNGP:
[Equation]
, with base Θ (1) ( x , x (cid:48) )
=
Σ (1) ( x
[Equation]
x
[Equation]
and ˙Σ ( l ) ( x , x (cid:48) )
=
E z ∼N ( 0 , Σ ( l − 1) | x , x (cid:48) (cid:2) ˙ σ ( z 1 ) ˙
σ ( z 2 ) (cid:3) .
Training inﬁnitely-wide networks with gradient descent thus reduces to kernel learning with the NTK Θ ( L ) .
Arora et al. (2019a); Lee et al.
(2019) extended this result to a non-asymptotic setting, showing that very wide networks trained with small learning rates become indistinguishable from kernel learning.
3. L EAVE - ONE - OUT E RROR
The goal of learning is to choose a model f ∈ F with minimal generalization error L ( f ) .
Since the data distribution D is usually not known, practitioners obtain an estimate for L ( f ) by using an additional test set S test =
{ ( x t 1 , y t 1 ) , . .
, ( x tb , y tb ) } not used as input to the learning algorithm Q F , and calculate
[Equation]
While this is a simple approach, practitioners cannot always afford to put a reasonably sized test set aside.
This observation led the ﬁeld of statistics to develop the idea of a leave-one-out error (LOO), allowing one to obtain an estimate of L ( f ) without having to rely on additional data (Lachenbruch, 1967; Mosteller & Tukey, 1968).
We give a formal deﬁnition below.
Deﬁnition 3.1.
Consider a learning algorithm Q F and a training set S
= { ( x i , y i ) } ni
=1 .
Let S − i =
S \ { ( x i , y i ) } be the training set without the i -th data point ( x i , y i ) and deﬁne ˆ f − i :
= Q F ( S − i ) , the corresponding model obtained by training on S − i .
The leave-one-out loss/accuracy of Q F on S is then deﬁned as the average loss/accuracy of ˆ f − i incurred on the left out point ( x i , y i ) , given by
[Equation]
Notice that calculating L LOO can indeed be done solely based on the training set.
On the other hand, using the training set to estimate L ( f ) comes at a cost.
Computing ˆ f − i for i
= 1 , . .
. , n requires ﬁtting the given model class n times.
While this might be feasible for smaller models such as random forests, the method clearly reaches its limits for larger models and datasets used in deep learning.
For instance, evaluating L LOO for ResNet-152 on ImageNet requires ﬁtting a model with 6 × 10parameters 10 7 times, which clearly becomes extremely inefﬁcient.
At ﬁrst glance, the concept of leave-one-out error therefore seems to be useless in the context of deep learning.
However, we will now show that the recently established connection between deep networks and kernel learning gives us an efﬁcient way to compute L LOO .
In the following, we will use the short-cut ˆ f λ S :
= ˆ f λ .
Surprisingly, it turns out that the space of kernels using Q λ F with mean-squared loss admits a closed-form expression for the leave-one-out error, only relying on a single model obtained on the full training set ( X , Y ) :
Theorem 3.2.
Consider a kernel K : R d × R d −→ R and the associated objective with regularization parameter λ > 0 under mean squared loss.
Deﬁne A
=
K
( K + λ 1 n ) − 1 .
Then it holds that
[Equation]
where the residual ∆ λik ∈ R for i = 1 , . . . , n , k = 1 , . .
, K is given by ∆ λik = Y ik − ˆ f λk ( x i ) 1 − A ii .
For the remainder of this text, we will use the short-cut L λ LOO :=
L LOO (cid:0) Q λ F (cid:1) .
While the expression for L λ LOO for K = 1 has been known in the statistics community for decades (Stone, 1974) and more recently for general K (Tacchetti et al., 2012), the formula for A λ LOO is to the best of our knowledge a novel result.
We provide a proof of Theorem 3.2 in the Appendix A.1.
Notice that Theorem 3. not only allows for an efﬁcient computation of L λ LOO and A λ LOO but also provides us with a simple formula to assess the generalization behaviour of the model.
It turns out that L λ LOO captures a great variety of generalization phenomena, typically encountered in deep learning.
Leveraging the neural tangent kernel together with Theorem 3.2 allows us to investigate their origin.
As a ﬁrst step, we consider the zero regularization limit λ −→ 0 , i.e. L 0 LOO since this is the standard setting considered in deep learning.
A quick look at the formula reveals however that care has to be taken; as soon as the kernel is invertible, we have a 00 situation.
Corollary 3.3.
Consider the eigendecomposition K = V diag( ω ) V T for V ∈ O ( n ) and ω ∈ R n .
Denote its rank by r = rank( K ) .
Then it holds that the residuals ∆ λik ∈ R can be expressed as
[Equation]
Moreover in the zero regularization limit, i.e. λ −→ 0 , it holds that
[Equation]
[Figure]
Remark.
At ﬁrst glance for λ = 0 , a distinct phase transition appears to take place when moving from r = n − 1 to r = n .
Notice however that a small eigenvalue ω n will allow for a smooth interpolation, i.e. ∆ ik ( n ) ω n −→ 0 −−−−→ ∆ ik ( n − 1) for i = 1 , . . . , n , k = 1 , . .
, K .
A similar result for the special case of linear regression and r = n has been derived in Hastie et al.
We now explore how well these formulas describe the complex behaviour encountered in deep models in the kernel regime.
L EAVE -O NE -O UT AS A G ENERALIZATION P ROXY
In this section, we study the derived formulas in the context of deep learning, through the lens of the neural tangent kernel and random feature models.
Through empirical studies, we investigate LOO for varying sample sizes, different amount of noise in the targets as well as different regimes of overparametrization, in the case of inﬁnitely-wide networks.
We evaluate the models on the benchmark vision datasets MNIST (LeCun & Cortes, 2010) and CIFAR10 (Krizhevsky & Hinton, 2009).
To estimate the true generalization error, we employ the test sets as detailed in equation 1.
We provide theoretical insights into double descent, establishing the explosion in loss around the interpolation threshold.
Finally, we demonstrate the utility of LOO for state-of-the-art networks by predicting their transferability to new tasks.
We provide more experiments for varying depth and double descent with random labels in the Appendix C. All experiments are performed in Jax (Bradbury et al., 2018) using the neural-tangents library (Novak et al., 2020).
4. F UNCTION OF S AMPLE S IZE
While known to exhibit only a small amount of bias, the LOO error may have a high variance for small sample sizes (Varoquaux, 2018).
Motivated by the consistency results obtained for ridge regression in Patil et al.
(2021), we study how the LOO error (and its variance) of an inﬁnitely-wide network behaves as a function of the sample size n .
In the following we thus interpret LOO as a function of n .
In order to quantify how well LOO captures the generalization error for different sample regimes, we evaluate a 5 -layer fully-connected NTK model for varying sample sizes on
MNIST and CIFAR10 .
We display the resulting test and LOO losses in Figure 1.
We plot the average result of 5 runs along with conﬁdence intervals.
We observe that both L LOO and A LOO closely follow their corresponding test quantity, even for very small sample sizes of around n = 500 .
As we increase the sample size, the variance of the estimator decreases and the quality of both LOO loss and accuracy improves even further, offering a very precise estimate at n = 20000 .
4. R ANDOM L ABELS
Next we study the behaviour of the generalization error when a portion of the labels is randomized.
In Zhang et al.
, the authors investigated how the performance of deep neural networks is affected when the functional relationship between inputs x and targets y is destroyed by replacing y by a random label.
More speciﬁcally, given a dataset X ∈ R n × d and one-hot targets y =
e k for k ∈ { 1 , . . .
, K } , we ﬁx a noise level p ∈ [0 , 1] and replace a subset of size pn of the targets with random ones, i.e. y = e ˜ k for ˜ k ∼ U ( { 1 , . . .
, K } ) .
We then train the model using the noisy targets ˜
y
while measuring the generalization error with respect to the clean target distribution.
As observed in
[Figure]
Zhang et al.
Rolnick et al.
(2018), neural networks manage to achieve very small empirical error even for p = 1 whereas their generalization consistently worsens with increasing p .
Random labels has become a very popular experiment to assess the quality of generalization bounds (Arora et al., 2019b; 2018; Bartlett et al., 2017).
We now show how the leave-one-out error exhibits the same behaviour as the test error under label noise, thus serving as an ideal test bed to analyze this phenomenon.
In the following, we consider the leave-one-out error as a function of the labels, i.e.
L LOO = L LOO ( y ) .
The phenomenon of noisy labels is of particular interest when the model has enough capacity to achieve zero empirical error.
For kernels, this is equivalent to rank( K ) =
n and we will thus assume in this section that the kernel matrix has full rank.
However, a direct application of Theorem 3.2 is not appropriate, since randomizing some of the training labels also randomizes the “implicit” test set.
In other words, our aim is not to evaluate ˆ f − i ˜
S against ˜ y
i since ˜ y
i might be one of the randomized labels.
Instead, we want to compare ˆ f − i ˜
S with the true label y
i , in order to preserve a clean target distribution.
To ﬁx this issue, we derive a formula for the leave-one-out error for a model trained on labels ˜
Y but evaluated on Y :
Proposition 4.1.
Consider a kernel with spectral decomposition K = V diag( ω ) V T for V ∈ R n × n orthogonal and ω ∈ R n .
Assume that rank( K ) = n .
Then it holds that the leave-one-out error L LOO ( ˜ Y ; Y ) for a model trained on ˜ S
= { ( x i , ˜ y i ) } ni =1 but evaluated on S =
{ ( x i , y i ) } ni = is given by
[Equation]
where ˜∆ ik = ∆ ik ( ˜ Y ) ∈ R is deﬁned as in Corollary 3.3.
We defer the proof to the Appendix A.4.
Attentive readers will notice that we indeed recover Theorem 3.2 when ˜ Y
= Y .
Using this generalization of LOO, we can study the setup in Zhang et al.
(2017) to test whether LOO indeed correctly reﬂects the behaviour of the generalization error.
To this end, we perform a random label experiment on MNIST and CIFAR10 with n = 20000 for a NTK model of depth 5 , where we track test and LOO losses while increasingly perturbing the targets with noise.
We display the results in Figure 2.
We see an almost perfect match between test and LOO loss as well test and LOO accuracy, highlighting how precisely LOO reﬂects the true error across all settings.
4. D OUBLE D ESCENT
Originally introduced by Belkin et al.
(2019), the double descent curve describes the peculiar shape of the generalization error as a function of the model complexity.
The error ﬁrst follows the classical U -shape, where an increase in complexity is beneﬁcial until a speciﬁc threshold.
Then, increasing the model complexity induces overﬁtting, which leads to worse performance and eventually a strong spike around the interpolation threshold.
However, as found in Belkin et al.
(2019), further increasing the complexity again reduces the error, often leading to the overall best performance.
A large body of works provide insights into the double descent curve but most of them rely on very restrictive assumptions such as Gaussian input distributions (Harzli et al., 2021;
Adlam & Pennington, 2020; d’Ascoli et al., 2020) or teacher models (Bartlett et al., 2020; Harzli et al., 2021; Adlam & Pennington, 2020; Mei & Montanari, 2021;
Hastie et al., 2020) and require very elaborate proof techniques.
Here we show under arguably weaker assumptions that the LOO loss also exhibits a spike around the interpolation threshold.
Our proof is very intuitive, relying largely on Equation 2 and standard probability tools.
In order to reason about the effect of complexity, we have to consider models with a ﬁnite dimensional feature map φ m : R d −→ R m .
Increasing the dimensionality of the feature space R m naturally overparametrizes the model and induces a dynamic in the rank r ( m ) :
= rank( K ) =
rank ( φ m ( X )) .
The exact rank dynamics r ( m ) are obviously speciﬁc to the employed feature map φ m , but by construction always monotonic.
We restrict our attention to models that admit an interpolation point, i.e. ∃ m ∗ ∈ N such that r ( m ∗ ) = n .
While the developed theory holds for any kernel, we focus the empirical evaluations largely to the random feature model due to its aforementioned connection to neural networks.
In the following, we will interpret the leave-one-out-error as a function of m and n , i.e. L LOO = L n LOO ( m ) .
Notice that L n LOO ( m ) measures the generalization error of a model trained on n − 1 points, instead of n .
The underlying interpolation point thus shifts, i.e. we denote m ∗ ∈ N such that r ( m ∗ )
=
n − 1 .
In the following we will show that
[Equation]
Intuition.
We give a high level explanation for our theoretical insights in this paragraph.
For m < m ∗ the dominating term in L n LOO ( m ) is
[Equation]
We observe a blow-up for small λ due to the unit vector nature of both V i • and V • j .
For instance, considering the interpolation point where r
=
n − 1 and λ = 0 , we get g ( n − 1 , 0) =
(cid:80) ni
=1 1 V 2 in , the sum over the elements of V • n , which likely has a small entry due to its unit norm nature.
On the other hand, reducing r or increasing λ has a dampening effect as we combine multiple positive terms together, increasing the likelihood to move away from the singularity.
A similar dampening behaviour in L n LOO ( m ) can also be observed for m > m ∗ when ω n >> 0 .
In order to mathematically prove the spike at the interpolation threshold, we need an assumption on the spectral decomposition of K :
Assumption 4.2.
Consider again the eigendecomposition K = V diag ( ω ) V T .
We impose the following distributional assumption on the rotation matrix V :
[Equation]
where U ( O n ) is the Haar measure on the orthogonal group O n .
Under Assumption 4.2, we can prove that at the interpolation point, the leave-one-out error exhibits a spike, which worsens as we increase the amount of data:
Theorem 4.3.
For large enough n ∈ N and λ −→ 0 , it holds that
[Equation]
where A ∼ Γ( 12 , 1) is independent of n .
L nLOO ( m ∗ ) hence diverges a.s. with n −→ ∞ .
It is surprising that Assumption 4.2 sufﬁces to prove Theorem 4.3, highlighting the universal nature of double descent.
We provide empirical evidence for the double descent behaviour of LOO in
Figure 3.
We ﬁt a random feature model of varying width to the binary (labels) MNIST dataset with n = 5000 .
We observe that the LOO error indeed closely follows the test error, exhibiting the spike as predicted by Theorem 4.3.
4. T RANSFER L EARNING
Finally, we study more realistic networks often employed for large-scale computer vision tasks.
We study the task of transfer learning (Yosinski et al., 2014) by considering classiﬁers trained on
[Figure]
ImageNet (Krizhevsky et al., 2012) and ﬁne-tuning their top-layer to the CIFAR10 dataset.
Notice that this corresponds to training a kernel with a data-dependent feature map φ data induced by the non-output layers.
For standard kernels however, the feature map is not dependent on any training data and consistency results such as Patil et al.
(2021) indeed rely on this fact.
One could thus expect that the data-dependent feature map might lead to a too optimistic LOO error.
Our experiments on
ResNet18 (He et al., 2016), AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2015) and DenseNet (Huang et al., 2017) however demonstrate that transferring between tasks indeed preserves the stability of the networks and leads to a very precise estimation of the test accuracy.
We display the results in Table 1.
We consider a small data regime where n = 10000 , as often encountered in practice for transfer learning tasks.
In order to highlight the role of pre-training, we also evaluate random feature maps φ rand where we use standard initialization for all the parameters in the network.
Indeed we clearly observe the beneﬁts of pre-training, leading to a high increase in performance for all considered models.
[Table]
D ISCUSSION
In this work we investigated the concept of leave-one-out error as a measure for generalization in the context of deep learning.
Through the use of NTK, we derive new expressions for the LOO loss and accuracy of deep models in the kernel regime and observe that they capture the generalization behaviour for a variety of settings.
Moreover, we mathematically prove that LOO exhibits a spike at the interpolation threshold, needing only a minimal set of theoretical tools.
Finally, we demonstrate that LOO accurately predicts the predictive power of deep models when applied to the problem of transfer learning.
Notably, the simple form of LOO might open the door to new types of theoretical analyses to better understand generalization, allowing for a disentanglement of the roles of various factors at play such as overparametrization, noise and architectural design.
A O MITTED P ROOFS
We list all the omitted proofs in the following section.
A. P ROOF OF T HEOREM 3.
Theorem 3.2.
Consider a kernel K : R d × R d −→ R and the associated objective with regularization parameter λ > 0 under mean squared loss.
Deﬁne A
=
K
( K + λ 1 n ) − 1 .
Then it holds that
[Equation]
where the residuals ∆ λik ∈ R for i = 1 , . . . , n , k = 1 , . .
, K is given by ∆ λik = Y ik −
[Equation]
λk
[Equation]
Proof.
Recall that ˆ f λ S solves the optimization problem
[Equation]
and predicting on the training data takes the form ˆ f S ( X ) =
AY for some A ∈ R n × n .
Now consider the model f − i λ :
= ˆ f λ S − i obtained from training on S − i .
W.L.O.G. assume that i = n .
We want to understand the quantity ˆ f − n λ,k ( x n ) , i.e. the k -th component of the prediction on x i .
To that end, consider the dataset Z := S − n ∪ (cid:110)
[Equation]
x
[Equation]
, ˆ
[Equation]
n λ ( x
[Equation]
) (cid:17) (cid:111) .
Notice that for any f ∈ F , it holds
[Equation]
where the ﬁrst inequality follows because ˆ f − n minimizes L λ S − n by deﬁnition.
Thus ˆ f − n λ also minimizes L λ Z and hence also takes the form
[Equation]
where ˜ Y ∈ R n × K such that ˜ Y
ik = (cid:40)
Y ik if i (cid:54) =
n ˆ f − n λ,k ( x n ) else .
Now we care about the n -th prediction which is
[Equation]
Solving for ˆ f − n λ,k ( x n ) gives
[Equation]
Then, subtracting Y nk leads to
Y
[Equation]
Squaring and summing the expression over n and k results in the formula.
For the accuracy, we know that we correctly predict if the maximal coordinate of ˆ f − n λ ( x n ) agrees with the maximal coordinate of y n , i.e.
[Equation]
From equation 3, we notice that
[Equation]
We thus have to check the indicator 1 { ( y n − ∆ n • ) ∗
=
y ∗ n } and sum it over n to obtain the result.
A. B INARY C LASSIFICATION
Here we state the corresponding results in the case of binary classiﬁcation.
The formulation for the accuracy changes slightly as now the sign of the classiﬁer serves as the prediction.
Proposition A.1.
Consider a kernel K : R d × R d −→ R and the associated objective with regularization parameter λ > 0 under mean squared loss.
Deﬁne A
=
K
( K + λ 1 n ) − 1 .
Then it holds that
[Equation]
where the residuals ∆ λi ∈ R for i = 1 , . .
. , n is given by ∆ λi = y
i − ˆ f λ ( x i )
1 − A ii .
Proof.
Notice that the result for L LOO is analogous to the proof for Theorem 3.2 by setting K = 1 .
For binary classiﬁcation we use the sign of the classiﬁer as a decision rule, i.e. the classiﬁer predicts correctly if y ˆ f λ ( x ) > 0 .
We can thus calculate that
[Equation]
Thus, the n -th sample is correctly classiﬁed
if and only if
[Equation]
We now just count the correct predictions for the accuracy, i.e.
[Equation]
A. P ROOF OF C OROLLARY 3.
Corollary 3.3.
Consider the eigendecomposition K = V diag( ω ) V T for V ∈ O ( n ) and ω ∈ R n .
Denote its rank by r = rank( K ) .
Then it holds that the residuals ∆ λik ∈ R can be expressed as
[Equation]
Moreover for zero regularization, i.e. λ −→ 0 , it holds that
[Equation]
if r = n
Proof.
Deﬁne A
= K ( K + λ 1 n ) − 1 ∈ R n × n .
Recall that ˆ f λ S ( X ) =
Ay and thus ˆ f λk ( x i ) =
(cid:80) n j =
A ij Y jk .
Let us ﬁrst simplify A :
[Equation]
We can characterize the off-diagonal elements for i (cid:54) =
j as follows:
[Equation]
where we made use of the fact that V i • ⊥ V j • .
The diagonal elements i =
j on the other hand can be written as
[Equation]
where we have made use of the fact that V i • is a unit vector.
Plugging-in the quantities into L LOO results in
[Equation]
Now crucially, in the full rank case r = n , we have empty sums, i.e. (cid:80) nl = r +1 = (cid:80) nl = n +1 =
and we obtain
[Equation]
On the other hand, in the rank deﬁcient case r < n we can cancel the regularization term:
[Equation]
Plugging this into the formulas for L λ LOO and A λ LOO concludes the proof.
A. P ROOF OF P ROPOSITION 4.
Proposition 4.1.
Consider a kernel with spectral decomposition K = V diag( ω ) V T for V ∈ R n × n orthogonal and ω ∈ R n .
Assume that rank( K ) = n .
Then it holds that the leave-one-out error L LOO ( ˜ Y ; Y ) for a model trained on ˜ S
= { ( x i , ˜ y i ) } ni =1 but evaluated on S =
{ ( x i , y i ) } ni = is given by
[Equation]
where ˜∆ ik = ∆ ik ( ˜ Y ) ∈ R is deﬁned as in Corollary 3.3.
Proof.
Denote by ˜ S − i the dataset { ( x j , ˜ y j ) } nj (cid:54) =
i .
Denote by ˜ f − i λ the model trained on ˜ S − i .
Recall from the proof of Theorem 3.2 that
[Equation]
Instead of subtracting the same label ˜
Y nk , we now subtract the evaluation label Y
nk :
[Equation]
The second term ˜∆ λik is now the summand of the standard leave-one-out error where we evaluate on ˜ Y .
We can hence re-use Theorem 3.3 to decompose it.
Squaring and summing over n concludes the
LOO loss result.
For the accuracy, we notice that a similar derivation as for Theorem 3.2 applies:
[Equation]
We thus have to check the indicator against the true label y n ,
i
[Equation]
e. 1 { ( ˜ y n − ˜∆ n • ) ∗ =
y ∗ n } and sum it over n to obtain the result.
A. P ROOF OF T HEOREM 4.
Theorem 4.3.
For large enough n ∈ N
[Equation]
we can estimate as
[Equation]
where A ∼ Γ( 1 2 , 1) is independent of n .
L n LOO ( m ∗ ) hence diverges a
[Equation]
s. with n −→ ∞ .
Proof.
First we notice that for m = m ∗ , by deﬁnition it holds that r = n − 1 , which simpliﬁes the
LOO expression to
[Equation]
(cid:32) n (cid:88) i =
y i V in (cid:33)
For notational convenience, we will introduce v ∈ R n such that v i :
= V in .
We will now bound the both factors one-by-one.
The ﬁrst part is a simple application of Proposition B.1 and Proposition
B.5:
[Equation]
Now for large enough n , we can use Lemma B.6 to make the following approximation in distribution:
[Equation]
where A ∼ Γ( 12 , 1) .
Thus for large enough n , it holds that
[Equation]
As the approximation becomes exact for larger and larger n , we conclude that
[Equation]
B A DDITIONAL L EMMAS
In this section we present the additional technical Lemmas needed for the proofs of the main claims in A.
Lemma B.1.
Consider a unit vector v ∈ S n − 1 .
Then it holds that
[Equation]
Proof.
Let’s parametrize each v i as
[Equation]
for i = 1 , . . . , n and z ∈ R n .
One can easily check that ||
v || 2 = 1 and hence v ∈ S n − 1 .
Plugging this in, we arrive at
[Equation]
We can re-arrange the sum into pairs
[Equation]
for a 2 =
z
2 j z 2 i > 0 and using the fact that x + 1 x ≥ 2 for x ≥ 0 .
We can ﬁnd n ( n − 1)such summands, and thus
[Equation]
Lemma B.2.
Consider v ∼ U (cid:0) S n − 1 (cid:1) and any ﬁxed orthogonal matrix U ∈
O ( n ) .
Then it holds that
[Equation]
Proof.
This is a standard result and can for instance be found in Vershynin (2018).
Lemma B.3.
Consider w ∼ N ( 0 , 1 n ) .
Then it holds that
[Equation]
Proof.
This is a standard result and can for instance be found in Vershynin (2018).
Lemma B.4.
Consider two independent Gamma variables X ∼ Gamma( α, ν ) and Y ∼
Gamma( β, ν ) .
Then it holds that
[Equation]
Proof.
This is a standard result and can for instance be found in Bowman et al.
Lemma B.5.
Consider v ∼ U (cid:0) S n − 1 (cid:1) .
Then it holds that
[Equation]
Proof.
First realize that we can write
[Equation]
where ˜ 1 n = (cid:16) 1 √ n , . . .
, 1 √ n (cid:17) with || ˜ 1 n ||
=
and the fact that v (cid:12) y
( d ) = v for ﬁxed y ∈ {− 1 , 1 } n .
The idea is now to choose U ∈ O ( n ) such that U T ˜ 1 n = e 1 = (1 , 0 , . . . , 0) .
Then by using Lemma
B.2, it holds
[Equation]
Thus, surprisingly, it sufﬁces to understand the distribution of v 1 .
By Lemma B.3, we know that
[Equation]
where z ∼ N ( 0 , 1 n ) .
We are interested in the square of this expression,
[Equation]
where we deﬁne w = (cid:80) n i =
z 2 i , clearly independent of z 21 .
Moreover, it holds that z 2 1 ∼
Gamma (cid:0) 12 , 12 (cid:1) and w ∼ Gamma (cid:0) n − 1 2 , 12 (cid:1) .
Thus, by Lemma B.4 we can conclude that
[Equation]
Lemma B.6.
Consider the sequence of Beta distributions X n ∼ Beta( k, n ) .
Then it holds that
[Equation]
Proof.
This is a standard result and can for instance be found in Walck (1996).
C F URTHER E XPERIMENTS
In this section we present additional experimental results on the leave-one-out error.
[Figure]
C. R ANK D YNAMICS FOR D IFFERENT K ERNELS
In this section, we illustrate how the rank rank( K ) of the kernel evolves as a function of the underlying complexity.
As observed in various works, the spike in the double descent curve for neural networks usually occurs around the interpolation threshold, i.e. the point in complexity where the model is able to achieve zero training loss.
Naturally, for the kernel formulation, interpolation is achieved when the kernel matrix K has full rank.
We illustrate in the following how the dynamics of the rank change for different architectures, displaying a similar behaviour as ﬁnite width neural networks.
In Fig.
4 (a) we show a depth 1 random feature model with feature maps σ ( W x ) where
W ∈ R m × d and σ is the ReLU non-linearity.
We use MNIST with n = 100 .
We measure the complexity through the width m of the model.
We observe that in this special case, the critical complexity exactly coincides with the number of samples, i.e. rank( K )
= n as soon as m = n .
This seems counter-intuitive at ﬁrst as an exact match of the complexity with the sample size is usually not observed for standard neural networks.
We show however in Fig.
4 (b) and (c) that the rank dynamics indeed become more involved for larger depth.
In Fig.
4 (b) and (c), we use a 2 -layer random feature model with feature map σ ( V σ ( W x )) for weights W ∈ R m 1 × d and V ∈ R m 2 × m 1 .
In (b) we show the change in rank as m 1 increases, while m 2 = 120 is ﬁxed.
We observe that indeed the dynamics change and the critical complexity is not at m 1 = n .
Similarly in (c) we ﬁx m 1 =and vary m 2 .
Also there we observe that the critical threshold is not located at n but rather a bigger width is needed to reach interpolation.
Finally we also study the linearizations of networks that also give rise to kernels.
More concretely, we consider feature maps of the form
[Equation]
where f θ is a fully-connected network with parameters θ ∈ R p , as introduced before.
Our doubledescent framework also captures this scenario.
In Fig.
5, we display the rank dynamics in case of a 2 layer network f θ ( x ) =
w T σ ( U σ ( V x )) where w ∈ R m 2 , U ∈
R m 2 × m 1 , V ∈ R m 1 × d and σ is the ReLU non-linearity.
Again we observe that the rank dynamics change and do not coincide with the number of samples n .
C. LOO AS FUNCTION OF DEPTH L
We study how the depth L of the NTK kernel Θ ( L ) affects the performance of LOO loss and accuracy.
We use the datasets MNIST and CIFAR10 with n = 5000 and evaluate NTK models with depth ranging from 3 to 20 .
We present our ﬁndings in Figure 6.
Again we see a very close match between LOO and the corresponding test quantity for CIFAR10 .
Interestingly the performance is slightly worse for very shallow models.
For MNIST we see a gap between LOO loss and test loss, which is due to the very zoomed-in nature of the plot (the gap is actually only 0 . 015 ) as the loss values are very small in general.
Indeed we observe an excellent match between the test and LOO accuracy.
[Figure]
C. D OUBLE D ESCENT WITH R ANDOM L ABELS
Here we demonstrate how the spike in double descent is a very universal phenomenon as demonstrated by Theorem 4.3.
We consider a random feature model of varying width m on binary MNIST with n = 2000 , where the labels are fully randomized ( p = 1 ), destroying thus any relationship between the inputs and targets.
Of course, there will be no double descent behaviour in the test accuracy as the network has to perform random guessing at any width.
We display this in Figure 7.
We observe that indeed the model is randomly guessing throughout all the regimes of overparametrization.
Both the test and LOO loss however, exhibit a strong spike around the interpolation threshold.
This underlines the universal nature of the phenomenon, connecting with the fact that Theorem 4. does not need any assumptions on the targets.
C. T RANSFER L EARNING L OSS
We report the corresponding test and leave-one-out losses, moved to the appendix due to space constraints.
We display the scores in Table 2.
Again we observe a very good match between the test and LOO losses.
Moreover, we again ﬁnd that pre-training on ImageNet is beneﬁcial in terms of the achieved loss values.
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]