P HENOMENOLOGY OF D OUBLE D ESCENT IN F INITE -W IDTH N EURAL N ETWORKS
A BSTRACT
‘Double descent’ delineates the generalization behaviour of models depending on the regime they belong to: under- or over-parameterized.
The current theoretical understanding behind the occurrence of this phenomenon is primarily based on linear and kernel regression models — with informal parallels to neural networks via the Neural Tangent Kernel.
Therefore such analyses do not adequately capture the mechanisms behind double descent in finite-width neural networks, as well as, disregard crucial components — such as the choice of the loss function.
We address these shortcomings by leveraging influence functions in order to derive suitable expressions of the population loss and its lower bound, while imposing minimal assumptions on the form of the parametric model.
Our derived bounds bear an intimate connection with the spectrum of the Hessian at the optimum, and importantly, exhibit a double descent behaviour at the interpolation threshold.
Building on our analysis, we further investigate how the loss function affects double descent — and thus uncover interesting properties of neural networks and
their
Hessian spectra near the interpolation threshold.
I NTRODUCTION
Double-descent (DD) (Belkin et al., 2019) refers to the phenomenon of population (or test) loss exhibiting a second descent when the model is over-parameterized beyond a certain threshold (dubbed as the interpolation threshold).
While such a behaviour has been originally observed in multiple contexts before (Loog et al., 2020), this terminology has been made commonplace by the influential work of (Belkin et al., 2019), who posited DD as a way to reconcile the traditional statistical wisdom with modern-day usage in machine learning.
In particular, when models such as deep neural networks are employed in the over-parameterized regime, this second descent in test loss is seen as an extension of the usual U-shaped bias-variance curve — thus providing a basis to reason about the amazing generalization abilities of neural networks despite their extreme surplus of parameters.
As a result, there have been a vast number of studies that investigate DD further and others that seek to explain the underlying mechanisms.
The former category of works (Nakkiran et al., 2019) has served to make this phenomenon conspicuous even in the context of frequently employed deep networks and shown the existence of such DD like behaviour based on other axes such as the amount of data or number of epochs (Nakkiran, 2019; Nakkiran et al., 2019); while the latter category of studies (Hastie et al., 2019; Mei & Montanari, 2019; Advani et al., 2020; Bartlett et al., 2020) has provided the theoretical grounds for the occurrence of such a phenomenon, but almost always, in linear or kernel regression.
Although the connection of (infinite-width) neural networks to Neural Tangent Kernel (Jacot et al., 2018) provides interesting parallels, its applicability to finite-width neural networks — i.e., models of actual practical significance — is unclear.
Hence, our aim is to go beyond such informal correspondences, and instead, thoroughly understand and characterize the phenomenology of double descent in finite-width neural networks .
In other words, we develop a theoretical analysis that can explain the source of double descent in the class of models that originally brought this phenomenon to the limelight, as well as study the impact of crucial training-related aspects, such as the choice of the loss function.
To this end, we derive an expression for the population risk by considering the change in the loss at a particular sample — when excluded from the training set and when included.
We do so by utilizing the notion of influence functions from robust statistics, which provides us with a closed-form estimate of the change in parameters.
Subsequently, we establish a lower bound to the population risk and leverage results from Random Matrix Theory to show that it diverges at the interpolation threshold — thereby demonstrating the presence of double descent.
The use of influence functions (Hampel et al., 1986; Efron & Stein, 1981) makes our analysis applicable to a broad class of parametric estimators such as maximum-likelihood-type — which includes the class of neural networks but also applies to other settings such as linear and kernel regression as a special case (see Section A.4).
As a result, this approach allows us to probe the effect of the loss functions used to train neural networks — which, as we will see, clearly influences the nature of DD — and reveals novel insights into the properties of neural networks near the interpolation threshold.
Contributions.
(a) Section 3: We derive a generalization expression based on a ‘add-one-in’ (akin to ‘leave-one-out’) procedure that requires minimal assumptions and, in principle, allows us to analyze the population risk of any neural network or linear/kernel regression model.
(b) Section 4: We show how this yields a suitable asymptotic lower-bound, which diverges at the interpolation threshold and helps explain double descent for neural networks via the Hessian spectra at the optimum.
We additionally support the theoretical arguments by an empirical investigation of the involved quantities, which also suggests its applicability in the non-asymptotic setting. (c)
Section 5: We thoroughly study the effect of the loss function by exploring in detail the double descent behaviour for cross-entropy loss, such as the location of the interpolation threshold, Hessian spectra near interpolation, as well as discuss novel insights implied by these observations.
(d) Section 6: Lastly, as a by-product of our influence functions based approach, we derive a generalized closed-form expression for leave-one-out loss that applies to finite-width neural networks.
S ETUP AND BACKGROUND
Let us consider the general setting of statistical estimation.
Assume that the input samples z ∈ Z are drawn i.i.d.
from some (unknown) distribution D .
A parametric model can be defined as a family of probability distributions D θθθ on the sample space Z , where θθθ ∈ ΘΘΘ .
The main task is to find an estimate (cid:98) θθθ of the parameters such that “ D (cid:98) θθθ ≈ D ”, but given a finite set of samples
S = { z 1 , · · · , z n } drawn i.i.d from D .
The parameter estimator, (cid:98) θθθ , is considered to be provided by some statistic T ( z 1 , · · · , z n ) .
Alternatively, if we denote the empirical distribution of the dataset S by D n , then the parameter estimator can be written as a functional, (cid:98) θθθ D n := (cid:98) θθθ
( D n ) =
T ( D n ) .
2. I NFLUENCE F UNCTIONS : A QUICK PRIMER
Quite often we are interested in analyzing how a slight contamination in the distribution affects, say, the estimated parameters (cid:98) θθθ , or some other statistic T .
Specifically, when this contamination can be expressed in the form of a Dirac distribution δ z , the (standardized) change in the statistic T can be obtained via the concept of influence functions.
Definition 1.
(Hampel et al.
The influence function IF of a statistic T based on some distribution D , is given by,
[Equation]
, evaluated on a point z ∈ Z where such a limit exists.
Essentially, the influence function (IF) involves a directional derivative of T at D along the direction of δ z .
More generally, one can view influence functions from the perspective of a Taylor series expansion (or what is referred to in this context as the first-order von Mises expansion) of T at D , evaluated on some distribution (cid:101) D close to D
[Equation]
So, as evident from this, it is also possible to utilize higher-order influence functions as considered in Debruyne et al.
However, the first-order term is usually the dominating term and to ensure tractability when we later consider neural networks — we will restrict our attention to firstorder influence functions hereafter.
Influence functions have a long history as an important tool, particularly in robust statistics (Hampel et al., 1986), but lately also in deep learning (Koh & Liang, 2017).
Not only do they have useful properties (see Appendix B.1 for a detailed background), but they form a natural tool to analyze changes in distribution (e.g., leave-one-out estimation).
2. I NFLUENCE FUNCTION FOR M AXIMUM LIKELIHOOD TYPE E STIMATORS
In general, we do not explicitly have the analytic form of the estimator (cid:98) θθθ — but implicitly as the solution to an optimization problem (e.g., parameters of neural networks obtained via training).
So, let us consider influence functions for the class of ‘maximum likelihood type’ estimators (or Mestimators)
, i.e. (cid:98) θθθ satisfying the implicit equation, E z ∼D (cid:104) ψ ( z ; (cid:98) θθθ ) (cid:105)
= 0 .
Let us assume that the function ψ ( z ; θθθ ) := ∇ θθθ ℓ ( z ,θθθ ) , where the (loss) function ℓ : Z × ΘΘΘ (cid:55)→ R is twice-differentiable in the parameters θθθ .
When the loss ℓ is the negative log-likelihood, we recover the usual maximum likelihood estimator, (cid:98) θθθ = arg min θθθ ∈ ΘΘΘ E z ∼D [ ℓ ( z ,θθθ )] .
The following proposition describes the influence function for such an estimator (cid:98) θθθ (all the omitted proofs can be found in Appendix A).
Proposition 2.
The (first-order) influence function IF of the M-estimator (cid:98) θθθ D based on the distribution D , evaluated at a point z , takes the following form: [Equation] ( (cid:98)
[Equation]
) (cid:105)
[Equation] ∇ [Equation] ℓ ( z , (cid:98) θθθ D ) , where, the Hessian matrix H L ( (cid:98) θθθ D ) := ∇ 2 θθθ L ( (cid:98) θθθ D ) contains the second derivative of the loss L ( θθθ ) :
= E z
∼D [ ℓ ( z ,θθθ )] with respect to the parameters θθθ .
Remark.
The Hessian in neural networks is typically rank deficient (Sagun et al., 2017; Singh et al., 2021), so for the sake of our analysis we will consider an additive regularization term
λ
I , λ > 0 , alongside the H L ( (cid:98) θθθ D ) term in the above-mentioned IF formula and call this modification IF λ .
Later, we will take the limit λ → 0 .
E XPRESSION OF THE POPULATION RISK
As we lack access to the true distribution, we usually take the route of empirical risk minimization and consider (cid:98) θθθ S :=
arg min θθθ ∈ ΘΘΘ L S ( θθθ ) with L S ( θθθ ) :
= 1 n (cid:80) ni =1 ℓ ( z i ,θθθ ) and recall, S =
{ z i } ni =1 is the training set of size n .
However, in regards to performance, our actual concern is the population risk L ( θθθ ) :
= E z ∼ D
[ ℓ ( z ,θθθ )] , which is measured empirically via the loss L S ′ ( θθθ ) on some unseen (test) set S ′ .
To analyze double descent (Belkin et al., 2019) — i.e., with increasing model capacity, the population risk exhibits a peak before descending again (besides the usual first descent) — we first derive a suitable expression of the population risk that will also apply to neural networks.
‘Add-one-in’ procedure.
Let us consider how the parameter estimate (cid:98) θθθ S changes when an additional sample z ′ ∼ D is included in the training set S .
We can cast this as a contamination of the corresponding empirical distribution D n to yield an altered distribution D ′ n +1 =
(1 − ϵ )
D n + ϵ δ z ′ , with the contamination amount ϵ = 1 n +1 .
Thanks to the influence function of the parameter estimate (Proposition 2 for D = D n ), we do not have to retrain and explicitly measure the change in final parameters, but have an analytic expression given as follows, IF λ ( z ′ ; (cid:98) θθθ S , D n )
= − (cid:104)
H S L ( (cid:98) θθθ S )
+ λ I (cid:105) − 1 ∇ θθθ ℓ ( z ′ , (cid:98) θθθ S ) , where the superscript S in H S L denotes the computation of the
Hessian on the set S .
Now using the chain rule of influence functions, we can express the influence on ℓ z ′ :=
ℓ ( z ′ ,θθθ ) as, IF λ [Equation]
θθθ S (cid:1) .
When | S | = n is large enough to ignore O ( n − 2 ) terms, the (infinitesimal) definition of influence function is equivalent to using the finite-difference form.
This allows us to directly express the change in loss which we leverage to derive an expression of the test loss, in Theorem 3 below.
Theorem 3.
Consider the parameter estimator (cid:98) θθθ S based on the set of input samples S of | S | = n .
Then the population risk L ( (cid:98) θθθ S ) :
= E z ∼ D (cid:104) ℓ ( z , (cid:98) θθθ S ) (cid:105) ) takes the following form,
[Equation]
(1) where (cid:101) L S ( (cid:98) θθθ S ) :
= E z ′ ∼D
[Equation]
ℓ [Equation] z ′ , [Equation]
θθθ S ∪{ z ′ } (cid:17)(cid:105) is the expectation of ‘one-sample training loss’ and
C DL ( (cid:98) [Equation]
[Equation] E
[Equation] ∼D [Equation] ℓ
[Equation] ) ⊤ [Equation] is the (uncentered) covariance of loss gradients.
Remark.
Note, the first term in the right-hand side of eq.
1 is not exactly the training loss, but deviates by a negligible quantity related to the expected difference between the loss of a training sample and the average loss on the remaining training set (as discussed in Section A.1.2).
When trained sufficiently long so that the loss on individual samples is close to zero (i.e., the interpolation setting), this quantity becomes inconsequential for the purpose of analyzing double descent.
Related work.
The more interesting quantity in eq.
1 is the second term on the right, which is reminiscent of the Takeuchi Information Criterion (Takeuchi, 1976; Stone, 1977) and a similar term appears in several works on neural networks (Murata et al., 1994; Thomas et al., 2019) as well as in the analyses of least-squares regression (Flammarion & Bach, 2015; D´efossez & Bach, 2015;
Pillaud-Vivien et al., 2018).
However, an important difference is that in our expression the Hessian
H S L evaluated on the training set appears, whereas it is based on the entire true data distribution in prior works.
This seemingly minor difference is in fact crucial, since studying double descent necessarily involves analyzing the relation between the training set size and the model capacity (such as the number of parameters).
Also, directly taking the results from previous works and merely approximating the involved quantities based on the training set, such as the Hessian and the covariance C DL , does not work either.
Since in such a scenario, when the limit of the regularization strength λ → 0 , results from prior works reduce to something ineffective for further analysis.
E.g., for mean-squared loss, to the rank of Hessian at the optimum (see Section A.2.2 for more details).
L OWER BOUND ON THE POPULATION RISK
As a brief outline, our strategy to theoretically illustrate the double descent behaviour will be to show that the lower bound of the population risk diverges around a certain threshold.
Before proceeding further, we would like to emphasize that so far we have not employed any assumptions on the structural form of the model or the neural network , as well as neither on the data distribution.
So let us now introduce some relevant notations to describe the precise setting for our upcoming result.
We assume that the samples z are tuples ( x , y ) , where the input x ∈ R d has dimension d and the targets y ∈ R K are of dimension K .
Let us consider the neural network function is f θθθ ( x ) :
= f ( x ,θθθ ) : R d × R p (cid:55)→ R K , where we have taken θθθ ∈ R p .
The parameter estimator, (cid:98) θθθ S , in this context refers to the parameters obtained by training to convergence using the training set S , and which we will henceforth denote by θθθ ⋆ := (cid:98) θθθ S (to emphasize the fact that we are at the local optimum).
Also, the loss function ℓ ( z ,θθθ ) , with a slight abuse of notation, refers to ℓ (cid:0)
f θθθ ( x ) , y (cid:1) in this setting.
Let us additionally define a shorthand ℓ i :=
ℓ ( f θθθ ( x i ) , y i ) .
Next, let us discuss in more detail the two matrices that appear in Theorem 3, before we introduce the assumptions we require.
First, the Hessian matrix of the loss , can in general be decomposed as a sum of two other matrices (Schraudolph, 2002): outer-product Hessian H
So ( θθθ ) and functional Hessian H Sf ( θθθ ) , i.e.,
[Equation]
[Equation]
where, ∇ θθθ f θθθ ∈ R p × K is the Jacobian of the function and ∇ 2 f ℓ ∈ R K × K is the Hessian of the loss with respect to the function.
Next, the other matrix that appears in eq.
1 is the (uncentered) covariance of loss gradients , C DL ( θθθ ) =
E ( x , y ) ∼D (cid:104) ∇ θθθ ℓ (cid:0)
f θθθ ( x ) , y (cid:1) ∇ θθθ ℓ (cid:0)
f θθθ ( x ) , y (cid:1) ⊤ (cid:105) , with ∇ θθθ ℓ (cid:0)
f θθθ ( x ) , y (cid:1) =
∇ θθθ
f θθθ ( x ) ∇ f ℓ (cid:0) f θθθ ( x ) , y (cid:1) where ∇ f ℓ ∈ R K is the gradient of the loss with respect to the function.
Similar to the covariance of loss gradients, let us define the covariance of function Jacobians, which we will denote by C S f when computed over the samples in set S and which can be expressed as C S f ( θθθ )
= 1 | S | Z S ( θθθ ) Z S ( θθθ ) ⊤ , with Z S :=
[ ∇ θθθ f θθθ ( x 1 ) · · · ∇ θθθ f θθθ ( x | S | )]
⊤ ∈ R p × K | S | .
4. L OWER B OUND
We employ the following assumption to obtain an appropriate lower bound of the population risk:
Assumption A1.
There exists a sample ( x , y ) ∼ D with non-zero probability α such that σ 2( x , y ) :
= ∥∇ f ℓ ( f θθθ ⋆ ( x ) , y ) ∥ 2 > 0 .
Finally, we are in a position to state our main theorem (all the proofs are in Appendix A).
Theorem 4.
Under the assumption A1 and taking the limit of external regularization λ → 0 and
K = 1 , we obtain the following lower bound on the population risk (eq. 1) at the minimum θθθ ⋆
[Equation]
where we consider the convention λ 1 ≥ · · · ≥ λ r for the eigenvalues with r := rank( H S L ( θθθ ⋆ )) , and σ 2 min denotes the minimum σ 2( x , y )
= ∥∇ f ℓ (cid:0) f θθθ ⋆ ( x ) , y (cid:1) ∥ 2 over (cid:101) D , i.e., ( x , y ) ∼ D : σ 2( x , y ) > 0 .
The key takeaway of this theorem is that the lower bound on the population risk is inversely proportional to the minimum non-zero eigenvalue λ r of the Hessian at the optimum H S L ( θθθ ⋆ ) , which — as we will see shortly — largely characterizes the double descent like the behaviour of the population risk.
Besides, we would like to emphasize that the primary purpose of the lower bounds is to isolate the source of double descent, and thus the practical applicability of the lower bounds — which is in itself an open research area — is of secondary concern.
As a result, in our analysis we lower bound the quantity r (cid:80) i =
1 λ r ( H S L ( θθθ ⋆ ) ) by 1 λ r ( H S L ( θθθ ⋆ ) ) , although it might be of interest to keep the original quantity in a different context.
Remark.
The assumption A1 just requires the existence of such a point from the true distribution
D with non-zero probability.
Notice, otherwise, we would have zero population risk, which is obviously of no interest.
The benefit of this assumption is that we can analyze C (cid:101)
D f ( θθθ ⋆ ) instead of
C DL ( θθθ ⋆ ) by taking the minimum non-zero σ 2 min outside of the corresponding expression.
4. D OUBLE DESCENT BEHAVIOUR
Having established this lower bound, we will utilize it to demonstrate the existence of the double descent behaviour.
Let us consider the case of mean-squared error (MSE) loss, ℓ ( f θθθ ( x ) , y )
= 12 ∥
y − f θθθ ( x ) ∥ 2 , and where the Hessian of the loss with respect to the function is just the identity, i.e., ∇ 2 f ℓ = I .
We will employ the following additional assumptions:
Assumption A2.
The functional Hessian at the optimum θθθ ⋆ is zero, i.e., H Sf ( θθθ ⋆ ) = 0 .
Assumption A3.
The minimum non-zero eigenvalue λ min of covariance of function Jacobians at optimum θθθ ⋆ is bounded by the corresponding one at initialization θθθ 0 over some common set S , i.e.,
A λ min ( C S f ( θθθ 0 )) ≤ λ min ( C S f ( θθθ ⋆ ))
≤ B λ min ( C S f ( θθθ 0 )) , with constants 0 < A, B < ∞ .
Assumption A4.
The columns of Z S ( θθθ 0 ) are sub-Gaussian independent random vectors at initialization θθθ 0 .
Note on the assumptions.
Assumption A2 is known from prior works (Sagun et al., 2017; Singh et al., 2021) to hold empirically —
in particular, c.f. Figures 5, S2 of Singh et al.
(2021) where it is shown that the rank of the functional Hessian converges to 0 when trained sufficiently (besides, c.f. Appendix C.8).
Also, in the setting of double descent, the individual losses (and their gradients) are themselves close to zero near the interpolation threshold, thereby making the functional Hessian vanish at the optimum (recall eq. 2).
Next, the assumption A3 essentially guarantees that, at the optimum, the minimum non-zero eigenvalue of the covariance of function gradients does not change much relative to that at initialization.
Importantly, this is purely for the purposes of the lower bound — not something that we impose as a constraint during training (vis-`a-vis the NTK regime).
The existence of the constants mentioned in this assumption A3 can be theoretically justified by the fact that the map A (cid:55)→ λ
i ( A ) is Lipschitz-continuous on the space of Hermitian matrices, which follows from Weyl’s inequality (Tao, 2012, p. 56).
Besides, in Figure 2a, we empirically justify this assumption, and in the adjoining Figure 2b we also validate our lower bound to the population risk throughout the double descent curve (additional details of which can be found in Appendix C.10).
The last assumption may seem more demanding but is mild in comparison to that in prior work (Pennington & Bahri, 2017), where all entries of the covariance of function gradients at the optimum are considered to be independent and identically distributed.
In contrast, our assumption just requires sub-gaussianity only at initialization .
Similar sub-gaussianity assumptions are also common in the regression-based analyses of double descent (Muthukumar et al., 2019; Bartlett et al., 2020).
The benefit of such an assumption is that it allows us to precisely characterize the behaviour of the minimum eigenvalue that appears in Theorem 4 using results from Random Matrix Theory (Vershynin, 2010), and thereby that of double descent, as described in the upcoming Theorem 5.
Lastly, before proceeding ahead, let us mention that the Appendix A.4 discusses concrete settings where all the above assumptions hold simultaneously for finite-width neural networks.
Theorem 5.
For the MSE loss, under the setting of Theorem 4 and the assumptions A2, A3, A4, the population risk takes the following form and diverges almost surely to ∞ at p =
[Equation]
, in the asymptotic regime of p, n → ∞ but their ratio is a fixed constant, and where c > 0 is a constant that depends only on the sub-Gaussian norm of columns of Z S ( θθθ 0 ) .
Takeaways.
(a) Firstly, the point where the second descent occurs, i.e, the interpolation threshold, is typically p ≈ n in the setting of K = 1 .
So our result from Theorem 5 (which is in this setting) not only implies a divergence at the interpolation threshold, but further illustrates that the complexity term will get smaller as its denominator increases for p >> n , — thereby also capturing the overall trend of population risk.
(b) Second, while the above result holds in the asymptotic setting, we empirically show that such a behaviour also takes place for p and n as small as a few hundreds or thousands, as discussed in Section 4.3 ahead.
Interpretation of the interpolation threshold location.
An interesting empirical observation, very briefly alluded to in prior works (Belkin et al., 2019; Greydanus, 2020), is that for the MSE loss with K targets, the interpolation threshold is instead located at p
≈ Kn .
This can be reconciled by looking the rank r of the Hessian at the optimum, as inherently the divergence at the interpolation threshold is because the Hessian’s minimum non-zero eigenvalue λ r vanishes.
More intuitively, double descent occurs at the transition when the Hessian rank starts being dictated by the # of samples (i.e., when over-parameterized) rather than being governed by the # of parameters (i.e., when under-parameterized), e.g., for MSE when p ≈ Kn and note Kn is precisely the rank of the
Hessian in the over-parameterized regime ( Kn > p ).
As a matter of fact, in practice, there might be redundancies due to either duplicates or linearly dependent features/samples, as well as parameters.
For instance, in Figure 15 we show a simple example of linear regression where the interpolation threshold can be moved arbitrarily by changing the extent of redundancy in the design matrix.
But, as demonstrated therein, thinking in terms of the Hessian rank can help avoid such inconsistencies.
Other facets of double descent.
(i)
Our analysis additionally explains the empirical observation of label noise accentuating the peak (Nakkiran et al., 2019), since the complexity term contains a multiplicative factor of σ 2 min which increases proportionately with label noise.
(ii)
The exact term that appears in our proof, before we take the limit of regularization λ → 0 , is (cid:0) λ r
( H S L ( θθθ ⋆ ))
+ λ (cid:1) − 1 .
This reveals why, when using a regularization of a suitable magnitude, double descent is not prominent or disappears, as also noted in (Nakkiran et al., 2019; 2020), — since this term can no longer explode.
4. E MPIRICAL VERIFICATION
[Figure]
[Figure] [Figure] To empirically demonstrate the validity of our theoretical results, we carry out the entire procedure of obtaining double descent for neural networks.
Namely, this involves training a large set of neural networks — with layer widths sampled in regular intervals — for sufficiently many number of epochs.
However, in our case, there is another factor which makes this whole process even more arduous — the lower bound depends on the minimum non-zero eigenvalue of the Hessian.
As a result, we cannot resort to efficient Hessian approximations based on, say, Hessian-vector products (Pearlmutter, 1994), but rather we need to compute entire Hessian spectrum — which has
O ( p 3 ) computational and memory costs.
Hence, we cannot but restrict our empirical investigation to smaller network sizes.
Nevertheless, on the positive side we can thoroughly assure the accuracy of our empirical investigations — since we compute the exact Hessian and its spectrum, and that too in F LOAT 64 precision.
In terms of the dataset, we primarily utilize MNIST1D (Greydanus, 2020), which is a downscaled version of MNIST yet designed to be significantly harder than the usual version.
However, we also present results on CIFAR10 and the usual (easier) MNIST, which alongside other empirical details, can be found in Appendix C. Figure 1 shows the results of running the double descent experiments for the settings of two and three layer fully-connected networks with ReLU activation trained for 5K epochs via SGD.
Alongside the population loss (empirically measured on a test set) — which peaks at the interpolation threshold of p ≈ Kn — we plot the trend of the minimum non-zero eigenvalue.
As predicted by our theory, this eigenvalue indeed tends to zero (note the log scale), in both the settings, around the precise neighborhood where the population loss takes its maximum value.
[Figure] [Figure]
3000 # of parameters 0.
L o ss ( l o g s c a l e )
L o w e r b o un d
( l o g s c a l e )
[Figure]
T HE C ASE OF C ROSS -E NTROPY
Greydanus (2020) notes that in the case of cross-entropy, the population loss peaks at p ≈ n (unlike at p
≈ Kn for the MSE loss).
To better understand this aspect, let us start by checking the form of the Hessian at the optimum for the cross-entropy loss, which by our assumption A2, will be given by the outer-product term.
as in eq. 2.
Now, the main difference is that the Hessian of the loss with respect to the function (named ‘output-Hessian’), instead of being identity like in MSE, is given by ∇ 2 f ℓ ( f θθθ ( x ) , y )
=
diag( p ) − pp ⊤ , where p = softmax( f θθθ ( x )) denotes the predicted classprobabilities obtained from applying the softmax operation.
In general, this output-Hessian matrix of size K × K , is rank-deficient with rank K − 1 .
Thus, the rank( H So ( θθθ )) =
min ( p, ( K − 1) n ) for any θθθ , and the initial surmise would be that the interpolation threshold is at p
≈ ( K − 1) n .
But, this is not in line with the stated observation from Greydanus (2020).
Hence, let us take another closer look at the Hessian, and in particular, ∇ 2 f ℓ .
Notice that near interpolation, when the loss on individual training samples tends to zero, the predicated class-probability p will tend to 1 for the correct class (as per the training label) and 0 elsewhere.
This suggests that Hessian matrix collapses to 0 since ∇ 2 f ℓ ( f θθθ ( x ) , y )
=
diag( p ) − pp ⊤ → 0 .
And indeed, this is true based on our empirical results, displayed in Figure 3 given the network is trained sufficiently long.
We clearly observe that the population loss diverges at p ≈ n , however the test error, although not as conspicuous, still shows a slight peak (similar to the curves in (Nakkiran et al., 2019) without label-noise).
Next, from the right sub-figure (plotted in log-scale), it also becomes evident that the entire Hessian spectrum — from the maximum to the minimum eigenvalue — collapses to zero near the interpolation threshold.
Fact 6.
Thus, we have that for the cross-entropy loss, near the interpolation threshold of p ≈ n , the entire Hessian matrix vanishes at the optimum, i.e., H S L ( θθθ ⋆ )
= 0 .
5000 # of parameters
P o p u l a t i o n
L o ss
E rr o r
[Figure]
interpolation threshold (best) Test error
Train error     5000 # of parameters
P o p u l a t i o n
L o ss ( l o g )
T r a i n L
o ss
a n d E i g e n v a l u e s ( l o g )
[Figure]
loss
[Figure]
1 ( H S ) r
( H S )
Importantly, the above fact (and our empirical results) report such a behaviour only around p ≈ n , and not well into the over-parameterized regime p >> n .
Therefore, a straightforward consequence of the above fact and empirical observations is that λ r ( H S L ( θθθ ⋆ ))
= 0 near this interpolation threshold, implying that the population loss in case of cross-entropy loss diverges at p ≈ n .
Vanishing Hessian hypothesis.
Figure 3 right, seems to suggest, rather surprisingly, that even the training loss has the same trend as these Hessian eigenvalue statistics.
Note, this is not the case that networks were not trained long enough — rather, we run them for 40 , 000 epochs.
In fact, we even trained models, right to the interpolation threshold, up to 80 , 000 epochs.
Yet, the training loss only drops by factor of 2 and is still > 1 e − 5 , compared to exact 0 at machine precision in half the epochs for networks lying in the region where the test loss diverges.
This leads us to posit the following hypothesis: around p ≈ n , SGD finds critical points with zero Hessian for neural networks trained with CE loss and over-parameterization beyond the interpolation threshold helps completely avoid or significantly decelerate convergence to such critical points.
Further investigation into this hypothesis is beyond the current scope, but forms an exciting direction for future work.
L EAVE - ONE - OUT ESTIMATE VIA INFLUENCE FUNCTIONS
Leave-one-out (LOO) is also known to provide a reasonable estimate of the population loss (Pontil, 2002). The principle behind it is to leave behind one sample, optimize the model from scratch on the remaining n − 1 samples, then evaluate the loss on the left-out sample, and finally average over the choice of the left-out sample.
In our setting for double descent, we prefer the add-one-in procedure since it directly gives the population loss itself.
However, LOO can still be useful from a practical perspective as it relies only on the training set — assuming we can analytically estimate it via some closed-form expression which avoids the need to train n models in the otherwise naive manner.
Similar to the add-one-in procedure from before, leave-one-out can be cast as a slight contamination of the distribution.
We can express the new distribution D \ in − 1 with the i -th sample removed as follows, D \ i n − 1 = (1 − ϵ )
D n + ϵ δ z i with ϵ = − 1 n − 1 , where D n refers to the original empirical distribution over the training set.
Now, we can carry out similar steps like for add-one-in, and derive the change in the parameter estimate as well as the change in loss over the left-out sample.
But, here we additionally analyze the effect of incorporating the second-order influence function , apart from the usual first-order influences.
This provides us with estimates LOO (1) and LOO (2) , the expressions of which can be found in Appendix A.6.1.
As a quick test-bed, we investigate the fidelity of these two approaches relative to the exact formula LOO LS in the case of least-squares.
Theorem 7.
Consider the particular case of the ordinary-least squares (OLS) with the training inputs gathered into the data matrix X ∈ R n × d and the targets collected in the vector y ∈ R n .
Under the assumption that the number of samples n is large enough, we have that LOO (2) =
LOO LS = 1 n n (cid:80) i = (cid:32) y
i − θθθ ⊤ x i 1 − A ii (cid:33), and LOO (1) = 1 n n (cid:80) i = (cid:16)
y i − θθθ ⊤ x
i (cid:17)
A ii 1 − A ii .
where, A ii = x
⊤ i ( X ⊤ X ) − 1 x i denotes the i -th diagonal entry of the matrix A
= X
( X ⊤ X ) − 1 X ⊤ (i.e., the so-called ‘hat-matrix’) and θθθ denotes the usual solution of ( X ⊤ X ) − 1 X ⊤ y obtained via OLS.
This result is reassuring as it shows that LOO expressions from influence function analysis are accurate, and we recover the least-squares formula as a special case through LOO (2) .
Further,
Corollary 8.
For any finite-width neural network, the first and second-order influence function give similar formulas for LOO like that in Theorem 7, but with A =
Z S ( θθθ ⋆ ) ⊤ (cid:0) Z S ( θθθ ⋆ ) Z S ( θθθ ⋆ ) ⊤ (cid:1) − 1 Z S ( θθθ ⋆ ) , where Z S ( θθθ ⋆ ) :
=
[ ∇ θθθ f θθθ ⋆ ( x 1 ) , · · · , ∇ θθθ f θθθ ⋆ ( x n )] and θ ⋆ are the parameters at convergence for MSE loss.
Finally, the above result raises a concern about the sub-optimality of first-order influence functions when used in the leave-one-out framework.
While this is not necessarily a significant concern for a theoretical analysis, say that of double descent, this can be relevant from a practical viewpoint (Basu et al., 2020).
However, an empirical investigation on this front remains beyond the current scope.
D ISCUSSION
Summary.
We derived an expression of the population risk via influence functions and obtained a lower bound to it — with fairly minimal assumptions — that applies to any finite-width neural network trained with commonly used loss functions.
The lower bound is inversely related to the smallest non-zero eigenvalue of the Hessian of the loss at the optimum.
When specialized to the MSE loss, this provably exhibits a double descent behaviour in the asymptotic regime and we empirically demonstrated that this holds even in much smaller non-asymptotic settings.
We also analyzed the intriguing phenomenology of double descent across other losses — through our Hessian-based framework — which explained existing empirical observations as well as uncovered novel aspects of neural networks near interpolation.
Finally, as a by-product, we presented theoretical results for leave-one-out estimation using influence functions in the case of neural networks.
Related theoretical work on Double Descent.
We carve out a niche in the growing set of studies on double descent by focusing primarily on — finite-width neural networks .
For the linear/kernel regression setting (or lately, the nearly equivalent two-layer network with frozen hidden-layer), there is a plethora of existing work (Advani et al., 2020; Bartlett et al., 2020; Mei & Montanari, 2019;
Muthukumar et al., 2019; Geiger et al., 2020; Ba et al., 2020), that analyzes double descent.
Thus, our aim is not to make these existing analyses tighter, but rather to take a step towards developing analyses that directly hold for finite-width neural networks.
Therefore, unlike the above works, we do not impose any restrictive assumptions on the structure of neural network, like two-layer networks, or the optimization methods used to train them, like gradient flow.
Yet, our work also bears a natural connection between the matrix whose spectrum comes to be of concern in the prior works — the input-covariance or kernel matrix in linear or kernel regression — while that of the outer-product Hessian in our work for MSE loss (which has the same spectrum as the ‘empirical’
NTK).
But our strategy also makes our work applicable to cross-entropy, e.g., where we bring to light the interesting observations near the interpolation threshold.
A closely related recent work, Kuzborskij et al.
(2021), links the population risk for least-squares to the minimum non-zero eigenvalue of the input covariance matrix — but although via an upperbound, which is insufficient for illustrating double descent.
Nevertheless, in analogy to their regression result, they study the minimum eigenvalue of the covariance matrix consisting of penultimatelayer features and conjecture this as a possible extension to neural networks.
However, the inputcovariance matrix in least-squares is also the Hessian, and as we have thoroughly established the
Hessian (at the optimum) is indeed the relevant object — thus contradicting their conjecture.
Limitations and directions for future work.
There are many important aspects surrounding double descent that remain unanswered, in the context of neural networks: (a) Analogous to (Hastie et al., 2019) for linear regression, what are the conditions for the global optimum to lie in the overparameterized regime instead of under-parameterized?
(b) Given the vanishing Hessian hypothesis, is there a qualification to the regime where the flat-minima generalizes better hypothesis (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016) holds — since the Hessian is the flattest possible here.
(d) On the technical side: better characterization of the mentioned cross-entropy phenomenon as well as non-asymptotic results.
Overall, we hope that our work will encourage foray into further studies of double descent, that are specifically built for finite-width neural networks.
R EPRODUCIBILITY STATEMENT
• All the omitted proofs to the theoretical results can be found in the Appendix A. • In regards to empirical results, we provide all the relevant details and additional results in the Appendix C. • The corresponding code for the experiments is located at https://github.com/sidak/DD.
A CKNOWLEDGEMENTS
We would like to thank Simon Buchholz for proofreading an early draft of the paper.
Besides, we thank the members of DA lab for useful comments.
Sidak Pal Singh would also like to acknowledge the financial support from Max Planck ETH Center for Learning Systems and the travel support from ELISE (GA no 951847)
A O MITTED P ROOFS
Proposition 2.
The influence function IF of the M-estimator (cid:98) θθθ D based on the distribution D , evaluated at point z , takes the following form:
[Equation]
where, the Hessian matrix H L ( (cid:98) θθθ D ) :
= ∇ 2 θθθ L ( (cid:98) θθθ D ) is the matrix of second-derivatives of the loss
L ( θθθ ) :
= E z
∼D [ ℓ ( z ,θθθ )] with respect to the parameters θθθ .
Proof.
Let (cid:98) θθθ D = arg min θθθ ∈ ΘΘΘ
E z ∼D [ ℓ ( z ,θθθ )] .
Instead of this formulation, we can also define the estimator as one that satisfies the following implicit equation (assuming the derivative can be moved inside expectation),
[Equation]
which is nothing but the first-order stationary point condition.
Now, consider a contaminated distribution (cid:101) D :
=
(1 − ϵ ) D
+ ϵ δ z .
The corresponding implicit equation, to be satisfied by the estimator (cid:98) θθθ (cid:101) D corresponding to (cid:101) D , can be written as follows: [Equation]
[Equation]
Take the derivative of the above expression with respect to ϵ (plus interchanging derivative and expectation), we get:
[Equation]
Since, (cid:98) θθθ (cid:101) D satisfies the above eq.
[Equation] let us substitute it in place of θθθ and analyze the case for ϵ → [Equation]
We are left with the following (after removing terms multiplied with ϵ ):
[Equation]
Now, the first term goes to zero as ϵ →
[Equation]
because eq. 6 holds for (cid:98) θθθ (cid:101) D , as shown below :
[Equation]
Further, as ϵ → 0 , one can replace (cid:98) θθθ (cid:101) D by (cid:98) Then, assuming that the Hessian H L (
[Equation]
) := [Equation]
∼D [ [Equation]
=
[Equation]
∼D
[Equation] ∇ [Equation] θθθ ℓ ( z , (cid:98) θθθ D ) (cid:105) is invertible, this yields:
[Equation]
A. E XPRESSION OF THE POPULATION RISK
Theorem 3.
Consider the parameter estimator (cid:98) θθθ S based on the set of input samples S of | S | = n .
Then the population risk L ( (cid:98) θθθ S ) :
= E z ∼ D (cid:104) ℓ ( z , (cid:98) θθθ S ) (cid:105) ) takes the following form,
[Equation]
) where (cid:101) L S ( (cid:98) θθθ S ) :=
E z ′
∼D (cid:104) ℓ (cid:16) z ′ , (cid:98)
θθθ S ∪{ z ′ } (cid:17)(cid:105) denotes the expectation of ‘one-sample training loss’ and C DL
( (cid:98) θθθ S ) :
= E z ′
∼D (cid:20)
[Equation]
( z ′ , (cid:98)
[Equation] [Equation]
ℓ ( z
[Equation]
, (cid:98) θθθ S ) ⊤ (cid:21) is the (uncentered) covariance of loss gradients.
Proof.
Let us recall the expression of influence function for the loss on the new sample z ′ ,
IF
[Equation]
When | S | = n is large enough to ignore O ( n − 2 ) terms, the (infinitesimal) definition of influence function is equivalent to using the finite-difference form.
Then the change in loss can be expressed as,
[Equation]
where we have multiplied both sides by ∆ ϵ = ϵ −
= 1 n +1 .
We leverage this relation to derive an expression of the test loss as follows.
Starting from eq.
12, we average out over the choice of an additional sample z ′ ∼ D :
[Equation]
Using properties of the trace and moving expectation inside
[Equation]
we get
E
[Equation]
where, C DL ( θθθ ) :=
E z ′
∼D (cid:104) ∇ θθθ ℓ ( z ′ ,θθθ ) ∇ θθθ ℓ ( z ′ ,θθθ ) ⊤ (cid:105) .
Now the term on the left-hand side is nothing but the population risk L ( (cid:98) θθθ S ) :
= E z ∼ D (cid:104) ℓ ( z , (cid:98) θθθ S ) (cid:105) ) , and we coin the first term on the right-hand side as the expectation of ‘one-sample training loss’.
A.1. R EMARKS ON THE FIRST - ORDER INFLUENCE FUNCTION USAGE
Let us better understand when using the first-order influence function suffices by analyzing the respective expression for the change in loss.
First, let us apply the influence function of the parameter estimator from eq.
[Equation]
(14) for the add-one-in case of sample z ′ to the training set S discussed above.
Next, we can substitute ϵ = 1 n +1 , and thereby obtain the change in parameters ∆ (cid:98) θθθ as:
[Equation] [Equation]
Then, via first-order influences, we get the change in loss over the sample z ′ as:
[Equation]
In contrast for the second-order influence
[Equation]
we obtain the change in loss over the sample z ′ as:
[Equation]
Later on, we take the expectation over the distribution, i.e., z ′ ∼ D , but it is not relevant for analyzing the scale of the above mentioned change in loss obtained via first or second order influences.
Note, the Hessian is itself O (1) in terms of number of samples n , as it is an average of the persample Hessians.
Also, the numerator in both the above equations is a trace of a p × p matrix and, overall, the numerator scales as O ( p ) .
Whereas, if we look at the denominator, the extra term in ∆ ℓ (2) scales as O ( n − 2 ) , while the common term in ∆ ℓ (2) and ∆ ℓ (1) is of O ( n − 1 ) .
Hence, when n is large enough such that n 2 >> p , then the O ( n − 2 ) terms can be ignored, we can simply consider the first-order influences.
A.1. I NTERPRETATION OF THE ‘ ONE - SAMPLE TRAINING LOSS ’
First, note that the first term on the RHS of eq.
13 is not exactly the training loss but rather some related quantity.
To see this better, let us rewrite as follows:
[Equation]
where, ℓ ( S,θθθ ) = (cid:80) | S | i =1 ℓ ( z i ,θθθ ) is the sum of the loss over the samples in S .
Further, the expression ∆ TR ( z ′ , S ∪ { z ′ } ) refers to the deviation between the loss of a training sample (here, z ′ ) relative to the average loss on rest of the training samples.
The ‘TR’ in this symbol refers to this deviation being computed on the given training set.
[Equation]
Notice, the last term in the expression above is the same as the term on the left hand side — albeit with an additional scaling factor of 1 | S | and negative sign in front.
Rearranging this results in the following equation:
[Equation]
The only extra thing we have done is to multiply and divide by | S ∪ { z ′ }| = | S | + 1 in the last term in the right hand side.
Notice the rightmost term is an expectation (over z ′ ) of the average training loss of the training set S ∪ { z ′ } and to which we assign the shorthand TR ( S ∪ { z ′ } ) .
Also, it is evident from here that the ‘one-sample training loss’ is a quantity very much related to the training loss.
Lastly, considering the large | S | = n limit, we have lim n →∞ n +1 n = 1 , and which thereby yields:
[Equation]
A. L OWER BOUND TO THE POPULATION RISK
Lemma 9.
Consider two matrices A ∈ R m × m and B ∈ R m × m , where A is symmetric and B is symmetric and positive semi-definite.
Then the following holds,
[Equation]
Proof.
See Fang et al.
Theorem 4.
Under the assumption A1 and taking the limit of external regularization λ → 0 and
K = 1 , we obtain the following lower bound on the population risk ( eq. 1), at the minimum θθθ ⋆ L ( [Equation]
[Equation]
(cid:101) [Equation] [Equation] where we consider the convention λ 1 ≥ · · · ≥ λ r for the eigenvalues with r := rank( H S L ( θθθ ⋆ )) , and σ 2 min denotes the minimum σ 2( x , y )
=
∥∇ f
[Equation]
., ( x , y )
[Equation]
D : σ 2( x , y ) > 0 .
Proof.
Let us recall the expression for the population risk that we proved in Theorem 3, for (cid:98) θθθ S =
θθθ ⋆ ,
[Equation]
In particular, we would like to analyze the complexity term T :=
Tr (cid:16)(cid:2) H S L
[Equation]
θθθ ⋆ )
+ λ
I
[Equation] − 1 C DL [Equation] θθθ ⋆ ) (cid:17) on the right-hand side and lower bound it.
The full expression of this term T is given by,
[Equation]
Now using the chain rule, we have that ∇ θθθ ℓ ( f θθθ ⋆ ( x ) , y ) =
∇ θθθ
f θθθ ⋆
[Equation] x ) ∇ f ℓ [Equation] f θθθ ⋆ ( x ) , y (cid:1) .
Then, for
K = 1 , the above equation is equivalent to,
[Equation]
Next, we take the lower bound by considering the minimum over all non-zero σ 2( x , y )
=
∥∇ f ℓ (cid:0) f [Equation] ⋆ ( x ) , y (cid:1) ∥ [Equation]
[Equation]
where σ 2 min = min ( x , y ) ∼D : σ 2( x , y ) > 0 σ 2( x , y ) and whose non-zero probability α is guaranteed by the assumption A1.
Finally, in the last line, we use the cyclic property of the trace once again and move the expectation inside the trace, obtaining the covariance of function gradients C (cid:101)
D f ( θθθ ⋆ ) .
Proceeding further, we again make use of Lemma 9 in the eq.
22, since these matrices are also symmetric positive semi-definite.
This yields,
[Equation]
where, the notation λ i ( · ) denotes the i -th eigenvalue of the corresponding matrix and we will use the convention that for some matrix in R m × m ,
[Equation]
Let us suppose r denotes the rank of the Hessian at the optimum, i.e., r := rank( H S L ( θθθ 0 )) .
Since the Hessian is positive semi-definite by the second-order necessary conditions of local minima, all the eigenvalues are non-negative and we can further lower bound the previous expression to T as follows:
[Equation]
where in the second line, we have used the fact that a sum of non-negative numbers can be lower bounded by the maximum summand.
The maximum here will correspond to using the inverse of the minimum non-zero eigenvalue
[Equation]
Now, substituting the following lower bound together with expression of population risk and taking the limit of λ → 0 , finishes the proof.
[Equation]
Note.
For the purposes of this lower bound, we ignored the O (cid:0) 1 n 2 (cid:1) part in Theorem 3.
This is because they take the form, e.g. for second-order influences as shown in Section A.6.1 and A.1.1.
A.2. A NALOGOUS UPPER BOUND
We can in fact derive an upper bound to the population risk by following analogous steps to that in the lower bound.
First, let us recall the expression of the complexity term T ,
[Equation]
The existence of a non-zero residual via the σ 2 min assumption A1 also implies that there exists an analogous σ 2 max , defined as follows:
[Equation]
Thus, we get the following upper bound by also considering the chain rule of ∇ θθθ ℓ ( f θθθ ⋆ ( x ) , y ) =
∇ θθθ
f θθθ ⋆ ( x ) ∇ f ℓ (cid:0) f θθθ ⋆ ( x ) , y (cid:1) and repeating similar steps as before,
[Equation]
[Equation]
Then we can use Lemma 9 as both the matrices inside trace are symmetric positive semi-definite, which gives us our initial upper bound:
[Equation]
While additional steps can be further carried out, depending on the required context, but the objective of this discussion is to show that many of the steps in our lower bound can be likewise generalized to get a corresponding upper bound.
A.2. E MPIRICAL APPROXIMATIONS OF TIC LIKE EXPRESSIONS
Consider the case when both the Hessian and covariance in the complexity term that shows up in our lower-bound expression,
[Equation]
are based/approximated on the training set.
Let us further assume the case of MSE loss and that we are at the optimum, where by A2, H S L ( θθθ ⋆ ) =
C S f
( θ ⋆ ) .
Under these set of assumptions, the term above reduces to,
[Equation]
Since we can always express a positive semi-definite matrix as some ZZ ⊤ .
Let us substitute this in the above expression and consider the limit of regularization λ → 0 .
[Equation]
(30) where in the last line we have used the result from Tikhonov regularization, and Z † denotes the pseudo-inverse of Z .
But this expression is nothing but the rank( Z ) and thus shows the ineffectiveness of such an analysis where both the Hessian and the covariance of function gradients are based on the training set.
A. D OUBLE DESCENT BEHAVIOUR
Theorem 5.
For the MSE loss, under the setting of Theorem 4 and the assumptions A2, A3, A4, the population risk takes the following form and diverges almost surely to ∞ at p = 1 √ c n ,
L ( θθθ
[Equation]
≥
[Equation]
S
[Equation] + [Equation] 2 , in the asymptotic regime of p, n → ∞ but their ratio is a fixed constant, and where c > 0 is a constant that depends only on the sub-Gaussian norm of columns of Z S ( θθθ 0 ) .
Proof.
Let us start from the lower bound shown in Theorem 4.
[Equation]
For the MSE loss, we have the Hessian H S L ( θθθ ⋆ ) =
C S f
( θθθ ⋆ ) , from assumption A2.
We then use the assumption A3 to bound the minimum non-zero eigenvalue (here, λ r ) of C S f
( θ θ θ ⋆ ) to the corresponding minimum non-zero eigenvalue at initialization.
This results in the following lower bound,
[Equation]
Again we utilize the assumption A3 to upper bound the minimum non-zero eigenvalue of C D f ( θθθ ⋆ ) to the corresponding minimum non-zero eigenvalue at initialization, thus obtaining:
[Equation]
Now notice that, via assumption A4, C S f
( θθθ 0 )
= 1 | S | Z S Z ⊤ S is a covariance matrix with the columns of Z S containing independent, sub-gaussian random vectors in R p .
Thus, we can leverage the results of Vershynin (2010) on the extremal eigenvalues of covariance matrices.
Specifically, given a random matrix, Z ∈ R m × n whose columns are isotropic, independent, sub-gaussian random vectors in R m , Vershynin (2010) states that the extremal eigenvalues ofn ZZ ⊤ , in the asymptotic regime where m, n → ∞ but their ratio mn → γ ∈ (0 , 1] , we have:
[Equation]
where c is a constant that depends on the subgaussian norm.
Since the above result holds in the isotropic case, let us first ensure this aspect.
Consider the matrix (cid:101) Z S :
= C D f ( θθθ 0 ) − 12 Z S , which is possible since C D f ( θθθ 0 ) is clearly positive semi-definite and its spectrum being bounded away from zero is a typical assumption, c.f.
Du et al.
; Nguyen et al.
As a result, the columns of this new matrix (cid:101) Z S are isotropic, besides being independent, subgaussian random vectors.
Thus, we apply the above RMT result to the matrix 1 | S | (cid:101) Z S (cid:101) Z ⊤ S , and we then obtain the following relation on λ r (cid:16) C S
f
( θθθ 0 ) (cid:17) :
[Equation]
This is because, s min ( AB ) ≤ ∥ A ∥ 2 s min ( B ) , where s denotes the singular value, and follows from using the definition of spectral norm together with min-max characterization of singular values
(see Theorem 3.3.16 in Horn & Johnson (1991) for more).
Also, we know that
λ
i ( M ⊤ M )
=
λ i ( MM ⊤ )
=
s 2 i ( M ) for some matrix M ∈ R m × n and for 1 ≤ i ≤ min( m, n ) .
Therefore, using this for A := C D f
( θθθ 0 ) 12 , and B := C D f
( θθθ 0 ) − 12 Z S gives the above bound.
Finally, combining all these together yields, [Equation]
[Equation]
Then, by a simple rearrangement and noting that we are in the asymptotic regime where n → ∞ , we recover our desired lower bound, thus finishing the proof.
Remark 1.
In the over-parameterized case, where γ > 1 , the similar procedure follows by using the random matrix theory result on 1 | S | Z ⊤ S Z S and the fact that in our lower bound we anyways have minimum non-zero eigenvalue λ r .
Remark 2.
The ratio of the terms ∥ C D f
( θθθ 0 )
∥ 2 and λ min (cid:16) C (cid:101) D
f ( θθθ 0 ) (cid:17) looks like a condition number, but notice the the minimum eigenvalue is for the covariance over (cid:101) D and not D .
Thus, if we are to write in the form of condition number, the above result will take the form:
[Equation]
A. O NE - HIDDEN LAYER NEURAL NETWORK WITH TRAINED OUTPUT WEIGHTS
In this section, we discuss the concrete case of a one-hidden layer neural network, but where only the output layer weights v are trained (akin to a random feature model).
Also, for simplicity we assume that the input x ∈ R d is sampled from a sub-Gaussian distribution (i.e. a distribution with a tail decay).
Linear case.
Let us begin with the case of a linear neural network, and then we can write the network function as follows,
[Equation]
Notice, since ∂f ∂ v i =
[ W x ] i , the second-derivatives ∂ 2 f ∂ v 2 i = 0 , ∀ i .
Thus, Assumption A2 holds trivially for all parameter configurations.
Next, note that the columns of Z S , consisting of the vectors ∇ θθθ f θθθ ( x ) =
∂f ( x ) ∂ v , are sub-Gaussian random vectors when conditioned on the initialization weights W , thus satisfying assumption A4.
Lastly, since the trainable parameters are only v and the matrix W remains fixed, then the covariance of network Jacobians remains fixed as well during training, i.e., C S f ( θθθ 0 ) =
1 | S | Z S
( θθθ 0 )
Z S ( θθθ 0 ) ⊤ =
1 | S | Z S ( θθθ ⋆ ) Z S ( θθθ ⋆ ) ⊤ =
C S f
( θθθ ⋆ ) .
Hence, assumption A3 holds trivially with equality and both constants equal to 1 .
Non-linear case.
Now, consider the general case where we have an elementwise non-linearity ϕ .
So, the network function can be expressed as:
[Equation]
The functional Hessian is still zero, as the gradient of function with respect to the parameters does not depend on the parameters.
Similarly, the covariance of network Jacobian will remain the same during training, as the matrix W is fixed.
Thus both assumptions A2, A3 hold.
The subgaussian assumption holds as well for a wide set of activation functions, including for instance the ReLU nonlinearity, ϕ ( z )
= max(0 , z ) .
Indeed, each component of the columns of Z S is still independent and random.
The difference relative to the linear case is that we squash to zero the part of the distribution which is in the second quadrant, which gives a sub-Gaussian distribution as its tail is dominated by a Gaussian.
We note that if we assume the input x to be Gaussian, then we get the well-known rectified Gaussian distribution (Harva & Kab´an, 2007), which is itself sub-Gaussian.
In summary, these two examples illustrate concrete scenarios of finite-width neural networks where all the assumptions are clearly satisfied simultaneously .
A. L EAVE - ONE - OUT DERIVATION
Influence calculation Let us recall the equation which the estimator (cid:98) θθθ
[Equation]
D should satisfy:
[Equation]
Here, G :=
(1 − ϵ ) F + ϵδ z 0 .
For leave-one-out, we have that G = D \ in − 1 and F = D n .
Without loss of generality, assume that the removed sampled index i = n and so z i = z n , and consider the shorthand ˜ D
= D \ nn − 1 .
Overall, then we are considering the contamination: ˜ D :=
(1 − ϵ )
D n + ϵδ z n , with ϵ = − 1 n − 1 .
Let us substitute all of this back into the equation above:
[Equation]
The last equation holds because notice that in the second term z ∼ (cid:98) θθθ ˜ D (where ˜ D
= (1 − ϵ ) D n + ϵδ z n ) and not z ∼ D n .
Since, both D n and ˜ D are empirical distributions
[Equation]
we can compute the expectation as a finite sum, −
[Equation]
[Equation]
We can split the term involving the gradient of the loss into two parts: one based on the n − 1 samples (of which (cid:98) θθθ
˜ D is also the parameter estimator) and the other based on the left-out sample with index n
[Equation]
The first term is zero since (cid:98) θθθ
˜ D is the parameter estimator of the first n − 1 samples, and will satisfy the first-order stationarity conditions.
Now, rearranging terms results in:
[Equation]
The term on the left is nothing but the Hessian computed over ˜ D or in other words, the first n −samples.
Let us denote it by H \ n L (
(cid:98) θθθ ˜ D ) , where the \ n in the superscript emphasizes that the Hessian is computed over samples excluding the n -th sample.
Further, ∆ ϵ = ϵ − 0 =
ϵ = − 1 n − 1 .
Also, ∆ θθθ = (cid:98) θθθ ˜ D − (cid:98) θθθ D .
Thus we obtain that the change in parameter estimator (i.e., influence on the parameter estimator) is,
[Equation]
Linearizing the loss at the left-out sample.
In order to look at the change in loss evaluated on the n -th sample (left-out sample), we can consider linearizing the loss at (cid:98) θθθ ˜ D .
[Equation]
But, ∆ ℓ n = ℓ [Equation] [Equation]
And, so we have:
[Equation]
Now, we would like to average over the choice of the left-out sample n .
However, we have to be careful and remember that ˜ D
= D \ in − 1 depends on the particular left-out sample i .
Thus, the full expression we get is the following:
[Equation]
Let us recall the change in parameters with the leave-one-out contamination.
From eq.
43, this was
[Equation]
where, ˜ D := D \ nn − 1 = (1 − ϵ )
D n + ϵδ z n , with ϵ = − 1 n − 1 and assuming n is large enough.
Under this asymptotic n scenario, it is reasonable to estimate the Hessian H \ n L and ∇ θθθ ℓ at the distribution D n instead of ˜ D :
= D \ nn − 1 .
We thereby get the following (where we also make the dependence of ∆ θθθ explicit on the particular left-out sample index),
[Equation]
Now, we have a nice expression — in the sense that everything on the right-hand side is in terms of the parameters (cid:98) θθθ S obtained by training with the usual (empirical) distribution D n .
Since, we are interested in the leave-one-out loss, let us analyze how the above change in the estimated parameters (i
[Equation]
e., those obtained at convergence) brings about a change in the loss incurred on the n -th sample itself ∆ ℓ ( n ) .
To this end, we consider a quadratic approximation of the n -th sample loss at (cid:98) θθθ S , as shown below (this can alternatively be thought of :
[Equation]
Next, we consider this procedure for every choice of the left-out-sample, and then average out to get the expression of overall leave-one-out loss.
We will also assume ∥ ∆ θθθ ( n ) ∥ 3 and thus ignore the third-order term.
[Equation]
To go further, we need to plug in the change in parameters computed in eq.
44 above.
However, we first establish the validity of this general expression via the following result.
A. L EAVE - ONE - OUT FORMULA PROOFS
Theorem 7.
Consider the particular case of the ordinary least-squares, where ℓ ( z ,θθθ ) :=
ℓ (( x , y ) ,θθθ ) =
( y − θθθ ⊤ x ) 2 .
We assume that we are given a training set S
= { ( x i , y i ) } ni =1 of points sampled i.i.d from the uniform empirical distribution D n .
Let the inputs be gathered into the data matrix X ∈ R n × d where d is the input dimension and the targets are collected in the vector y ∈ R n .
Under the assumption that the number of samples n is large enough such that n ≈ n − 1 , we have that the leave-one-out expression from eq.
46 (with parameter change given in eq. 44) is equal to the widely-known closed-form formula of leave-one-out for least-squares LOO LS .
Mathematically, we have that,
[Equation]
where, A ii = x
⊤ i ( X ⊤ X ) − 1 x i denotes the i -th diagonal entry of the matrix A
= X
( X ⊤ X ) − 1 X ⊤ (i.e., the so-called ‘hat-matrix’) and θθθ denotes the usual solution of θθθ =
( X ⊤ X ) − 1 X ⊤ y obtained via ordinary least-squares.
Proof.
We will first separately analyze the terms in the summation corresponding to the first-order and second-order parts in the eq. 46, which have been accorded the shorthand B i and C i respectively.
Also, to make expressions simpler, we will define the residual as r i := θθθ ⊤ x i − y i and denote the matrix X ⊤ X by Σ .
Let us write down the individual loss gradients and Hessian in this case.
We have
[Equation]
and, ∇ 2 ℓ i := ∇ 2 θθθ ℓ (( x i , y i ) ,θθθ )
=
2 x i x ⊤ i
[Equation]
The term B i in summation corresponding to the first-order part can be computed as follows.
[Equation]
Essentially, we have used the Sherman-Morrison-Woodbury formula in the fourth line above and rest is mere manipulation.
Similarly
[Equation]
we compute the term C i from the second-order part:
[Equation]
In the last line, we just reuse the computation that we already did in the previous part for x ⊤ i (cid:0) Σ − x i x ⊤ i (cid:1) − 1 x i .
Finally, let us combine everything we have got with eq.
[Equation]
Thus, remembering our shorthand r i := θθθ ⊤ x i − y
i , we conclude our proof.
Corollary 8.
For any finite-width neural network, the first and second-order influence function give similar formulas for LOO like that in Theorem 7, but with A =
Z S ( θθθ ⋆ ) ⊤ (cid:0) Z S ( θθθ ⋆ ) Z S ( θθθ ⋆ ) ⊤ (cid:1) − 1 Z S ( θθθ ⋆ ) , where Z S ( θθθ ⋆ ) :
=
[ ∇ θθθ f θθθ ⋆ ( x 1 ) , · · · , ∇ θθθ f θθθ ⋆ ( x n )] and θ ⋆ are the parameters at convergence for MSE loss.
Proof.
Simply replace x i to ∇ θθθ f θθθ ( x i ) in the proof of Theorem 7 and repeat the procedure, since all the steps hold under the assumption of MSE loss and considering the Hessian H S L ( θθθ ⋆ ) =
H
So ( θθθ ⋆ ) provided for by the assumption A2.
A.6. O VER - PARAMETERIZED CASE
In our above discussion, we considered that Σ
= X ⊤ X was invertible (and likewise in the corollary we considered Z S ( θθθ ⋆ ) Z S ( θθθ ⋆ ) ⊤ was invertible).
However, in the over-parameterized case this many not necessarily be the case.
Nevertheless, there is one simple fix to this issue, as often carried out in the literature.
We consider (cid:98) Σ
= Σ
+ λ I in place of Σ for λ > 0 , and in regards to influence functions, this would basically amount to having an ℓ 2 regularization in the loss function.
As a result, we can exactly repeat the same steps in the proof of the Theorem 7, except with (cid:98) Σ instead.
Eventually, we recover the same formulas but now the expression of the resulting matrix A is slightly different as mentioned below:
[Equation]
But, we can further use the well-known push-through identity and obtain A = (cid:0) XX ⊤ + λ I (cid:1) − 1 XX ⊤ , where we consider the shorthand K :=
XX ⊤ to indicate the kernel or the gram matrix.
Rewriting this gives,
A =
( K + λ I ) − 1 K which is the familiar expression as seen in regularized kernel regression.
For the case of Corollary 8, the same extension can be carried out, except that now the matrix K will be the empirical Neural Tangent Kernel (NTK)
(Jacot et al., 2018) matrix at the optimum θθθ ⋆ .
B I NFLUENCE F UNCTIONS : A PRIMER
In this primer on influence functions, we closely follow the textbook of Hampel et al.
The key objective of influence function is to investigate the infinitesimal behaviour of functionals, such as T ( D n ) .
In particular, this is defined when the change in underlying distribution can be expressed in the form of a Dirac distribution.
Formally, this is defined as follows:
Definition 10.
The influence function IF of the estimator T , at some distribution F , evaluated at a point z
(where such a limit exists) is given by,
[Equation]
Essentially, the influence function (IF) involves a directional derivative of T at F along the direction of δ z .
Further, in order to interpret the above definition, substitute F by D n − 1 and put ϵ = 1 n .
This implies that (IF) measures n times the change of statistic T due to an additional observation z .
In other words, it describes the (standardized) effect of an infinitesimal contamination on the estimate
T .
E.g., in the case of parameter estimator, the change in estimated parameters due to presence of an additional datapoint.
By now, the astute reader can already see its natural application for leave-oneout error, but we ask for some patience so as to discuss some other important aspects of influence functions.
More generally, one can view influence functions from the perspective of a Taylor series expansion (to be precise, the first-order von Mises expansion) of T at F , evaluated on some distribution G “close” to F ,
[Equation]
So, as evident from this, it is also possible to consider higher-order influence functions and use them in a combined manner, as considered in Debruyne et al.
However, since the first-order term is often the dominating term — as well as to ensure tractability when we later consider neural networks — we will restrict our attention to only first-order influence functions hereafter.
B. P ROPERTIES OF INFLUENCE FUNCTIONS
(i) Zero expectation.
The expectation of influence function over the same distribution is zero, i.e., (cid:82)
IF( z ; T, F ) dF
( z )
= 0 .
This should actually be quite intuitive as we are basically averaging out all possible deviations of the estimator.
But, the formal reasoning is that influence function is essentially akin to G ˆ ateaux derivative at the distribution F ,
[Equation]
Now, replace G by F in the above equation and the stated property follows.
(ii) Variance of IF provides asymptotic variance of corresponding estimator.
When the observations z i are sampled i.i.d.
according to F , then the empirical distribution D n converges to F, for n sufficiently large, by Glivenko-Cantelli theorem.
So, replacing G by D n in eq. 47 and using the first property, we get:
[Equation]
Integrating over the empirical distribution D n , we obtain:
[Equation]
The term on the right-hand side involving IF is asymptotically normal by (multi-variate)
Central
Limit Theorem.
Further, the remainder terms can often be neglected for n → ∞ , and thereby the estimator T n is also asymptotically normal.
Thus, √ n ( T n ( D n ) − T ( F ))
d → N p (0 , V ( T, F )) , with the asymptotic variance (more accurately, covariance matrix) V ( T, F ) as follows:
[Equation]
For the one-dimensional case, i.e., T ( F ) ∈ R , we simply get V ( T, F ) =
(cid:82)
IF( z ; T, F ) 2 dF ( z ) .
(iii) Chain rule for influence functions.
Suppose our estimator (cid:98) θθθ ( F ) depends on some other estimators, i.e., (cid:98) θθθ ( F ) :
= T (cid:16)(cid:98) θθθ 1 ( F ) , · · · , (cid:98) θθθ k ( F ) (cid:17) , then we can expression the influence function for (cid:98) θθθ ( F ) as,
[Equation]
Remark.
We refer the mathematically-oriented reader to Huber (2004) for details on the regularity conditions needed to ensure the existence of IF and the like.
C E MPIRICAL RESULTS AND DETAILS C. E MPIRICAL DETAILS
We train all the networks via SGD with learning rate 0 .
5 and learning rate decay by a factor of 0 .after each quarter of the target number of epochs.
For the experiments based on MNIST1D (Greydanus, 2020), the input dimension size is d =while number of classes is K = 10 .
Double-descent model sizes (a) For the 3-layer double descent plot, we consider hidden widths m 1 , m 2 ∈ { 10 , 20 , 30 , 40 , 50 } and choose all those pairs as experiments where | m 1 − m 2 | ≤ 20 .
(b) For the 2-layer plot in MSE, we take the hidden layer widths from [1 , 151] at an interval of 10 .
(c) For the 2-layer plot corresponding to CE, we take hidden layer sizes from [2 , 32] at intervals of 2 and then for reducing the computation load when the network sizes increase, we take coarser intervals with a gap 5 and for even bigger sizes, at a gap of later 10 .
But this is purely to reduce computational load that comes with Hessian computation.
In all the double descent curves, we ensure that near the interpolation threshold all models are at interpolation, in the sense that the training accuracy is 100% .
C.
V ERIFICATION OF ASSUMPTION
A
[Figure]
n o n
[Figure]
z e r o e i g e n v a l u e r
( H
S o )
( l o g s c a l e )
[Figure] [Figure]
C.
C ROSS E NTROPY DOUBLE DESCENT
[Figure]
C. MSE DOUBLE DESCENT
For the sake of visualization in the 3-layer case
[Figure]
we smooth the quantities involved by considering a moving average of them over three successive model sizes
[Figure]
P o p u l a t i o n
L o ss
[Figure]
Also, in the unsmoothed version of the above plots, we also show the corresponding train loss.
It must be emphasized that near interpolation the training accuracy is 100% , even though there might be some non-zero training loss.
C. R ESULTS ON ADDITIONAL DATASETS
[Figure]
P o p u l a t i o n
L o ss      r
( H S o )
( l o g s c a l e )
[Figure]
i
[Figure]
n [Figure]
[Figure]
Empirical details.
The optimization details are the same as mentioned before in the setting of
MNIST1D, except that we train longer for 20 K epochs.
We downsample the input dimension for CIFAR10 and MNIST to about the same size so as to ensure consistency, and in particular, d = 48 for CIFAR10 (by downsampling 32 × 32 × 3 images to 4 × 4 × 3 and then flattening) while d = 49 for MNIST (by downsampling 28 × 28 images to 7 × 7 and then flattening).
We find that on harder datasets like these, it takes even longer to drive the networks to interpolation and thus consider n = 40 samples.
We consider the sizes of the hidden layer from the set { 1 , 3 , 5 , 7 , 9 , 11 , 21 , 31 , 41 , 51 , 61 , 71 , 81 , 91 , 101 , 111 } .
In other words, we sample at a finer granularity near the interpolation threshold ( ≈ 400 ), while increase the gaps later on.
Also, we find that networks with hidden layer width m = 1 fails to train and gives NANs, so we exclude its result.
Overall, we find that the population risk and the minimum non-zero eigenvalue of the Hessian at the optimum show a very similar trend like in the case of MNIST1D.
As a result, this implies that our empirical findings also generalize to other datasets as well
[Figure]
C. 2- HIDDEN LAYER RESULTS FOR LARGE DIFFERENCE IN SUCCESSIVE LAYER SIZES
[Figure]
Previously, in our double descent experiments for 2-hidden layers, we considered that hidden widths m 1 , m 2 ∈ { 10 , 20 , 30 , 40 , 50 } and chose all those pairs where | m 1 − m 2 | ≤ 20 .
Now, a question might arise what happens in the case when the successive hidden layer sizes are very ‘imbalanced’ (or non-uniform), i.e., they have a big difference between their sizes.
The Figure 8 shows the setting of double descent with 2 hidden layers with sizes m 1 and m 2 , which are chosen as per m 1 ∈ { 20 , 40 } and m 2 ∈ { 20 , 40 , 60 , 80 , 100 , 120 , 140 } .
Rest of the empirical details regarding training and dataset are identical to the MNIST1D setting considered earlier.
We find the population risk and minimum non-zero eigenvalue behaviour which is very similar to Figure 1b and in accordance with our theoretical predictions.
This also confirms that our theoretical analysis generalizes to diverse architecture patterns, not just the ‘balanced’ setting in which double descent was has been considered before, which leads to the following remark.
Closing remark.
Before we finish this discussion, let us emphasize that almost all prior works on double descent Nakkiran et al.
; Nakkiran (2019) consider the case of ’balanced’ hidden layer sizes.
In other words, these works consider a fixed architectural pattern, say { m } → { 2 m } → { m } and then increase the common width multiplier m .
In this respect, we are one of the first works that not just demonstrates the occurrence of double descent in the imbalanced hidden-layer settings — but also explains the behaviour via the trend of the minimum non-zero eigenvalue of the Hessian at the optimum.
C.
N ATURE OF CONSTANTS FOR THE EIGENVALUE ASSUMPTION
In the assumption A3, we assume the existence of constants 0 <
A 0 θθθ < B 0 θθθ < ∞ such that the following holds:
[Equation]
So in this section, how these constants actually behave in practice and if they are O (1) across varying network sizes?
Or, equivalently, how the ratio λ min ( C S f ( θθθ ⋆ )) λ min ( C S f ( θθθ 0 )) behaves and if it is O (1) ?
[Figure]
[Figure]
We consider the setting of one-hidden layer neural networks trained with MSE loss for 5 K epochs as discussed in the main text (for other training details, see Appendix C.1).
To remind, we consider networks of hidden-layer sizes sampled uniformly from [1 , 151] at an interval of 10 .
The Figureplots this desired ratio.
We find that indeed this ratio is O (1) .
More precisely, the Table 1 details the summary statistics of this ratio.
Overall, the trend in Figure 9 and the precise summary statistics in the above Table 1 even implies the existence of universal constants for A 0 θθθ and B 0 θθθ (e.g., one setting would be to 0 .
3 and 12 respectively).
This ratio naturally depends on the problem, so it is likely that it will change slightly (e.g., on CIFAR10, the mean changes to 7 . 812 ) but we are able to confirm the existence of universal constants [Table] [Table]
across all our experiments.
Lastly, there is also a theoretical basis to it, as remarked in the main text, since the map
A (cid:55)→ λ
i ( A ) is Lipschitz-continuous on the space of Hermitian matrices, which follows from Weyl’s inequality (Tao, 2012, p. 56).
Final remarks.
Importantly, we would thus like to reiterate that we never impose constraints on the minimum eigenvalue of the covariance of the function Jacbians.
But, rather inspired by the above empirical observation, we consider the existence of such constants.
C. C ONSTITUENT OF H ESSIAN AT THE OPTIMUM
For our analysis
[Figure]
we made the assumption A2 based on the prior works of (Sagun et al., 2017; Singh et al., 2021).
We also find in our own experiments that the very same observation made in these papers holds — namely, that the Hessian H L at the optimum is only composed of the outer-product
Hessian H o , while the functional Hessian H f = 0 .
To show this
[Figure]
we plot in Figure 10 the nuclear norm, spectral norms [Figure] as well as the minimum nonzero eigenvalue of the the overall loss Hessian H L and the outer-product Hessian H o for one-hidden layer networks trained for 20K epochs and with rest of the training details identical to all other experiments
[Figure]
[Figure]
L OWEST FEW EIGENVALUES ARE THE DOMINATING FACTOR BEHIND DIVERGENCE NEAR INTERPOLATION THRESHOLD
We consider the settings of CIFAR10, MNIST, and MNIST1D, all of which correspond to the double descent shown in Figures 7a, 7b, and 1 respectively.
To demonstrate how many of the lowest eigenvalues capture the double descent trend, for each model in all of the above settings, we plot the % of the trace (of the Hessian inverse) captured by the lowest eigenvalues of the Hessian, since the lower bounds exhibit this dependence.
In order to ensure consistent comparisons across varying model sizes, we consider the lowest eigenvalues in 0 .
5% , 1% , 2% , 5% , 10% of the total number of eigenvalues of that model.
The results can be found in the following plots:
Observations.
We see that across all these cases just 0 .
5% of the lowest eigenvalues are enough to capture the double descent behaviour — capturing a minimum of 60% of the trace near interpolation across all these settings.
[Figure]
Further > 98% of the trace is captured as soon as we have 1% of the lowest eigenvalues for CIFAR10, 0 .
5% for MNIST, and 5% for MNIST1D.
This clearly shows that the double descent behaviour is indeed captured by a very small handful of the lowest eigenvalues.
(As a matter of fact, near the interpolation threshold, even just using the minimum non-zero eigenvalue alone captures 85 .
56% of the trace
CIFAR10 and
C. L OWER BOUND COMPUTATION
At an initial glance, it might seem that the computed lower bounds could be very small in magnitude to be of use — as the bound is scaled by σ 2 min .
However, while practically computing one can consider a tighter lower bound, by using a particular tolerance τ .
Let us illustrate by considering evaluating this bound on a test set S ′ (just like we actually do).
This is because in the lower bound we have the expression,
[Equation]
Now, since Tr (cid:0) A ( x , y ) (cid:1) is non-negative, if we are given a tolerance τ , we filter out those samples with ∥∇ f ℓ (cid:0) f θθθ ⋆ ( x ) , y (cid:1) ∥ 2 < τ
[Equation]
where, 1 is the indicator function.
This helps us compute lower bounds which can be evaluated in practice.
[Figure]
C. E MPIRICAL OBSERVATIONS FROM LINEAR REGRESSION C.11. H ESSIAN STATISTICS ON TRAINING SET
[Figure]
C.11. H ESSIAN STATISTICS ON TEST SET
[Figure]
C.11. D RAWING DD PLOTS AT ARBITRARY THRESHOLDS
[Figure]
The input design matrix is constructed in such a way so that given some d ′ many input features, the number of base features is d .
But we additionally have β d many redundant features (composed of linear combinations of the base features).
Also, as the number of features is increased in the above test loss curve, these redundant features follow in succession to the base features.
Therefore, at any point in these graphs along the x-axis for some number of features d ′ , the effective number of features are [Figure] ′ ( β + 1) .
Hence, the peak occurs when the effective number of features equals the (effective) number of samples, i.e., at d ⋆ β + 1 = n , which is nothing but d ⋆ = n ( β + 1) .
Thereby, looking from the perspective of rank (the Hessian here is simply the covariance matrix), the position of the interpolation threshold can be reconciled.