W IDE
N EURAL
N
ETWORKS ARE I NTERPOLATING K ERNEL M ETHODS : I MPACT OF I NITIALIZATION ON G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The recently developed link between strongly overparametrized neural networks (NNs) and kernel methods has opened a new way to understand puzzling features of NNs, such as their convergence and generalization behaviors.
In this paper, we make the bias of initialization on strongly overparametrized NNs under gradient descent explicit.
We prove that fully-connected wide ReLU-NNs trained with squared loss are essentially a sum of two parts:
The ﬁrst is the minimum complexity solution of an interpolating kernel method, while the second contributes to the test error only and depends heavily on the initialization.
This decomposition has two consequences: (a) the second part becomes negligible in the regime of small initialization variance, which allows us to transfer generalization bounds from minimum complexity interpolating kernel methods to
NNs; (b) in the opposite regime, the test error of wide NNs increases signiﬁcantly with the initialization variance, while still interpolating the training data perfectly.
Our work shows that – contrary to common belief – the initialization scheme has a strong effect on generalization performance, providing a novel criterion to identify good initialization strategies.
I NTRODUCTION
Neural networks (NNs) have celebrated many successes over the past decade and achieved state of the art performance across various domains and tasks.
From a theoretical standpoint, however, many aspects of neural networks are not well understood, as for example illustrated by Zhang et al.
Neural networks seem to contradict classical learning theory as, in many scenarios, they are able to ﬁt random labels perfectly while still generalizing well when trained on the true labels.
In addition, overparametrized neural networks frequently exhibit even improved test performance when the number of parameters is increased further (Belkin et al., 2019).
NN models thus often seem to avoid overﬁtting.
Very recently there have been advances in understanding the training and evaluation behavior of neural networks in the inﬁnitely wide limit (Jacot et al., 2018;
Hayou et al., 2019) and also in the strongly overparametrized regime (Du et al., 2018b; Li & Liang, 2018; Allen-Zhu et al., 2018a), which is close to the inﬁnite limit NN.
Both lines of work express the behavior of the neural network in terms of the so-called neural tangent kernel (NTK).
In particular, Du et al.
(2018b) showed in this way that, under mild conditions, strongly over-parametrized neural networks converge to a global minimum of zero training error.
In another line of research, Belkin et al.
(2018b) suggested a new picture of learning in the overparametrized regime, introducing an “interpolating kernel method” which successfully learns in this regime.
This method selects the least complex function that interpolates all data points perfectly, as opposed to traditional methods which balance the function’s complexity against its goodness-of-ﬁt.
It is suggested that this picture of overparametrized learning could help understand neural networks.
If overparametrized NNs are indeed linked to such kernel methods, then zero (or small) training error alone might not tell us much about the test performance, as the measure of function complexity could be determined by the NN architecture, initialization, and the optimization method.
A bad NN design may lead to an unfavorable complexity measure that could result in a large generalization gap.
The question therefore remains in which way exactly are overparametrized neural networks linked to interpolating kernel methods?
And how can this connection help us better understand neural networks and their puzzling features?
1. O UR C ONTRIBUTIONS
In this paper we answer the ﬁrst question by making the link to interpolating kernel methods explicit for strongly overparametrized NNs.
After that, we exhibit two implications for NNs which this connection allows us to draw.
First, for strongly overparametrized ReLU-NNs we make the bias explicit that is imposed by the NN initialization and by training with discrete gradient descent steps on the squared loss.
We achieve this by decomposing the learned NN function as well as the test error into two terms: The ﬁrst term corresponds to a minimum complexity interpolating kernel method, whereas the second term is proportional to the initialized weights and gives a non-zero contribution only if the test input is not part of the training set.
Second, we are able to bound the difference between a NN with standard initialization of small variance and the solution of a minimum complexity interpolating kernel method based on the NTK.
This bound provides a new way to transfer existing test error bounds from interpolating kernel methods to NNs, and vice versa.
Our third contribution shows that without putting any constraints on the NN initialization, low training error does not imply low test error.
This is because the second term in the above test error decomposition grows as a power of the initialization variance, as we show, whereas the ﬁrst term is independent of this variance.
Thus, the test error grows dramatically with increasing initialization variance, despite (close to) zero training error.
Our ﬁndings provide an additional theoretical way to analyze initialization schemes, besides the existing heuristics and intention to prevent vanishing and exploding gradients (He et al., 2015; Glorot & Bengio, 2010).
Our theoretical bounds and insights are nicely corroborated in several experimental settings.
1. N OTATION
We consider a supervised learning setting with data D
= { ( x i , y i ) } Ni =1 ⊂ B d 1 (0)
× R consisting of
N training points, i.e. the inputs come from the bounded d -dimensional ball B d 1 (0) =
{ x ∈ R d : || x || 2 ≤ 1 } 1 .
We often view the training inputs X = ( x 1 , . . .
, x N ) T ∈ R N × d and corresponding labels Y =
( y 1 , . . .
, y N ) T ∈ R N in matrix form.
For any function g with domain R d , we deﬁne g ( X ) as the row-wise application of g .
Similarly, for any set D test ⊂ R d × R of N test test points we deﬁne X test ∈
R N test ×
d and Y test ∈ R N test .
We call a dataset D consistent if x i = x
j implies y i =
y j .
We denote the output of a fully-connected NN with L layers and parameters θ by
[Equation]
where a ( h ) :
= max { h, 0 } is the ReLU-activation function applied componentwise, m l denotes the number of neurons in layer l , and W l ∈ R
m l × m l − 1 and b l ∈ R m l are the weights and biases.
For simplicity we choose the number of units in each hidden layer to be the same 2 m l = m for l = 1 , . . .
, L − 1 and m L = 1 .
θ l =
[ W l , b l ] denotes the vector of weights and biases in layer l , and
θ =
[ θ 1 , . .
, θ L ] the vector of all parameters, whose total number is M = (cid:80) Ll =1 ( m l − 1 + 1) m l .
We keep the “initialization variance” σ 2 (equivalently, σ > 0 ) as an explicit scalar parameter, that can be varied.
Our parametrization (1) together with a standard normal initialization W li,j , b li ∼ N (0 , 1) is the so-called NTK-parametrization (e.g. Du et al.
(2018b)).
In a forward pass this is equivalent to the standard parametrization (setting σ/ √ m l → 1 in Eq.
(1)) with the weights and biases initialized at variances σ 2 /m l and σ 2 , respectively (He et al., 2015).
During gradient training the two parametrizations are equivalent up to a parameter-dependent scaling factor of the learning rate (Lee et al., 2019).
We consider the squared error loss (cid:96) (ˆ y, y ) =
| ˆ y [Equation] y | 2 / 2 and train by minimizing the empirical risk
[Equation]
using gradient descent (GD).
That is, starting from the initialization θ = θ
the weights are updated according to the discrete iteration θ t +
=
θ t − η ∇ θ L ( θ t ) for t = 0 , 1 , . . .
, where η is a learning rate.
We connect NNs to interpolating kernel methods w.r.t.
two speciﬁc kernel functions that are deﬁned for any initialization θ = θ 0 of a NN (1).
The ﬁrst kernel Ψ : R d × R d → R is associated with training only the last layer θ L of a NN (Lee et al., 2019),
[Equation]
is the corresponding feature map.
The second is the so-called neural tangent kernel (NTK) Φ :
R d × R d → R associated with training all NN parameters (Jacot et al., 2018; Lee et al., 2019), Φ( x, x (cid:48) )
=
φ ( x ) φ ( x (cid:48) )
T , where φ ( x ) = ∇ θ f NN θ ( x ) (cid:12)(cid:12) θ
= θ 0 ∈ R 1 × M (4) again represents the feature map.
The “minimum complexity interpolating kernel method” ﬁts a consistent training set D perfectly, using a function of minimum kernel norm w.r.t.
a kernel K :
[Equation]
subject to Y = f
( X ) , (5) where H K is the reproducing kernel Hilbert space (RKHS) associated with K (Belkin et al.
(2018b)
; see also App. B).
Additionally, we denote the solution of a fully converged fully-connected ReLUNN trained with gradient descent, squared loss and random initialization by f NN .
We denote the test losses for these two predictors by L methtest =
(cid:107) f meth
( X test ) − Y test (cid:107) 2 / ( N test ) for meth =
NN , int.
R ELATED W ORK
There are two main lines of research related to our work.
The ﬁrst line investigated in which way strongly overparametrized NNs, where the number m of units per hidden layer scales polynomially with the number N of training data points, converge to arbitrary small training error during training.
The ﬁrst papers to investigate this for 1 hidden layer were Daniely (2017), Li & Liang (2018) and
Du et al. (2018b).
Later, Allen-Zhu et al.
(2018a),
Allen-Zhu et al.
(2018b),
Zou et al.
and Du et al.
(2018a) extended the work to deeper networks, CNNs and RNNs (Goodfellow et al., 2016).
Li & Liang (2018) mostly focus on the cross entropy loss, whereas Du et al.
(2018b) focuses on the squared error as we do here.
Additionally, Du et al.
(2018b) is the ﬁrst work to connect the training behaviour of ﬁnite NNs (not only in the limit of inﬁnite width m ) to the NTK-kernel introduced by Jacot et al.
This connection is important for our work.
In a similar vein, Arora et al.
(2019b) developed data dependent generalization bounds based on the fact that the weights do not move far from their initialization, as shown in the above works.
Furthermore, Arora et al. (2019a) experimentally evaluate the generalization behaviour of the inﬁnite width solution of NNs in the
NTK-limit. Most of these contributions focus on showing that the training loss converges to a global minimum or that the training dynamics is close to a trajectory that converges to zero training loss.
We build upon these works, but in contrast our focus is the connection between the test error of a trained
NN and the error of an interpolating kernel method.
We achieve this by resolving the implicit bias that the initialization and gradient descent (GD) has on the test loss.
The initialization bias on the training loss is minor compared to the test loss.
The most related work to ours is by Zhang et al.
They investigate a similar issue as we do but use a different method to weaken the effects of non-zero initialization and also do not quantify the impact of initialization on generalization but give evidence for such an effect.
The recent contribution Lee et al.
, building upon Lee et al.
(2017), Matthews et al.
and Neal (1996), explicitly solves the training dynamics and shows that the solution in the trained inﬁnite model seems to be similar to a Gaussian Process, even though its covariance is not the Bayesian posterior of a simple prior.
The solution of their continuous training time analysis is similar to the linearized model in the proof of our Thm. 2.
We perform a discrete-time analysis, as in actual NN training, instead of the continuous-time (ODE-like) analysis of other works.
Woodworth et al.
(2019) uses a continuous time analysis as well to focus on the effect of noise in the gradients.
Borovykh (2019) investigates the effect of the scaling between the “deep” and the “kernel” regime.
The second line of related work introduces interpolating kernel methods and the idea of an overparametrized learning regime which does not suffer from overﬁtting even though the number of parameters is be much larger than the number of training samples.
This line of work was mainly introduced by Belkin et al.
(2018b)
and the follow-up works Ma et al.
; Belkin et al.
(2018c;a).
The main idea is, instead of regularizing the function while trying to ﬁt the data points well, to regularize (the complexity of) a function that perfectly interpolates all data points.
This newly identiﬁed overparametrized learning regime challenges the conventional thinking about the biasvariance-tradeoff (Vapnik, 2013) from the underparametrized regime.
Belkin et al.
(2019) introduces a bouble-dip picture to describe the transition between the two regimes.
Liang & Rakhlin (2018) have derived generalization bounds for these interpolating kernel methods using novel techniques.
Using our explicit link to NNs, this will provide an alternative to NN generalization bounds like margin or PAC-Bayes-based bounds (e.g., Bartlett et al.
; Neyshabur et al.
While a connection between overparametrized NNs and interpolating kernel methods has been suggested in the papers above, the link has not been made explicit in a rigorous way, to the best of our knowledge.
The implicit bias of (stochastic) gradient descent ((S)GD) and random initialization on the trained
NN has been the topic of many previous works, e.g. Soudry et al.
; Glorot & Bengio (2010);
Daniely et al.
; Rahaman et al.
; Oymak & Soltanolkotabi (2018), which all elucidate certain aspects of the implicit bias.
Our work is unique in that it investigates the effect (implicit bias of (S)GD and random initialization) on the test error of trained NNs, exhibiting NNs where – despite always vanishing training error – the test error can be either very good or arbitrarily large.
Without gradient descent training and in a deterministic fashion, such examples have been constructed in (M ¨ ucke & Steinwart, 2019).
L INKING WIDE NN S AND I NTERPOLATING K ERNEL M ETHODS
We now develop the link between strongly overparametrized neural networks and minimum complexity interpolating kernel methods mathematically.
We ﬁrst illustrate the main idea with the simpler case where only the last NN layer is trained, before extending the result to general NN training.
3. P ARADIGMATIC LINEAR CASE : T RAINING ONLY THE LAST NN LAYER
As the NN function f NN θ ( x ) (1) is linear in the parameters θ L of the last layer, the associated features ψ ( x ) = ∇ θ L[-+
-] f NN θ ( x ) (cid:12)(cid:12) θ 0 and corresponding kernel Ψ from (3) do not depend on the last-layer initialization θ L 0 (but depend on the other layers θ 1: L − 1), and f NN θ ( x )
= (cid:104) θ L , ψ ( x ) (cid:105) holds exactly for all θ and x .
Starting NN gradient descent at θ L 0 = 0 , we get the following simple link:
Theorem 1 Let f NN be the fully converged solution of a fully-connected strongly overparametrized
ReLU-NN (1) with L layers, where only the last layer θ L =
[ W L , b L ] has been trained, using gradient descent under squared loss on a consistent dataset D , after it had been initialized at θ L 0 = 0 , and all other layers have been kept at their random initialization ∼ N (0 , 1) .
Then it holds that
[Equation]
where f int Ψ is the minimum complexity interpolating kernel solution (5) of D w.r.t.
kernel Ψ (3).
Proof.
Abbreviating the last-layer parameters by ϑ = θ L and using the aforementioned linearity f θ ( x ) =
(cid:104) ϑ, ψ ( x ) (cid:105) in ϑ , the gradient descent update rule reads:
[Equation]
[Equation]
This iteration can be solved explicitly using induction and the binomial theorem (Shah et al., 2018):
[Equation]
where ( ψ ( X ) ψ
( X ) T ) − 1 denotes the pseudoinverse.
If the learning rate satisﬁes η < 2 /λ max ( ψ ( X ) ψ ( X ) T ) such that gradient descent converges, we get (cid:0) 1 − ηψ ( X ) ψ
( X ) T (cid:1) t →in the limit t → ∞ as the spectral radius is smaller than one and the term in square brackets can be simpliﬁed to 1 .
To be precise this holds since D is consistent and we are in the strongly overparametrized regime, so that Y (and ψ ( X ) ϑ 0 ) lies in the range of ψ ( X ) ψ ( X ) T almost surely (over the random initialization θ 1: L − 1, on which ψ depends), see also Du et al.
(2018b).
Setting ϑ 0 = 0 as presupposed, we obtain the following predictor at full convergence ( t = ∞ ):
[Equation]
The same value ϑ ∞ can be obtained as the solution to the following least squares interpolation:
[Equation]
subject to Y = ψ ( X
[Equation]
ϑ
[Equation]
This can in turn be written as minimizing the RKHS norm of an interpolation function w.r.t.
the kernel Ψ from Eq.
(3) (for details on this see App.
A and B):
[Equation]
subject to Y =
f
( X ) [Equation] [Equation] The claim has thus been proven by deriving the explicit solution (9) and showing it to equal f int Ψ .
(cid:3)
3. G ENERAL CASE : T RAINING THE FULL N EURAL N ETWORK
In the case of training the full NN, we want to follow similar steps as in Sect.
For this, we ﬁrst approximate f NN θ ( x ) afﬁne-linearly by f lin θ around its initialization θ = θ 0 , using recent results on wide NNs (Du et al., 2018b).
We further drop the requirement that any components of θ 0 must vanish; note, e.g., that the suggestive requirement θ 0 =
(analogous to Sect.
3.1) would, for L ≥ 1 , lead to vanishing features φ ( x ) = ∇ θ f NN θ ( x ) (cid:12)(cid:12) θ 0 and a degenerate kernel Φ = 0 .
Nonzero θ 0 will however result in an initialization bias in f lin θ as we will see (Eq.
Theorem 2 Let f NN be the fully converged solution of a strongly overparametrized ReLU-NN (1) with L layers, each with m neurons, where all layers θ have been trained using gradient descent under squared loss on a consistent dataset D after random initialization θ 0 ∼ N (0 , 1) . Then there exists a N ∈ N such that for all NN with m ≥ N and ∀ x ∗
∈ B d 1 (0) it holds with probability at least 1 − 2 δ over the initialization that
[Equation]
(cid:19) (12) where f int Φ is the minimum complexity interpolating kernel solution (5) of D w.r.t.
kernel Φ (4).
For the proof of Thm.
2 we describe here especially those steps which are new compared to Thm. 1.
The ﬁrst property we need is the approximate linearity of the NN in its parameters θ during the whole gradient training.
This is the reason why we restrict to strongly overparametrized fully-connected NNs.
For such networks, a combination of the works (Jacot et al.
Du et al.
(2018b); Lee et al.
(2019)) shows that the solution θ t of the training dynamics t = 0 , 1 , . . .
stays close to the initialization θ 0 , and the deviation from a linear model vanishes with growing width m :
Lemma 1 (Lee et al.
Du et al.
(2018b))
Denote the linearization of an NN (1) around its initialization θ 0 by f lin θ ( x ) :
=
f NN θ 0 ( x ) + ( ∇ θ f
NN θ ( x ) (cid:12)(cid:12) θ 0 )( θ − θ 0 ) .
Further, for t ≥ 0 , let ˜ θ t and θ t be the parameters obtained by gradient descent training starting from θ 0 with sufﬁciently small step size η of the full NN and its linearisation respectively.
Then, there exists some N ∈ N such that for all m ≥ N it holds with probability at least 1 − δ over the initialization that: sup t | f NN ˜ θ t ( x ) − f lin θ t ( x ) | [-
-]2 ≤
O (1 /m ) for all x ∈ B d 1 (0) .
Similar linearisation hold for strongly overparametrized RNNs and CNNs as well, not only for fully-connected NNs.
The theorem proves that for every sufﬁciently wide NN the linearised solution is close to the trained full NN.
For a more detailed analysis of the dependence of N on the width m , variance σ and δ we refer to Lee et al.
Du et al. (2018b).
Using this result, we continue our analysis with f lin which we can rewrite as f lin θ ( x ) = f NN [Equation]
0 ( x ) + (cid:104) [Equation] [Equation] x )
[Equation]
=
[Equation]
f NN
[Equation] [Equation] − [Equation]
, φ [Equation] ) (cid:105) (cid:1) + (cid:104) θ, φ ( x ) (cid:105) were φ ( x ) = ∇ θ f NN θ ( x ) | θ 0 is the feature vector deﬁned in Eq.
Similar to the proof of Thm. 1, gradient descent on f lin θ with the loss function (cid:107)
f lin θ ( X ) − Y
(cid:107) 2 /and sufﬁciently small step size η < 2 /λ max ( ψ ( X ) ψ ( X ) T ) leads to the ﬁnal ( t → ∞ ) parameter θ lin
[Equation]
[-θ 
-]X
[Equation] X [Equation]
(cid:1) [Equation] [Equation] NN
[Equation] 0 ( X )) , resulting in the function
[Equation]
To connect Eq.
(13), and thereby f NN , to the solution f int Φ of the interpolating model, two steps remain to be done.
First we show that the two summands in Eq.
(13) which are not proportional to the labels Y can be simpliﬁed into a single form for fully-connected ReLU-NNs; this can be done by showing f θ 0 ( x ) =
1 L (cid:104) θ 0 , Φ( x ) (cid:105) (see Lemma 2 for more details).
Secondly, we can bound this single expression from above in terms of the initialization variance (Lemma 3; for proofs see Appendix.
The term proportional to Y is then identiﬁed with f int .
Lemma
For a fully-connected ReLU-NN f NN θ
(Eq. 1) with L layers, the trained linearized version f lin from Eq.
(13) can be written as follows:
[Equation]
Thus, the trained f NN ( x ) =
f lin ( x ) + O (1 / √ m ) decomposes essentially into the two terms of Eq.
(14) (for x ∈ B d 1 (0) ).
As | f NN ( x ) − f lin ( x ) | ≤ O (1 / √ m ) , the expression (14) holds for the trained f NN in the wide limit m → ∞ as well.
This technical result enables all of our results in Thms. 2, 3, and 4.
Eq.
(14) can be understood intuitively (in the case of a consistent D ):
The ﬁrst term makes the model interpolate the training data D perfectly
( f lin ( X ) = Y ), independently of the initialization θ 0 , whereas the second term vanishes on the training data x = X
and thus only contributes to test inputs x = x ∗ .
The second term furthermore depends on the initialization θ 0 , more precisely on the component of θ 0 that is orthogonal to the feature manifold spanned by the data D , i.e. orthogonal to the range of φ ( X ) T .
Note that (cid:2) 1 − φ ( X ) T ( φ ( X ) φ ( X ) T ) − 1 φ ( X ) (cid:3) is the projector onto the kernel of φ ( X ) , about which the dataset D does not give any information.
Lemma
For a fully-connected ReLU-NN f NN θ , it holds for all x ∈ B d 1 (0) with probability at least 1 − δ over the initialization θ 0 that 1 L | φ ( x ) (cid:0) [Equation] − φ ( X [Equation]
T ( φ ( X ) φ ( X
[Equation] φ ( X ) [Equation] As in the proof of Thm.
(see Eqs.
the ﬁrst part of the decomposition in Eq.
(14) is f int Φ .
By Lemma 3, we see that for small σ 2 the distance between f lin and f int Φ becomes small with high probability, and f lin and f NN are close too because of m (cid:29) N .
The triangle inequality | f NN ( x ∗ ) − f int Φ ( x ∗ ) | ≤ | f NN ( x ∗ ) − f lin /Phi ( x ∗ ) | + | f lin ( x ∗ ) − f int ( x ∗ ) | gives the desired result.
Thus, strongly overparametrized neural networks initialized with small variance and trained under gradient descent correspond to solutions of a minimum complexity interpolating kernel method.
(cid:3)
I MPLICATIONS FOR NEURAL NETWORKS
Now that we have established the link between NNs and interpolating kernel methods, we show two implications this has for the test loss of neural networks.
First, we derive an upper bound on the NN test loss, which – in the regime of small NN initialization variance – allows us to transfer generalization bounds from interpolating kernel methods to neural networks.
Second, a lower bound on the NN test loss gives – in the regime of large NN initialization variance – examples of NNs, trained under gradient descent after random initialization, that generalize badly despite (close to) zero training loss.
4. T RANSFERRING G ENERALIZATION B OUNDS TO N EURAL N ETWORKS
From Thm.
2, it is now almost straightforward to connect the test error of the interpolating kernel method to the test error (w.r.t.
squared loss) of the neural network.
Theorem 3 Let f NN be the fully-converged solution of a strongly overparametrized NN (1) with L layers and m hidden units per layer and initialization variance σ 2 , where all layers θ have been trained using gradient descent under squared loss on a consistent dataset D after random initialization θ 0 ∼ N (0 , 1) .
Then it holds with probability at least 1 − 2 δ over the random initialization that (cid:112)
[Equation] To prove this theorem we use the fact that | f lin ( x ) − f NN ( x ) | ≤ O (1 / √ m ) holds uniformly for all x ∈ B d 1 (0) and that (cid:107) f lin
( X test ) − f int ( X test ) (cid:107) 2 /N test ≤ σ 2 L /δ holds with high probability; the last fact follows in the same way as in the proof of Lemma 3 by, instead of applying Markov’s inequality to the squared deviation for a single test point x , applying it to averaged sum of squared deviations over all test points.
Then, we make use of the triangle inequality and evaluate the square to prove the desired result.
This inequality now enables us to transfer bounds for the test error from one formalism to the other and gives us a new grip on ﬁnding bounds on the test error for NNs.
Good generalization bounds on minimum complexity interpolating models have been established in (Liang & Rakhlin (2018)).
On the other hand, Belkin et al.
(2018b) showed that conventional bounds in the underparametrized regime might be of little use for minimum complexity kernel methods.
4. Z ERO NN TRAINING ERROR ALONE DOES NOT IMPLY SMALL TEST ERROR
In the past many people trained neural networks with the goal to achieve a low training error with the hope that this might also result in a low test error.
The initialization was merely thought of as a necessity to converge faster to a low training error or to give a favourable starting point for the training dynamics.
We now show with our analysis that random initialization alone is not enough, but instead we need certain constraints on the initialization to be able to give generalization bounds.
Before we prove this let us derive some simple properties.
Let Φ( x, σ ) be the feature vector of a NN initialized with NTK-parametrization with initialization variance σ .
Then it is easy to see that Φ( x, σ ) =
σ L Φ( x, 1) because of the homogeneity of the derivative w.r.t constants.
Further we deﬁne J ( X test , σ ) :
= (cid:107) 1 L Φ( X test ) (cid:0) 1 − Φ( X ) T (Φ( X )Φ( X ) T ) − 1 Φ( X ) (cid:1) θ 0 (cid:107) 2 / √ N test and J ( X test ) :
=
J ( X test , 1) .
With this notation, we can show the following theorem:
Theorem 4 Let f NN be the fully-converged solution of a strongly overparametrized NN (1) with L layers and m hidden units per layer and initialization variance σ 2 , where all layers θ have been trained using gradient descent under squared loss on a consistent dataset D after random initialization θ 0 ∼ N (0 , 1) .
Then it holds with probability at least 1 − δ over random initialization (cid:112) L
[Equation]
For general datasets D , it is highly likely that J ( X test , σ ) (cid:54) = 0 .
To prove the statement we use the decomposition of Lemma 2.
Now instead of making the variance small to bound the second term we can use the homogeneity of Φ( x, σ ) to show that J ( X test , σ ) =
σ L J ( X test ) .
Now using twice the reverse triangle inequality we and the lemma 1 get the desired result.
(cid:3)
Now looking at the expression of Thm. 4 and using again the homogeneity of Φ we can see that (cid:112)
L inttest does not change when changing σ because the different factors cancel out and the term corresponding to the linear approximation should also not impact the result too much as long as we choose m large and do not increase σ signiﬁcantly.
Respecting this constraint we can now easily increase the variance to make the second term as large as we want.
Thus, by increasing σ we are able to increase the test loss arbitrarily while not changing the training error at all because the second term in our decomposition does not contribute during training time.
This shows that without considering an explicit initialization the training error of a NN might tell us very little about the actual test loss.
Thus, our results underline the importance of ﬁnding good initialization strategies.
E XPERIMENTS
In this section we give experimental evidence for our theoretical ﬁndings: the inﬂuence of NN initialization on the generalization performance of fully-trained wide NNs (Sect. 4) as well as the link to interpolating kernel methods in the ﬁrst place (Sect. 3).
We perform experiments (Fig. 1) in three different settings 3 : (1) A toy experiment, ﬁtting a 2Ddataset with N = 10 datapoints sampled uniformly from f ( x ) = exp( −(cid:107) x (cid:107) 2 ) , x ∈ [ − 1 , 1] 2 , with a NN of 1 hidden layer ( L = 2) with m = 10000 neurons; here, the “wide NN” requirement m (cid:29) N is satisﬁed to the highest degree.
Fitting N =
MNIST digits 0 and 1 by a single-hidden-layer network with m = 4000 hidden units.
In line with our framework and with other works on wide
NNs (e.g., (Du et al., 2018b; Arora et al., 2019b)), we ﬁt these classiﬁcation labels using squared error loss.
(3) Fitting MNIST digits 0 and 1 with 2 hidden layers ( L = 3) of m = 1500 neurons each.
The reason for using small training sets is to maintain the overparametrization limit m (cid:29) N .
In each setting we vary the variance σ 2 of the NN initialization, performing always 10 repetitions.
We stop training at the same “close to zero” training loss for all σ 2 , chosen in each setting so small that empirically the test loss remains almost constant (while the training loss is decreasing further).
The left panels (Fig. 1) show the test losses of the trained NN vs. the linearization f lin of the initialized NN (Lemma 1), at different NN initialization variances σ 2 .
Both curves are very close, conﬁrming the basic premise that the approximation of wide NNs by linear models is quite accurate for our NNs, even though we are not in the guaranteed overparametrized regime (e.g., m ≥ O ( N 6 ) from Du et al.
(2018b)).
The approximation is better for the two MNIST experiments, where we are actually further from the this regime, whereas the larger repetition variances for the 2D bump are due to sampling the training inputs from a small volume of low dimension.
Most strikingly, the test loss of the trained NN depends heavily on its weight initialization θ 0 even though all NNs have been trained to the same low training error.
This is shown by the increase at larger initialization variance σ 2 .
Via the link to f lin , this can be understood from the test behavior of strongly overparametrized linear models f lin with big initialization θ
(Eq.
(14) and Sect. 4.2).
The right panels underline this behavior predicted by our theory quantitatively:
The test error of trained ReLU-NNs as well our our analytical lower bound on it (Thm. 4) both behave at big σ 2 like Θ( σ 2 L ) (see also Eq.
(12)), with L the number of layers.
This in particular shows how one can ﬁnd natural examples of (overparametrized) NNs, i.e. initialized randomly and trained by gradient descent, that interpolate the training data perfectly, yet with arbitrarily large test error.
In the regime of small σ 2 on the other hand, the right panels conﬁrm Thm. 3, namely that the test error is close to the (low) test error of the interpolating kernel method f int Φ (Eq.
Note that its test error in the ﬁgures is almost constant in σ 2 , as f int does not depend directly on the weight initialization θ
but on the (many and random)
NN features φ ( x ) , which via the kernel Φ only determine function smoothness.
This is a more reasonable modeling behavior than that of wide NNs initialized at large σ 2 .
D ISCUSSION
In this paper we make the bias explicit that is imposed by gradient descent and random initialization on fully-connected strongly overparametrized NNs.
We achieve this by decomposing a gradient descent-trained network with (close to) zero training error into an interpolating kernel term and a second term that vanishes on training inputs but not on test inputs.
Because this second term is independent of the training labels but depends strongly on the initialization, it is favorable for good generalization to reduce its impact.
This can be achieved by choosing the variance of the NN initialization to be sufﬁciently small, as we have shown.
On the other hand, too small an initialization variance limits the expressivity of the NN and its features.
Our work furthers the practical understanding of modern NNs by utilizing recent theoretical advances in strongly overparametrized NNs.
In particular, we saw that the initialized weights are not merely a starting point for training, but signiﬁcantly inﬂuence the performance of the NN trained to [Figure]
At small σ 2 , on the other hand, the NN test error is close to that of the interpolating kernel method f int Φ (Eq.
(5)), conﬁrming Thm. 3.
zero empirical error. Similarly, focussing on the ﬁrst term of our decomposition, learning-theoretical results on interpolating kernel methods lead to a better understanding of puzzling features of NNs.
Our work provides intuition for the “essentially no barriers” phenomenon ﬁrst observed in Draxler et al.
; Garipov et al.
(2018): When scaling all initialization weights θ 0 by a common continuous factor (like σ ), all of the ﬁnally trained NNs f NN σ are connected continuously and their training error stays at (close to) zero.
This is because Φ changes by only a scalar factor σ 2 L , which continously changes the second (test) term in the linearization f NN ≈ f lin (Eq. 14) but cancels out of the ﬁrst (training) term.
It would be interesting for further work to investigate varying θ 0 more generally in the hypersurface of zero training loss, which induces non-scalar changes in Φ .
The connection of NNs with minimum complexity interpolating kernel methods opens up more directions for future research.
An important next step would be to go from strongly overparametrized
NNs to merely overparametrized NNs, where the (inﬁnitesimal) kernel dynamics effected by NN training is to be understood in addition to ﬁxed-kernel interpolation.
Further, the connection remains to be made explicit for CNNs and RNNs, as well as for other (classiﬁcation) loss functions, whose strict minimization often requires inﬁnite weights.
For SGD at least, instead of full gradient descent, the basic decomposition into a training part and its orthogonal component (Eq. 14) remains valid.
Another direction needed to fully leverage the link to NNs made explicit in Sect.
4.1, is to understand better the properties – e.g. the generalization behavior (Liang & Rakhlin, 2018) – of interpolating kernel methods, whose close connection to large modern-day NNs we have highlighted in this paper.
A A PPENDIX A M INIMUM 2-N ORM SOLUTION
Let us consider an afﬁne linear model ˆ f ( x ) =
Φ( x ) θ , where Φ( x ) is our feature map and θ our weights.
We want to choose the weights as a solution of the following problem:
[Equation]
subject to Y
[Equation]
Φ
[Equation]
First, we use the method of Lagrange multipliers to and deﬁne the following Lagrange function
[Equation]
Taking the derivative with respect to θ and setting it to zero gives us ˜ θ T =
λ T Φ( X ) .
Now plugging this into the Lagrangian we get
[Equation]
Minimizing this term with respect to λ we get λ
= (cid:0) Φ( X )Φ( X ) T (cid:1) − 1 Y .
Inserting this relation we get
[Equation]
B C ONNECTING THE INTERPOLATING MINIMUM 2-N ORM SOLUTION TO THE MINIMUM RKHS-N ORM INTERPOLATING KERNEL METHOD
Building upon the previous section we now want to show that a linear predictor where the weights are the solution of a minimum 2-norm interpolation with feature map Φ( x ) is equivalent to a minimum complexity interpolating kernel method with
[Equation]
were the kernel of the RKHS is given by K ( x, y )
= (cid:104) Φ( x ) , Φ( y ) (cid:105) and Φ is the same function used for the linear ansatz in the minimum 2-norm solution.
We can ﬁnd the solution of the minimum complexity interpolating kernel method by minimizing the following Lagrangian:
[Equation]
Because the RKHS-Norm is bounded by deﬁnition (for non degenerate data) we can use the representer theorem ( f = (cid:80) Ni {+=
 α i K+} ( x i , . ) with α i ∈ R ) and the reproducibility of the kernel K to arrive at
[Equation]
Now taking the derivative with respect to α we get α = λ .
Inserting this into our Lagrangian we get
[Equation]
This Lagrangian is equivalent to the Lagrangian of the minimal two norm solution in the dual representation and due to the convexity of the problem the solutions must also be equivalent.
C L EMMA
In this section we want to give the proof for lemma 2.
Let f NN be a fully-connected NN as deﬁned in the notation section.
For simplicity we switch from the NTK initialization to the He et al.
(2015) initialization.
The proof is equivalent in both parametrizations. The only difference is that we can suppress some of the scaling factors in the initialization of the weights.
We again assume a ReLUactivation function.
Before showing the actual lemma we will ﬁrst show the following expression which we will use to proof the lemma
[Equation]
were the scalar product is deﬁned as the Hilbert-Schmid-product with (cid:104) A, B (cid:105) = tr (cid:2)
A T B (cid:3) and
W = diag ( W L , ..., W 1 ) is the diagonal matrix with all the weigh matrices on the diagonal.
We restrict ourselves to networks without biases but the proof with biases is almost identical.
To prove the lemma we can ﬁrst proof that ∀ l ∈ { 1 , ..., L } we have f NNW l ( x ) [Equation] l f ( x ) (cid:105) .
If this statement is true it is easy to see that the formula above is also true.
A ReLU Network can be written in the following way
[Equation]
where l h l ( x ) ≥ 0 is the diagonal matrix with the step function on the diagonal corresponding to the components of h l ( x ) ≥ 0 .
Now, we can deﬁne a ( W L : l , x ) :=
W L l h
[Equation]
[Equation]
W L
[Equation]
1 ... l h l ( x ) ≥ 0 and b ( W l :1 , x ) : = l h l − 1 ( x ) ≥ 0 W l − 1 ... to get
[Equation]
Using the fact that ∂a
[Equation]
= 0 because we deﬁne ∂ ( I hl ( x )=0 ) ij ∂ ( W l ) mn = 0 .
Deﬁning the derivative of the stepfunction at the step to be zero is also done in most of the standard frameworks like tensorﬂow. Additionally, the region where the stepfunction jumps is a d − 1 dimensional subspace and thus the expression holds almost everywhere anyway.
Now, since our weights do not move much and only a neglectable number change the step function we can just deﬁne this derivative to be zero.
The derivative of b ( W l :1 , x ) with respect to W l is clearly zero because it does not depend on W l .
Using this we can see that almost everywhere
[Equation]
Applying this result L -times we have get that almost everywhere f NN ( x )
=
1 L (cid:104) W, ∇ W f ( x ) (cid:105) .
Thus, we get almost everywhere
[Equation]
and therefore also
[Equation]
D L EMMA
In this section we want to show the statement of lemma 3.
We want to show that
[Equation]
First, let us focus on the operator (cid:0)
[Equation]
The second term has the properties of a projection operator (symmetric and equal to its square).
Thus, all the eigenvalues are either zero or one.
The operator projects M -dimensional vectors into a N -dimensional subspace and then back to the M -dimensional space.
Thus, it has N eigenvalues which are one and all the rest are zero.
Now, looking at the whole operator we see that it has M − N eigenvalues which are one and the rest are zero.
Therefore, we can neglect the projection operator because this will only make our result larger due to the neglected N dimensions which would be projected out.
Using this and the formula we have derived in th proof for lemma 2 we get almost everywhere
[Equation]
Now, we can use Markov’s inequality to arrive at
[Equation]
Thus, the only thing that is left to do is to show that E (cid:2)
f NN θ 0 ( x ) (cid:3) ≤ σ 2 L .
To start of we can again write the Neural Network as a product of matrices
[Equation]
We use a similar approach as Du & Hu (2019) and generalize this approach to non-linear networks.
We make use of the fact that for for a random matrix A ∈ R d 1 × d 2 with i.i.d.
N (0 , 1) entries and a arbitrary non-zero vector v ∈ R d 1 / 0 the distribution of (cid:107) Av (cid:107) 2 (cid:107) v (cid:107) 2 is χ 2 d 2 distributed.
Additionally, it is easy to see that
[Equation]
we thus get
[Equation]
Inserting the result for E [ f 0 ( x, θ 0 )] we get
[Equation]
E D ETAILS ON OUR E XPERIMENTS
In this section we want to talk about how we conducted the experiments.
Since the experimental settings are not very complicated this should be more than enough to easily reproduce our results.
As described in the experimental section we used three different setting to investigate the dependence of the Neural Network solution on the initialization variance.
We used a standard He et al.
(2015)- initialization which is, as mention in the notation section, equivalent to the NTK-parametrization and standard normal initialization when we rescale the learning rate.
Since we choose the theoretical value of η = 1 . 9 λ max (Ψ( X )Ψ( X ) T ) for the learning rate the settings are equivalent.
For all experiments we use Tensorﬂow with ReLU-activation functions, set the bias initially to zero and train with GD.
For all networks we use a low number of training data points to make sure that we are in or close to the over-parametrized regime.
This is necessary to assure the validity of the linearisation.
The ﬁrst Network for the 2D exponential Bump is a single hidden layer NN with 10000 hidden units.
We use 10 training data points and 300 testing points.
We train 10 networks for each variance value and until the training loss drops ﬁrst bellow a loss of 10 − 5 .
The ﬁnal training loss of the curves plotted in the experimental section can be seen in ﬁgure 2.
We choose the stopping point for the training error in such a way that even though the training loss is still decreasing further the test loss is almost constant.
[Figure]
For the single layer MNIST network we use 4000 hidden units and for the two layer network two layers of each a 1500 hidden units.
We use 100 training samples of 0 and 1 labelled examples of the training set and 100 samples from the test set to calculate the test error.
For the MNIST Network we train the networks to reach a training error smaller than 10 − 4 .
The plot of the ﬁnal training errors that corresponds to the curves in the Experimental section are shown in ﬁgure 3.
[Figure]