I MPLICIT B IAS OF MSE G RADIENT O PTIMIZATION IN U NDERPARAMETERIZED N EURAL N ETWORKS
A BSTRACT
We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient ﬂow.
We show that in the underparameterized regime the network learns eigenfunctions of an integral operator T K ∞ determined by the Neural Tangent Kernel (NTK) at rates corresponding to their eigenvalues.
For example, for uniformly distributed data on the sphere S d − 1 and rotation invariant weight distributions, the eigenfunctions of T K ∞ are the spherical harmonics.
Our results can be understood as describing a spectral bias in the underparameterized regime.
The proofs use the concept of “Damped Deviations”, where deviations of the NTK matter less for eigendirections with large eigenvalues due to the occurence of a damping factor.
Aside from the underparameterized regime, the damped deviations point-of-view can be used to track the dynamics of the empirical risk in the overparameterized setting, allowing us to extend certain results in the literature.
We conclude that damped deviations offers a simple and unifying perspective of the dynamics when optimizing the squared error.
I NTRODUCTION
A surprising but well established empirical fact is that neural networks optimized by gradient descent can ﬁnd solutions to the empirical risk minimization (ERM) problem that generalize.
This is surprising from an optimization point-of-view because the ERM problem induced by neural networks is nonconvex (Sontag & Sussmann, 1989; 1991) and can even be NP-Complete in certain cases (Blum & Rivest, 1993).
Perhaps even more surprising is that the discovered solution can generalize even when the network is able to ﬁt arbitrary labels (Zhang et al., 2017), rendering traditional complexity measures such as Rademacher complexity inadequate.
How does deep learning succeed in the face of pathological behavior by the standards of classical optimization and statistical learning theory?
Towards addressing generalization, a modern line of thought that has emerged is that gradient descent performs implicit regularization, limiting the solutions one encounters in practice to a favorable subset of the model’s full capacity (see, e.g., Neyshabur et al., 2015; 2017; Gunasekar et al., 2017;
Wu et al., 2017).
An empirical observation is that neural networks optimized by gradient descent tend to ﬁt the low frequencies of the target function ﬁrst, and only pick up the higher frequencies later in training (Rahaman et al., 2019; Ronen et al., 2019; Basri et al., 2020; Xu et al., 2019).
A closely related theme is gradient descent’s bias towards smoothness for regression problems (Williams et al., 2019; Jin & Mont´ufar, 2021).
For classiﬁcation problems, in suitable settings gradient descent provably selects max-margin solutions (Soudry et al., 2018; Ji & Telgarsky, 2019).
Gradient descent is not impartial, thus understanding its bias is an important program in modern deep learning.
Generalization concerns aside, the fact that gradient descent can succeed in a nonconvex optimization landscape warrants attention on its own.
A brilliant insight made by Jacot et al.
(2018) is that in function space the neural network follows a kernel gradient descent with respect to the “Neural
Tangent Kernel” (NTK).
This kernel captures how the parameterization biases the trajectory in function space, an abstraction that allows one to largely ignore parameter space and its complications.
This is a proﬁtable point-of-view, but there is a caveat.
The NTK still depends on the evolution of the network parameters throughout time, and thus is in general time-dependent and complicated to analyze.
However, under appropriate scaling of the parameters in the inﬁnite-width limit it remains constant (Jacot et al., 2018).
Once the NTK matrix has small enough deviations to remain strictly positive deﬁnite throughout training, the optimization dynamics start to become comparable to that of a linear model (Lee et al., 2019).
For wide networks (quadratic or higher polynomial dependence on the number of training data samples n and other parameters) this property holds and this has been used by a variety of works to prove global convergence guarantees for the optimization (Du et al., 2019b;
Oymak & Soltanolkotabi, 2020; Du et al., 2019a; Allen-Zhu et al., 2019a;b; Zou et al., 2020; Zou & Gu, 2019; Song & Yang, 2020; Dukler et al., 2020) 1 and to characterize the solution throughout time (Arora et al., 2019; Basri et al., 2020).
The NTK has been so heavily exploited in this setting that it has become synonymous with polynomially wide networks where the NTK is strictly positive deﬁnite throughout training.
This begs the question, to what extent is the NTK informative outside this regime?
While the NTK has hitherto been associated with the heavily overparameterized regime, we demonstrate that reﬁned analysis is possible in the underparameterized setting.
Our theorems primarily concern a one-hidden layer network, however unlike many NTK results appearing in the literature our network has biases and both layers are trained.
In fact, the machinery we build is strong enough to extend some existing results in the overparameterized regime appearing in the literature to the case of training both layers.
1. R ELATED W ORK
There has been a deluge of works on the Neural Tangent Kernel since it was introduced by Jacot et al.
(2018), and thus we do our best to provide a partial list.
Global convergence guarantees for the optimization, and to a lesser extent generalization, for networks polynomially wide in the number of training samples n and other parameters has been addressed in several works (Du et al., 2019b; Oymak & Soltanolkotabi, 2020; Du et al., 2019a; Allen-Zhu et al., 2019a;b; Zou et al., 2020;
Zou & Gu, 2019; Song & Yang, 2020; Arora et al., 2019).
To our knowledge, for the regression problem with arbitrary labels, quadratic overparameterization m (cid:38) n 2 is state-of-the art (Oymak & Soltanolkotabi, 2020; Song & Yang, 2020; Nguyen & Mondelli, 2020).
E et al.
(2020) gave a fairly comprehensive study of optimization and generalization of shallow networks trained under the standard parameterization.
Under the standard parameterization, changes in the outer layer weights are more signiﬁcant, whereas under the NTK parameterization both layers have roughly equal effect.
Since we study the NTK parameterization in this work, we view the analysis as complementary.
Our work is perhaps most closely connected with Arora et al.
In Theorem 4.1 in that work they showed that for a shallow network in the polynomially overparameterized regime m (cid:38) n 7 , the training error along eigendirections of the NTK matrix decay linearly at rates that correspond to their eigenvalues.
Our main Theorem 3.5 can be viewed as an analogous statement for the actual risk (not the empirical risk) in the underparameterized regime: eigenfunctions of the NTK integral operator
T K ∞ are approximately learned linearly at rates that correspond to their eigenvalues.
In contrast with
Arora et al.
(2019), we have that the requirements on width m and number of samples n required to learn eigenfunctions with large eigenvalues are smaller compared to those with small eigenvalues.
Surprisingly the machinery we build is also strong enough to prove in our setting the direct analog of Theorem 4.1.
Note that Arora et al.
(2019) train the hidden layer of a ReLU network via gradient descent, whereas we are training both layers with biases for a network with smooth activations via gradient ﬂow.
Due to the different settings, the results are not directly comparable.
This important detail notwithstanding, our overparameterization requirement ignoring logarithmic factors is smaller by a factor of n 2 dδ 4 where n is the number of input samples, d is the input dimension, and δ is the failure probability.
Basri et al.
(2020) extended Theorem 4.1 in Arora et al.
(2019) to deep ReLU networks without bias where the ﬁrst and last layer are ﬁxed, with a higher overparameterization requirement than the original (Arora et al., 2019).
Since the ﬁrst and last layers are ﬁxed this cannot be specialized to get a guarantee for training both layers of a shallow network even with ReLU activations.
Although it was not our focus, the tools to prove Theorem 3.5 are enough to prove analogs of
Theorem 4 and Corollary 2 in the work of Su & Yang (2019).
Theorem 4 and Corollary 2 of Su &
Yang (2019) are empirical risk guarantees that show that for target functions that participate mostly in the top eigendirections of the NTK integral operator T K ∞ , moderate overparameterization is possible.
Again in this work they train the hidden layer of a ReLU network via gradient descent, whereas we are training both layers with biases for a network with smooth activations via gradient ﬂow.
Again due to the different settings, we emphasize the results are not directly comparable.
In our results the bounds and requirements are comparable to Su & Yang (2019), with neither appearing better.
Nevertheless we think it is important to demonstrate that these results hold for training both layers with biases, and we hope our “Damped Deviations” approach will simplify the interpretation of the aforementioned works.
Cao et al.
(2020, Theorem 4.2) provide an analogous statement to our Theorem 3.5 if you replace our quantities with their empirical counterparts.
While our statement concerns the projections of the test residual onto the eigenfunctions of an operator associated with the Neural Tangent Kernel, their statement concerns the inner products of the empirical residual with those eigenfunctions.
Their work was a crucial step towards explaining the spectral bias from gradient descent, however we view the difference between tracking the empirical quantities versus the actual quantities to be highly nontrivial.
Another difference is they consider a ReLU network whereas we consider smooth activations; also they consider gradient descent versus we consider gradient ﬂow.
Due to the different settings we would like to emphasize that the scalings of the different parameters are not directly comparable, nevertheless the networks they consider are signiﬁcantly wider.
They require at least m ≥ ˜ O
(max { σ − 14 k , (cid:15) − 6 } ) , where σ k is a cutoff eigenvalue and (cid:15) is the error tolerance.
By contrast in our work, to have the projection onto the top k eigenvectors be bounded by epsilon in L norm requires m = ˜Ω( σ − 4 k (cid:15) − 2 ) .
Another detail is their network has no bias whereas ours does.
1. O UR C ONTRIBUTIONS
The key idea for our work is the concept of “Damped Deviatons”, the fact that for the squared error deviations of the NTK are softened by a damping factor, with large eigendirections being damped the most.
This enables the following results.
• In Theorem 3.5 we characterize the bias of the neural network to learn the eigenfunctions of the integral operator T K ∞ associated with the Neural Tangent Kernel (NTK) at rates proportional to the corresponding eigenvalues.
• In Theorem 3.7 we show that in the overparameterized setting the training error along different directions can be sharply characterized, showing that Theorem 4.1 in Arora et al.
(2019) holds for smooth activations when training both layers with a smaller overparameterization requirement.
• In Theorem 3.8 and Corollary 3.9 we show that moderate overparameterization is sufﬁcient for solving the ERM problem when the target function has a compact representation in terms of eigenfunctions of T K ∞ .
This extends the results in Su & Yang (2019) to the setting of training both layers with smooth activations.
G RADIENT D YNAMICS AND D AMPED D EVIATIONS 2. N OTATIONS
We will use (cid:107)•(cid:107) 2 and (cid:104)• , •(cid:105) 2 to denote the L 2 norm and inner product respectively (for vectors or for functions depending on context).
For a symmetric matrix
A ∈ R k × k , λ i ( A ) denotes its i th largest eigenvalue, i.e. λ 1 ( A ) ≥ λ 2 ( A ) ≥ · · · ≥ λ k ( A ) .
For a matrix A , (cid:107) A (cid:107) op :
=
sup (cid:107) x
(cid:107) 2 ≤ 1 (cid:107)
Ax (cid:107)is the operator norm induced by the Euclidean norm.
We will let (cid:104)• , •(cid:105) R n denote the standard inner product on R n normalized by 1 n , namely (cid:104) x, y (cid:105) R n = 1 n (cid:104) x, y (cid:105) 2 = 1 n (cid:80) ni
=1 x i y i .
We will let (cid:107) x (cid:107) R n = (cid:112) (cid:104) x, x (cid:105) R n be the associated norm.
This normalized inner product has the convenient property that if v ∈ R n such that v i
= O (1) for each i then (cid:107) v (cid:107) R n = O (1) , where by contrast (cid:107) v (cid:107) 2 =
O ( √ n ) .
This is convenient as we will often consider what happens when n → ∞ .
(cid:107)•(cid:107) ∞ will denote the supremum norm with associated space L ∞ .
We will use the standard big O and Ω notation with ˜ O and ˜ Ω hiding logarithmic terms.
2. G RADIENT D YNAMICS A ND T HE NTK I NTEGRAL O PERATOR
We will let f ( x ; θ ) denote our neural network taking input x ∈ R d and parameterized by θ ∈ R p .
The speciﬁc architecture of the network does not matter for the purposes of this section.
Our training data consists of n input-label pairs { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } where x i ∈ R d and y i ∈ R .
We focus on the setting where the labels are generated from a ﬁxed target function f ∗ , i.e. y i =
f ∗ ( x i ) .
We will concatenate the labels into a label vector y ∈ R n , i.e. y i =
f ∗ ( x i ) .
We will let ˆ r ( θ ) ∈ R n be the vector whose i th entry is equal to f ( x i ; θ ) − f ∗ ( x i ) .
Hence ˆ r ( θ ) is the residual vector that measures the difference between our neural networks predictions and the labels.
We will be concerned with optimizing the squared loss
[Equation]
Optimization will be done by gradient ﬂow
[Equation]
which is the continuous time analog of gradient descent.
We will denote the residual at time t , ˆ r ( θ t ) , as ˆ r t for the sake of brevity and similarly we will let f t ( x ) =
f ( x ; θ t ) .
We will let r t ( x ) :
= f t ( x ) − f ∗ ( x ) denote the residual off of the training set for an arbitrary input x .
We quickly recall some facts about the Neural Tangent Kernel and its connection to the gradient dynamics.
For a comprehensive tutorial we suggest Jacot et al.
The analytical NTK is the kernel given by
[Equation]
where the expectation is taken with respect to the parameter initialization for θ .
We associate K ∞ with the integral operator T K ∞ :
L 2 ρ ( X ) →
L 2 ρ ( X ) deﬁned by
[Equation]
where X is our input space with probability measure ρ .
Our training data x i ∈
X are distributed according to this measure x i ∼ ρ .
By Mercer’s theorem we can decompose
[Equation]
where { φ i } ni =1 is an orthonormal basis of L 2 , { σ i } ∞
i =1 is a nonincreasing sequence of positive values, and each φ i is an eigenfunction of T K ∞ with eigenvalue
σ i > 0 .
When X = S d − 1 is the unit sphere, ρ is the uniform distribution, and the weights of the network are from a rotation invariant distribution (e.g. standard Gaussian), { φ i } ∞ i =1 are the spherical harmonics (which in d = 2 is the
Fourier basis) due to K ∞ being rotation-invariant (see Bullins et al., 2018, Theorem 2.2).
We will let κ :=
max x ∈ X K ∞ ( x, x ) which will be a relevant quantity in our later theorems.
In our setting κ will always be ﬁnite as K ∞ will be continuous and X will be bounded.
The training data inputs { x 1 , . . .
, x n } induce a discretization of the integral operator T K ∞ , namely
[Equation]
where ρ n = 1 n (cid:80) ni
=
δ x i is the empirical measure.
We recall the deﬁnition of the time-dependent
NTK 2 ,
[Equation]
We can look at the version of T n corresponding to K t , namely
[Equation]
We recall that the residual r t ( x ) :
= f
( x ; θ ) − f ∗ ( x ) follows the update rule
[Equation]
We will let ( H t ) i,j :=
K t ( x i , x j ) and H ∞ i,j := K ∞ ( x i , x j ) denote the Gram matrices induced by these kernels and we will let G t := 1 n H t and G ∞ :
= 1 n H ∞ be their normalized versions 3 .
Throughout we will let u 1 , . .
, u n denote the eigenvectors of G ∞ with corresponding eigenvalues λ 1 , . . .
, λ n .
The u 1 , . . .
, u n are chosen to be orthonormal with respect to the inner product (cid:104)• , •(cid:105) R n .
When restricted to the training set we have the update rule
[Equation]
2. D AMPED D EVIATIONS
The concept of damped deviations comes from the very simple lemma that follows (the proof is provided in Appendix D).
The lemma compares the dynamics of the residual ˆ r ( t ) on the training set to the dynamics of an arbitrary kernel regression exp( − Gt )
ˆ r (0) :
Lemma 2.1.
Let G ∈ R n × n be an arbitrary positive semideﬁnite matrix and let G s be the time dependent NTK matrix at time s .
Then
[Equation]
Let’s specialize the lemma to the case where G = G ∞ .
In this case the ﬁrst term is exp( − G ∞ t )ˆ r 0 , which is exactly the dynamics of the residual in the exact NTK regime when G t =
G ∞ for all t .
The second term is a correction term that weights the NTK deviations ( G ∞ − G s ) by the damping factor exp( − G ∞ ( t − s )) .
We see that damping is largest along the large eigendirections of G ∞ .
The equation becomes most interpretable when projected along a speciﬁc eigenvector.
Fix an eigenvector u i of G ∞ corresponding to eigenvalue λ
i .
Then the equation along this component becomes
[Equation]
The ﬁrst term above converges to zero at rate λ i .
The second term is a correction term that weights the deviatiations of the NTK matrix G s from G ∞ by the damping factor exp( − λ i ( t − s )) .
The second term can be upper bounded by
[Equation]
where we have used the property (cid:107) ˆ r s (cid:107) R n ≤ (cid:107) ˆ r 0 (cid:107) R n from gradient ﬂow.
When f ∗ =
O (1) we have that (cid:107) ˆ r 0 (cid:107) R n =
O (1) , thus whenever (cid:107) G ∞ − G s (cid:107) op is small relative to λ
i this term is negligible.
It has been identiﬁed that the NTK matrices tend to have a small number of outlier large eigenvalues and exhibit a low rank structure (Oymak et al., 2020; Arora et al., 2019).
In light of this, the dependence of the above bound on the magnitude of λ i is particularly interesting.
We reach following important conclusion.
Observation 2.2.
The dynamics in function space will be similar to the NTK regime dynamics along eigendirections whose eigenvalues are large relative to the deviations of the time-dependent
NTK matrix from the analytical NTK matrix.
The equation in Lemma 2.1 concerns the residual restricted to the training set, but we will be interested in the residual for arbitrary inputs.
Recall that r t ( x ) =
f
( x ; θ t ) − f ∗ ( x ) denotes the residual at time t for an arbitrary input.
Then more generally we have the following damped deviations lemma for the whole residual (proved in Appendix C.3).
Lemma 2.3.
Let K ( x, x (cid:48) ) be an arbitrary continuous, symmetric, positive-deﬁnite kernel.
Let [ T K h ]( • ) =
(cid:82) X K ( • , s ) h ( s ) dρ ( s ) be the integral operator associated with K and let [ T sn h ]( • )
=
n (cid:80) ni
=1 K s ( • , x i ) h ( x i ) denote the operator associated with the time-dependent NTK K s .
Then
[Equation]
where the equality is in the L 2 sense.
For our main results we will specialize the above lemma to the case where K = K ∞ .
However there are other natural kernels to compare against, say K 0 or the kernel corresponding to some subset of parameters.
We will elaborate further on this point after we introduce the main theorem.
When specializing Lemma 2 .
3 to the case K = K ∞ , we have that T K ∞ and T sn are the operator analogs of
G ∞ and G s respectively.
From this statement the same concepts holds as before, the dynamics of r t will be similar to that of exp( − T K ∞ t ) r 0 along eigendirections whose eigenvalues are large relative to the deviations ( T K ∞ − T sn ) .
In the underparameterized regime we can bound the second term and make it negligible (Theorem 3.5) and thus demonstrate that the eigenfunctions φ i of T K ∞ with eigenvalues σ i will be learned at rate σ i .
When the input data are distributed uniformly on the sphere
S d − 1 and the network weights are from a rotation-invariant distribution, the eigenfunctions of T K ∞ are the spherical harmonics (which is the Fourier basis when d = 2 ).
In this case the network is biased towards learning the spherical harmonics that correspond to large eigenvalues of T K ∞ .
It is in this vein that we will demonstrate a spectral bias.
M AIN R ESULTS
Our theorems will concern the shallow neural network
[Equation]
where W ∈ R m × d , a, b ∈ R m and b 0 ∈ R and w (cid:96) =
W (cid:96), : denotes the (cid:96) th row of W and σ : R → R is applied entry-wise.
θ =
( a T , vec ( W ) T , b T , b 0 )
T ∈ R p where p = md
+ 2 m + 1 is the total number of parameters.
Here we are utilizing the NTK parameterization (Jacot et al., 2018).
For a thorough analysis using the standard parameterization we suggest E et al.
We will consider two parameter initialization schemes.
The ﬁrst initializes W i,j (0) ∼ W , b (cid:96) (0) ∼ B , a (cid:96) (0) ∼ A , b 0 ∼ B (cid:48) i.i.d., where W , B , A , B (cid:48) represent zero-mean unit variance subgaussian distributions.
In the second initialization scheme we initialize the parameters according to the ﬁrst scheme and then perform the following swaps W (0) → (cid:20) W (0)
W (0) (cid:21) , b (0) → (cid:20) b (0) b (0) (cid:21) , a (0) → (cid:20) a (0) − a (0) (cid:21) , b 0 → 0 and replace the 1 √ m factor in the parameterization with 1 √ 2 m .
This is called the “doubling trick” (Chizat et al., 2019; Zhang et al., 2020) and ensures that the network is identically zero f ( x ; θ 0 ) ≡ 0 at initialization.
We will explicitly state where we use the second scheme and otherwise will be using the ﬁrst scheme.
The following assumptions will persist throughout the rest of the paper:
Assumption 3.1.
σ is a C 2 function satisfying
[Equation]
Assumption 3.2.
The inputs satisfy
(cid:107) x (cid:107) 2 ≤ M .
The following assumptions will be used in most, but not all theorems.
We will explicitly state when they apply.
Assumption 3.3.
The input domain X is compact with strictly positive Borel measure ρ .
Assumption 3.4.
T K ∞ is strictly positive, i.e., (cid:104) f, T K ∞ f (cid:105) 2 > 0 for f (cid:54) = 0 .
Most activation functions other than ReLU satisfy Assumption 3.1, such as Softplus σ ( x ) =
ln(1 + e x ) , Sigmoid σ ( x ) = 1 1+ e − x , and Tanh σ ( x )
=
e x − e − x e x + e − x .
Assumption 3.2 is a mild assumption which is satisﬁed for instance for RGB images and has been commonly used (Du et al., 2019b;a;
Oymak & Soltanolkotabi, 2020).
Assumption 3.3 is so that Mercer’s decomposition holds, which is often assumed implicitly.
Assumption 3.4 is again a mild assumption that is satisﬁed for a broad family of parameter initializations (e.g. Gaussian) anytime σ is not a polynomial function, as we will show in Appendix G. Assumption 3.4 is not strictly necessary but it simpliﬁes the presentation by ensuring T K ∞ has no zero eigenvalues.
We will track most constants that depend on parameters of our theorems such as M , the activation function σ , and the target function f ∗ .
However, constants appearing in concentration inequalities such as Hoeffding’s or Bernstein’s inequality or constants arising from δ/ 2 or δ/ 3 arguments will not be tracked.
We will reserve c, C > 0 for untracked constants whose precise meaning can vary from statement to statement.
In the proofs in the appendix it will be explicit which constants are involved.
3. U NDERPARAMETERIZED R EGIME
Our main result compares the dynamics of the residual r t ( x ) =
f ( x ; θ t ) − f ∗ ( x ) to that of exp( − T K ∞ t ) r 0 in the underparameterized setting.
Note that (cid:104) exp( − T K ∞ t ) r 0 , φ i (cid:105) 2 = exp( − σ i t ) (cid:104) r 0 , φ i (cid:105) 2 , thus exp( − T K ∞ t ) r 0 learns the eigenfunctions φ i of T K ∞ at rate σ i .
Therefore exp( − T K ∞ t ) r 0 exhibits a bias to learn the eigenfunctions of T K ∞ corresponding to large eigenvalues more quickly.
To our knowledge no one has been able to rigorously relate the dynamics in function space of the residual r t to exp( − T K ∞ t ) r 0 , although that seems to be what is suggested by
Ronen et al.
; Basri et al.
The existing works we are aware of (Arora et al., 2019;
Basri et al., 2020; Cao et al., 2020) characterize the bias of the empirical residual primarily in the heavily overparameterized regime (Cao et al.
(2020) stands out as requiring wide but not necessarily overparameterized networks).
By contrast, we characterize the bias of the whole residual in the underparameterized regime.
Theorem 3.5.
Assume that Assumptions 3.3 and 3.4 hold.
Let P k be the orthogonal projection in
L 2 onto span { φ 1 , . .
. , φ k } and let D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } .
If we are doing the doubling trick set S (cid:48)
=
and otherwise set S (cid:48)
=
O (cid:18)(cid:113)
˜
O
( d ) + log( c/δ ) (cid:19) ,
[Equation]
Also let T > 0 .
Assume m ≥ D 2 (cid:107) y (cid:107) 2 R n T 2 , and m ≥ O
(log( c/δ )+ ˜ O ( d )) max (cid:8) T 2 , 1 (cid:9) .
Then with probability at least 1 − δ we have that for all t ≤ T and k ∈ N (cid:107) P k
( r t − exp( − T K ∞ t ) r 0 ) (cid:107) 2 ≤ 1 − exp(
[Equation]
σ k t ) σ k
[Equation]
O (cid:32)
[Equation]
tS
[Equation]
[Equation]
(cid:33) and (cid:107) r t − exp( − T K ∞ t ) r 0 (cid:107) 2 ≤
[Equation]
˜ O (cid:32)
[Equation] [Equation] [Equation]
(cid:33) .
Theorem 3.5 will be proved in Appendix C.
The proof uses the uniform deviation bounds for the
NTK to bound T n − T sn and tools from empirical process theory to show convergence of T n to T K ∞ uniformly over a class of functions corresponding to networks with bounded parameter norms.
To interpret the results, we observe that to track the dynamics for eigenfunctions corresponding to eigenvalue σ k and above, the expression under the ˜ O needs to be small relative to 1 σ k .
Thus the bias towards learning the eigenfunctions corresponding to large eigenvalues appears more pronounced.
When t = log( (cid:107) r 0 (cid:107) 2 /(cid:15) )
/σ
k , we have that (cid:107) P k exp( − T K ∞ t ) r 0 (cid:107) 2 ≤ (cid:15) .
Thus by applying this stopping time we get that to learn the eigenfunctions corresponding to eigenvalue σ k and above up to (cid:15) accuracy we need t 2 √ m (cid:46) (cid:15) and t 2 √ p √ n (cid:46) (cid:15) which translates to m (cid:38) σ − 4 k (cid:15) − 2 and n (cid:38) pσ − 4 k (cid:15) − 2 .
In typical NTK works the width m needs to be polynomially large relative to the number of samples n , where by contrast here the width depends on the inverse of the eigenvalues for the relevant components of the target function.
From an approximation point-of-view this makes sense; the more complicated the target function the more expressive the model must be.
We believe future works can adopt more precise requirements on the width m that do not require growth relative to the number of samples n .
To further illustrate the scaling of the parameters required by Theorem 3.5, we can apply Theorem 3.5 for an appropriate stopping time to get a bound on the test error.
Corollary 3.6.
Assume Assumptions 3.3 and 3.4 hold.
Suppose that f ∗ =
O (1) and assume we are performing the doubling trick where f 0 ≡
so that r 0 = − f ∗ .
Let k ∈ N and let P k be the orthogonal projection onto span { φ 1 , . .
. , φ k } .
Set t = log(
[Equation]
k
Then we have that m = ˜Ω( d(cid:15)σ 4 k ) and n = ˜Ω (cid:16) pσ 4 k
[Equation]
(cid:17) sufﬁces to ensure with probability at least 1 − δ
[Equation]
If one specialized to the case where f ∗ is a ﬁnite sum of eigenfunctions of T K ∞ (when the data is uniformly distributed on the sphere S d − 1 and the network weights are from a rotation invariant distribution this corresponds to a ﬁnite sum of spherical harmonics, which in d = 2 is equivalently a bandlimited function) one can choose k such that (cid:107) ( I − P k ) f ∗ (cid:107) 22 = 0 .
It is interesting to note that in this special case gradient ﬂow with early stopping achieves essentially the same rates with respect to m and n (up to constants and logarithms) as the estimated network in the classical approximation theory paper by Barron (1994).
It is also interesting to note that the approximation results by Barron (1994) depend on the decay in frequency domain of the target function f ∗ via their constant C f ∗ , and similarly for us the constant 1 /σ 4 k grows with the bandwidth of the target function in the case of uniform distribution on the sphere S 1 which we mentioned parenthetically above.
While in Theorem 3.5 we compared the dynamics of r t against that of exp( − T K ∞ t ) r 0 , the damped deviations equation given by Lemma 2.3 enables you to compare against exp( − T K t ) r 0 for an arbitrary kernel K .
There are other natural choices for K besides K = K ∞ , the most obvious being
K = K 0 .
In Appendix C.8 we prove a version of Theorem 3.5 where K = K 0 and θ 0 is an arbitrary deterministic parameter initialization.
This could be interesting in scenarios where the parameters are initialized from a pretrained network or one has a priori knowledge that informs the selection of θ 0 .
One could let K be the kernel corresponding to some subset of parameters, such as the random feature kernel (Rahimi & Recht, 2008b) corresponding to the outer layer.
This would compare the dynamics of training all layers to that of training a subset of the parameters.
If one wanted to account for adaptations of the kernel K t one could try to set K
= K t 0 for some t 0 > 0 .
However since θ t 0 depends on the training data it is not obvious how one could produce a bound for T
sn − K t 0 .
Nevertheless we leave the suggestion open as a possibility for future work.
3. O VERPARAMETERIZED R EGIME
Once one has deviation bounds for the NTK so that the quantity (cid:107) G ∞ − G s (cid:107) op is controlled, the damped deviations equation (Lemma 2.1) allows one to control the dynamics of the empirical risk.
In this section we will demonstrate three such results that follow from this approach.
The following is our analog of Theorem 4.1 from Arora et al.
(2019) in our setting, proved in Appendix E.
The result demonstrates that when the network is heavily overparameterized, the dynamics of the residual ˆ r t follow the NTK regime dynamics exp( − G ∞ t )ˆ r 0 .
Theorem 3.7.
Assume m = ˜Ω( dn 5 (cid:15) − 2 λ n ( H ∞ ) − 4 ) and m ≥ O
(log( c/δ ) + ˜ O ( d )) and f ∗ =
O (1) .
Assume we are performing the doubling trick so that ˆ r 0 = − y .
Let v 1 , . .
. , v n denote the eigenvectors of G ∞ normalized to have unit L2 norm (cid:107) v i (cid:107) 2 = 1 .
Then with probability at least 1 − δ
[Equation]
, where sup t ≥ 0 (cid:107) δ ( t ) (cid:107) 2 ≤ (cid:15) .
In particular (cid:107) ˆ r t (cid:107) 2 = (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = exp( − 2 λ i t ) |(cid:104)
y, v i (cid:105) 2 | 2 ± (cid:15).
In the work of Arora et al.
the requirement is m = Ω( n 7 λ n ( H ∞ ) 4 κ
δ 4 (cid:15) 2 ) and κ = O ( (cid:15)δ √ n ) where w (cid:96) ∼ N (0 , κ 2 I ) (not to be confused with our deﬁnition of κ := max x K ∞ ( x, x ) ).
By contrast our weights have unit variance, which for Gaussian initialization corresponds to w (cid:96) ∼ N (0 , I ) .
They require κ to be small to ensure the neural network is small in magnitude at initializeation.
To achieve the same effect we can perform antisymmetric initializeation to ensure the network is equivalentlyat initializeation.
Our overparameterization requirement ignoring logarithmic factors is smaller by a factor of n 2 dδ 4 .
Again due to the different settings we do not claim superiority over this work.
The following is our analog of Theorem 4 by Su & Yang (2019) proved in Appendix F.
This shows that when the target function has a compact representation in terms of eigenfunctions of T K ∞ , a more moderate overparametrization is sufﬁcient to approximately solve the ERM problem.
Theorem 3.8.
Assume Assumptions 3.3 and 3.4 hold.
Furthermore assume m = ˜Ω (cid:16) (cid:15) − 2 dT 2 (cid:107) f ∗ (cid:107) 2 ∞ (1 + T (cid:107) f ∗ (cid:107) ∞ ) 2 (cid:17) where T > 0 is a time parameter and m ≥ O
(log( c/δ ) + ˜ O ( d )) and n ≥ 128 κ 2 log(2 /δ )
( σ k − σ k +1 ) 2 .
Also assume f ∗ ∈
L ∞ ( X ) ⊂
L 2 ( X ) and let P T K ∞ be the orthogonal projection onto the eigenspaces of T K ∞ corresponding to the eigenvalue α ∈ σ ( T K ∞ ) and higher.
Assume that (cid:13)(cid:13)
( I − P T K ∞ )
f ∗ (cid:13)(cid:13) ∞ ≤ (cid:15) (cid:48) for some (cid:15) (cid:48) ≥ 0 .
Pick k so that σ k = α and σ k +1 <
α , i.e. k is the index of the last repeated eigenvalue corresponding to α in the ordered sequence { σ i } i .
Also assume we are performing the doubling trick so
that ˆ r (0) = − y .
Then we have with probability at least 1 − 3 δ over the sampling of x 1 , . . .
, x n and θ 0 that for t ≤ T (cid:107) ˆ r t (cid:107)
[Equation]
n
[Equation]
[Equation]
) (cid:107) y (cid:107) R n
[Equation]
(cid:107) f [Equation] (cid:107) [Equation]
+ (cid:15).
Su & Yang (2019) have (cid:107) f ∗ (cid:107) 2 ≤ (cid:107) f ∗ (cid:107) ∞ ≤ 1 , κ ≤ 12 and they treat d as a constant.
Taking these into account we do not see the overparameterization requirements or bounds of either work being superior to the other.
From Theorem 3.8, setting (cid:15) = 4 κ (cid:107) f ∗ (cid:107)√ 10log(2 /δ )
( σ k − σ k +1 ) √ n and (cid:15) (cid:48)
=
we immediately get the analog of Corollary 2 in the work of Su & Yang (2019).
This explains how in the special case that the target function is a ﬁnite sum of eigenfunctions of T K ∞ , the width m and the number of samples n can grow at the same rate, up to logarithms, and still solve the ERM problem.
This is an ERM guarantee for m = ˜Ω( n ) and thus attains moderate overparameterization.
Corollary 3.9.
Assume Assumptions 3.3 and 3.4 hold.
Furhtermore assume m =
[Equation]
Let f ∗ , P T K ∞ , and k be the same as in the hypothesis of Theorem 3.8.
Furthermore assume that (cid:13)(cid:13)
( I − P [Equation] K ∞ )
f ∗ (cid:13)(cid:13) ∞
= 0 .
Also assume we are performing the doubling trick so that ˆ r (0) = − y .
Set T = log( √ n (cid:107)
ˆ r (0) (cid:107) R n ) /λ k .
Then we have with probability at least 1 − 3 δ over the sampling of x 1 , . . .
, x n and θ 0 that for t ≤ T
[Equation]
Note Su & Yang (2019) are training only the hidden layer of a ReLU network by gradient descent, by contrast we are training both layers with biases of a network with smooth activations by gradient ﬂow.
For Corollary 2 by Su & Yang (2019) they have the overparameterization requirement m (cid:38) n log n (cid:16) 1 λ 4 k + log 4 n log 2 (1 /δ )
( λ k − λ k +1 ) 2 n 2 λ 4 k (cid:17) .
Thus both bounds scale like nλ 4 k .
Our bound has the extra factor ( σ k − σ k +1 ) 2 in front which could make it appear smaller at ﬁrst glance but their Theorem 4 is strong enough to include this factor in the corollary they just chose not to.
Thus we view both overparameterization requirements as comparable with neither superior to the other.
C ONCLUSION AND F UTURE D IRECTIONS
The damped deviations equation allows one to compare the dynamics when optimizing the squared error to that of an arbitrary kernel regression.
We showed how this simple equation can be used to track the dynamics of the test residual in the underparameterized regime and extend existing results in the overparameterized setting.
In the underparameterized setting the neural network learns eigenfunctions of the integral operator T K ∞ determined by the Neural Tangent Kernel at rates corresponding to their eigenvalues.
In the overparameterized setting the damped deviations equation combined with NTK deviation bounds allows one to track the dynamics of the empirical risk.
In this fashion we extended existing work to the setting of a network with smooth activations where all parameters are trained as in practice.
We hope damped deviations offers a simple interpretation of the MSE dynamics and encourages others to compare against other kernels in future work.
A CKNOWLEDGMENTS
Benjamin Bowman was at the Max Planck Institute for Mathematics in the Sciences while working on parts of this project.
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no 757983)
A PPENDIX A A DDITIONAL N OTATIONS
We let [ k ] := { 1 , 2 ,
. , k } .
For a set A we let | A | denote its cardinality.
(cid:107)•(cid:107)
F denotes the
Frobenius norm for matrices and for two matrices A, B ∈ R n × m we will let (cid:104) A, B (cid:105) =
Tr ( A T B ) =
(cid:80) ni
=1 (cid:80) mj
=
A i,j B i j denote the Frobenius or entry-wise inner product.
We will let B R :=
{ x : (cid:107) x (cid:107) 2 ≤ R } to be the Euclidean ball of radius R > 0 .
B NTK D EVIATION AND P ARAMETER N ORM B OUNDS
Let Γ > 1 .
At the end of this section we will prove a high probability bound of the form sup ( x
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Ideally we would like to use the results in Huang & Yau (2020) where they prove for a deep feedforward network without biases:
[Equation]
However there are three problems that prevent this.
The ﬁrst is that the have a constant under the ˜ O above that depends on the training data.
Speciﬁcally their Assumption 2.2 requires that the smallest singular value of the data matrix [ x α 1 , . . .
, x α r ] is greater than c r > 0 where 1 ≤ α 1 , . . .
, α r ≤ n are arbitrary distinct indices.
As you send the number of samples to inﬁnity you will have c r → 0 , thus it is not clear how the bound will scale in the large sample regime.
The second is that their bound only holds on the training data, whereas we need a bound that is uniform over all inputs.
The ﬁnal one is their network does not have biases.
In the following section we will overcome these issues.
The main difference between our argument and theirs is how we prove convergence at initialization.
In their argument for convergence at initialization they make repeated use of a Gaussian conditioning lemma as they pass through the layers, and this relies on their Assumption 2.2.
By contrast we will use Lipschitzness of the NTK and convergence over an (cid:15) net to prove convergence at initialization.
As we see it, our deviation bounds for the time derivative
∂ t K t are proved in a very similar fashion and the rest of the argument is very much inspired by their approach.
At the time of submission of this manuscript, we were made aware of the work by Liu et al.
(2020) that provides an alternative uniform NTK deviation bound by providing a uniform bound on the operator norm of the Hessian.
Their work is very nice, and it opens the door to extending the results of this paper to the other architectures they consider.
Nevertheless, we proceed with our original analysis below.
This section is conceptually simple but technical.
We will take care to outline the high level structure of each section to prevent the technicalities from obfuscating the overall simplicity of the approach.
Our argument runs through the following steps:
• Control parameter norms throughout training.
• Bound the Lipschitz constant of the NTK with respect to spatial inputs.
• Use concentration of subexponential random variables (Bernstein’s inequality) to show that | K ∞ ( z (cid:48) )
− K
( z (cid:48) )
| =
˜ O (1 / √ m ) (roughly) for all z (cid:48) in an (cid:15) net of the spatial domain.
Combine with the Lipschitz property of the NTK to show convergence over all inputs, namely sup z ∈ B M | K ∞ ( z ) − K
( z ) | =
˜ O (1 / √ m ) (roughly).
• Produce the bound sup z ∈ B M × B M | ∂ t K t ( z ) | =
˜ O (1 / √ m ) (roughly).
• Conclude that sup z ∈ B M × B M | K t ( z ) − K
( z ) | =
˜ O ( t/ √ m ) (roughly).
B. I MPORTANT E QUATIONS
The following list contains the equations that are relevant for this section.
We found it easier to read the following proofs by keeping these equations on a separate piece of paper or in a separate tab.
We write a ⊗ x = ax T .
Also throughout this section the training data will be considered ﬁxed and thus the randomness of the inputs is not relevant to this section.
The randomness will come entirely from the parameter initialization θ 0 .
[Equation]
[Table]
B. A P RIORI P ARAMETER N ORM B OUNDS
In this section we will provide bounds for the following quantities:
[Equation]
Here w (cid:96) =
W (cid:96), : ∈ R d is the vector of input weights to the (cid:96) th unit.
These quantities appear repeatedly throughout the rest of the proofs of this section and thus need to be controlled.
The parameter norm bounds will also be useful for the purpose of the covering number argument in
Section C.6.
This section is broken down as follows:
• Prove Lemma B. • Bound ξ ( t )
• Bound ˜ ξ ( t )
The time derivatives throughout will repeatedly be of the form 1 n (cid:80) {+ni =
+} ˆ r
i ( t ) v i .
Lemma B.1 provides a simple bound that we will use over and over again.
Lemma B.1.
Let (cid:107)•(cid:107) be any norm over a vector space V .
Then for any v 1 , . . .
, v n ∈ V we have
[Equation]
Proof.
Note that
[Equation]
(cid:13)(cid:13)(cid:13) n n (cid:88) i = ˆ r i ( t ) v i (cid:13) (cid:13)
[Equation]
last inequality
(cid:107)
ˆ r
[Equation]
t )
[Equation]
R n ≤
[Equation]
ˆ r
[Equation]
0) (cid:107) R n gradient ﬂow
[Equation]
We now proceed to bound ξ ( t ) .
Lemma B.2.
Let ξ ( t )
=
max { 1 √ m (cid:107) W ( t ) (cid:107) op , 1 √ m (cid:107) b ( t )
[Equation]
2 , 1 √ m (cid:107) a ( t ) (cid:107) 2 , 1 } and
[Equation]
Then for any initial conditions W (0) , b (0) , a (0) we have for all t
[Equation]
Proof.
Recall that
[Equation]
We will show that each of the above derivatives is (cid:46) (cid:107)
ˆ r ( t ) (cid:107) R n ξ ( t ) then apply Gr¨onwall’s inequality.
By Lemma B.1 it sufﬁces to show that the terms multiplied by ˆ r i in the above sums are (cid:46) ξ ( t ) .
First we note that
[Equation]
Second we have that
[Equation]
Finally we have that
[Equation]
Thus by Lemma B.1 and the above bounds we have
[Equation]
Let v ( t ) be a placeholder for one of the functions 1 √ m a ( t ) , 1 √ m W ( t ) , 1 √ m b ( t ) with corresponding norm (cid:107)•(cid:107) [Equation]
Then we have that
[Equation]
This inequality holds for any of the three choices of v thus we get that
[Equation]
Therefore by Gr¨onwall’s inequality we get that
[Equation]
D √ m (cid:107) ˆ r (0) (cid:107) R n t (cid:19) ξ (0) .
We will now bound ˜ ξ ( t ) using essentially the same argument as in the previous lemma.
Lemma B.3.
Let ˜ ξ ( t ) = max { max (cid:96) ∈
[ m ] (cid:107) w (cid:96) ( t ) (cid:107) 2 , (cid:107) a ( t ) (cid:107) ∞ , (cid:107) b ( t ) (cid:107) ∞ , 1 } and
[Equation]
Then for any initial conditions W (0) , b (0) , a (0) we have for all t
[Equation]
Proof.
The proof is basically the same as Lemma B.2.
We have that
[Equation]
Now note
[Equation]
√ m
[Equation]
Thus by Lemma B.1 we have that
[Equation]
On the other hand
[Equation]
with
[Equation]
Thus again by Lemma B.1 we have
[Equation]
Finally we have
[Equation]
with
[Equation]
Again applying Lemma B.1 one last time we get
[Equation]
Therefore by the same argument as in Lemma B.2 using Gr¨onwall’s inequality we get that
[Equation]
B. NTK I S L IPSCHITZ W ITH R ESPECT T O S PATIAL I NPUTS
The NTK being Lipschitz with respect to spatial inputs is essential to our proof.
The Lipschitz property means that to show convergence uniformly for all inputs it sufﬁces to show convergence on an (cid:15) net of the spatial domain.
Since the parameters are changing throughout time, the Lipschitz constant of the NTK will change throughout time.
We will see that the Lipschitz constant depends on the quantities ξ ( t ) and ˜ ξ ( t ) from the previous Section B.2.
The NTK K t ( x, x (cid:48) ) is a sum of terms of the form g ( x ) T g ( x (cid:48) ) where g is one of the derivatives ∂ a f ( x ; θ t ) , ∂ b f ( x ; θ t ) , ∂
W f ( x ; θ t ) , ∂ b 0 f ( x ; θ t ) .
Since ∂ b 0 f ( x ; θ t ) ≡ 1 this term can be ignored for the rest of the section.
The upcomming Lemma B.4 shows that if g is Lipschitz and bounded then ( x, x (cid:48) )
(cid:55)→ g ( x ) T g ( x (cid:48) ) is Lipschitz.
This lemma guides the structure of this section:
• Prove Lemma B
• Show that ∂ a f ( x ; θ ) , ∂ b f ( x ; θ ) , ∂ W f ( x ; θ ) are bounded and Lipschitz • Conclude the NTK is Lipschitz
Lemma B.4.
Let g : R k
→ R l be L -Lipschitz with respect to the 2-norm, i.e.
(cid:107) g
( x ) − g ( z ) (cid:107) 2 ≤ L (cid:107) x − z (cid:107)and satisfy (cid:107) g ( x ) (cid:107) 2 ≤ M for all x in some set X .
Then K g :
X × X → R
K g ( x, x (cid:48) ) :
= g ( x ) T g
[Equation]
is ML -Lipschitz with respect to the norm
[Equation]
Proof.
We have
[Equation]
By the previous Lemma B.4, to show that the NTK is Lipschitz it sufﬁces to show that ∂ a f ( x ; θ ) , ∂ b f ( x ; θ ) , ∂ W f ( x ; θ ) , ∂ b 0 f
( x ; θ ) are bounded and Lipschitz.
The following lemma bounds the norms of the derivatives ∂ a f ( x ; θ ) , ∂ W f ( x ; θ ) , ∂ b f ( x ; θ ) .
Lemma B.5.
Let D = 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } and ξ = max { 1 √ m (cid:107) W (cid:107) op , 1 √ m (cid:107) b (cid:107) 2 , 1 √ m (cid:107) a (cid:107) 2 , 1 } .
Then (cid:107) ∂ a f ( x ; θ ) (cid:107) 2 , (cid:107) ∂
W f ( x ; θ ) (cid:107) F , (cid:107) ∂
b f ( x ; θ ) (cid:107) 2 ≤ Dξ.
Proof.
We have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
The following lemma demonstrates that ∂ a f ( x ; θ ) , ∂ W f ( x ; θ ) , and ∂ b f ( x ; θ ) are Lipschitz as functions of the input x .
Lemma B.6.
Let
[Equation]
Then ∂ a f ( x ; θ ) , ∂ b f ( x ; θ ) , ∂ W f ( x ; θ ) are all L -Lipschitz with respect to the Euclidean norm (cid:107)•(cid:107) 2 .
In symbols:
[Equation]
Proof.
We have
[Equation]
[Table]
Finally we can prove that the Neural Tangent Kernel is Lipschitz.
Theorem B.7.
Consider the Neural Tangent Kernel
[Equation]
and let
[Equation]
Then the Neural Tangent Kernel is Lipschitz with respect to the norm
[Equation]
with Lipschitz constant L := 6 DD (cid:48) ξ 2 ˜ ξ .
In symbols:
[Equation]
Proof.
By Lemma B.5, we have that the gradients are bounded
[Equation]
Also by Lemma B.6 the gradients are Lipschitz with Lipschitz constant 2 ξ ˜ ξD (cid:48) .
Thus these two facts combined with Lemma B.4 tell us that each of the three terms (cid:104) ∂ a f ( x ; θ ) , ∂ a f ( y ; θ ) (cid:105) , (cid:104) ∂
b f ( x ; θ ) , ∂ b f ( y ; θ ) (cid:105) , and (cid:104)
∂ W f ( x ; θ ) , ∂ W f ( y ; θ ) (cid:105) are individually Lipschitz with constant ( Dξ ) · (2 ξ ˜ ξD (cid:48) ) .
Thus the Lipschitz constant of the NTK itself is bounded by the sum of the 3 Lipschitz constants, for a total of 6 DD (cid:48) ξ 2 ˜ ξ .
Using that the NTK at time zero K 0 ( x, y ) is Lipschitz we can prove that the analytical NTK
K ∞ =
E [ K 0 ( x, y )] is Lipschitz.
We will use this primarily as a qualitative statement, meaning that the estimate that we derive for the Lipschitz constant will not be used as it is not very explicit.
Rather, in theorems where we use the fact that K ∞ is Lipschitz we will simply take the Lipschitz constant of K ∞ as an external parameter.
Theorem B.8.
Assume that W i,j (0) ∼ W , b (cid:96) (0) ∼ B , a (cid:96) (0) ∼ A are all i.i.d.
zero-mean, unit variance subgaussian random variables.
Let
[Equation]
[Equation]
Then the analytical Neural Tangent Kernel K ∞ ( x, y ) =
E [ K 0 ( x, y )] is Lipschitz with respect to the norm
[Equation]
with Lipschitz constant ≤ 6 DD (cid:48) E [ ξ 2 ˜ ξ ] < ∞ .
If one instead does the doubling trick then the same conclusion holds.
Proof.
First assume we are not doing the doubling trick.
We note that
[Equation]
) (cid:107) where the last line follows from the Lipschitzness of K 0 provided by Theorem B.7.
Using that (cid:107) W (0) (cid:107) op ≤ (cid:107) W (0) (cid:107) F and the fact that the Euclidean norm of a vector with i.i.d.
subgaussian entries is subgaussian (Vershynin, 2018, Theorem 3.1.1), we have that ξ (0) and ˜ ξ (0) are maximums of subgaussian random variables.
Since a maximum of subgaussian random variables is subgaussian, we have that ξ (0) and ˜ ξ (0) are subgaussian.
From the inequality ab ≤ 12 ( a 2 + b 2 ) we get
E [ ξ 2 ˜ ξ ] ≤ 12 E [ ξ 4 ] + 12 E [˜ ξ 2 ] < ∞ since moments of subgaussian random variables are all ﬁnite.
Since the doubling trick does not change the distribution of K 0 , the same conclusion holds under that initialization scheme.
B. NTK C ONVERGENCE A T I NITIALIZATION
In this section we prove that sup z ∈ B M × B M | K 0 ( z ) − K ∞ ( z ) | =
˜ O (1 / √ m ) .
Our argument traces the following steps:
• Show that K 0 is sum of averages of m independent subexponential random variables • Use subexponential concentration to show that sup z (cid:48) ∈ ∆ | K 0 ( z (cid:48) )
− K ∞ ( z (cid:48) )
| =
˜ O (1 / √ m ) for all z (cid:48) in an (cid:15) net ∆ of B M × B M • Use that K 0 is Lipschitz and convergence over the epsilon net ∆ to show that
[Equation]
We recall the following deﬁnitions 2.5.6 and 2.7.5 from Vershynin (2018).
Deﬁnition B.9.
(Vershynin 2018)
Let Y be a random variable.
Then we deﬁne the subgaussian norm of Y to be
[Equation]
If (cid:107) Y (cid:107) ψ 2 < ∞
, then we say Y is subgaussian.
Deﬁnition B.10.
(Vershynin 2018)
Let Y be a random variable.
Then we deﬁne the subexponential norm of Y to be
[Equation]
If (cid:107) Y (cid:107) ψ 1 < ∞ , then we say Y is subexponential.
We also recall the following useful lemma Vershynin (2018, Lemma 2.7.7).
Lemma B.11.
(Vershynin 2018)
Let X and Y be subgaussian random variables.
Then XY is subexponential.
Moreover
[Equation]
We recall one last deﬁnition Vershynin (2018, Deﬁnition 3.4.1)
Deﬁnition B.12.
(Vershynin 2018)
A random vector Y ∈
R k is called subgaussian if the one dimensional marginals (cid:104) Y, x (cid:105) are subgaussian random variables for all x ∈ R k .
The subgaussian norm of Y is deﬁned as
[Equation]
The typical example of a subgaussian random vector is a random vector with independent subgaussian coordinates.
The following lemma demonstrates that the NTK at initializeation is a sum of terms that are averages of independent subexponential random variables, which will enable us to use concentration arguments later.
Theorem B.13.
Let w (cid:96) , b (cid:96) , a (cid:96) all be independent subgaussian random variables with subgaussian norms satisfying (cid:107)•(cid:107) ψ 2 ≤ K .
Furthermore assume (cid:107) 1 (cid:107) ψ 2 ≤ K .
Also let
[Equation]
Then for ﬁxed x, y , each of the following
[Equation]
is an average of m independent subexponential random variables with subexponential norms bounded by D 2 K 2 .
Proof.
We ﬁrst observe that
[Equation]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
Note that
[Equation] [Table]
(cid:107)
[Equation]
Also
[Equation]
therefore
[Equation]
Finally
[Equation]
ψ 2 ≤ M (cid:107) σ (cid:48) (cid:107) ∞ (cid:107) a (cid:96) (cid:107) ψ 2 ≤ DK.
It follows by Lemma B.11 that each of (cid:104) ∂ a f ( x ; θ ) , ∂ a f ( y ; θ ) (cid:105) , (cid:104)
∂
W f ( x ; θ ) , ∂ W f
( y ; θ ) (cid:105) , and (cid:104) ∂
b f ( x ; θ ) , ∂ b f ( y ; θ ) (cid:105) is an average of m independent subexponential random variables with subexponential norm (cid:107)•(cid:107) ψ 1 ≤ D 2 K 2 .
We now recall the following Theorem from Vershynin (2012, Theorem 5.39) which will be useful.
Lemma B.14 (Vershynin 2012) .
Let A be an N × n matrix whose rows A i are independent subgaussian isotropic random vectors in R n .
Then for every t ≥ 0 , with probability at least 1 − 2 exp( − ct 2 ) one has the following bounds on the singular values
[Equation]
Here C =
C K > 0 depends only on the subgaussian norms K = max
i (cid:107) A
i (cid:107) ψ 2 of the rows.
Also the following special case of Vershynin (2012, Lemma 5.5) will be useful for us.
Lemma B.15 (Vershynin 2012) .
Let Y be subgaussian.
Then
[Equation]
It will be useful to remind the reader that C, c > 0 denote absolute constants whose meaning will vary from statement-to-statement, as this abuse of notation becomes especially prevalent during the concentration of measure arguments of the rest of the section.
The following lemma provides a concentration inequality for the maximum of subgaussian random variables which will be useful for bounding ξ and ˜ ξ later which is necessary for bounding the Lipschitz constant of K 0 .
Lemma B.16.
Let Y 1 , . . .
, Y n be subgaussian random variables with (cid:107) Y
i (cid:107) ψ 2 ≤ K for i ∈ [ n ] .
Then there exists absolute constants c, c (cid:48) , C > 0 such that
[Equation]
Proof.
Since each Y i is subgaussian we have for any t ≥
(Lemma B.15)
[Equation]
By the union bound,
[Equation]
Thus by setting c (cid:48) :
= c − 1 we get the desired result.
We now introduce a high probability bound for ξ .
Lemma B.17.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) w (cid:96)
[Equation]
ψ
[Equation]
b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let
[Equation]
Then with probability at least 1 − δ
[Equation]
+
[Equation]
m
Proof.
Note that by setting t =
(cid:112) c − 1 log(2 /δ ) in Lemma B.14 we have that with probability at least 1 − δ
[Equation]
Also by Theorem 3.1.1 in (Vershynin, 2018)
[Equation]
Thus by Lemma B.15 and a union bound we have with probability at least 1 − 2 δ
[Equation]
Thus by replacing every δ in the above arguments with δ/ 3 and using the union bound we have with probability at least 1 − δ
[Equation]
Similarly we now introduce a high probability bound for ˜ ξ .
Lemma B.18.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let
[Equation]
Then with probability at least 1 − δ we have
[Equation]
[Table]
Proof.
By Theorem 3.1.1 in (Vershynin, 2018)
we have
[Equation]
Well then by Lemma B
[Equation]
16 there is a constant c (cid:48) >
so that
[Equation]
Thus by setting t = CK 2 (cid:112) log( c/δ ) we have with probability at least 1 − δ
[Equation]
where we have absorbed the constant √ c (cid:48) into C .
Similarly by Lemma B.16 and a union bound we get with probability at least 1 − 2 δ that
[Equation]
Thus by replacing each δ with δ/ 3 in the above arguments and using the union bound we get with probability at least 1 − δ
[Equation]
We are now ﬁnally ready to prove the main theorem of this section.
Theorem B.19.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let
[Equation]
Deﬁne
[Equation]
Let L ( K ∞ ) denote the Lipschitz constant of K ∞ .
If
[Equation]
then with probability at least 1 − δ
[Equation]
If one instead does the doubling trick then the same conclusion holds.
Proof.
First assume we are not doing the doubling trick.
Recall that by Theorem B.7 that K 0 is
Lipschitz with constant at most
[Equation]
where ξ and ˜ ξ are deﬁned as in the theorem.
Well then by Lemmas B.17, B.18 and a union bound we have with probability at least 1 − 2 δ
[Equation]
Let L ( K 0 ) , L ( K ∞ ) denote the Lipschitz constant of K 0 and K ∞ respectively.
Then assuming the above inequality holds we have that
[Equation]
For conciseness from now on we will suppress the arguments of ρ .
Now set
[Equation]
Let N γ ( B M ) be the cardinality of a maximal γ -net of the ball B M = { x : (cid:107) x (cid:107) 2 ≤ M } with respect to the L2 norm (cid:107)•(cid:107) 2 .
By a standard volume argument we have that
[Equation]
By taking the product of two γ/ 2 nets of B M it follows that we can choose a γ net of B M × B M , say ∆ , with respect to the norm
[Equation]
such that
[Equation]
By Theorem B.13 for ( x, y ) ∈ B M × B M ﬁxed each of the following (cid:104) ∂ a f ( x ; θ ) , ∂ a f ( y ; θ ) (cid:105) 2 , (cid:104) ∂ b f ( x ; θ ) , ∂ b f ( y ; θ ) (cid:105) 2 , (cid:104) [Equation] ( x
[Equation]
θ )
[Equation] ∂ W [Equation] ( y
[Equation]
θ ) (cid:105) is an average of m subexponential random variables with subexponential norm at most D 2 K 2 .
Therefore separately from the randomness discussed before by Bernstein’s inequality Vershynin (2018, Theorem 2.8.1) and a union bound we have
[Equation]
Thus for t ≤ D 2 K 2 we have
[Equation]
Then by a union bound and the previous inequality we have that for t ≤ D 2 K
[Equation]
Thus by setting t = CD 2 K
[Equation]
(note that the condition on m in the hypothesis ensures that t ≤ D 2 K 2 ) we get that with probability 1 − δ
[Equation]
Now ﬁx z ∈ B M × B M and choose z
(cid:48) ∈
∆ such that (cid:107)
z − z (cid:48) (cid:107) ≤ γ .
Then
[Equation]
Note that this argument runs through for any z ∈ B M × B M therefore
[Equation]
) | ≤ 2 max { L ( K 0 ) , L ( K ∞ ) } γ + t.
Well by replacing δ with δ/ 3 in the previous arguments by taking a union bound we can assume that equations (1) and (2) hold simultaneously.
In which case
[Equation]
where we have used the deﬁnition of M γ in the second-to-last equality and the deﬁnition of γ in the last equality.
Since the doubling trick does not change the distribution of K 0 , the same conclusion holds under that initialization scheme.
We immediately get the following corollary.
Corollary B.20.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a [Equation] (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Then
[Equation]
sufﬁces to ensure that with probability at least 1 − δ
[Equation]
If one instead does the doubling trick then the same conclusion holds.
B. C ONTROL OF N ETWORK A T I NITIALIZATION
Many of our previous results depend on the quantity (cid:107)
ˆ r (0) (cid:107) R n which depends on the network at initialization.
Before we proceed we must control the inﬁnity norm of the network at initialization and work out a few consequences of this.
The following lemma controls (cid:107) f
( • ; θ 0 ) (cid:107) ∞ .
Lemma B.21.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A , b 0 ∼ B (cid:48) are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) 1 (cid:107) ψ 2 , (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤
K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let
[Equation]
Assume that
[Equation]
Then with probability at least 1 − δ sup
[Equation]
∈
[Equation]
Proof.
First we note that
[Equation]
where ξ (0) is deﬁned as in Lemma B.
[Equation]
Thus f ( • ; θ 0 ) is Lipschitz with constant L = √ m (cid:107) σ (cid:48) (cid:107) ∞
ξ (0) 2 .
Well then by Lemma B.17 we have with probability at least 1 − δ
[Equation]
When the above holds we have that f ( • ; θ 0 ) is Lipschitz with constant
[Equation]
On the other hand note that
[Equation]
Thus
[Equation]
Therefore by Lemma B.11 we have
[Equation]
Thus for each x ﬁxed we have by Bernstein’s inequality Vershynin (2018, Theorem 2.8.1)
[Equation]
Thus for t ≤ √ mDK 2 this simpliﬁes to
[Equation]
Let ∆ be a γ net of the ball B M = { x : (cid:107) x (cid:107) 2 ≤ M } with respect to the Euclidean (cid:107)•(cid:107) 2 norm.
Then by a standard volume argument we have that
[Equation]
Thus by a union bound we have for t ≤ √ mDK
[Equation]
Thus by setting t = CDK 2 (cid:112) log( c M γ /δ )
assuming t ≤ √ mDK 2 we have with probability at least 1 − δ
[Equation]
On the other hand by Lemma B.15 our prior deﬁnition of t is large enough (up to a redeﬁnition of the constants c, C ) to ensure that with probability at least 1 − δ | b 0 | ≤ t.
When (4) and (5) hold simultaneously we have that max x
(cid:48) ∈ ∆ | f
( x (cid:48) , θ 0 )
| ≤ 2 t .
By a union bound we have with probability at least 1 − 3 δ that (3), (4), (5) hold simultaneously.
Well then for any x ∈ B M we may choose x (cid:48) ∈ ∆
so that (cid:107) x − x (cid:48) (cid:107) 2 ≤ γ .
Then
[Equation]
2 t + Lγ.
Therefore
[Equation]
and this argument runs through for any γ > 0 .
We will set γ = 1 /L .
Note that for this choice of γ the hypothesis on m ensures that t ≤ √ mDK 2 .
Thus the preceding argument goes through in this case.
Thus by replacing δ with δ/ 3 in the previous argument we get the desired conclusion up to a redeﬁnition of c, C .
We quickly introduce the following lemma.
Lemma B.22.
[Equation]
Proof.
Let ˆ y ∈ R n be the vector whose i th entry is equal to f ( x i ; θ 0 ) .
Well then note that (cid:107) ˆ y (cid:107) R n ≤ (cid:107) f ( • ; θ 0 ) (cid:107) ∞ .
Therefore
[Equation]
Finally we prove one last lemma that will be useful later.
Lemma B.23.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) 1 (cid:107) ψ 2 , (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let Γ > 1 , D = 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } , L ( m, [Equation] [Equation]
:=
[Equation]
∞
[Equation]
Suppose
[Equation]
m ≥
[Equation]
[Equation]
(cid:17)
[Equation]
Then with probability at least 1 − δ
[Equation]
where ξ ( t ) and ˜ ξ ( t ) are deﬁned as in Lemmas B.2 and B.3.
If one instead does the doubling trick the second hypothesis on m can be removed and the conclusion holds with probability 1 .
Proof.
First assume we are not doing the doubling trick.
Well from the condition m ≥ (cid:0) ρDK 2 (cid:1) 2 we have by Lemma B.21 that with probability at least 1 − δ sup
[Equation]
Also by Lemma B.22 and the above bound we have
[Equation]
Well in this case
[Equation]
where we have used the hypothesis on m in the last inequality.
Therefore by Lemmas B.2, B.3 we have in this case that
[Equation]
Now assume we are performing the doubling trick
so that f ( • ; θ 0 ) ≡ 0 .
Then ρ in the previous argument can simply be replaced with zero and the same argument runs through, except using Lemma
B.21 is no longer necessary (and thus the second hypothesis on m is not needed).
Without using Lemma B.21 the whole argument is deterministic so that the conclusion holds with probability 1 .
B. NTK T IME D EVIATIONS B OUNDS
In this section we bound the deviations of the NTK throughout time.
This section runs through the following steps
• Bound sup ( x,y ) ∈
B M × B M | ∂ t K t ( x, y )
| • Bound sup ( x
,y ) ∈
B M × B M | K t ( x, y ) − K 0 ( x, y )
|
In the following lemma we will provide an upper bound on the NTK derivative sup x,y ∈ B M × B M | ∂ t K t ( x, y ) | .
Lemma B
[Equation]
Let
[Equation]
Then for any initial conditions W (0) , b (0) , a (0) we have for all t
[Equation]
Proof.
We need to bound the following time derivatives
[Equation]
Note that
[Equation]
Thus by Lemma B.
[Equation]
On the other hand
[Equation]
Well
[Equation]
Thus by Lemma B.1 we have that
[Equation]
Finally we have
[Equation]
Well
[Equation]
Thus we ﬁnally by Lemma B [Equation] 1 again we get that
[Equation]
It follows that
[Equation]
and similarly
[Equation]
Thus in total we can say
[Equation]
It thus follows by the chain rule and Lemma B.5 that
[Equation]
Using the previous lemma we can now bound the deviations of the NTK .
Theorem B.25.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let Γ > 1 and T > 0 be positive constants,
[Equation]
and assume
[Equation]
y (cid:107) 2 R n T[log(Γ)]and m ≥ max (cid:40) 4 D 2 O (log( c/δ ) + ˜ O ( d )) T[log(Γ)], O (log( c/δ ) + ˜ O ( d )) (cid:41) .
Then with probability at least 1 − δ
[Equation]
If one instead does the doubling trick then the second condition on m can be removed from the hypothesis and the same conclusion holds.
Proof.
First assume we are not doing the doubling trick.
By Lemmas B.17, B.18 and a union bound we have with probability at least 1 − δ
[Equation]
Note that ρ as deﬁned in Lemma B.23 satisﬁes ρ 2 =
O (log( c/δ )+ ˜ O ( d )) .
Thus the hypothesis on m is strong enough to apply Lemma B.23, therefore by applying this lemma we have with probability at least 1 −
[Equation]
Thus by replacing δ with δ/ 2 and taking a union bound we have that with probability at least 1 − δ (6) and (7) hold simultaneously.
Then using Lemma B.24 and the fact that (cid:107) ˆ r ( t ) (cid:107) R n ≤ (cid:107) ˆ r
[Equation]
[Equation] n we have for t ≤ T
[Equation]
Therefore by the fundamental theorem of calculus for t ≤ T
[Equation]
Now consider if one instead does the doubling trick where one does the following swaps W (0) → (cid:20) W (0) W
[Equation]
[Equation]
b (
[Equation]
(cid:21)
[Equation]
a (
[Equation]
→ (cid:20)
[Equation] [Equation]
− a (
[Equation]
(cid:21) and m → 2 m where W (0) , b (0) , and a (0) are initialized as before.
Then ξ (0) and ˜ ξ (0) do not change.
We can then run through the same exact proof as before except when we apply Lemma B.23 the second hypothesis on m is no longer needed.
Theorem B.26.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Let Γ > 1 and T > 0 be positive constants and let D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } . Assume m [Equation] D
[Equation] (cid:107) y (cid:107) [Equation] T[log(Γ)]and m ≥ max
[Equation]
log
[Equation]
[Equation]
Then with probability at least 1 − δ we have for t ≤ T
[Equation]
If one instead does the doubling trick then one can remove the assumption m ≥
[Equation]
and have the same conclusion hold.
Proof.
The condition m ≥ O (log( c/δ )
+ ˜ O
( d )) is sufﬁcient to satisfy the hypothesis of Theorem
B.19.
The condition on m also immediately satisﬁes the hypothesis of Theorem B.25.
The desired result then follows from a union bound.
Theorem B.27.
Under the same assumptions as Theorem B.26 we have that with probability at least 1 − δ for all t ≤ T
[Equation]
Proof.
Recall that for a matrix A ∈ R m × n (cid:107) A (cid:107) op ≤ √ mn max i,j |
A
i,j | .
Thus by Theorem B. with probability at least 1 − δ
[Equation]
The second bound follows from G s = 1 n H s and G ∞ = 1 n H ∞ .
B. NTK D EVIATIONS F OR R E LU A PPROXIMATIONS
The NTK deviation bounds given in the previous subsections assumed (cid:107) σ (cid:48)(cid:48) (cid:107) ∞ < ∞ .
For ReLU this assumption is not satisﬁed.
It is natural to ask to what extent we might expect the results to hold when the activation function is σ ( x ) = ReLU ( x ) = max { 0 , x } .
The closest we can get to ReLU without modifying the proofs is to use the Softmax approximation to ReLU, namely σ ( x ) = 1 α ln(1 + exp( αx )) , and consider what happens as α → ∞ .
For this choice of σ we have that (cid:107) σ (cid:48)(cid:48) (cid:107) ∞
=
O ( α ) .
In Subsection B.6 where you will pay the biggest penalty is in Theorem B. via the constant D (cid:48)
=
O ( (cid:107) σ (cid:48)(cid:48) (cid:107) 2 ∞ ) =
O ( α 2 ) .
Since the ﬁnal bound depends on the ratio D (cid:48)
√ m you will have that m will grow like O ( α 4 ) .
This is no moderate penalty, although we might expect the results to hold for wide ReLU networks if a ﬁnite α provides a reasonable approximation.
In particular Softmax ln(1 + exp( x )) leads to a ﬁxed constant for D (cid:48) .
C U NDERPARAMETERIZED R EGIME
In this section we build the tools to study the implicit bias in the underparameterized case.
Our ultimate goal is prove Theorem 3.5.
Outline of this section
• Review operator theory • Prove damped deviations equation • Bound (cid:107) ( T K ∞ − T sn )
r t (cid:107)–
Bound (cid:107) ( T n − T sn ) r t (cid:107) 2 using NTK deviation results (comparatively easy) – Bound (cid:107) ( T K ∞ − T n ) r t (cid:107)*
Derive covering number for a class of functions C
* Use covering number to bound sup g ∈C (cid:107) ( T K ∞ − T n ) g
(cid:107) *
Show that r t is in class C • Prove Theorem 3.
C. RKHS AND M ERCER ’ S T HEOREM
We recall some facts about Reproducing Kernel Hilbert Spaces (RKHS) and Mercer’s Theorem.
For additional background we suggest Berlinet & Thomas-Agnan (2004).
Let X ⊂ R d be a compact space equipped with a strictly positive (regular Borel) probability measure ρ .
Let K : X × X
→ R be a continuous, symmetric, positive deﬁnite function.
We deﬁne the integral operator T K : L 2 ρ ( X ) →
L 2 ρ (
[Equation]
In this setting T K is a compact, positive, self-adjoint operator.
By the spectral theorem there is a countable nonincreasing sequence of nonnegative values { σ i } ∞ i =1 and an orthonormal set { φ i } ∞ i = in L 2 such that T K φ i =
σ i φ i .
We will assume that T K is strictly positive, i.e. (cid:104) f, T K f (cid:105) 2 >for f (cid:54) = 0 , so that we have further that { φ i } ∞ i =1 is an orthonormal basis of L 2 and σ i > 0 for all i .
Moreover since K is continuous we may select the φ i so that they are continuous functions, i.e. φ i ∈ C
( X ) for each i .
Then by Mercer’s theorem we can decompose
[Equation]
where the convergence is uniform.
Furthermore the RKHS H associated with K is given by the set of functions
[Equation]
where the inner product on H is given by
[Equation]
Note that in this setting {√ σ i φ i } ∞ i =1 is an orthonormal basis of H .
Deﬁne K x := K ( • , x ) .
Recall the RKHS has the deﬁning properties
[Equation]
We will let κ := sup x ∈ X K ( x, x ) < ∞ .
From this we will have the useful inequality: for h ∈ H
[Equation]
Furthermore the elements of H are bounded continuous functions and H is seperable.
C. H ILBERT -S CHMIDT AND T RACE C LASS O PERATORS
We will recall some deﬁnitions from (Rosasco et al., 2010).
A bounded operator on a separable
Hilbert space with associated norm (cid:107)•(cid:107) is called Hilbert-Schmidt if ∞ (cid:88) i =
[Equation]
Ae i (cid:107) 2 <
[Equation]
for some (any) orthonormal basis { e i } i .
For such an operator we deﬁne its Hilbert-Schmidt norm (cid:107) A (cid:107) HS to be the square root of the above sum.
The Hilbert-Schmidt norm is the analog of the
Frobenius norm for matrices.
It is useful to note that every Hilbert-Schmidt operator is compact.
The space of Hilbert-Schmidt operators is a Hilbert space with respect to the inner product
[Equation]
A stronger notion is that of a trace class operator.
We say a bounded operator on a separable Hilbert space is trace class if
[Equation]
for some (any) orthonormal bases { e i } i .
For such an operator we may deﬁne
[Equation]
By Lidskii’s theorem the above sum is also equal to the sum of the eigenvalues of A repeated by multiplicity.
The space of trace class operators is a Banach space with the norm (cid:107) A (cid:107) TC
=
Tr ( √ A ∗ A ) .
The following inequalities will be useful
[Equation]
Furthermore if A is Hilbert-Schmidt and B is bounded we have
[Equation]
Note that in our setting we have
[Equation]
where the interchange of integration and summation is justiﬁed by the monotone convergence theorem.
Thus T K is a trace class operator and we have the inequality
[Equation]
which will prove useful later.
C. D AMPED D EVIATIONS
Let x (cid:55)→ g s ( x )
∈
L 2 for each s ∈ [0 , t ] such that s (cid:55)→ (cid:104) φ
i , g s (cid:105) 2 is measureable for each i and (cid:82) t 0 (cid:107) g s (cid:107) 22 < ∞ .
Then we deﬁne the integral
[Equation]
coordinatewise, meaning that (cid:82) t 0 g s ds is the L 2 function h such that
[Equation]
Using this deﬁnition, we can now prove the “Damped Deviations” lemma.
Lemma 2.3.
Let K ( x, x (cid:48) ) be an arbitrary continuous, symmetric, positive-deﬁnite kernel.
Let [ T K h ]( • ) =
(cid:82) X K ( • , s ) h ( s ) dρ ( s ) be the integral operator associated with K and let [ T sn h ]( • )
=
n (cid:80) ni
=1 K s ( • , x i ) h ( x i ) denote the operator associated with the time-dependent NTK K s .
Then
[Equation]
where the equality is in the L 2 sense.
Proof.
We have that
[Equation]
where the equality is pointwise over x .
∂ s r s ( x ) is a continuous function of x since K s is continuous and is thus in L 2 .
Therefore we can consider
[Equation]
By the continuity of s (cid:55)→ θ s we have the parameters are locally bounded in time and thus by Lemma
B.5 we have that (cid:107) K s (cid:107) ∞ is also locally bounded therefore for any δ > 0 , s 0 : sup | s − s 0 |≤ δ (cid:107) K s (cid:107) ∞ < ∞ .
Note then that
[Equation]
It follows that (cid:107) ∂ s r s (cid:107) ∞ is bounded locally uniformly in s .
Therefore the following differentiation under the integral sign is justiﬁed
[Equation]
Thus combined with our previous equality we get
[Equation]
where we have used that T K is self-adjoint.
Therefore
[Equation]
Multiplying by the integrating factor exp( σ i s ) we get
[Equation]
Therefore applying the fundamental theorem of calculus after rearrangement we get
[Equation]
which is just the coordinatewise version of the desired result
[Equation]
C. C OVERING N UMBER OF C LASS
We will now estimate the covering number of the class of shallow networks with bounds on their parameter norms.
This lemma is slightly more general than what we will use but we will particularize it latter as it’s general formulation presents no additional difﬁculty
[Equation]
Lemma C.1.
Let
[Equation]
and
[Equation]
Then the (proper) covering number of C in the uniform norm satisﬁes
[Equation]
where p = md + 2 m + 1 is the total number of parameters and C (cid:48) equals
[Equation]
where C > 0 is an absolute constant.
Proof.
We will bound the pertubation of the function when changing the weights, speciﬁcally we will bound
[Equation]
Let x (1) = 1 √ m σ ( Wx + b ) and ˜ x (1) = 1 √ m σ ( ˜ Wx + ˜ b ) .
Then note that we have
[Equation]
Well then
[Equation]
[Table]
[Equation]
Therefore
[Equation]
Thus if we have
[Equation]
then
[Equation]
Therefore
[Equation]
and this bound holds for any x ∈ B M .
If add biases b 0 and ˜ b 0 such that | b 0 − ˜ b 0 | ≤ (cid:15)/ 2 we simply get by the triangle inequality
[Equation]
Thus to get a cover we can simply cover the sets
[Equation]
[Equation]
in the Euclidean norm and multiply the covering numbers.
Recall that the (cid:15) covering number for a
Euclidean ball of radius R in R s , say N (cid:15) , using the Euclidean norm satisﬁes
[Equation]
for two absolute constants c, C > 0 .
Therefore we get that
[Equation]
The desired result follows from
[Equation]
We can now prove the following corollary which is the version of the previous lemma that we will actually use for our neural network
[Equation]
Corollary C.2.
Let
[Equation]
and
[Equation]
and assume m ≥ d .
Then the (proper) covering number of C in the uniform norm satisﬁes
[Equation]
where
[Equation]
Proof.
The idea is to apply Lemma C.1 with a (cid:48)
= 0 , W (cid:48)
= 0 , b (cid:48) = 0 , and b (cid:48)
= 0 .
Note that (cid:107) W (cid:107) F ≤ √ d (cid:107) W (cid:107) op ≤ √ mdA .
The result then follows by applying lemma with ρ 1 = √ mA , ρ 2 = √ mdA , ρ 3 = √ mA and ρ 4 = B and ρ (cid:48) 1 = ρ (cid:48)
2 = ρ (cid:48)
= A .
C. U NIFORM C ONVERGENCE O VER T HE C LASS
We now show that (cid:107) ( T n − T K ∞ ) g (cid:107) 2 is uniformly small for all g in a suitable class of functions C (cid:48) .
Ultimately we will show that r t ∈ C (cid:48) and thus this result is towards proving that (cid:107)
( T n − T K ∞ ) r t (cid:107)is small.
Lemma C.3.
Let K ( x, x (cid:48) ) by a continuous, symmetric, positive-deﬁnite kernel and let κ = max x ∈ X K ( x, x ) < ∞ .
Let T K h ( • )
=
(cid:82) X K ( • , s ) h ( s ) dρ ( s ) and T n h ( • ) =
n (cid:80) ni
=1 K ( • , x i ) h
( x i ) be the associated operators.
Let σ 1 denote the largest eigenvalue of T K .
Let C and Ψ( m, d ) be deﬁned as in Corollary C.2.
We let C (cid:48)
= { g − f ∗ : g ∈ C} ∩ { g : (cid:107) g (cid:107) ∞ ≤ S } be the set where C is translated by the target function f ∗ then intersected with the L ∞ ball of radius
S > 0 .
Then with probability at least 1 − δ over the sampling of x 1 , . . .
, x n sup
[Equation]
√ σ
[Equation]
√ n .
Proof.
Let g ∈ C (cid:48) .
We introduce the random variables Y i :=
K x i g ( x i ) − E x ∼ ρ
[ K x g ( x )] taking values in the Hilbert space H for i ∈ [ n ] where H is the RKHS associated with K .
Note that for any
[Equation]
Thus (cid:107) Y i (cid:107) H ≤ 2 Sκ 1 / {+
 a [Equation] s.+}
Thus by Hoeffding’s inequality for random variables taking values in a separable Hilbert space (see Rosasco et al.
(2010, Section 2.4))
we have
[Equation]
Note that by basic properties of the covering number we have that N ( C (cid:48) , (cid:15), (cid:107)(cid:107) ∞ ) ≤ N ( C , (cid:15)/ 2 , (cid:107)(cid:107) ∞ ) , thus by Corollary C.2 the covering number of C (cid:48) satisﬁes (up to a redeﬁnition of C )
[Equation]
Let ∆ be an (cid:15) net of C (cid:48) in the uniform norm.
Note that 1 n (cid:80) ni =1 Y i =
( T n − T K ) g .
Thus by taking a union bound we have
[Equation]
Note that for any probability measure ν and h ∈ L ∞
[Equation]
It follows that for any h ∈ L ∞
[Equation]
Note for any g ∈ C (cid:48)
we can pick ˆ g in ∆ such that (cid:107) g − ˆ g (cid:107) ∞ ≤ (cid:15) . Then (cid:107) ( T [Equation]
− T [Equation] [Equation] ≤
[Equation]
where we have used the fact that (cid:107)•(cid:107) 2 ≤ √ σ 1 (cid:107)•(cid:107) H and (cid:107)•(cid:107) 2 ≤ (cid:107)•(cid:107) ∞ in the second inequality.
Thus by setting
[Equation]
n we have with probability at least 1 − δ
[Equation]
This argument runs through for any (cid:15) > 0 .
Thus by setting (cid:15) =
[Equation]
√ n we get the desired result.
C. N EURAL N ETWORK I S I N T HE C LASS
In this section we demonstrate that the neural network in such a class as C as deﬁned in Lemma C.1.
Once we have this we can use Lemma C.3 to show that (cid:107) ( T K ∞ − T n ) r t (cid:107) 2 is uniformly small.
The ﬁrst step is to bound the parameter norms, hence the following lemma.
Lemma C.4.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) 1 (cid:107) ψ 2 , (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let Γ > 1 , T > 0 , D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } , and ξ ( t ) =
max { 1 √ m (cid:107) W (cid:107) op , 1 √ m (cid:107) b (cid:107) 2 , 1 √ m (cid:107) a (cid:107) 2 , 1 } .
Furthermore assume
[Equation]
[log(Γ)]and m ≥ max (cid:40)
[Equation]
D
[Equation]
O (log( c/δ ) + ˜ O ( d )) T[log(Γ)], O (log( c/δ )
+ ˜ O
( d )) (cid:41) .
Then with probability at least 1 − δ max t ∈ [0 ,T ] ξ ( t ) ≤ Γ (cid:34) [Equation] m
[Equation]
If one instead does the doubling trick then the second condition on m can be removed from the hypothesis and the same conclusion holds.
Proof.
First assume we are not doing the doubling trick.
Note that the hypothesis on m is strong enough to satisfy the hypothesis of Lemma B.23, therefore we have with probability at least 1 − δ
[Equation]
Well then separately by Lemma B.17 with probability at least 1 − δ
[Equation]
Thus by replacing δ with δ/ 2 in the previous statements and taking a union bound we have with probability at least [Equation] δ
[Equation]
which is the desired result.
Now suppose instead one does the doubling trick.
We recall that the doubling trick does not change ξ (0) .
Thus we can run through the exact same argument as before except when we apply Lemma B.23 we can remove the second condition on m from the hypothesis.
The following lemma bounds the bias term.
Lemma C.5.
For any initial conditions we have
[Equation]
Proof.
Note that
[Equation]
Thus by the fundamental theorem of calculus
[Equation]
The following lemma demonstrates that the residual r t = f t − f ∗ is bounded.
Lemma C.6.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) 1 (cid:107) ψ 2 , (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let Γ > 1 , T > 0 , D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } , and assume [Equation] [log(Γ)] [Equation]
and m ≥ max (cid:40)
[Equation] D [Equation] O (log( c/δ ) + ˜ O ( d )) T
[Equation]
[log(Γ)], O (log( c/δ ) + ˜ O ( d )) (cid:41) .
Then with probability at least 1 − δ for t ≤ T
[Equation]
If one instead does the doubling trick then the second condition on m can be removed from the hypothesis and the same conclusion holds.
Proof.
Recall that
[Equation]
K
[Equation]
Thus
[Equation]
Well by Lemma B.5 we have that (cid:107) K t (cid:107) ∞ ≤ CD 2 ξ 2 ( t ) where
[Equation]
Well by Lemma C.4 we have that with probability at least 1 − δ
[Equation]
Thus by the fundamental theorem of calculus for t ≤ T
[Equation]
Thus by taking the supremum over x we get
[Equation]
which is the desired conclusion.
We can now ﬁnally prove that (cid:107) ( T K ∞ − T n ) r t (cid:107) 2 is uniformly small.
Lemma C.7.
Let K ( x, x (cid:48) ) by a continuous, symmetric, positive-deﬁnite kernel and let κ = max x K ( x, x ) < ∞ .
Let T K h ( • )
=
(cid:82) X K ( • , s ) h ( s ) dρ ( s ) and T n h ( • ) =
1 n (cid:80) ni =1 K ( • , x i ) h
( x i ) be the associated operators.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance. Furthermore assume (cid:107) [Equation] [Equation]
ψ
[Equation]
, (cid:107)
[Equation] (cid:96) (cid:107) ψ 2 , (cid:107) [Equation] 0 (cid:107) ψ 2 ≤ K (cid:48) for each (cid:96) ∈ [ m ] where K (cid:48) ≥ 1 .
Let Γ > 1 ,
T > 0 , D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } , and assume m
[Equation]
D [Equation] (cid:107) [Equation]
(cid:107)
[Equation]
T[log(Γ)]and m ≥ max (cid:40)
[Equation]
d
[Equation]
[log(Γ)]
[Equation]
, O (log( c/δ ) + ˜ O ( d ))
(cid:41) .
If we are doing the doubling trick set S (cid:48)
=
and otherwise set
S (cid:48)
= CD ( K (cid:48) ) 2 (cid:113) d log( CM ˜ O ( √ m )) + log( c/δ ) = ˜ O ( √ d ) .
Then with probability at least 1 − δ sup
[Equation]
n (cid:19) and (cid:107)
[Equation]
If we are performing the doubling trick the second condition on m can be removed and the same conclusion holds.
Proof.
By Lemma C.4 we have with probability at least 1 − δ
[Equation]
Also by Lemma C.
[Equation]
If we are doing the doubling trick then b 0 (0) = 0 .
Otherwise by Lemma B.15 we have with probability at least 1 − δ
[Equation]
Furthermore by Lemma B.22 we have
[Equation]
Let L be deﬁned as in Lemma B.21, i
[Equation]
e.
[Equation]
If we are not performing the doubling trick set
[Equation]
Otherwise if we are performing the doubling trick set S (cid:48)
= 0 .
In either case by Lemma B.21 we have with probability at least 1 − δ
[Equation]
In particular by Lemma B.22 we have
[Equation]
Thus we can say
[Equation]
and this holds whether or not we are performing the doubling trick.
Thus up until time T the neural network is in class C as deﬁned in Corollary C.2 with parameters A and B as deﬁned above.
Moreover by Lemma C.6 separate from the randomness before we have that with probability at least
[Equation]
Well note that when (9) holds we have
[Equation]
Thus
[Equation]
Thus by taking a union bound and redeﬁning δ we have by an application of Lemma C [Equation] 3 with S as deﬁned in the hypothesis of the current theorem that with probability at least 1 − δ [Equation]
n
[Equation]
where we have used that S =
˜ O
([ (cid:107) f ∗ (cid:107) ∞ + S (cid:48) ]
[1 + T Γ 2 ]) .
C. P ROOF O F T HEOREM 3.
We are almost ready to prove Theorem 3.5.
However ﬁrst we must introduce a couple lemmas.
The following lemma uses the damped deviations equation to bound the difference between r t and exp( − T K t ) r 0 .
Lemma C.8.
Let K ( x, x (cid:48) ) by a continuous, symmetric, positive-deﬁnite kernel with associated operator T K h ( • )
= (cid:82) X K ( • , s ) h ( s ) dρ ( s ) .
Let T sn h ( • )
= 1 n (cid:80) ni
=1 K s ( • , x i ) h ( x i ) denote the operator associated with the time-dependent NTK .
Then
[Equation]
and
[Equation]
Proof.
From Lemma [Equation] 3 we have
[Equation]
Thus for any k ∈ N
[Equation]
Therefore
[Equation]
Similarly
[Equation]
[Table]
[Equation]
In light of the previous lemma we would like to have a bound for (cid:107) ( T K − T sn ) r s (cid:107) 2 .
This is accomplished by the following lemma.
Lemma C.9.
Let K ( x, x (cid:48) ) by a continuous, symmetric, positive-deﬁnite kernel.
Let T K h ( • )
=
(cid:82)
X K ( • , s ) h ( s ) dρ ( s ) and T n h ( • ) = 1 n (cid:80) ni
=1 K ( • , x i ) h
( x i ) be the associated operators.
Let
T sn h ( • )
=
1 n (cid:80) ni =1 K s ( • , x i ) h ( x i ) denote the operator associated with the time-dependent NTK .
Then
[Equation]
Proof.
We have that
[Equation]
Now observe that
[Equation]
Therefore
[Equation]
Thus
[Equation]
We are almost ready to ﬁnally prove Theorem 3.5.
We must prove one ﬁnal lemma that combines
Lemma C.7 with the NTK deviation bounds in Theorem B.26 to show that (cid:107) ( T K ∞ − T sn ) r s (cid:107) 2 is uniformly small.
Lemma C.10.
Assume that W i,j ∼ W , b (cid:96) ∼ B , a (cid:96) ∼ A are all i.i.d zero-mean, subgaussian random variables with unit variance.
Furthermore assume (cid:107) 1 (cid:107) ψ 2 , (cid:107) w (cid:96) (cid:107) ψ 2 , (cid:107) a (cid:96) (cid:107) ψ 2 , (cid:107) b (cid:96) (cid:107) ψ 2 ≤ K for each (cid:96) ∈ [ m ] where K ≥ 1 .
Let Γ > 1 , T > 0 , D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } , and assume [Equation] [log(Γ)]and m ≥ max
[Equation]
If we are doing the doubling trick set S (cid:48)
=
and otherwise set
[Equation]
Then with probability at least 1 − δ
[Equation]
If we are performing the doubling trick the condition m ≥ 4 D 2 O (log [Equation] c/δ )+ ˜ O [Equation]
d ))
T 2 [log [Equation] )]can be removed and the same conclusion holds. Proof.
Note by Lemma C.9 we have
[Equation]
Well then by Theorem B.26 we have with probability at least 1 − δ that
[Equation]
Separately by Lemma C.7 we have with probability at least 1 − δ sup s ≤ T (cid:107) ( T
[Equation]
and
[Equation]
The result follows then from taking a union bound and replacing δ with δ/ 2 .
We now proceed to prove the main theorem of this paper.
Theorem 3.5.
Assume that Assumptions 3.3 and 3.4 hold.
Let P k be the orthogonal projection in
L 2 onto span { φ 1 , . .
. , φ k } and let D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } .
If we are doing the doubling trick set S (cid:48)
=
and otherwise set S (cid:48)
=
O (cid:18)(cid:113)
˜
O
( d ) + log( c/δ ) (cid:19) ,
S = (cid:107) f ∗ (cid:107) ∞ + S (cid:48) .
Also let T > 0 .
Assume m ≥ D 2 (cid:107) y (cid:107) 2 R n T 2 , and m ≥ O
(log( c/δ )+ ˜ O ( d )) max (cid:8) T 2 , 1 (cid:9) .
Then with probability at least 1 − δ we have that for all t ≤ T and k ∈ N [Equation] k
[Equation] σ k [Equation] O
[Equation]
√ p [Equation] n (cid:33) and [Equation]
Proof.
By Lemma C.8 we have for any k ∈ N
[Equation]
and furthermore
[Equation]
Well the conditions on m in the hypothesis sufﬁce to apply Lemma C.10 with Γ =
e 2 ensure that with probability at least 1 − δ
[Equation]
Since κ and σ 1 only depend on K ∞ which is ﬁxed we will treat them as constants for simplicity of presentation of the main result (note that they were tracked in all previous results for anyone interested in the speciﬁc constants).
The desired result follows from plugging in the above expression into the previous bounds after setting σ 1 and κ as constants.
Theorem 3.5 is strong enough to get a bound on the test error, which is demonstrated by the following corollary.
Corollary 3.6.
Assume Assumptions 3.3 and 3.4 hold.
Suppose that f ∗ =
O (1) and assume we are performing the doubling trick where f 0 ≡
so that r 0 = − f ∗ .
Let k ∈ N and let P k be the orthogonal projection onto span { φ 1 , . .
. , φ k } .
Set t = log(
[Equation]
Then we have that
[Equation]
and n = ˜Ω (cid:16) pσ
[Equation]
sufﬁces to ensure with probability at least 1 − δ
[Equation]
Proof.
Set t =
[Equation]
Note that
[Equation]
Note that
[Equation]
We want to apply Theorem 3.5 with T = t .
We need m ≥ D 2 (cid:107) y (cid:107) 2 R n t 2 and m ≥ O
(log( c/δ ) + ˜ O ( d )) max { t 2 , 1 } .
Note that since f ∗ =
O (1) we have that (cid:107) ˆ r (0) (cid:107) R n = (cid:107) y (cid:107) R n = O (1) .
Then
[Equation]
thus our condition on m is strong enough to satisfy the ﬁrst condition.
Also O (log( c/δ ) + ˜ O
( d )) max { t 2 , 1 } =
˜ O ( dt 2 ) which is satisﬁed by our condition on m .
Thus by an application of Theorem 3.5 with T
= t we have with probability at least 1 − δ
[Equation]
Recall that f ∗ =
O (1) .
Thus the ﬁrst term above is
[Equation]
Thus setting m = ˜Ω( d(cid:15)σ 4 k ) sufﬁces to ensure the ﬁrst term is bounded by (cid:15) 1 / 2 / (2 √ 2) .
Similarly the second term is
[Equation]
Thus setting n = ˜Ω (cid:16) pσ 4 k (cid:15) (cid:17) sufﬁces to ensure that the second term bounded by (cid:15) 1 / 2 / (2 √ 2) .
Thus in this case we have
[Equation]
Thus we have
[Equation]
C. D ETERMINISTIC I NITIALIZATION
In this section we will prove a version of Theorem 3.5 where instead of θ 0 being chosen randomly we take θ 0 to be some deterministic value.
θ 0 could represent the parameters given by the output of some pretraining procedure that is independent of the training data, or selected with a priori knowledge.
Lemma C.11.
Let θ 0 be a ﬁxed parameter initialization.
Let Γ > 1 , T > 0 , D := 3 max {| σ (0)
[Equation]
Furthermore assume
[Equation]
Then
[Equation]
Proof.
By the hypothesis on m we have that for t ≤ T
[Equation]
Therefore by Lemmas B
[Equation]
2 and B.3 the desired result holds.
Lemma C.12.
Let θ 0 be a ﬁxed parameter initialization.
Let Γ > 1 , T > 0 , D :=
[Equation]
and assume
[Equation]
Then for t ≤ T (cid:107) f t − f ∗ (cid:107) ∞ ≤ (cid:107) f 0 − f ∗ (cid:107) ∞ + t (cid:107) ˆ r (0) (cid:107) R n CD 2 Γ 2 ξ (0) 2 .
Proof.
Recall that
[Equation]
Thus
[Equation]
Well by Lemma B.5 we have that (cid:107) K t (cid:107) ∞ ≤ CD 2 ξ 2 ( t ) .
Also by Lemma C.11 we have that
[Equation]
Thus by the fundamental theorem of calculus for t ≤ T
[Equation]
Thus by taking the supremum over x we get
[Equation]
which is the desired conclusion.
Lemma C.13.
Let θ
be a ﬁxed parameter initialization.
Let K 0 denote the time-dependent NTK at initialization θ 0 .
Let T K 0 h ( • ) =
(cid:82) X K 0 ( • , s ) h
( s ) dρ ( s ) and T n h ( • )
= 1 n (cid:80) ni =1 K 0 ( • , x i ) h
( x i ) be the associated operators.
Let κ = max x K 0 ( x, x ) and let σ 1 denote the largest eigenvalue of T K 0 .
Let Γ > 1 , T > 0 , D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } ,
[Equation]
and assume
[Equation] [Equation]
Then with probability at least 1 − δ over the sampling of x 1 , . . .
, x n we have that
[Equation]
Proof.
First note that
[Equation]
Thus our hypothesis on m is strong enough to apply Lemma C.11 so that we have
[Equation]
[Equation]
Also by Lemma C.
[Equation]
n , therefore
[Equation]
Thus up until time T the neural network is in class C as deﬁned in Corollary C.2 with parameters A and B as deﬁned above.
Furthermore by Lemma C.12 we have
[Equation]
Well then by (10) and the above we have
[Equation]
Thus by an application of Lemma C.3 with K = K
we have with probability at least 1 − δ over the sampling of x 1 , . . .
, x n that
[Equation]
where we have used that S =
˜ O
([ (cid:107) f ∗ (cid:107) ∞ + (cid:107) f 0 (cid:107) ∞ ][1 + T Γ 2 ξ (0) 2 ]) .
Lemma
C.14.
Let θbe a ﬁxed parameter initialization.
Let Γ > 1 ,
[Equation]
>
[Equation]
Furthermore assume
[Equation]
Then for t ≤ T (cid:107) K [Equation] tCDD
[Equation]
˜ ξ
[Equation]
Proof.
Note by Lemma B.24 we have that
[Equation]
Now applying Lemma C.11 and the fact that (cid:107) ˆ r ( t ) (cid:107) R n ≤ (cid:107) ˆ r
[Equation]
(cid:107) R n from the above we get that for t ≤ T
[Equation]
Thus by the fundamental theorem of calculus we have that for t ≤ T
[Equation]
Theorem C.15.
Let θ 0 be a ﬁxed parameter initialization.
Assume that Assumption 3.3 holds.
Let { φ i } i denote the eigenfunctions of T K 0 corresponding to the nonzero eigenvalues, which we enumerate σ 1 ≥ σ 2 ≥ · · · .
Let P k be the orthogonal projection in L 2 onto span { φ 1 , . .
. , φ k } and let D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } .
Also let T > 0 and set
[Equation]
Assume
[Equation]
Then with probability at least 1 − δ over the sampling of x 1 , . . .
, x n we have that for all t ≤ T and k ∈ N
[Equation]
and
[Equation]
Proof.
By Lemma C.8 we have for any k ∈ N
[Equation]
and furthermore
[Equation]
Let T n h ( • )
=
1 n (cid:80) ni =1 K 0 ( • , x i ) h
( x i ) be the discretization of T K 0 .
Thus by Lemma C.9 we have
[Equation]
Note from the inequality
[Equation]
the hypothesis on m is strong enough to apply Lemma C.14 with Γ = e .
Well then by an application of Lemma C.14 with Γ =
e we have that
[Equation]
Separately by Lemma C.13 we have with probability at least 1 − δ
[Equation]
n
[Equation]
Combining these results we get that
[Equation]
The desired result follows from plugging in the above expression into the previous bounds
[Equation]
D D AMPED D EVIATIONS ON T RAINING S ET
The damped deviations lemma for the training set is incredibly simple to prove and yet is incredibly powerful as we will see later.
Here is the proof.
Lemma 2.1.
Let G ∈ R n × n be an arbitrary positive semideﬁnite matrix and let G s be the time dependent NTK matrix at time s .
Then
[Equation]
Proof.
Note that we have the equation
[Equation]
Thus by multiplying by the integrating factor exp( Gt ) and using the fact that exp( Gt ) and G commute we have that
[Equation]
Therefore by the fundamental theorem of calculus
[Equation]
which after rearrangement gives
[Equation]
ˆ r s ds.
Throughout we will let u 1 , . .
, u n denote the eigenvectors of G ∞ with corresponding eigenvalues λ 1 , . . .
, λ n , normalized to have unit norm in (cid:107)•(cid:107) R n , i.e. (cid:107)
u
i (cid:107) R n = 1 .
The following corollary demonstrates that if one is only interested in approximating the top eigenvectors, then the deviations of the NTK only need to be small relative to the cutoff eigenvalue
λ i that you care about.
Corollary D.1.
Let P k be the orthogonal projection onto span { u 1 , . . .
, u k } .
Then for any k ∈ [ n ]
[Equation]
In particular
[Equation]
Proof.
Note by Lemma 2.1 we have that
[Equation]
Therefore for any k ∈ [ n ]
[Equation]
where we have used the inequality 1 + x ≤ exp( x ) in the last inequality.
By specializing to the case k = n since span { u 1 , . . .
u n } =
R n we have
[Equation]
This completes the proof.
Theorem 3.5 uses the concept of damped deviations to compare r t with exp( − T K ∞ t ) r 0 .
We can also prove the analogous statement on the training set that compares ˆ r t to exp( − G ∞ t )ˆ r 0 .
The following is the analog of Theorem 3.5 on the training set.
Theorem D.2.
Let D := 3 max {| σ (0) | , M (cid:107) σ (cid:48) (cid:107) ∞ , (cid:107) σ (cid:48) (cid:107) ∞ , 1 } .
Also let Γ > 1 , T > 0 .
Furthermore assume
[Equation]
Let P k be the orthogonal projection onto span { u 1 , . . .
, u k } .
Then with probability at least 1 − δ we have for any k ∈ [ n ] and t ≤ T
[Equation]
in particular
[Equation]
If one instead does the doubling trick the condition 4 D 2 O
(log( c/δ )+ ˜ O ( d )) T 2 [log(Γ)]can be removed from the hypothesis and the same conclusion holds.
Proof.
By Corollary D.1 we have
[Equation]
Well by Theorem B.27 we have with probability at least 1 − δ
[Equation]
The desired result follows from plugging this in to the previous bounds.
E P ROOF OF T HEOREM 3.
We can now quickly prove our analog of Theorem 4.1 from Arora et al.
Theorem 3.7.
Assume m = ˜Ω( dn 5 (cid:15) − 2 λ n ( H ∞ ) − 4 ) and m ≥ O
(log( c/δ ) + ˜ O ( d )) and f ∗ =
O (1) .
Assume we are performing the doubling trick so that ˆ r 0 = − y .
Let v 1 , . .
. , v n denote the eigenvectors of G ∞ normalized to have unit L2 norm (cid:107) v i (cid:107) 2 = 1 .
Then with probability at least 1 − δ ˆ [Equation] = [Equation]
G
[Equation] − y [Equation] +[-δ ( t ) , where sup t ≥ 
-]
[Equation] t ) , where sup t ≥ 0 (cid:107) δ ( t ) (cid:107) 2 ≤ (cid:15) .
In particular (cid:107) ˆ r t (cid:107) 2 = (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = exp( − 2 λ i t ) |(cid:104)
y, v i (cid:105) 2 | 2 ± (cid:15).
Proof.
Set T = log( (cid:107)
ˆ r (0) (cid:107) R n √ n/(cid:15) ) /λ n .
Note that since f ∗ =
O (1) and we are performing the doubling trick we have that (cid:107)
ˆ r 0 (cid:107) R n = (cid:107) y (cid:107) R n = O (1) .
Recall that λ n := 1 n λ n ( H ∞ ) therefore m = ˜Ω( dn 5 (cid:15) − 2 λ n ( H ∞ ) − 4 ) = ˜Ω( dn(cid:15) − 2 λ − 4 n ) is strong enough to ensure that
[Equation]
Then by an application of Theorem D.2 with Γ = 2 we have with probability at least 1 − δ that for all t ≤ T
[Equation]
Since f ∗ =
O (1) we have that (cid:107) ˆ r 0 (cid:107) R n = (cid:107) y (cid:107) R n =
O (1) therefore the above bound is
[Equation]
Thus m = ˜Ω( dn 5 (cid:15) − 2 λ n ( H ∞ )
[Equation]
= ˜Ω( dn(cid:15) − 2 λ − 4 n ) sufﬁces to make the above term bounded by (cid:15)/ √ n .
Thus in this case
[Equation]
Let δ ( t ) =
ˆ r t − exp( − G ∞ t )ˆ r 0 .
We have just shown that sup t ≤ T (cid:107) δ ( t ) (cid:107) R n ≤ (cid:15)/ √ n .
We will now bound δ ( t ) for t ≥ T .
Note that for t ≥ T
[Equation]
n
Also for t ≥ T
[Equation]
where we have used that (cid:107) ˆ r t (cid:107) R n is nonincreasing for gradient ﬂow.
Therefore for t ≥ T
[Equation]
Thus we have shown
[Equation]
The desired result follows from replacing (cid:15) with (cid:15)/ 3 in the previous argument and using the fact that (cid:107)•(cid:107) 2 = √ n (cid:107)•(cid:107) R n and ˆ r 0 = − y
[Equation]
F P ROOF OF T HEOREM 3.
Using some lemmas that we leave to the following section, we can prove Theorem 3.8 quite quickly using the damped deviations equation and the NTK deviation bounds.
F. M AIN T HEOREM
Theorem 3.8.
Assume Assumptions 3.3 and 3.4 hold.
Furthermore assume m =
[Equation]
∞ ) 2 (cid:17) where T > 0 is a time parameter and
m ≥ O
(log( c/δ ) + ˜ O ( d )) and n ≥ 128 κ 2 log(2 /δ )
( σ k − σ k +1 ) 2 .
Also assume f ∗ ∈
L ∞ ( X ) ⊂
L 2 ( X ) and let P T K ∞ be the orthogonal projection onto the eigenspaces of T K ∞ corresponding to the eigenvalue α ∈ σ ( T K ∞ ) and higher.
Assume that (cid:13)(cid:13)
( I − P T K ∞ )
f ∗ (cid:13)(cid:13) ∞ ≤ (cid:15) (cid:48) for some (cid:15) (cid:48) ≥ 0 .
Pick k so that σ k = α and σ k +1 <
α , i.e. k is the index of the last repeated eigenvalue corresponding to α in the ordered sequence { σ i } i .
Also assume we are performing the doubling trick so
that ˆ r (0) = − y .
Then we have with probability at least 1 − 3 δ over the sampling of x 1 , . . .
, x n and θ 0 that for t ≤ T
[Equation]
Proof.
Recall that we have (cid:107) ˆ r (0) (cid:107) R n =
(cid:107)− y (cid:107) R n ≤ (cid:107) f ∗ (cid:107) ∞ .
Note that m =
˜ Ω (cid:16) (cid:15) − 2 dT 2 (cid:107) f ∗ (cid:107) 2 ∞ (1 + T (cid:107) f ∗ (cid:107) ∞ ) 2 (cid:17) and m ≥ O
(log( c/δ ) + ˜ O ( d )) are strong enough to ensure the hypothesis of Theorem D.2 is satisﬁed with Γ = 2 .
From now on Γ = 2 =
O (1) and will be treated as a constant.
Then by Theorem D.2 with probability at least 1 − δ we have for t ≤ T
[Equation]
Thus using the fact from the doubling trick that (cid:107) ˆ r (0) (cid:107) R n = (cid:107) y (cid:107) R n ≤ (cid:107) f ∗ (cid:107) ∞ setting m = ˜Ω (cid:16) (cid:15) − 2 dT 2 (cid:107) f ∗ (cid:107) 2 ∞ (1 + T (cid:107) f ∗ (cid:107) ∞ ) 2 (cid:17) sufﬁces to ensure that (cid:107) ˆ r t − exp( − G ∞ t )ˆ r 0 (cid:107) R n ≤ (cid:15) for t ≤ T .
Let P k be the orthogonal projection onto span { u 1 , . . .
, u k } .
Well then for t ≤ T (cid:107) ˆ r t
[Equation]
By Theorem F.6 we have with probability at least 1 − 2 δ over the sampling of x 1 , . . .
, x n that
[Equation]
Since we are using the doubling trick we have ˆ r 0 = − y .
Thus we have
[Equation]
Thus by taking a union bound we have with probability at least 1 − 3 δ for all t ≤ T
[Equation]
The desired result follows from ˆ r 0 = − y .
F. C ONTROL OF I NITIAL R ESIDUAL
We will use some of the notation and operator theory from Section C.1 and Section C.2 in this section, thus it is recommended to have read those sections ﬁrst.
Let u 1 , . .
, u n denote the eigenvectors of G ∞ normalized to have unit norm in (cid:107)•(cid:107) R n , i.e. (cid:107) u
i (cid:107) R n = 1 .
Let P k be the orthogonal projection onto span { u 1 , . . .
, u k } .
The goal of this section is to upper bound the extent to which the labels y participate in the bottom eigendirections of G ∞ , i.e. to show that (cid:107) ( I − P k ) y (cid:107) R n is small.
Let P T K ∞ be some projection onto the top eigenspaces of T K ∞ .
The idea is to show that if (cid:13)(cid:13) ( I − P T K ∞ ) f ∗ (cid:13)(cid:13) 2 is small then by picking P k
so that rank ( P k ) =
rank ( P T K ∞ ) then (cid:107) ( I − P k ) y (cid:107) R n is also small with high probability.
The results in this section essentially all appear in the proofs in Su & Yang (2019).
We repeat the arguments here for completeness and due to differences in notation and constants.
We use some of the same machinery in (Rosasco et al., 2010).
We deﬁne operators L H : H → H and T n : H → H by
[Equation]
Note that T H is equal to T K ∞ on H and T n is simply the operator you get if you replace ρ in the deﬁntion of T H with the empirical measure 1 n (cid:80) ni
=
δ x i .
We deﬁne the “restriction” operator
R n : H → R n by
[Equation]
Note here the domain of R n is H but in other parts of this paper we will allow R n to take more general functions as input.
Deﬁne R ∗ n : R n → H by
[Equation]
It can be seen that
[Equation]
and thus R ∗ n is the adjoint of R n .
Using these operators we may write T n = R ∗ n R n and G ∞ =
R n R ∗ n .
It will follow that T n and G ∞ have the same eigenvalues (up to some zero eigenvalues) and their eigenvectors are related.
We recall the following result from (Rosasco et al., 2010)
(Proposition 9)
Theorem F.1.
(Rosasco et al., 2010)
The following hold
• The operator T n is ﬁnite rank, self-adjoint and positive, and the matrix G ∞ is symmetric and semi-positive deﬁnite.
In particular the spectrum σ ( T n ) of T n has ﬁnitely many nonzero elements and they are contained in [0 , κ ] .
• The spectrum of T n and G ∞ are the same up to zero, speciﬁcally σ ( G ∞ ) \ { 0 }
= σ ( T n ) \ { 0 } .
Moreover if λ i is a nonzero eigenvalue and u i and v i are the corresponding eigenvector and eigenfunction for G ∞ and T n respectively (normalized to norm 1 in (cid:107)•(cid:107) R n and (cid:107)•(cid:107) H respectively), then
[Equation]
where ( u i )
j is the j th component of the vector u i .
• The following decompositions hold
[Equation]
where k = rank ( G ∞ )
=
rank ( T n ) and both sums run over the positive eigenvalues.
{ u i } ki =1 is an orthonormal basis for ker ( G ∞ )
⊥
and
{ v i } ki =1 is an orthonormal basis for ker ( T n ) ⊥ .
We will make use of the following lemma from Rosasco et al.
(2010, Proposition 6):
Lemma F.2.
(Rosasco et al., 2010) Let α 1 > α 2 > . . .
> α N > α N +1 be the top N + 1 distinct eigenvalues of T H .
Let P T H be the orthogonal projection onto the eigenfunctions of T H corresponding to eigenvalues α N and above.
Let P T n be the projection onto the top k eigenvectors of T n so that k = dim ( range ( T n ))
=
dim ( range ( T H )) .
Assume further that
[Equation]
Then
[Equation]
The following lemma will be useful.
Lemma F.3.
Let f ∗ ∈
L 2 and let P T H and P T n be deﬁned as in Lemma F.2.
Then n
[Equation]
Proof.
We repeat the same proof as in (Su & Yang, 2019) for completeness and to remove confusion that may arise from differences in notation.
The proof was originally given in (Rosasco et al., 2010) albeit with a minor error involving missing multiplicative factors.
Note that
[Equation]
Therefore
[Equation]
Applying Cauchy-Schwarz we get
[Equation]
Well then note that k
[Equation]
Therefore n (cid:88) i =
[Equation]
+
[Equation]
[Equation]
22 λ
[Equation]
=
[Equation]
|
[Equation]
On the other hand
[Equation]
Note that for 1 ≤ j ≤ k and k + 1 ≤ i ≤ n we have
[Equation]
So
[Equation]
To summarize we have shown
[Equation]
Combining this with (12) we get the ﬁnal result n (cid:88)
[Equation]
We can use Lemma F.3 to produce the following bound.
Lemma F.4.
Let f ∗ ∈
L 2 and let P T H and P T n be deﬁned as in Lemma F.2.
Then n
[Equation]
Proof.
We have that
[Equation]
Thus from the inequality ( a + b ) 2 ≤ 2( a 2 + b 2 ) we get n (cid:88)
i
[Equation]
To control the ﬁrst term we have
[Equation]
Then by applying Lemma F.3 to the second term we get the desired result.
We recall the following lemma from Rosasco et al.
(2010, Theorem 7)
Lemma F.5.
(Rosasco et al., 2010)
With probability at least 1 − δ over the sampling of x 1 , . . .
, x n
[Equation]
Now ﬁnally we can provide a bound on the labels participation in the bottom eigendirections.
Theorem F.6.
Assume f ∗ ∈
L 2 ( X ) and let P T K ∞ be the orthogonal projection onto the eigenspaces of T K ∞ corresponding to the eigenvalue α ∈ σ ( T K ∞ ) and higher.
Assume that
[Equation]
for some (cid:15) (cid:48) ≥ 0 .
Pick k so that σ k = α and σ k +1 <
α , i.e. k is the index of the last repeated eigenvalue corresponding to α in the ordered sequence { σ i } i .
Let P k denote the orthogonal projection onto span { u 1 , . . .
, u k } .
Finally assume
[Equation]
Then we have with probability at least 1 − 2 δ over the sampling of x 1 , . . .
, x n that
[Equation]
Proof.
From Lemma F.4 we have n
[Equation]
i
[Equation]
| [Equation] [Equation]
By assumption we have that the ﬁrst term is bounded by 2( (cid:15) (cid:48) ) 2 .
Now we must control the term
[Equation]
By Lemma F.5 we have with probability at least 1 − δ
[Equation]
Then
[Equation]
sufﬁces so that the right hand side above is less than or equal to σ k − σ k +1.
Thus by Lemma F.2 we have that
[Equation]
Thus from the above inequality we get that
[Equation]
By Proposition 10 in Rosasco et al.
we have separately with probability at least 1 − δ
[Equation]
Note that
[Equation]
implies that
[Equation]
therefore
[Equation]
Thus
[Equation]
Thus combined with our previous results we ﬁnally get that
[Equation]
Thus from the inequality √ a + b ≤ √ 2( √ a + √ b ) which holds for all a, b ≥
we have
[Equation]
Since y =
R n f ∗ this provides the desired conclusion.
G T K ∞ I S S TRICTLY P OSITIVE Note that
K
[Equation]
where the expectation is taken with respect to the parameter initialization.
It sufﬁces to show that the kernel corresponding to the ﬁrst term above
[Equation]
induces a strictly positive operator T K a f ( x )
=
(cid:82) X K a ( x, s ) f
( s ) dρ ( s ) .
From the discussion in
Section C.1 it sufﬁces to show that the RKHS corresponding to K a is dense in L 2 .
In Proposition 4.1 in Rahimi & Recht (2008a) they showed that the RKHS associated with K a has dense subset
[Equation]
where µ is the measure for the parameter initialization, i.e. ( w, b ) ∼ µ .
Since C ( X ) is dense in
L 2 ( X ) it sufﬁces to show that F is dense in C ( X ) which is provided by the following theorem:
Theorem G.1.
Let σ be L -Lipschitz and not a polynomial.
Assume that µ is a strictly positive measure supported on all of R d +1 .
Also assume that
[Equation]
Then F is dense in C ( X ) under the uniform norm.
Proof.
We ﬁrst show that F ⊂ C ( X )
[Equation]
Suppose we have f ∈ F and write f ( x ) =
(cid:82)
R d +1 a ( w, b ) σ ( (cid:104) w, x (cid:105) 2 + b ) dµ ( w, b ) .
Well then
[Equation]
Thus f is Lipschitz and thus continuous.
Now suppose that F is not dense in C ( X ) .
Then by the
Riesz representation theorem there exists a nonzero signed measure ν ( x ) with ﬁnite total variation such that (cid:82) X f ( x ) dν ( x ) = 0 for all f ∈ F .
Well then writing f ( x )
= (cid:82) R d +1 a ( w, b ) σ ( (cid:104) w, x (cid:105) 2 + b ) dµ ( w, b ) as before we have
[Equation]
Note that
[Equation]
where we have used Cauchy-Schwarz and the hypothesis on the integrability of (cid:107) w (cid:107) 22 , (cid:107) b (cid:107) 22 in the last step.
Thus the integrand in (13) is µ × ν integrable thus by Fubini’s theorem we may interchange the order of integration.
To get that
[Equation]
and the above holds for any a ∈ L 2 ( R d +1 , µ ) .
Thus (cid:82) X σ ( (cid:104) w, x (cid:105) 2 + b ) dν ( x ) = 0 for µ -almost every w, b .
However by essentially the same proof as when we showed F ⊂ C ( X ) we may show that (cid:82)
X σ ( (cid:104) w, x (cid:105) 2 + b ) dν ( x ) = 0 is a continuous function of ( w, b ) .
Thus since µ is a strictly positive measure on R d +1 this implies that (cid:82) X σ ( (cid:104) w, x (cid:105) 2 + b ) dν ( x ) = 0 for every ( w, b ) ∈ R d +1 .
However by Theorem 1 in Leshno et al.
we have that span { σ ( (cid:104) w, x (cid:105) 2 + b ) : ( w, b ) ∈ R d +1 } is dense in C ( X ) .
However by our previous conclusion and linearity we have that (cid:82) g ( x ) dν ( x ) = 0 for any g in span { σ ( (cid:104) w, x (cid:105) 2 + b ) : ( w, b ) ∈ R d +1 } , which implies then that ν must equal 0 .
Thus F is dense in C ( X ) .
Since Gaussians are supported on all of R d
+1 we have the following corollary:
Corollary G.2.
If ( w, b ) ∼ N (0 , I d +1 ) then K ∞ is strictly positive.