U NIFORM G ENERALIZATION B OUNDS FOR O VERPARAMETERIZED N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
An interesting observation in artiﬁcial neural networks is their favorable generalization error despite typically being extremely overparameterized.
It is well known that the classical statistical learning methods often result in vacuous generalization errors in the case of overparameterized neural networks.
Adopting the recently developed Neural Tangent (NT) kernel theory, we prove uniform generalization bounds for overparameterized neural networks in kernel regimes, when the true data generating model belongs to the reproducing kernel Hilbert space (RKHS) corresponding to the NT kernel.
Importantly, our bounds capture the exact error rates depending on the differentiability of the activation functions.
In order to establish these bounds, we propose the information gain of the NT kernel as a measure of complexity of the learning problem.
Our analysis uses a Mercer decomposition of the NT kernel in the basis of spherical harmonics and the decay rate of the corresponding eigenvalues.
As a byproduct of our results, we show the equivalence between the RKHS corresponding to the NT kernel and its counterpart corresponding to the Mat´ern family of kernels, showing the NT kernels induce a very general class of models.
We further discuss the implications of our analysis for some recent results on the regret bounds for reinforcement learning and bandit algorithms, which use overparameterized neural networks.
I NTRODUCTION
Neural networks have shown an unprecedented success in the hands of practitioners in several ﬁelds.
Examples of successful applications include classiﬁcation (Krizhevsky et al., 2012; LeCun et al., 2015) and generative modeling (Goodfellow et al., 2014; Brown et al., 2020).
This practical efﬁciency has accelerated developing analytical tools for understanding the properties of neural networks.
A long standing dilemma in the analysis of neural networks is their favorable generalization error despite typically being extremely overparameterized.
It is well known that the classical statistical learning methods for bounding the generalization error, such as Vapnik–Chervonenkis (VC) dimension (Bartlett, 1998; Anthony & Bartlett, 2009) and Rademacher complexity (Bartlett & Mendelson, 2002; Neyshabur et al., 2015), often result in vacuous bounds in the case of neural networks (Dziugaite & Roy, 2017).
Motivated with the problem mentioned above, we investigate the generalization error in overparameterized neural networks.
In particular, let D n = { ( x i , y i ) } ni
=1 be a possibly noisy dataset collected from a true model f : y i = f
( x i ) + (cid:15) i , where (cid:15) i are well behaved zero mean noise terms.
Let ˆ f n be the learnt model using a gradient based training of an overparameterized neural network on D n .
We are interested in bounding | f ( x ) − ˆ f n ( x ) | uniformly at all test points x belonging to a particular domain X .
A recent line of work has shown that gradient based learning of certain overparameterized neural networks can be analyzed using a projection on the tangent space at random initialization.
This leads to the equivalence of the solution of the neural network model to a kernel ridge regression, that is a linear regression in the possibly inﬁnite dimensional feature space of the corresponding kernel.
The particular kernel arising from this approach is called the neural tangent (NT) kernel (Jacot et al., 2018).
This equivalence is based on the observation that, during training, the parameters of the neural network remain within a close vicinity of the random initialization.
This regime is sometimes referred to as lazy training (Chizat et al., 2019).
1. C ONTRIBUTIONS
Here, we summarize our contributions.
We propose the maximal information gain (MIG), denoted by γ k ( n ) , of the NT kernel k , as a measure of complexity of the learning problem for overparameterized neural networks in kernel regimes.
The MIG is closely related to the effective dimension of the kernel (for the formal deﬁnition of the MIG and its connection to the effective dimension of the kernel, see Section 4).
We provide kernel speciﬁc bounds on the MIG of the NT kernel.
Using these bounds, for networks with inﬁnite number of parameters, we analytically prove novel generalization error bounds decaying with respect to the dataset size n .
Furthermore, our results capture the dependence of the decay rate of the error bound on the smoothness properties of the activation functions.
Our analysis crucially uses Mercer’s theorem and the spectral decomposition of the NT kernel in the basis of spherical harmonics (which are special functions on the surface of the hypersphere; see
Section 3).
We derive bounds on the decay rate of the Mercer eigenvalues (referred to hereafter as eigendecay ) based on the differentiability of the activation functions.
These eigendecays are obtained by careful characterization of a recent result relating the eigendecay to the asymptotic endpoint expansions of the kernel, given in (Bietti & Bach, 2020).
In particular, we consider s − 1 times differentiable activation functions a s ( u ) =
(max(0 , u )) s , s ∈ N .
We show that, with these activation functions and a d dimensional hyperspherical support, the eigenvalues λ
i of the corresponding NT kernel decay at the rate λ i ∼ i − d +2 s − 2 d − 1 1 (up to certain parity constraints on i in special cases; see Section 2 for details).
We then use the eigendecay of the NT kernel to establish novel bounds on its MIG, which are of the form γ k ( n ) =
˜ O ( n d − 1 d +2 s − 2 ) .
Then, we show this implies high probability error bounds of the form sup x ∈X | f ( x ) − ˆ f n ( x ) | =
˜ O ( n − 2 s +1 2 d +4 s − 4 ) , when f belongs to the corresponding reproducing kernel
Hilbert space (RKHS), and D n is distributed over X in a sufﬁciently informative way.
In particular, we consider a D n that is collected sequentially in a way that maximally reduces the uncertainty in the model.
That results in a quasi-uniform dataset over the entire domain X (see Section 5.2 for details).
A summary of our results on MIG and error bounds is reported in Table 1.
As a byproduct of our results, we show that the RKHS of the NT kernel with activation function a s ( . ) is equivalent to the RKHS of a Mat ´ ern kernel with smoothness parameter ν = s − 12 .
This result was already known for a special case.
In particular, Geifman et al.
; Chen & Xu (2021) recently showed the equivalence between the RKHS of the NT kernel with ReLU activation function, a 1 ( . ) , and that of the Mat´ern kernel with ν = 12 (also known as the Laplace kernel).
Our contribution includes generalizing this result to the equivalence between the RKHS of the NT kernel under various activation functions and that of a Mat ´ ern kernel with the corresponding smoothness (see Section 3.2).
We note that the members of the RKHS of Mat ´ ern family of kernels can uniformly approximate almost all continuous functions (Srinivas et al., 2010).
Therefore, our results suggest that the NT kernels induce a very general class of models.
As an intermediate step, we also consider the simpler random feature (RF) model which is the result of partial training of the neural network.
Also referred to as conjugate kernel or neural networks
Gaussian process (NNGP) (Cho & Saul, 2009; Daniely et al., 2016; Lee et al., 2018; de G. Matthews et al., 2018), this model corresponds to the case where only the last layer of the neural network is trained (see Section 2 for details).
kernels, with a s ( . ) activation functions, when the true model f belongs to the RKHS of the corresponding kernel and is supported on the hypersphere
S d − 1 ⊂ R d .
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] We note that the decay rate of the error bound is faster in the case of RF kernel with the same activation function.
Also, the decay rate of the error bound is faster when s is larger, for both NT and RF kernels.
In interpreting these results, we should however take into account the RKHS of the corresponding kernels.
The RKHS of the RF kernel is smaller than the RKHS of the NT kernel.
Also, when s is larger, the RKHS is smaller for both NT and RF kernels.
In Section 6, we provide experimental results on the error rate for synthetic functions belonging to the RKHS of the NT kernel, which are consistent with the analytically predicted behaviors.
Our bounds on MIG may be of independent interest.
In Section 7, we comment on the implications of our results for the regret bounds in reinforcement learning (RL) and bandit problems, which use overparameterized neural network models (e.g., Yang et al., 2020; Gu et al., 2021).
O VERPARAMETERIZED N EURAL N ETWORKS IN K ERNEL R EGIMES
In this section, we brieﬂy overview the equivalence of overparameterized neural network models to kernel methods.
It has been recently shown that, in overparameterized regimes, gradient based training of a neural network reaches a global minimum, where the weights remain within a close vicinity of the random initialization (Jacot et al., 2018).
In particular, let f ( x, W ) denote a neural network with a large width m parameterized by W .
The model can be approximated with its linear projection on the tangent space at random initialization W 0 , as
[Equation]
The approximation error can be bounded by the second order term ξ m (cid:107) W − W 0 (cid:107) 2 , and shown to be diminishing as m grows, where ξ m is the spectral norm of the Hessian matrix of f (Liu et al., 2020).
The approximation becomes an equality when the width of the hidden layers approaches inﬁnity.
Known as lazy training regime (Chizat et al., 2019), this model is equivalent to the NT kernel (Jacot et al., 2018), given by
[Equation]
2. N EURAL K ERNELS FOR A 2 L AYER N ETWORK
Consider the following 2 layer neural network with width m and a proper normalization
[Equation]
where a ( . ) is the activation function, c is a constant, x ∈
X is a d dimensional input to the network, w j ∈ R
d are the weights in the ﬁrst layer, and v j ∈ R are the weights in the second layer.
The weights w j are initialized randomly according to N ( 0 d , I d ) and v j are initialized randomly according to N (0 , 1) .
Then, the NT kernel of this network can be expressed as (Jacot et al., 2018)
[Equation]
When w j are ﬁxed at initialization, and only v j are trained with (cid:96) 2 regularization, the model is equivalent to a certain Gaussian process (GP) (Neal, 2012) that is often referred to as a random feature (RF) model.
The RF kernel is given as (Rahimi et al., 2007)
[Equation]
Note that the RF kernel is the same as the second term in the expression of the NT kernel given in equation 2.
2. R OTATIONAL I NVARIANCE
When the input domain X is the hypersphere S d − 1 , by the spherical symmetry of the Gaussian distribution, the neural kernels are invariant to unitary transformations 2 and can be expressed as a function of x (cid:62) x (cid:48) .
We thus use the notations κ NT ,s ( x (cid:62) x (cid:48) ) and κ s ( x (cid:62) x (cid:48) ) to refer to the NT and
RF kernels, as deﬁned in equation 2 and equation 3, respectively.
In this notation, s speciﬁes the smoothness of the activation function, a s ( u ) =
(max(0 , u )) s .
The particular form of the neural kernels depends on s .
For example, for the special case of s = 1 , which corresponds to the ReLU activation function, the following closed form expressions can be derived from the expressions given in equation 2 and equation 3.
[Equation]
In deriving the expressions above, the constant c in equations 2 and 3 is set to √ 2 .
In general, we choose c 2 = 2 (2 s − 1)!!
, that normalizes the maximum value of the RF kernel to 1 : κ s (1) = 1 , ∀ s ≥ 1 .
2. N EURAL K ERNELS FOR M ULTILAYER N ETWORKS
For an l > 2 layer network, the NT kernel is recursively given as (see, Jacot et al., 2018, Theorem 1 )
[Equation]
Here, κ ls ( . ) corresponds to the RF kernel of an l layer network, as shown in Cho & Saul (2009);
Daniely et al.
; Lee et al.
de G. Matthews et al.
In our notation, when l = 2 , we drop the layer index l from the superscript, as in the previous subsection.
An expression for the derivative κ (cid:48) s ( . ) of the RF kernel in a 2 layer network, based on κ s − 1 , is given in Lemma 1.
That is needed to see the equivalence of the expressions in equation 4 to the one given in Theorem 1 of Jacot et al.
T HE RKHS OF THE N EURAL K ERNELS
When the domain is S d − 1 , as discussed earlier, the neural kernels have a rotationally invariant (dot product) form.
The rotationally invariant kernels can be diagonalized in the basis of spherical harmonics, which are special functions on the surface of the hypersphere.
Specifically, the Mercer decomposition of the kernels (for all l ≥ 2 and s ≥ 1 ) can be given as
[Equation]
where the eigenfunction ˜ φ i,j is the j ’th spherical harmonic of degree i , ˜
λ
i is the corresponding eigenvalue, N d,
[Equation]
=
[Equation]
is the multiplicity of ˜ λ
i (meaning the number of spherical harmonics of degree i , which all share the same eigenvalue ˜ λ i ), and the convergence of the series holds uniformly on X ×X .
As a consequence of Mercer’s representation theorem, the corresponding RKHS can be written as
[Equation]
Formal statements of Mercer’s theorem and Mercer’s representation theorem, as well as a formal deﬁnition of Mercer eigenvalues and eigenfunctions are given in Appendix B.
We use the notation λ
i (in contrast to ˜ λ i ) to denote the decreasingly ordered eigenvalues, when their multiplicity is taken into account (for all i : (cid:80) i (cid:48)
− 1 i (cid:48)(cid:48)
=1 N d,i (cid:48)(cid:48) <
i ≤ (cid:80) i (cid:48)
i (cid:48)(cid:48)
=1 N d,i (cid:48)(cid:48) , we deﬁne λ
i = ˜
λ
i (cid:48) ).
3. T HE D ECAY R ATE OF THE E IGENVALUES
As discussed, both RF and NT kernels can be represented in the basis of spherical harmonics.
Our analysis of the MIG relies on the decay rate of the corresponding eigenvalues.
In this section, we derive the eigendecay depending on the particular activation functions.
In the special case of ReLU activation function, several recent works (Geifman et al., 2020; Chen & Xu, 2021; Bietti & Bach, 2020) have proven a ˜
λ i ∼ i − d eigendecay for the NT kernel on the hypersphere S d − 1 .
In this work, we use a recent result from Bietti & Bach (2020) to derive the eigendecay based on the differentiability properties of the activation functions.
Proposition 1 Consider the neural kernels on the hypersphere S d − 1 , and their Mercer decomposition in the basis of spherical harmonics.
Then, the eigendecays are as reported in Table 2.
NT l =
For i of the same parity of s : ˜ λ i ∼ i − d − 2 s +2 , λ i ∼ i − d +2 s − 2 d −
For i of the opposite parity of s : ˜ λ i
=
o
( i − d − 2 s +2 ) , λ i
=
o
( i − d +2 s − 2 d − 1 ) l >˜ λ
i ∼ i − d − 2 s +2 , λ i ∼ i
[Table]
RF l
=
For i of the opposite parity of s : ˜ λ i ∼ i
[Table]
s , λ i ∼
[Table]
For i of the same parity of s : ˜ λ i
=
o
( i − d − 2 s ) , λ i
=
o
( i − d +2 s
[Table] [Table]
>
[Table]
Proof Sketch.
The proof follows from Theorem 1 of Bietti & Bach (2020), which proved that the eigendecay of a rotationally invariant kernel κ ( . ) :
[ − 1 , 1] → R can be determined based on its asymptotic expansions around the endpoints, ± 1 .
In particular, when
[Equation]
for polynomial p ± 1 and non-integer θ , the eigenvalues decay at the rate ˜
λ i ∼ ( c +1 + c − 1 )
i − d − 2 θ +1 , for even i , and ˜ λ i ∼ ( c +1 − c − 1 ) i − d − 2 θ +1 , for odd i .
A formal statement of this result is given in
Appendix E. They derived these expansions for the case of the ReLU activation function.
For completeness, we derive the asymptotic endpoint expansions for the neural kernels with s − 1 times differentiable activation functions, in the case of l = 2 and l > 2 layer networks.
Our derivations is based on the following Lemma.
Lemma 1 For the RF kernel deﬁned in equation 3, with activation function a s ( . ) , and c =
[Equation]
s − [Equation]
, we have
[Equation]
The proof is given in Appendix D. Lemma 1 allows us to recursively derive the endpoint expansions of κ s from those of κ s − 1 , through integration.
We thus obtain θ and c ± 1 for s > 1 , when l = 2 , using a recursive relation over s .
We then use the compositional forms given in equation 4 to obtain the same, when l > 2 , resulting in the eigendecays reported in Table 2.
[Equation]
Remark
The parity constraints in Table 2 may imply undesirable behavior for 2 layer inﬁnitely wide networks.
Ronen et al.
(2019) studied this behavior and showed that adding a bias term removes these constraints, for the NT kernel.
Although the parity constrained eigendecay affects the representation power of the neural kernels, it does not affect our analysis of MIG and error rates, since for that analysis we only use ˜ λ
i =
O ( i − d − 2 s +2 ) .
Remark
We note that our results do not necessarily apply to deep neural networks in the sense that the implied constants in Table 2 grow exponentially in l , for s > 1 (see Appendix E for the expressions of the constants).
When s = 1 , however, the constants grow at most as fast as l 2 (as shown in Bietti & Bach, 2020).
3. E QUIVALENCE OF THE RKHS S OF THE N EURAL AND M AT ´ ERN KERNELS
Our bounds on the eigendecay of the neural kernels show the connection between the RKHS of the neural kernels, and that of Met´ern family of kernels.
Let H k ν denote the RKHS of the Mat´ern kernel with smoothness parameter ν .
Theorem 1 On a hyperspherical domain, when l > 2 , H κ l NT ,s ( H κ ls ) is equivalent to H k ν , and, when l = 2 , H κ l NT ,s ⊂ H k ν
( H κ ls ⊂ H k ν ), where ν
= s − 12 ( ν = s + 12 ).
The statement is summarized in
Table 3.
Kernel
[Table]
The formal deﬁnition of equivalence ( ≡ ) and subset ( ⊂ ) relation between two normed spaces is given in Appendix A.
Proof Sketch.
The proof follows from the eigendecay of the neural kernels given in Table 2, the eigendecay of the Mat ´ ern family of kernels on manifolds proven in Borovitskiy et al.
(2020), and the Mercer’s representation theorem.
Details are given in Appendix F.
(cid:3)
Remark
The observations that when l = 2 , we only have H κ l NT ,s ⊂ H k s − 12 , and not vice versa, is a result of parity constraints in Table 2.
The subset relation can change to equivalence, with a bias term which removes the parity constraints (see, Ronen et al., 2019).
A special case of Theorem 1, when s = 1 , was recently proven in (Geifman et al., 2020; Chen &
Xu, 2021).
This special case pertains to the ReLU activation function and the Mat´ern kernel with smoothness parameter ν = 12 , that is also referred to as the Laplace kernel.
Remark
It is known that the RKHS of a Mat ´ ern kernel with smoothness parameter ν is equivalent to a Sobolev space with parameter ν
+ d 2 (e.g., see, Kanagawa et al., 2018, Section 4 ).
This observation provides an intuitive interpretation for the RKHS of the neural kernels as a consequence of
Theorem 1.
That is (cid:107) f (cid:107) κ l NT ,s ( (cid:107) f (cid:107) κ ls ) is proportional to the cumulative L 2 norm of the weak derivatives of f up to s + d − 12 ( s + d +12 ) order.
I.e., f ∈ H κ l NT ,s ( f ∈ H κ ls ) translates to the existence of weak derivatives of f
up to s + d − 12 ( s + d +12 ) order, which can be understood as a versatile measure for the smoothness of f controlled by s .
For the details on the deﬁnition of weak derivatives and
Sobolev spaces see, e.g., Hunter & Nachtergaele (2011).
M AXIMAL I NFORMATION G AIN OF THE N EURAL K ERNELS
In this section, we use the eigendecay of the neural kernels shown in the previous section to derive bounds on their information gain, which will then be used to prove uniform bounds on the generalization error.
4. I NFORMATION G AIN : A M EASURE OF C OMPLEXITY
To deﬁne information gain, we need to introduce a ﬁctitious GP model.
Assume F is a zero mean
GP indexed on X with kernel k .
Information gain then refers to the mutual information I ( Y n ; F ) between the data values Y n =
[ y i ] ni =1 and F .
From the closed from expression of mutual information between two multivariate Gaussian distributions (Cover, 1999), it follows that
[Equation]
Here K n is the kernel matrix [ K n ] i,j
= k ( x i , x j ) , and λ > 0 is a regularization parameter.
We also deﬁne the data independent and kernel speciﬁc maximal information gain (MIG) as follows.
[Equation]
The MIG is also closely related to the effective dimension of the kernel.
In particular, although the feature space of the neural kernels are inﬁnite dimensional, for a ﬁnite dataset, the number of features with a signiﬁcant impact on the regression model can be ﬁnite.
The effective dimension of a kernel is often deﬁned as (Zhang, 2005; Valko et al., 2013)
[Equation] [Equation]
It is known that the information gain and the effective dimension are the same up to logarithmic factors.
Speciﬁcally, ˜ d k ( n ) ≤ I ( Y n ; F ) , and I ( Y n ; F )
=
O ( ˜ d k ( n ) log( n ))
(Calandriello et al., 2019).
Relating the information gain to the effective dimension provides some intuition into why it can capture the complexity of the learning problem.
Roughly speaking, our bounds on the generalization error are analogous to the ones for a linear model, which has a feature dimension of ˜ d k ( n ) .
4. B OUNDING THE I NFORMATION G AIN OF THE N EURAL K ERNELS
In the following theorem, we provide a bound on the maximal information gain of the neural kernels.
Theorem 2 For the neural kernels with all l ≥ 2 , on a hyperspherical domain X = S d − 1 , we have
[Equation]
A special case of our Theorem 2, for the special case of the ReLU activation function, was recently proved in Kassraie & Krause (2021).
Proof Sketch.
The main components of the analysis include the eigendecay given in Proposition 1, a projection on a ﬁnite dimensional RKHS technique proposed in Vakili et al.
(2021b), and a bound on the sum of spherical harmonics of degree i determined by the Legendre polynomials, that is sometimes referred to as the Legendre addition theorem (Maleˇcek & N´aden´ık, 2001).
Details are given in Appendix G.
(cid:3)
U NIFORM B OUNDS ON THE G ENERALIZATION E RROR
In this section, we use the bound on MIG from previous section to establish uniform bounds on the generalization error.
5. I MPLICIT E RROR B OUNDS
We ﬁrst overview the implicit (data dependent) error bounds for the kernel methods under the following assumptions.
Assumption 1 Assume the true data generating model f is in the RKHS corresponding to a neural kernel κ .
In particular, (cid:107) f (cid:107) H κ ≤ B , for some B > 0 .
In addition, assume that the observation noise (cid:15) i are independent R sub-Gaussian random variables.
That is E [exp( η(cid:15) i )] ≤ exp( η 2 R 2 2 ) , ∀ η ∈ R , ∀ i ≥ 1 , where y i
=
f
( x i ) + (cid:15)
i , ∀
i ≥ 1 .
Under Assumption 1, for a ﬁxed x ∈ X , we have, with probability at least 1 − δ (Vakili et al., 2021a),
[Equation] [Equation]
where ˆ f n ( x ) = k (cid:62) n ( x )( K n + λ 2 I n ) − 1 Y n is the solution to the kernel ridge regression using κ and D n , σ 2 n ( x ) =
k ( x, x ) − k (cid:62) n ( x )( K n + λ 2 I n ) − 1 k n ( x ) , k (cid:62) n ( x ) =
[ κ ( x (cid:62) x i )]
ni =1 , and β ( δ ) =
B + Rλ (cid:113) 2 log( 2 δ ) .
Equation 7 provides a high probability bound on | f ( x ) − ˆ f n ( x ) | .
The decay rate of this bound based on n , however, is not explicit.
5. E XPLICIT E RROR B OUNDS
In this section, we use the bounds on MIG and the implicit error bounds from the previous subsection to provide explicit (in n ) bounds on the error.
It is clear that with no assumption on the distribution of the data, generalization error cannot be nontrivially bounded.
For example, if all the data points are collected from a small region of the input domain, it is not expected for the error to be small far from this small region.
We here consider a dataset that is distributed over the input space in a sufﬁciently informative way.
For this purpose, we introduce a data collection module which collects the data points based on the current uncertainty level of the kernel model.
In particular, consider a dataset ˜ D n collected as follows: x i = arg max x ∈X σ i − 1 ( x ) , where σ i ( . ) is deﬁned above.
We have the following result.
Theorem 3 Consider the neural kernels with l ≥ 2 , on a hyperspherical domain X = S
d − 1 .
Consider a model ˆ f n trained on ˜ D n .
Under Assumption 1, with probability at least 1 − δ , uniformly in x ∈ X ,
[Equation]
[Equation]
Proof Sketch.
The proof follows the same steps as in the proof of Theorem 3 of Vakili et al.
(2021a).
The key step is bounding the total uncertainty in the model with the information gain that is (cid:80)
[Equation]
( Y
[Equation]
F ) (Srinivas et al., 2010).
This allows us to bound the σ n in the right hand side of equation 7 by (cid:112) γ κ ( n ) /n up to multiplicative absolute constants.
Plugging in the bounds on γ κ ( n ) from Theorem 2, and using the implicit error bound given in equation(with a union bound on a discretization of the domain with size O ( n d ) ), we obtain equation 8. A detailed proof is given in Appendix H.
[Equation]
E XPERIMENTS
In this section, we provide experimental results on the error rates.
In our experiments, we create synthetic data from a true model f that belongs to the RKHS of a NT kernel κ .
For this purpose, we create two random vectors ˆ X n 0 ∈ X {+n 
+}
and ˆ Y n 0 ∈ R n 0 , and let f ( . )
=
ˆ k (cid:62) n 0 ( . )
( ˆ K n 0 + δ
I n 0 ) − 1 ˆ Y n 0 , where ˆ k n 0 and ˆ K n 0 are deﬁned similar to k n and K n in Subsection 5.1.
We then generate datasets
D n of various sizes n = 2 i , with i = 1 , 2 , . . .
, 13 , according to the underlying model f .
We train the model to obtain ˆ f n .
Figure 2 shows the error rate versus the size n of the dataset for various d and s .
The experiments show that the error converges to zero at a rate satisfying the bounds given in Theorem 3.
In addition, as analytically predicted, the absolute value of the error rate exponent increases with s and decreases with d .
Our experiments use the neural-tangents library (Novak et al., 2019).
See Appendix I for more details.
D ISCUSSION
Our bounds on MIG may be of independent interest in other problems.
Recent works on RL and bandit problems, which use overparameterized neural network models, derived an ˜ O ( γ k ( n ) √ n ) regret bound (see, e.g., Zhou et al., 2020;
Yang et al., 2020; ZHANG et al., 2021; Gu et al., 2021).
These works however did not characterize the bound on γ k ( n ) , leaving the regret bounds implicit.
A consequence of our bounds on γ k ( n ) is an explicit (in n ) regret bound for these RL and bandit problems.
Our results however have mixed implications by showing that the existing regret bounds are not necessarily sublinear in n .
In particular, plugging in our bound on γ κ NT ,s ( n ) into the existing regret bounds, we get ˜ O
( n 1 .
5 d
+ s − 2 d +2 s − 2 ) , which is sublinear only when s > d 2 (that, e.g., excludes the
ReLU activation function).
Our analysis of various activation functions is thus essential in showing sublinear regret bounds for at least some cases.
As recently formalized in Vakili et al.
(2021c)
, it remains an open problem whether the analysis of RL and bandit algorithms in kernel regimes can improve to have an always sublinear regret bound.
We note that this open problem and the analysis of RL and bandit problems were not considered in this work.
Our remark is mainly concerned with the consequences of our bound on MIG, which appears in the regret bounds.
The recent work of Wang et al.
(2020) proved that the L 2 norm of the error is bounded as (cid:107)
f − ˆ f n (cid:107) L 2 =
O ( n − d 2 d − 1 ) , for a two layer neural network with ReLU activation functions in kernel regimes.
Bordelon et al.
(2020) decomposed the L 2 norm of the error into a series corresponding to eigenfunctions of the NT kernel and provided bounds based on the corresponding eigenvalues.
Our results are stronger as they are given in terms of absolute error instead of L 2 norm.
In addition, we provide explicit error bounds depending on differentiability of the activation functions.
The MIG has been studied for popular GP kernels such as Mat´ern and Squared Exponential (Srinivas et al., 2010; Vakili et al., 2021b).
Our proof technique is similar to that of Vakili et al.
(2021b).
Their analysis however does not directly apply to the NT kernel on the hypersphere.
The reason is that Vakili et al.
(2021b) assume uniformly bounded eigenfunctions.
In our analysis, we use a
Mercer decomposition of the NT kernel in the basis of spherical harmonics.
Those are not uniformly bounded.
Technical details are provided in the analysis of Theorem 2.
Applying a proof technique similar to Srinivas et al.
(2010) results in suboptimal bounds in our case.
Our work may be relevant to the recent developments in the ﬁeld of implicit neural representations (Mildenhall et al., 2020;
Sitzmann et al., 2020; Fathony et al., 2020).
In this line of work, the input to a neural network often represents the coordinates of a plane (image), a camera position or angle, and the function to be approximated is the image or the scene itself.
Interestingly, it has been observed that smooth activation functions perform better than ReLU in these applications, since the models are also smooth (Sitzmann et al., 2020).
A G ENERAL N OTATION
In this section, we formally deﬁne our general notations.
We use the notations 0 n and I n to denote the zero vector and the square identity matrix of dimension n , respectively.
For a matrix M (a vector v ), M (cid:62) ( v (cid:62) ) denotes its transpose.
In addition, det and log det are used to denote determinant of M and its logarithm, respectively.
The notation (cid:107) v (cid:107) l 2 is used to denote the l 2 norm of a vector v .
For v ∈ R d and a symmetric positive deﬁnite matrix Σ ∈ R d × d , N ( v, Σ) denotes a normal distribution with mean v and covariance Σ .
The Kronecker delta is denoted by δ i,j .
The notation S d − 1 denotes the d dimensional hypersphere in R d .
For example, S 2 ⊂ R 3 is the usual sphere.
The notations o and O denote the standard mathematical orders, while ˜ O is used to denote O up to logarithmic factors.
For two sequences a n , b n : N → R , we use the notation a n ∼ b n , when a n = O ( b n ) and b n = O ( a n ) .
For s ∈ N , we deﬁne (2 s − 1)!!
= (cid:81) si =1 (2 i − 1) .
For example, 3!!
= 3 and 5!!
= 15 .
For a normed space H , we use (cid:107) .
(cid:107) H to denote the norm associated with H .
For two normed spaces
H 1 , H 2 , we write H 1 ⊂ H 2 , if the following two conditions are satisﬁed.
First, H 1 ⊂ H 2 as sets.
Second, there exist a constant c 1 > 0 , such that (cid:107) f (cid:107) H 2 ≤ c 1 (cid:107) f (cid:107) H 1 , for all f ∈ H 1 .
We also write
H 1 ≡ H 2 , when both H 1 ⊂ H 2 and H 2 ⊂ H 1 .
The derivative of a : R → R is denoted by a (cid:48) .
We deﬁne 0
= 0 , so that a 0 ( x ) = (max(0 , x ))corresponds to the step function: a 0 ( x ) = 0 ,
when x ≤ 0 , and a 0 ( x ) = 1 , when x > 0 .
B M ERCER ’ S T HEOREM
In this section, we give an overview of Mercer’s theorem, as well as the the reproducing kernel Hilbert spaces (RKHSs) associated with the kernels.
Mercer’s theorem (Mercer, 1909) provides a spectral decomposition of the kernel in terms of an inﬁnite dimensional feature map (see, e.g. Steinwart & Christmann (2008), Theorem 4 . 49 ).
Theorem 4 (Mercer’s Theorem ) Let X be a compact domain.
Let k be a continuous square integrable kernel with respect to a ﬁnite Borel measure µ .
Deﬁne a positive deﬁnite operator
T k
[Equation]
Then, there exists a sequence of eigenvalue-eigenfunction pairs { ( λ i , φ i ) } ∞ i =1 such that λ i ∈ R + , and T k φ i
= λ
i φ i , for i ≥ 1 .
Moreover, the kernel function can be represented as
[Equation]
where the convergence of the series holds uniformly on X × X .
The λ i and φ i deﬁned in Mercer’s theorem are referred to as Mercer eigenvalues and Mercer eigenfunctions, respectively.
Let H k denote the RKHS corresponding to k , deﬁned as a Hilbert space equipped with an inner product (cid:104) .
(cid:105)
H k satisfying the following: k ( ., x ) ∈ H k , ∀ x ∈ X , and (cid:104) f, k ( ., x ) (cid:105)
H k
= f ( x ) , ∀ x ∈ X , ∀ f ∈ H k (reproducing property).
As a consequence of Mercer’s theorem, H k can be represented in terms of { ( λ i , φ i ) } ∞ i =1 , that is often referred to as Mercer’s representation theorem (see, e.g., Steinwart & Christmann (2008), Theorem 4 . 51 ).
Theorem 5 (Mercer’s Representation Theorem) Let { ( λ i , φ i ) } ∞
i =1 be the Mercer eigenvalueeigenfunction pairs.
Then, the RKHS corresponding to k is given by
[Equation]
Mercer’s representation theorem indicates that { λ 12 i φ i } ∞ i =1 form an orthonormal basis for H k : (cid:104)
λ
12 i φ i , λ 12 i (cid:48) φ i (cid:48) (cid:105) H k
= δ i,i (cid:48) .
It also provides a constructive deﬁnition for the RKHS as the span of this orthonormal basis, and a constructive deﬁnition for the (cid:107) f (cid:107) H k as the l 2 norm of the weights [ w i ] ∞ i =1 vector.
C C LASSICAL A PPROACHES TO B OUNDING THE G ENERALIZATION E RROR
There are classical statistical learning methods which address the generalization error in machine learning models.
Two notable approaches are VC dimension (Bartlett, 1998; Anthony & Bartlett, 2009) and Rademacher complexity (Bartlett & Mendelson, 2002; Neyshabur et al., 2015).
In the former, the expected generalization loss is bounded by the square root of the VC dimension divided by the square root of n .
In the latter, the expected generalization loss scales with the Rademacher complexity of the hypothesis class.
In the case of a 2 layer neural network of width m , it is shown that both of these approaches result in a generalization bound of O ( (cid:112) m/n ) that is vacuous for overparameterized neural networks with a very large m .
A more recent approach to studying the generalization error is PAC Bayes.
It provides non-vacuous error bounds that depend on the distribution of parameters (Dziugaite & Roy, 2017).
However, the distribution of parameters must be known or be estimated in order to use those bounds.
Taking a different approach, Arora et al.
(2019) provides bounds on expected generalization error of wellconditioned 2 layer neural networks based on the Gram matrix of the input data, which does not depend on the distribution of model parameters as in PAC Bayes.
In contrast to all of these results, our bounds are explicit and are much stronger in the sense that they hold uniformly in x .
D P ROOF OF L EMMA
Lemma 1 offers a recursive relation over s for the RF kernel.
We prove the lemma by taking the derivative of κ s ( . ) , and applying the Stein’s lemma (given at the end of this section).
Let x = [0 , 0 , . .
, 0 , 1] (cid:62) and x (cid:48) =
. , √ 1 − u 2 , u ] (cid:62) , so
that x (cid:62) x (cid:48) = u , and x, x (cid:48)
∈ S
d − 1 .
We have
∂ ∂u
[Equation]
The ﬁrst equation is obtained by taking derivative of the term inside the expectation.
The second equation is obtained by applying Stein’s lemma to E [ a s ( w (cid:62) x ) sa s − 1 ( w (cid:62) x (cid:48) )
w d ] , where w d is the normally distributed random variable, also to E [ a s ( w (cid:62) x ) sa s − 1 ( w (cid:62) x (cid:48) )
uw
d − 1 √ 1 − u 2 ] , where w d − 1 is the normally distributed random variable.
Taking into account the constant normalization of the RF kernel c 2 = 2 (2 s − 1)!
! , we have κ
[Equation]
which proves the lemma.
Lemma 2 (Stein’s Lemma) Suppose X ∼ N (0 , 1) is a normally distributed random variable.
Consider a function g : R → R such that both E [ Xg ( X )] and E [ g (cid:48)
( X )] exist.
We then have
E [ Xg ( X )] =
E [ g (cid:48)
( X )] .
The proof of Stein’s lemma follows from an integration by parts (see, e.g., Stein, 1986; Nourdin &
Peccati, 2009; Chen et al., 2011).
E P ROOF OF P ROPOSITION
This proposition follows from Theorem 1 of Bietti & Bach (2020), which proved that the eigendecay of a rotationally invariant kernel κ : [ − 1 , 1] → ∞ can be determined based on its asymptotic expansions around the endpoints ± 1 . Recall that a rotationally invariant kernel k can be represented as κ ( x (cid:62) x (cid:48) ) = k ( x, x (cid:48) ) .
Their result is formally given in the following lemma.
Lemma 3 (Theorem 1 in Bietti & Bach (2020))
Assume κ : [ − 1 , 1] → R is C ∞ and has the following asymptotic expansions around ±
[Equation]
for t > 0 , where p ± 1 are polynomials, and θ > 0 is not an integer.
Also, assume that the derivatives of κ admit similar expansions obtained by differentiating the above ones.
Then, there exists C d,θ such that, when i is even, if c +1 (cid:54) =
− c − 1 : ˜
λ i ∼ ( c +1 + c − 1 ) C d,θ i − d − 2 θ
+1 , and, when i is odd, if c +1 (cid:54) =
c − 1 : ˜
λ i ∼ ( c
+1 − c − 1 ) C d,θ l − d − 2 θ
+1 .
In the case | c +1 | = | c − 1 | , then we have ˜ λ
i =
o ( l − d − 2 θ +1 )
for one of the two parities.
If κ is inﬁnitely differentiable on [ − 1 , 1] so that no such θ exists, the ˜ λ i decays faster than any polynomial.
Building on this result, the eigendecays can be obtained by deriving the endpoint expansions for the
RF and NT kernels.
That is derived in Bietti & Bach (2020) for the ReLU activation function.
For completeness, here, we derive the endpoint expansions for RF and NT kernels associated with s −times differentiable activation functions a s ( . ) .
Recall that for a 2 layer network, the corresponding RF and NT kernels are given by
[Equation]
For the special cases of s = 0 and s = 1 , the following closed form expressions can be derived by taking the expectations.
[Equation]
where u = x (cid:62) x (cid:48) .
At the endpoints ± 1 , κ 0 ( u ) , κ 1 ( u ) have the following asymptotic expansions. [Equation]
[Equation]
These endpoint expansions where used in Bietti & Bach (2020) to obtain the eigendecay of the RF and NT kernels with ReLU activation functions.
We ﬁrst extend the results on the endpoint expansions and the eigendecay to a 2 layer neural network with s > 1 , in Subsection E.1.
Then, we extend the derivation to l > 2 layer neural networks, in
Subsection E.2.
In Subsection E.3, we show how the eigendecays in terms of ˜ λ
i can be translated to the ones in terms of λ i .
E. E NDPOINT E XPANSIONS FOR 2 L AYER N ETWORKS WITH s >
We ﬁrst note that the normalization constant c 2 = 2 (2 s − 1)!!
, suggested in Section 2, ensures κ s (1)
= 1 , for all s ≥ 1 .
The reason is that, by the well known values for the even moments of the normal distribution, we have
[Equation]
when x (cid:62) x (cid:48)
= 1 .
Also notice that κ s ( − 1) = 0 , for all s ≥ 1 .
The reason is that when x (cid:62) x (cid:48) = − 1 , at least one of w (cid:62) x and w (cid:62)
x (cid:48) is non-positive.
We note that the normalization constant is considered only for the convenience of some calculations, and does not affect the exponent in the eigendecays.
Speciﬁcally, scaling the kernel with a constant factor, scales the corresponding Mercer eigenvalues with the same constant.
The constant factor scaling does not affect the Mercer eigenfunctions.
From Lemma 1, recall
[Equation]
This is a key component in deriving the endpoint expansions for s > 1 .
In particular, this allows us to recursively obtain the endpoint expansions of κ s ( . ) from those of κ s − 1 ( . ) , by integration.
That results in the following expansions for κ s .
[Equation]
Here, p +1 ,s
[Equation]
t )
= − s 2 2 s − 1 (cid:82) p +1 ,s − 1 ( t ) dt , subject to p +1 ,s (0) =
1 (that follows form κ s (1) = 1 ).
Thus, p +1 ,s ( t ) can be obtained recursively, starting from p +1 , 1 ( t ) = 1 − t (see equation 9).
For example,
[Equation]
and so on.
Similarly, p − 1 ,s ( t ) can be expressed in closed form using p − 1 ,s ( t )
= s 2 2 s − 1 (cid:82)
p − 1 ,s − 1 ( t ) dt subject to p − 1 ,s (0)
=
(that follows from κ s ( − 1) = 0 ), and starting from p − 1 , 1 ( t ) = 0 , ∀ t (see equation 9).
That leads to p − 1 ,s ( t )
= 0 , ∀ s > 1 , t .
The exact expressions of p ± 1 ,s ( t ) however do not affect the eigendecay.
Instead, the constants c ± 1 ,s are important for the eigendecay, based on Lemma 3.
The constants can also be obtained recursively.
Starting from c +1 , 1 = 2 √ 2 3 π (see equation 9), c
+1 ,s = − 2 s 2 c
+1 ,s − 1 (2 s − 1)(2 s +1) .
Also starting from c − 1 , 1 = 2 √ 2 3 π (see equation 9), c − 1 ,s =
s 2 c
+1 ,s − 1 (2 s − 1)(2 s +1) .
This recursive relation leads to
[Equation]
[Equation]
and c +1 ,s = ( − 1) s − 1 c − 1 ,s .
In the case of RF kernel, applying Lemma 3, we have ˜ λ
i ∼ i − d − 2 s , for i having the opposite parity of s , and ˜ λ
i =
o ( i − d − 2 s ) for i having the same parity of s .
For the NT kernel , using 2, we have
[Equation]
Using the expansion of the RF kernel, we get
[Equation]
13) where
[Equation]
and
[Equation]
Thus, in the case of NT kernel, applying Lemma 3, we have ˜ λ
i ∼ i − d − 2 s +2 , for i having the same parity of s , and ˜ λ
i =
o ( i − d − 2 s +2 ) for i having the opposite parity of s .
E. E NDPOINT E XPANSIONS FOR l > 2 L AYER N ETWORKS
We here extend the endpoint expansions and the eigendecays to l > 2 later networks.
Recall κ ls ( u ) =
κ s ( κ l − 1 s ( u )) .
Using this recursive relation over l , we prove the following expressions
[Equation]
where p l ± 1 ,s are polynomials and c l ± 1 ,s are constants.
The notations p +1 ,s and c +1 ,s in Subsection E.1 correspond to p 2+1 ,s c 2+1 ,s , where we drop the superscript specifying the number l of layers, for 2 layer networks.
For the endpoint expansion at 1 , we have
[Equation]
Thus, we have
[Equation]
14) where q l,i +1 ,s is the coefﬁcient of t i in p l +1 ,s ( t ) .
For these coefﬁcients, we have
[Equation]
Starting from q 2 , 1+1 ,s = − s 2 2 s − 1 (which can be seen from the recursive expression of p +1 ,s given in
Section E.1), we get q l, 1+1 ,s = − s
( l − 1) (2 s − 1) l − 1 .
We thus have
[Equation]
That implies
[Equation]
when s > 1 .
The characterization of c l +1 ,s shows that our results do not apply to deep neural networks when s > 1 , in the sense that the constants grow exponentially in l (as stated in Remark 2).
For the endpoint expansion at − 1 , we have
[Equation]
where q l,i − 1 ,s is the coefﬁcient of t i in p l − 1 ,s ( t ) .
From the expression above we can see that
[Equation]
Thus, 0 ≤ q l, 0 − 1 ,s ≤ 1 .
In addition, the expression above implies
[Equation]
From Lemma 1, κ (cid:48) s ( u ) ≤ s
s − 1 , for all u .
Therefore,
[Equation]
when, s > 1 .
Comparing c l ± 1 ,a , we can see that for l > 2 , we have | c l +1 ,s | (cid:54) =
| c l − 1 ,s | .
Thus, for the RF kernel with l > 2 , applying Lemma 3, we have ˜
[Equation]
i ∼ i − d − 2 s .
For the NT kernel, recall
[Equation]
The second term is exactly the same as the RF kernel.
Recall the expression of κ (cid:48)
s based on κ s −from Lemma 1.
To ﬁnd the endpoint expansions of the ﬁrst term, we prove the following expressions for κ s − 1 ( κ l − 1 s ( u )) .
[Equation]
We use the notation used for the RF kernel to write the following expansion around
[Equation]
Thus, we have
[Equation] [Equation]
Recall, form the analysis of the RF kernel that q l, 1+1 ,s = − s 2 ( l − 1) (2 s − 1) l − 1 .
We thus have
[Equation]
That implies
[Equation]
For the endpoint expansion at − 1 , we have
[Equation]
We thus have
[Equation] [Equation]
Since κ (cid:48) s − 1 ≤ ( s − 1)
2 2 s −
[Equation]
we have
[Equation]
Now we use the end point expansions of κ ls and κ s − 1 ( κ l − 1 s ) to obtain the following endpoint expansions for κ l NT ,s .
[Equation]
We have
[Equation]
and,
[Equation]
Since κ s (1) =
[Equation]
by induction we have κ ls (1) = 1 , ∀ l ≥ 2 .
In addition, κ s − 1 ( κ l − 1 s (1))
=
κ s − 1 (1) = 1 , ∀ l ≥ 3 .
Therefore q (cid:48) l, 0+1 ,s , q l, 0+1 ,s = 1 .
Thus,
[Equation] [Equation]
Starting from q (cid:48)(cid:48) 2 , 0 +1 ,s = s 2 2 s − 1 +
[Equation]
we can derive the following expression for q (cid:48)(cid:48) l, 0 +1 ,s , when s > 1 ,
[Equation]
When s = 1 , q (cid:48)(cid:48) l, 0 +1 ,s = l , that is consistent with the results in Bietti & Bach (2020).
For c (cid:48)(cid:48)
l +1 ,s , we thus have
[Equation]
Replacing the expressions for q (cid:48)(cid:48) l − 1 , 0 +1 ,s and c (cid:48) l +1 ,s derived above, we obtain
[Equation]
For the constant in the expansion around − 1 , we have
[Equation]
27) where q (cid:48) l, 0 − 1 ,s =
κ s − 1 ( q l − 1 , 0 − 1 ,s ) is bounded between 0 and 1 .
Thus, starting from c (cid:48)(cid:48) 2 − 1 ,s = c − 1 ,s − 1 , we obtain
[Equation]
Comparing c (cid:48)(cid:48) l ± 1 ,a , we can see that for l > 2 , we have | c (cid:48)(cid:48)
l +1 ,s | (cid:54) =
| c (cid:48)(cid:48)
l − 1 ,s | .
Thus, for the NT kernel with l > 2 , applying Lemma 3, we have ˜ λ
i ∼ i − d − 2 s +2 .
E. T HE E IGENDECAYS IN T ERMS OF λ i
Recall the multiplicity N d,i =
i + d − 2 i (cid:0) i + d − 3 d − 2 (cid:1) of ˜ λ
i .
Here we take into account this multiplicity to give the eigendecay expressions in terms of λ i .
Using ( nk ) k ≤ (cid:0) nk (cid:1) ≤ ( nek ) k , for all k, n ∈ N , we have, for all i > 1 and d > 2 ,
[Equation]
where we used 2 < 2 i + d − 2 i < d +22 .
We thus have N d,i ∼ ( i − 1) d − 2 , for the scaling of N d,i with i , with constants given above.
Recall we deﬁne λ i = ˜
λ
i (cid:48) , for i and i (cid:48) which satisfy (cid:80) i (cid:48) − 1 i (cid:48)(cid:48)
=1 N d,i (cid:48)(cid:48) <
i ≤ (cid:80) i (cid:48)
i (cid:48)(cid:48)
=1 N d,i (cid:48)(cid:48) .
Using
N d,i ∼ ( i − 1) d − 2 , we have (cid:80) i (cid:48)
i (cid:48)(cid:48)
=1 N d,i (cid:48)(cid:48) ∼ i (cid:48)
d − 1 .
Thus λ i ∼ ˜ λ
i (cid:48)
=
˜
λ
i 1 d − 1 .
Replacing i with i 1 d − 1 , in the expressions derived for ˜ λ i in this section, we obtain the eigendecay expressions for λ
i reported in Table 2.
F P ROOF OF T HEOREM
The Mat´ern kernel can also be decomposed in the basis of spherical harmonics.
In particular,
Borovitskiy et al.
(2020) proved the following expression for the Mat´ern kernel with smoothness parameter ν on the hypersphere S
d − 1 (see, also Dutordoir et al., 2020, Appendix B
[Equation]
where ˜ φ i,j are the spherical harmonics.
Theorem 5 implies that the RKHS of Met´ern kernel is also constructed as a span of spherical harmonics.
Thus, in order to show the equivalence of the RKHSs of the neural kernels with various activation functions and a Mat´ern kernel with the corresponding smoothness, we show that the ratio between their norms is bounded by absolute constants.
Let f ∈ H lk NT ,s , with l ≥ 2 .
As a result of Mercer’s representation theorem, we have
[Equation]
Note that
[Equation]
For the NT kernel, from Proposition 1, ˜ λ i
=
O
( i − d − 2 s +2 ) .
Thus, when ν = s − 12 ,
[Equation]
So, with ν = s − 12 we have
[Equation]
That proves H k l NT ,s ⊂ H k ν .
A similar proof shows that when ν = s + 12 , H lk s ⊂ H k ν .
Now, let f ∈ H k ν .
As a result of Mercer’s representation theorem, we have f ( · ) = ∞ (cid:88) i =
[Equation]
=
[Equation]
=
[Equation]
˜
[Equation]
For the NT kernel with l > 2 , from Proposition 1, ˜ λ l ∼ l − d − 2 s +2 .
Thus, when ν = s − 12 , (
[Equation]
) ˜ λ
[Equation]
O (1) .
So, with ν = s − 12 we have
[Equation]
=
N d
[Equation] [Equation]
λ
[Equation]
= O ( ∞ (cid:88)
i =
N d
[Equation]
i
[Equation]
=
[Equation]
w
[Equation] [Equation]
=
O ( (cid:107) f (cid:107) 2 H kν ) .
That proves, for l > 2 , when ν = s − 12 , H k ν ⊂ H lk NT ,s .
A similar proof shows that for l > 2 , when ν = s + 12 , H k ν ⊂ H lk s , which completes the proof of Theorem 1.
G P ROOF OF T HEOREM
Recall the deﬁnition of the information gain for a kernel κ ,
[Equation]
To bound the information gain, we deﬁne two new kernels resulting from the partitioning of the eigenvalues of the neural kernels to the ﬁrst M eigenvalues and the remainder.
In particular, we deﬁne
[Equation]
We present the proof for the NT kernel.
A similar proof applies to the RF kernel.
The truncated kennel at M eigenvalues, ˜ κ ( x (cid:62) x ) , corresponds to the projection of the RKHS of κ onto a ﬁnite dimensional space with dimension (cid:80) Mi
=1 (cid:80) N d,i j =1 .
We use the notations ˜ K n and ˜˜ K n to denote the kernel matrices corresponding to ˜ κ and ˜˜ κ , respectively.
As proposed in Vakili et al.
(2021b), we write
[Equation]
The analysis in Vakili et al.
(2021b) crucially relies on an assumption that the eigenfunctions are uniformly bounded.
In the case of spherical harmonics however (see, e.g., Stein & Weiss, 2016,
Corollary 2 . 9 )
[Equation]
Thus, their analysis does not apply to the case of neural kernels on the hypersphere.
To bound the two terms on the right hand side of equation 30, we ﬁrst prove in Lemma 4 that ˜˜ κ ( x, x (cid:48) )
approaches 0 as M grows (uniformly in x and x (cid:48) ), with a certain rate.
Lemma 4 For the NT kernel, we have ˜˜ κ ( x (cid:62) x (cid:48) )
=
O ( M − 2 s +1 ) , for all x, x (cid:48)
∈ X .
Proof of Lemma 4: Recall that the eigenspace corresponding to ˜ λ
i has dimension N d,i and consists of spherical harmonics of degree i .
Legendre addition theorem (Maleˇcek & N´aden´ık, 2001) states
[Equation] [Equation]
where d is the geodesic distance on the hypersphere, C ( d − 2) /
2 i are Gegenbauer polynomials (those are the same as Legendre polynomials up to a scaling factor), and the constant c i,d is
[Equation]
We thus have
[Equation]
Here, the inequality follows from C ( d − 2) /
i
[Equation]
(1) , because the Gegenbauer polynomials attain their maximum at the endpoint 1 .
For the fourth line we used N d,i ∼ i d − 2 .
The implied constants include the implied constants in N d,i ∼ i d − 2 given in Section E.3, and Γ(( d − 2) / 2) 2 π (( d − 2) / 2) .
(cid:3)
We also introduce a notation N M = (cid:80) Mi
=1 N d,i for the number of eigenvalues corresponding to the spherical harmonics of degree up to M , taking into account their multiplicities, that satisﬁes
N M ∼ M d − 1 (see Section E.3).
We are now ready to bound the two terms on the right hand side of equation 30.
Let us deﬁne Φ n,N M =
[ φ N M ( x 1 ) , φ N M ( x 2 ) , . .
. , φ N M ( x n )] (cid:62) , an n × N M matrix which stacks the feature vectors φ N M ( x i ) =
[ φ j ( x i )] N M j =1 , i = 1 , . .
. , n , at the observation points, as its rows.
Notice that
[Equation]
where Λ N M is the diagonal matrix of the eigenvalues deﬁned as [Λ N M ] i,j = λ
i δ i,j .
Now, consider the Gram matrix
[Equation]
As it was shown in Vakili et al.
(2021b), by matrix determinant lemma, we have
[Equation]
To upper bound the second term on the right hand side of equation 30, we use Lemma 4.
In particular, since
[Equation]
and [ ˜˜ K n ] i,i
=
O
( M − 2 s +1 ) , we have
[Equation]
Therefore log
[Equation]
where for the last line we used log(1 + z ) ≤ z which holds for all z ∈ R .
We thus have γ k ( n ) =
O ( M d − 1 log( n ) + nM − 2 s +1 ) .
Choosing M ∼ n 1 d
+
s
− 2 (log( n )) − 1 d
+2 s − 2 , we obtain
[Equation]
[Equation]
For example, with ReLU activation functions, we have
[Equation]
H P ROOF OF T HEOREM
As stated in the paper, the proof follows the same steps as in the proof of Theorem 3 of Vakili et al.
(2021a).
Their theorem holds for general kernels provided the bound on MIG.
We have adopted their theorem for the special case of neural kernels, inserting our novel bounds on MIG of the neural kernels given in Theorem 2.
For completeness, we include a detailed proof here.
The proof consists of tow components.
First, we bound the uncertainty σ n ( x ) in terms of γ k ( n ) .
Our novel bounds on γ k ( n ) of neural kernels given in Theorem 2 allows us to derive explicit bounds on σ n ( x ) .
Then, we bound the error | f ( x ) − ˆ f n ( x ) | in terms of σ n ( x ) , using implicit error bounds.
Recall the way that the dataset ˜ D n is collected: x i = arg max x ∈X σ i − 1 ( x ) .
This ensures that, ∀ x ∈ X , σ i − 1 ( x ) ≤ σ i − 1 ( x i ) .
Due to positive deﬁniteness of the kernel matrix, conditioning on a larger dataset reduces the uncertainty.
Thus ∀ x ∈ X and ∀ i ≤ n , σ n ( x ) ≤ σ i ( x ) .
Therefore σ 2 n ( x ) is upper bounded by the average of σ 2 i − 1 ( x i ) over i : ∀ x ∈ X
[Equation]
[Equation]
It is shown that (e.g., see, Srinivas et al., 2010, Lemmas 5 . 3 , 5 . 4 )
[Equation]
Thus, combining equation 35 and equation 36, and by deﬁnition of γ k ( n ) , we have, ∀ x ∈ X
[Equation]
n
[Equation]
Now, inserting our bounds on γ κ l NT ,s ( n ) and γ κ ls ( n ) of the neural kernels from Theorem 2, we obtain, ∀ x ∈ X
[Equation]
The second component of the proof is summarized in the following lemma.
{+Lemma 
+}
Under Assumption 1, we have, with probability at least 1 − δ , ∀ x ∈ X
[Equation]
[Equation]
where C is an absolute constant, and B is the upper bound on RKHS norm of f given in Assumption 1.
Lemma 5 follows from equation 7 and a probability union bound over a discretization of the domain.
A detailed proof of this lemma is given at the end of this section.
Inserting the bounds on σ n ( x ) from equation 38 in Lemma 5, we have, with probability at least 1 − δ , uniformly in x ,
[Equation]
the case of RF kernel .
That completes the proof.
Proof of Lemma
Recall the implicit error bound given in equation 7: For a ﬁxed x ∈ X , under Assumption 1, we have, with probability at least 1 − δ (Vakili et al., 2021a),
[Equation]
where β ( δ ) =
B + Rλ (cid:113) 2 log( 2 δ ) .
Lemma 5 extends this inequality to a uniform bound in x , using a probability union bound on a discretization of the domain.
For f ∈ H k , with (cid:107) f (cid:107) H k ≤ B , and for n ∈ N , there exists a ﬁne discretization X n of X with size | X n | such that f ( x ) − f
([ x ] n ) ≤ 1 √ n , where [ x ] n = arg min x (cid:48)
∈
X n (cid:107) x (cid:48) − x (cid:107) l 2 is the closest point in X n to x , and | X n | ≤ cB d − 1 n ( d − 1) / 2 , where c is an absolute constant independent of n and
B (Chowdhury & Gopalan, 2017; Vakili et al., 2021a).
Under Assumption 1, it can be shown that: with probability at least 1 − δ/ 2 (Vakili et al., 2021a,
Lemma 4 )
[Equation]
Note that ˆ f n is a random function, where the randomness comes from the randomness in observation noise.
We thus conclude that there is a discretization ˜ X n with size | ˜ X n | ≤ c ( U ( δ ))
d − 1 n ( d − 1) / 2 such that f ( x ) − f
([ x ] n ) ≤ 1 √ n , and with probability at least 1 − δ/ 2 ,
[Equation]
Let δ (cid:48)
=
δ
[Equation]
n
[Equation]
A probability union bound over the discretization ˜ X n implies that: with probability at least 1 − δ/ 2 , uniformly in x ∈ ˜ X n ,
[Equation]
Accounting for the discretization error from equation 42, and a probability union bound over equation 41 and equation 43, we have, with probability 1 − δ , uniformly in x ∈ X ,
[Equation]
The ﬁrst line holds by triangle inequality, the second line comes from the discretization error, and the third line holds by equation 43.
Inserting the value of U ( δ )
=
B + R √ nλ (cid:113) 2 log( 2 nδ ) in δ (cid:48) , and the value of δ (cid:48) =
[Equation]
[Equation]
in β ( · )
=
B + Rλ (cid:113) 2 log( 2 · ) , we arrive at the lemma.
I D ETAILS ON THE E XPERIMENTS
In this section, we provide further details on the experiments shown in the main text, Section 6.
The code will be made available upon the acceptance of the paper.
We consider NT kernels κ NT ,s ( . ) , with s = 1 , 2 , 3 , which correspond to wide fully connected 2 layer neural networks with activation functions a s ( . ) .
In the ﬁrst step, we create a synthetic function f belonging to the RKHS of a NT kernel κ .
For this purpose, we randomly generate n 0 = 100 points on the hypersphere S d − 1 .
Let ˆ X n
=
[ˆ x i ] n 0 i =1 denote the vector of these points.
We also randomly sample ˆ Y n 0 =
[ˆ y i ] n
i =1 from a multivariate Gaussian distribution
N ( 0 n 0 , K n 0 ) , where [ ˆ K n 0 ] i,j = κ (ˆ x (cid:62) i ˆ x j ) .
We deﬁne a function g ( . )
=
ˆ k (cid:62) n 0 ( . )
( ˆ K [-n 0 + δ 
-][Equation]
ˆ Y
[Equation] where δ 2 = 0 . 01 and [ ˆ K n 0 ( x )] i = κ ( x (cid:62) ˆ x i ) .
We then normalize g with its range to obtain f ( . )
=
g ( . )
max x ∈X g ( x ) − min x ∈X g ( x ) .
For a ﬁxed ˆ X n
and ˆ Y n 0 , g is a linear combination of partial applications of the kernel.
Thus g is in the RKHS of κ , and its RKHS norm can be bounded as follows.
[Equation]
The second line follows from the reproducing property.
We thus can see that for each ﬁxed ˆ X n 0 and ˆ Y n 0 , g (and consequently f ) belong to the RKHS of κ .
The values of max x ∈X g ( x ) and min x ∈X g ( x ) are numerically approximated by sampling 10 ,points on the hypersphere, and choosing the maximum and minimum over the sample.
We then generate the training datasets D n of the sizes n = 2 i , with i = 1 , 2 , . .
. , 13 , by sampling n points X n =
[ x i ] ni =1 on the hypersphere, uniformly at random.
The values Y n =
[ y i ] ni = are generated according to f .
We then train the neural network model to obtain ˆ f n ( . ) .
The error max x ∈X | f ( x ) − ˆ f n ( x ) | is then numerically approximated by sampling 10 , 000 random points on the hypersphere and choosing the maximum of the sample.
We have considered 9 different cases for the pairs of the kernel and the input domain.
In particular, the experiments are run for each κ NT ,s , s = 1 , 2 , 3 on all S d − 1 , d = 2 , 3 , 4 .
In addition, each one of these 9 experiments is repeated 20 times ( 180 experiments in total).
In Figure 2 we plot max x ∈X | f ( x ) − ˆ f n ( x ) | versus n , averaged over 20 repetitions, for each one of the 9 experiments.
Note that for max x ∈X | f ( x ) − ˆ f n ( x ) | ∼ n α , we have log(max x ∈X | f ( x ) − ˆ f n ( x ) | )
=
α log( n ) + constant.
Thus, in our log scale plots, the slope of the line represents the exponent of the error rate.
As predicted analytically, we see all the exponents are negative (the error converges to 0 ).
In addition, the absolute value of the exponent is larger, when s is larger or d is smaller.
The bars in the plot on the left in Figure 2, show the standard deviation of the exponents.
For training of the model, we have used neural-tangents library (Novak et al., 2019) that is based on JAX (Bradbury et al., 2018).
The library is primarily suitable for κ NT , 1 ( . )
corresponding to the
ReLU activation function.
We thus made an amendment by directly feeding the expressions of the
RF kernels, κ s , s = 2 , 3 , to the stax.
Elementwise layer provided in the library.
Below we give these expressions
[Equation]
where θ = arccos( u ) .
We derived these expressions using Lemma 1 in a recursive way starting from κ 1 ( . ) .
Also, see Cho & Saul (2009), which provides a similar expression for κ 2 ( . ) and a general method to obtain κ s ( . ) for other values of s .
We note that we only need to supply κ s ( . ) to the neural-tangents library.
The
NT kernel κ NT ,s ( . ) will then be automatically created.
Our experiments run on a single GPU machine ( GeForce RTX 2080 Ti ) with 11 GB of VRAM memory.
Each one of the 180 experiments described above takes approximately 4 minutes to run.