C HARACTERIZING THE SPECTRUM OF THE NTK VIA A POWER SERIES EXPANSION
A BSTRACT
Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the inﬁnite width limit.
We provide expressions for the coefﬁcients of this power series which depend on both the Hermite coefﬁcients of the activation function as well as the depth of the network.
We observe faster decay of the Hermite coefﬁcients leads to faster decay in the NTK coefﬁcients and explore the role of depth.
Using this series, ﬁrst we relate the effective rank of the NTK to the effective rank of the input-data Gram.
Second, for data drawn uniformly on the sphere we study the eigenvalues of the NTK, analyzing the impact of the choice of activation function.
Finally, for generic data and activation functions with sufﬁciently fast Hermite coefﬁcient decay, we derive an asymptotic upper bound on the spectrum of the NTK.
I NTRODUCTION
Neural networks currently dominate modern artiﬁcial intelligence, however, despite their empirical success establishing a principled theoretical foundation for them remains an active challenge.
The key difﬁculties are that neural networks induce nonconvex optimization objectives (Sontag &
Sussmann, 1989) and typically operate in an overparameterized regime which precludes classical statistical learning theory (Anthony & Bartlett, 2002).
The persistent success of overparameterized models tuned via non-convex optimization suggests that the relationship between the parameterization, optimization, and generalization is more sophisticated than that which can be addressed using classical theory.
A recent breakthrough on understanding the success of overparameterized networks was established through the Neural Tangent Kernel (NTK) (Jacot et al., 2018).
In the inﬁnite width limit the optimization dynamics are described entirely by the NTK and the parameterization behaves like a linear model (Lee et al., 2019).
In this regime explicit guarantees for the optimization and generalization can be obtained (Du et al., 2019a;b; Arora et al., 2019a; Allen-Zhu et al., 2019; Zou et al., 2020).
While one must be judicious when extrapolating insights from the NTK to ﬁnite width networks (Lee et al., 2020), the NTK remains one of the most promising avenues for understanding deep learning on a principled basis.
The spectrum of the NTK is fundamental to both the optimization and generalization of wide networks.
In particular, bounding the smallest eigenvalue of the NTK Gram matrix is a staple technique for establishing convergence guarantees for the optimization (Du et al., 2019a;b; Oymak &
Soltanolkotabi, 2020).
Furthermore, the full spectrum of the NTK Gram matrix governs the dynamics of the empirical risk (Arora et al., 2019b), and the eigenvalues of the associated integral operator characterize the dynamics of the generalization error outside the training set (Bowman & Montufar, 2022; Bowman & Mont´ufar, 2022).
Moreover, the decay rate of the generalization error for Gaussian process regression using the NTK can be characterized by the decay rate of the spectrum (Caponnetto & De Vito, 2007;
Cui et al., 2021; Jin et al., 2022).
The importance of the spectrum of the NTK has led to a variety of efforts to characterize its structure via random matrix theory and other tools (Yang & Salman, 2019; Fan & Wang, 2020).
There is a broader body of work studying the closely related Conjugate Kernel, Fisher Information Matrix, and
Hessian (Poole et al., 2016; Pennington & Worah, 2017; 2018; Louart et al., 2018; Karakida et al., 2020).
These results often require complex random matrix theory or operate in a regime where the input dimension is sent to inﬁnity.
By contrast, using a just a power series expansion we are able to characterize a variety of attributes of the spectrum for ﬁxed input dimension and recover key results from prior work.
1. C ONTRIBUTIONS
In Theorem 3.1 we derive coefﬁcients for the power series expansion of the NTK under unit variance initialization, see Assumption 2.
Consequently we are able to derive insights into the NTK spectrum, notably concerning the outlier eigenvalues as well as the asymptotic decay.
• In Theorem 4.1 and Observation 4.2 we demonstrate that the largest eigenvalue λ 1 ( K ) of the NTK takes up an Ω(1) proportion of the trace and that there are O (1) outlier eigenvalues of the same order as λ 1 ( K ) .
• In Theorem 4.3 and Theorem 4.5 we show that the effective rank
Tr ( K ) /λ 1 ( K ) of the NTK is upper bounded by a constant multiple of the effective rank
Tr ( XX T ) /λ 1 ( XX T ) of the input data Gram matrix for both inﬁnite and ﬁnite width networks.
• In Corollary 4.7 and Theorem 4.8 we characterize the asymptotic behavior of the NTK spectrum for both uniform and nonuniform data distributions on the sphere.
1. R ELATED WORK
Neural Tangent Kernel (NTK): the NTK was introduced by Jacot et al.
(2018), who demonstrated that in the inﬁnite width limit neural network optimization is described via a kernel gradient descent.
As a consequence, when the network is polynomially wide in the number of samples, global convergence guarantees for gradient descent can be obtained (Du et al., 2019a;b; Allen-Zhu et al., 2019; Zou & Gu, 2019; Lee et al., 2019; Zou et al., 2020; Oymak & Soltanolkotabi, 2020; Nguyen & Mondelli, 2020; Nguyen, 2021).
Furthermore, the connection between inﬁnite width networks and Gaussian processes, which traces back to Neal (1996), has been reinvigorated in light of the NTK.
Recent investigations include Lee et al.
de G. Matthews et al.
; Novak et al.
Analysis of NTK Spectrum: theoretical analysis of the NTK spectrum via random matrix theory was investigated by Yang & Salman (2019); Fan & Wang (2020) in the high dimensional limit.
Velikanov & Yarotsky (2021) demonstrated that for ReLU networks the spectrum of the NTK integral operator asymptotically follows a power law, which is consistent with our results for the uniform data distribution.
Basri et al.
(2019) calculated the NTK spectrum for shallow ReLU networks under the uniform distribution, which was then expanded to the nonuniform case by Basri et al.
Geifman et al.
(2022) analyzed the spectrum of the conjugate kernel and NTK for convolutional networks with ReLU activations whose pixels are uniformly distributed on the sphere.
Geifman et al.
; Bietti & Bach (2021); Chen & Xu (2021) analyzed the reproducing kernel Hilbert spaces of the NTK for ReLU networks and the Laplace kernel via the decay rate of the spectrum of the kernel.
In contrast to previous works, we are able to address the spectrum in the ﬁnite dimensional setting and characterize the impact of different activation functions on it.
Hermite Expansion: Daniely et al.
(2016) used Hermite expansion to the study the expressivity of the Conjugate Kernel.
Simon et al.
(2022) used this technique to demonstrate that any dot product kernel can be realized by the NTK or Conjugate Kernel of a shallow, zero bias network.
Oymak &
Soltanolkotabi (2020) use Hermite expansion to study the NTK and establish a quantitative bound on the smallest eigenvalue for shallow networks.
This approach was incorporated by Nguyen &
Mondelli (2020) to handle convergence for deep networks, with sharp bounds on the smallest NTK eigenvalue for deep ReLU networks provided by Nguyen et al. (2021).
The Hermite approach was utilized by Panigrahi et al.
(2020) to analyze the smallest NTK eigenvalue of shallow networks under various activations.
Finally, in a concurrent work Han et al.
(2022) use Hermite expansions to develop a principled and efﬁcient polynomial based approximation algorithm for the NTK and
CNTK.
In contrast to the aforementioned works, here we employ the Hermite expansion to characterize both the outlier and asymptotic portions of the spectrum for both shallow and deep networks under general activations.
P RELIMINARIES
For our notation, lower case letters, e.g., x, y , denote scalars, lower case bold characters, e.g., x , y are for vectors, and upper case bold characters, e.g., X , Y , are for matrices.
For natural numbers k 1 , k 2 ∈ N we let [ k 1 ] = { 1 , . . .
, k 1 } and [ k 2 , k 1 ] =
{ k 2 , . .
. , k 1 } .
If k 2 > k 1 then [ k 2 , k 1 ] is the empty set.
We use (cid:107)·(cid:107) p to denote the p -norm of the matrix or vector in question and as default use (cid:107)·(cid:107) as the operator or 2-norm respectively.
We use 1 m × n ∈ R m × n to denote the matrix with all entries equal to one.
We deﬁne δ p =
c to take the value 1 if p = c and be zero otherwise.
We will frequently overload scalar functions φ : R → R by applying them elementwise to vectors and matrices.
The entry in the i th row and j th column of a matrix we access using the notation [ X ] ij .
The Hadamard or entrywise product of two matrices X , Y ∈ R m × n we denote X (cid:12) Y as is standard.
The p th Hadamard power we denote X (cid:12) p and deﬁne it as the Hadamard product of X with itself p times,
[Equation]
Given a Hermitian or symmetric matrix X ∈ R n × n , we adopt the convention that λ i ( X ) denotes the i th largest eigenvalue,
[Equation]
Finally, for a square matrix X ∈ R n × n we let Tr ( X ) =
(cid:80) n i =
[ X ] ii denote the trace.
2. H ERMITE E XPANSION
We say that a function f : R → R is square integrable with respect to the standard Gaussian measure γ ( z ) = 1 √ 2 π e − z 2 / 2 if E X ∼N (0 , 1) [ f ( X ) 2 ] < ∞ .
We denote by L 2 ( R , γ ) the space of all such functions.
The normalized probabilist’s Hermite polynomials are deﬁned as
[Equation]
and form a complete orthonormal basis in L 2 ( R , γ ) (O’Donnell, 2014, §11).
The Hermite expansion of a function φ ∈ L 2 ( R , γ ) is given by φ ( x )
= (cid:80) ∞
k =0 µ k ( φ ) h k ( x ) , where µ k ( φ ) =
E
X ∼N (0 , 1) [ φ ( X ) h k
( X )] is the k th normalized probabilist’s Hermite coefﬁcient of φ .
2. NTK P ARAMETRIZATION
In what follows, for n, d ∈ N let X ∈ R n × d denote a matrix which stores n points in R d row-wise.
Unless otherwise stated, we assume d ≤ n and denote the i th row of X n as x i .
In this work we consider fully-connected neural networks of the form f ( L +1) :
R d → R with L ∈ N hidden layers and a linear output layer.
For a given input vector x ∈ R d , the activation f ( l ) and preactivation g ( l ) at each layer l ∈ [ L + 1] are deﬁned via the following recurrence relations,
[Equation]
The parameters W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈
R m l are the weight matrix and bias vector at the l th layer respectively, m 0 =
d , m L +
= 1 , and φ :
R → R is the activation function applied elementwise.
The variables γ w , σ w ∈ R > 0 and γ b , σ b ∈ R ≥
correspond to weight and bias hyperparameters respectively.
Let θ l ∈ R p denote a vector storing the network parameters ( W ( h ) , b ( h ) )
lh =1 up to and including the l th layer.
The Neural Tangent Kernel (Jacot et al., 2018) ˜Θ ( l ) : R d × R d
→ R associated with f ( l ) at layer l ∈
[ L + 1] is deﬁned as
[Equation]
We will mostly study the NTK under the following standard assumptions.
Assumption 1.
NTK initialization.
At initialization all network parameters are distributed as N (0 , 1) and are mutually independent.
The activation function satisﬁes φ ∈ L 2 ( R , γ ) , is differentiable almost everywhere and its derivative, which we denote φ (cid:48) , also satisﬁes φ (cid:48)
∈
L 2 ( R , γ ) .
The widths are sent to inﬁnity in sequence, m 1 → ∞ , m 2 → ∞ , . . .
, m L → ∞ .
We refer to this regime as the sequential inﬁnite width limit.
Under Assumption 1, for any l ∈ [ L +1] , ˜Θ ( l ) ( x , y ) converges in probability to a deterministic limit Θ ( l ) : R d × R d → R (Jacot et al., 2018) and the network behaves like a kernelized linear predictor during training; see, e.g., Arora et al. (2019b); Lee et al.
; Woodworth et al.
Given access to the rows ( x i ) ni =1 of X the NTK matrix at layer l ∈
[ L + 1] , which we denote K l , is the n × n matrix with entries deﬁned as
[Equation]
E XPRESSING THE NTK AS A POWER SERIES
The following assumption allows us to study a power series for the NTK of deep network and with general activation functions. We remark that power series for the NTK of deep networks with positive homogeneous activation functions, namely ReLU, have been studied in prior works Han et al. ; Chen & Xu (2021); Bietti & Bach (2021); Geifman et al.
We further remark that while these works focus on the asymptotics of the NTK spectrum we also study the large eigenvalues.
Assumption 2.
The hyperparameters of the network satisfy γ 2 w + γ 2 b = 1 , σ 2 w E Z ∼N (0 , 1) [ φ ( Z ) 2 ] ≤and σ 2 b = 1 − σ 2 w E Z ∼N (0 , 1) [ φ ( Z ) 2 ] .
The data is normalized so that (cid:107) x i (cid:107) = 1 for all i ∈ [ n ] .
Recall under Assumption 1 that the preactivations of the network are centered Gaussian processes (Neal, 1996; Lee et al., 2018).
Assumption 2 ensures the preactivation of each neuron has unit variance and thus is reminiscent of the LeCun et al.
(2012), Glorot & Bengio (2010)
and
He et al.
(2015) initializations, which are designed to avoid vanishing and exploding gradients.
We refer the reader to Appendix A.3 for a thorough discussion.
Under Assumption 2 we will show it is possible to write the NTK not only as a dot-product kernel but also as an analytic power series on [ − 1 , 1] and derive expressions for the coefﬁcients.
In order to state this result recall, given a function f ∈ L 2 ( R , γ ) , that the p th normalized probabilist’s Hermite coefﬁcient of f is denoted µ p ( f ) , we refer the reader to Appendix A.4 for an overview of the Hermite polynomials and their properties.
Furthermore, letting ¯ a = ( a j ) ∞ j =0 denote a sequence of real numbers, then for any p, k ∈ Z ≥
we deﬁne
[Equation]
where
[Equation]
Here J ( p, k ) is the set of all k -tuples of nonnegative integers which sum to p and F ( p, k, ¯ a ) is therefore the sum of all ordered products of k elements of ¯ a whose indices sum to p .
We are now ready to state the key result of this section, Theorem 3.1, whose proof is provided in Appendix B.1.
Theorem 3.1.
Under Assumptions 1 and 2, for all l ∈ [ L + 1]
[Equation]
The series for each entry n [ K l ] ij converges absolutely and the coefﬁcients κ p,l are nonnegative and can be evaluated using the recurrence relationships
[Equation]
where
[Equation]
and
[Equation]
are likewise nonnegative for all p ∈ Z ≥ 0 and l ∈ [2 , L + 1] .
As already remarked, power series for the NTK have been studied in previous works, however, to the best of our knowledge Theorem 3.1 is the ﬁrst to explicitly express the coefﬁcients at a layer in terms of the coefﬁcients of previous layers.
To compute the coefﬁcients of the NTK as per Theorem 3.1, the Hermite coefﬁcients of both φ and φ (cid:48) are required.
Under Assumption 3 below, which has minimal impact on the generality of our results, this calculation can be simpliﬁed.
In short, under Assumption 3 υ p, 2 = ( p + 1) α p +1 , 2 and therefore only the Hermite coefﬁcients of φ are required.
We refer the reader to Lemma B.3 in Appendix B.2 for further details.
Assumption 3.
The activation function φ : R → R is absolutely continuous on [ − a, a ] for all a > 0 , differentiable almost everywhere, and is polynomially bounded, i.e., | φ ( x ) |
=
O ( | x | β ) for some β > 0 .
Further, the derivative φ (cid:48) : R → R satisﬁes φ (cid:48)
∈
L 2 ( R , γ ) .
We remark that ReLU, Tanh, Sigmoid, Softplus and many other commonly used activation functions satisfy Assumption 3.
In order to understand the relationship between the Hermite coefﬁcients of the activation function and the coefﬁcients of the NTK, we ﬁrst consider the simple two-layer case with L = 1 hidden layers.
From Theorem 3.
[Equation]
As per Table 1, a general trend we observe across all activation functions is that the ﬁrst few coefﬁcients account for the large majority of the total NTK coefﬁcient series. [Table]
[Table]
However, the asymptotic rate of decay of the NTK coefﬁcients varies signiﬁcantly by activation function, due to the varying behavior of their tails.
In Lemma 3.2 we choose ReLU, Tanh and
Gaussian as prototypical examples of activations functions with growing, constant, and decaying tails respectively, and analyze the corresponding NTK coefﬁcients in the two layer setting.
typographical ease we denote the zero mean Gaussian density function with variance σ 2 as ω σ ( z ) :
= (1 / √ 2 πσ 2 ) exp (cid:0) − z 2 / (2 σ 2 ) (cid:1) .
Lemma 3.2.
Under Assumptions 1 and 2,
if φ ( z ) =
ReLU ( z ) , then κ p, 2 = δ ( γ b > 0) ∪ ( p even )
Θ( p − 3 / 2 ) , 2.
if φ ( z ) =
Tanh ( z ) , then κ p, 2 = O (cid:16) exp (cid:16) − π √ p − 1(cid:17)(cid:17) , 3.
if φ ( z ) =
ω σ ( z ) , then κ p, 2 = δ ( γ b > 0) ∪ ( p even )
Θ( p 1 / 2 ( σ 2 + 1) − p ) .
The trend we observe from Lemma 3.2 is that activation functions whose Hermite coefﬁcients decay quickly, such as ω σ , result in a faster decay of the NTK coefﬁcients.
We remark that analyzing the rates of decay for l ≥ 3 is challenging due to the calculation of F ( p, k,
¯ α l − 1 ) (4). In Appendix B. we provide some preliminary results in this direction, illustrating how, in a speciﬁc setting, depth can lead to slower coefﬁcient decay. Additionally, in Appendix B.4 we show that the zeroth coefﬁcient strictly increases with depth.
Finally, we brieﬂy pause here to highlight the potential for using a truncation of (5) in order to perform efﬁcient numerical approximation of the inﬁnite width NTK.
We remark that this idea is also addressed in a concurrent work by Han et al.
(2022), albeit under a somewhat different set of assumptions 1 .
As per our observations thus far that the coefﬁcients of the NTK power series (5) typically decay quite rapidly, one might consider approximating Θ ( l ) by computing just the ﬁrst few terms in each series of (5).
Figure 2 in Appendix B.3 displays the absolute error between the truncated ReLU NTK and the analytical expression for the ReLU NTK, which is also deﬁned in Appendix B.3.
Letting ρ denote the input correlation then the key takeaway is that while for | ρ | close to one the approximation is poor, for | ρ | < 0 .
5 , which is arguably more realistic for real-world data, with just 50 coefﬁcients machine level precision can be achieved.
We refer the interested reader to Appendix B.3 for a proper discussion.
A NALYZING THE SPECTRUM OF THE NTK VIA ITS POWER SERIES
In this section, we consider a general kernel matrix power series of the form n K = (cid:80) ∞
p =
c p ( XX T ) (cid:12) p where { c p } ∞ p =0 are coefﬁcients and X is the data matrix.
According to Theorem 3.1, the coefﬁcients of the NTK power series (5) are always nonnegative, thus we only consider the case where c p are nonnegative.
We will also consider the kernel function power series, which we denote as K ( x 1 , x 2 ) =
(cid:80) ∞
p =
c p (cid:104) x 1 , x 2 (cid:105) p .
Later on we will analyze the spectrum of kernel matrix
K and kernel function K .
4. A NALYSIS OF THE UPPER SPECTRUM AND EFFECTIVE RANK
In this section we analyze the upper part of the spectrum of the NTK, corresponding to the large eigenvalues, using the power series given in Theorem 3.1.
Our ﬁrst result concerns the effective rank (Huang et al., 2022) of the NTK.
Given a positive semideﬁnite matrix A ∈ R n × n we deﬁne the effective rank of A to be
[Equation]
The effective rank quantiﬁes how many eigenvalues are on the order of the largest eigenvalue.
This follows from the Markov-like inequality
[Equation]
and the eigenvalue bound
[Equation]
Our ﬁrst result is that the effective rank of the NTK can be bounded in terms of a ratio involving the power series coefﬁcients.
As we are assuming the data is normalized so that (cid:107) x
i (cid:107) = 1 for all i ∈ [ n ] , then observe by the linearity of the trace
[Equation]
where we have used the fact that Tr (( XX T ) (cid:12) p ) = n for all p ∈ N .
On the other hand,
[Equation]
Combining these two results we get the following theorem.
Theorem 4.1.
Assume that we have a kernel Gram matrix K of the form n K = (cid:80) ∞
p =
c p ( XX T ) (cid:12) p where c 0 (cid:54) = 0 .
Furthermore, assume the input data x i are normalized so that (cid:107) x i (cid:107) =
1 for all i ∈ [ n ] .
Then
[Equation]
on functions the By Theorem 3.1 c 0 (cid:54) = 0 provided the network has biases or the activation function has nonzero
Gaussian expectation (i.e., µ 0 ( φ ) (cid:54) = 0 ).
Thus we have that the effective rank of K is bounded by an
O (1) quantity.
In the case of ReLU for example, as evidenced by Table 1, the effective rank will be roughly 2 .
3 for a shallow network.
By contrast, a well-conditioned matrix would have an effective rank that is Ω( n ) .
Combining Theorem 4.1 and the Markov-type bound (10) we make the following important observation.
Observation 4.2.
The largest eigenvalue λ 1 ( K ) of the NTK takes up an Ω(1) fraction of the entire trace and there are O (1) eigenvalues on the same order of magnitude as λ 1 ( K ) , where the O (1) and Ω(1) notation are with respect to the parameter n .
While the constant term c 0 1 n × n in the kernel leads to a signiﬁcant outlier in the spectrum of K , it is rather uninformative beyond this.
What interests us is how the structure of the data X manifests in the spectrum of the kernel matrix K .
For this reason we will examine the centered kernel matrix (cid:101) K :=
K − c 0 n 1 n × n .
By a very similar argument as before we get the following result.
Theorem 4.3.
Assume that we have a kernel Gram matrix K of the form n K = (cid:80) ∞
p =
c p ( XX T ) (cid:12) p where c 1 (cid:54) = 0 .
Furthermore, assume the input data x i are normalized so that (cid:107) x i (cid:107) =
1 for all i ∈ [ n ] .
Then the centered kernel (cid:101) K :=
K − c 0 n 1 n × n satisﬁes
[Equation]
Thus we have that the effective rank of the centered kernel (cid:101) K is upper bounded by a constant multiple of the effective rank of the input data Gram XX T .
Furthermore, we can take the ratio
[Equation]
as a measure of how much the NTK inherits the behavior of the linear kernel XX T : in particular, if the input data gram has low effective rank and this ratio is moderate then we may conclude that the centered NTK must also have low effective rank.
Again from Table 1, in the shallow setting we see that this ratio tends to be small for many of the common activations, for example, for ReLU it is roughly 1.3.
To summarize then from Theorem 4.3 we make the important observation.
Observation 4.4.
Whenever the input data are approximately low rank, the centered kernel matrix (cid:101) K
= K − c 0 n 1 n × n is also approximately low rank.
It turns out that this phenomenon also holds for ﬁnite-width networks at initialization.
Consider the shallow model
[Equation]
where x ∈ R d and w (cid:96) ∈ R d , a (cid:96) ∈
R for all (cid:96) ∈ [ m ] .
The following theorem demonstrates that when the width m is linear in the number of samples n then eﬀ( K ) is upper bounded by a constant multiple of eﬀ( XX T ) .
Theorem 4.5.
Assume φ ( x ) =
ReLU ( x ) and n ≥ d . Fix (cid:15) > 0 small.
Suppose that w 1 , . . .
, w m ∼
N (0 , ν 21 I d ) i.i.d.
and a 1 , . .
. , a m ∼ N (0 , ν 22 ) .
Set M = max i ∈
[ n ] (cid:107) x i (cid:107) 2 , and let Σ :=
E w ∼ N (0 ,ν 21 I ) [ φ ( Xw ) φ ( w T X T )] .
Then
[Equation]
sufﬁces to ensure that, with probability at least 1 − (cid:15) over the sampling of the parameter initialization,
[Equation]
where C > 0 is an absolute constant.
Many works consider the model where the outer layer weights are ﬁxed and have constant magnitude and only the inner layer weights are trained.
This is the setting considered by Xie et al.
(2017), Arora et al. (2019a), Du et al.
(2019b), Oymak et al.
, Li et al.
(2020), and Oymak & Soltanolkotabi (2020).
In this setting we can reduce the dependence on the width m to only be logarithmic in the number of samples n , and we have an accompanying lower bound.
See Theorem C.5 in the
Appendix C.2.3 for details.
In Figure 1 we empirically validate our theory by computing the spectrum of the NTK on both
Caltech101 (Li et al., 2022) and isotropic Gaussian data for feedforward networks.
We use the functorch 2 module in PyTorch (Paszke et al., 2019) using an algorithmic approach inspired by
Novak et al. (2022).
As per Theorem 4.1 and Observation 4.2, we observe all network architectures exhibit a dominant outlier eigenvalue due to the nonzero constant coefﬁcient in the power series.
Furthermore, this dominant outlier becomes more pronounced with depth, as can be observed if one carries out the calculations described in Theorem 3.1.
Additionally, this outlier is most pronounced for ReLU, as the combination of its Gaussian mean plus bias term is the largest out of the activations considered here.
As predicted by Theorem 4.3, Observation 4.4 and Theorem 4.5, we observe real-world data, which has a skewed spectrum and hence a low effective rank, results in the spectrum of the NTK being skewed.
By contrast, isotropic Gaussian data has a ﬂat spectrum, and as a result beyond the outlier the decay of eigenvalues of the NTK is more gradual.
These observations support the claim that the NTK inherits its spectral structure from the data.
We also observe that the spectrum for Tanh is closer to the linear activation relative to ReLU: intuitively this should not be surprising as close to the origin Tanh is well approximated by the identity.
Our theory provides a formal explanation for this observation, indeed, the power series coefﬁcients for Tanh networks decay quickly relative to ReLU.
We provide further experimental results in Appendix C.3, including for CNNs where we observe the same trends.
We note that the effective rank has implications for the generalization error.
The Rademacher complexity of a kernel method (and hence the NTK model) within a parameter ball is determined by its its trace (Bartlett & Mendelson, 2002).
Since for the
NTK λ 1 ( K ) =
O (1) , lower effective rank implies smaller trace and hence limited complexity.
matrix K the data Gram matrix XX T for Caltech101 and isotropic Gaussian datasets. To compute the NTK we randomly initialize feedforward networks of depths 2 and 5 with width 500 .
We use the standard parameterization and Pytorch’s default Kaiming uniform initialization in order to better connect our results with what is used in practice.
We consider a batch size of n = 200 and plot the ﬁrst 100 eigenvalues.
The thick part of each curve corresponds to the mean across 10 trials, while the transparent part corresponds to the 95% conﬁdence interval
A NALYSIS OF THE LOWER SPECTRUM
In this section, we analyze the lower part of the spectrum using the power series.
We ﬁrst analyze the kernel function K which we recall is a dot-product kernel of the form K ( x 1 , x 2 ) =
(cid:80) ∞
p =
c p (cid:104) x 1 , x 2 (cid:105) p .
Assuming the training data is uniformly distributed on a hypersphere it was shown by Basri et al.
; Bietti & Mairal (2019) that the eigenfunctions of K are the spherical harmonics.
Azevedo & Menegatto (2015) gave the eigenvalues of the kernel K in terms of the power series coefﬁcients.
Theorem 4.6.
[Azevedo & Menegatto (2015)] Suppose that the training data are uniformly sampled from the unit hypersphere S d , d ≥ 2 .
If the dot-product kernel function has the expansion K ( x 1 , x 2 ) = (cid:80) ∞[-p =
-]
{+p =
+} c p (cid:104) x 1 , x 2 (cid:105) p where c p ≥ 0 , then the eigenvalue of every spherical harmonic of frequency k is given by
[Equation]
where Γ is the gamma function.
A proof of Theorem 4.6 is provided in Appendix C.4 for the reader’s convenience.
This theorem connects the coefﬁcients c p of the kernel power series with the eigenvalues λ k of the kernel.
In particular, given a speciﬁc decay rate for the coefﬁcients c p one may derive the decay rate of λ k : for example, Scetbon & Harchaoui (2021) examined the decay rate of λ k if c p admits a polynomial decay or exponential decay.
The following Corollary summarizes the decay rates of λ k corresponding to two layer networks with different activations.
Corollary 4.7.
Under the same setting as in Theorem 4.6,
if c p =
Θ( p − a ) where a ≥ 1 , then λ k
=
Θ
( k − d − 2 a +2 ) , 2.
if c p = δ ( p even ) Θ( p − a ) , then λ k
= δ ( k even ) Θ( k − d − 2 a +2 ) , 3.
if c p = O (cid:0) exp (cid:0) − a √ p (cid:1)(cid:1) , then λ k
= O (cid:16) k − d +1 / 2 exp (cid:16) − a √ k (cid:17)(cid:17) , 4.
if c p =
Θ( p 1 / 2 a − p ) , then λ k
= O (cid:0) k − d +1 a − k (cid:1) and λ k = Ω (cid:0) k − d/ 2+1 2 − k a − k (cid:1) .
In addition to recovering existing results for ReLU networks Basri et al.
(2019); Velikanov & Yarotsky (2021);
Geifman et al.
; Bietti & Bach (2021), Corollary 4.7 also provides the decay rates for two-layer networks with Tanh and Gaussian activations.
As faster eigenvalue decay implies a smaller RKHS Corollary 4.7 shows using ReLU results in a larger RKHS relative to Tanh or Gaussian activations.
Numerics for Corollary 4.7 are provided in Figure 4 in Appendix C.3.
Finally, in Theorem 4.8 we relate a kernel’s power series to its spectral decay for arbitrary data distributions.
Theorem 4.8 (Informal) .
Let the rows of X ∈ R n × d be arbitrary points on the unit sphere.
Consider the kernel matrix n K = (cid:80) ∞ p =
c p (cid:0) XX T (cid:1) (cid:12) p and let r ( n ) ≤ d denote the rank of XX T .
Then
if c p =
O ( p − α ) with α > r ( n ) + 1 for all n ∈ Z ≥ 0 then λ n ( K )
=
O (cid:16) n − α − 1 r ( n ) (cid:17) , 2.
if c p =
O ( e − α √ p ) then λ n ( K ) =
O (cid:16) n 1 2 r ( n ) exp (cid:16) − α (cid:48) n 1 2 r ( n ) (cid:17)(cid:17) for any α (cid:48) <
α 2 − 1 / 2 r ( n ) , 3.
if c p =
O ( e − αp ) then λ n ( K )
=
O (cid:16) exp (cid:16) − α (cid:48) n 1 r ( n ) (cid:17)(cid:17) for any α (cid:48) <
α 2 − 1 / 2 r ( n ) .
Although the presence of the factor 1 /r ( n ) in the exponents of n in these bounds is a weakness,
Theorem 4.8 still illustrates how, in a highly general setting, the asymptotic decay of the coefﬁcients of the power series ensures a certain asymptotic decay in the eigenvalues of the kernel matrix.
A formal version of this result is provided in Appendix C.5 along with further discussion.
C ONCLUSION
Using a power series expansion we derived a number of insights into both the outliers as well as the asymptotic decay of the spectrum of the NTK, in particular highlighting the role of the activation function.
We performed our analysis without recourse to a high dimensional limit or the use of random matrix theory.
Interesting avenues for future work include better analyzing the role of depth as well as characterizing the outlier eigenvalues and spectrum as a whole for networks with convolutional, residual or transformer layers.
Reproducibility Statement To ensure reproducibility, we make the code public at https:// github.com/bbowman223/data_ntk
A PPENDIX
The appendix is organized as follows.
• Appendix A gives background material on Gaussan kernels, NTK, unit variance intitialization, and Hermite polynomial expansions.
• Appendix B provides details for Section 3.
• Appendix C provides details for Section 4.
A B ACKGROUND MATERIAL A. G AUSSIAN KERNEL
Observe by construction that the ﬂattened collection of preactivations at the ﬁrst layer ( g (1) ( x i ))
ni = form a centered Gaussian process, with the covariance between the α th and β th neuron being described by
[Equation]
Under the Assumption 1, the preactivations at each layer l ∈ [ L + 1] converge also in distribution to centered Gaussian processes (Neal, 1996; Lee et al., 2018).
We remark that the sequential width limit condition of Assumption 1 is not necessary for this behavior, for example the same result can be derived in the setting where the widths of the network are sent to inﬁnity simultaneously under certain conditions on the activation function (de G. Matthews et al., 2018).
However, as our interests lie in analyzing the limit rather than the conditions for convergence to said limit, for simplicity we consider only the sequential width limit.
As per Lee et al.
(2018, Eq. 4), the covariance between the preactivations of the α th and β th neurons at layer l ≥ 2 for any input pair x , y ∈ R are described by the following kernel,
[Equation]
We refer to this kernel as the Gaussian kernel.
As each neuron is identically distributed and the covariance between pairs of neurons is 0 unless α = β , moving forward we drop the subscript and discuss only the covariance between the preactivations of an arbitrary neuron given two inputs.
As per the discussion by Lee et al.
(2018, Section 2.3), the expectations involved in the computation of these Gaussian kernels can be computed with respect to a bivariate Gaussian distribution, whose covariance matrix has three distinct entries: the variance of a preactivation of x at the previous layer, Σ ( l − 1) ( x , x ) , the variance of a preactivation of y at the previous layer, Σ ( l ) ( y , y ) , and the covariance between preactivations of x and y , Σ ( l − 1) ( x , y ) .
Therefore the Gaussian kernel, or covariance function, and its derivative, which we will require later for our analysis of the NTK, can be computed via the the following recurrence relations, see for instance (Lee et al., 2018;
Jacot et al., 2018; Arora et al., 2019b; Nguyen et al., 2021),
[Equation]
A. N EURAL
T ANGENT K ERNEL (NTK)
As discussed in the Section 1, under Assumption 1 ˜Θ ( l ) converges in probability to a deterministic limit, which we denote Θ ( l ) .
This deterministic limit kernel can be expressed in terms of the Gaussian kernels and their derivatives from Section A.1 via the following recurrence relationships (Jacot et al., 2018, Theorem 1),
[Equation]
A useful expression for the NTK matrix, which is a straightforward extension and generalization of
Nguyen et al.
(2021, Lemma 3.1)
[Equation]
is provided in Lemma A.1 below.
Lemma A.1.
(Based on Nguyen et al. 2021, Lemma 3.1) Under Assumption
[Equation]
a sequence of positive semideﬁnite matrices ( G l ) L +1 l =1 in R n × n , and the related sequence ( ˙ G l ) L +1 l =2 also in R n × n , can be constructed via the following recurrence relationships,
[Equation]
The sequence of NTK matrices ( K l ) L +1 l =1 can in turn be written using the following recurrence relationship,
[Equation]
Proof.
For the sequence ( G l ) L +1 l =1 it sufﬁces to prove for any i, j ∈ [ n ] and l ∈
[ L + 1] that
[Equation]
and G l is positive semi-deﬁnite.
We proceed by induction, considering the base case l = 1 and comparing (13) with (11) then it is evident that
[Equation]
In addition, G 1 is also clearly positive semi-deﬁnite as for any u ∈ R n
[Equation]
We now assume the induction hypothesis is true for G l − 1 .
We will need to distinguish slightly between two cases, l = 2 and l ∈ [3 , L + 1] .
The proof of the induction step in either case is identical.
To this end, and for notational ease, let V
=
X , w ∼ N (0 , I d ) when l = 2 , and
V = (cid:112) G l − 1 , w ∼ N (0 , I n ) for l ∈ [3 , L + 1] .
In either case we let v i denote the i th row of V .
For any i, j ∈ [ n ]
[Equation]
Now let B 1 = v Ti w , B 2 = v Tj w and observe for any α 1 , α 2 ∈ R that α 1 B 1 + α 2 B 2 = (cid:80) nk ( α 1 v ik + α 2 v jk ) w k ∼ N (0 , (cid:107)
α 1 v i
+ α 2 v j (cid:107) 2 ) .
Therefore the joint distribution of ( B 1 , B 2 ) is a meanbivariate normal distribution.
Denoting the covariance matrix of this distribution as ˜ A ∈ R 2 × 2 , then [ G l ] ij can be expressed as
[Equation]
To prove [ G l ] i,j = Σ ( l ) it therefore sufﬁces to show that ˜
A =
A ( l ) as per (11).
This follows by the induction hypothesis as
[Equation]
Finally, G l is positive semi-deﬁnite as long as E w [ φ ( Vw ) φ ( Vw ) T ] is positive semi-deﬁnite.
Let
M ( w ) =
φ ( Vw ) ∈
R n × n and observe for any w that M ( w ) M ( w ) T is positive semi-deﬁnite.
Therefore E w [ M ( w ) M ( w ) T ] must also be positive semi-deﬁnite.
Thus the inductive step is complete and we may conclude for l ∈
[ L + 1] that
[Equation]
For the proof of the expression for the sequence ( ˙ G l ) L +1 l =2 it sufﬁces to prove for any i, j ∈ [ n ] and l ∈ [ L + 1] that
[Equation]
By comparing (13) with (11) this follows immediately from (15).
Therefore with (13) proven (14) follows from (12).
A. U NIT VARIANCE INITIALIZATION
The initialization scheme for a neural network, particularly a deep neural network, needs to be designed with some care in order to avoid either vanishing or exploding gradients during training
Glorot & Bengio (2010); He et al.
(2015); Mishkin & Matas (2016)
; LeCun et al.
Some of the most popular initialization strategies used in practice today, in particular LeCun et al.
and Glorot & Bengio (2010) initialization, ﬁrst model the preactivations of the network as Gaussian random variables and then select the network hyperparameters in order that the variance of these idealized preactivations is ﬁxed at one.
Under Assumption 1 this idealized model on the preactivations is actually realized and if we additionally assume the conditions of Assumption 2 hold then likewise the variance of the preactivations at every layer will be ﬁxed at one.
To this end, and as in
Poole et al.
; Murray et al.
(2022), consider the function V : R ≥
→ R ≥ 0 deﬁned as
[Equation]
Noting that V is another expression for Σ ( l ) ( x , x ) , derived via a change of variables as per Poole et al.
(2016), the sequence of variances (Σ ( l ) ( x , x ))
Ll =2 can therefore be generated as follows,
[Equation]
The linear correlation ρ ( l ) : R d × R d → [ − 1 , 1] between the preactivations of two inputs x , y ∈
R
d we deﬁne as
[Equation]
Assuming Σ ( l ) ( x , x ) = Σ ( l ) ( y , y )
= 1 for all l ∈
[ L + 1] , then ρ ( l ) ( x , y ) =
Σ ( l ) ( x , y ) .
Again as in Murray et al.
(2022) and analogous to (16), with Z 1 ,
Z 2 ∼ N (0 , 1) independent, U 1 :
=
Z 1 ,
U 2 ( ρ ) :
= ( ρZ 1 + (cid:112)
[Equation]
Z 2 ) 3 we deﬁne the correlation function R : [ − 1 , 1] →
[ − 1 , 1] as
[Equation]
Noting under these assumptions that R is equivalent to Σ ( l ) ( x , y ) , the sequence of correlations ( ρ ( l ) ( x , y ))
L l =2 can thus be generated as
[Equation]
As observed in Poole et al.
; Schoenholz et al.
(2017), R (1)
=
V (1) = 1 , hence ρ = 1 is a ﬁxed point of R .
We remark that as all preactivations are distributed as N (0 , 1) , then a correlation of one between preactivations implies they are equal.
The stability of the ﬁxed point ρ = 1 is of particular signiﬁcance in the context of initializing deep neural networks successfully.
Under mild conditions on the activation function one can compute the derivative of R , see e.g., Poole et al.
(2016); Schoenholz et al.
; Murray et al.
(2022), as follows,
[Equation]
Observe that the expression for ˙Σ ( l ) and R (cid:48) are equivalent via a change of variables (Poole et al., 2016), and therefore the sequence of correlation derivatives may be computed as
[Equation]
With the relevant background material now in place we are in a position to prove Lemma A.2.
Lemma A.2.
Under Assumptions 1 and 2 and deﬁning χ = σ 2 w E Z ∼N (0 , 1) [ φ (cid:48)
( Z ) 2 ] ∈ R > 0 , then for all i, j ∈ [ n ] ,
l ∈
[ L + 1]
• [ G n,l ] ij ∈ [ − 1 , 1] and [ G n,l ] ii = 1 , • [ ˙ G n,l ] ij ∈ [ − χ, χ ] and [ ˙ G n,l ] ii = χ .
Furthermore, the NTK is a dot product kernel, meaning Θ( x i , x j ) can be written as a function of the inner product between the two inputs, Θ( x Ti x j ) .
Proof.
Recall from Lemma A.1 and its proof that for any l ∈ [ L + 1] , i, j ∈ [ n ] [ G n,l ] ij = Σ ( l ) ( x i , x j ) and [ ˙ G n,l ] ij = ˙Σ ( l ) ( x i , x j ) .
We ﬁrst prove by induction Σ ( l ) ( x i , x i )
=
for all l ∈
[ L + 1] .
The base case l = 1 follows as
[Equation]
Assume the induction hypothesis is true for layer l − 1 .
With Z ∼ N (0 , 1) , then from (16) and (17)
[Equation]
thus the inductive step is complete.
As an immediate consequence it follows that [ G l ] ii = 1 .
Also, for any i, j ∈ [ n ] and l ∈ [ L + 1] ,
[Equation]
Thus we can consider Σ ( l ) as a univariate function of the input correlation Σ : [ − 1 , 1] →
[ − 1 , 1] and also conclude that [ G l ] ij ∈ [ − 1 , 1] .
Furthermore,
[Equation]
which likewise implies ˙Σ is a dot product kernel.
Recall now the random variables introduced to deﬁne R : Z 1 , Z 2 ∼ N (0 , 1) are independent and U
= Z 1 , U
= ( ρZ 1 + (cid:112) 1 − ρ 2 Z 2 ) .
Observe
U 1 , U 2 are dependent but identically distributed as U 1 , U 2 ∼ N (0 , 1) .
For any ρ ∈ [ − 1 , 1] then applying the Cauchy-Schwarz inequality gives
[Equation]
As a result, under the assumptions of the lemma ˙Σ ( l ) : [ − 1 , 1] → [ − χ, χ ] and ˙Σ ( l ) ( x i , x i ) = χ .
From this it immediately follows that [ ˙ G l ] ij ∈ [ − χ, χ ] and [ ˙ G l ] ii
=
χ as claimed.
Finally, as Σ : [ − 1 , 1] → [ − 1 , 1] and ˙Σ : [ − 1 , 1] → [ − χ, χ ] are dot product kernels, then from (12) the NTK must also be a dot product kernel and furthermore a univariate function of the pairwise correlation of its input arguments.
The following corollary, which follows immediately from Lemma A.2 and (14), characterizes the trace of the NTK matrix in terms of the trace of the input gram.
Corollary A.3.
Under the same conditions as Lemma A.2, suppose φ and σ 2 w are chosen such that χ = 1 .
Then
[Equation]
A. H ERMITE E XPANSIONS
We say that a function f : R → R is square integrable w.r.t.
the standard Gaussian measure γ =
e − x 2 / 2 / √ 2 π if E x ∼N (0 , 1) [ f ( x ) 2 ] < ∞ .
We denote by L 2 ( R , γ ) the space of all such functions.
The probabilist’s Hermite polynomials are given by
[Equation]
The ﬁrst three Hermite polynomials are H 0 ( x )
= 1 , H 1 ( x )
= x , H 2 ( x ) =
( x 2 − 1) .
Let h k ( x )
=
H k ( x ) √ k !
denote the normalized probabilist’s Hermite polynomials.
The normalized Hermite polynomials form a complete orthonormal basis in L 2 ( R , γ ) (O’Donnell, 2014, §11): in all that follows, whenever we reference the Hermite polynomials, we will be referring to the normalized
Hermite polynomials.
The Hermite expansion of a function φ ∈ L 2 ( R , γ ) is given by
[Equation]
22) where
[Equation]
) is the k th normalized probabilist’s Hermite coefﬁcient of φ .
In what follows we shall make use of the following identities.
[Equation]
if k is even , where k !!
=
[Equation]
We also remark that the more commonly encountered physicist’s Hermite polynomials, which we denote ˜ H k , are related to the normalized probablist’s polynomials as follows,
[Equation]
The Hermite expansion of the activation function deployed will play a key role in determining the coefﬁcients of the NTK power series.
In particular, the Hermite coefﬁcients of ReLU are as follows.
Lemma A.4.
Daniely et al.
For φ ( z ) = max { 0 , z } the Hermite coefﬁcients are given by
[Equation]
B E XPRESSING THE NTK AS A POWER SERIES B. D ERIVING A POWER SERIES FOR THE NTK
We will require the following minor adaptation of Nguyen & Mondelli (2020, Lemma D.2).
We remark this result was ﬁrst stated for ReLU and Softplus activations in the work of Oymak &
Soltanolkotabi (2020, Lemma H.2).
Lemma B.1.
For arbitrary n, d ∈ N , let A ∈ R n × d .
For i ∈ [ n ] , we denote the i th row of A as a i , and further assume that (cid:107) a i (cid:107) = 1 .
Let φ : R → R satisfy φ ∈ L 2 ( R , γ ) and deﬁne
[Equation]
Then the matrix series
[Equation]
converges uniformly to M as K → ∞ .
The proof of Lemma B.1 follows exactly as in (Nguyen & Mondelli, 2020, Lemma D.2), and is in fact slightly simpler due to the fact we assume the rows of A are unit length and w ∼ N (0 , I d ) instead of √ d and w ∼ N (0 , 1 d I d ) respectively.
For the ease of the reader, we now recall the following deﬁnitions, which are also stated in Section 3.
Letting ¯ α l := ( α p,l ) ∞ p =0 denote a sequence of real coefﬁcients, then [Equation] p
[Equation]
where
J
[Equation]
p [Equation] for all p ∈ Z ≥ 0 , k ∈ Z ≥ 1 .
We are now ready to derive power series for elements of ( G l ))
L +1 l =1 and ( ˙ G l ))
L +1 l =2 .
Lemma B.2.
Under Assumptions 1 and 2, for all l ∈
[Equation]
where the series for each element [ G l ] ij converges absolutely and the coefﬁcients α p,l are nonnegative.
The coefﬁcients of the series (29) for all p ∈ Z ≥ 0 can be expressed via the following recurrence relationship,
[Equation]
Furthermore,
[Equation]
where likewise the series for each entry [ ˙ G l ] ij converges absolutely and the coefﬁcients υ p,l for all p ∈ Z ≥ 0 are nonnegative and can be expressed via the following recurrence relationship,
[Equation]
Proof.
We start by proving (29) and (30).
Proceeding by induction, consider the base case l = 2 .
From Lemma A.
[Equation]
By the assumptions of the lemma, the conditions of Lemma B.1 are satisﬁed and therefore
[Equation]
[Table]
Observe the coefﬁcients ( α k, 2 )
k ∈
Z ≥ 0 are nonnegative.
Therefore, for any i, j ∈ [ n ] using
Lemma A.2 the series for [ G l ] ij satisﬁes
[Equation]
and so must be absolutely convergent.
With the base case proved we proceed to assume the inductive hypothesis holds for arbitrary G l with l ∈ [2 , L ] .
Observe
[Equation]
where A is a matrix square root of G l , meaning G l = AA .
Recall from Lemma A.1 that G l is also symmetric and positive semi-deﬁnite, therefore we may additionally assume, without loss of generality, that A ∈ R n × n is symmetric, which conveniently implies G n,l = AA T .
Under the assumptions of the lemma the conditions for Lemma A.2 are satisﬁed and as a result [ G n,l ] ii = (cid:107) a i (cid:107) =
1 for all i ∈ [ n ] , where we recall a i denotes the i th row of A .
Therefore we may again apply
Lemma A.1,
[Equation]
where the ﬁnal equality follows from the inductive hypothesis.
For any pair of indices i, j ∈ [ n ]
[Equation]
By the induction hypothesis, for any i, j ∈ [ n ] the series (cid:80) ∞ m [-=
-][Equation]
α m,l (cid:104) x i , x j (cid:105) m is absolutely convergent.
Therefore, from the Cauchy product of power series and for any k ∈ Z ≥
we have
[Equation]
where F ( p, k, ¯ α l ) is deﬁned in (4).
By deﬁnition, F ( p, k, ¯ α l ) is a sum of products of positive coefﬁcients, and therefore | F ( p, k, ¯ α l ) | =
F ( p, k, ¯ α l ) .
In addition, recall again by Assumptionand Lemma A.2 that [ G l ] ii = 1 .
As a result, for any k ∈ Z ≥ 0 , as |(cid:104) x i , x j (cid:105)| ≤
[Equation]
and therefore the series (cid:80) ∞
p =
F ( p, k, ¯ α l ) (cid:104) x i , x j (cid:105) p converges absolutely.
Recalling from the proof of the base case that the series (cid:80) ∞ p =1 α p, 2 is absolutely convergent and has only nonnegative elements, we may therefore interchange the order of summation in the following,
[Equation]
Recalling the deﬁnition of F ( p, k, l ) in (4), in particular F (0 , 0 , ¯ α l ) = 1 and F ( p, 0 , ¯ α l ) = 0 for p ∈ Z ≥ 1 , then
[Equation]
As the indices i, j ∈ [ n ] were arbitrary we conclude that
[Equation]
as claimed.
In addition, by inspection and using the induction hypothesis it is clear that the coefﬁcients ( α p,l +1 ) ∞
p =0 are nonnegative.
Therefore, by an argument identical to (33), the series for each entry of [ G l +1 ] ij is absolutely convergent.
This concludes the proof of (29) and (30).
We now turn our attention to proving the (31) and (32).
Under the assumptions of the lemma the conditions for Lemmas A.1 and B.1 are satisﬁed and therefore for the base case l =
[Equation]
By inspection the coefﬁcients ( υ p, 2 ) ∞
p =0 are nonnegative and as a result by an argument again identical to (33) the series for each entry of [ ˙ G 2 ] ij is absolutely convergent.
For l ∈ [2 , L ] , from (29) and its proof there is a matrix A ∈ R n × n such that G l = AA T .
Again applying Lemma B.
[Equation]
(cid:12) p (cid:33) (cid:12) k
Analyzing now an arbitrary entry [ ˙ G l +1 ] ij , by substituting in the power series expression for G l from (29) and using (34) we have
[Equation]
Note that exchanging the order of summation in the third equality above is justiﬁed as for any k ∈
Z ≥ 0 by (35) we have
[Equation]
≤ 1 and therefore (cid:80) ∞
k =0 (cid:80) ∞
p =
υ k, 2 F ( p, k, ¯ α l ) (cid:104) x i , x j (cid:105) p converges absolutely.
As the indices i, j ∈ [ n ] were arbitrary we conclude that
[Equation]
as claimed.
Finally, by inspection the coefﬁcients ( υ p,l +1 ) ∞
p =0 are nonnegative, therefore, and again by an argument identical to (33), the series for each entry of [ ˙ G n,l +1 ] ij is absolutely convergent.
This concludes the proof.
We are now prove the key result of Section 3.
Theorem 3.1.
Under Assumptions 1 and 2, for all l ∈ [ L + 1]
[Equation]
The series for each entry n [ K l ] ij converges absolutely and the coefﬁcients κ p,l are nonnegative and can be evaluated using the recurrence relationships
[Equation]
where α p,l = (cid:26) σ
[Equation]
and
[Equation]
are likewise nonnegative for all p ∈ Z ≥ 0 and l ∈ [2 , L + 1] .
Proof.
We proceed by induction.
The base case l = 1 follows trivially from Lemma A.1.
We therefore assume the induction hypothesis holds for an arbitrary l − 1 ∈ [1 , L ] .
From (14) and
Lemma B.
[Equation]
T (cid:1) (cid:12) p (cid:33) + (cid:32) n ∞ (cid:88) q = κ q,l
[Equation]
(cid:0) XX T (cid:1) (cid:12) q (cid:33) (cid:12) (cid:32) ∞ (cid:88) w = υ w,l (cid:0) XX T (cid:1) (cid:12) w (cid:33) .
Therefore, for arbitrary i, j ∈ [ n ]
[Equation]
Observe n (cid:80)
[Equation]
[Equation]
j
[Equation]
q
[Equation]
Θ
[Equation]
1) ( x i , x j ) and therefore the series must converge due to the convergence of the NTK.
Furthermore, (cid:80) ∞ [Equation] = [Equation]
w
[Equation] l [Equation] x i , x j (cid:105) w
=
[ ˙ G n,l ] ij and therefore is absolutely convergent by Lemma B.2.
As a result, by Merten’s Theorem the product of these two series is equal to their Cauchy product.
Therefore
[Equation]
p , from which the (5) immediately follows.
B. A NALYZING THE COEFFICIENTS OF THE NTK POWER SERIES
In this section we study the coefﬁcients of the NTK power series stated in Theorem 3.1.
Our ﬁrst observation is that, under additional assumptions on the activation function φ , the recurrence relationship (6) can be simpliﬁed in order to depend only on the Hermite expansion of φ .
Lemma B.3.
Under Assumption 3 the Hermite coefﬁcients of φ (cid:48) satisfy
[Equation]
for all k ∈ Z ≥ 0 .
Proof.
Note for each n ∈ N as φ is absolutely continuous on [ − n, n ] it is differentiable a.e. on [ − n, n ] .
It follows by the countable additivity of the Lebesgue measure that φ is differentiable a.e. on R .
Furthermore, as φ is polynomially bounded we have φ ∈ L 2 ( R , e − x 2 / 2 / √ 2 π ) .
Fix a > 0 .
Since φ is absolutely continuous on [ − a, a ] it is of bounded variation on [ − a, a ] .
Also note that h k ( x ) e − x 2 / 2 is of bounded variation on [ − a, a ] due to having a bounded derivative.
Thus we have by Lebesgue-Stieltjes integration-by-parts (see e.g. Folland 1999, Chapter 3)
[Equation]
where in the last line above we have used the fact that (24) and (25) imply that xh k ( x ) − h (cid:48)
k
( x ) = √ k
+ 1 h k +1 ( x ) .
Thus we have shown
[Equation]
We note that since | φ ( x ) h k ( x ) | =
O ( | x | β + k ) we have that as a → ∞ the ﬁrst two terms above vanish.
Thus by sending a → ∞ we have
[Equation]
After dividing by √ 2 π we get the desired result.
In particular, under Assumption 3, and as highlighted by Corollary B.4, which follows directly from
Lemmas B.2 and B.3, the NTK coefﬁcients can be computed only using the Hermite coefﬁcients of φ .
Corollary B.4.
Under Assumptions 1, 2 and 3, for all p ∈ Z ≥
[Equation]
With these results in place we proceed to analyze the decay of the coefﬁcients of the NTK for depth two networks.
As stated in the main text, the decay of the NTK coefﬁcients depends on the decay of the Hermite coefﬁcients of the activation function deployed.
This in turn is strongly inﬂuenced by the behavior of the tails of the activation function.
To this end we roughly group activation functions into three categories: growing tails, ﬂat or constant tails and ﬁnally decaying tails.
Analyzing each of these groups in full generality is beyond the scope of this paper, we therefore instead study the behavior of ReLU, Tanh and Gaussian activation functions, being prototypical and practically used examples of each of these three groups respectively.
We remark that these three activation functions satisfy Assumption 3.
For typographical ease we let ω σ ( z ) :
= (1 / √ 2 πσ 2 ) exp (cid:0) − z 2 / (2 σ 2 ) (cid:1) denote the Gaussian activation function with variance σ 2 .
Lemma B.5.
Under Assumptions 1 and 2,
if φ ( z ) =
ReLU ( z ) , then κ p, 2 = δ ( γ b > 0) ∪ ( p even )
Θ( p − 3 / 2 ) , 2.
if φ ( z ) =
Tanh ( z ) , then κ p, 2 = O (cid:16) exp (cid:16) − π √ p − 1(cid:17)(cid:17) , 3.
if φ
( z ) =
ω σ ( z ) , then κ p, 2 = δ ( γ b > 0) ∪ ( p even )
Θ( p 1 / 2 ( σ 2 + 1) − p ) .
Proof.
Recall (9),
[Equation]
In order to bound κ p, 2 we proceed by using Lemma A.4 to bound the square of the Hermite coefﬁcients.
We start with ReLU.
Note Lemma
A.4 actually provides precise expressions for the
Hermite coefﬁcients of ReLU, however, these are not immediately easy to interpret.
Observe from
Lemma A.4 that above index p = 2 all odd indexed Hermite coefﬁcients are 0 .
It therefore sufﬁces to bound the even indexed terms, given by
[Equation]
Observe from (26) that for p even
[Equation]
therefore
[Equation]
Analyzing now | h p (0) | ,
[Equation]
Here, the expression inside the square root is referred to in the literature as the Wallis ratio, for which the following lower and upper bounds are available Kazarinoff (1956), (cid:115) [Equation]
( p [Equation]
[Equation]
p !!
[Equation]
As a result |
[Equation]
and therefore
[Equation]
As ( p + 1) − 3 / 2 =
Θ( p − 3 / 2 ) , then from (9)
[Equation]
as claimed in item 1.
We now proceed to analyze φ ( z ) =
Tanh ( z ) .
From Panigrahi et al.
(2020, Corollary F.7.1)
[Equation]
As Tanh satisﬁes the conditions of Lemma B
[Equation] [Equation]
Therefore the result claimed in item 2. follows as
[Equation]
Finally, we now consider φ ( z ) =
ω σ ( z ) where ω σ ( z ) is the density function of N (0 , σ 2 ) .
Similar to
ReLU, analytic expressions for the Hermite coefﬁcients of ω σ ( z ) are known (see e.g., Davis, 2021,
Theorem 2.9),
[Equation]
[Equation]
For p even (
[Equation]
Therefore p !
[Equation]
As a result, for p even and using (37), it follows that
[Equation]
σ
[Equation]
+ 1) [Equation] p ) . Finally, since ( p + 1) 1 / 2 ( σ 2 + 1) − p − 1 = Θ( p 1 / 2 ( σ 2 + 1) − p ) , then from (9)
[Equation]
as claimed in item 3.
B. N UMERICAL APPROXIMATION VIA A TRUNCATED NTK POWER SERIES AND INTERPRETATION OF F
Currently, computing the inﬁnite width NTK requires either a) explicit evaluation of the Gaussian integrals highlighted in (13), b) numerical approximation of these same integrals such as in Lee et al.
(2018), or c) approximation via a sufﬁciently wide yet still ﬁnite width network, see for instance Engel et al.
; Novak et al.
These Gaussian integrals (13) can be solved solved analytically only for a minority of activation functions, notably ReLU as discussed for example by
Arora et al. (2019b), while the numerical integration and ﬁnite width approximation approaches are relatively computationally expensive.
The truncated NTK power series we deﬁne as analogous to (5) but with the series involved being computed only up to the T th element.
Once the top T coefﬁcients are computed, then for any input correlation the NTK can be approximated by evaluating the corresponding ﬁnite degree T polynomial.
Deﬁnition B.6.
For an arbitrary pair x , y ∈ S d − 1 let ρ = x
T y denote their linear correlation.
Under Assumptions 1, 2 and 3, for all l ∈ [2 , L + 1] the T -truncated NTK power series ˆΘ ( l ) T : [ − 1 , 1] → R is deﬁned as
[Equation]
and whose coefﬁcients are deﬁned via the following recurrence relation,
[Equation]
Here, with ¯ˆ α l − 1 = (ˆ α p,l − 1 ) Tp =0 , ˆ
[Equation]
and ˆ
[Equation]
[Equation] In order to analyze the performance and potential of the truncated NTK for numerical approximation, we compute it for ReLU and compare it with its analytical expression Arora et al. (2019b).
To recall this result, let
[Equation]
Under Assumptions 1 and 2, with φ ( z ) =
ReLU ( z ) , γ 2 w = 1 , σ 2 w = 2 , σ 2 b = γ 2 b = 0 , x , y ∈ S d and ρ 1 := x T y , then Θ 1 ( x , y )
=
ρ and for all l ∈ [2 , L + 1]
[Equation]
[Equation]
Turning our attention to Figure 2, we observe particularly for input correlations | ρ | ≈ 0 .
5 and below then the truncated ReLU
NTK power series achieves machine level precision.
For | ρ | ≈higher order coefﬁcients play a more signiﬁcant role.
As the truncated ReLU NTK power series approximates these coefﬁcients less well the overall approximation of the ReLU NTK is worse.
We remark also that negative correlations have a smaller absolute error as odd indexed terms cancel with even index terms: we emphasize again that in Figure 2 we plot the absolute not relative error.
In addition, for L = 1 there is symmetry in the absolute error for positive and negative correlations as α p, 2 = 0 for all odd p .
One also observes that approximation accuracy goes down with depth, which is due to the error in the coefﬁcients at the previous layer contributing to the error in the coefﬁcients at the next, thereby resulting in an accumulation of error with depth.
Also, and certainly as one might expect, a larger truncation point T results in overall better approximation.
Finally, as the decay in the Hermite coefﬁcients for ReLU is relatively slow, see e.g., Table 1 and Lemma 3.2, we expect the truncated ReLU NTK power series to perform worse relative to the truncated NTK’s for other activation functions.
B. T HE ZEROTH COEFFICIENT OF THE NTK STRICTLY INCREASES WITH DEPTH
The following lemmas show, somewhat unsurprisingly, that the zeroth coefﬁcient of the NTK strictly grows with depth. Lemma B.7. Let the data, hyperparameters and activation function φ be such that Assumptions 1, 2 and 3 are satisﬁed.
Then for all l ≥ 3 it holds that α 0 ,l > α 0 ,l − 1 and υ 0 ,l > υ 0 ,l − 1 .
Proof.
We ﬁrst consider the zeroth coefﬁcients of the Gaussian Process kernel at each layer.
From
Theorem 3.1 for l ≥ 3 recall that
[Equation]
We proceed by induction.
For l = 3 we have
[Equation]
The base case α 0 , 3 > α 0 , 2 is established by dividing both sides by α 0 , 2 and recalling, again from
Theorem 3.1, that the coefﬁcients are nonnegative,
[Equation]
Assume now that the induction hypothesis holds at some layer l ≥ 3 , i.e., α 0 ,l > α 0 ,l − 1 .
Then
[Equation]
and thus the inductive step is complete.
We now turn our attention to analyzing the zeroth coefﬁcient of the derivative of the Gaussian Process kernel.
From Theorem 3.1 and for l ≥
[Equation]
The fact that υ 0 , 3 > υ 0 , 2 follows as
[Equation]
For l ≥ 3 , as α 0 ,l > α 0 ,l − 1 , then the inequality υ 0 ,l +1 > υ 0 ,l can be derived by observing
[Equation]
Lemma B.8.
Let the data, hyperparameters and activation function φ be such that Assumptions 1, 2 and 3 are satisﬁed.
Furthermore, assume the choice of hyperparameters and activation function are such that they satisfy the inequality σ 2 w µ 20 ( φ ) + σ 2 b + γ 2 b σ 2 w µ 21 ( φ ) > γ 2 b .
Then for all l ≥ 2 it follows that κ l > κ l − 1 .
Proof.
Observe, under the assumptions of the lemma, from Theorem 3.
[Equation]
We proceed by induction.
For the base case l = 2 observe
[Equation]
For the inductive step, assume that κ 0 ,l > κ 0 ,l − 1 for some l ≥ 2 .
Then using Lemma B.
[Equation]
This concludes the proof.
Recalling the upper bound in Theorem 4.1 and Corollary A.3, then, under additional assumptions on the activation function and by comparing the size of κ 0 ,l +1 relative to κ 0 ,l (1 + 1 l ) , one could potentially characterize the impact of depth on the upper bound of the effective rank.
We leave such analyses to future work.
B. C HARACTERIZING NTK POWER SERIES COEFFICIENT DECAY RATES FOR DEEP NETWORKS
In general, Theorem 3.1 does not provide a straightforward path to analyzing the decay of the NTK power series coefﬁcients for depths greater than two.
This is due to the difﬁculty in analyzing
F ( p, k, ¯ α l − 1 ) , which we recall is the sum of all ordered products of k elements of ¯ α l − 1 whose indices sum to p , deﬁned in (4).
However, in the setting where the squares of the Hermite coefﬁcients, and therefore the series ( α p, 2 ) ∞
p =0 , decay at an exponential rate, this quantity can be characterized and therefore an analysis of the impact of depth conducted.
Although admittedly limited in scope, we highlight that this setting is relevant for the study of Gaussian activation functions and radial basis function (RBF) networks, which to the best of our knowledge have not been analyzed in prior works.
Under an additional assumption that the activation function has zero bias, which helps simplify the analysis, the following lemma precisely describes the evolution of the coefﬁcients of the related Gaussian Process kernel.
We leave relaxing this zero bias assumption, which we expect to not be materially different from the non-zero bias setting, as well as only enforcing exponential decay asymptotically and exploring other decay patterns, to future work.
Lemma B.9.
Let α 0 , 2 = 0 and α p, 2 = C 2 η − pfor p ∈ Z ≥ 1 , where C 2 and η 2 are constants such that (cid:80) ∞
p =1 α p, 2 = 1 .
Then for all l ≥ 2 and p ∈
Z ≥
[Equation]
p
[Equation]
(43) where the constants η l +1 and C l +1 are deﬁned as
[Equation]
Proof.
We proceed by induction to show that if (43) and (44) are true at layer l then (43) and (44) are true at layer l + 1 .
Observe for l = 2 that (43) and (44) hold by the assumptions of the lemma.
We therefore deploy l = 2 as our base case and proceed by assuming (43) and (44) are true at some layer l ∈ [2 , L ] .
Recall the deﬁnition of F from (4): as α 0 ,l = 0 then with p ≥ 1 and 1 ≤ k ≤ p
[Equation]
here
[Equation]
which is the set of all k -tuples of positive (instead of non-negative) integers which sum to p .
Substituting α p,l = C l η − p l then
[Equation]
where the ﬁnal equality follows from a stars and bars argument.
Now observe for k > p that at least one of the indices in ( j i ) ki
=1 must be 0 and therefore (cid:81) ki
=1 α j i , 2 = 0 .
As a result under the assumptions of the lemma
[Equation]
otherwise
[Equation]
Substituting (45) into (7) it follows that
[Equation]
and for p ≥
[Equation]
as claimed.
We now analyze the coefﬁcients of the derivative of the Gaussian Process kernel.
Lemma B.10.
In addition to the assumptions of Lemma B.9, assume also that φ satisﬁes Assumption 3.
Then υ p, 2 = C 2 η 2 (1 + p ) η − p 2 .
Furthermore, for all l ≥ 2 and p ∈
Z ≥
[Equation]
[Equation]
where the constants V (cid:48)
l +1 and V l +1 are deﬁned as
[Equation]
and C l and η l are deﬁned in (44) .
Proof.
{+Under Assumption 3 then for all p ∈ Z ≥ 
+}
we have
[Equation]
For l ≥ 2 and p = 0 it therefore follows that
[Equation]
For l ≥ 2 and p ≥ 1 then υ p,l +1 =
[Equation]
(cid:88)
[Equation]
as claimed.
With the coefﬁcients of both the Gaussian Process kernel and its derivative characterized, we proceed to analyze the the asymptotics of the coefﬁcients of the NTK.
Lemma B.11.
Let the data, hyperparameters and activation function φ be such that Assumptions 1, 2 and 3 are satisﬁed along with the conditions of of Lemma B.9.
Then for any l ≥ 2 there exist positive constants K l and K (cid:48)
l such that
[Equation]
where η l and C l are deﬁned in Lemma B.9.
Proof.
We proceed by induction starting with the base case l = 2 .
Applying the results of Lemmas
B.9 and B.10 to (6) then
[Equation]
where K 0 , 2 = γ 2 b C 2 η − 1 2 , K 1 , 2 := C 2 ( η − 1 2 γ 2 b + γ 2 w ) and K 2 , 2 = C 2 (1 + η − 1 2 γ 2 b ) are positive constants.
{+Therefore, letting K (cid:48) l = K 2 , 2 then for all p ∈ Z ≥ 
+}
it is clear that
[Equation]
and thus the inequality in (48) is trivially true for l = 2 .
With the base case established we proceed to assume the induction hypothesis holds at some layer l ≥ 2 .
From (6)
[Equation]
Observe from (46) that for any layer l ≥ 2 and p ∈
{+Z ≥ 
+}
there are positive constants A, B such that
[Equation]
As we do not care about the speciﬁc values of the constants we will use A and B to denote generic constants involved with these coefﬁcients.
To derive the upper bound,
[Equation]
p and thus with K (cid:48)
l +1 =
C l A + C l B + AK (cid:48)
l + BK (cid:48)
l we have proven the upper bound.
To derive the lower bound,
[Equation]
Therefore with K l +1 =
AC l + AK l we have also proven the lower bound and as a result the proof by induction is complete.
In the setting studied in B.11 Lemma B.9 implies η l +1 < η l , therefore Lemma B.11 implies that the coefﬁcients of the NTK corresponding to deeper networks decay slower than the coefﬁcients of shallower networks.
C A NALYZING THE SPECTRUM OF THE NTK VIA ITS POWER SERIES C. E FFECTIVE RANK OF POWER SERIES KERNELS
Recall that for a positive semideﬁnite matrix A we deﬁne the effective rank Huang et al.
via the following ratio eﬀ( A ) :
[Equation]
We consider a kernel Gram matrix K ∈ R n × n that has the following power series representation in terms of an input gram matrix XX T
[Equation]
Whenever c 0 (cid:54) =
the effective rank of K is O (1) , as displayed in the following theorem.
Theorem 4.1.
Assume that we have a kernel Gram matrix K of the form n K = (cid:80) ∞
p =
c p ( XX T ) (cid:12) p where c 0 (cid:54) = 0 .
Furthermore, assume the input data x i are normalized so that (cid:107) x i (cid:107) =
1 for all i ∈ [ n ] .
Then
[Equation]
Proof.
By linearity of trace we have that
[Equation]
where we have used the fact
that Tr (( XX T ) (cid:12) i ) =
n for all
i ∈ N .
On the other hand
[Equation]
Thus we have that
[Equation]
The above theorem demonstrates that the constant term c 0 1 n × n in the kernel leads to a signiﬁcant outlier in the spectrum of K .
However this fails to capture how the structure of the input data
X manifests in the spectrum of K .
For this we will examine the centered kernel matrix (cid:101) K :=
K − c 0 n 11 T .
Using a very similar argument as before we can demonstrate that the effective rank of (cid:101) K is controlled by the effective rank of the input data gram XX T .
This is formalized in the following theorem.
Theorem 4.3.
Assume that we have a kernel Gram matrix K of the form n K = (cid:80) ∞
p =
c p ( XX T ) (cid:12) p where c 1 (cid:54) = 0 .
Furthermore, assume the input data x i are normalized so that (cid:107) x i (cid:107) =
1 for all i ∈ [ n ] .
Then the centered kernel (cid:101) K :=
K − c 0 n [Equation] n × n satisﬁes
[Equation]
Proof.
By the linearity of the trace we have that
[Equation]
where we have used the fact that Tr (( XX T ) (cid:12) i ) =
Tr ( XX T ) = n for all i ∈ [ n ] .
On the other hand we have that
[Equation]
Thus we conclude
[Equation]
C. E FFECTIVE RANK OF THE NTK FOR FINITE WIDTH NETWORKS C.2. N OTATION AND DEFINITIONS
We will let [ k ] := { 1 , 2 , . . .
, k } .
We consider a neural network m (cid:88) (cid:96)
[Equation]
a (cid:96)
[Equation]
( (cid:104)
[Equation]
(cid:96)
[Equation]
(cid:105) ) where x ∈ R d and w (cid:96) ∈ R d , a (cid:96) ∈
R for all (cid:96) ∈ [ m ] and φ is a scalar valued activation function.
The network we present here does not have any bias values in the inner-layer, however the results we will prove later apply to the nonzero bias case by replacing x with [ x T , 1] T .
We let W ∈ R m ×
d be the matrix whose (cid:96) -th row is equal to w (cid:96) and a ∈ R m be the vector whose (cid:96) -th entry is equal to a (cid:96) .
We can then write the neural network in vector form
[Equation]
where φ is understood to be applied entry-wise.
Suppose we have n training data inputs x 1 , . . .
, x n ∈ R d .
We will let X ∈ R n × d be the matrix whose i -th row is equal to x i .
Let θ inner = vec ( W ) denote the row-wise vectorization of the innerlayer weights.
We consider the Jacobian of the neural networks predictions on the training data with respect to the inner layer weights:
[Equation]
Similarly we can look at the analagous quantity for the outer layer weights
[Equation]
Our ﬁrst observation is that the per-example gradients for the inner layer weights have a nice Kronecker product representation
[Equation]
For convenience we will let
[Equation]
where the dependence of Y i on the parameters W and a is suppressed (formally Y i
=
Y i ( W , a ) ).
This way we may write
[Equation]
We will study the NTK with respect to the inner-layer weights
[Equation]
and the same quantity for the outer-layer weights
[Equation]
For a hermitian matrix A we will let λ i ( A ) denote the i th largest eigenvalue of A so that λ 1 ( A ) ≥ λ 2 ( A ) ≥ · · · ≥ λ n ( A ) .
Similarly for an arbitrary matrix A we will let σ i ( A ) to the i th largest singular value of A .
For a matrix A ∈ R r × k we will let σ min ( A ) =
σ min( r,k ) .
C.2. E FFECTIVE RANK
For a positive semideﬁnite matrix A we deﬁne the effective rank (Huang et al., 2022) of A to be the quantity
[Equation]
The effective rank quantiﬁes how many eigenvalues are on the order of the largest eigenvalue.
We have the Markov-like inequality
[Equation]
and the eigenvalue bound
[Equation]
Let A and B be positive semideﬁnite matrices.
Then we have
[Equation]
Thus the effective rank is subadditive for positive semideﬁnite matrices.
We will be interested in bounding the effective rank of the NTK.
Let K
= JJ T =
J outer J Touter +
J inner J Tinner =
K outer + K inner be the NTK matrix with respect to all the network parameters.
Note that by subadditivity
[Equation]
In this vein we will control the effective rank of K inner and K outer separately.
C.2. E FFECTIVE RANK OF INNER - LAYER NTK
We will show that the effective rank of inner-layer NTK is bounded by a multiple of the effective rank of the data input gram XX T .
We introduce the following meta-theorem that we will use to prove various corollaries later
Theorem C.1.
Set α := sup (cid:107) b (cid:107) =1 (cid:2)
min j ∈ [ n ] |(cid:104) Y j , b (cid:105)| (cid:3) .
Assume α > 0 .
Then
[Equation]
[Equation]
Proof.
We will ﬁrst prove the upper bound.
We ﬁrst observe that
[Equation]
Recall that {+λ 
+}
( K inner ) =
λ 1 (cid:0) J inner J Tinner (cid:1) =
λ 1 (cid:0) J Tinner J inner (cid:1) .
Well
J Tinner J inner = n (cid:88) i = ∂f ( x i ) ∂θ inner ∂f ( x i ) ∂θ inner T = n (cid:88) i =
[ Y i ⊗ x i ] [ Y i ⊗ x i ] T = n (cid:88) i = (cid:2) Y i Y Ti (cid:3) ⊗ (cid:2)
x
i x Ti (cid:3)
Well then we may use the fact that
[Equation]
Let b 1 ∈ R m and b 2 ∈ R d be vectors that we will optimize later satisfying (cid:107) b 1 (cid:107) 2 (cid:107) b 2 (cid:107) 2 = 1 .
Then we have that (cid:107) b 1 ⊗ b 2 (cid:107) = 1 and
[Equation]
[Equation]
=
[Equation]
Pick b 2 so that (cid:107) b 2 (cid:107) = 1 and
[Equation]
Thus for this choice of b 2 we have
[Equation]
Now note that α 2 = sup (cid:107) b 1 (cid:107) =1 (cid:2) min j ∈ [ n ] b T 1 Y j Y
Tj b 1 (cid:3) .
Thus by taking the sup over b 1 in our previous bound we have
[Equation]
Thus combined with our previous result we have
[Equation]
We now prove the lower bound.
[Equation]
Let Y ∈ R n × m be the matrix whose i th row is equal to Y i .
Then observe that
[Equation]
where (cid:12) denotes the entry-wise Hadamard product of two matrices.
We now recall that if A and B are two positive semideﬁnite matrices we have (Oymak & Soltanolkotabi, 2020, Lemma 2)
[Equation]
Applying this to K inner we get that
[Equation]
Combining this with our previous result we get
[Equation]
We can immediately get a useful corollary that applies to the ReLU activation function
Corollary C.2.
Set α :=
sup (cid:107) b (cid:107) =1 (cid:2) min j ∈ [ n ] |(cid:104) Y j , b (cid:105)| (cid:3) and γ max := sup x ∈ R | φ (cid:48)
( x ) | .
Assume α > 0 and γ max < ∞ .
Then
[Equation]
Proof.
Note that the hypothesis on | φ (cid:48) | gives (cid:107) Y
i (cid:107) 22 ≤ γ 2 max (cid:107) a (cid:107) 22 for all i ∈ [ n ] .
Moreover by
Cauchy-Schwarz we have that min i ∈ [
n ] (cid:107) Y i (cid:107) 2 ≥ α .
Thus by theorem C.
we get the desired result.
If φ is a leaky ReLU type activation (say like those used in Nguyen & Mondelli (2020))
Theorem
C.1 translates into an even simpler bound
Corollary C.3.
Suppose φ (cid:48)
( x ) ∈ [ γ min , γ max ] for all x ∈ R where γ min > 0 .
Then γ 2 min [Equation] XX T
[Equation]
Proof.
We will lower bound
[Equation]
so that we can apply Corollary C.2.
Set b = a / (cid:107) a (cid:107) 2 .
Then we have that
[Equation]
Thus α ≥ γ min (cid:107) a (cid:107) 2 .
The result then follows from Corollary C.
To control α in Theorem C.1 when φ is the ReLU activation function requires a bit more work.
To this end we introduce the following lemma.
Lemma C.4.
Assume φ ( x ) =
ReLU ( x ) .
Let R min , R max > 0 and deﬁne τ
= { (cid:96) ∈ [ m ] : | a (cid:96) |
∈ [ R min , R max ] } .
Set T = min i ∈
[ n ] (cid:80) (cid:96)
∈
τ I [ (cid:104) x i , w (cid:96) (cid:105) ≥ 0] .
Then
[Equation]
T | τ |
[Equation]
Proof.
Let a τ be the vector such that ( a τ ) (cid:96) =
a (cid:96) I
[ (cid:96) ∈ τ ] .
Then note that
[Equation]
Roughly what Lemma C.4 says is that α is controlled when there is a set of inner-layer neurons that are active for each data point whose outer layer weights are similar in magnitude.
Note that in
Du et al. (2019b), Arora et al. (2019a), Oymak et al.
, Li et al.
(2020), Xie et al.
(2017) and
Oymak & Soltanolkotabi (2020)
the outer layer weights all have ﬁxed constant magnitude.
Thus in that case we can set R min =
R max in Lemma C.4 so that τ = [ m ] .
In this setting we have the following result.
Theorem C.5.
Assume φ ( x ) =
ReLU ( x ) .
Suppose | a (cid:96) | =
R > 0 for all (cid:96) ∈ [ m ] .
Furthermore suppose w 1 , . . .
, w m are independent random vectors such that w (cid:96) / (cid:107) w (cid:96) (cid:107) has the uniform distribution on the sphere for each (cid:96) ∈ [ m ] .
Also assume m ≥ 4log( n/(cid:15) ) δfor some δ, (cid:15) ∈ (0 , 1) .
Then with probability at least 1 − (cid:15) we have that
[Equation]
Proof.
Fix j ∈ [ n ] .
Note by the assumption on the w (cid:96) ’s
we have that I [ (cid:104) w 1 , x j (cid:105) ≥ 0] , . . .
, I [ (cid:104) w m , x j (cid:105) ≥ 0] are i.i.d.
Bernouilli random variables taking the values 0 and 1 with probability 1 / 2 .
Thus by the Chernoff bound for Binomial random variables we have that
P
[Equation]
Thus taking the union bound over every j ∈ [ n ] we get that if m ≥ 4log( n/(cid:15) ) δthen
[Equation]
holds with probability at least 1 − (cid:15) .
Now note that if we set R min =
R max =
R we have that τ
=
[ m ] where τ is deﬁned as it is in Lemma C.4.
In this case by our previous bound we have that
T as deﬁned in Lemma C.4 satisﬁes T ≥ m 2 (1 − δ ) with probability at least 1 − (cid:15) .
In this case the conclusion of Lemma C.4 gives us
[Equation]
Thus by Corollary C.2 and the above bound for α we get the desired result.
We will now use Lemma C.4 to prove a bound in the case of Gaussian initialization.
Lemma C.6.
Assume φ ( x ) =
ReLU ( x ) .
Suppose that a (cid:96) ∼ N (0 , ν 2 ) for each (cid:96) ∈ [ m ] i.i.d.
Furthermore suppose w 1 , . . .
, w m are random vectors independent of each other and a such that w (cid:96) / (cid:107) w (cid:96) (cid:107) has the uniform distribution on the sphere for each (cid:96) ∈ [ m ] .
Set p
= P z ∼ N (0 , 1) ( | z | ∈ [1 / 2 , 1])
≈ 0 .
Assume
[Equation]
for some (cid:15), δ ∈ (0 , 1) .
Then with probability at least (1 − (cid:15) ) 2 we have that
[Equation]
Proof.
Set R min = ν/ 2 and R max
= ν .
Now set
[Equation]
z
[Equation]
N [Equation] [Equation] Now deﬁne τ = { (cid:96) ∈ [ m ] : | a (cid:96) | ∈ [ R min , R max ] } .
We have by the Chernoff bound for binomial random variables
[Equation]
Thus if m ≥ log (cid:0) 1 (cid:15) (cid:1) 2 pδ 2 (a weaker condition than the hypothesis on m ) then we have that | τ | ≥ (1 − δ ) mp with probability at least 1 − (cid:15) .
From now on assume such a τ has been observed and view it as ﬁxed so that the only remaining randomness is over the w (cid:96) ’
s. Now set
T = min i ∈ [ n ] (cid:80) (cid:96) ∈
τ I [ (cid:104) x i , w (cid:96) (cid:105) ≥ 0] .
By the Chernoff bound again we get that for ﬁxed i ∈ [ n ]
P (cid:32)(cid:88)
[Equation]
|
[Equation] | [Equation] Thus by taking the union bound over i ∈ [ n ] we get
[Equation]
Thus if we consider τ as ﬁxed and m ≥ 4log( n/(cid:15) )
δ 2 (1 − δ ) p then with probability at least 1 − (cid:15) over the sampling of the w (cid:96) ’s
we have that
[Equation]
In this case by lemma C.4 we have that
[Equation]
Thus the above holds with probability at least (1 − (cid:15) ) 2 .
This lemma now allows us to bound the effective rank of K inner in the case of Gaussian initialization.
Theorem C.7.
Assume φ ( x ) =
ReLU ( x ) .
Suppose that a (cid:96) ∼ N (0 , ν 2 ) for each (cid:96) ∈ [ m ] i.i.d.
Furthermore suppose w 1 , . . .
, w m are random vectors independent of each other and a such that w (cid:96) / (cid:107) w (cid:96) (cid:107) has the uniform distribution on the sphere for each (cid:96) ∈ [ m ] .
Set p
= P z ∼ N (0 , 1) ( | z | ∈ [1 / 2 , 1])
≈ 0 .
Let (cid:15), δ ∈ (0 , 1) .
Then there exists absolute constants c, K >such that if
[Equation]
then with probability at least 1 − 3 (cid:15) we have that
[Equation]
where
[Equation]
Proof.
By Bernstein’s inequality
[Equation]
where c is an absolute constant.
Set t = max { c − 1 K log(1 /(cid:15) ) , mK } so that the right hand side of the above inequality is bounded by (cid:15) .
Thus by Lemma C.6 and the union bound we can ensure that with probability at least
[Equation]
that (cid:107) a /ν (cid:107) 22 ≤ m + t and the conclusion of Lemma C.6 hold simultaneously.
In that case
[Equation]
Thus by Corollary C.2 we get the desired result.
By ﬁxing δ > 0 in the previous theorem we get the immediate corollary
Corollary C.8.
Assume φ ( x ) =
ReLU ( x ) .
Suppose that a (cid:96) ∼ N (0 , ν 2 ) for each (cid:96) ∈ [ m ] i.i.d.
Furthermore suppose w 1 , . . .
, w m are random vectors independent of each other and a such that w (cid:96) / (cid:107) w (cid:96) (cid:107) has the uniform distribution on the sphere for each (cid:96) ∈ [ m ] .
Then there exists an absolute constant C > 0 such that m = Ω(log( n/(cid:15) )) ensures that with probability at least 1 − (cid:15)
[Equation]
C.2. E FFECTIVE RANK OF OUTER - LAYER NTK
Throughout this section φ ( x ) =
ReLU ( x ) .
Our goal of this section, similar to before, is to bound the effective rank of K outer by the effective rank of the input data gram XX T .
In this section we will use often make use of the basic identities
[Equation]
To begin bounding the effective rank of K outer
[Equation]
we prove the following lemma.
Lemma C.9.
Assume φ ( x ) =
ReLU ( x ) and W is full rank with m ≥ d .
Then
[Equation]
Proof.
First note that
[Equation]
{+W (cid:107) 
 Tr ( XX T ) .+} Pick b ∈ R d such that (cid:107) b (cid:107) 2 = 1 and (cid:107) Xb (cid:107)
= (cid:107) X (cid:107) 2 .
Since W T is full rank we may set u
= ( W T ) † b so that W T u
=
b where (cid:107) u (cid:107) 2 ≤ σ min ( W T ) − 1 where σ min ( W T ) is the smallest nonzero singular value of W T .
Well then
[Equation]
Now using the fact that x = φ ( x ) − φ ( − x ) we have that
[Equation]
Thus combined with our previous results gives
[Equation]
Therefore
[Equation]
which gives us the desired result.
Corollary C.10.
Assume φ ( x ) =
ReLU ( x ) and W is full rank with m ≥ d [Equation]
Then
[Equation]
Proof.
Using the fact that
[Equation]
and lemma C
[Equation]
9 we have that
[Equation]
Note that the right hand side and the denominator of the left hand side do not change when you replace W with − W .
Therefore by using the above bound for both W and − W as the weight matrix separately we can conclude
[Equation]
Corollary C.11.
Assume φ ( x ) =
ReLU ( x ) and m ≥ d .
Suppose W and − W have the same distribution.
Then conditioned on W being full rank we have that with probability at least [Equation]
[Equation]
Proof.
Fix W where W is full rank.
We have by corollary C.10 that either
[Equation]
holds or
[Equation]
(the ﬁrst holds in the case where (cid:13)(cid:13) φ [Equation] WX T [Equation]
2 2 ≥
[Equation]
φ
[Equation] − WX T ) [Equation] 2 2 and the second in the case (cid:13)(cid:13) φ
[Equation] WX T ) [Equation]
[Equation]
φ
[Equation]
WX T ) (cid:13)(cid:13) 2 2 ). Since W and − W have the same distribution, it follows that the ﬁrst inequality must hold at least 1 / 2 of the time. From
[Equation]
we get the desired result.
We now note that when W is rectangular shaped and the entries of W are i.i.d.
Gaussians that W is full rank with high probability and σ min ( W ) − 2 (cid:107) W (cid:107) 22 is well behaved.
We recall the result from
Vershynin (2012)
Theorem C.12.
Let A be a N × n matrix whose entries are independent standard normal random variables.
Then for every t ≥ 0 , with probability at least 1 − 2 exp( − t 2 / 2) one has √
[Equation]
Corollary C.11 gives us a bound that works at least half the time.
However, we would like to derive a bound that holds with high probability.
We will have that when m (cid:38) n we have sufﬁcient concentration of the largest singular value of φ ( WX T ) to prove such a bound.
We recall the result from Vershynin (2012) (Remark 5.40)
Theorem C.13.
Assume that A is an N × n matrix whose rows A i are independent sub-gaussian random vectors in R n with second moment matrix Σ .
Then for every t ≥ 0 , the following inequality holds with probability at least 1 − 2 exp( − ct 2 ) (cid:13)(cid:13)(cid:13)(cid:13)
N A ∗ A − Σ (cid:13)(cid:13)(cid:13)(cid:13)
≤ max( δ, δ 2 ) where δ = C (cid:114) n
N + t √
N where C = C K , c = c K > 0 depend only on K := max i (cid:107)
A i (cid:107) ψ 2 .
We will use theorem C.13 in the following lemma.
Lemma C.14.
Assume φ ( x ) =
ReLU ( x ) .
Let A = φ ( WX T ) and M = max i ∈
[ n ] (cid:107) x i (cid:107) 2 .
Suppose that w 1 , . . .
w m ∼ N (0 , ν 2 I d ) i.i.d.
Set K =
Mν √ n and deﬁne Σ :=
E w ∼ N (0 ,ν 2 I ) [ φ ( Xw ) φ ( w T X T )]
Then for every t ≥
the following inequality holds with probability at least 1 − 2 exp( − c K t 2 ) (cid:13)(cid:13)(cid:13)(cid:13)m A T
[Equation]
− Σ (cid:13)(cid:13)(cid:13)(cid:13) ≤ max( δ, δ 2 ) where δ = C K (cid:114) n m + t √ m, where c K , C K > 0 are absolute constants that depend only on K .
Proof.
We will let A (cid:96) : denote the (cid:96) th row of A (considered as a column vector).
Note that
[Equation]
We immediately get that the rows of A are i.i.d.
We will now bound (cid:107) A (cid:96) : (cid:107) ψ 2 .
Let b ∈ R n such that (cid:107) b (cid:107) 2 = 1 .
Then
[Equation]
where C > 0 is an absolute constant.
Set K :=
Mν √ n .
Well then by theorem C.13 we have the following.
For every t ≥ 0 the following inequality holds with probability at least 1 − 2 exp( − c K t 2 )
[Equation]
≤ max( δ, δ 2 ) where δ = C K (cid:114) n m + t √ m
We are now ready to prove a high probability bound for the effective rank of K outer .
Theorem C.15.
Assume φ ( x ) =
ReLU ( x ) and m ≥ d .
Let M = max i ∈
[ n ] (cid:107) x i (cid:107) 2 .
Suppose that w 1 , . . .
w m ∼ N (0 , ν 2 I d ) i.i.d.
Set K =
Mν √ n
[Equation]
where (cid:15) > 0 is small.
Now assume
[Equation]
and
[Equation]
Then with probability at least 1 − 3 (cid:15)
[Equation]
Proof.
By theorem C.12 with t 1 = (cid:112) 2 log(2 /(cid:15) )
we have that with probability at least 1 − (cid:15) that
[Equation]
The above inequalities and the hypothesis on m imply that W is full rank.
Let A = φ ( WX T ) and ˜ A = φ ( − WX T ) .
Set t 2 = (cid:113) log(2 /(cid:15) )
c K where c K is deﬁned as in theorem
C.14.
Note that A and ˜ A are identical in distribution.
Thus by theorem C.14 and the union bound we get that with probability
at least 1 − 2 (cid:15)
[Equation]
where
[Equation]
By our previous results and the union bound we can ensure with probability at least 1 − 3 (cid:15) that the bounds (50) and (51) all hold simultaneously.
In this case we have
[Equation]
1 m A T A (cid:13)(cid:13)(cid:13)(cid:13) (cid:34) 1 + 2 ρ (cid:13)(cid:13)
1 m A T A (cid:13)(cid:13)(cid:35) ≤ (cid:13)(cid:13)(cid:13)(cid:13) 1 m A T A (cid:13)(cid:13)(cid:13)(cid:13) (cid:20) 1 + 2 ρ λ 1 (Σ) − ρ (cid:21)
Assuming ρ ≤ λ 1 (Σ) / 2 we have by the above bound (
cid:13)(cid:13)(cid:13)(cid:13)m ˜ A T ˜
A (cid:13)(cid:13)(cid:13)(cid:13) ≤ 3 (cid:13)(cid:13)(cid:13)(cid:13) 1 m A T A (cid:13)(cid:13)(cid:13)(cid:13) . Now note that (cid:13)(cid:13) A T A (cid:13)(cid:13) 2 = (cid:13)(cid:13) φ ( WX T ) (cid:13)(cid:13) 2(cid:13)(cid:13)(cid:13) ˜ A T ˜
{+A (cid:13)(cid:13)(cid:13) 
+}
= (cid:13)(cid:13) φ ( − WX T ) (cid:13)(cid:13) 2so that our previous bound implies (cid:13)(cid:13) φ [Equation] [Equation] ≤
[Equation]
(cid:13)(cid:13) φ
[Equation]
WX T ) (cid:13)(cid:13)
2then we have by corollary C.10 that
[Equation]
From the above theorem we get the following corollary.
Corollary C.16.
Assume φ ( x ) =
ReLU ( x ) and n ≥ d . Suppose that w 1 , . . .
w m ∼ N (0 , ν 2 I d ) i.i.d.
Fix (cid:15) > 0 small.
Set M = max i ∈
[ n ] (cid:107) x i (cid:107) 2 .
Then
[Equation]
and
[Equation]
sufﬁces to ensure that with probability at least 1 − (cid:15)
[Equation]
where C > 0 is an absolute constant.
C.2. B OUND FOR THE COMBINED NTK
Based on the results in the previous two sections, we can now bound the effective rank of the combined NTK gram matrix K = K inner + K outer .
Theorem 4.5.
Assume φ ( x ) =
ReLU ( x ) and n ≥ d . Fix (cid:15) > 0 small.
Suppose that w 1 , . . .
, w m ∼
N (0 , ν 21 I d ) i.i.d.
and a 1 , . .
. , a m ∼ N (0 , ν 22 ) .
Set M = max i ∈
[ n ] (cid:107) x
i (cid:107) 2 , and let
[Equation]
Then
[Equation]
sufﬁces to ensure that, with probability at least 1 − (cid:15) over the sampling of the parameter initialization,
[Equation]
where C > 0 is an absolute constant.
Proof.
This follows from the union bound and Corollaries C.8 and C.16.
C.2. M AGNITUDE OF THE SPECTRUM
By our results in sections C.2.3 and C.2.4 we have that m (cid:38) n sufﬁces to ensure that
[Equation]
Well note that
[Equation]
If i (cid:29) d then λ i ( K ) /λ 1 ( K ) is small.
Thus the NTK only has O ( d ) large eigenvalues.
The smallest eigenvalue λ n ( K ) of the NTK has been of interest in proving convergence guarantees (Du et al., 2019a;b; Oymak & Soltanolkotabi, 2020).
By our previous inequality
[Equation]
Thus in the setting where m (cid:38) n (cid:29) d we have that the smallest eigenvalue will be driven to zero relative to the largest eigenvalue.
Alternatively we can view the above inequality as a lower bound on the condition number
[Equation]
We will ﬁrst bound the analytical NTK in the setting when the outer layer weights have ﬁxed constant magnitude.
This is the setting considered by Xie et al.
(2017), Arora et al. (2019a), Du et al.
(2019b),
Oymak et al.
, Li et al.
(2020), and Oymak & Soltanolkotabi (2020).
Theorem C.17.
Let φ ( x ) =
ReLU ( x ) and assume X (cid:54) = 0 .
Let K ∞ inner ∈ R n × n be the analytical
NTK, i.e.
[Equation]
Then
[Equation]
Proof.
We consider the setting where | a (cid:96) | = 1 / √ m for all (cid:96) ∈ [ m ] and w (cid:96) ∼ N (0 , I d ) i.i.d..
As was shown by Jacot et al.
(2018), Du et al.
(2019b) in this setting we have that if we ﬁx the training data X and send m → ∞ we have that
[Equation]
[Table]
in probability.
Therefore by continuity of the effective rank we have that
[Equation]
in probability.
Let η > 0 .
Then there exists an M ∈ N such that m ≥ M implies that
[Equation]
with probability greater than 1 / 2 .
Now ﬁx δ ∈ (0 , 1) .
On the other hand by Theorem C.5 with (cid:15) = 1 / 4 we have that if m ≥ 4 δ 2 log(4 n ) then with probability at least 3 / 4 that
[Equation]
Thus if we set m = max( 4 δ 2 log(4 n ) , M ) we have with probability at least 3 / 4 − 1 / 2 = 1 / 4 that (52) and (53) hold simultaneously.
In this case we have that
[Equation]
Note that the above argument runs through for any η > 0 and δ ∈ (0 , 1) .
Thus we may send η → 0 + and δ → 0 + in the above inequality to get
[Equation]
We thus have the following corollary about the conditioning of the analytical NTK.
Corollary C.18.
Let φ ( x ) =
ReLU ( x ) and assume X (cid:54) = 0 .
Let K ∞ inner ∈ R n × n be the analytical
NTK, i.e.
[Equation]
Then
[Equation]
E XPERIMENTAL VALIDATION OF RESULTS ON THE NTK SPECTRUM
We experimentally test the theory developed in Section 4.1 and its implications by analyzing the spectrum of the NTK for both fully connected neural network architectures (FCNNs), the results of which are displayed in Figure 1, and also convolutional neural network architectures (CNNs), shown in Figure 3.
For the feedforward architectures we consider networks of depth 2 and 5 with the width of all layers being set at 500.
With regard to the activation function we test linear, ReLU and Tanh, and in terms of initialization we use Kaiming uniform (He et al., 2015), which is very common in practice and is the default in PyTorch (Paszke et al., 2019).
For the convolutional architectures we again consider depths 2 and 5, with each layer consisting of 100 channels with the ﬁlter size set to 5x5.
In terms of data, we consider 40x40 patches from both real world images, generated by applying Pytorch’s RandomResizedCrop transform to a random batch of Caltech101 images (Li et al., 2022), as well as synthetic images corresponding to isotropic Gaussian vectors.
The batch sized is ﬁxed at 200 and we plot only the ﬁrst 100 normalized eigenvalues.
Each experiment was repeated 10 times.
Finally, to compute the NTK we use the functorch 4 module in PyTorch using an algorithmic approach inspired by Novak et al.
The results for convolutional neural networks show the same trends as observed in feedforward neural networks, which we discussed in Section 4.1.
In particular, we again observe the dominant outlier eigenvalue, which increases with both depth and the size of the Gaussian mean of the activation.
We also again see that the NTK spectrum inherits its structure from the data, i.e., is skewed for skewed data or relatively ﬂat for isotropic Gaussian data.
Finally, we also see that the spectrum for Tanh is closer to the spectrum for the linear activation when compared with the ReLU spectrum.
In terms of differences between the CNN and FCNN experiments, we observe that the spread of the 95% conﬁdence interval is slightly larger for convolutional nets, implying a slightly larger variance between trials.
We remark that this is likely attributable to the fact that there are only 100 channels in each layer and by increasing this quantity we would expect the variance to reduce.
In summary, despite the fact that our analysis is concerned with FCNNs, it appears that the broad implications and trends also hold for CNNs.
We leave a thorough study of the NTK spectrum for CNNs and other network architectures to future work.
To test our theory in Section 4.2, we numerically plot the spectrum of NTK of two-layer feedforward networks with ReLU, Tanh, and Gaussian activations in Figure 4.
The input data are uniformly drawn from S 2 .
Notice that when d = 2 , k = Θ( (cid:96) 1 / 2 ) .
Then Corollary 4.7 shows that for the
ReLU activation λ (cid:96) =
Θ( (cid:96) − 3 / 2 ) , for the Tanh activation λ (cid:96) =
O (cid:0) (cid:96) − 3 / 4 exp( − π 2 (cid:96) 1 / 4 ) (cid:1) , and for the
Gaussian activation λ (cid:96) =
O ( (cid:96) − 1 / 2 2 − (cid:96) 1 / 2 ) .
These theoretical decay rates for the NTK spectrum are veriﬁed by the experimental results in Figure 4.
C. A NALYSIS OF THE LOWER SPECTRUM : UNIFORM DATA
Theorem 4.6.
[Azevedo & Menegatto (2015)] Suppose that the training data are uniformly sampled from the unit hypersphere S d , d ≥ 2 .
If the dot-product kernel function has the expansion K ( x 1 , x 2 ) = (cid:80) ∞[-p =
-]
{+p =
+} c p (cid:104) x 1 , x 2 (cid:105) p where c p ≥ 0 , then the eigenvalue of every spherical harmonic of frequency k is given by
[Equation]
where Γ is the gamma function.
Proof.
Let θ ( t ) =
(cid:80) ∞
p =
c p t p , then K ( x 1 , x 2 )
= θ ( [Equation] x 1 , x 2 (cid:105) )
According to Funk Hecke theorem (Basri et al., 2019, Section 4.2), we have
[Equation]
where Vol( S d − 1 ) =
2 π d/ 2 Γ( d/ 2) is the volume of the hypersphere S d − 1 , and P k,d ( t ) is the Gegenbauer polynomial, given by [Equation]
[Equation]
and Γ is the gamma function.
From (54) we have
[Equation]
Using integration by parts
[Equation]
we have
[Equation]
where the last line in (56) holds because d k
[Equation]
= 0 when t = 1 or t = − 1 .
When p < k , repeat the above procedure (56) p times, we get
[Equation]
When p ≥ k , repeat the above procedure (56) k times, we get (cid:90)
[Equation]
When p − k is odd, t p − k (1 − t 2 ) k
+( d − 2) / 2 is an odd function, then (cid:90)
[Equation]
When p − k is even,
[Equation]
where B is the beta function.
Plugging (60) , (57) and (59) into (58), we get (cid:90) [Equation] [Equation] p
[Equation]
p
[Equation]
[Equation]
p
[Equation]
k + 1)
[Equation]
p
[Equation]
k +
[Equation]
)Γ( k + d/ 2) Γ( p
[Equation]
k + [Equation] + k + d/ 2) , p − k is even and p ≥ k, 0 , otherwise . [Equation]
Plugging (61) into (55), we get λ k = 2 π d/
[Equation]
Corollary C.19.
Under the same setting as in Theorem 4.6,
if c p =
Θ( p − a ) where a ≥ 1 , then λ k
=
Θ
( k − d − 2 a +2 ) , 2.
if c p = δ ( p even ) Θ( p − a ) , then λ k
= δ ( k even ) Θ( k − d − 2 a +2 ) , 3.
if c p = O (cid:0) exp (cid:0) − a √ p (cid:1)(cid:1) , then λ k
= O (cid:16) k − d +1 / 2 exp (cid:16) − a √ k (cid:17)(cid:17) , 4.
if c p =
Θ( p 1 / 2 a − p ) , then λ k
= O (cid:0) k − d +1 a − k (cid:1) and λ k = Ω (cid:0) k − d/ 2+
Proof of Corollary C.4, part 1.
We ﬁrst prove λ k
= O ( k − d − 2 a +2 ) .
Suppose that c p ≤ Cp − a for some constant C , then according to Theorem 4.6 we have
[Equation]
According to Stirling’s formula, we have
[Equation]
Then for any z ≥ 12 , we can ﬁnd constants C 1 and C 2 such that
[Equation]
Then
[Equation]
[Equation]
We deﬁne
[Equation]
By applying the chain rule to e log f a ( p ) , we have that the derivative of f a is f (cid:48) a ( p ) =
( p + 1) p + 12 p − a 2( p − k + 1) p − k
+ ( p + k + d + 1) p + k + d· (cid:18) − 2 a p − k + d
( p + 1)( p + k + d + 1) + log(1 +
[Equation]
Let g a ( p ) =
− 2 ap −
[Equation]
+
[Equation]
( p
[Equation]
+ k 2 − d p − k +1)
[Equation]
p − k +
[Equation]
)( p + k + d +1) ) .
Then g a ( p ) and f (cid:48) a ( p ) have the same sign.
Next we will show that g a ( p ) ≥ 0 for k ≤ p ≤ k 2 d +24 a when k is large enough.
First when p ≥ k and
[Equation]
, we have
[Equation]
when k is sufﬁciently large.
Second when p ≥ k and 0 ≤ k
[Equation]
d
[Equation]
k +
[Equation]
[Equation]
+ k + d +1) ≤ 1 , since log(1 + x ) ≥ x 2 for 0 ≤ x ≤ 1 , we have
[Equation]
+
[Equation]
When p ≤ k 2 d +24 a , we have k 2 − dp ≥ 24 ap .
Then
[Equation]
when k is sufﬁciently large.
Also we have k
[Equation]
when k is sufﬁciently large.
Combining all the arguments above, we conclude that g a ( p ) ≥ 0 and f (cid:48) a ( p ) ≥ 0 when k ≤ p ≤ k 2 d +24 a .
Then when k ≤ p ≤ k 2 d +24 a , we have
[Equation]
When p ≥ k 2 d +24 a , we have
[Equation]
If k 2 − d ( p − k +1) < 0 , (cid:16) 1 − k 2 − d ( p − k +1) ( p +1)
[Equation]
(cid:17) p − k + ≥ 1 .
If k 2 − d ( p − k +1) ≥ 0 , i.e., p ≤ k 2 + dk − d d
[Equation]
for sufﬁciently large k , we have
[Equation]
which is a constant independent of k .
Then for p ≥ k 2 d +24 a , we have
[Equation]
Finally we have
[Equation]
Next we prove λ k = Ω( k − d − 2 a +2 ) .
Since c p are nonnegative and c p =
Θ( p − a ) , we have that c p ≥ C (cid:48)
p − a for some constant C (cid:48) .
Then we have
[Equation]
[Equation] According to Stirling’s formula (62) and (63), using the similar argument as (64) we have λ k
[Equation]
π [Equation]
[Equation]
where f a ( p ) is deﬁned in (65).
When p ≥ k 2 , we have f
[Equation]
For sufﬁciently large k , k 2 − d ( p − k + 1) < 0 .
Then we have (cid:18)
[Equation]
which is a constant independent of k .
Also, for sufﬁciently large k , we have
[Equation]
Then for p ≥ k 2 , we have f a ( p ) ≥ e − d 2 − 3 2 ( p + 1) − a − d 2 .
Finally we have
[Equation]
Overall, we have λ k =
Θ( k − d − 2 a +2 ) .
Proof of Corollary C.4, part 2.
It is easy to verify that λ k = 0 when k is even because c p = 0 when p ≥ k and p − k is even.
When k is odd, the proof of Theorem 4.6 still applies.
Proof of Corollary C.4, part 3.
Since c p = O (cid:0) exp (cid:0) − a √ p (cid:1)(cid:1) , we have that c p ≤ Ce − a √ p for some constant C .
Similar to (64), we have
[Equation]
Use the deﬁnition in (65) and let a = 0 , we have
[Equation]
Then according to (68) and (69), for sufﬁciently large k , we have f 0 ( p ) ≤ f 0 (cid:16) k 2 d (cid:17) when k ≤ p ≤ k 2 d and f 0 ( p ) ≤ C 3 p − d 2 for some constant C 3 when p ≥ k 2 d .
Then when k ≤ p ≤ k 2 d , we have f 0 ( p ) ≤ f 0 (cid:16) k 2 d (cid:17) ≤ C 3 (cid:16) k 2 d (cid:17) − d 2 .
When p ≥ k 2 d , we have f 0 ( p ) ≤ C 3 p − d 2 ≤ C 3 (cid:16) k 2 d (cid:17) − d 2 .
Overall, for all p ≥ k , we have
[Equation]
Then we have
[Equation]
Proof of Corollary C.4, part 4.
Since c p = Θ( p 1 / 2 a − p ) , we have that c p ≤ Cp 1 / 2 a − p for some constant C .
Similar to (64), we have
[Equation]
Use the deﬁnition in (65) and let a = 0 , we have
[Equation]
Then according to (68) and (69), for sufﬁciently large k , we have f 0 ( p ) ≤ f 0 (cid:16) k 2 d (cid:17) when k ≤ p ≤ k 2 d and f 0 ( p ) ≤ C 3 p − d 2 for some constant C 3 when p ≥ k 2 d .
Then when k ≤ p ≤ k 2 d , we have p 1 / 2 f 0 ( p ) ≤ p 1 / 2 f 0 (cid:16) k 2 d (cid:17) ≤ C 3 (cid:16) k 2 d (cid:17) 1 / 2 (cid:16) k 2 d (cid:17) − d 2 .
When p ≥ k 2 d , we have p 1 / 2 f 0 ( p ) ≤
C 3 p 1 / 2 p − d 2 ≤ C 3 (cid:16) k 2 d (cid:17) − d 2 + 12 .
Overall, for all p ≥ k , we have
[Equation]
Then we have
[Equation]
On the other hand, since c p =
Θ( p 1 / 2 a − p )
, we have that c p ≥ C (cid:48)
p 1 / 2 a − p for some constant C (cid:48) .
Similar to (72), we have
[Equation]
Since ( k + 1) k = k k
(1+1 /k )
k
= Θ( k k ) .
Similarly, ( k + k + 1 + d ) k = Θ((2 k ) k ) . Then we have [Equation] [Equation]
For the NTK of a two-layer ReLU network with γ b > 0 , then according to Lemma 3.2 we have c p = κ p, 2 =
Θ ( p − 3 / 2 ) . Therefore using Corollary 4.7 λ k =
Θ
( k − d − 1 ) .
Notice here that k refers to the frequency, and the number of spherical harmonics of frequency at most k is Θ( k d ) .
Therefore, for the (cid:96) th largest eigenvalue λ (cid:96) we have λ (cid:96) = Θ( (cid:96) − ( d +1) /d ) . This rate agrees with Basri et al. and Velikanov & Yarotsky (2021).
For the NTK of a two-layer ReLU network with γ b = 0 , the eigenvalues corresponding to the even frequencies are 0 , which also agrees with Basri et al.
Corollary 4.7 also shows the decay rates of eigenvalues for the NTK of two-layer networks with Tanh activation and Gaussian activation.
We observe that when the coefﬁcients of the kernel power series decay quickly then the eigenvalues of the kernel also decay quickly.
As a faster decay of the eigenvalues of the kernel implies a smaller RKHS, Corollary 4.7 demonstrates that using ReLU results in a larger RKHS relative to using either Tanh or Gaussian activations.
We numerically illustrate Corollary 4.7 in Figure 4, Appendix C.3.
C. A NALYSIS OF THE LOWER SPECTRUM : NON - UNIFORM DATA
The purpose of this section is to prove a formal version of Theorem 4.8.
In order to prove this result we ﬁrst need the following lemma.
Lemma C.20.
Let the coefﬁcients ( c j ) ∞ j =0 with c j ∈ R ≥ 0 for all j ∈ Z ≥ 0 be such that the series (cid:80) ∞
j =
c j ρ j converges for all ρ ∈ [ − 1 , 1] .
Given a data matrix X ∈ R n × d with (cid:107) x
i (cid:107) =
1 for all i ∈ [ n ] , deﬁne r := rank( X ) ≥ 2 and the gram matrix G :=
XX T .
Consider the kernel matrix
[Equation]
For arbitrary m ∈
Z ≥ 1 , let the eigenvalue index k satisfy n ≥ k > rank
( H m ) , where H m :=
(cid:80) m −j =
c j G (cid:12) j .
Then
[Equation]
Proof.
We start our analysis by considering λ k ( n K ) for some arbitrary k ∈ N ≤ n .
Let
[Equation]
be the m -head and m -tail of the Hermite expansion of n K : clearly n K = H m + T m for any m ∈ N .
Recall that a constant matrix is symmetric and positive semi-deﬁnite, furthermore, by the Schur product theorem, the Hadamard product of two positive semi-deﬁnite matrices is positive semi-deﬁnite.
As a result, G (cid:12) j is symmetric and positive semi-deﬁnite for all j ∈ Z ≥ 0 and therefore
H m and T m are also symmetric positive semi-deﬁnite matrices.
From Weyl’s inequality (Weyl, 1912, Satz 1) it follows that
[Equation]
In order to upper bound λ 1 ( T m ) , observe, as T m is square, symmetric and positive semi-deﬁnite, that λ 1 ( T m )
= (cid:107) T m (cid:107) .
Using the non-negativity of the coefﬁcients ( c j ) ∞ j =0 and the triangle inequality we have
[Equation]
By the assumptions of the lemma [ G ] ii = 1 and therefore [ G ] jii = 1 for all j ∈ Z ≥ 0 .
Furthermore, for any pair of positive semi-deﬁnite matrices A , B ∈ R n × n and k ∈ [ n ]
[Equation]
Schur (1911).
Therefore, as max i ∈ [ n ] [ G ] ii = 1 ,
[Equation]
for all j ∈ N .
As a result
[Equation]
Finally, we now turn our attention to the analysis of λ k
( H m ) .
Upper bounding a small eigenvalue is typically challenging, however, the problem simpliﬁes when and k exceeds the rank of H m , as is assumed here, as this trivially implies λ k ( H m ) = 0 .
Therefore, for k > rank( H m )
[Equation]
as claimed.
In order to use Lemma C.20 we require an upper bound on the rank of H m .
To this end we provide
Lemma C.21. Lemma C.21.
Let G ∈ R n × n be a symmetric, positive semi-deﬁnite matrix of rank 2 ≤ r ≤ d .
Deﬁne H m ∈ R n × n as
[Equation]
where ( c j ) m − 1 j =0 is a sequence of real coefﬁcients.
Then
[Equation]
Proof.
As G is a symmetric and positive semi-deﬁnite matrix [Equation] its eigenvalues are real and nonnegative and its eigenvectors are orthogonal.
Let { v i } ri =1 be a set of orthogonal eigenvectors for
G and γ i the eigenvalue associated with v i ∈ R n .
Then G may be written as a sum of rank one matrices as follows,
[Equation]
As the Hadamard product is commutative [Equation] associative and distributive over addition [Equation]
for any j ∈ Z ≥
G (cid:12) j can also be expressed as a sum of rank 1 matrices,
[Equation]
Note the fourth equality in the above follows from v i v Ti = v i ⊗ v i and an application of the mixedproduct property of the Hadamard product.
As matrix rank is sub-additive, the rank of G (cid:12) j is less than or equal to the number of distinct rank-one matrix summands.
This quantity in turn is equal to the number of vectors of the form (cid:0) v i 1 (cid:12) v i 2 (cid:12) · · · (cid:12) v i j (cid:1) , where i 1 , i 2 , . .
, i j ∈ [ r ] .
This in turn is equivalent to computing the number of j -combinations with repetition from r objects.
Via a stars and bars argument this is equal to (cid:0) r + j − 1 j
[Equation]
It therefore follows that
[Equation]
The rank of H m can therefore be bounded via subadditivity of the rank as
[Equation]
+ m − 1 (cid:88)
j[-− 1) r −(cid:19) r −≤ e r − 1 (cid:18) 
-]
= (2 e ) r −(cid:32) δ [Equation] r
[Equation] + δ [Equation] (cid:18)
j r [Equation] (cid:19)
[Equation]
(cid:33) ≤ 1 + min { r − 1 , m − 1 } (2 e ) r −+ max { 0 , m − r } (cid:18)
2 e r −(cid:19) r −( m − 1) r − 1 .
As our goal here is to characterize the small eigenvalues, then as n grows we need both k and therefore m to grow as well.
As a result we will therefore be operating in the regime where m > r .
To this end we provide the following corollary.
Corollary C.22.
Under the same conditions and setup as Lemma C.21 with m ≥ r ≥ 7 then
[Equation]
Proof.
If r ≥ 7 > 2 e + 1 then r − 1 > 2 e .
As a result from Lemma C.
[Equation]
as claimed.
Corollary C.22 implies for any k ≥ 2 m r , k ≤ n that we can apply Lemma C.20 to upper bound the size of the k th eigenvalue.
Our goal is to upper bound the decay of the smallest eigenvalue.
To this end, and in order to make our bounds as tight as possible, we therefore choose the truncation point m ( n ) =
(cid:98) ( n/ 2) 1 /r (cid:99) , note this is the largest truncation which still satisﬁes 2 m ( n ) r ≤ n .
In order to state the next lemma, we introduce the following pieces of notation: with L := { (cid:96) : R ≥
→ R ≥ 0 } deﬁne U :
L × Z ≥ 1 → R ≥ 0 as
[Equation]
Lemma C.23.
Given a sequence of data points ( x i ) i ∈
Z ≥ 1 with x i ∈ S d for all i ∈ Z ≥ 1 , construct a sequence of row-wise data matrices ( X n ) n ∈ Z ≥ 1 , X n ∈ R n × d , with x i corresponding to the i th row of X n .
The corresponding sequence of gram matrices we denote G n := X n X Tn .
Let m ( n ) :
= (cid:98) ( n/ 2) 1 /r ( n ) (cid:99)
where r ( n ) :=
rank( X n ) and suppose for all sufﬁciently large n that m ( n ) ≥ r ( n ) ≥ 7 .
Let the coefﬁcients ( c j ) ∞ j =0 with c j ∈ R ≥ 0 for all j ∈ Z ≥ 0 be such that 1) the series (cid:80) ∞
j {+=
+}
c j ρ j converges for all ρ ∈ [ − 1 , 1] and 2) ( c j ) ∞
j {+=
+}
= O ( (cid:96) ( j )) , where (cid:96)
∈ L satisﬁes U ( (cid:96), m ( n )) < ∞ for all n and is monotonically decreasing.
Consider the sequence of kernel matrices indexed by n and deﬁned as
[Equation]
With ν : Z ≥ 1 → Z ≥ 1 suppose (cid:13)(cid:13)(cid:13) G (cid:12) m ( n ) n (cid:13)(cid:13)(cid:13) =
O ( n − ν ( n )+1 ) , then
[Equation]
Proof.
By the assumptions of the Lemma we may apply Lemma C.20 and Corollary C.22, which results in
[Equation]
Additionally, as ( c j ) ∞ j =
=
O
( (cid:96) ( j ))
then
[Equation]
as claimed.
Based on Lemma C.20 we provide Theorem C.24, which considers three speciﬁc scenarios for the decay of the power series coefﬁcients inspired by Lemma 3.2.
Theorem C.24.
In the same setting, and also under the same assumptions as in Lemma C.23, then
if c p =
O ( p − α ) with α > r ( n ) + 1 for all n ∈ Z ≥ 0 then λ n ( K n ) =
O (cid:16) n − α − 1 r ( n ) (cid:17) , 2.
if c p =
O ( e − α √ p ) , then λ n ( K n ) =
O (cid:16) n 1 2 r ( n ) exp (cid:16) − α (cid:48) n 1 2 r ( n ) (cid:17)(cid:17) for any α (cid:48) <
α 2 − 1 / 2 r ( n ) , 3.
if c p =
O ( e − αp ) , then λ n ( K n ) =
O (cid:16) exp (cid:16) − α (cid:48) n 1 r ( n ) (cid:17)(cid:17) for any α (cid:48) <
α 2 − 1 / 2 r ( n ) .
Proof.
First, as [ G n ] ij ≤ 1 then
[Equation]
Therefore, to recover the three results listed we now apply Lemma C.23 with ν ( n ) = 0 .
First, to prove 1., under the assumption (cid:96) ( x ) = x − α with α > 0 then
[Equation]
As a result
[Equation]
To prove ii), under the assumption (cid:96) ( x )
=
e − α √ x with α > 0 then (cid:90) ∞
[Equation]
As a result
[Equation]
for any α (cid:48) <
α 2 − 1 / 2 r ( n ) .
Finally, to prove iii), under the assumption (cid:96) ( x ) =
e − αx with α > 0 then (cid:90)
[Equation]
Therefore
[Equation]
again for any α (cid:48) <
α 2 − 1 / 2 r ( n ) . Unfortunately, the curse of dimensionality is clearly present in these results due to the 1 /r ( n ) factor in the exponents of n .
However, although perhaps somewhat loose we emphasize that these results are certainly far from trivial.
In particular, while trivially we know that λ n ( K n ) ≤
Tr ( K n ) /n =
O ( n − 1 ) , in contrast, even the weakest result concerning the power law decay our result is a clear improvement as long as α > r ( n ) + 1 .
For the other settings, i.e., those speciﬁed in 2.
and 3., our results are signiﬁcantly stronger.