T OWARDS U NDERSTANDING THE S PECTRAL B IAS OF D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
An intriguing phenomenon observed during training neural networks is the spectral bias, where neural networks are biased towards learning less complex functions.
The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias.
However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias.
In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work.
We prove that the training process of neural networks can be decomposed along different directions deﬁned by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue.
We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks.
I NTRODUCTION
Over-parameterized neural networks have achieved great success in many applications such as computer vision (He et al., 2016), natural language processing (Collobert and Weston, 2008) and speech recognition (Hinton et al., 2012).
It has been shown that over-parameterized neural networks can ﬁt complicated target function or even randomly labeled data (Zhang et al., 2016) and still exhibit good generalization performance when trained with real labels.
Intuitively, this is at odds with the traditional notion of generalization ability such as model complexity.
In order to understand neural network training, a line of work (Soudry et al., 2017; Gunasekar et al., 2018b;a) has made efforts in the perspective of “implicit bias”, which states that training algorithms for deep learning implicitly pose an inductive bias onto the training process and lead to a solution with low complexity measured by certain norms in the parameter space of the neural network.
Among many attempts to establish implicit bias, Rahaman et al.
(2018) pointed out an intriguing phenomenon called spectral bias , which says that during training, neural networks tend to learn the components of lower complexity faster.
The concept of spectral bias is appealing because this may intuitively explain why over-parameterized neural networks can achieve a good generalization performance without overﬁtting.
During training, the networks ﬁt the low complexity components ﬁrst and thus lie in the concept class of low complexity.
Arguments like this may lead to rigorous guarantee for generalization.
Great efforts have been made in seek of explanations about the spectral bias. Rahaman et al.
(2018) evaluated the Fourier spectrum of ReLU networks and empirically showed that the lower frequencies are learned ﬁrst; also lower frequencies are more robust to random perturbation.
Andoni et al.
(2014) showed that for a sufﬁciently wide two-layer network, gradient descent with respect to the second layer can learn any low degree bounded polynomial.
Xu (2018) provided Fourier analysis to twolayer networks and showed similar empirical results on one-dimensional functions and real data.
Nakkiran et al.
(2019) used information theoretical approach to show that networks obtained by stochastic gradient descent can be explained by a linear classiﬁer during early training.
All these studies provide certain explanations about why neural networks exhibit spectral bias in real tasks.
But explanations in the theoretical aspect, if any, are to some extent restricted.
For example, the popular Fourier analysis is usually done in the one-dimensional setting, and thus lacks generality.
Meanwhile, a recent line of work (Jacot et al., 2018; Du et al., 2018b; Li and Liang, 2018; Chizat and Bach, 2018) has shed light on new approaches to analyze neural networks.
In particular, they show that under certain over-parameterized condition, the neural network trained by gradient descent behaves similar to the kernel regression predictor using the neural tangent kernel (NTK) (Jacot et al., 2018).
Du et al.
(2018b) showed that the convergence is provably guaranteed under certain over-parameterization conditions determined by the smallest eigenvalue of NTK.
Arora et al.
(2019b) further gave a ﬁner characterization of error convergence based on the eigenvalues of NTK’s Gram matrix.
Su and Yang (2019) improved the convergence guarantee in terms of the k -th largest eigenvalue for certain target functions.
Inspired by these works mentioned above, we can present a theoretical explanation for spectral bias.
Under NTK regime, we establish a precise characterization for the training process of neural networks.
More speciﬁcally, we theoretically prove that over-parameterized neural networks’ training process can be controlled by the eigenvalues of the integrating operator deﬁned by the NTK.
Under the speciﬁc case of uniform distribution on unit sphere, we give an exact calculation for these eigenvalues and show that the lower frequencies have larger eigenvalues, which thus leads to faster convergence.
We also conduct experiments to corroborate the theory we establish.
Our contributions are highlighted as follows:
We prove a generic theorem for arbitrary data distributions, which states that under certain sample complexity and over-parameterization conditions, the error term’s convergence along different directions actually relies on the corresponding eigenvalues.
This theorem gives a more precise control on the regression residual than Su and Yang (2019), where the authors focused on the case when the labeling function is close to the subspace spanned by the ﬁrst few eigenfunctions.
We present a more general result about the spectra of the neural tangent kernel.
In particular, we show that the order of eigenvalues appears as µ k “ Ω p max t
k ´ d ´ 1 , d ´ k ` 1 uq . Our result is better than the bound Ω p k ´ d ´ 1 q derived in Bietti and Mairal (2019) when d " k , which is clearly a more practical setting.
We establish a rigorous explanation for the spectral bias based on the aforementioned theoretical results without any speciﬁc assumptions on the target function.
We show that the error terms from different frequencies are provably controlled by the eigenvalues of the NTK, and the lowerfrequency components can be learned with less training examples and narrower networks with a faster convergence rate.
As far as we know, this is the ﬁrst attempt to give a comprehensive theory justifying the existence of spectral bias.
1. A DDITIONAL R ELATED W ORK
Recently, there is a rich literature about the property of neural tangent kernel.
Jacot et al.
ﬁrst showed that during training, the network function follows a descent along the kernel gradient with respect to the Neural Tangent Kernel (NTK) under inﬁnity width setting.
Li and Liang (2018) and
Du et al.
(2018b) implicitly built connection between Neural Tangent Kernel and gradient descent by showing that GD can provably optimize sufﬁciently wide two-layer neural networks.
In Du et al.
(2018b), it is shown that gradient descent can achieve zero training loss at a linear convergence rate for training two-layer ReLU network with square loss.
Allen-Zhu et al.
Du et al.
(2018a);
Zou et al.
; Cao and Gu (2019b); Arora et al. (2019a); Zou and Gu (2019); Cao and Gu (2019a) further studied the optimization and generalization of deep neural networks. These papers are all in the so-called neural tangent kernel regime, and their requirements on the network width depend either implicitly or explicitly on the smallest eigenvalue of the kernel Gram matrix.
Later, Su and Yang (2019) showed that this smallest eigenvalue actually scales in the number of samples n and will eventually converge to 0 .
In order to obtain constant convergence rate, Su and Yang (2019) assumed that the target function f ˚ can be approximated by the ﬁrst few eigenfunctions of the integrating operator L κ f p s q : “ ş
S d κ p x, s q f p s q dτ p s q where κ p¨ , ¨q is the NTK function and τ p s q is the input distribution, and proved linear convergence rate up to the this approximation error.
A few theoretical results have been established towards understanding the spectra of neural tangent kernels.
Bach (2017) studied two-layer ReLU networks by relating it to kernel methods, and proposed a harmonic decomposition for the functions in the reproducing kernel Hilbert space which we utilize in our proof.
Based on the technique in Bach (2017), Bietti and Mairal (2019) studied the eigenvalue decay of integrating operator L κ f p x q deﬁned by NTK on unit sphere by using spheri- cal harmonics.
Vempala and Wilmes (2018) calculated the eigenvalues of NTK corresponding to two-layer neural networks with sigmoid activation function.
Basri et al.
(2019) established similar results as Bietti and Mairal (2019), but considered the case of training the ﬁrst layer parameters of a two-layer networks with bias terms.
Yang and Salman (2019) studied the the eigenvalues of integral operator with respect to the NTK on Boolean cube by Fourier analysis.
The rest of the paper is organized as follows.
We state the notation, problem setup and other preliminaries in Section 2 and present our main results in Section 3.
In Section 4, we present experimental results to support our theory.
Proofs of our main results can be found in the appendix.
P RELIMINARIES
In this section we introduce the basic problem setup including the neural network structure and the training algorithm, as well as some background on the neural tangent kernel proposed recently in
Jacot et al.
(2018) and the corresponding integral operator.
2. N OTATION
We use lower case, lower case bold face, and upper case bold face letters to denote scalars, vectors and matrices respectively.
For a vector v “ p v 1 , . . .
, v d q T P R d and a number 1 ď p ă 8 , we denote its p ´ norm by } v } p “ p ř di “ 1 | v i | p q 1 { p .
We also deﬁne inﬁnity norm by } v } 8 “ max i | v i | .
For a matrix A “ p A i,j q m ˆ n , we use } A } 0 to denote the number of non-zero entries of A , and use } A } F “ p ř di,j “ 1 A 2 i,j q 1 { 2 to denote its Frobenius norm.
Let } A } p “ max } v } p ď 1 } Av } p for p ě 1 , and } A } max “ max i,j | A
i,j | .
For two matrices A , B P R m ˆ n , we deﬁne x
A , B y
“ Tr p A J B q .
We use A ľ B if A ´ B is positive semi-deﬁnite.
In addition, we deﬁne the asymptotic notations O p¨q , r
O p¨q , Ω p¨q and r Ω p¨q as follows.
Suppose that a n and b n be two sequences.
We write a n “ O p b n q if lim sup n Ñ8 | a n { b n | ă 8 , and a n “ Ω p b n q if lim inf n Ñ8 | a n { b n | ą 0 .
We use r O p¨q and r Ω p¨q to hide the logarithmic factors in O p¨q and Ω p¨q .
2. P ROBLEM S ETUP
Here we introduce the basic problem setup.
We consider two-layer fully connected neural networks of the form
[Equation]
where W 1 P R m ˆp d ` 1 q , W 2 P R 1 ˆ m 1 are the ﬁrst and second layer weight matrices respectively, and σ p¨q “ max t 0 , ¨u is the entry-wise ReLU activation function.
The network is trained according to the square loss on n training examples S “ tp x i , y i q : i P r n su :
[Equation]
where θ is a small coefﬁcient to control the effect of initialization, and the data inputs t x i u ni “ 1 is assumed to follow some unknown distribution τ on the unit sphere S d P R d ` 1 .
Without loss of generality, we also assume that | y i | ď 1 .
We ﬁrst randomly initialize the parameters of the network, and then apply gradient descent to optimize both layers.
We present our detailed neural network training algorithm in Algorithm 1.
Algorithm 1 GD for DNNs starting at Gaussian initialization
Input: Number of iterations T , step size η .
Generate each entry of W p 0 q 1 and W p 0 q 2 from N p 0 , 2 { m q
and N p 0 , 1 { m q respectively.
for t “ 0 , 1 , . . .
T ´ 1 do
Update W p [Equation] The initialization scheme for W p 0 q given in Algorithm 1 is known as He initialization (He et al., 2015).
This scheme generates each entry of the weight matrices from a Gaussian distribution with mean zero.
The variances of the Gaussian distributions in initialization are chosen following the principle that the initialization does not change the magnitudes of inputs in each layer.
The second layer parameter is not associated with the ReLU activation function, thus it is initialized with variance 1 { m instead of 2 { m .
2. N EURAL T ANGENT K ERNEL
Many attempts have been made to study the convergence of gradient descent assuming the width of the network is extremely large (Du et al., 2018b; Li and Liang, 2018).
When the width of the network goes to inﬁnity, with certain initialization on parameters, the inner product of gradients of the output function would converge to a limiting kernel, i.e., Neural Tangent Kernel (Jacot et al., 2018).
In this paper, we denote it by κ p x , x 1 q “ lim m Ñ8 m ´ 1 x ∇ W f W p
q p x q , ∇ W f W p 0 q p x 1 qy
and we have
[Equation]
1) where
[Equation]
Since we apply gradient descent to both layers, the Neural Tangent Kernel is the sum of two different kernel functions and clearly it can be reduced to one layer training setting.
These two kernels are arc-cosine kernels of degree 0 and 1 (Cho and Saul, 2009).
Their explicit expressions are given as
[Equation]
where t “ x x , x 1 y{ p} x } } x 1 }q .
2. I NTEGRAL O PERATOR
The theory of integral operator with respect to kernel function has been well studied in machine learning (Smale and Zhou, 2007; Rosasco et al., 2010) thus we only give a brief introduction here.
Let L 2 τ p X q be the Hilbert space of square-integrable functions with respect to a Borel measure τ from X Ñ R .
For any continuous kernel function κ : X ˆ X Ñ R and τ we can deﬁne an integral operator
L κ on L 2 τ p X q by
[Equation]
It has been pointed out in Cho and Saul (2009) that arc-cosine kernels are positive semi-deﬁnite.
Thus the kernel function κ deﬁned by (2.1) is positive semi-deﬁnite being a product and a sum of positive semi-deﬁnite kernels.
Clearly this kernel is also continuous and symmetric.
Thus we know that the neural tangent kernel κ is a Mercer kernel.
M AIN R ESULTS
In this section we present our main results.
In Section 3.1, we give a general result on the convergence rate of gradient descent along different eigendirections of neural tangent kernel.
Motivated by this result, in Section 3.2, we give a case study on the spectrum of L κ when the input data are uniformly distributed over the unit sphere S d .
In Section 3.3, we combine the spectrum analysis with the general convergence result to give explicit convergence rate for uniformly distributed data on the unit sphere.
3. C ONVERGENCE A NALYSIS OF G RADIENT D ESCENT
In this section we study the convergence of Algorithm 1.
Instead of studying the standard convergence of loss function value, we aim to provide a reﬁned analysis on the speed of convergence along different directions deﬁned by the eigenfunctions of L κ .
We ﬁrst introduce the following deﬁnitions and notations.
Let t λ i u i ě 1 with λ 1 ě λ 2 ě ¨ ¨ ¨ be the strictly positive eigenvalues of L κ , and φ 1 p¨q , φ 2 p¨q , . . be the corresponding orthonormal eigenfunctions.
Set v i “ n ´ 1 { 2 p φ i p x 1 q , . .
, φ i p x n qq J , i “ 1 , 2 , . . . .
Note that L κ may have eigenvalues with multiplicities larger than
and λ i , i ě 1 are not distinct.
Therefore for any integer k , we deﬁne r k as the sum of the multiplicities of the ﬁrst k distinct eigenvalues of L κ .
Deﬁne V r k “ p v 1 , . .
. , v r k q .
By deﬁnition, v i , i P r r k s are rescaled restrictions of orthonormal functions in L 2 τ p S d q on the training examples.
Therefore we can expect them to form a set of almost orthonomal bases in the vector space R n .
The following lemma follows by standard concentration inequality.
Lemma 3.1.
Suppose that | φ i p x q| ď M for all x P S d and i P r r k s .
For any δ ą 0 , with probability at least 1 ´ δ ,
[Equation]
where C is an absolute constant.
Denote y “ p y 1 , . .
, y n q J and p y p t q “ θ ¨ p f W p t q p x 1 q , . .
, f W p t q p x n qq , t “ 0 , . . .
, T .
Then
Lemma 3.1 shows that the convergence rate of } V J r k p
y ´ p y p t q q} 2 roughly represents the speed gradient descent learns the components of the target function corresponding to the ﬁrst r k eigenvalues.
The following theorem gives the convergence guarantee of } V J r k p y ´ p y p t q q} 2 .
Theorem 3.2.
Suppose | φ j p x q| ď M for j P r r k s and x P S d .
For any (cid:15), δ ą
and integer k , if n ě r Ω p (cid:15) ´ 2 ¨ max tp λ r k
´ λ
r k ` 1 q ´ 2 , M 4 r 2 k uq , m ě r Ω p poly p T, λ ´ 1 r k , (cid:15) ´ 1 qq , then with probability at least 1 ´ δ , Algorithm 1 with η “ r O pp mθ 2 q ´ 1 q , θ “ r O p (cid:15) q satisﬁes [Equation]
¨
[Equation]
Remark 3.3.
Theorem 3.2 theoretically reveals the spectral bias of deep learning.
Speciﬁcally, as long as the network is wide enough and the sample size is large enough, gradient descent ﬁrst learns the target function along the the eigendirections of neural tangent kernel with larger eigenvalues, and then learns the rest components corresponding to smaller eigenvalues.
Moreover, by showing that learning the components corresponding to larger eigenvalues can be done with smaller sample size and narrower networks, our theory pushes the study of neural networks in the NTK regime towards a more practical setting.
For these reasons, we believe that Theorem 3.2 to certain extent explains the empirical observations given in Rahaman et al.
(2018), and demonstrates that the difﬁculty of a function to be learned by neural network can be characterized in the eigenspace of neural tangent kernel: if the target function has a component corresponding to a small eigenvalue of neural tangent kernel, then learning this function up to good accuracy takes longer time, and requires more examples and a wider network.
3. S PECTRAL ANALYSIS OF N EURAL T ANGENT K ERNEL FOR U NIFORM D ISTRIBUTION
After presenting a general theorem (without assumptions on data distribution) in the previous subsection, we now study the case when the data inputs are uniformly distributed over the unit sphere.
We present our results (an extension of Proposition 5 in Bietti and Mairal (2019)) of spectral analysis of neural tangent kernel.
We show Mercer decomposition of neural tangent kernel for two-layer setting.
We give explicit expression of eigenvalues and show orders of eigenvalues in both cases when d " k and k " d .
Theorem 3.4.
For any x , x 1 P S d Ă R d ` 1 , we have the Mercer decomposition of the neural tangent kernel κ : S d ˆ S d Ñ R ,
[Equation]
where Y k,j for j “ 1 , ¨ ¨ ¨ , N p d, k q are linearly independent spherical harmonics of degree k in d `variables with N p d, k q “ 2 k ` d ´ 1 k
[Equation]
d ´ 1 ˘ and orders of µ k are given by
[Equation]
where j P N ` .
More speciﬁcally, we have µ k “ Ω ` k ´ d ´ 1 ˘ when k " d and µ k “ Ω ` d ´ k ` 1 ˘ when d " k , k “ 2 , 4 , 6 , . . . .
Remark 3.5.
In the above theorem, the coefﬁcients µ k are actually different eigenvalues of the integral operator L κ on L 2 τ d p S d q deﬁned by
[Equation]
where τ d is the uniform probability measure on unit sphere S d .
Therefore the λ r k in Theorem 3.2 is just µ k ´ 1 given in Theorem 3.4 when τ d is uniform distribution.
Remark 3.6.
Vempala and Wilmes (2018) studied two-layer neural networks with sigmoid activation function, and {+established guarantees to achieve (cid:15) 
+} ` (cid:15) error with iteration complexity
T “ p d ` 1 q O p k q log p} f ˚ } 2 { (cid:15) [-to achieve (cid:15) 
-]q under the over-parameterization condition m “ p d ` 1 q O p k q poly p} f ˚ } 2 { (cid:15) q , where f ˚ is the target function, and (cid:15) 0 is certain function approximation error.
Another highly related work is Bietti and Mairal (2019), which gives µ k “ Ω p k ´ d ´ 1 q .
The order of eigenvalues we present appears as µ k “ Ω p max p k ´ d ´ 1 , d ´ k ` 1 qq .
This is better when d " k , which is closer to the practical setting.
3. E XPLICIT C ONVERGENCE R ATE FOR U NIFORMLY D ISTRIBUTED D ATA
In this subsection, we combine our results in the previous two subsections and give explicit convergence rate for uniformly distributed data on the unit sphere.
Note that the ﬁrst k distinct eigenvalues of NTK have spherical harmonics up to degree k ´ 1 as eigenfunctions.
Corollary 3.7.
Suppose that k " d , and the sample t x i u ni “ 1 follows the uniform distribution τ d on the unit sphere S d .
For any (cid:15), δ ą {+
+} and integer k , if n [Equation]
r Ω p poly p T, k d ` 1 , (cid:15) ´ 1 qq , then with probability at least 1 ´ δ , Algorithm 1 with η “ r O pp mθ 2 q ´ 1 q , θ “ r O p (cid:15) q satisﬁes
[Equation]
where r k “ ř k ´ 1 k 1 “ 0 N p d, k 1 q and V r k “ p n ´ 1 { 2 φ j p x
i qq n ˆ r k with φ 1 , . . . , φ r k being a set of orthonomal spherical harmonics of degrees up to k ´ 1 .
Corollary 3.8.
Suppose that d " k , and the sample t x i u ni “ 1 follows the uniform distribution τ d on the unit sphere S d .
For any (cid:15), δ ą {+
+} and integer k , if n ě r Ω p (cid:15) ´ 2 d 2 k ´ 2 r 2 k q , m ě r Ω p poly p T, d k ´ 2 , (cid:15)
´
1 qq , then with probability at least 1 ´ δ , Algorithm 1 with η “ r O pp mθ 2 q ´ 1 q , θ “ r O p (cid:15) q satisﬁes
[Equation]
where r k “ ř k ´ 1 k 1 “ 0 N p d, k 1 q and V r k “ p n ´ 1 { 2 φ j p x i qq n ˆ r k with φ 1 , . . . , φ r k being a set of orthonomal spherical harmonics of degrees up to k ´ 1 . Corollaries 3.7 and 3.8 further illustrate the spectral bias of neural networks by providing exact calculations of λ r k , V r k and M in Theorem 3.2.
They show that if the input distribution is uniform over unit sphere, then spherical harmonics with lower degrees are learned ﬁrst by over-parameterized neural networks. Remark 3.9. In Corollaries 3.7 and 3.8, it shows that the conditions on n and m depend exponentially on either k or d .
We would like to emphasize that such exponential dependency is reasonable and unavoidable. In our case, we can take the d " k setting as an example. The exponential dependency in k is a natural consequence of the fact that in high dimensional space, there are a large number of linearly independent polynomials even for very low degrees.
It is apparently only reasonable to expect to learn less than n independent components of the true function, which means that it is unavoidable to assume [Equation] Similar arguments can apply to the requirement of m and the k " d setting.
E XPERIMENTS
In this section we illustrate our results by training neural networks on synthetic data.
Across all tasks, we train a two-layer hidden neural networks with 4096 neurons and initialize it exactly as deﬁned in the setup.
The optimization method is vanilla full gradient descent.
We sampletraining data which is uniformly sampled from the unit sphere in R 10 .
4. L EARNING COMBINATION OF SPHERICAL HARMONICS
First, we show a result when the target function is exactly linear combination of spherical harmonics.
The target function is explicitly deﬁned as
[Equation]
where the P k p t q is the Gegenbauer polynomial, and ζ k , k “ 1 , 2 , 4 are ﬁxed vectors that are independently generated from uniform distribution on unit sphere in R 10 in our experiments.
Note that according to the addition formula ř N p d,k q j “ [Equation]
[Equation] x , y yq , every normalized Gegenbauer polynomial is a spherical harmonic, so f ˚ p x q is a linear combination of spherical harmonics of order 1,2 and 4. The higher odd-order Gegenbauer polynomials are omitted because the spectral analysis showed that µ k “ 0 for k “ 3 , 5 , 7 . . . .
Following the setting in section 3.1, we denote v k “ n ´ 1 { 2 p P k p x 1 q , . . . , P k p x n qq .
By Lemma 3.
v k ’s are almost orthonormal with high probability. So we can deﬁne the (approximate) projection length of residual r p t q onto v k at step t as
[Equation]
where r
[Equation]
x n qq and f W p t q p x q is the neural network function.
Remark 4.1.
Here p a k is the projection length onto an approximate vector.
In the function space, we can also project the residual function r p x q “ f ˚ p x q ´ θf W p t q p x q onto the orthonormal Gegenbauer functions P k p x q . Replacing the training data with randomly sampled data points x i can lead to a random estimate of the projection length in function space. We provide the corresponding result for freshly sampled points in Appendix E.1.
[Figure]
The results are showned in Figure 1.
It can be seen that at the beginning of training, the residual at the lowest frequency ( k “ 1 ) converges to zero ﬁrst and then the second lowest ( k “ 2 ).
The highest frequency component is the last one to converge.
Following the setting of Rahaman et al.
we assign high frequencies a larger scale, expecting that larger scale will introduce a better descending speed.
Still, the low frequencies are regressed ﬁrst.
4. L EARNING FUNCTIONS OF SIMPLE FORM
Apart from the synthesized low frequency function [Figure] we also showed the dynamics of normal functions’ projection to P k p x q .
These functions, though in a simple form, have non-zero components in almost all frequencies.
In this subsection we further show our results still apply when all frequencies exist in the target function, which is given by
f ˚ p x q “ ř i cos p a i x ζ , x yq or f ˚ p x q “ ř i x ζ , x y p i , where ζ is a ﬁxed unit vector.
The coefﬁcients of given components are calculated in the same way as in Section 4.
[Figure]
n l
[Figure]
Figure 2 shows that even for arbitrarily chosen functions of simple form, the networks can still ﬁrst learn the low frequency components of the target function.
Notice that early in training not all the curves may descend, we believe this is due to the unseen components’ inﬂuence on the gradient.
Again, as the training proceeds, the convergence is controlled at the predicted rate.
Remark 4.2.
The reason why we only use cosine function and even polynomial is that the only odd basis function with non-zero eigenvalue is P 1 p x q .
To show a general tendency it is better to restrict the target function in the even function space.
C ONCLUSION AND D ISCUSSION
In this paper, we give theoretical justiﬁcation for spectral bias through a detailed analysis of the convergence behavior of two-layer neural networks with ReLU activation function.
We show that the convergence of gradient descent in different directions depends on the corresponding eigenvalues and essentially exhibits different convergence rates.
We show Mercer decomposition of neural tangent kernel and give explicit order of eigenvalues of integral operator with respect to the neural tangent kernel when the data is uniformly distributed on the unit sphere S d .
Combined with the convergence analysis, we give the exact order of convergence rate on different directions.
We also conduct experiments on synthetic data to support our theoretical result.
So far, we have considered the upper bound for convergence with respect to low frequency components and present comprehensive theorem to explain the spectral bias.
One desired improvement is to give the lower bound of convergence with respect to high frequency components, which is essential to establish tighter characterization of spectral-biased optimization.
It is also interesting to extend our result to other training algorithms like Adam, where the analysis in Wu et al.
; Zhou et al.
(2018) might be implemented with a more careful quantiﬁcation on the projection of residual along different directions.
Another potential improvement is to generalize the result to multi-layer neural networks, which might require different techniques since our analysis heavily rely on exactly computing the eigenvalues of the neural tangent kernel.
It is also an important direction to weaken the requirement on over-parameterization, or study the spectral bias in a non-NTK regime to furthur close the gap between theory and practice.
A R EVIEW ON SPHERICAL HARMONICS
In this section, we give a brief review on relevant concepts in spherical harmonics.
For more detials, see Bach (2017), Bietti and Mairal (2019), Frye and Efthimiou (2012) and Atkinson and Han (2012) for references.
We consider the unit sphere S d “ (cid:32) x
P R d ` 1 : } x } 2 “ 1 ( , whose surface area is given by ω d “ 2 π p d ` 1 q{ 2 { Γ pp d ` 1 q{ 2 q and denote τ d the uniform measure on the sphere.
For any k ě 1 , we consider a set of spherical harmonics !
[Equation]
They form an orthonormal basis and satisfy the following equation x Y ki , Y sj y
[Equation]
Moreover, since they are homogeneous functions of degree k , it is clear that Y k p x q has the same parity as k .
We have the addition formula
[Equation]
where P k p t q is the Legendre polynomial of degree k in d ` 1 dimensions, explicitly given by (Rodrigues’ formula)
[Equation]
We can also see that P k p t q , the Legendre polynomial of degree k shares the same parity with k .
By the orthogonality and the addition formula (A.1) we have, ż
[Equation]
Further we have the recurrence relation for the Legendre polynomials,
[Equation]
for k ě 1 and tP 0 p t q “ P 1 p t q for k “ 0 .
The Hecke-Funk formula is given for a spherical harmonic Y k of degree k ż
[Equation]
B P ROOF OF M AIN T HEOREMS B. P ROOF OF T HEOREM 3.
In this section we give the proof of Theorem 3.2.
The core idea of our proof is to establish connections between neural network gradients throughout training and the neural tangent kernel.
To do so, we ﬁrst introduce the following deﬁnitions and notations.
[Equation]
lim m Ñ8 K p 0 q .
Let t p
λ
i u ni “ 1 , p λ 1 ě ¨
¨
¨ ě p λ n be the eigenvalues of n ´ 1 K 8 , and p v 1 , . .
. , p v n be the corresponding eigenvectors.
Set p V r k “ p p v 1 , . .
. , p v r k q , p V K r k “ p p v r k ` 1 , . .
, p v n q .
For notation simplicity, we denote ∇ W f W {+p 
 q p x i q+}
[Equation]
The following lemma is a direct application of Proposition 1 in Smale and Zhou (2009) or Proposition 10 in Rosasco et al.
It bounds the difference between the eigenvalues of NTK and their ﬁnite-width counterparts.
Lemma B.1.
For any δ ą 0 , with probability at least 1 ´ δ ,
[Equation]
The following lemma is partly summarized from the proof of equation (44) in Su and Yang (2019).
Its purpose is to further connect the eigenfunctions of NTK with their ﬁnite-width, ﬁnite-sample counterparts.
Lemma B.2.
Suppose that | φ i p x q| ď M for all x P S d .
There exist absolute constants C, C 1 , c 2 ą 0 , such that for any δ ą 0 and integer k with r k ď n , if n ě C p λ r k ´ λ r k ` 1 q ´ 2 log p 1 { δ q , then with probability at least 1 ´ δ ,
[Equation]
The following two lemmas are key to characterize the dynamics of the residual throughout training.
These two results, especially Lemma B.4, are the ones that distinguish our analysis from previous works on neural network training in the NTK regime (Du et al., 2018b; Su and Yang, 2019), since our analysis provides more careful characterization on the residual along different directions.
Lemma B.3.
Suppose that the iterates of gradient descent W p 0 q , . . .
, W p t q are inside the ball
B p W p [Equation] q , ω q .
If ω ď O pr log p m qs ´ 3 { 2 q , then with probability at least 1 ´ O p n 2 q ¨ exp r´ Ω p mω 2 { 3 qs , y ´
[Equation]
[Equation]
I ´ p
[Equation]
2 { n
[Equation]
y
[Equation]
p y
[Equation]
for all t 1 “ 0 , . . . , t ´ 1 , where y “ p y 1 , . . . , y n q J , p y p t 1 q “ θ ¨ p f W p t 1q p x 1 q , . . . , f W p t 1q p x n qq J .
Lemma B.4.
Suppose that the iterates of gradient descent W p 0 q , . . .
, W p t q are inside the ball B
[Equation]
W p 0 q , ω q .
If ω ď r O p min tr log p m qs ´ 3 { 2 , λ 3 r k , p ηm q ´ 3 uq and n ě r O p λ ´ 2 r k q , then with probability at least 1 ´ O
[Equation]
for all t 1 “ 0 , . . .
, t ´ 1 .
Now we are ready to prove Theorem 3.2.
Proof of Theorem 3.2.
Deﬁne ω “ CT {p λ r k ? m
q for some small enough absolute constant C .
Then by union bound, as long as m ě r Ω p poly p T, λ ´ 1 r k , (cid:15) ´ 1 qq , the conditions on ω given in Lemmas D.2, D.4, D.5, B.3 and B.4 are all satisﬁed.
We ﬁrst show that all the iterates W p 0 q , . . .
, W p T q are inside the ball B p W p 0 q , ω q .
We prove this result by inductively show that W p t q P B p W p 0 q , ω q , t “ 0 , . . .
, T .
First of all, it is clear that W p 0 q P
B p W p 0 q , ω q .
Suppose that W p 0 q , . .
, W p t q P B p W p 0 q , ω q .
Then the results of Lemmas D.2, D.4, D and
Denote
“ y ´ y
T .
Then we have } W
[Equation]
where the second inequality follows by Lemma D.4.
By Lemma B.4, we have t ÿ
[Equation]
It then follows by the choice ω “ CT {p θλ r k ?
m
q , η “ r O pp mθ 2 λ r k q ´ 1 q , θ “ r O p (cid:15) q and the assumption m ě r O p poly p T, λ ´ 1 r k , {+(cid:15) ´ 1 qq that } W p t ` 1 q l ´ W p 
+}
q l } F ď ω , l “ 1 , 2 .
Therefore by induction, we see that with probability at least 1 ´ O p T 3 n 2 q¨ exp r´ Ω p mω 2 { 3 qs , W p 0 q , . . .
, W p T q P B p W p 0 q , ω q .
Applying Lemma B.4 then gives
[Equation]
Now by ω “ CT {p λ r k ? m q , η “ r O p θ 2 m q ´ 1 and the assumption that m ě m ˚ “ r O p λ ´ 14 r k ¨ (cid:15) ´
6 q , we obtain
[Equation]
By Lemma 3.1, θ “ r O p (cid:15) q and the assumptions n ě r Ω p max t
[Equation]
´ 1 p λ r k ´ λ
r k ` 1 q ´ 1 , (cid:15) ´
2 M 2 r 2 k uq , the eigenvalues of V J r k V r k are all between 1 {?
and ? 2 .
Therefore by Lemma B.2 we have
[Equation]
Similarly,
[Equation]
By Lemma 3.1, with probability at least 1 ´ δ , } V J r k } 2 ď 1 ` Cr k Ma log p r k { δ q{ n .
{+Combining this result with Lemma D.3 gives } V J r k p y p 
+}
q } 2 ď θ O p a n
[Equation]
Plugging the above estimates into (B.4) gives
[Equation]
Applying union bounds completes the proof.
B. P ROOF OF THE T HEOREM 3.
Proof of the Theorem 3.4.
The idea of the proof is close to that of Proposition 5 in (Bietti and Mairal, 2019) where they consider k " d
and we present a more general case including k " d and d " k .
For any function g : S d Ñ R , by denoting g 0 p x q “ ş S d g p y q dτ d p y q , it can be decomposed as
[Equation]
where we project function g to spherical harmonics in the second equality and apply the addition equation in the last equality.
For a positive-deﬁnite dot-product kernel κ p x , x 1 q : S d ˆ S d Ñ R which has the form κ p x , x 1 q “ p κ px x , x 1 yq for p κ : r´ 1 , 1 s Ñ R , we can present a decomposition by (B.5) if we consider g p x q “ φ px x , z yq for z P S d and φ : r´ 1 , 1 s Ñ R ,
[Equation]
x 1 q
[Equation]
where we apply the Hecke-Funk formula and addition formula.
By denoting λ k “ p ω [Equation] ´
[Equation]
d q ş
[Equation]
p
[Equation]
q P
[Equation]
p
[Equation]
qp
[Equation]
q{ 2 dt and the addition formula, we have
[Equation]
This formula (B.6) is the Mercer decomposition for the kernel function κ p x , x 1 q and µ k is exactly the eigenvalue of the integral operator L K on L 2 p S d q deﬁned by
[Equation]
By using same technique as κ p x , x 1 q , we can derive a similar expression for σ px w , x yq “ max tx w , x y , 0 u and σ 1 px w , x yq “ 1 tx w , x y ą 0 u , since they are essentially dot-product function on L 2 p S d q .
We deliver the expression below without presenting proofs.
[Equation]
where β 1 ,k “ p ω d ´ 1 { ω d q ş´ 1 σ p t q P k p t qp 1 ´ t 2 q p d ´ 2 q{ 2 dt and β 2 ,k “ p ω d ´ 1 { ω d q ş´ 1 σ 1 p t q P k p t qp 1 ´ t 2 q p d ´ 2 q{ 2 dt .
We add more comments on the values of β 1 ,k and β 2 ,k .
It has been pointed out in Bach (2017) that when k ą α and when k and α have same parity, we have β α ` 1 ,k “ 0 .
This is because the Legendre polynomial P k p t q is orthogonal to any other polynomials of degree less than k with respect to the density function p p t q “ p 1 ´ t 2 q p d ´ 2 q{ 2 .
Then we clearly know that β 1 ,k “ 0 for k “ 2 j and β 2 ,k “ 0 for k “ 2 j ` 1 with j P N ` .
For two kernel function deﬁned in (2.2), we have
[Equation]
The ﬁrst equality holds because σ 1 is 0-homogeneous function and the second equality is true since the normalized direction of a multivariate Gaussian random variable satisﬁes uniform distribution on the unit sphere.
Similarly we can derive
[Equation]
By combing (A.2), (B.7), (B.8), (B.9) and (B.10), we can get
[Equation]
and
[Equation]
Comparing (B.6), (B.11) and (B.12), we can easily show that
[Equation]
In Bach (2017)
[Equation]
explicit expressions for β 1 ,k and β 2 ,k for k ě α ` 1 are presented by
[Equation]
q . By Stirling formula Γ p x q « x x ´ 1 { 2 e ´ x ? 2 π , we have following expression of β α ` 1 ,k for k ě α `β
[Equation]
k ´ α ´ 1p k ` d q ´ k ´ d ´ α¯ where C p α q {+“ ? 2 α ! 
+}
π exp t α ` 1 u .
Also β α ` 1 ,“ d ´ 1 4 π Γ p α ` 12 q Γ p d 2 q Γ p d ` α ` q , β 1 ,“ d ´ 1 2 dπ and β 2 ,“ d ´ 1 4 πd Γ p 12 q Γ p d ` 22 q Γ p d
[Equation]
32 q . Thus combine (B.13) we know that µ α ` 1 ,k “ Ω
[Equation]
By considering (A.3) and (B.6), we have
[Equation]
and µ
[Equation]
for k ě 1 and k ‰ k 1 .
From the discussion above, we thus know exactly that for k ěµ k
[Equation]
This ﬁnishes the proof.
B. P ROOF OF THE C OROLLARIES 3.7 AND 3.
Proof of the Corollaries 3.7 and 3.8. We only need to bound | φ j p x q| for j P r r k s to ﬁnish the proof.
Since now we assume input data follows uniform distribution on the unit sphere S d , φ j p x q would be spherical harmonics of order at most k for j P r r k s .
For any spherical harmonics Y k of order k
and any point on S d , we have an upper bound (Proposition 4.16 in Frye and Efthimiou (2012))
[Equation]
Thus we know that | φ j p x q| ď a
N p d, k q .
For k " d , we have N p d, k q “ 2 k ` d ´ 1 k ` k ` d ´ 2 d ´ 1 ˘ “
O p k d ´ 1 q .
For d " k , we have N p d, k q “ 2 k ` d ´ 1 k ` k ` d ´ 2 d ´ 1 ˘ “
O p d k q .
This completes the proof.
C P ROOF OF T ECHNICAL L EMMAS C. P ROOF OF L EMMA B.
Proof of Lemma B.2.
The ﬁrst inequality directly follows by equation (44) in Su and Yang (2019) [Equation]
To prove the second bound [Equation] we write V r k “ p V r k A ` p V K r k B [Equation]
where A P R r k ˆ r k , B P R p n ´ r k qˆ r k .
Let ξ 1 “ C 1 p λ r k ´ λ
r k ` 1 q ´ 1 ¨ a log p 1 { δ q{ n , ξ 2 “ C 3 Ma log p r k { δ q{ n be the bounds given in the ﬁrst inequality and Lemma [Equation]
By the ﬁrst inequality [Equation] we have
[Equation]
Moreover
[Equation]
since V J r k V r k “ A J A ` B J B [Equation] by Lemma 3.1 we have
[Equation]
Therefore
[Equation]
Plugging in the deﬁnition of ξ 1 and ξ 2 completes the proof.
C. P ROOF OF L EMMA B.
Proof of Lemma B.3.
The gradient descent update formula gives
[Equation]
For any j P r n s , subtracting W p t q and applying inner product with θ ∇ W f W p t q p x j q on both sides gives
[Equation]
Further rearranging terms then gives
[Equation]
where
I
[Equation]
For I 1 ,j,t , by Lemma D. [Equation] we have
[Equation]
with probability at least 1 ´ O p n q ¨ exp r´ Ω p mω 2 { 3 qs . For I 2 ,j,t , by Bernstein inequality and union bound, with probability at least 1 ´ O p n 2 q ¨ exp p´ Ω p mω 2 { 3 qq , we have
[Equation]
for all i, j P r n s .
Therefore
[Equation]
I
[Equation]
For I 3 ,j,t , we have
[Equation]
where the ﬁrst inequality follows by Lemmas D.2, the second inequality is obtained from (C.1), and the third inequality follows by Lemma D.4.
Setting the j -th entry of e p t q as I 1 ,j,t ` I 2 ,j,t ` I 3 ,j,t and writing (C.2) into matrix form completes the proof.
C. P ROOF OF L EMMA B.
Proof of Lemma B.4.
Denote u p t q “ y ´ p y
p
t q , t P T .
Then we have
[Equation]
where the ﬁrst inequality follows by Lemma B.
[Equation]
and the second inequality follows by Lemma D. [Equation]
Therefore we have
[Equation]
for t 1 “ 0 , . . .
, t .
This completes the proof of (B.1).
Similarly, we have
[Equation]
for t
[Equation]
. , t ´ 1 , where the third inequality is by Lemma B.1 and the assumption that ω ď r
O p λ 3 r k q , n ě r O p λ ´ 2 r k q
[Equation]
and the fourth inequality is by (B.1).
Therefore we have
[Equation]
This completes the proof of (B.2).
Finally, for (B.3), by assumption we have ω 1 { 3 ηmθ 2 ď r O p 1 q .
Therefore
[Equation]
for t 1 “ 0 , . . .
, t ´ 1 , where the third inequality is by Lemma B.1 and the assumption that ω ď r
O p λ 3 r k q [Equation] and the fourth inequality follows by (B.1).
Therefore we have
[Equation]
This ﬁnishes the proof.
D A UXILIARY L EMMAS
In this section we list several auxiliary lemmas on the properties of over-parameterized neural networks we need in our proof of Theorem 3.2.
These results are mostly summarized from Allen-Zhu et al.
(2018) and Cao and Gu (2019a).
D. A UXILIARY L EMMAS
Denote
[Equation]
Lemma D.1 (Allen-Zhu et al.
If ω ď O pr log p m qs ´ 3 { 2 q , then with probability at least 1 ´ O p n q ¨ exp
[Equation]
for all W P B p W p 0 q , ω q , i P r n s .
Lemma D.2 (Cao and Gu (2019a)) .
There exists an absolute constant κ such that, with probability at least 1 ´ O p n q ¨ exp r´ Ω p mω 2 { 3 qs over the randomness of W p 1 q , for all i P r n s and W , W 1 P
B p W p 0 q , ω q with ω ď κ r log p m qs ´ 3 { 2 , it holds uniformly that
[Equation]
Lemma D.3 (Cao and Gu (2019a)) .
For any δ ą 0 , if m ě C log p n { δ q for a large enough absolute constant C , then with probability at least 1 ´ δ , | f W p
q p x i q| ď O p a log p n { δ qq for all i P r n s .
Lemma D.4 (Cao and Gu (2019a)) .
There exists an absolute constant C such that, with probability at least 1 ´ O p n q ¨ exp r´ Ω p mω 2 { 3 qs , for all i P r n s , l P r L s and W P B p W p 0 q , ω q with ω ď
C r log p m qs ´ 3 , it holds uniformly that
[Equation]
The following lemma provides a uniform bound of the neural network function value over
B p W p 0 q , ω q .
Lemma D.5.
Suppose that m ě Ω p ω ´ 2 { 3 log p n { δ qq and ω ď O pr log p m qs ´ 3 q .
Then with probability at least 1 ´ δ , | f W p x i q| ď O p a log p n { δ q ` ω ?
m q for all W P B p W p 0 q ,
ω q i P r n s .
Lemma D.6.
If ω ď O pr log p m qs ´ 3 { 2 q , then with probability at least 1 ´ O p n q ¨ exp r´ Ω p mω 2 { 3 qs , }
[Equation]
|
[Equation]
∇ W f W p
x
i q , ∇ W f W p x j qy ´ x ∇ W f W {+p 
+}
q p x i q , ∇ W f W p {+
+}
q p x j
qy| ď O p ω 1 { 3 m q for all W P B p W p 0 q , ω q
and i P r n s .
D. P ROOFS OF L EMMAS D.5 AND D.
Proof of Lemma D.5.
By Lemmas D.2 and D.4, we have
[Equation]
where the last inequality is by the assumption ω ď r log p m qs ´ 3 .
Applying triangle inequality and
Lemma D [Equation] 3 then gives
[Equation]
This completes the proof.
Proof of Lemma D.6.
By direct calculation, we have
[Equation]
Therefore we have
[Equation]
By Lemma 7.4 in Allen-Zhu et al.
(2018) and Lemma D.1, with probability at least 1 ´ n ¨ exp r´ Ω p m qs , ? m ¨ } [Equation] p
[Equation]
q [Equation] p
[Equation]
p [Equation] q [Equation]
´ D i q} F ď O p ω 1 { 3 ? m q for all i P r n s .
Moreover, clearly {+}p W p 0 q 2 ´ W 2 q D i } F ď } W p 
 q ´ W 2 } F ď ω .+}
Therefore } ∇ W
[Equation]
p x i
[Equation]
for all i P r n s .
This proves the bound for the ﬁrst layer gradients.
For the second layer gradients, we have
[Equation]
It therefore follows by the 1 -Lipschitz continuity of σ p¨q that
[Equation]
This completes the proof of the ﬁrst inequality.
The second inequality directly follows by triangle inequality and Lemma D.4:
[Equation]
This ﬁnishes the proof.
E A DDITIONAL E XPERIMENTAL R ESULTS E. E STIMATING THE PROJECTION LENGTH IN FUNCTION SPACE
As mentioned in Section 4.1, when using freshly sampled points, we are actually estimating the projection length of residual function r p x q “ f ˚ p x q ´ θf W p t q p x q onto the given Gegenbauer polynomial P k p x q .
Here we present in Figure 3 a comparison between the projection length using training data and that using test data.
An interesting phenomenon is that the network generalizes well on the lower-order Gegenbauer polynomial and along the highest-order Gegenbauer polynomial the network suffers overﬁtting
[Figure]
low-order Gegenbauer polynomials, the network learns the function while for the high-order Gegenbauer polynomial, the network overﬁts the training data.
E. N EAR - LINEAR CONVERGENCE BEHAVIOR
In this subsection, we present the same curve shown in Section 4.1 in logarithmic scale instead of linear scale.
As shown in Figure 4 we can see that the convergence of different projection length is close to linear convergence, which is well-aligned with our theorem.
Note that we performed a moving average of range 20 on these curves to avoid the heavy oscillation especially in late stage.
[Figure]