T HE
T HREE
S TAGES
OF L EARNING
D YNAMICS
IN
H IGH DIMENSIONAL K ERNEL M ETHODS
Anonymous authors
Paper under double-blind review
A BSTRACT
To understand how deep learning works, it is crucial to understand the training dynamics of neural networks.
Several interesting hypotheses about these dynamics have been made based on empirically observed phenomena, but there exists a limited theoretical understanding of when and why such phenomena occur.
In this paper, we consider the training dynamics of gradient ﬂow on kernel leastsquares objectives, which is a limiting dynamics of SGD trained neural networks.
Using precise high-dimensional asymptotics, we characterize the dynamics of the ﬁtted model in two “worlds”: in the Oracle World the model is trained on the population distribution and in the Empirical World the model is trained on a sampled dataset.
We show that under mild conditions on the kernel and L 2 target regression function the training dynamics undergo three stages characterized by the behaviors of the models in the two worlds.
Our theoretical results also mathematically formalize some interesting deep learning phenomena.
Speciﬁcally, in our setting we show that SGD progressively learns more complex functions and that there is a “deep bootstrap” phenomenon: during the second stage, the test error of both worlds remain close despite the empirical training error being much smaller.
Finally, we give a concrete example comparing the dynamics of two different kernels which shows that faster training is not necessary for better generalization.
I NTRODUCTION
In order to fundamentally understand how and why deep learning works, there has been much effort to understand the dynamics of neural networks trained by gradient descent based algorithms.
This effort has led to the discovery of many intriguing empirical phenomena (e.g. Frankle et al.
Fort et al.
; Nakkiran et al.
(2019a;b; 2020))
that help shape our conceptual framework for understanding the learning process in neural networks.
Nakkiran et al.
(2019b) provides evidence that
SGD starts by ﬁrst learning a linear classiﬁer and over time learns increasingly complex functions.
Nakkiran et al.
(2020) introduces the “deep bootstrap” phenomenon: for some deep learning tasks the empirical world test error remains close to the oracle world error 1 for many SGD iterations, even if the empirical training and test errors display a large gap.
To better understand such phenomena, it is useful to study training dynamics in related but mathematically tractable settings.
One approach for theoretical investigation is to study kernel methods, which were recently shown to have a tight connection with over-parameterized neural networks (Jacot et al., 2018; Du et al., 2018).
Indeed, consider a sequence of neural networks ( f N ( x ; θ )) N ∈ N with the widths of the layers going to inﬁnity as N → ∞ . Assuming proper parametrization and initialization, for large N the SGD dynamics on f N is known to be well approximated by the corresponding dynamics on the ﬁrst-order Taylor expansion of f N around its initialization θ 0 ,
[Equation]
Thus, in the large width limit it sufﬁces to study the dynamics on the linearization f N, lin . When using the squared loss, these dynamics correspond to optimizing a kernel least-squares objective with the neural tangent kernel K
[Equation]
[Figure]
Over the past few years, researchers have used kernel machines as a tractable model to investigate many neural network phenomena including benign overﬁtting, i.e., generalization despite the interpolation of noisy data (Bartlett et al., 2020; Liang & Rakhlin, 2020) and double-descent, i.e, risk curves that are not classically U-shaped (Belkin et al., 2020; Liu et al., 2021).
Kernels have also been studied to better understand certain aspects of neural network architectures such as invariance and stability (Bietti & Mairal, 2017; Mei et al., 2021b).
Although kernel methods cannot be used to explain some phenomena such as feature learning, they can still be conceptually useful for understanding other neural networks properties.
1. T HREE STAGES OF KERNEL DYNAMICS
Despite much classical work in the study of gradient descent training of kernel machines (e.g. Yao et al.
; Raskutti et al.
there has been limited work understanding the high-dimensional setting, which is the setting of interest in this paper.
Although solving the linear dynamics of gradient ﬂow is simple, the statistical analysis of the ﬁtted model requires involved random matrix theory arguments.
In our analysis we study the dynamics of the Oracle World , where training is done on the (usually inaccessible) population risk, and the Empirical World , where training is done on the empirical risk (as is done in practice).
Associated with the oracle world model f or t and the empirical world model ˆ f t are the following quantities of interest: the empirical training error (cid:98) R n ( ˆ f t ) , the empirical test error R ( ˆ f t ) , and the oracle error R ( f or t ) deﬁned in Eqs.
(1), (2), (3) for which we derive expressions that are accurate in high dimensions.
Informally, our main results show that under reasonable conditions on the regression function and the kernel the training dynamics undergo the following three stages:
• Stage one : the empirical training error, the empirical test error, and the oracle error are all close.
• Stage two : the empirical training error decays to zero, but the empirical test error and the oracle error stay close and keep approximately constant.
• Stage three : the empirical training error is still zero, the empirical test error stays approximately constant, but the oracle test error decays to the approximation error.
We conceptually illustrate the error curves of the oracle and empirical world in Fig. 1 and provide intuition for the evolution of the learned models in Fig.
The existence of the ﬁrst and third stages are not unexpected: at the beginning of training the model has not ﬁt the dataset enough to distinguish the oracle and empirical world and at the end of training an expressive enough model with inﬁnite samples will outperform one with ﬁnitely many.
The most interesting stage is the second one where the empirical model begins to “overﬁt”
the training set while still remaining close to the non-interpolating oracle model in the L 2 sense (see Fig. 2).
In Section 2 we discuss some related work.
In Section 3 we elaborate our description of the three stages and give a mathematical characterization for two particular settings in Theorems 1 and 2. [Figure]
Although the three stages arise fairly generally, we remark that certain stages will vanish if the problem parameters are chosen in a special way (c.f. Remark 1).
We connect our theoretical results to related empirical deep learning phenomena in Remark 3 and discuss the relation to deep learning in practice in Remark 4.
In Section 4 we provide numerical simulations to illustrate the theory more concretely and in Section 5 we end with a summary and discussion of the results.
R ELATED L ITERATURE
The generalization error of the kernel ridge regression (KRR) solution has been well-studied in both the ﬁxed dimension regime (Wainwright, 2019, Chap. 13), (Caponnetto & De Vito, 2007) and the high-dimensional regime (El Karoui, 2010; Liang & Rakhlin, 2020; Liu et al., 2021; Ghorbani et al., 2020; 2021; Mei et al., 2021a;b).
Most closely related to our results is the setting of (Ghorbani et al., 2021; Mei et al., 2021a;b).
Analysis of the entire KRR training trajectory has also been done (Yao et al., 2007; Raskutti et al., 2014; Cao et al., 2019) but only for the ﬁxed dimensional setting.
Classical non-parametric rates are often obtained by specifying a strong regularity assumption on the target function (e.g. the source condition in Fischer & Steinwart (2020)), whereas in our work the assumption on the target function is mild.
Another line of work directly studies the dynamics of learning in linear neural networks (Saxe et al., 2013; Li et al., 2018; Arora et al., 2019; Vaskevicius et al., 2019).
Similar to us, these works show that some notion of complexity (typically effective rank or sparsity) increases in the linear network over the course of optimization.
The relationship between the speed of iterative optimization and gap between population and empirical quantities has been studied before in the context of algorithmic stability (Bousquet & Elisseeff, 2002; Hardt et al., 2016; Chen et al., 2018).
These analyses certify good empirical generalization by using stability in the ﬁrst few iterations to upper bound the gap between train and test error.
In contrast, our analysis directly computes the errors at an arbitrary time t (c.f. Remark 2).
The relationship between oracle and empirical training dynamics has been considered before in Bottou &
LeCun (2004) and Pillaud-Vivien et al.
R ESULTS
In this section we introduce the problem and present a specialization of our results to two concrete settings: dot product and group invariant kernels on the sphere (Theorems 1 and 2 respectively).
The more general version of our results is described in Appendix A.3.
3. P ROBLEM SETUP
We consider the supervised learning problem where we are given i.i.d. data ( x i , y i ) i ≤ n .
The covariate vectors ( x i ) i ≤ n ∼ iid Unif( S d − 1 ( √ d )) and the real-valued noisy responses y i =
f d ( x i ) + ε i for some unknown target function f d ∈ L 2 ( S d − 1 ( √ d )) and ( ε i ) i ≤ n ∼ iid N (0 , σ 2 ε ) .
Given a function f ∈ L 2 ( S d − 1 ( √ d )) , we deﬁne its test error R ( f ) and its training error (cid:98) R n ( f ) as
[Equation]
where ( x new , y new ) is i.i.d. with ( x i , y i ) i ≤ n .
The test error R ( f ) measures the ﬁt of f on the population distribution and the training error (cid:98) R n ( f ) measures the ﬁt of f to the training set.
For a kernel function H d : S d − 1 ( √ d ) × S d − 1 ( √ d ) → R , we analyse the dynamics of the following two ﬁtted models indexed by time t : the oracle model f or t and the empirical model ˆ f t , which are given by the gradient ﬂow on R and (cid:98) R n over the associated RKHS H d respectively
[Equation]
with zero initialization f or 0 ≡ ˆ f 0 ≡ 0 .
These dynamics are motivated from the neural tangent kernel perspective of over-parameterized neural networks (Jacot et al., 2018; Du et al., 2018).
A precise mathematical deﬁnition and derivation of these two dynamics are provided in Appendix E.1.
For our results we make some assumptions on the spectral properties of the kernels H d similar to those in Mei et al.
(2021a) that are discussed in detail in Section A.2.
At a high-level we require that the diagonal elements of the kernel concentrate, that the kernel eigenvalues obey certain spectral gap conditions, and that the top eigenfunctions obey a hyperconctractivity condition which says they are “delocalized”.
For the speciﬁc settings of Theorems 1 and 2 we give more speciﬁc conditions on the kernels that are more easily veriﬁed and imply the required spectral properties.
3. D OT P RODUCT K ERNELS
In our ﬁrst example, we consider dot product kernels H d of the form
[Equation]
for some function h d : [ − 1 , 1] → R .
Our results apply to general dot product kernels under weak conditions on h d given in Appendix C.2.
In particular they apply to the random feature and neural tangent kernels associated to certain fully connected neural networks (Jacot et al., 2018).
Before presenting our results for this setting we introduce some notation.
Denote by P ≤ (cid:96) the orthogonal projection onto the subspace of L 2 ( S d − 1 ( √ d )) spanned by polynomials of degree less than or equal to (cid:96) .
The projectors P (cid:96) and P >(cid:96) are deﬁned analogously (see Appendix G for details).
We use o d ( · ) for standard little-o relations, where the subscript d emphasizes the asymptotic variable. The statement f ( d ) = ω
d ( g ( d )) is equivalent to g ( d ) =
o d ( f ( d )) . We use o d, P ( · ) in probability relations.
Namely for two sequences of random variables Z 1 ( d ) and Z 2 ( d ) , Z 1 ( d )
= o d, P ( Z 2 ( d )) if for any ε, C ε > 0 there exists d ε ∈ Z > 0 , such that P ( | Z 1 ( d ) /Z
( d ) | < C ε ) ≤ ε for all d ≥ d ε .
The asymptotic notations O d , O d, P etc. are deﬁned analogously.
Theorem 1 (Dot Product Kernels) .
Let { f d ∈
L
( S d − 1 ( √ d )) }
d ≥ 1 be a sequence of functions such that for some η > 0 , (cid:107) f d (cid:107) L 2+ η = O d (1) , and let { H d } d ≥ 1 be a sequence of dot product kernels satisfying Assumption 4.
Assume that for some ﬁxed integers j , s ≥ 0 and some δ > 0 that [Equation]
Then we have the following characterizations,
(a) (Oracle World)
The oracle model learns every degree component of f d as time progresses
[Equation]
(b) (Empirical World – Train) Empirical training error follows oracle error then goes to zero
[Equation]
[Figure] [Figure]
[Figure]
the empirical model learns
the
degree- s component of
f d
[Equation]
The results are conceptually illustrated in an example in Fig. 3a which shows the stair-case phenomenon in high-dimensions and the three learning stages.
We see that in both the oracle world and the empirical world, the prediction model increases in complexity over time.
More precisely, the model learns the best polynomial ﬁt to the target function (in an L 2 sense) of increasingly higher degree.
In the empirical world the maximum complexity is determined by the sample size n , which is in contrast to the oracle world where there are effectively inﬁnite samples.
The results imply that generally (but not always
c.f. Remark 1) there will be three stages of learning.
In the ﬁrst stage the oracle and empirical world models are close in L 2 and ﬁt a polynomial with degree determined by t .
The ﬁrst stage lasts from t = 0 to t = nd − ε (cid:28) n for some small ε > 0 .
As t approaches n , there is a phase transition and the empirical world training error goes to zero at t = nd ε (cid:29) n .
From time nd − ε till at least d s +1 is the second stage where the empirical and oracle models remain close in L 2 but the gap between test and train error can be large.
If the sample size n is not large enough for ˆ f t to learn the target function, then at some large enough t we will enter a third stage where f or t improves in performance, outperforming ˆ f t which remains the same.
On synthetic data in ﬁnite dimensions we can see a resemblance of the staircase shape which becomes sharper with increasing d (c.f. Appendix F).
Remark 1 (Degenerate stages of Learning) .
For special problem parameters we will not observe the second and/or third stages.
The second stage will disappear if n (cid:16) d q for some q ∈ N (see Fig.
3b), or if f d is a degree- s polynomial.
The third stage will not occur if P > s f d lies in the orthogonal complement of the RKHS H d .
3. G ROUP I NVARIANT K ERNELS
We now consider our second setting which concerns the invariant function estimation problem introduced in Mei et al.
(2021b).
As before, we are given i.i.d. data ( x i , y i ) i ≤ n where the feature vectors ( x i ) i ≤ n ∼ iid Unif( S d − 1 ( √ d )) and noisy responses y i =
f d ( x i ) + ε i .
We now assume that the target function f
d satisﬁes an invariant property.
We consider a general type of invariance, deﬁned by a group G d that is represented as a subgroup of the orthogonal group in d dimensions.
The group element g ∈ G d acts on a vector x ∈ R d via x (cid:55)→ g · x .
We say that f (cid:63) is G d -invariant if f (cid:63) ( x ) =
f (cid:63) ( g · x ) for all g ∈ G d .
We denote the space of square integrable G d -invariant functions by L 2 ( S d − 1 ( √ d ) , G d ) .
We focus on groups G d that are groups of degeneracy α as deﬁned below.
As an example, we consider the cyclic group
Cyc d = { g 0 , g 1 , . . .
, g d − 1 } where for any x = ( x 1 , . . .
, x d ) T ∈ S
d − 1 ( √ d ) , the group action is deﬁned by g i · x = ( x i +1 , x i +2 , . . .
, x d , x 1 , x 2 , . . .
, x i ) T .
The cyclic group has degeneracy 1 .
Deﬁnition 1 (Groups of degeneracy α ) .
Let V d,k be the subspace of degree-k polynomials that are orthogonal to polynomials of degree at most ( k − 1) in L 2 ( S d − 1 ( √ d )) , and denote by V d,
k ( G d ) the subspace of V d,k formed by polynomials that are G d -invariant.
We say that G d has degeneracy α if for any integer k ≥ α we have dim( V d,k /V d,k
( G d )) (cid:16) d α (i.e., there exists 0 < c k ≤ C k < + ∞ such that c k ≤ dim( V d,k /V d, k ( G d )) ≤ C k for any d ≥ 2 ).
To encode invariance in our kernel we consider G d -invariant kernels H d of the form
[Equation]
where π d is the Haar measuare on G d .
Such kernels satisfy the following invariance property: for all g, g (cid:48) ∈ G d and for H d ( x 1 , x 2 ) =
H d ( g · x 1 , g (cid:48) · x 2 ) for every x 1 , x 2 .
For the cyclic group, π d is the uniform measure.
We now present our results for the group invariant setting.
Theorem 2 (Group Invariant Kernels) .
Let G d be a group of degeneracy α ≤ 1 according to Definition 1.
Let { f d ∈
L ( S d − 1 ( √ d ) , G d ) } d ≥ 1 a sequence of G d -invariant functions such that for some η > 0 , (cid:107) f d (cid:107) L 2+ η = O d (1) , and let { H d } d ≥ 1 be a sequence of G d -invariant kernels satisfying Assumption 5.
Assume that for some ﬁxed integers j ≥ 0 , s ≥ 1 and some δ > 0 that
[Equation]
Then we have the following characterizations,
(a) (Oracle World)
The oracle model learns every degree component of f d as time progresses
[Equation]
(b) (Empirical World – Train) Empirical training error follows oracle error then goes to zero
[Equation]
(c) (Empirical World – Test) Empirical test error follows oracle error until the empirical model learns the degree- s component of f d
[Equation]
With respect to the dot product kernel setting (c.f. Theorem 1), in this setting the behavior of the oracle world is unchanged, but the empirical world behaves as if it has d α times as many samples.
This is illustrated graphically in Fig. 3c.
It can be shown that using an invariant kernel is equivalent to using a dot product kernel and augmenting the dataset to { ( g · x i , y i ) : g ∈ G d , i ∈ [ n ] } (c.f. Appendix E.2).
Hence for the cyclic group which has size d α = d , we arrive at the following intriguing conclusion: if the target function is invariant, then using a dot product kernel and augmenting n i.i.d samples to nd many samples is asymptotically equivalent to using nd i.i.d samples.
Remark 2 (Optimization Speed versus Generalization) .
Interestingly, training with an invariant kernel is slower than with a dot product kernel and takes longer to interpolate the dataset despite eventually generalizing better on invariant function estimation tasks (c.f. Fig. 3c).
This conclusion is not an artifact of the continuous time analysis (c.f. Appendix E.3) and is observed empirically in Section 4.2 for discrete-time SGD.
This example highlights the limitation of stability based analyses (e.g. Hardt et al.
(2016)) which argue that faster SGD training leads to better generalization.
While a faster rate leads to better generalization in the ﬁrst stage when stability can control the gap between train and test error, our analysis shows that the duration length of the ﬁrst stage also impacts the ﬁnal generalization error.
Remark 3 (Connection with Deep Phenomena) .
The dynamics of high-dimensional kernel regression display behaviors that parallel some empirically observed phenomena in deep learning.
For kernel regression, we have shown that the complexity of the empirical model, measured as the number of learned eigenfunctions, depends on the time optimized when t (cid:28) n and the sample size when t (cid:29) n .
At a high-level, we also expect a similar story for neural networks but for some other notion of complexity.
It is believed that neural networks ﬁrst learn simple functions and then progressively more complex ones, until the complexity saturates after interpolating at some time proportional to n (Nakkiran et al., 2019b).
We have also shown that in kernel regression there is a non-trivial “deep boostrap” phenomenon (Nakkiran et al., 2020) during the second learning stage: the gap between the oracle world and empirical world test errors is negligible whereas the train and test errors exhibit a substantial gap.
The gradient ﬂow results for kernel regression can also provide insight into the deep bootstrap for random feature networks SGD as these results can approximately predict their behavior (see Section 4.2).
Remark 4 (Connection with Deep Learning Practice) .
Although we believe our results conceptually shed light on some of the interesting behaviors observed in the training dynamics of deep learning, due to our stylized setting we may not exactly see the predicted phenomena in practice.
Accurately observing the three stages of kernel regression requires sufﬁciently high-dimensional data in order for the kernel eigenvalues to obey a staircase-like decay and for training to be sufﬁciently long as the time axis should be in log-scale.
Our results hold for regression whereas for classiﬁcation the empirical model may continue improving after classifying the train set correctly.
Despite these caveats, certain conclusions can be observed in some realistic settings (c.f. Appendix E.4).
N UMERICAL S IMULATIONS
As mentioned previously, Fig. 3 is a “cartoon” of the conclusions stated in Theorems 1 and 2.
In this section, we verify the qualitative predictions of our theorems using synthetic data.
Concretely, throughout this section we take d = 400 and n = d 1 . 5 = 8000 , and following our theoretical setup (c.f. Section 3.1) generate covariates ( x i ) i ≤ n ∼ iid Unif( S d − 1 ( √ d )) and responses y i =
f (cid:63) ( x i )
+ ε i with ( ε i ) i ≤ n ∼ iid N (0 , σ 2 ε ) , for different choices of target function f (cid:63) .
All simulations in this section are for the noiseless case σ 2 ε =
but a noisy example is given in Appendix F.
In Section 4.1, we simulate the gradient ﬂows of kernel least-squares with dot product kernels and cyclic kernels (Fig. 4) to reproduce the three stages as shown in Fig.
In Section 4.2 we show that SGD of (dot product and cyclic) random-feature models (Fig. 5) exhibit similar three stages phenomena, in which the second stage behaviors are consistent with the deep bootstrap phenomena observed in deep learning experiments (Nakkiran et al., 2020).
Empirical quantities are averaged over 10 trials and the shaded regions indicate one standard deviation from the mean.
4. G RADIENT F LOW OF K ERNEL L EAST -S QUARES
Under the synthetic data set-up mentioned earlier, we ﬁrst simulate the oracle world and empirical world errors curves of gradient ﬂow dynamics using dot product kernels of the form
[Equation]
) for some activation function σ .
We will examine a few different choices of f (cid:63) and kernel H , which are speciﬁed in the descriptions of each ﬁgure.
The oracle world error is computed analytically but the empirical world errors require sampling train and test datasets.
We compute empirical world errors by averaging over 10 trials.
The results are visualized both in log-scale and linear-scale on the time axis.
The log-scale plots allow for direct comparison with the cartoons in Fig.
The linear-scale plots are zoomed into the region 0 < t ≤ nd 0 .
4 since: 1) plotting the full interval squeeze all curves to the left boundary which is uninformative 2) in practice one would not optimize for very long after interpolation.
In Figs. 4a and 4b we take the target function to be a polynomial of the form
[Equation]
where He i ( t ) is the i th Hermite polynomial (c.f. Appendix G.4) and the approximate equality in Eq. (7) holds in high-dimensions.
In panel (4a) we consider use the ReLU activation function σ ( t ) =
max( t, 0) , and take f (cid:63) to be a quadratic polynomial with ( a 0 , a 1 , a 2 ) =
(1 / 2 , 1 / √ 2 , 1 / √ 8) .
With
[Figure]
such a choice of parameters, we can see the three stages phenomenon.
In panel (4b) we choose f (cid:63) to be a cubic polynomial with ( a 0 , a 1 , a 2 , a 3 ) =
(1 / 2 , 1 / √ 2 , 0 , 1 / √ 24) and σ ( t ) = max( t, 0) + 0 .
He 3 ( t ) (we need the third Hermite coefﬁcient of σ to be non-zero for stage 3 to occur).
This choice of coefﬁcients for f (cid:63) is such that (cid:107) P > 1 f (cid:63) (cid:107)
L 2 ≈ (cid:107) P > 2 f (cid:63) (cid:107) 2 L 2 , so that the second stage in (4b) is longer compared to (4a).
In Fig. 4c, we take the target function to be a cubic cyclic polynomial [Equation] i
[Equation]
x
[Equation]
x [Equation] (cid:33) (8) where the subindex addition in x i + k is understood to be taken modulo d .
We compare the performance of the dot product kernel H and its invariant version H inv (c.f. Eq.
(5)) with activation function σ ( t ) =
max( t, 0) + 0 .
He 3 ( t ) .
The kernel H inv with n samples performs equivalently to
H with nd samples (c.f. Remark 2), but is more computationally efﬁcient since the size of the kernel matrix is still n × n .
Using H inv elongates the ﬁrst stage by a factor d , delaying the later stages and ensuring that the empirical world model improves longer.
Although in the simulations, the dimension d is not yet high enough to see a totally sharp staircase phenomenon as in the illustrations of Fig. 3, even for this d we are still able to clearly see the three predicted learning stages and deep bootstrap phenomenon across a range of settings.
To better understand the effect of dimension we show similar plots with varying d in Appendix F.
4. SGD FOR T WO - LAYER R ANDOM -F EATURE M ODELS
To more closely relate with deep learning practice and the deep bootstrap phenomenon (Nakkiran et al., 2020), we simulate the error curves of SGD training on random-feature (RF) models (i.e. twolayer networks with random ﬁrst-layer weights and trainable second-layer weights), in the same synthetic data setup as before.
In particular, we look at dot product RF models
[Equation]
and cyclic invariant RF models
[Equation]
[Figure]
For all following experiments we take the activation σ to be ReLU and N = 4 × 10 5 ≈ n
For a given data distribution and RF model we train two ﬁtted functions, one on a ﬁnite dataset (empirical world) and the other on the data distribution (oracle world).
More speciﬁcally, the training of the empirical model is done using multi-pass SGD on a ﬁnite training set of size n with learning rate η = 0 .
1 and batch size b = 50 .
The training of the oracle model is done using one-pass SGD with the same learning rate η and batch size b , but at each iteration a fresh batch is sampled from the population distribution.
Both models ˆ f t , f or t are initialized with a i = 0 for i ∈
[ N ] .
To speed up and stabilize optimization we use momentum β = 0 .
Note that if we took N → ∞ , η → 0 , and β =
we would be exactly in the dot product kernel gradient ﬂow setting.
In Fig. 5, the data generating distributions of panels (5a), (5b), (5c) are respectively the same as that of panels (4a), (4b), and (4c) from Section 4.1.
The top row of Fig. 5 shows SGD for { dot product, cyclic } RF models, and the bottom row shows the corresponding gradient ﬂow for { dot product, cyclic } kernel least-squares.
We see that the corresponding curves in these two rows exhibit qualitatively the same behaviors.
Additionally, the results in panel (5c) show that as predicted, even for discrete SGD dynamics the dot product RF optimizes faster but fails to generalize, whereas the invariant RF optimizes slower but generalizes better.
S UMMARY AND D ISCUSSION
In this paper, we used precise asymptotics to study the oracle world and empirical world dynamics of gradient ﬂow on kernel least-squares objectives for high-dimensional regression problems.
Under reasonable conditions on the target function and kernel, we showed that in this setting there are three learning stages based on the behaviors of the empirical and oracle models and also connected our results to some empirical deep learning phenomena.
Although our setting already captures some interesting aspects of deep learning training dynamics, there are some limitations which would be interesting to resolve in future work.
We require very high-dimensional data in order for the asymptotics to be accurate, but real data distributions have low-dimensional structure.
We work in a limiting regime of neural network training where the dynamics are linear and the step-size is inﬁnitesimal.
It is an important direction to extend this analysis to the non-linear feature learning regime and to consider discrete step-size minibatch SGD, as these are considered important aspects of network training.
Our results hold for the square-loss in regression problems, but many deep learning problems involve classiﬁcation using cross-entropy loss.
Lastly, our analysis holds speciﬁcally for gradient ﬂow, so it would be also interesting to consider other iterative learning algorithms such as boosting.
A G ENERAL S ETTING
In this section we present our theory for training dynamics of kernel regression in an abstract setting similar to that of Mei et al.
(2021a).
We ﬁrst introduce the setting of interest, then state the relevant assumptions, and ﬁnally we provide our theoretical results.
We provide proofs of these results in
Appendix B.
A. P ROBLEM S ETUP
Consider a sequence of Polish probability spaces ( X d , ν d ) , where ν d is a probability measure on the conﬁguration space X d , indexed by an integer d .
We denote by L 2 ( X d ) =
L 2 ( X d , ν d ) the space of square integrable functions on ( X d , ν d ) .
For p ≥ 1 , we denote (cid:107) f (cid:107) L p ( X d ) =
E x ∼ ν d [ | f ( x ) p | ] 1 /p the L p norm of f .
Let D d
⊆
L 2 ( X d ) be a closed linear subspace.
In some simple applications we will consider D d
= L 2 ( X d ) , but the extra generality will be useful in certain applications.
We are concerned with a supervised learning problem where we are given i.i.d data ( y i , x i ) i ≤ n .
The feature vectors x i ∼ iid ν
d are in X d and the empirical-valued noisy responses y i are given by y i =
f d ( x i ) + ε i for some unknown target function f d ∈ D d and ε i ∼ iid N (0 , σ 2 ε ) .
We consider a general RKHS deﬁned on ( X d , ν d ) via the compact self-adjoint positive deﬁnite operator H d : D d → D d which admits the representation
[Equation]
where H d ∈ L 2 ( X d × X d ) with the property that (cid:82) X
d H
[Equation]
( x
[Equation]
x
[Equation]
( x [Equation] ν d {+(d x (cid:48) ) = 
+}
for g ∈ D ⊥ d . By the spectral theorem of compact operators, there exists an orthonormal basis ( ψ j ) j ≥ 1 , span( ψ j , j ≥ 1) =
D d ⊆
L 2 ( X d ) and empirical eigenvalues ( λ d,j ) j ≥ 1 with nonincreasing absolute values | λ d, 1 | ≥ | λ d, 2 | ≥ · · · and (cid:80) j ≥ 1 λ 2 d,j < ∞ such that
[Equation]
where convergence holds in L 2 ( X d × X d ) .
For S ⊆ { 1 , 2 , . . .
} we denote P S to be the projection operator from L 2 ( X d ) onto D d,S :=
span( ψ j , j ∈ S ) .
We denote H d,S to be the operator
[Equation]
and H d,S the corresponding kernel.
If S = { j ∈ N : j ≤ (cid:96) } we will write as short-hand H d, ≤ (cid:96) and analogously for S
= { j ∈ N : j > (cid:96) } .
The trace of this operator is given by
[Equation]
d [ H d
[Equation]
S ( x , x )] < ∞ .
Deﬁne the test error R :
L 2 ( X d )
→ R and training error (cid:98) R n : L 2 ( X d ) → R
R (
[Equation]
[Equation]
new
[Equation]
y new ) { ( y new − f ( x new )) 2 } , (cid:98) R n ( f ) ≡ 1 n n (cid:88)
i =
( y i − f ( x i ))
(9) where ( x 1 , y 1 ) , . . .
, ( x n , y n ) , ( x new , y new ) are iid.
For a kernel H d ∈ L 2 ( X d ×X d ) we will consider the oracle model f or t and the empirical model ˆ f t which satisfy the following gradient ﬂows
[Equation]
A. G ENERAL A SSUMPTIONS
We now state our assumptions on the kernel and the sequence of probability spaces ( X d , ν d ) .
Assumption 1 ( { n ( d ) , m ( d ) } d ≥ 1 -Kernel Concentration Property) .
We say that the sequence of operators { H d } d ≥ 1 satisﬁes the Kernel Concentration Property (KCP) with respect to the sequence { n ( d ) , m ( d ) } d ≥ 1 if there exists a sequence of integers { r ( d ) } d ≥ 1 with r ( d ) ≥ m ( d ) , such that the following conditions hold.
(a) (Hypercontractivity of ﬁnite eigenspaces.)
For any ﬁxed q ≥ 1 , there exists a constant C such that for any h ∈ D d, ≤ r ( d ) = span( ψ s , 1 ≤ s ≤ r ( d )) , we have
[Equation]
(b) (Properly decaying eigenvalues)
There exists ﬁxed δ 0 > 0 , such that, for all d large enough, n
[Equation]
(c) (Concentration of diagonal elements of kernel)
For ( x i ) i ∈ [ n ( d )] ∼ iid ν d , we have: max
[Equation]
Assumption 1(a) can be interpreted as requiring that the top eigenfunctions of H d are delocalized.
Assumption 1(b) concerns the tail of eigenvalues of H d and is a mild assumption in high-dimensions.
Lastly, 1(c) essentially requires that “most points” in X d behave similarly in the sense of having similar values of the kernel diagonal H d ( x , x ) .
Assumption 2 (Eigenvalue condition at level { ( n ( d ) , m ( d )) } d ≥ 1 ) .
We say that the sequence of kernel operators { H d } d ≥ 1 satisﬁes the Eigenvalue Condition at level { ( n ( d ) , m ( d )) } d ≥ 1 if the following conditions hold for all d large enough
(a) There exists a ﬁxed δ 0 > 0 , such that
[Equation]
(b) There exists a ﬁxed δ 0 > 0 , such that
[Equation]
(c) There exists a ﬁxed δ 0 > 0 , such that
[Equation]
Assumptions 2(a) and 2(b) can be seen as a spectral gap assumption.
This ensures a clear separation between the eigenvalues in the subspace D d, ≤ m ( d ) and the subspace D d,> m ( d ) .
The technical requirement 2(c) is mild.
In the asymptotic setting, we will be interested in understanding the model learned at a time t
= t ( d ) which scales with the dimension.
The next set of assumptions give requirements for a valid scaling.
Assumption 3 (Admissible Time at { t ( d ) , u ( d ) , n ( d ) , m ( d ) } d ≥ 1 ) .
We say that the sequence { t ( d ) , u ( d ) , n ( d ) , m ( d ) } d ≥ 1 is an Admissible Time for the sequence of kernel operators { H d } d ≥ 1 if the following conditions hold
(a) There exists a ﬁxed δ 0 > 0 , such that for d large enough
[Equation]
(b) If u ( d ) < m ( d ) for inﬁnitely many d , then
[Equation]
(c) There exists a constant C such that
[Equation] Assumption 3(a) is similar to the spectral gap condition Assumption 2(a), 2(b) for ( t ( d ) , u ( d ))
d ≥ 1 .
Assumption 3(b) relates the ordering of the indices u ( d ) , m ( d ) to the relative growth of t ( d ) , n ( d ) .
Assumption 3(c) requires that the eigenvalue tail does not decay too abruptly.
A. M AIN R ESULTS
In this section we give the main theoretical results.
Recall the problem set-up and notation from
Appendix A.1. We will characterize the gradient ﬂow dynamics dynamics of the oracle model f or t Eq. (10) and the empirical model ˆ f t Eq.
(11) for a general kernel H d .
Theorem 3 (Oracle World) .
Let { f d ∈ D d } d ≥ 1 be a sequence of functions and { H d } d ≥ 1 be a sequence of kernel operators such that { ( H d , t ( d ) , u ( d )) }
d ≥ 1 satisﬁes Assumption 3(a), then [Equation] or
[Equation]
where P ≤ u ( d ) and P > u ( d ) are the projection operators onto the subspace spanned by the top u ( d ) kernel eigenfunctions and the orthogonal complement respectively, as deﬁned in Appendix A.1.
The error of the oracle model is determined solely by optimization time t through u ( d ) .
Due to the spectral gap assumption 3(a) learning only occurs along the top u ( d ) eigenfunctions.
The next results describe the empirical model.
First we characterize the training error.
Theorem 4 (Empirical World - Train) .
Let { f d ∈ D d } d ≥ 1 be a sequence of functions, ( x i ) i ∈
[ n ( d )] ∼ ν d independently, and { H d } d ≥ 1 be a sequence of kernel operators such that { ( H d , n ( d ) , m ( d ) , t ( d ) , u ( d )) }
d ≥ 1 satisﬁes { ( n ( d ) , m ( d )) }
d ≥ 1 -KPCP
(Assumption 1), eigenvalue condition at level { ( n ( d ) , m ( d )) }
d ≥ 1 (Assumption 2), and { ( n ( d ) , m ( d ) , t ( d ) , u ( d )) } d ≥ 1 is a valid time (Assumption 3).
Deﬁne κ H =
Tr (cid:0) H d,> m ( d ) (cid:1) and (cid:96) ( d ) = min { u ( d ) , m ( d ) } .
Then for any η > 0 we have,
[Equation]
R n ( ˆ f t ) =
(cid:13)(cid:13) P >(cid:96) ( d ) f
d (cid:13)(cid:13) 2 L 2 + σ 2 ε + o d, P (1) · ( (cid:107) f
{+d (cid:107) 
+}
L 2+ η + σ 2 ε ) , if t = o d ( n/κ H ) , (cid:98) R n ( ˆ f t )
=
o d, P (1) · ( (cid:107) f
{+d (cid:107) 
 L 2+ η + σ 2 ε ) , if t = ω d ( n/κ H ) .+} In the early-time regime t (cid:28) n/κ H the training error may be non-zero and matches the oracle world error if also u ( d ) ≤ m ( d ) .
In the late-time regime t (cid:29) n/κ H the training error is negligible and the model interpolates the training set.
The quantity κ H arises since the empirical kernel matrix can be decomposed as H =
H ≤ m + H > m and the second component is approximately a multiple of the identity: H > m ≈
Tr( H > m ) · I n = κ H · I n .
This term acts as a self-induced ridge-regularizer.
Our ﬁnal result characterizes the test error of the empirical model.
Theorem 5 (Empirical World - Test) .
Let { f d ∈ D d } d ≥ 1 be a sequence of functions, ( x i ) i ∈
[ n ( d )] ∼ ν d independently, and { H d } d ≥ 1 be a sequence of kernel operators such that { ( H d , n ( d ) , m ( d ) , t ( d ) , u ( d )) }
d ≥ 1 satisﬁes { ( n ( d ) , m ( d )) }
d ≥ 1 -KPCP
(Assumption 1), eigenvalue condition at level { ( n ( d ) , m ( d )) }
d ≥ 1 (Assumption 2), and { ( n ( d ) , m ( d ) , t ( d ) , u ( d )) } d ≥ 1 is a valid time (Assumption 3).
Deﬁne (cid:96) ( d ) = min { u ( d ) , m ( d ) } .
Then for any η > 0 we have,
[Equation]
The result shows that the empirical model is essentially the projection of the regression function onto the ﬁrst (cid:96) ( d ) eigenfunctions.
The quantity (cid:96) ( d ) controls the complexity of the model which increases with time t up until u ( d ) ≥ m ( d ) after which the complexity is limited by n .
B P ROOF OF G ENERAL S ETTING B. O RACLE W ORLD - P ROOF OF T HEOREM
The oracle model ODE Eq.
(10) with initialization f or0 ≡ 0 can be solved (c.f. Appendix E.1) to yield the solution
[Equation]
Therefore the excess risk is given by
[Equation]
where λ 2 d
[Equation]
k are the kernel eigenvalues and ˆ f k :
=
(cid:104) f d , ψ k (cid:105) L 2 are the Fourier coefﬁcients of f d in the kernel eigenbasis (c
[Equation]
f. Appendix A.1).
We can control this quantity as follows,
[Equation]
where the last inequality follows from Assumption 3(a).
This shows the ﬁrst theorem statement,
[Equation]
or
[Equation]
Now observe that
[Equation]
hence
[Equation]
where the second inequality follows from the fact that (1 − e − x ) 2 ≤ 1 − e − 2 x and the ﬁnal equality is from the proof of the ﬁrst part of the theorem.
Thus,
[Equation]
completing the proof
[Equation]
B. E MPIRICAL W ORLD – P RELIMINARIES
We will introduce some useful notations and the objects of analysis for studying the empirical world.
B.2. T RAINING D YNAMICS
For the training of the empirical world c.f. Eq. (11), if u ( t ) =
( ˆ f t ( x 1 ) , . . .
, ˆ f t ( x n ))
∈ R n then from Eq.
(44) if u (0) {+= 
+}
then
[Equation]
14) where H ∈ R n × n with the ( i, j ) th element given by H ij =
H d ( x i , x j ) and y = f
+ ε ∈ R n with f = ( f d ( x 1 ) , . .
, f d ( x n ))
T ∈ R n and ε = ( ε 1 , . . . , ε n ) T ∈ R n . For x ∈ R d , deﬁne h ( x ) =
( H d ( x 1 , x ) , . . .
, H d ( x n , x )) T ∈ R n .
The training and test errors as deﬁned in Eq. (1) can be written as
[Equation]
Expanding R ( ˆ f t ) yields
[Equation]
where E =
( E 1 , . . .
E n ) T ∈ R n , M = ( M ij ) i,j ∈ [ n ] ∈ R n × n , and H =
( H ij ) i,j ∈ [ n ] ∈ R n × n with
[Equation]
B.2. D ECOMPOSITIONS AND N OTATIONS
In this section we recall some useful decompositions of empirical quantities from Mei et al.
(2021a).
As mentioned earlier the eigendecomposition of H d is given by
[Equation]
We write the orthogonal decomposition of f d in the basis { ψ k } k ≥ 1 as
[Equation]
Deﬁne
[Equation]
We have the following orthogonal basis decompositions of f , H , E and M
[Equation]
By Lemma 6 below
[Equation]
under Assumptions 1 and 2(a) the matrices H and M can be written as
[Equation]
[Equation]
where
[Equation]
and
[Equation]
We will use α as shorthand for the scalar valued dimension dependent quantity e − ( t/n ) κ H and take
[Equation]
[Equation]
We also introduce the shrinkage matrix deﬁned as
[Equation]
where s j = λ 2 d,j [Equation] d [Equation]
κ H n [Equation] If unspeciﬁed we will typically use ∆ , ∆ (cid:48) , etc. to denote matrices with operator norm o d, P (1) . For positive integers (cid:96) < m , deﬁne [ (cid:96), m ] = { (cid:96) + 1 , . . .
, m } .
We will use the following notation
[Equation]
[Equation]
for s j deﬁned in Eq.
[Equation]
B.2. A UXILIARY L EMMAS
Here we collect some lemmas which will be of use to us.
Lemma 1 (Matrix Exponential Perturbation Inequality) .
For matrix operator norm (cid:107) · (cid:107) , if A , B ∈ R n × n are symmetric then
[Equation]
For general square matrices A , B ∈ R n × n we have
[Equation]
Lemma 2.
Let Z ∈ R m × n , Ψ ∈ R m × p , D ∈ R p × p and t ∈ R .
Denote A = Z T Ψ ∈ R n × p and B = Ψ T Ψ ∈ R p × p .
Then,
[Equation]
The notations in Lemmas 3-8 all follow the notations given in Section B.2.2.
Lemma 3 (Lemma 12 from Mei et al.
(2021a) with λ = 0 ) . Let Assumptions 1 and 2 hold. Then, [Equation]
= o d
[Equation]
P [Equation] [Equation] Lemma 4 (Theorem 6(b) from Mei et al.
(2021a)) .
Let Assumptions 1(a), 2(c) hold.
Then,
[Equation]
Ψ
[Equation]
/n
[Equation]
I m (cid:13)(cid:13) op =
o d
[Equation]
P (1) . Lemma 5 (Lemma 13 from Mei et al. (2021a) with λ = 0 ) . Let Assumptions 1 and 2 hold.
Then,
[Equation]
Lemma 6 (Theorem 6 from Mei et al. (2021a)) . Let Assumptions 1 and 2(a) hold.
Then we can decompose the kernel matrices as follows
[Equation]
where
[Equation]
and [Equation] Lemma 7. Let Assumption 1(a) hold.
Let S, T be disjoint subsets of N .
Let D ∈ R | S |×| S | be a diagonal matrix.
Then we have that for any η > 0 , there exists C ( η ) (independent of d ), such that
[Equation]
where the expectation is with respect to the randomness in Ψ S , Ψ T .
Proof.
Let ι : S →
[ | S | ] be the bijection such that Ψ S = ( ψ ι − 1 ( k ) ( x i )) i ∈ [ n ] ,k ∈ [ | S | ] then we have E [
[Equation]
where the second to last inequality is by Holder’s inequality and the last inequality used the hypercontractivity assumption as in Assumption 1(a).
Lemma 8 (Exponential Kernel Decomposition) .
Assume the conditions of Lemma 6 hold and assume there exists δ 0 > 0 , such that ( t ( d ) , u ( d )) satisﬁes the condition
[Equation]
Let j ( d ) satisfy min { u ( d )
[Equation]
m ( d ) } ≤ j ( d ) ≤ m ( d ) then
[Equation]
Proof.
First consider the regime t =
O d ( n/κ H ) .
Recall the decomposition,
[Equation]
where (cid:107) ∆ H (cid:107) op
=
o d, P (1) .
By Lemma 1 and Lemma 4,
[Equation]
[Table]
[Equation]
[Table]
[Equation] where equality ( a ) holds by assumption Eq.
Now if t = ω d ( n/κ H ) , then it is easy to see that [Equation]
[Equation]
and therefore
[Equation]
B. E MPIRICAL W ORLD - T RAIN
If t = ω d ( n/κ H ) , then by Eq. (17) it is easy to see that (cid:13)(cid:13)
e − 2( t/n ) H (cid:13)(cid:13) op
=
o d, P (1) , hence (cid:98) R n ( ˆ f t ) =
o d, P (1) · (cid:107) f
d (cid:107) 2 L 2 .
From now on we focus on the case that t = o d ( n/κ H ) .
Let us decompose the training error as follows
[Equation]
where
[Equation]
B.3. T ERM R
Let us start by analysing R 1 .
We can write
[Equation]
where
[Equation]
Let us analyse the term T 1 ,
[Equation]
By Lemma 2, denoting B = Ψ T ≤ (cid:96) Ψ ≤ (cid:96) /n , [Equation] We will show that [Equation]
[Equation]
By Lemma 4, since (cid:107) B (cid:107) op =
Θ d, P (1) , for d large B has a positive-deﬁnite square root B 1 / 2 w.h.p. Therefore we can write [Equation]
and bound the operator norm
[Equation]
since by Assumption 3(a), tλ min ( D 2 ≤ (cid:96) ) = Ω( t δ 0 ) . Therefore T 1 = o d, P (1) · (cid:107) P ≤
[Equation]
f d
[Equation]
L 2 . Now we analyse the term T 3 .
Observe that by the inequality 1 − x ≤ e − x ≤ 1 ,
[Equation]
We have by Lemma 7,
[Equation]
where the last quantity is o d, P (1)
[Equation]
(cid:107) P >(cid:96) f
d {+(cid:107) 
+}
L 2+ η by Assumption 3(c).
Thus we see that
[Equation]
Observe that by the Cauchy-Schwarz inequality
[Equation]
Putting everything together we see that,
[Equation]
B.3. T ERM R
Turning to R 2 , we take the second-moment with respect to ε
[Equation]
By Markov’s inequality R 2 =
o d, P (1) · (cid:107)
f d (cid:107)
[Equation]
B.3. T ERM R
Now let us analyse R 3 .
Recalling the deﬁnition B = Ψ T ≤ (cid:96) Ψ ≤ (cid:96) /n
[Equation]
we can compute the expectation
[Equation]
Equality ( a ) follows from Lemma 8.
For ( b ) we used the inequality 1 − e − x ≤ x and that trace is the sum of eigenvalues.
Equality ( c ) uses the cyclic property of trace. In the last equality we used that by Lemma 4, (cid:107) B (cid:107) op
= O d, P (1) and Assumption 3(c) implies that ( t/n ) Tr (cid:0) D 2 ≤ (cid:96) (cid:1) = O d (( t/n ) κ H ) = o d (1) since by assumption we consider t = o d ( n/κ H ) .
Turning to the variance
[Equation]
where the ﬁrst inequality uses that (cid:13)(cid:13)
e − 2( t/n ) H (cid:13)(cid:13) op ≤ 1 and ( a ) uses that for ε i ∼ N (0 , σ 2 ε ) , E [ ε 4 i ] = 3 σ 4 ε .
Therefore by Chebyshev’s inequality, R 3 = σ 2 ε (1 + o d, P (1)) .
Putting everything together yields
[Equation]
B. E MPIRICAL W ORLD - T EST
Recalling u ( t ) from Eq. (14), let
[Equation]
where
[Equation]
Recall the expansion of the test error from Eq.
[Equation]
where
[Equation]
The proof for the test error is the most involved, but will follow a similar strategy of analysing each term in the expansion.
We ﬁrst begin by analysing T 2 in Section B.4.1, then T 1 in Section B.4.2, and ﬁnally terms T 3 , T 4 , T 5 , which are all simpler than the ﬁrst two, in Section B.4.3.
At the end of this Appendix section we present the proof of Theorem 5.
B.4. T ERM T
As before, we will analyze each term separately.
We begin with term T 2 .
Proposition 1 (Term T 2 ) .
[Equation]
where we recall the shrinkage matrix S ≤ m deﬁned [Table]
Proof of Proposition 1.
To analyze T 2 we further decompose it into the following terms
[Equation]
where
[Equation]
Using Lemma 9 and 10 proven below [Equation] by the Cauchy-Schwarz inequality [Equation]
and hence
[Equation]
Lemma 9 (Term T 21 Eq.
[Equation]
Proof of Lemma 9.
Recall the notation α and K ≤ (cid:96) from Eq. Deﬁne ∆ such that
[Equation]
which by Lemma 8 satisﬁes (cid:107) ∆ (cid:107) op =
o d, P (1) . Then we can split T 21 into T 21 =
[Equation]
211 +
[Equation] 212 + [Equation] 213 + Twhere
[Equation]
We will show that the dominant term is T 211 and the others are of lower order.
By Lemma 3,
[Equation]
and since K ≤ (cid:96) (cid:22) I n , T 213 = o [Equation] , P
[Equation]
[Equation]
f d (cid:107) 2 L 2 .
By Lemma 3, Lemma 4 and since S ≤ m (cid:22) I m [-and Lemma 14 it is easy to see that T 
-], [Equation] hence [Table]
o [Table] (cid:107) [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] where the ﬁrst inequality used S 2 (cid:96) m (cid:22) I m − (cid:96) , we used Lemma 2 in ( a ) , and the last equality used
Lemma 4 (note that Ψ T ≤ (cid:96) Ψ (cid:96) m /n is an off-diagonal block of Ψ T ≤ m Ψ ≤ m /n
which corresponds to an all zeroes submatrix of I m ).
Finally we look at the main term T 211 .
Deﬁning the matrices
[Equation]
we can write
[Equation]
Now observe that by Lemma 2,
[Equation]
and by the same argument used to show Eq.
(22), (cid:107) A (cid:107) op
=
o d, P (1) .
Since B = I n + ∆ (cid:48) and
S ≤ (cid:96) (cid:22) I (cid:96) , we have
[Equation]
hence combining terms
[Equation]
Lemma 10 (Term T 23 Eq.
[Equation]
Proof of Lemma 10.
Let us deﬁne the matrix
[Equation]
Using similar reasoning from Lemma 9 we can write
[Equation]
for matrices ∆ [Equation] ∆ (cid:48) satisfying max {(cid:107) ∆ (cid:107) op , (cid:107) ∆
[Equation]
(cid:107) op } =
o d, P (1) .
We can split T 23 into
[Equation]
where
[Equation]
Let us ﬁrst start with analysing T 231 , deﬁning B = Ψ T ≤ m Ψ ≤ m /n we have
[Equation]
where the ﬁrst equality is by Lemma 2 and the last line follows from Lemma 7, Markov’s inequality, and by the fact that m /n
=
o d (1) by Assumption 2(c).
To see that inequality ( a ) holds note that
[Equation]
Since S 2 ≤ m (cid:22) I m , [Equation] where the last equality is by Lemma 4. Now let us turn to T 233 , which we can further split into
[Equation]
28) where
[Equation]
Redeﬁning B = Ψ T ≤ (cid:96) Ψ ≤ (cid:96) /n
and using a similar argument as for T 231 , the ﬁrst term can be seen as
[Equation]
Turning to the second term T
[Equation]
let
[Equation]
then we can write this term as
[Equation]
where the last inequality holds since A (cid:22) Ψ T (cid:96) m Ψ (cid:96) m /n = I m − (cid:96) + ∆ by Lemma 4 and so
[Equation]
By the inequality 1 − e − x ≤ x
[Equation]
in the PSD order we see that
[Equation]
Therefore from Eq. (30) we have
[Equation]
For the second term on the right
[Equation]
by Assumption 3(b) and by Lemma 4,
[Equation]
[Equation]
therefore by Markov’s inequality
[Equation]
Hence combining terms
[Equation]
By the Cauchy-Schwarz inequality
[Equation]
η . Putting everything together we see that
[Equation]
B.4. T ERM T
Now we will analyse term T 1 .
Proposition 2 (Term T 1 ) .
[Equation]
Proof of Proposition 2.
We break T 1 into the following terms
[Equation]
where
[Equation]
Using Lemma 11 and recalling Lemma 10, [Equation] where T 23 is as given in Eq. From the analysis of T 11 in Lemma 12 and T 13 in Lemma 13 we combine everything to get Proposition
[Equation]
Lemma 11.
For a vector v ∈ R n ,
[Equation]
Proof of Lemma 11.
Recall that E ≤ m = Ψ
[Equation]
By the Cauchy-Schwarz inequality,
[Equation]
where the last equality follows from Lemma 3. Lemma 12 (Term T 11 Eq. [Equation]
Proof of Lemma 12. First note that by Lemma 8 for some ∆ such that (cid:107) ∆ (cid:107) op
=
o d, P (1) ,
[Equation]
11 =
[Equation]
[Equation]
Twhere
[Equation]
By Lemma 11,
[Equation]
We now consider
T
= T 1111 − Twhere
[Equation]
Note that by Lemma 5,
[Equation]
where 0 is a (cid:96) × ( m − (cid:96) ) matrix of zeros.
Hence for the ﬁrst term T 1111 ,
[Equation]
Deﬁne H ≤ (cid:96) : = Ψ ≤
[Equation] D 2 ≤ [Equation] Ψ T ≤ (cid:96) .
For the second term T 1112 , by Lemma
[Equation]
34) where
[Equation]
Deﬁne A := 1 n Ψ T ≤
[Equation]
e −
[Equation]
/n )
H ≤
[Equation]
Ψ ≤
[Equation]
We have
[Equation]
[Equation]
since as noted before in Eq. (22), (cid:107) A (cid:107) op
=
o d, P (1) .
Therefore
[Equation]
Lemma 13 (Term T 13 Eq.
[Equation]
Proof of Lemma 13. We have
[Equation]
Note that we have E [ (cid:107) f (cid:107) 22 ] = n (cid:107) f d (cid:107) 2 L 2 .
Further by Eq. (17), we have (cid:13)(cid:13) H − 1 (cid:13)(cid:13) op ≤ 2 /κ H with high probability.
Finally, recalling the deﬁnition of E > m from Eq. (16), we have [Equation] As a result, we have
[Equation]
where the last equality used Eq.
(13) in Assumption 2(a).
B.4. T ERMS T 3 , T 4 , T
To analyse the terms T 3 , T 4 , T 5 we can adapt the corresponding steps for the proof of Theorem 4 in Mei et al.
(2021a).
For the following analysis we recall the deﬁnition of ε ( t ) from Eq. Lemma 14 (Term T 3 ) . [Equation] Proof of Lemma 14.
[Equation] where ( a ) used Lemma 3, ( b ) used S ≤ m (cid:22) I m , and ( c ) used Lemma 4 and Assumption 2(c). The lemma then follows from Markov’s inequality. Lemma 15 (Term T 4 ) .
[Equation]
Proof of Lemma
[Equation] [Equation] Notice that M (cid:23) Ψ ≤ L D 4 ≤ L Ψ T ≤ L for any L ∈ N , by the decomposition of Eq. Therefore
[Equation]
where the last inequality follows from Lemma 3. Hence,
[Equation]
where ( a ) follows from the deﬁnition of E ≤ L , ( b ) follows from the deﬁnition of operator norm, and ( c ) follows from Eq.
Therefore we get
[Equation]
Lemma 16 (Term T 5 ) . [Equation] Proof of Lemma 16. We can write term T 5 as
[Equation]
where
[Equation]
Note as in Eq. (35), that by Lemma 3 and Lemma 4,
[Equation]
and taking the second moment of T 51 yields
[Equation]
where T 21 is as given in Eq.
Similarly we get that
[Equation]
where T 23 is as given in Eq.
By Markov’s inequality we deduce that
[Equation]
Proof of Theorem
[Equation]
By Assumption 2(b), κ H /n
=
o d (1) · [Equation] j ≤ [Equation]
λ
[Equation]
hence
[Equation]
and as a result we obtain the ﬁrst part of the theorem
[Equation]
Now observe that similar to Eq.
(15) we have the following decomposition [Equation] where u ( t ) is given in Eq. Therefore we can write
[Equation]
We now focus on the term
[Equation]
By choosing L = (cid:96) in the proof of Lemma
[Equation] it follows that [Equation] hence combining with the bound for T 4 in Lemma 15 yields
[Equation]
We will now show that
[Equation]
If (cid:96) ( d ) = m ( d ) , then Eq.
(38) follows from Lemma 13.
Otherwise, consider the case (cid:96) ( d ) =
u ( d ) .
Following similar logic to the proof of Lemma 13, because E [ (cid:107) f (cid:107) 22 ] = n (cid:107) f d (cid:107) 2 L 2 and
[Equation]
we also get Eq.
(38) since
[Equation]
where ( a ) used the inequality (1 − e − x ) /x ≤ 1 and ( b ) used Assumption 3(a).
Therefore u ( t ) T H − 1 E >(cid:96) =
o d, P (1) · ( σ 2 ε + (cid:107) f
d (cid:107) 2 L 2 ) , hence by Eq.
(36) and Eq.
(37) we obtain the ﬁnal part of the theorem
[Equation]
C D OT P RODUCT K ERNELS ON S d − 1 ( √ d ) C. S ETTING
We now apply our general theorems to the setting of dot product kernels on the sphere.
Concretely we take X d = S d − 1 ( √ d ) and ν d = Unif( S d − 1 ( √ d )) and consider dot product kernels H d which take the form of Eq.
Note that by Eq.
(61) any dot product kernel h d can be decomposed as
[Equation]
[Equation]
[Equation]
for some activation function σ d .
We state mild assumptions on σ d and show that under these conditions we can apply the results in Section A.3.
C. A SSUMPTIONS
We state our assumptions on σ d after some deﬁnitions.
See Appendix G for additional background.
Denote by P ≤ (cid:96) the orthogonal projection onto the subspace of L 2 ( S d − 1 ( √ d )) spanned by polynomials of degree less than or equal to (cid:96) .
The projectors P (cid:96) and P >(cid:96) are deﬁned analogously.
Let us emphasize that the projectors P ≤ (cid:96) are related but distinct from the P ≤ m : while P ≤ (cid:96) projects onto the eigenspace of polynomials of degree at most (cid:96) , P ≤ m projects onto the top m -eigenfunctions.
The assumptions given on the activations are the same as Assumption 3 of Mei et al.
(2021a).
Assumption 4 (Assumptions for Dot Product Kernels at level s ∈ N ) .
Let { H d } d ≥ 1 be a sequence of dot product kernels with associated activation functions { σ d } d ≥ 1 as in Eq.
We assume the following hold (a) There exists an integer k and constants c 1 < 1 and c 0 > 0 , such that | σ d ( x ) | ≤ c 0 exp (cid:0) c 1 x 2 / (4 k ) (cid:1) . (b) We have [Equation]
where e ∈ S d − 1 is a ﬁxed vector (it is easy to see that these quantities do not depend on e ).
Consider t ( d ) , n ( d ) such that [Equation] for some j , s ∈ N and δ 0 > 0 . We now verify that if { σ d } d ≥ 1 satisﬁes Assumption 4 at level s , then for an appropriate choice of ( u ( d ) , m ( d )) the conditions in Appendix A.2 are satisﬁed and lead to
Theorem 1.
We set u ( d ) and m ( d ) to be the number of eigenvalues associated to spherical harmonics of degree less than or equal to j and s respectively
[Equation]
The veriﬁcation of Assumption 1 (Kernel Concentration Property) and Assumption 2 (Eigenvalue
Condition) at level { ( n ( d ) , m ( d ) } is the same as the treatment in Theorem 2 of Mei et al.
(2021a). We only need to verify Assumption 3. To see part 3(a), note that 1 /λ 2 d, u ( d ) = Θ d ( d j ) and 1 /λ 2 d, u ( d ) =
Θ d ( d j +1 ) .
For part 3(b), the condition holds because u ( d ) < m ( d ) for large d if and only if j < s .
Assumption 3(c) is easily seen to hold since the trace of the kernel operator
Tr( H d ) =
Θ d (1) .
D G ROUP I NVARIANT K ERNELS ON S d − 1 ( √ d ) D. S ETTING
We now apply our general theorems to the setting of group invariant kernels on the sphere.
Concretely we take X d = S d − 1 ( √ d ) and ν d = Unif( S d − 1 ( √ d )) and consider kernels H d which take the form of Eq.
(5) for some function h .
By Eq.
(61), for some activation function σ d
[Equation]
We state mild assumptions on σ d and show that under these conditions we can apply the results in
Appendix A.3.
For additional technical background refer to Appendix G.
D. A SSUMPTIONS
We will assume that σ d = σ for all d and make the following assumptions on σ which are the same as Assumption 1 in Mei et al.
(2021b).
Assumption 5 (Assumption on Group Invariant Kernel at level s ) .
Let { H d } d ≥ 1 be a sequence of invariant kernels with associated activation functions σ d = σ as in Eq.
We assume the following conditions hold
(a) For G d = Cyc d , we assume σ to be ( s + 1) ∨ 3 differentiable and there exists constants c 0 > 0 and c 1 < 1 such that | σ ( k ) |
≤ c 0 e c 1 u 2 / 2 for any 2 ≤ k ≤ ( s + 1) ∨ 3 .
For general G d , we assume that σ is a (ﬁnite degree) polynomial function.
(b) The Hermite coefﬁcients µ k ( σ ) (c.f. Appendix) verify µ k (cid:54) = 0 for any 0 ≤ k ≤ s .
(c) We assume that σ is not a polynomial with degrees less than or equal to s .
Consider t ( d ) , n ( d ) such that
[Equation]
for some j , s ∈ N and δ 0 > 0 .
We now verify that if σ satisﬁes Assumption 5 at level s , then the conditions given in Appendix A.2 are satisﬁed for an appropriate choice of ( u ( d ) , m ( d )) which leads to Theorem 2.
We set u and m to be the number of eigenvalues invariant polynomials of degree less than or equal to j and s respectively
[Equation]
where D ( d, k ) is the dimension of the subspace of invariant polynomials of degree k ,
c.f. Appendix
G.6.
The veriﬁcation of Assumption 1 (Kernel Concentration Property) and Assumption 2 (Eigenvalue Condition) at level { ( n ( d ) , m ( d ) } is exactly the same as in Theorem 1 in Mei et al.
(2021b).
We must verify Assumption 3.
To see part 3(a), note that 1 /λ 2 d, u ( d ) =
Θ d ( d j ) and 1 /λ 2 d, u ( d ) =
Θ d ( d j +1 ) .
For part 3(b), the condition holds because u ( d ) < m ( d ) for large d if and only if j < s in which case t ( d )
[Equation]
Assumption 3(c) can be seen to hold from the fact that Tr (cid:0) H d,> m ( d ) (cid:1) =
Θ( d s − α ) and m (cid:88)
j
[Equation]
from which it follows that for some constant C
[Equation]
E A UXILIARY R ESULTS E. S OLUTION TO K ERNEL D YNAMICS
Recall that we are interested in the following dynamics given in Eqs.
[Equation]
with zero initialization f or0 ≡ ˆ f 0 ≡ 0 .
In this section we clarify the derivation, validity [Equation] and solution of these dynamics.
Let us consider the maps R : H d → R and (cid:98) R n : H d
→ R deﬁned in Eq. [Equation] First
[Equation]
we recall the deﬁnition of the Fr ´ echet derivative of a functional V : H d → R at f .
The Fr ´ echet derivative DV ( f ) is the linear functional such that for g ∈ H d ,
[Equation]
The gradient ∇ V ( f ) ∈ H d is deﬁned such that [Equation] exists uniquely by the Riesz representation theorem. The gradients of the risk functionals are
[Equation]
where
H x i ( x ) :
= H d ( x i , x ) and H d is the kernel operator as in Section A.1.
A proof of this fact is given in Proposition 2.1 of Yao et al.
Taking f or t ( x ) as shorthand for f or ( t, x ) where f or ( t, · ) ∈ H d is the oracle model at time t and similarly for ˆ f t ( x ) , the following gradient ﬂows with zero initialization are well-deﬁned for t ≥ 0 ,
[Equation]
The oracle model ODE Eq. (41) is simply a linear differential equation which has the following solution involving the operator exponential exp( A ) :
= ∞ (cid:80)
[Equation]
For the empirical model ODE Eq.
(42) we ﬁrst consider the system of scalar differential equations induced at the points { ( x i , y i ) } i ∈ [ n ] .
Letting u ( t ) =
( ˆ f t ( x 1 ) , . . .
, ˆ f t ( x n )) T , y = ( y 1 , . . . , y n ) T , and H
=
( H d ( x i , x j ))
i,j ∈ [ n ] we have
[Equation]
with initial condition u (0) = 0 .
As this a linear ODE, the solution is given by
[Equation]
For x ∈ R d , deﬁne h ( x ) =
( H d ( x , x 1 ) , . . . ,
H d ( x , x n ))
T ∈ R n .
Let a ( t ) =
H − 1 u ( t ) ∈ R n .
We will show that the function ˆ f t ( · ) :=
(cid:104) h ( · ) , a ( t ) (cid:
) ∈ H d , which satisﬁes ( ˆ f t ( x 1 ) , . . .
, ˆ f t ( x n )) T = u ( t ) , satisﬁes the following equation
[Equation]
which is Eq. (42) at point x .
Indeed, by the chain rule
[Equation]
which is what we wanted to show.
E. E QUIVALENCE BETWEEN I NVARIANT K ERNELS AND D ATA A UGMENTATION
In this section we will show an equivalence between the (time rescaled) gradient ﬂows for training invariant kernels and using an augmented dataset.
Speciﬁcally consider a group G and a kernel H that is G -equivariant, that is
[Equation]
Given a G -equivariant kernel H , we deﬁne a G -invariant kernel H inv as the group averaged kernel
[Equation]
for the Haar measure π on G (c.f. Eq.
Note that any dot product kernel is G -equivariant for G a subgroup the orthogonal group e.g. the cyclic group Cyc (c.f. Section 3.3).
Given a dataset ( X , y )
= { ( x i , y i ) : i ∈ [ n ] } consider the augmented dataset [Equation] We consider the (rescaled c.f.
Remark 5) empirical dynamics Eq.
(3) of the gradient ﬂow on ( X , y ) using H inv which we denote ˆ f t, inv [Equation] and the empirical dynamics of the gradient ﬂow on ( X G , y G ) using H which we denote ˆ f t, aug [Equation]
Proposition 3.
Let G be a ﬁnite group with m elements.
Given a G -equivariant kernel H , if π is the uniform measure on G then
[Equation]
Proof.
Let G
=
{ g 1 , . .
g m } where g 1 is the identity.
Deﬁne the output vectors
[Equation]
Furthermore, deﬁne the kernel matrices
[Equation]
Note that by deﬁnition H inv = 1 m m (cid:80)
[Equation]
We will show that
[Equation]
From this the result follows by Theorem 4.1 in Li et al.
(2019) since ˆ f t, inv , ˆ f t, aug are given by kernel regressions with targets u inv ( t ) , u aug ( t ) and kernels H inv , H respectively.
By Eq. (42), we can write
[Equation]
where y G =
( y , . . .
, y ) ∈ R mn .
By expanding the matrix exponential series and using linearity, it sufﬁces to show that
[Equation]
in order to show Eq.
(45) holds.
We prove the above by induction on k .
For k = 1 , observe that
[Equation]
where the second equality follows from G -equivariance of H .
Assume the inductive hypothesis holds for k .
Then
[Equation]
where the second equality applies the induction hypothesis and the third equality uses equivariance.
Thus the inductive claim is proved and the proof is complete.
Remark 5.
The scaling factor m in the gradient ﬂow for ˆ f t, inv , leads to a natural comparison with ˆ f t, aug as elaborated in Appendix E.3.
As argued in that section, in the gradient descent discretization, it is natural to take a step-size inversely proportional to the maximum kernel eigenvalue.
In the case of high-dimensional invariant kernels, note that
[Equation]
hence the step-size for the invariant kernel ﬂow should be m times larger.
E. D ISCRETIZING T IME
Comparing different “speeds” of optimization algorithms only makes sense for discrete-time algorithms.
Consider the following gradient descent dynamics with step-size η , obtained as the discretization of the empirical gradient ﬂow Eq.
[Equation]
[Equation] We will argue that it is natural to take η ∼ n/λ max ( H ) where H is the kernel matrix. Deﬁne the sampling operator S : H d → R n by S ( f ) = ( f ( x i ))
ni =1 ∈ R n and let S ∗ : R n → H d be its adjoint, deﬁned by S ∗ ( y )
=
1 n n (cid:80)
[Equation] (see Yao et al. Appendix B for more details). Then we can rewrite the gradient descent equation Eq.
(46) as
[Equation]
Let T =
S ∗ S and deﬁne H :=
SS ∗ = 1 n H to be the normalized kernel matrix.
Let b := S ∗ H − 1 y ∈ H d and note that since Tb = S ∗ y , we can rewrite Eq.
(47) as
[Equation]
Denote the eigenvalues of H as λ 1 ≥ λ 2 ≥ . . .
≥ λ n > 0 .
By the spectral theorem, there exists a set of orthornomal eigenvectors φ 1 , . .
. , φ n ∈ H d such that T = n (cid:80) i =
1 λ
i φ i φ ∗ i .
Deﬁne α
i ( k ) =
(cid:104)
ˆ f k − b, φ i (cid:105) ∈ R .
By taking Eq.
(48) then subtracting b and taking the inner product with φ i on both sides, we get the coordinate evolution equations for i = 1 , . . . , n
[Equation]
where the second equality holds since T is self-adjoint and the last equality is since φ i is an eigenvector of T . It is easy to see that α i ( k ) = (1 − ηλ i ) k
α i (0) .
Therefore we see that gradient descent Eq.
(46) is guaranteed to converge if η < 1 / (2 λ 1 ) and may not otherwise. Therefore it is natural to choose the step-size η to scale asymptotically as η ∼ 1 /λ max ( H ) .
For a dot product kernel H and its corresponding invariant kernel H inv the kernel matrices have operator norms of the same order
[Equation]
hence no time rescaling is need to compare the corresponding optimization speeds asymptotically.
E. S IMILARITIES WITH E MPIRICAL P HENOMENA
In this section we elaborate upon Remark 4 and mention some connections with empirical observations in Nakkiran et al.
Although the metric in our setting is the squared loss, we can still observe three stages in classiﬁcation problems when measuring the soft error.
In Fig.
6a taken from Nakkiran et al. we can observe stage 1 and stage 2.
Either training has not continued long enough to observe stage 3 or n is large enough so that the models have converged to the approximation error of the neural network class (c.f. Remark 1).
In Fig.
6b taken from Nakkiran et al. (2020), although the train errors are not plotted, by extrapolating from Fig.
6a, presumably for each n stage 1 and stage 2 occur. From the dimmer curves in Fig. 6b we can see that for n < 50000 stageoccurs as well.
In Fig.
7a, we see a parallel between the use of cyclic versus dot product kernels and the use data augmentation versus not for a Resnet-18 trained on CIFAR-5m (note that using a cyclic kernel is equivalent to using a dot product kernel with data-augmentation c.f. Appendix E.2). In both our theoretical results and in the empirical results of Nakkiran et al. we observe that the ideal world optimization speed of augmented and non-augmented training are the same, but for augmented training the real world training speed is slowed down, eventually leading to better generalization for long enough training. [Figure]
[Figure]
F A DDITIONAL F IGURES
To see the effects of varying the dimension d , in Fig. 8 we replicate the log-scale plots of kernel gradient ﬂow with dot product kernels from Fig.
We take n = d 1 . 5 and vary d ∈ { 50 , 100 , 200 , 400 } .
Each plot is averaged over 10 runs with the shaded region representing one standard deviation around the mean.
We can see that as d increasing the standard deviation decreases and the curves approach the theoretical high-dimensional prediction. [Figure]
G T ECHNICAL BACKGROUND G. N OTATIONS
For a positive integer, we denote by [ n ] the set { 1 , 2 , . . . , n } .
For vectors u , v ∈ R d , we denote (cid:104) u , v (cid:105) =
u 1 v 1 + . . .
+ u d v d their scalar product, and (cid:107) u (cid:107) 2 = (cid:104) u , u (cid:105) 1 / 2 the (cid:96) 2 norm.
Given a matrix A ∈ R n × m , we denote (cid:107) A (cid:107) op = max (cid:107) u (cid:107) 2 =1 (cid:107)
Au (cid:107)
its operator norm and by (cid:107) A (cid:107) F = (cid:0) (cid:80) i,j A 2 ij (cid:1) 1 / 2 its Frobenius norm.
If A ∈ R n × n is a square matrix, the trace of A is denoted by
Tr( A ) =
(cid:80) i ∈ [ n ] A ii .
We use O d ( · ) (resp.
o d ( · ) ) for the standard big-O (resp.
little-o) relations, where the subscript d emphasizes the asymptotic variable.
Furthermore, we write f = Ω d ( g ) if g ( d ) =
O d ( f ( d )) , and f = ω
d ( g ) if g ( d ) =
o d ( f ( d )) .
Finally, f = Θ d ( g ) if we have both f = O d ( g ) and f = Ω d ( g ) .
We use O d, P ( · ) (resp.
o
d, P ( · ) )
the big-O (resp.
little-o) in probability relations.
Namely, for h 1 ( d ) and h 2 ( d ) two sequences of random variables, h 1 ( d ) =
O d, P ( h 2 ( d ))
if for any ε > 0 , there exists
C ε > 0 and d ε ∈ Z > 0 , such that
[Equation]
( d ) /h 2 ( d ) | > C ε ) ≤ ε,
∀ d ≥ d ε , and respectively: h 1 ( d ) =
o d, P ( h 2 ( d )) , if h 1 ( d ) /h 2 ( d ) converges to 0 in probability.
Similarly, we will denote h 1 ( d ) =
Ω d, P ( h 2 ( d )) if h 2 ( d ) =
O d, P ( h 1 ( d )) , and h 1 ( d ) =
ω d, P ( h 2 ( d )) if h 2 ( d ) =
o d, P ( h 1 ( d )) .
Finally, h 1 ( d ) =
Θ d, P ( h 2 ( d )) if we have both h 1 ( d ) =
O d, P ( h 2 ( d )) and h 1 ( d ) =
Ω d, P ( h 2 ( d )) .
G. F UNCTIONAL SPACES OVER THE SPHERE
For d ≥ 3 , we let S d − 1 ( r ) =
{ x ∈ R d : (cid:107) x (cid:107)
=
r
} denote the sphere with radius r in R d .
We will mostly work with the sphere of radius √ d , S d − 1 ( √ d ) and will denote by τ d the uniform probability measure on S d − 1 ( √ d ) .
All functions in this section are assumed to be elements of
L 2 ( S d − 1 ( √ d ) , τ d ) , with scalar product and norm denoted as (cid:104) · , · (cid:105) L 2 and (cid:107) · (cid:107) L 2 : (cid:104) f, g (cid:105) [Equation] [Equation] (cid:90)
[Equation]
For (cid:96) ∈
Z ≥ 0 , let ˜ V d,(cid:96) be the space of homogeneous harmonic polynomials of degree (cid:96) on R d (i.e. homogeneous polynomials q ( x ) satisfying ∆ q ( x ) = 0 ), and denote by V d,(cid:96) the linear space of functions obtained by restricting the polynomials in ˜ V d,
[Equation]
to S d − 1 ( √ d ) .
With these deﬁnitions, we have the following orthogonal decomposition
[Equation]
The dimension of each subspace is given by
[Equation]
For each (cid:96)
∈
Z ≥ 0 , the spherical harmonics { Y ( d ) (cid:96),j } 1 ≤ j ≤ B ( d,(cid:96) ) form an orthonormal basis of V d,(cid:96) :
[Equation]
Note that our convention is different from the more standard one, that deﬁnes the spherical harmonics as functions on S d − 1 (1) .
It is immediate to pass from one convention to the other by a simple scaling.
We will drop the superscript d and write Y (cid:96),j = Y ( d ) (cid:96),j whenever clear from the context.
We denote by P k the orthogonal projections to V d,k in L 2 ( S d − 1 ( √ d ) , τ d ) .
This can be written in terms of spherical harmonics as
[Equation]
We also deﬁne P ≤ (cid:96) ≡ (cid:80) (cid:96)k =
P k , P >(cid:96) ≡ I − P ≤ (cid:96) =
(cid:80) ∞
k = (cid:96) +1 P k , and P <(cid:96) ≡ P ≤ (cid:96)
[Table]
G. G EGENBAUER POLYNOMIALS
The (cid:96) -th
Gegenbauer polynomial Q ( d ) (cid:96) is a polynomial of degree (cid:96) .
Consistently with our convention for spherical harmonics, we view Q ( d ) (cid:96) as a function Q ( d ) (cid:96) : [ − d, d ] → R .
The set { Q ( d ) (cid:96) } (cid:96) ≥ 0 forms an orthogonal basis on L 2 ([ − d, d ] , ˜ τ 1 d ) , where ˜ τ 1 d is the distribution of √ d (cid:104) x , e 1 (cid:105) when x ∼ τ d , satisfying the normalization condition:
[Equation]
In particular, these polynomials are normalized so that Q ( d ) (cid:96) ( d ) = 1 .
As above, we will omit the superscript ( d ) in Q ( d ) (cid:96) when clear from the context.
Gegenbauer polynomials are directly related to spherical harmonics as follows.
Fix v ∈ S d − 1 ( √ d ) and consider the subspace of V (cid:96) formed by all functions that are invariant under rotations in R d that keep v unchanged.
It is not hard to see that this subspace has dimension one, and coincides with the span of the function Q ( d ) (cid:96) ( (cid:104) v , · (cid:105) ) .
We will use the following properties of Gegenbauer polynomials
For x , y ∈ S
d − 1 ( √ d )
[Equation]
For x , y ∈ S
d − 1 ( √ d )
[Equation]
These properties imply that, up to a constant, Q ( d ) k
( (cid:104) x , y (cid:105) ) is a representation of the projector onto the subspace of degree- k spherical harmonics
[Equation]
[Equation] For a function σ ∈ L 2 ([ −√ d, √ d ] , τ 1 d ) (where τ 1 d is the distribution of (cid:104) e 1 , x (cid:105) when x ∼
Unif( S d − 1 ( √ d )) ), denoting its spherical harmonics coefﬁcients ξ d,k ( σ ) to be [Equation]
[Equation]
then we have the following equation holds in L 2 ([ −√ d, √ d ] , τ 1 d ) sense
[Equation]
For any rotationally invariant kernel H d ( x 1 , x 2 ) = h d (
[Equation] x 1 , x 2 (cid:105) /d ) , with h d ( √ d · ) ∈
L 2 ([ −√ d, √ d ] , τ 1 d ) , we can associate a self adjoint operator H d
[Equation]
L
[Equation]
S
[Equation]
[Equation]
By rotational invariance, the space V k of homogeneous polynomials of degree k is an eigenspace of H d , and we will denote the corresponding eigenvalue by ξ d,k ( h d ) .
In other words H d f ( x ) ≡ (cid:80) ∞
k =0 ξ d,k ( h d ) P k f .
The eigenvalues can be computed via
[Equation]
[Equation]
For a dot product kernel H d ( x , y )
=
h d (
[Equation]
x , y (cid:105) /d ) consider the Gegenbauer expansion of h d in
L
[Equation]
Using Eq.
(54) we can equivalently write the kernel as an expectation over random features for some activation σ d
[Equation]
61) by taking
[Equation]
Note that σ d ∈ L 2 ([ −√ d, √ d ] , τ 1 d ) as long as h (1) < ∞ .
G. H ERMITE POLYNOMIALS
The Hermite polynomials { He k } k ≥ 0 form an orthogonal basis of L 2 ( R , γ ) , where γ (d x ) =
e − x 2 / 2 d x/ √ 2 π is the standard Gaussian measure, and He k has degree k .
We will follow the classical normalization (here and below, expectation is with respect to G ∼ N (0 , 1) ):
[Equation]
As a consequence, for any function g ∈ L 2 ( R , γ ) , we have the decomposition
[Equation]
The Hermite polynomials can be obtained as high-dimensional limits of the Gegenbauer polynomials introduced in the previous section.
Indeed, the Gegenbauer polynomials (up to a √ d scaling in domain) are constructed by Gram-Schmidt orthogonalization of the monomials { x k } k ≥ 0 with respect to the measure ˜ τ
1 d , while Hermite polynomial are obtained by Gram-Schmidt orthogonalization with respect to γ .
Since ˜ τ 1 d ⇒ γ (here ⇒ denotes weak convergence), it is immediate to show that, for any ﬁxed integer k ,
[Equation]
Here and below, for P a polynomial, Coeﬀ { P ( x ) } is the vector of the coefﬁcients of P .
As a consequence, for any ﬁxed integer k , we have
[Equation]
where µ k ( σ ) and ξ d,k ( σ ) are given in Eq.
(64) and Eq.
G. T HE INVARIANT FUNCTION CLASS AND THE SYMMETRIZATION OPERATOR
Let G d be a group that is isomorphic to a subgroup of O ( d ) , the orthogonal group in d dimension.
That means, each element of G d can be identiﬁed with a matrix in O ( d ) ⊆
R d × d , and the group addition operation in G d can be regarded as matrix multiplications in O ( d ) .
For any x ∈ S d − 1 ( √ d ) and g ∈ G d , we deﬁne group action g · x to be the multiplication of matrix representation of g with the vector x .
We equip G d with a probability measure π d , which is the uniform probability measure on G d .
More speciﬁcally, the Borel sigma algebra on G d is deﬁned as the Borel sigma algebra of O ( d ) restricted on G d .
The uniform probability measure π d satisﬁes the property that, for any Borel-measurable set B ⊆ G d and any g ∈ G d , we have π d ( B ) =
π d ( gB ) .
Let L 2 ( S d − 1 ( √ d )) be the class of L 2 functions on S d − 1 ( √ d ) equipped with uniform probability measure Unif( S d − 1 ( √ d )) .
We deﬁne the invariant function class to be
L [Equation]
[Equation]
−
[Equation] √ d ) , G d ) = [Equation]
[Equation]
[Equation]
G
[Equation]
We deﬁne the symmetrization operator S :
L 2 ( S d − 1 ( √ d ))
→ L 2 ( S d − 1 ( √ d ) , G d ) to be
[Equation]
G. O RTHOGONAL POLYNOMIALS ON INVARIANT FUNCTION CLASS
We deﬁne V d, ≤ k ⊆
L 2 ( S d − 1 ( √ d )) to be the subspace spanned by all the degree (cid:96) polynomials, V d,>k ≡ V ⊥ d, ≤ k ⊆ L 2 ( S d − 1 ( √ d )) to be the orthogonal complement of V d, ≤ k , and V d,k = V d, ≤ k ∩ V ⊥ d, ≤ k − 1 .
In words, V d,k contains all degree k polynomials that orthogonal to all polynomials of degree at most k − 1 .
We further deﬁne V d,<k =
V d, ≤ k − 1 and V d, ≥ k = V d,>k − 1 . Let P ≤ (cid:96) to be the projection operator on L 2 ( S d − 1 ( √ d ) , Unif) that project a function onto V d, ≤ (cid:96) , the space spanned by all the degree (cid:96) polynomials.
Then it is easy to see that P ≤ (cid:96) and S operator commute.
This means, for any f ∈ L 2 ( S d − 1 ( √ d )) , we have
[Equation]
≤ (cid:96) [Equation] f )] . Similarly, we can deﬁne P (cid:96) , P <(cid:96) , P >(cid:96) , P ≥ (cid:96) , which commute with S .
We denote V d,(cid:96)
( G d ) ≡
P (cid:96) ( S d − 1 ( √ d ) , G d ) to be the space of polynomials in the images of P (cid:96) S .
Then we have
[Equation]
We denote D ( d ; k ) =
D ( S d − 1 ( √ d ); G d ; k ) ≡ dim( P k ( S d − 1 ( √ d ) , G d )) to be the dimension of
P k ( S d − 1 ( √ d ) , G d ) .
We denote { Y ( d ) kl } l ∈ [ D ( S d − 1 ( √ d ); k )] to be a set of orthonormal polynomial basis in P k
( S d − 1 ( √ d ) , G d ) .
That means
[Equation]
, and
[Equation]