L EARNING CURVES FOR G AUSSIAN PROCESS REGRESSION WITH POWER - LAW PRIORS AND TARGETS
Hui Jin
UCLA huijin@ucla.edu
Pradeep Kr.
Banerjee
MPI MiS pradeep@mis.mpg.de
Guido Montúfar
UCLA & MPI MiS montufar@math.ucla.edu
A BSTRACT
We characterize the power-law asymptotics of learning curves for Gaussian process regression (GPR) under the assumption that the eigenspectrum of the prior and the eigenexpansion coefﬁcients of the target function follow a power law.
Under similar assumptions, we leverage the equivalence between GPR and kernel ridge regression (KRR) to show the generalization error of KRR.
Inﬁnitely wide neural networks can be related to GPR with respect to the neural network GP kernel and the neural tangent kernel, which in several cases is known to have a power-law spectrum.
Hence our methods can be applied to study the generalization error of inﬁnitely wide neural networks.
We present toy experiments demonstrating the theory.
I NTRODUCTION
Gaussian processes (GPs) provide a ﬂexible and interpretable framework for learning and adaptive inference, and are widely used for constructing prior distributions in non-parametric Bayesian learning.
From an application perspective, one crucial question is how fast do GPs learn, i e., how much training data is needed to achieve a certain level of generalization performance.
Theoretically, this is addressed by analyzing so-called “learning curves”, which describe the generalization error as a function of the training set size n .
The rate at which the curve approaches zero determines the difﬁculty of learning tasks and conveys important information about the asymptotic performance of GP learning algorithms.
In this paper we study the learning curves for Gaussian process regression.
Our main result characterizes the asymptotics of the generalization error in cases where the eigenvalues of the
GP kernel and the coefﬁcients of the eigenexpansion of the target function have a power-law decay.
In the remainder of this introductory section, we review related work and outline our main contributions.
Gaussian processes A GP model is a probabilistic model on an inﬁnite-dimensional parameter space (Williams and Rasmussen, 2006; Orbanz and Teh, 2010).
In GP regression (GPR), for example, this space can be the set of all continuous functions.
Assumptions about the learning problem are encoded by way of a prior distribution over functions, which gets transformed into a posterior distribution given some observed data.
The mean of the posterior is then used for prediction.
The model uses only a ﬁnite subset of the available parameters to explain the data and this subset can grow arbitrarily large as more data are observed.
In this sense, GPs are “non-parametric” and contrast with parametric models, where there is a ﬁxed number of parameters.
For regression with Gaussian noise, a major appeal of the GP formalism is that the posterior is analytically tractable.
GPs are also one important part in learning with kernel machines (Kanagawa et al., 2018) and modeling using GPs has recently gained considerable traction in the neural network community.
Neural networks and kernel learning From a GP viewpoint, there exists a well known correspondence between kernel methods and inﬁnite neural networks (NNs) ﬁrst studied by Neal (1996).
Neal showed that the outputs of a randomly initialized one-hidden layer neural network (with appropriate scaling of the variance of the initialization distribution) converges to a GP over functions in the limit of an inﬁnite number of hidden units.
Follow-up work extended this correspondence with analytical expressions for the kernel covariance for shallow NNs by Williams (1997), and more recently for deep fully-connected NNs (Lee et al., 2018; de G. Matthews et al., 2018), convolutional NNs with many channels (Novak et al., 2019; Garriga-Alonso et al., 2019), and more general architectures (Yang, 2019).
The correspondence enables exact Bayesian inference in the associated GP model for inﬁnite-width NNs on regression tasks and has led to some recent breakthroughs in our understanding of overparameterized NNs (Jacot et al., 2018;
Lee et al., 2019; Arora et al., 2019; Belkin et al., 2018;
Daniely et al., 2016; Yang and Salman, 2019; Bietti and Mairal, 2019).
The most prominent kernels associated with inﬁnite-width NNs are the Neural Network Gaussian Process (NNGP) kernel (Lee et al., 2018; de G. Matthews et al., 2018), and the Neural Tangent Kernel (NTK)
(Jacot et al., 2018).
Empirical studies have shown that inference with such inﬁnite network kernels is competitive with standard gradient descent-based optimization for fully-connected architectures (Lee et al., 2020).
Learning curves A large-scale empirical characterization of the generalization performance of state-of-the-art deep NNs showed that the associated learning curves often follow a power law of the form n − β with the exponent β ranging between 0.07 and 0.35 depending on the data and the algorithm (Hestness et al., 2017; Spigler et al., 2020).
Power-law asymptotics of learning curves have been theoretically studied in early works for the Gibbs learning algorithm (Amari et al., 1992; Amari and
Murata, 1993; Haussler et al., 1996) that showed a generalization error scaling with exponent β =0 .
5 , 1 or 2 under certain assumptions.
More recent results from statistical learning theory characterize the shape of learning curves depending on the properties of the hypothesis class (Bousquet et al., 2021).
In the context of GPs, approximations and bounds on learning curves have been investigated in several works (Sollich, 1999; Sollich and Halees, 2002; Sollich, 2001; Opper and Vivarelli, 1999;
Opper and Malzahn, 2002; Williams and Vivarelli, 2000; Malzahn and Opper, 2001a;b; Seeger et al., 2008; Van Der Vaart and Van Zanten, 2011; Le Gratiet and Garnier, 2015), with recent extensions to kernel regression from a spectral bias perspective (Bordelon et al., 2020; Canatar et al., 2021).
For a review on learning curves in relation to its shape and monotonicity, see Loog et al.
; Viering et al.
(2019); Viering and Loog (2021).
A related but complementary line of work studies the convergence rates and posterior consistency properties of Bayesian non-parametric models (Barron, 1998; Seeger et al., 2008; Van Der Vaart and Van Zanten, 2011).
Power-law decay of the GP kernel eigenspectrum
The rate of decay of the eigenvalues of the
GP kernel conveys important information about its smoothness.
Intuitively, if a process is “rough” with more power at high frequencies, then the eigenspectrum decays more slowly.
On the other hand, kernels that deﬁne smooth processes have a fast-decaying eigenspectrum (Stein, 2012; Williams and
Rasmussen, 2006).
The precise eigenvalues ( λ p ) p ≥ 1 of the operators associated to many kernels and input distributions are not known explicitly, except for a few special cases (Williams and Rasmussen, 2006).
Often, however, the asymptotic properties are known.
The asymptotic rate of decay of the eigenvalues of stationary kernels for input distributions with bounded support is well understood (Widom, 1963; Ritter et al., 1995).
Ronen et al.
(2019) showed that for inputs distributed uniformly on a hypersphere, the eigenfunctions of the arc-cosine kernel are spherical harmonics and the eigenvalues follow a power-law decay.
The spectral properties of the NTK are integral to the analysis of training convergence and generalization of NNs, and several recent works empirically justify and rely on a power law assumption for the NTK spectrum (Bahri et al., 2021;
Canatar et al., 2021; Lee et al., 2020;
Nitanda and Suzuki, 2021).
Velikanov and Yarotsky (2021) showed that the asymptotics of the NTK of inﬁnitely wide shallow ReLU networks follows a power-law that is determined primarily by the singularities of the kernel and has the form
λ p ∝ p − α with α =1+ 1 d , where d is the input dimension.
Asymptotics of the generalization error of kernel ridge regression (KRR)
There is a well known equivalence between GPR and KRR with the additive noise in GPR playing the role of regularization in KRR (Kanagawa et al., 2018).
Analysis of the decay rates of the excess generalization error of
KRR has appeared in several works, e.g, in the noiseless case with constant regularization (Bordelon et al., 2020;
Spigler et al., 2020;
Jun et al., 2019), and the noisy optimally regularized case (Caponnetto and De Vito, 2007; Steinwart et al., 2009; Fischer and Steinwart, 2020) under the assumption that the kernel eigenspectrum, and the eigenexpansion coefﬁcients of the target function follow a power law.
These assumptions, which are often called resp.
the capacity and source conditions are related to the effective dimension of the problem and the difﬁculty of learning the target function (Caponnetto and De Vito, 2007; Blanchard and Mücke, 2018).
Cui et al.
(2021) present a unifying picture of the excess error decay rates under the capacity and source conditions in terms of the interplay between noise and regularization illustrating their results with real datasets.
Contributions In this work, we characterize the asymptotics of the generalization error of GPR and KRR under the capacity and source conditions.
Our main contributions are as follows:
• When the eigenspectrum of the prior decays with rate α and the eigenexpansion coefﬁcients of the target function decay with rate β , we show that with high probability over the draw of n input samples, the negative log-marginal likelihood behaves as Θ( n max { 1 α , 1 − 2 β α +1 } )
(Theorem 7) and the generalization error behaves as Θ( n max { 1 α − 1 , 1 − 2 β α } )
(Theorem 9).
In the special case that the model is correctly speciﬁed, i.e., the GP prior is the true one from which the target functions are actually generated, our result implies that the generalization error behaves as O ( n 1 α − 1 ) recovering as a special case a result due to Sollich and Halees (2002) (vide Remark 10).
• Under similar assumptions as in the previous item, we leverage the equivalence between GPR and KRR to show that the excess generalization error of KRR behaves as Θ( n max { 1 α − 1 , 1 − 2 β α } )
(Theorem 12).
In the noiseless case with constant regularization, our result implies that the generalization error behaves as Θ( n 1 − 2 β α ) recovering as a special case a result due to Bordelon et al.
Specializing to the case of KRR with Gaussian design, we recover as a special case a result due to Cui et al.
(2021) (vide Remark 14).
For the unrealizable case, i e., when the target function is outside the span of the eigenfunctions with positive eigenvalues, we show that the generalization error converges to a constant.
• We present a few toy experiments demonstrating the theory for GPR with arc-cosine kernel without biases(resp.
withbiases)whichistheconjugatekernelofaninﬁnitelywideshallownetworkwithtwoinputs and one hidden layer without biases (resp.
with biases) (Cho and Saul, 2009; Ronen et al., 2019).
B AYESIAN LEARNING AND GENERALIZATION ERROR FOR GP S
In GP regression, our goal is to learn a target function f : Ω (cid:55)→ R between an input x ∈ Ω and output y ∈ R based on training samples D n = { ( x i , y i ) } ni =1 .
We consider an additive noise model y i =
f ( x i ) + (cid:15) i , where (cid:15)
i i.i.d.
∼ N (0 ,σ 2true ) .
If ρ denotes the marginal density of the inputs x i , then the pairs ( x i , y i ) are generated according to the density q ( x, y )
=
ρ ( x ) q ( y | x ) , where q ( y | x )=
N ( y | f ( x ) ,σ 2true ) .
We assume that there is a prior distribution Π 0 on f which is deﬁned as a zero-mean GP with continuous and bounded covariance function k :Ω × Ω → R , i.e., f ∼GP (0 ,k ) .
This means that for any ﬁnite set x =( x 1 ,...,x n ) T , the random vector f ( x )=( f
( x 1 ) ,...,f ( x n ))
T follows the multivariate normal distribution N (0 ,K n ) with covariance matrix K n =( k
( x i ,x j ))
ni,j =1 ∈ R n × n .
By Bayes’ rule, the posterior distribution over f given the training data is given by
[Equation]
where Π 0 is the prior distribution, Z ( D n )
= (cid:82) (cid:81)
ni
=1 N ( y i | f ( x i ) ,σ 2model )
d Π 0 ( f ) is the marginal likelihood or model evidence and σ model is the sample variance used in GPR.
In practice, we do not know the exact value of σ true and so our choice of σ model can be different from σ true .
The GP prior and the Gaussian noise assumption allows for exact Bayesian inference and the posterior distribution over functions is again a GP with mean and covariance function given by
[Equation]
where K x x =( k ( x 1 ,x ) ,...,k ( x n ,x )) T and y =( y 1 ,...,y n ) T ∈ R n (Williams and Rasmussen, 2006, Eqs.
The performance of GPR depends on how well the posterior approximates f as the number of training samples n tends to inﬁnity.
The distance of the posterior to the ground truth can be measured in various ways.
We consider two such measures, namely the Bayesian generalization error (Seeger et al., 2008;
Haussler and Opper, 1997; Opper and Vivarelli, 1999) and the excess mean squared error (Sollich and Halees, 2002; Le Gratiet and Garnier, 2015; Bordelon et al., 2020; Cui et al., 2021).
Deﬁnition 1 (Bayesian generalization error) .
The Bayesian generalization error is deﬁned as the
Kullback-Leibler divergence between the true density q ( y | x ) and the Bayesian predictive density p n ( y | x,D n )=
[Equation]
[Equation]
A related quantity of interest is the stochastic complexity (SC), also known as the free energy , which is just the negative log-marginal likelihood.
We shall primarily be concerned with a normalized version of the stochastic complexity which is deﬁned as follows:
[Equation]
The generalization error (3) can be expressed in terms of the normalized SC as follows (Watanabe, 2009, Theorem 1.2):
[Equation]
where D n +1 =
D n ∪{ ( x n +1 ,y n +1 ) } is obtained by augmenting D n with a test point ( x n +1 ,y n +1 ) .
If we only wish to measure the performance of the mean of the Bayesian posterior, then we can use the excess mean squared error:
Deﬁnition 2 (Excess mean squared error) .
The excess mean squared error is deﬁned as
[Equation]
Proposition 3 (Normalized stochastic complexity for GPR) .
Assume that σ 2model = σ 2true =
σ 2 .
The normalized SC F 0 ( D n ) (4) for GPR with prior GP (0 ,k ) is given as
[Equation]
) where (cid:15) =( (cid:15) 1 ,...,(cid:15) n ) T .
The expectation of the normalized SC w.r.t.
the noise (cid:15) is given as
[Equation]
This is a basic result and has applications in relation to model selection in GPR (Williams and
Rasmussen, 2006).
For completeness, we give a proof of Proposition 3 in Appendix B. Seeger et al.
(2008, Theorem 1) gave an upper bound on the normalized stochastic complexity for the case when f lies in the reproducing kernel Hilbert space (RKHS) of the GP prior.
It is well known, however, that sample paths of GP almost surely fall outside the corresponding RKHS (Van Der Vaart and Van Zanten, 2011) limiting the applicability of the result.
We next derive the asymptotics of E (cid:15) F 0 ( D n ) , the expected generalization error E (cid:15) G ( D n ) =
E (cid:15) E ( x n +1 ,y n +1 ) F
( D n +1) − E (cid:15) F 0 ( D n ) , and the excess mean squared error E (cid:15) M ( D n ) .
A SYMPTOTIC ANALYSIS OF GP REGRESSION WITH POWER - LAW PRIORS
We begin by introducing some notations and assumptions.
We assume that f ∈ L 2 (Ω ,ρ ) .
By the generalization of Mercer’s theorem (Steinwart and Scovel, 2012, Corollary 3.2), the covariance function of the GP prior can be decomposed as k ( x 1 , x 2 ) =
(cid:80) ∞
p =1 λ p φ p ( x 1 )
φ p ( x 2 ) ρ -almost surely, where ( φ p ( x ))
p ≥ 1 are the eigenfunctions of the operator L k :
L 2 (Ω , ρ ) (cid:55)→
L 2 (Ω , ρ ) ; ( L k f )( x )=
(cid:82) Ω k ( x,s ) f
( s )d ρ ( s ) , and ( λ p ) p ≥ 1 are the corresponding positive eigenvalues.
We index the sequence of eigenvalues in decreasing order, that is λ 1 ≥ λ 2 ≥··· > 0 .
The target function f ( x ) is decomposed into the orthonormal set ( φ p ( x ))
p ≥ 1 and its orthogonal complement { φ p ( x ): p ≥ 1 } ⊥ as
[Equation]
where µ =( µ 0 ,µ 1 ,...,µ p ,... )
T are the coefﬁcients of the decomposition, and φ 0 ( x ) satisﬁes (cid:107) φ 0 ( x ) (cid:107) 2 = 1 and φ 0 ( x ) ∈ { φ p ( x ) : p ≥ 1 } ⊥ .
For given sample inputs x , let φ p ( x ) =
( φ p ( x 1 ) ,...,φ p ( x n )) T , Φ = ( φ 0 ( x ) ,φ 1 ( x ) ,...,φ p ( x ) ,... ) and Λ = diag { 0 ,λ 1 ,...,λ p ,... } .
Then the covariance matrix K n can be written as K n =ΦΛΦ T , and the function values on the sample inputs can be written as f ( x )=
Φ µ .
We shall make the following assumptions in order to derive the power-law asymptotics of the normalized stochastic complexity and the generalization error of GPR:
Assumption 4 (Power law decay of eigenvalues) .
The eigenvalues ( λ p ) p ≥ 1 follow the power law
[Equation]
(10) where C λ , C λ and α are three positive constants which satisfy 0 <C λ ≤ C λ and α> 1 .
As mentioned in the introduction, this assumption, called the capacity condition, is fairly standard in kernel learning and is adopted in many recent works (Bordelon et al., 2020; Canatar et al., 2021;
Jun et al., 2019; Bietti et al., 2021; Cui et al., 2021).
Velikanov and Yarotsky (2021) derived the exact value of the exponent α when the kernel function has a homogeneous singularity on its diagonal, which is the case for instance for the arc-cosine kernel.
Assumption 5 (Power law decay of coefﬁcients of decomposition) .
Let C µ ,C µ > 0 and β > 1 / 2 be positive constants and let { p i } i ≥ 1 be an increasing integer sequence such that sup
i ≥ 1 ( p i +1 − p i ) < ∞ .
The coefﬁcients ( µ p ) p ≥ 1 of the decomposition (9) of the target function follow the power law
[Equation]
Since f ∈ L 2 (Ω ,ρ ) , we have (cid:80) ∞ p =0 µ 2 p < ∞ .
The condition β > 1 / 2 in Assumption 5 ensures that the sum (cid:80) ∞ p =0 µ 2 p does not diverge.
When the orthonormal basis ( φ p ( x )) p is the Fourier basis or the spherical harmonics basis, the coefﬁcients ( µ p ) p decay at least as fast as a power law so long as the target function f ( x ) satisﬁes certain smoothness conditions (Bietti and Mairal, 2019).
Velikanov and Yarotsky (2021) gave examples of some natural classes of functions for which Assumption 5 is satisﬁed, such as functions that have a bounded support with smooth boundary and are smooth on the interior of this support, and derived the corresponding exponents β .
Assumption 6 (Boundedness of eigenfunctions) .
The eigenfunctions ( φ p ( x ))
p ≥ 0 satisfy
[Equation]
φ 0 (cid:107)
[Equation]
where C φ and τ are two positive constants which satisfy τ < α − 1 2 .
The second condition in (12) appears, for example, in Valdivia (2018, Hypothesis H 1 ) and is less restrictive than the assumption of uniformly bounded eigenfunctions that has appeared in several other works in the GP literature, see, e.g., Braun (2006); Chatterji et al.
; Vakili et al.
Deﬁne
[Equation]
Using (8) and (5), we have E (cid:15) F 0 ( D n )=
T 1 ( D n )+ T 2 ( D n ) and E (cid:15)
G ( D n )=
G 1 ( D n )+ G 2 ( D n ) .
Intuitively, G 1 corresponds to the effect of the noise on the generalization error irrespective of the target function f , whereas G 2 corresponds to the ability of the model to ﬁt the target function.
As we will see next in
Theorems 9 and 11, if α is large, then the error associated with the noise is smaller.
When f is contained in the span of the eigenfunctions { φ p } p ≥ 1 , G 2 decreases with increasing n , but if f contains an orthogonal component, then the error remains constant and GP regression is not able to learn the target function.
3. A SYMPTOTICS OF THE NORMALIZED STOCHASTIC COMPLEXITY
We derive the asymptotics of the normalized SC (8) for the following two cases: µ 0 =0 and µ 0 > 0 .
When µ 0 =0 , the target function f ( x ) lies in the span of all eigenfunctions with positive eigenvalues.
Theorem 7 (Asymptotics of the normalized SC, µ 0 = 0 ) .
Assume that µ 0 = 0 and σ 2model = σ 2true
=
σ 2 = Θ(1) .
Under Assumptions 4, 5 and 6, with probability of at least 1 − n − q over sample inputs ( x i ) ni
=1 , where 0 ≤ q < min { (2 β − 1)( α − 1 − 2 τ ) 4 α
[Equation]
τ 2 α } , the expected normalized SC (8) has the asymptotic behavior:
[Equation]
The complete proof of Theorem 7 is given in Appendix D.1.
We give a sketch of the proof below.
In the sequel, we use the notations O and Θ to denote the standard mathematical orders and the notation ˜
O to suppress logarithmic factors.
Proof sketch of Theorem 7.
By (8), (13) and (14) we have E (cid:15) F 0 ( D n ) =
T 1 ( D n ) + T 2 ( D n ) .
In order to analyze the terms T 1 ( D n ) and T 2 ( D n ) , we will consider truncated versions of these quantities and bound the corresponding residual errors.
Given a truncation parameter R ∈ N , let Φ R
=
( φ 0 ( x ) ,φ 1 ( x ) ,...,
φ R ( x ))
∈ R n × R be the truncated matrix of eigenfunctions evaluated at the data points, Λ R = diag(0 ,λ 1 ,...,λ
R ) ∈ R ( R +1)
× ( R +1) and µ R = ( µ 0 ,µ 1 ,...,µ R ) ∈ R R
+1 .
We deﬁne the truncated version of T 1 ( D n ) as follows:
[Equation]
Similarly, deﬁne Φ >R =
( φ R +
( x ) , φ R +2 ( x ) , ... , φ p ( x ) , ... ) , Λ >R = diag( λ R +1 , ... , λ p , ... ) , f R ( x ) = (cid:80)
Rp =1 µ p φ p ( x ) , f R ( x ) =
( f R ( x 1 ) , ... , f R ( x n )) T , f >R ( x ) =
f ( x ) − f R ( x ) , and f >R ( x )=( f >R ( x 1 ) ,...,f >R ( x n )) T .
The truncated version of T 2 ( D n ) is then deﬁned as
[Equation]
The proof consists of three steps:
• Approximation step: In this step, we show that the asymptotics of T 1 ,R resp.
T 2 ,R dominates that of the residuals, | T 1 ,R ( D n ) − T 1 ( D n ) | resp.
| T 2 ,R ( D n ) − T 2 ( D n ) | (see Lemma 32).
This builds upon ﬁrst showing that (cid:107) Φ >R Λ >R Φ T>R (cid:107) 2 =
˜ O (max { nR − α ,n 12 R 1 − 2 α 2 ,R 1 − α } ) (see Lemma 25) and then choosing R = n 1 α
+ κ where 0 <κ< α − 1 − 2 τ 2 αwhen
we have (cid:107) Φ >R Λ >R Φ T>R (cid:107)
= o (1) .
Intuitively, the choice of the truncation parameter R is governed by the fact that λ R =
Θ( R − α )= n − 1+ κα = o
( n − 1 ) .
• Decomposition step: In this step, we decompose T 1 ,R into a term independent of Φ R and a series involving Φ TR Φ R − nI R , and likewise for T 2 ,R (see Lemma 34).
This builds upon ﬁrst showing using the Woodbury matrix identity (Williams and Rasmussen, 2006, §A.3)
that
[Equation]
and then Taylor expanding the matrix inverse ( σ 2 I R + Λ R Φ TR Φ R ) − 1 in (20) and (21) to show that the Φ R -independent terms in the decomposition of T 1 ,R and T 2 ,R are, respectively,
[Equation]
and n
[Equation]
µ
[Equation]
R
[Equation]
• Concentration step: Finally, we use concentration inequalities to show that these Φ R -independent terms dominate the series involving Φ TR Φ R − nI R (see Lemma 35) when we have
[Equation]
The key idea is to consider the matrix Λ 1 / 2 R
( I + nσ 2 Λ R ) − 1 / 2 Φ T R Φ R
( I + nσ 2 Λ R ) − 1 / 2 Λ 1 / 2 R and show that it concentrates around n Λ R
( I + nσ 2 ) − 1 (see Corollary 22).
Note that an ordinary application of the matrix Bernstein inequality to Φ TR Φ R − nI R yields (cid:107) Φ TR Φ R − nI (cid:107)
=
O
( R √ n ) , which is not sufﬁcient for our purposes, since this would give O ( R √ n )=
o ( n )
only when α>
In contrast, our results are valid for α> 1 and cover cases of practical interest, e.g., the NTK of inﬁnitely wide shallow
ReLU network (Velikanov and Yarotsky, 2021) and the arc-cosine kernels over high-dimensional hyperspheres (Ronen et al., 2019) that have α =1+ O ( 1 d ) , where d is the input dimension.
For µ 0 > 0 , we note the following result:
Theorem 8 (Asymptotics of the normalized SC, µ> 0 ) .
Assume µ> 0 and σ 2 model
=
σ 2 true =
σ 2 = Θ(1) .
Under Assumptions 4, 5 and 6, with probability of at least 1 − n − q over sample inputs ( x i ) ni
=1 , where 0 ≤ q< min { 2 β − 1 2 ,α }· min { α − 1 − 2 τ 2 α, 2 β − 1 α 2 } .
the expected normalized SC (8) has the asymptotic behavior: E (cid:15) F 0 ( D n )= 12 σ 2 µ 20 n + o ( n ) .
The proof of Theorem 8 is given in Appendix D.1 and follows from showing that when µ 0 > 0 ,
T 2 ,R
[Equation]
2 µ 20 n + o ( n ) (see Lemma 38), which dominates T 1 ( D n ) and the residual | T 2 ,R ( D n ) − T 2 ( D n ) | .
3. A SYMPTOTICS OF THE B AYESIAN GENERALIZATION ERROR
In this section, we derive the asymptotics of the expected generalization error E (cid:15) G ( D n ) by analyzing the asymptotics of the components G 1 ( D n ) and G 2 ( D n ) in resp.
(15) and (16) for the following two cases: µ 0 =0 and µ 0 > 0 .
First, we consider the case µ 0 =0 .
Theorem 9 (Asymptotics of the Bayesian generalization error, µ 0 = 0 ) .
Let Assumptions 4, 5, and 6 hold.
Assume that µ 0 = 0 and σ 2model = σ 2true
=
σ 2 = Θ( n t ) where 1 − α 1+2 τ < t < 1 .
Then with probability of at least 1 − n − q over sample inputs ( x i ) ni
=1 where 0 ≤ q <
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 α, the expectation of the Bayesian generalization error (3) w.r.t.
the noise (cid:15) has the asymptotic behavior:
[Equation]
The proof of Theorem 9 is given in Appendix D.2.
Intuitively, for a given t , the exponent (1 − α )(1 − t ) α in (22) captures the rate at which the model suppresses the noise, while the exponent (1 − 2 β )(1 − t ) α captures the rate at which the model learns the target function.
A larger β implies that the exponent (1 − 2 β )(1 − t ) α is smaller and it is easier to learn the target.
A larger α implies that the exponent (1 − α )(1 − t )
α is smaller and the error associated with the noise is smaller as well.
A larger α , however, also implies that the exponent (1 − 2 β )(1 − t ) α is larger (recall that α> 1 and β > 1 / 2 by Assumptions 4 and 5, resp.), which means that it is harder to learn the target.
Remark 10.
If f ∼
GP (0 , k ) , then using the Karhunen-Loève expansion we have f ( x ) = (cid:80) ∞
p =1 (cid:112) λ p ω p φ p ( x ) , where ( ω p ) ∞ p =1 are i.i.d.
standard Gaussian variables.
We can bound ω p almost surely as | ω p | ≤ C log p , where C = sup p ≥ 1 | ω p | log p is a ﬁnite constant.
Comparing with the expansion of f ( x ) in (9) , we ﬁnd that µ p = (cid:112) λ p ω p
= O ( p − α/ 2 log p ) =
O ( p − α/ 2+ ε ) where ε> 0 is arbitrarily small.
Choosing β
= α/ 2 − ε in (22) , we have E (cid:15) G ( D n )=
O ( n 1 α − 1+ 2 εα ) .
This rate matches that of an earlier result due to Sollich and Halees (2002), where it is shown that the asymptotic learning curve (as measured by the expectation of the excess mean squared error, E f M ( D n ) ) scales as n 1 α − 1 when the model is correctly speciﬁed, i.e., f is a sample from the same Gaussian process
GP (0 ,k ) , and the eigenvalues decay as a power law for large i , λ i ∼ i α .
For µ 0 > 0 , we note the following result:
Theorem 11 (Asymptotics of the Bayesian generalization error, µ 0 > 0 ) .
Let Assumptions 4, 5, and 6 hold.
Assume that µ 0 > 0 and σ 2model = σ 2true
=
σ 2 = Θ( n t ) where 1 − α 1+2 τ < t < 1 .
Then with probability of at least 1 − n − q over sample inputs ( x i ) ni
=1 , where 0 ≤ q <
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 α, the expectation of the Bayesian generalization error (3) w.r.t.
the noise (cid:15) has the asymptotic behavior: E
[Equation]
In general, if µ 0 > 0 , then the generalization error remains constant when n → ∞ .
This means that if the target function contains a component in the kernel of the operator L k , then GP regression is not able to learn the target function.
The proof of Theorem 11 is given in Appendix D.2.
3. A SYMPTOTICS OF THE EXCESS MEAN SQUARED ERROR
In this section we derive the asymptotics of the excess mean squared error in Deﬁnition 2.
Theorem 12 (Asymptotics of excess mean squared error) .
Let Assumptions 4, 5, and 6 hold.
Assume σ 2model =
Θ( n t ) where 1 − α 1+2 τ <t< 1 .
Then with probability of at least 1 − n − q over sample inputs ( x i ) ni
=1 , where 0 ≤ q<
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 α, the excess mean squared error (6) has the asymptotic:
[Equation]
The proof of Theorem 12 uses similar techniques as Theorem 9 and is given in Appendix D.3.
Remark 13 (Correspondence with kernel ridge regression) .
The kernel ridge regression (KRR) estimator arises as a solution to the optimization problem
[Equation]
where the hypothesis space H k is chosen to be an RKHS, and λ > 0 is a regularization parameter.
The solution to (23) is unique as a function, and is given by ˆ f ( x )
=
K T x x ( K n + nλI n ) − 1 y , which coincides with the posterior mean function ¯ m ( x ) of the GPR (1) if σ 2model = nλ (Kanagawa et al., 2018, Proposition 3.6).
Thus, the additive Gaussian noise in GPR plays the role of regularization in KRR.
Leveraging this well known equivalence between GPR and KRR we observe that Theoremalso describes the generalization error of KRR as measured by the excess mean squared error.
Remark 14.
Cui et al.
(2021) derived the asymptotics of the expected excess mean-squared error for different regularization strengths and different scales of noise.
In particular, for KRR with Gaussian design where Λ 1 / 2 R ( φ 1 ( x ) ,...,φ R ( x ))) is assumed to follow a Gaussian distribution N (0 , Λ R ) , and regularization λ = n t − 1 where 1 − α ≤ t , Cui et al.
(2021, Eq. 10) showed that
[Equation]
Let δ = n − q , where 0 ≤ q <
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 α.
By Markov’s inequality, this implies that with probability of at least 1 − δ , E (cid:15) M ( D n ) =
O ( 1 δ max { σ 2true n 1 − α − t α , n (1 − 2 β )(1 − t ) α } ) =
O ( n q max { σ 2true n 1 − α − t α ,n (1 − 2 β )(1 − t ) α } ) .
Theorem 12 improves upon this by showing that with probability of at least 1 − δ , we have an optimal bound E (cid:15)
M
[Equation] [Equation]
Furthermore, in contrast to the approach by Cui et al. (2021), we have no requirement on the distribution of φ p ( x ) , and hence our result is more generally applicable.
For example, Theoremcan be applied to KRR with the arc-cosine kernel when the Gaussian design assumption is not valid.
In the noiseless setting ( σ true =0 ) with constant regularization ( t =0 ), Theorem 12 implies that the mean squared error behaves as Θ( n 1 − 2 β α ) .
This recovers a result in Bordelon et al.
Our upper bound in Theorem 12 matches with the ones derived in (Steinwart et al., 2009;
Fischer and Steinwart, 2020).
Steinwart et al.
(2009) and Fischer and Steinwart (2020) also derived algorithm independent minmax lower bounds.
In contrast to their results, our Theorem 12 gives lower bounds for different regularization strengths λ .
E XPERIMENTS
We illustrate our theory on a few toy experiments.
We let the input x be uniformly distributed on a unit circle, i.e., Ω= S 1 and ρ =
U ( S 1 ) .
The points on S 1 can be represented by x =(cos θ, sin θ ) where θ ∈ [ − π,π ) .
We use the ﬁrst order arc-cosine kernel function without bias, k (1)w / o bias ( x 1 ,x 2 )
=
1 π (sin ψ + ( π − ψ )cos ψ ) , where ψ = (cid:104) x 1 ,x 2 (cid:105) is the angle between x 1 and x 2 .
Hence Assumption 4 is satisﬁed with α =4 . We consider the target functions in Table 1, which satisfy Assumption 5 with the indicated β , and µ 0 indicates whether the function lies in the span of eigenfunctions of the kernel.
For each target we conduct GPR 20 times and report the mean and standard deviation of the normalized SC and the Bayesian generalization error in Figure 1, which agree with the asymptotics predicted in Theoremsand 9.
The details of the experiments appear in Appendix A, where we also show more experiments conﬁrming our theory for zero- and second- order arc-cosine kernels, with and without biases.
C ONCLUSION
We described the learning curves for GPR for the case that the kernel and target function follow a power law.
This setting is frequently encountered in kernel learning and relates to recent advances on neural networks.
Our approach is based on a tight analysis of the concentration of the inner product of empirical eigenfunctions Φ T Φ around nI .
This allowed us to obtain more general results with more [Table] realistic assumptions than previous works.
In particular, we recovered some results on learning curves for GPR and KRR previously obtained under more restricted settings (vide Remarks 10 and 14).
We showed that when β ≥ α/ 2 , meaning that the target function has a compact representation in terms of the eigenfunctions of the kernel, the learning rate is as good as in the correctly speciﬁed case.
In addition, our result allows us to interpret β from a spectral bias perspective.
When 12 < β ≤ α 2 , the larger the value of β , the faster the decay of the generalization error.
This implies that low-frequency functions are learned faster in terms of the number of training data points.
By leveraging the equivalence between GPR and KRR, we obtained a result on the generalization error of KRR.
In the inﬁnite-width limit, training fully-connected deep NNs with gradient descent and inﬁnitesimally small learning rate under least-squared loss is equivalent to solving KRR with respect to the NTK (Jacot et al., 2018; Lee et al., 2019; Domingos, 2020), which in several cases is known to have a power-law spectrum (Velikanov and Yarotsky, 2021).
Hence our methods can be applied to study the generalization error of inﬁnitely wide neural networks.
In future work, it would be interesting to estimate the values of α and β for the NTK and the NNGP kernel of deep fully-connected or convolutional NNs and real data distributions and test our theory in these cases.
Similarly, it would be interesting to consider extensions to ﬁnite width kernels.
A CKNOWLEDGMENT
This project has received funding from the European Research Council (ERC) under the EU’s Horizon 2020 research and innovation programme (grant agreement n o 757983).
A PPENDIX A E XPERIMENTS FOR ARC - COSINE KERNELS OF DIFFERENT ORDERS
In our experiment, the input space and input distribution are Ω = S 1 and ρ
=
U ( S 1 ) and we use the ﬁrst order arc-cosine kernel function.
(Cho and Saul, 2009) showed that this kernel is the conjugate kernel of an inﬁnitely wide shallow ReLU network with two inputs and no biases in the hidden layer.
GP regression with prior GP (0 ,k ) corresponds to Bayesian training of this network (Lee et al., 2018).
Under this setting, the eigenvalues and eigenfunctions are λ 1 = 4 π 2 , λ
2 = λ 3 = 14 , λ 2 p = λ 2 p +
=
π 2 ((2 p − 2) 2 − 1) 2 , p ≥ 2 and φ 1 ( θ )=1 , φ 2 ( θ )= √ 22 cos θ , φ 3 ( θ )= √ 22 sin θ , φ 2 p ( θ ) = √ 22 cos(2 p − 2) θ , φ 2 p +1 ( θ ) = √ 22 sin(
p − 2) θ , p ≥ 2 .
Hence Assumption 4 is satisﬁed with α =4 , and the second part of Assumption 6 is satisﬁed with (cid:107) φ p (cid:107)≤ √ 22 , p ≥ 1 .
The training and test data are generated as follows: We independently sample training inputs x 1 ,...,x n and test input x n +1 from U ( S 1 ) and training outputs
y i , i = 1 ,...,n from N ( f ( x i ) ,σ 2 ) , where we choose σ = 0 .
The Bayesian predictive density conditioned on the test point x n +
N ( ¯ m ( x n +1 ) , ¯ k ( x n +1 ,x n +1 )) is obtained by (1) and (2).
We compute the normalized SC by (7) and the Bayesian generalization error by the Kullback-Leibler divergence between N ( f ( x n +1 ) ,σ 2 ) and
N ( ¯ m ( x n +1 ) , ¯ k ( x n +1 ,x n +1 )) .
Next we present experiment results for arc-cosine kernels of different orders and arc-cosine kernels with biases.
Consider the ﬁrst order arc-cosine kernel function with biases, k
[Equation]
[Equation]
( π
[Equation]
ψ )cos ¯ ψ ) , where ¯ ψ =arccos (cid:0) 12 ( (cid:104) x [Equation] (cid:105)
[Equation]
1) (cid:1) .
Ronen et al.
(2019) showed that this kernel is the conjugate kernel of an inﬁnitely wide shallow ReLU network with two inputs and one hidden layer with biases, whose eigenvalues satisfy Assumptionwith α = 4 .
The eigenfunctions of this kernel are the same as that of the ﬁrst-order arc-cosine kernel without biases, k
(1)w / o bias in Section 4.
We consider the target functions in Table 3, which satisfy Assumption 5 with the indicated β , and µ 0 indicates whether the function lies in the span of eigenfunctions of the kernel.
For each target we conduct GPR 20 times and report the mean and standard deviation of the normalized SC and the Bayesian generalization error in Figure 3, which agree with the asymptotics predicted in Theorems 7 and 9.
Table 2 summarizes all the different kernel functions that we consider in our experiments with pointers to the corresponding tables and ﬁgures.
Summarizing the observations from these experiments, we see that the smoothness of the activation function (which is controlled by the order of the arc-cosine kernel) inﬂuences the decay rate α of the
[Table]
α
[Table]
bias pointer
[Table]
max
[Table] [Table]
no
Table 1/Figure
[Table]
max { 0 ,x } yes
Table 3/Figure
[Table]
no
Table 4/Figure
[Table]
yes
Table 5/Figure
[Table] [Table]
no
Table 6/Figure
[Table]
yes
Table 7/Figure eigenvalues.
In general, when the activation function is smoother, the decay rate α is larger. Theoremthen implies that smooth activation functions are more capable in suppressing noise but slower in learning the target. We also observe that networks with biases are more capable at learning functions compared to networks without bias. For example, the function cos(2 θ ) cannot be learned by the zero order arc-cosine kernel without biases (see Table 6 and Figure 6), but it can be learned by the zero order arc-cosine kernel with biases (see Table 7 and Figure 7).
function value [Equation] ( D n ) E (cid:15) G ( D n ) fcos2 θ + ∞ Θ( n 1 / 4 ) Θ( n − 3 / 4 ) fθ  Θ( n 1 / 4 ) Θ( n − 3 / 4 )
f
( | θ |− π/ 2)  Θ( n 1 / 4 ) Θ( n − 3 / 4 ) f(cid:26) π/ 2 − θ, θ ∈ [0 ,π ) − π/ 2 − θ, θ ∈ [ − π, 0)  Θ( n 3 / 4 ) Θ( n − 1 / 4 ) [Table]
[Table]
β µ
E (cid:15) F 0 ( D n ) E (cid:15) G ( D n ) fcos2 θ + ∞ Θ( n 1 / 6 ) Θ( n − 5 / 6 ) fθ  Θ( n 1 / 2 ) Θ( n − 1 / 2 ) f ( | θ |− π/ 2)  Θ( n 1 / 2 ) Θ( n − 1 / 2 ) f(cid:26) π/ 2 − θ, θ ∈ [0 ,π ) − π/ 2 − θ, θ ∈ [ − π, 0)  Θ( n 5 / 6 ) Θ( n − 1 / 6 ) function value β µ E (cid:15) F 0 ( D n )
{+E (cid:15) G ( D n ) fcos2 θ + ∞ >Θ( n ) Θ(1) fsign( θ )  Θ( n 1 / 2 ) Θ( n − 1 / 2 ) fπ/ 
 −| θ |  Θ( n 1 / 2 ) Θ( n − 1 / 2 ) f(cid:26) π/ 2 − θ, θ ∈ [0 ,π ) − π/ 2 − θ, θ ∈ [ − π, 0) >Θ( n ) Θ(1)+}
[Table]
β µ E (cid:15) F 0 ( D n ) E (cid:15) G ( D n ) fcos2 θ + ∞ Θ( n 1 / 2 ) Θ( n − 1 / 2 ) fθ  Θ( n 1 / 2 ) Θ( n − 1 / 2 ) f ( | θ |− π/ 2)  Θ( n 1 / 2 ) Θ( n − 1 / 2 ) f(cid:26) π/ 2 − θ, θ ∈ [0 ,π ) − π/ 2 − θ, θ ∈ [ − π, 0)  Θ( n 1 / 2 ) Θ( n − 1 / 2 )
B P ROOFS RELATED TO THE MARGINAL LIKELIHOOD
Proof of Proposition 3.
Let ¯ y
=
(¯ y 1 ,..., ¯ y n ) T be the outputs of the GP regression model on training inputs x .
Under the GP prior [Equation] the prior distribution of ¯ y is N (0 ,K n ) .
Then the evidence of the model is given as follows:
[Equation]
Letting ˜ K − 1 n = K − 1 n + 1 σ 2 I and µ = 1 σ 2 ˜ K n y
[Equation]
we have
[Equation]
The normalized evidence is
[Equation]
So the normalized stochastic complexity is
[Equation]
After taking the expectation over noises (cid:15)
[Equation]
we get
[Equation]
This concludes the proof
[Equation]
C H ELPER LEMMAS
Lemma 15.
Assume that m → ∞ as n → ∞ .
Given constants a 1 ,a 2 ,s 1 ,s 2 > 0 , if s 1 > 1 and s 2 s 3 >s 1 − 1 , we have that
[Equation]
If s 1 > 1 and s 2 s
= s 1 − 1 , we have that
[Equation]
If s 1 > 1 and s 2 s 3 <s 1 − 1 , we have that
[Equation]
Overall, if s 1 > 1 and m →∞ ,
[Equation]
Proof of Lemma 15.
First, when s 1 > 1 and s 2 s 3 >s 1 −
[Equation]
we have that
[Equation]
[Equation]
On the other hand, we have
R
[Equation]
Second, when s 1 > 1 and s 2 s
= s 1 − 1 , we have that
R (cid:88)
[Equation]
On the other hand, we have
[Equation]
Third, when s 1 > 1 and s 2 s 3 <s 1 − 1 , we have that
[Equation]
On the other hand, we have
[Equation]
Overall, if s 1 > 1 ,
[Equation]
Lemma 16.
Assume that R = m
1 s
+ κ for κ> 0 .
Given constants a 1 ,a 2 ,s 1 ,s 2 > 0 , if s 1 ≤ 1 , we have that
[Equation]
[Equation]
Proof of Lemma 16.
First, when s 1 ≤ 1 and s 2 s 3 >s 1 − 1 , we have that
R (cid:88)
[Equation]
Second, when s 1 ≤ 1 and s 2 s 3 ≤ s 1 − 1 , we have that
R (cid:88)
[Equation]
Overall, if s 1 ≤ 1 ,
[Equation]
Lemma 17.
Assume that f ∈ L 2 (Ω ,ρ ) .
Consider the random vector f ( x ) =
( f ( x 1 ) ,...,f ( x n )) T , where x 1 ,...,x n are drawn i.i.d from ρ .
Then with probability of at least 1 − δ 1 , we have
[Equation]
where (cid:107) f
(cid:107) 22 = (cid:82) x ∈ Ω f 2 ( x )d ρ ( x ) .
Proof of Lemma 17.
Given a positive number C ≥(cid:107) f (cid:107) 22 , applying Markov’s inequality we have
[Equation]
Let A be the event that for all sample inputs ( x i ) ni =1 , f 2 ( x i ) ≤ C .
Then
P
[Equation]
Deﬁne ¯ f 2 ( x ) =
min { f 2 ( x ) ,C } .
Then E ¯ f 2 ( X ) ≤ E f 2 ( X ) =
(cid:107) f (cid:107) 22 .
So | ¯ f 2 ( X ) − E ¯ f 2 ( X ) | ≤ max { C, (cid:107) f
(cid:107) 22 } =
C
Since 0 ≤ ¯ f 2 ( x ) ≤ C , we have
[Equation]
So we have
[Equation] [Equation]
Applying Bernstein’s inequality, we have
P
[Equation]
Hence, with probability of at least 1 − δ 1 / 2 we have n (cid:88)
[Equation]
When event A happens, f 2 ( x i ) =
¯ f 2 ( x i ) for all sample inputs.
According to (38) and (41), with probability at least 1 − 1 C n (cid:107) f (cid:107) 22 − δ 1 / 2 , we have n
[Equation]
i
[Equation]
Choosing C = 2 δ 1 n (cid:107) f (cid:107) 22 , with probability of at least 1 − δ 1 we have n (cid:88)
[Equation]
Lemma 18.
Assume that f ∈ L 2 (Ω ,ρ ) .
Consider the random vector f ( x ) =
( f ( x 1 ) ,...,f ( x n )) T , where x 1 ,...,x n are drawn i.i.d from ρ .
Assume that (cid:107) f (cid:107) ∞ =
sup x ∈ Ω f ( x ) ≤ C .
With probability of at least 1 − δ 1 , we have
[Equation]
where (cid:107) f
(cid:107) 2 2 = (cid:82) x ∈ Ω f 2 ( x )d ρ ( x ) .
Proof of Lemma 18.
We have | f 2 ( X ) − E f 2 ( X ) | ≤ max { C 2 , (cid:107) f
(cid:107) 22 } =
C
Since 0 ≤ f 2 ( x ) ≤ C , we have
[Equation]
So we have
[Equation]
Applying Bernstein’s inequality, we have
[Equation]
Hence, with probability of at least 1 − δ 1 we have
[Equation]
For the proofs in the reminder of this section, the deﬁnitions of the relevant quantities are given in
Section 3.
Corollary 19.
With probability of at least 1 − δ 1 , we have
[Equation]
Proof of Corollary 19.
The L 2 norm of f >R ( x ) is given by (cid:107) f >R (cid:107)
= (cid:80) ∞
p =
R +1 µ 2 p ≤ C µ 2 β − 1 R 1 − 2 β .
Applying Lemma 17 we get the result.
Corollary 20.
For any ν ∈ R R , with probability of at least 1 − δ 1 we have
[Equation]
Proof of Corollary 20.
Let g ( x )=
(cid:80)
Rp =1 ν p
φ p ( x ) .
Then Φ R ν = g
( x ) .
The L 2 norm of g ( x ) is given by (cid:107) g (cid:107)
= (cid:80)
Rp =1 ν 2 p
= (cid:107) ν (cid:107) 22 .
Applying Lemma 17 we get the result.
Next we consider the quantity, Φ TR Φ R − nI .
The key tool that we use is the matrix Bernstein inequality that describes the upper tail of a sum of independent zero-mean random matrices.
Lemma 21.
Let D = diag { d 1 , ... , d R } , d 1 , ... , d R > 0 and d max = max { d 1 , ... , d R } .
Let
M =max { (cid:80)
Rp =
d 2 p (cid:107) φ p (cid:107) 2 ∞ ,d 2max } .
Then with probability of at least 1 − δ , we have
[Equation]
Proof of Lemma 21.
Let Y j
= ( φ 1 ( x j ) , ... , φ R ( x j )) T and Z j = DY j .
It is easy to verify that E ( Z j Z Tj )=
D 2 .
Then the left hand side of (45) is (cid:80) nj =
[ Z j Z Tj − E ( Z j Z Tj )] .
We note that
[Equation]
≤
[Equation]
22 ,d 2max } .
For (cid:107) Z j (cid:107) 22 , we have (cid:107) Z [Equation] (cid:107) 22 =
[Equation]
(cid:88) p = d
[Equation] p φ [Equation] p ( x
[Equation]
) ≤
R (cid:88) p =
d 2 p (cid:107) φ p (cid:107) 2 ∞ , (46) we have (cid:107) Z j
[Equation]
(cid:107) [Equation] p (cid:107)
[Equation]
[-≤ max { (cid:80) R p =
 d 2 p (cid:107) φ p (cid:107) 2-] ∞ ,d 2max } .
On the other hand,
[Equation]
Since
E [ (cid:107) [-Z j (cid:107) 
-][Equation]
[Equation]
R (cid:88) p =
d 2 p (cid:107) φ p (cid:107) 2 ∞
Z j Z Tj ] , ( by (46) ) =
R (cid:88) p =
d 2 p (cid:107) φ p (cid:107)
[Equation]
we have
[Equation]
Using the matrix Bernstein inequality (Tropp, 2012, Theorem 6.1)
[Equation]
we have
[Equation]
 
[Equation]


[Equation]
[Equation]
Then with probability of at least 1 − δ
[Equation]
we have
[Equation]
Corollary 22.
Suppose that the eigenvalues ( λ p ) p ≥ 1 satisfy Assumption [Equation] and the eigenfunctions satisfy Assumption 6.
Assume σ 2 = Θ( n t ) where 1 − α 1+2 τ < t < 1 Let γ be a positive number such that 1+ α +2 τ − (1+2 τ +2 α ) t 2 α (1 − t ) <
γ ≤ 1 .
Then with probability of at least 1 − δ , we have
[Equation]
Proof of Corollary 22.
Use the same notation as in Lemma 21.
Let D
=
( I + nσ 2 Λ R ) − γ/ 2 Λ γ/ 2 R .
Then d 2max ≤ σ 2 γ n γ and (cid:80)
Rp =
d 2 p (cid:107) φ p (cid:107) 2 ∞ ≤ (cid:80)
Rp =
C 2 φ [Equation] [Equation]
+ nσ [Equation] ) γ [Equation]
O
[Equation] ) , where the ﬁrst inequality follows from Assumptions 4 and 6 and the last equality from Lemma 15.
Then
[-M =max { (cid:80) Rp =
 d 2 p (cid:107) φ p (cid:107) 2 ∞ ,d 2max } = O (( nσ 2 ) 1 − γα +2 τ α ) .-][Equation]
Applying Lemma 21, we have
[Equation]
Corollary 23.
Suppose that the eigenvalues ( λ p ) p ≥ 1 satisfy Assumption 4, and the eigenfunctions satisfy Assumption 6.
Let ˜ Λ 1 ,R = diag { 1 ,λ 1 ,...,λ R } .
Assume σ 2 = Θ( n t ) where t < 1 Let γ be a positive number such that 1+
τ α <γ ≤ 1 .
Then with probability of at least 1 − δ , we have
[Equation]
Proof of Corollary 23.
Use the same notation as in Lemma 21.
Let D =(
I + nσ 2 Λ R ) − γ/ 2 ˜Λ γ/ 2 1 ,R .
Then d 2max ≤ 1 and[-(cid:80) Rp =
 d 2 p (cid:107) φ p (cid:107) 2 ∞ ≤ C 2 φ + (cid:80) Rp =1 C 2 φ λ-]
[Equation]
=
O (1) where the ﬁrst inequality follows from Assumptions 4 and 6 and the second equality from Lemma 15.
Then
[-M =max { (cid:80) Rp =
 d 2 p (cid:107) φ p (cid:107)-][Equation] 2 ∞ ,d 2max } =
O (1) .
Applying Lemma 21, we have
[Equation]
[Equation]
Corollary 24.
Suppose that the eigenvalues ( λ p ) p ≥ 1 satisfy Assumption 4, and the eigenfunctions satisfy Assumption 6.
Let Φ R +1: S =(
φ R +1 ( x ) ,...,φ S ( x )) , and Λ R +1: S =( λ
R +1 ,...,λ S ) .
Then with probability of at least 1 − δ , we have
[Equation]
Proof of Corollary 24.
Use the same notation as in Lemma 21.
Let D = Λ 1 / 2 R
+1: S .
Then d 2max ≤ C λ R − α
= O ( R − α ) and (cid:80) Sp
= R +1 C 2 φ d 2 p p 2 τ ≤ (cid:80) Sp
= R +1 C
φ C λ p − α p 2 τ
= O ( R 1 − α +2 τ ) , where the ﬁrst inequality follows from Assumptions 4 and 6.
Then M =max { (cid:80) Sp
= R +1 C 2 φ d 2 p p 2 τ ,d 2max } =
O ( R 1 − α +2 τ ) .
Applying Lemma 21, we have
[Equation]
Lemma 25.
Under the assumptions of Corollary 24, with probability of at least 1 − δ , we have
[Equation]
Proof of Lemma 25.
For S ∈ N , we have
[Equation]
Let S
=
R α α − 1 − 2 τ .
Then we get (cid:107) Φ >S Λ >S Φ T >S (cid:107)
= O ( nR − α ) .
Let Φ R +1: S =(
φ R +1 ( x ) ,...,φ S ( x )) , Λ R +1: S =( λ
R +1 ,...,λ S ) .
We then have (cid:107) Φ >R Λ >R Φ T>R (cid:107) 2 ≤(cid:107)
Φ >S Λ >S Φ T>S (cid:107)
+ (cid:107) Φ R
+1: S Λ R +1: S Φ TR
+1: S (cid:107)≤ O ( nR − α )+ (cid:107) Λ [Equation] [Equation]
[Equation] S Φ TR
[-+1: S Λ 1 / 2 R +1: S (cid:107)≤ O ( nR − α ) + n (cid:107) Λ R +1: S (cid:107) 
 + (cid:107) Λ 1 / 2 R-][Equation]
[Equation] S Φ R [Equation]
[Equation] S Λ 1 / 2 R + [Equation] S (cid:107)≤ O
[Equation] n
[Equation] Λ [Equation]
[Equation]
R [Equation] where in the fourth inequality we use Corollary [Equation]
Corollary 26.
Assume that σ 2 = Θ(1) .
If R = n 1 α + κ where 0 < κ < α − 1 − 2 τ α (1+2 τ ) , then with probability of at least 1 − δ , we have
[Equation]
Proof of Corollary 26.
By Lemma 25 and the assumption R = n 1 α + κ
[Equation]
we have
[Equation]
Lemma 27.
Assume that (cid:107) 1 σ 2 ( I + nσ 2 Λ R ) − γ/ 2 Λ γ/ 2 R (Φ TR Φ R − nI )Λ γ/ 2 R
( I + nσ 2 Λ R ) − γ/ 2 (cid:107) 2 < 1 where [-1+
-]{+
+}
+2 τ α <γ ≤ 1 .
We then have
[Equation]
Proof of Lemma 27.
First note that
[Equation]
Let ˜Λ (cid:15),R = diag { (cid:15),λ 1 ,...,λ R } .
Since Λ R = diag { 0 ,λ 1 ,...,λ R } , we have that when (cid:15) is sufﬁciently small, (cid:107) 1 σ
( I + nσ 2 ˜Λ
[Equation]
,R ) −
[Equation]
/ 2 ˜Λ 1 /
[Equation]
,R
[Equation]
Φ TR
[Equation]
nI )˜Λ 1 / 2 (cid:15),R
[Equation]
I
[Equation]
nσ 2 ˜Λ (cid:15),R ) − 1 / [Equation] 2 < 1 .
Since all diagonal entries of ˜Λ (cid:15),R are positive, we have
[Equation]
Letting (cid:15) → 0 , we get
[Equation]
This concludes the proof
[Equation]
Lemma 28.
If (cid:107) ( I + Φ R Λ R Φ TR σ) − 1 Φ >R Λ >R Φ
[Equation]
>R σ(cid:107) 2 < 1 , then we have
[Equation]
In particular, assume that σ 2 =Θ(1) .
Let R = n 1 α
+ κ where 0 <κ< α − 1 −
τ
α
(1+2 τ ) .
Then with probability of at least 1 − δ , for sufﬁciently large n , we have (cid:107) ( I + Φ R Λ R Φ TR σ) − 1 Φ >R Λ >R Φ T>R
[Equation]
(cid:107) 2 < 1 and (53) holds.
Proof of Lemma 28.
Deﬁne Φ >R =(
φ R +
( x ) ,φ R +2 ( x ) ,... ) , Λ >R =diag( λ R +1 ,λ R +2 ,... ) .
Then we have
[Equation]
By Corollary 26, for sufﬁciently large n , (cid:107) ( I + Φ R Λ R Φ TR σ) − 1 Φ >R Λ >R Φ T>R σ(cid:107) 2 < 1 with probability of at least 1 − δ .
Hence
[Equation]
Lemma 29.
Assume that µ 0 =0 and σ 2 =Θ( n t ) where 1 − α 1+2 τ <t< 1 .
Let R = n ( 1 α + κ )(1 − t )
where 0 <κ<
[Equation]
Then when n is sufﬁciently large, with probability of at least 1 − 2 δ we have
[Equation]
Proof of Lemma 29.
Let Λ 1: R = diag { λ 1 , ... , λ R } , Φ 1: R = ( φ 1 ( x ) , φ 1 ( x ) , ... , φ R ( x )) and µ 1: R = ( µ 1 , ... , µ R ) .
Since µ 0 = 0 , we have ( I + 1 σ 2 Φ R Λ R Φ TR ) − 1 f R ( x ) =
( I + 1 σ 2 Φ 1: R Λ 1: R Φ T 1: R ) − 1 Φ 1: R µ 1: R .
Using the Woodbury matrix identity, we have that ( I +
[Equation]
Let A
= ( I + nσ 2 Λ 1: R ) − 1 / 2 Λ 1 / 2 1: R (Φ T 1: R Φ 1: R − nI )Λ 1 / 2 1: R ( I + nσ 2 Λ 1: R ) − 1 / 2 .By
Corollary 22, with probability of at least 1 − δ , we have (cid:107) 1 σ 2 A (cid:107) 2 = (cid:113) log R δ n 1 − [Equation]
− (1+2 τ ) t 2 α .
When n is sufﬁciently large, (cid:107) 1 σ 2 A (cid:107)
= o (1) is less than 1 because 1 − α 1+2 τ <t< 1 .
By Lemma 27, we have
[Equation]
We then have
[Equation]
≤  
[Equation]
By Lemma 15 and Assumption 5, assuming that sup
i ≥ 1 p
i +1 − p i
=
h , we have
[Equation]
where k = (cid:26) 0 , 2 α (cid:54) =2 β − 1 , 1 , 2 α =2 β − 1 . .
Overall we have
[Equation]
k/ 2 n [-1 − α +
-]) .
Using the fact that (cid:107)
[Equation]
2 A (cid:107) 2 = (cid:113) log Rδ n
[Equation]
[Equation]
2 α − (1+2 τ ) t 2 α and (cid:107) ( I + nσ 2 Λ 1: R ) − 1 Λ 1: R (cid:107) 2 ≤ n − 1 , we have
[Equation]
By Lemma 16 and the assumption R = n (
[Equation]
+ κ
[Equation]
1 −
[Equation]
[Equation]
We then have
[Equation]
By (56), (57) and (60)
[Equation]
we have
[Equation]
By assumption κ< α − 1 − 2 τ +(1+2 τ ) t
α 2 (1 − t ) , we have that κ
[Equation]
Using (61)
[Equation]
we then get
[Equation]
By Corollary
[Equation]
with probability of at least 1 − δ , we have
[Equation]
From (55)
[Equation]
we get
[Equation]
This concludes the proof.
Lemma 30.
Assume that µ 0 > 0 and σ 2 = Θ( n t ) where 1 − α 1+2 τ < t < 1 .
Let R = n 1 α
+ κ where 0 <κ< α − 1 − 2 τ
+(1+2 τ ) t α.
Then when n is sufﬁciently large, with probability of at least 1 − 2 δ , we have
[Equation]
Proof of Lemma 30.
Using the Woodbury matrix identity, we have that
[Equation]
Let µ R, 1 =( µ 0 , 0 ,..., 0) and µ R, 2 =(0 ,µ 1 ,...,µ R ) .
Then µ R = µ R, 1 + µ R, 2 .
Then we have (cid:107) [Equation] I [Equation]
[Equation]
µ R
[Equation]
[Equation]
µ
[Equation]
(cid:107) 2 . According to (62) in the proof of Lemma 29, we have (cid:107) ( I + 1 σ 2 Λ R Φ TR Φ R ) − 1 µ R, 2 (cid:107) [Equation]
= ˜ O ( n max {− (1 − t ) , (1 − t )(1 − 2 β )
[Equation]
α } ) .
Next we estimate (cid:107)
[Equation]
Let
[Equation]
where 11 − t ( 1+ [Equation] + [Equation]
[Equation] − (1+2 τ +2 α ) t 2 α ) < γ < 1 .
Since 1 − α 1+2 τ < t < 1 , 11 − t ( 1+ α +2 τ 2 α − ( [Equation] [Equation] [Equation]
α ) <
so the range for γ is well-deﬁned.
By Corollary 22, with probability of at least 1 − δ , we have (cid:107)
[Equation]
= o (1) .
When n is sufﬁciently large, (cid:107) 1 σ 2 A (cid:107) 2 is less than 1 because 1 − α 1+2 τ <t< 1 .
By Lemma 27, we have
[Equation]
We then have (cid:107)
[Equation]

[Equation] [Equation]
By Lemma 15,
[Equation]
Let ˜Λ 1 ,R = diag { 1 , λ 1 , ... , λ R } and I 0 ,R = (0 , 1 , ... , 1) .
Then Λ R = ˜Λ 1 ,R I 0 ,R .
Let B
=
( I + nσ 2 Λ R ) − γ/ 2 ˜Λ γ/ 2 1 ,R (Φ TR Φ R − nI )˜Λ γ/ 2 1 ,R
( I + nσ 2 Λ R ) − γ/ 2 .
According to Corollary 23, we have (cid:107) B (cid:107) 2 =
O (cid:0)(cid:113) log Rδ n 12 (cid:1) .
Using the fact that (cid:107) 1 σ 2 A (cid:107) 2 = ˜
O (cid:0)(cid:113) log
Rδ n 1+ α +2 τ 2 α − (1+2 τ +2 α ) t 2 α − γ (1 − t ) (cid:1) , we have (cid:13)(cid:13)(cid:13)(cid:0)
[Equation]
2 ( I + nσ 2 Λ R ) − 1 Λ R (Φ TR Φ R [Equation] nI ) (cid:1)
[Equation]
( I
[Equation]
[Table]
=
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] ≤ [Table] [Equation]
[Table]
[Equation]
[Table]
[Table]
[Equation]
[Table] =
˜ O ( n − 12 + γ 2 (1 − t ) + [1 − α +2 τ − (1+2 τ ) t ]( j − 1) 2 α [-+
-]) .
Since 11 − t ( 1+ α +2 τ 2 α − (1+2 τ +2 α ) t 2 α ) <γ < 1 and − 12 + 11 − t ( 1+ α +2 τ 2 α − (1+2 τ +2 α ) t 2 α ) 1 − t 2 < 0 , we can let γ be a little bit larger than 11 − t ( 1+ α +2 τ
2 α − (1+2 τ +2 α ) t 2 α ) and make − 12 + γ 2 (1 − t ) < 0 holds.
By (67), (68), (69), we have
[Equation]
According to (66), we have (cid:107) (
[Equation]
O (1) .
By Corollary 20, with probability of at least 1 − δ , we have
[Equation]
From (65), we get (cid:107) ( I + 1 σ 2 Φ R Λ R Φ TR ) − 1 f R ( x ) (cid:107)
=
˜
O
(cid:16)(cid:113) ( 1 δ +1) n (cid:17) .
This concludes the proof.
Lemma 31.
Assume that σ 2 = Θ(1) .
Let R = n 1 α
+ κ where 0 < κ < α − 1 − 2 τ α.
Assume that µ 0 = 0 .
Then when n is sufﬁciently large, with probability of at least 1 − 3 δ we have
[Equation]
Assume that µ 0 > 0 .
Then when n is sufﬁciently large, with probability of at least 1 − 3 δ we have
[Equation]
Proof of Lemma 31.
We have
[Equation]
When µ 0 =
[Equation]
by Lemma [Equation] with probability of at least 1 − 2 δ [Equation]
we have
[Equation]
Since α − 1 − 2 τ α< α − 1 − 2 τ α (1+2 τ )
[Equation]
we apply Lemma 28 and Corollary 26 and get that with probability of at least 1 − δ , the second term in the right hand side of (73) is estimated as follows: [Equation] ˜
[Equation]
Overall, from (73), we have that with probability 1 − 3 δ ,
[Equation]
When µ 0 > 0 , using the same approach and Lemma 30, we can prove that (cid:107) ( I + ΦΛΦ T σ 2 ) − 1 f R ( x ) (cid:107) 2 = ˜
O ( (cid:113) ( 1 δ +1) n ) .
This concludes the proof.
D P ROOF OF THE MAIN RESULTS D. P ROOFS RELATED TO THE ASYMPTOTICS OF THE NORMALIZED STOCHASTIC COMPLEXITY
Lemma 32.
Under Assumptions 4, 5 and 6, with probability of at least 1 − 2 δ we have, we have
[Equation]
If R = n 1 α + κ where κ > 0 , we have | T 1 ,R ( D n ) − T 1 ( D n ) |
= o (cid:16) [Equation] 2 n 1 α (cid:17) .
If we further assume that 0 <κ< α − 1 − 2 τ α, µ 0 =0 and σ 2 =Θ(1) , then for sufﬁciently large n with probability of at least 1 − 4 δ we have
[Equation]
Proof of Lemma 32.
Deﬁne Φ >R =
( φ R[-+
-]
[Equation]
1 ( x ) , φ R +2 ( x ) , ... , φ p ( x ) , ... ) , and Λ >R = diag( λ R +1 ,...,λ p ,... ) .
We then have
[Equation]
[Equation]
As for the ﬁrst term in the right hand side of (76), we have
[Equation]
1 2logdet
[Equation]
Given a concave function h and a matrix B ∈ R n × n whose eigenvalues ζ 1 ,...
[Equation]
ζ n are all positive
[Equation]
we have that
[Equation]
where we used Jensen’s inequality.
Using h ( x )
=log(
+ x ) in (78)
[Equation]
with probability 1 − δ , we have
[Equation]
where in the second inequality we use the fact that Tr AB ≤(cid:107)
A (cid:107)
Tr B when A and B are symmetric positive deﬁnite matrices [Equation] and in the last inequality we use Lemma
[Equation]
As for the second term in the right hand side of (76)
[Equation]
let A =(
I + Φ R Λ R Φ TR σ) − 1 / 2 .
Then we have
[Equation]
where in the ﬁrst inequality we use the fact that (cid:107) A (cid:107) 2 < 1 and Tr ABA ≤ (cid:107) A (cid:107) 22 Tr B when A and B are symmetric positive deﬁnite matrices, in the second inequality we use h ( x )=1 − 1 / (1+ x ) in (78) and in the last equality we use the last few steps of (79).
This concludes the proof of the ﬁrst statement
[Equation]
As for | T 2 ( D n ) − T 2 ,R ( D n ) | , we have
[Equation]
For the ﬁrst term on the right-hand side of (80), we have
[Equation]
[-x ) T ( I + ΦΛΦ T σ 2 ) − 1 f >R ( x ) (cid:12)(cid:12)(cid:12) ≤ 2 (cid:107) f >R ( x ) (cid:107) 2 (cid:107) ( I + ΦΛΦ T σ 2 ) − 1 f R ( x ) (cid:107) 
 + (cid:107) f-]Applying Corollary 19 and Lemma
[Equation]
with probability of at least 1 − [-1 f R ( x ) (cid:107) 
 + (cid:107) f-]4 δ
[Equation]
we have
[Equation]
where the last equality holds because ( 1 α + κ ) [Equation] 2 < 1 − 2 β 2 α when κ> 0 .
As for the second term on the right-hand side of (80), according to Lemma 28, Corollary 26 and
Lemma 29, we have
[Equation]
By (80) [Equation] we have
[Equation]
This concludes the proof of the second statement.
In Lemma 32, we gave a bound for | T 2 ,R ( D n ) − T 2 ( D n ) | when n 1 α < R < n 1 α + α − 1 − 2 τ α.
For R > n , we note the following lemma:
Lemma 33.
Let R = n C and σ 2 = n t .
Assume that C ≥ 1 and C (1 − α + 2 τ ) − t < 0 .
Under
Assumptions 4, 5 and 6, for sufﬁciently large n and with probability of at least 1 − 3 δ we have
[Equation]
Proof of Lemma 33.
Deﬁne Φ >R =
[Equation]
[Equation]
, ... , φ p ( x ) , ... )
[Equation]
and Λ >R = diag( λ R +1 ,...,λ p ,... ) .
Then we have
[Equation]
For the ﬁrst term on the right-hand side of (83)
[Equation]
with probability 1 − 3 δ we have
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where we used Corollary 19 and Lemma 17 for the last inequality.
The assumption C (1 − α + 2 τ ) − t < 0 means that R 1 − α +2 τ σ= o (1) .
For the second term on the right-hand side of (83), by Lemmas 28 and 25, we have
[Equation] [Equation]
Using (83) [Equation] we have
[Equation]
Next we consider the asympototics of T 1 ,R ( D n ) and T 2 ,R ( D n ) .
Lemma 34.
Let A
= ( I + nσ 2 Λ R ) − [Equation] / 2 Λ γ/ 2 R [Equation]
Φ TR Φ R − nI )
Λ γ/ 2 R ( I + nσ 2 Λ R ) − γ/ 2 .
Assume that (cid:107) A (cid:107) 2 < 1 where 1+2 τ α <γ ≤ 1 .
Then we have
[Equation]
where
[Equation]
Proof of Lemma 34.
Let ˜Λ (cid:15),R = diag { (cid:15),λ 1 ,...,λ R } .
Since Λ R = diag { 0 ,λ 1 ,...,λ R } , we have that when (cid:15) is sufﬁciently small, (cid:107)[-1 σ 
 ( I + n-]
[Equation]
<
[Equation]
Since all diagonal entries of ˜Λ (cid:15),R are positive, we have
[Equation]
Using Lemma
[Equation]
we have
[Equation]
Letting (cid:15) → 0 , we get
[Equation]
This concludes the proof
[Equation]
Lemma 35.
Assume that σ 2 =Θ(1) .
Let R = n 1 α
+ κ where 0 <κ< α − 1 − 2 τ 2 α.
Under Assumptions 4, 5 and 6, with probability of at least 1 − δ , we have
[Equation]
Furthermore, if we assume µ 0 =0 , we have
[Equation]
Proof of Lemma 35.
Let
[Equation]
where 1+ α +2 τ 2 α <γ ≤ 1 .
By Corollary 22, with probability of at least 1 − δ , we have
[Equation]
[Equation]
When n is sufﬁciently large, (cid:107) A (cid:107) 2 is less than 1 .
Let B =(
I + nσ 2 Λ R ) − 1 / 2 Λ 1 / 2 R
(Φ TR Φ R − nI )Λ 1 / 2 R
( I + nσ 2 Λ R ) − 1 / 2 .
Then (cid:107) B (cid:107) 2 =
σ 2(1 − γ ) n 1 − γ (cid:107) A (cid:107) 2 =
˜
O
( n 1 − α +
τ 2 α ) .
Using the Woodbury matrix identity, we compute T 1 ,R ( D n ) as follows:
[Equation]
where in the last equality we apply Lemma
[Equation]
Let h ( x )=
log(1+ x ) − (1 − 11+ x ) .
It is easy to verify that h ( x ) is increasing on [0 , + ∞ ) .
As for the ﬁrst term on the right hand side of (91), we have
[Equation]
where in the last equality we use the fact that (cid:82) [0 , + ∞ ] h ( x − α )d x< ∞ .
On the other hand, we have
[Equation]
Overall, we have 12 logdet
[Equation]
As for the second term on the right hand side of (91), we have (cid:12)
[Equation]
=
[Equation]
−
[Equation]
As for the third term on the right hand side of (91), we have
[Equation]
Tr   ∞ (cid:88)
j
= ( − 1)
[Equation]
σ [Equation] I [Equation]
Λ R
[Equation] [Equation] ( I [Equation]
nσ
[Equation]
R
[Equation] − 1 /  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ [Equation]
j =
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
I
[Equation]
R ) − 1 / 2 (cid:17)(cid:12)(cid:12)(cid:12) ≤ R ∞ (cid:88)
j = (cid:13)(cid:13)(cid:13) [Equation] σ 2 j ( I + nσ 2 Λ R ) − 1 / 2 B j ( I + nσ 2 Λ R ) − 1 / 2 (cid:13)(cid:13)(cid:13)≤ R ∞ (cid:88)
j = (cid:13)(cid:13)(cid:13)
[Equation]
σ
[Equation]
j ( I
[Equation]
nσ [Equation] R ) − 1 / 2 B j ( I
[Equation]
nσ 2 Λ R ) − 1 / 2 (cid:13)(cid:13)(cid:13)≤ R ∞ (cid:88)
j = (cid:13)(cid:13) 1 σ
[Equation]
j B j (cid:13)(cid:13) 2 = ˜ O ( n 1 α + κ +1 − {+α +
+} τ 2 α ) .
Then the asymptotics of T 1 ,R ( D n ) is given by T 1 ,
[Equation]
where in the last inequality we use the assumption that κ< α − 1 − 2 τ 2 α . Since ˜ O ( n 1 α + κ + 1 − α +2 τ 2 α ) is lower order term compared to Θ( n 1 α ) , we further have
[Equation]
This concludes the proof of the ﬁrst statement.
Let Λ 1: R =diag { λ 1 ,...,λ R } , Φ 1: R =( φ 1 ( x ) ,φ 1 ( x ) ,...,
φ R ( x )) and µ 1: R =( µ 1 ,...,µ R ) .
Since µ 0 =0 , we have T 2 ,R ( D n )=
1 2 σ 2 µ T 1: R Φ T 1: R ( I + 1 σ 2 Φ 1: R Λ 1: R Φ T 1: R ) − 1 Φ 1: R µ 1: R .
According to Lemma 34, we have
[Equation]
where in the second to last equality we used the deﬁnition of A (89).
As for the ﬁrst term on the right hand side of (92) [Equation] by Lemma 15, Assumption 4 and Assumption 5, we have
[Equation]
On the other hand, by Assumption 5, assuming that sup
i ≥ 1 p
i +1 − p i
=
h , we have
[Equation]
Overall, we have n
[Equation]
max {
[Equation]
[Equation]
[Equation]
k n ) , where k = (cid:26) 0 , α (cid:54) =2 β − 1 , 1 , α =2 β − 1 .
By Lemma
[Equation]
we have
[Equation]
Using (90), the second term on the right hand side of (92) is computed as follows:
[Equation]
Since 1+ α +2 τ 2 α {+< 
+}
[-˜ O ( n max {− 2+ 1+ α +
-]
[-τ 2 α , 1 − 2 β α + 
-]
+ α +
[-+ κ (1 − 2 β + αγ ) } ) . Since 1+ α +2 τ 2 α < 
 + α +
-] τ α +1+2 τ =1 , we have − 2+ 1+ α +2 τ
2 α <
.Also
we have
[Equation]
where the last inequality holds because κ< α − 1 − 2 τ 2 αand γ ≤ 1 . Hence we have
[Equation]
n
[Equation]
where k = (cid:26) 0 , α (cid:54) =
2 β − 1 , 1 , α =2 β − 1 . .
Since ˜ O ( n max {− 2+ 1+ α +2 τ 2 α , 1 − 2 β α +
+ α +
τ
2 α
+ κ (1 − 2 β + αγ ) } ) is lower order term compared to Θ( n max { 0 , 1+ 1 −
2 β α } log k n ) , we further have
[Equation]
This concludes the proof of the second statement.
Lemma 36.
Under Assumptions 4, 5 and 6, with probability of at least 1 − 5 δ , we have
[Equation]
Furthermore, let δ = n − q where 0 ≤ q< min { (2 β − 1)( α − 1 − 2 τ
[Equation]
4 α, α − 1 − 2 τ 2 α } .
If we assume µ 0 =0 , we have
[Equation]
Proof of Lemma 36.
Let R = n 1 α
+ κ where 0 ≤ κ< α − 1 − 2 τ 2 α.
By Lemmas 32 and 35, with probability of at least 1 − 5 δ we have
[Equation]
[Equation]
and |
[Equation]
as well as
[Equation]
and
[Equation]
We then have
[Equation]
Since ˜ O ( n 1 α + κ (1 − α ) ) is lower order term compared to Θ( n 1 α ) [Equation] we further have
[Equation]
Besides, we have
[Equation]
Then we have log det( I + nσ 2 Λ R )
= log det( I + nσ 2 Λ)(1 + o (1)) .
Similarly we can prove
Tr (cid:0) I − ( I + nσ 2 Λ) − 1 (cid:1) =
Tr (cid:0) I − ( I + nσ 2 Λ R ) − 1 (cid:1) (1 + o (1)) .
This concludes the proof of the ﬁrst statement.
As for T 2 ( D n ) , we have T
[Equation]
where we use δ = n − q , k = (cid:26) 0 , α (cid:54) =
2 β − 1 , 1 , α =2 β − 1 . .
Since 0 ≤ κ< α − 1 − 2 τ 2 αand 0 ≤ q< min { (2 β − 1)( α − 1 − 2 τ ) 4 α, α − 1 − 2 τ 2 α } , we can choose κ< α − 1 − 2 τ
2 αand κ is arbitrarily close to α − 1 − 2 τ 2 αsuch that 0 ≤ q< min { (2 β − 1) κ,κα } .
Then we have ( 1 α + κ ) 1 − 2 β 2 + q< 0 , − 1 − κα + q< 0 , (1 − 2 β )
κ+ q< 0 and − κα + q< 0 .
So we have
[Equation]
Since ˜ O (cid:16)
[Equation]
} (cid:17) is lower order term compared to Θ( n max { 0 , 1+ 1 −
2 β α } log k n ) , we further have
[Equation]
Furthermore, we have
[Equation]
− 2 β α ) . Then we have µ T ( I + nσ 2 Λ) − 1 µ = µ TR ( I + nσ 2 Λ R ) − 1 µ R (1+ o (1)) .
This concludes the proof of the second statement.
Proof of Theorem 7.
Using Lemma 36 and noting that 1 α > 0 , with probability of at least 1 − 5˜ δ , we have
[Equation]
Letting δ
=5˜ δ , we get the result.
In the case of µ 0 > 0 , we have the following lemma:
Lemma 37.
Assume that σ 2 = Θ(1) .
Let R = n 1 α
+ κ where 0 < κ < α − 1 − 2 τ α.
Assume that µ 0 > 0 .
Under Assumptions 4, 5 and 6, for sufﬁciently large n with probability of at least 1 − 4 δ we have
[Equation]
Proof of Lemma 37.
As for | T 2 ( D n ) − T 2 ,R ( D n ) | , we have
[Equation]
For the ﬁrst term on the right-hand side of (103), we have
[Equation]
Applying Corollary 19 and Lemma
[Equation]
with probability of at least 1 − 4 δ
[Equation]
we have
[Equation]
As for the second term on the right-hand side of (80) [Equation]
according to Lemma
[Equation]
Corollary 26 and Lemma
[Equation]
we have
[Equation]
By (80), we have
[Equation]
Lemma 38.
Assume that σ 2 = Θ(1) .
Let R = n 1 α
+ κ where 0 < κ < min { α − 1 − 2 τ 2 α, 2 β − 1 α 2 } .
Assume that µ 0 > 0 .
Under Assumptions 4, 5 and 6, with probability of at least 1 − δ , we have
[Equation]
Proof of Lemma 38.
Let
[Equation]
where 1+ α +2 τ 2 α <γ ≤ 1 .
By Corollary 22, with probability of at least 1 − δ , we have
[Equation]
When n is sufﬁciently large, (cid:107) A (cid:107) 2 is less than 1 .
Let µ R, 1 = ( µ 0 , 0 ,..., 0) and µ R, 2 = (0 ,µ 1 ,...,µ R ) .
Then µ R = µ R, 1 + µ R, 2 .
Let ˜Λ 1 ,R =diag { 1 ,λ 1 ,...,λ R } and I 0 ,R =(0 , 1 ,..., 1) .
Then Λ R = ˜Λ 1 ,R I 0 ,R .
Let B =
( I + nσ 2 Λ R ) − 1 / 2 ˜Λ 1 / 2 1 ,R (Φ TR Φ R − nI )˜Λ 1 / 2 1 ,R
( I + nσ 2 Λ R ) − 1 / 2 .
By Corollary 23, we have (cid:107) B (cid:107)
= O ( (cid:113) log Rδ n 12 ) .
By Lemma 34, we have
T
[Equation]
As for the ﬁrst term on the right hand side of (108), by Lemma 15, we have
[Equation]
We deﬁne Q 1 ,j , Q 2 ,j and Q 3 ,j by
[Equation]
The quantity Q 3 ,j actually shows up in the case of µ 0 = 0 in the proof of Lemma 35.
By (92), (94) and (95), we have that
[Equation]
For Q 1 ,j , we have
[Equation]
where in the last equality we use {+(cid:107) B (cid:107) 
 = O ( (cid:113) log Rδ n 12 ) . For j ≥+}
[Equation]
we have
[Equation]
Then we have |
[Equation]
For Q [Equation] j , we have
[Equation]
Since (cid:107) ( I + [Equation] γ 2 1 ,R µ R, 2 (cid:107) 2 is actually the case of µ 0 = 0 , we can use (93) in the proof of
Lemma 35 and get
[Equation]
where in the last equality we use κ< 2 β − 1 α 2 .
Then we have |
[Equation]
Choosing γ = 12 (1+ 1+ α +2 τ 2 α )
= [-1+
-]1+
[Equation]
+2 τ 4 α < 1 , we have
[Equation]
Proof of Theorem 8.
Let R = n 1 α
+ κ where 0 < κ < min { α − 1 − 2 τ 2 α, 2 β − 1 α 2 } .
Since 0 ≤ q < min { 2 β − 1 2 , α } · min { α − 1 − 2 τ 2 α, 2 β − 1 α 2 } , we can choose κ < min { α − 1 − 2 τ 2 α, 2 β − 1 α 2 } and κ is arbitrarily close to κ < min {
α − 1 − 2 τ 2 α, 2 β − 1 α 2 } such that 0 ≤ q < min { (2 β − 1) κ,κα } .
Then we have ( 1 α + κ ) 1 − 2 β 2 + q< 0 , and − κα + q< 0 .
As for T 2 ( D n ) , we have
[Equation]
By Lemma 36, we have T 1 ( D n ) =
O ( n 1 α ) .
Hence E (cid:15) F 0 ( D n ) =
T 1 ( D n ) + T 2 ( D n ) = n 2 σ 2 µ 20 + o ( n ) .
D. P ROOFS RELATED TO THE ASYMPTOTICS OF THE GENERALIZATION ERROR
Lemma 39.
Assume σ 2 = Θ( n t ) where 1 − α 1+2 τ < t < 1 .
Let R = n ( 2 α − 1 α ( α − 1)
+1)(1 − t ) .
Under
Assumptions 4, 5 and 6, with probability of at least 1 − δ over sample inputs ( x i ) ni
=1 , we have
[Equation]
Proof of Lemma 39.
Let G 1 ,R ( D n ) =
E ( x n +1 ,y n +1 ) ( T 1 ,R ( D n +1 ) − T 1 ,R
( D n )) , where R = n C for some constant C. By Lemma 32, we have that
[Equation]
y n +1 )
[ T 1 ( D n +1 ) − T 1 ,R ( D n +1 )] − [ T 1 ( D n ) − T 1 ,R
( D n )] (cid:12) (cid:12) =
(cid:12)(cid:12) E ( x n +1 ,y n +1 )
O (( n +1) R 1 − α )
(cid:12)(cid:12) + (cid:12)(cid:12) O
( nR 1 − α )] (cid:12)(cid:12) =
O ( 1 σ 2 nR 1 − α ) .
Deﬁne η R =( φ 0 ( x n +1 ) ,φ 1 ( x n +1 ) ,...,φ R ( x n +1 ))
T and (cid:101)
Φ R =(Φ TR ,η R ) T .
As for G 1 ,R ( D n ) , we have
G 1 ,R ( D n )=
E ( x n +1 ,y n +1 ) ( T 1 ,R ( D n +1 )
[Equation]
T 1 ,R ( D n ))
[Equation]
[Equation]
As for the ﬁrst term in the right hand side (116), we have
[Equation]
Let
[Equation]
According to Corollary 22, with probability of at least 1 − δ , we have (cid:107) 1 σ 2 A (cid:107)=
O ( (cid:113) log
Rδ n 1 −
[Equation]
+
[Equation]
α − (1+2 τ ) t 2 α ) =
o (1) .
When n is sufﬁciently large, (cid:107) 1 σ 2 A (cid:107) 2 is less than 1 .
By
Lemma 27, we have
[Equation] [Equation]
where we use Lemma 15 in the last inequality.
Next we have
[Equation]
where in the last equality we use the fact that E ( x n +1 ,y n +1 )
η R η TR
= I .
By Lemma 27, we have
[Equation]
By Lemma
[Equation]
we have
[Equation]
Overall,
[Equation]
Since (cid:107) 1 σ 2 A (cid:107) j
=
o
[Equation]
we have that the absolute values of diagonal entries of 1 σ 2 j A j are at most o (1) .
Let ( A j ) p,p denote the ( p,p ) -th entry of the matrix A j .
Then we have
[Equation]
Tr
[Equation]
) where in the last step we used (119).
According to (119) and (120)
[Equation]
we have
[Equation]
Using the Woodbury matrix identity, the second term in the right hand side (116) is given by
[Equation]
where the last equality uses the Sherman–Morrison formula.
According to (118)
[Equation]
we get
[Equation]
where in the penultimate equality we use Tr( BB T )
= (cid:107) B (cid:107) 2 F , (cid:107) B (cid:107) F is the Frobenius norm of A
[Equation]
and in the last equality we use the deﬁnition of A (117).
Then we have
[Equation]
By Lemma
[Equation]
we have
[Equation]
Overall, we have
[Equation] [Equation]
Since (cid:107)
[Equation]
2 A (cid:107)
=
O ( (cid:113) log Rδ
n
[Equation]
= o
[Equation]
1) , we have
[Equation]
124) where in the ﬁrst inequality we use the fact that (cid:107) AB (cid:107) F ≤ (cid:107) A (cid:107) F (cid:107) B (cid:107) 2 when B is symmetric.
By
Lemma 15, we have
[Equation]
According to (123), (124) and (125)
[Equation]
we have
[Equation] [Equation]
+2TrΛ 1 /
[Equation]
Combining (121) and (126) we get that G 1 ,R
( D n )
= 1+ o (1) 2 σ
Tr( I + nσ 2 Λ R ) − 1 Λ R + (cid:107) Λ 1 / 2 R
( I + nσ 2 Λ R ) − 1 (cid:107) 2 F ) =
1 σ 2 Θ( n (1 − α )(1 − t ) α ) .
From (115) we have that G 1 ( D n ) ≤
G 1 ,R ( D n ) + | G 1 ( D n ) − G 1 ,R
( D n )
| =
σ 2 Θ( n (1 − α )(1 − t ) α )
+ O ( n 1 σ 2 R 1 − α ) .
Choosing
R = n
[Equation]
) we conclude the proof.
Lemma 40.
Assume σ 2 =Θ( n t ) where 1 − α 1+2 τ <t< 1 .
Let S = n D .
Assume that (cid:107) ξ (cid:107) 2 =1 .
When n is sufﬁciently large, with probability of at least 1 − 2 δ we have
[Equation]
Proof of Lemma 40.
Using the Woodbury matrix identity, we have that
[Equation]
Let A =
[Equation]
γ/ 2 , where γ > 1+ α +2 τ − (1+2 τ +2 α ) t 2 α (
[Equation]
By Corollary 22, with probability of at least 1 − δ , we have (cid:107) 1 σ 2 A (cid:107) 2 = ˜
[Equation]
( n
[Equation]
When n is sufﬁciently large, (cid:107) 1 σ 2 A (cid:107) 2 is less than 1 .
By Lemma 27, we have
[Equation]
Then we have
[Equation]
≤ 
[Equation]
For the ﬁrst term in the right hand side of the last equation, we have
[Equation] [Equation]
Using the fact that (cid:107) 1 σ 2 A (cid:107)
=
˜
O
( n 1+
[Equation]
+
[Equation]
− (1+
[Equation]
+
[Equation]
) t
[Equation]
− γ (1 − t ) ) and (cid:107) ( I + nσ 2 Λ S ) − 1 Λ S (cid:107) 2 ≤ n − 1 , we have
[Equation] [Equation]
= ˜
O
[Equation]
n
[Equation] [Equation]
Using (129)
[Equation]
(130) and (131)
[Equation]
we have
[Equation]
( n − 1+ (1 − α +2 τ − (1+2 τ ) t )
j 2 α ) 
 = (cid:16)
˜ O
( n − (1 − t ) )
+ ˜ O
( n − 1+ 1 − α +2 τ − (1+2 τ ) t 2 α ) (cid:17) =
˜ O ( n − (1 − t ) ) .
By Corollary 20, with probability of at least 1 − δ , we have
[Equation]
From (128) we get (cid:107) ( I + 1 σ 2 Φ S Λ S Φ TS ) − 1 f S ( x ) (cid:107)
= ˜ O
( (cid:113) ( 1 δ +1) n · n − (1 − t ) ) .
This concludes the proof.
Lemma 41.
Assume σ 2 = Θ( n t ) where 1 − α 1+2 τ < t < 1 .
Let δ = n − q where 0 ≤ q <
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 α.
Under Assumptions 4, 5 and 6, assume that µ 0 = 0 .
Let R = n ( 1 α + κ )(1 − t )
where 0 <κ< α − 1 − 2 τ +(1+2 τ ) t 2 α 2 (1 − t ) .
Then with probability of at least 1 − 6 δ over sample inputs ( x i ) ni
=1 , we have G 2 ( D n )=
(1+ o (1)) 2 σ
[Equation]
log k/ 2 n ) , where k = (cid:26) 0 , 2 α (cid:54) =
2 β − 1 , 1 , 2 α =2 β − 1 . .
Proof of Lemma 41.
Let S = n D .
Let G 2 ,S ( D n ) =
E ( x n +1 ,y n +1 )
( T 2 ,S ( D n +1 ) − T 2 ,S
( D n )) .
By
Lemma 33, when S >n max { 1 ,
[Equation]
t
[Equation]
) } with probability of at least 1 − 3 δ we have that
[Equation]
Let Λ 1: S = diag { λ 1 ,...,λ S } , Φ 1: S = ( φ 1 ( x ) ,φ 1 ( x ) ,...,φ S ( x )) and µ 1: S = ( µ 1 ,...,µ S ) .
Since µ 0 = 0 , we have T 2 ,S ( D n ) = 12 σ 2 µ T 1: S Φ T 1: S ( I + 1 σ 2 Φ 1: S Λ 1: S Φ T 1: S ) − 1 Φ 1: S µ 1: S .
Deﬁne η 1: S = ( φ 1 ( x n +1 ) ,...,φ S ( x n +1 ))
T and (cid:101) Φ 1: S =(Φ T 1: S ,η 1: S ) T .
In the proof of Lemma 34, we showed that
[Equation]
We have
[Equation]
where in the fourth to last equality we used the Sherman–Morrison formula, in the third inequality we used (118) , and in the last equality we used the fact that E ( x n +1 ,y n +1 )
η 1: S η T 1: S
= I .
Let ˆ µ 1: R =( µ 1 ,...,µ R , 0 ,..., 0) ∈ R S .
Then we have
[Equation]
Let R = n ( 1 α + κ )(1 − t )
where 0 < κ < α − 1 − 2 τ +(1+2 τ ) t 2 α 2 (1 − t ) .
In Lemma 29, (62), we showed that with probability of at least 1 − δ ,
[Equation]
where k
=
[Equation] [Equation]
=
[Equation]
, 1 , 2 α =2 β − 1 . .
The same proof holds if we replace Φ 1: R with Φ 1: S , Λ 1: R with Λ 1: S , and µ 1: R with ˆ µ 1: R .
We have
[Equation]
Next we bound (cid:107) ( I + 1 σ 2 Λ 1: S Φ T 1: S Φ 1: S ) − 1 ( µ 1: S − ˆ µ 1: R ) (cid:107) 2 .
By Assumption 5, we have that (cid:107) µ 1: S − ˆ µ 1: R (cid:107)
= O ( R 1 − 2 β 2 ) .
For any ξ ∈ R S and (cid:107) ξ (cid:107)
=1 , using the Woodbury matrix identity, with probability of at least 1 − 2 δ
we have
[Equation]
where in the second to last step we used Corollary 20 to show (cid:107) Φ 1: S ( µ 1: S − ˆ µ 1: R ) (cid:107) 2 =
O ( (cid:113) ( 1 δ +1)
nR 1 − 2 β 2 ) with probability of at least 1 − δ , and Lemma 40 to show that (cid:107) ( I + 1 σ 2 Φ 1: S Λ 1: S Φ T 1: S ) − 1 Φ 1: S Λ 1: S ξ (cid:107)
= O ( (cid:113) ( 1 δ +1) n · n − 1 ) with probability of at least 1 − δ .
Since R = n ( 1 α + κ )(1 − t ) , we have
[Equation]
(1 − t ) κ) .
Since ξ is arbitrary, we have (cid:107) ( I + 1 σ 2 Λ 1: S Φ T 1: S Φ 1: S ) − 1 ( µ 1: S − ˆ µ 1: R ) (cid:107)
=
O
(( 1 δ + 1) n (1 − 2 β )(1 − t )
α + (1 − 2 β )(1 − t ) κ) .
Since 0 ≤ q <
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 αand 0 < κ < α − 1 − 2 τ
+(1+2 τ ) t
α 2 (1 − t ) , we can choose κ < α − 1 − 2 τ
+(1+2 τ ) t
α 2 (1 − t ) and κ is arbitrarily close to κ < α − 1 − 2 τ
+(1+2 τ ) t
α 2 (1 − t )
such that
0 ≤
[Equation]
Then we have (1 − 2 β )(
[Equation]
+ q< 0 .
From (136) and (138), we have
[Equation]
Hence
[Equation]
Then by (133)
[Equation]
we have
[Equation]
Choosing S = n max
[Equation]
we get the result.
Proof of Theorem 9.
From Lemmas 39 and 41 and 1 α − 1 > −
[Equation]
we have that with probability of at least 1 − 7˜ δ ,
[Equation]
where k = (cid:26) 0 , 2 α (cid:54) =2 β −1 , 2 α =2 β − 1 , and R = n ( 1 α + κ )(1 − t ) , κ> 0 .
Furthermore
[Equation]
we have
[Equation]
Then we have
[Equation]
Similarly we can prove
[Equation]
Letting δ =
7˜ δ , the proof is complete.
In the case of µ 0 > 0 , we have the following lemma:
Lemma 42.
Let δ = n − q where 0 ≤ q <
[ α − (1+2 τ )(1 − t )](2 β − 1) 4 α.
Under Assumptions 4, 5 and 6, assume that µ 0 > 0 .
Then with probability of at least 1 − 6 δ over sample inputs ( x i ) ni
=1 , we have
G 2 ( D n )=
1 2 σ 2 µ 2
+ o (1) .
Proof of Lemma 42.
Let S = n D .
Let G 2 ,S ( D n ) =
E ( x n +1 ,y n +1 )
( T 2 ,S ( D n +1 ) − T 2 ,S
( D n )) .
By
Lemma 33, when S >n max { 1 , − t ( α − 1 − 2 τ ) } , with probability of at least 1 − 3 δ we have that
[Equation]
=
[Equation]
Let Λ S = diag { λ 1 , ... , λ S } , Φ S =
( φ 1 ( x ) , φ 1 ( x ) , ... , φ S ( x )) and µ S =
( µ 1 , ... , µ S ) .
Deﬁne η S =
( φ 0 ( x n +1 ) ,φ 1 ( x n +1 ) ,...,φ S ( x n +1 ))
T and (cid:101) Φ S
=
(Φ TS ,η S ) T .
By the same technique as in the proof of Lemma 34, we replace Λ R by ˜Λ (cid:15),R =diag { (cid:15),λ 1 ,...,λ R } , let (cid:15) → 0 and show the counterpart of the result (135) in the proof of Lemma 41:
[Equation]
where in the fourth to last equality we used the Sherman–Morrison formula, in the third inequality we used (118) , and in the last equality we used the fact that E ( x n +1 ,y n +1 )
η 1: S η T 1: S
= I .
Let ˆ µ R =( µ 0 ,µ 1 ,...,µ R , 0 ,..., 0) ∈ R S .
Then we have
[Equation]
Choose R = n ( 1 α + κ )(1 − t )
where 0 <κ< α − 1 − 2 τ +(1+2 τ ) t
α 2 (1 − t ) .
In Lemma 29, (62)
[Equation]
we showed that with probability of at least 1 − δ ,
[Equation]
where k = (cid:26) 0 , 2 α (cid:54) =2 β − 1 , 1 , 2 α =2 β − 1 . .
The same proof holds if we replace Φ 1: R with Φ 1: S , Λ 1: R with Λ 1: S , and µ 1: R with ˆ µ 1: R .
We have
[Equation]
So we have
[Equation]
Next we bound (cid:107) ( I + 1 σ 2 Λ S Φ TS Φ S ) − 1 ( µ S − ˆ µ R ) (cid:107) 2 .
By Assumption 5, we have that (cid:107) µ S − ˆ µ R (cid:107)
= O ( R 1 − 2 β 2 ) .
For any ξ ∈ R S and (cid:107) ξ (cid:107)
= 1 , using the Woodbury matrix identity, with probability of at least 1 − 2 δ
we have
[Equation]
whereinthesecondtolaststepweusedCorollary20toshow (cid:107) Φ
[Equation]
= O
[Equation]
[Equation] nR [Equation] with probability of at least 1 − δ , and Lemma 40 to show that (cid:107) ( I + 1 σ 2 Φ S Λ S Φ TS ) − 1 Φ S Λ S ξ (cid:107) 2 =
O ( (cid:113) ( 1 δ +1) n · n − (1 − t ) ) with probability of at least 1 − δ .
Since R = n ( 1 α + κ )(1 − t ) , we have
[Equation]
Since ξ is arbitrary, we have (cid:107) ( I
[Equation]
[Equation]
+
[Equation]
Since 0 ≤ q <
[ α − (1+2 τ
[Equation]
(1 − t )](2 β − 1) 4 αand 0 < κ < α − 1 − 2 τ
+(1+2 τ ) t
α 2 (1 − t ) , we can choose κ < α − 1 − 2 τ
+(1+2 τ ) t
α 2 (1 − t ) and κ is arbitrarily close to κ < α − 1 − 2 τ
+(1+2 τ ) t
α 2 (1 − t )
such that
[Equation]
Then we have (1 − 2 β )(1 − t )
[Equation]
+ q< 0 .
From (145) and (148), we have
[Equation]
Hence G 2 ,S ( D n ) =
+ o (1) 2 σ 2 (cid:107) ( I + 1 σ 2 Λ S
Φ TS Φ S )
− 1 µ S (cid:107) 22 = 12 σ 2 µ 20 + o (1) .
Then by (144),
G 2 ( D n )= 12 σ 2 µ 20 + o (1)+ ˜ O (cid:0) ( 1 δ +1) nS max { 1 / 2 − β, 1 − α } (cid:1) .
Choosing S = n max (cid:26) 1 , − t ( α − 1 − 2 τ ) , (cid:18) 1+ q +min { 2 , 2 β − 1 α } min { β − 1 / 2 ,α − 1 − 2 τ } +1 (cid:19) (1 − t ) (cid:27) , we get the result.
Proof of Theorem 11.
According to Lemma 42, G 2 ( D n ) = 12 σ 2 µ 20 + o (1) .
By Lemma 39, we have
G 1 ( D n )=
Θ( n (1 − α )(1 − t ) α ) .
Then E (cid:15) G ( D n )=
G 1 ( D n )+ G 2 ( D n )=
1 2 σ 2 µ 2
+ o (1) .
D. P ROOFS RELATED TO THE EXCESS MEAN SQUARED GENERALIZATION ERROR
Proof of Theorem 12.
For µ 0 =0 , we can show that
[Equation]
According to (139) from the proof of Lemma
[Equation]
the truncation procedure (133) and (143)
[Equation]
with probability of at least 1 − δ we have (cid:107) ( I +
[Equation]
where k
=
[Equation]
According to (121) and (126) from the proof of Lemma
[Equation]
the truncation procedure (115)
[Equation]
(141) and (142)
[Equation]
with probability of at least 1 − δ we have
[Equation]
Combining the above two equations we get
[Equation]
When µ 0 > 0 , according to (149) in the proof of Lemma 42 and the truncation procedure (133), with probability of at least 1 − δ we have
[Equation]