On the inability of Gaussian process regression to optimally learn compositional functions
Abstract
We rigorously prove that deep Gaussian process priors can outperform Gaussian process priors if the target function has a compositional structure.
To this end, we study information-theoretic lower bounds for posterior contraction rates for
Gaussian process regression in a continuous regression model.
We show that if the true function is a generalized additive function, then the posterior based on any mean-zero Gaussian process can only recover the truth at a rate that is strictly slower than the minimax rate by a factor that is polynomially suboptimal in the sample size n .
Introduction
Hierarchical methods are at the heart of modern machine learning and provide state of the art performance for a variety of different problems.
While the role of the depth in deep neural networks has been extensively investigated [37, 44, 45, 24, 33], the theory underlying deep Gaussian process priors is still in its infancy.
In this work we identify a structured recovery problem for which deep
Gaussian process priors are known to achieve near optimal behavior.
By proving that for this problem any Gaussian process prior will lead to a sub-optimal posterior shrinkage behavior, we provide a theoretical justiﬁcation for the use of deep methods for Bayesians.
Gaussian process (GP) priors are arguably the most widely used Bayesian nonparametric priors in machine learning, having found success in multiple settings [26].
Reasons for their popularity include their interpretability, ability to ﬂexibly model target functions, model spatial or temporal correlations, incorporate prior knowledge such as stationarity, periodicity or smoothness, and their provision of uncertainty quantiﬁcation.
This work establishes information-theoretic bounds on the inability of
GPs to optimally model generalized additive models, thereby quantifying certain limitations of GP methods.
We will study the performance of GPs in a continuous analogue of nonparametric regression via quantifying the speed of posterior concentration/contraction around the true regression function.
This is a frequentist assessment that measures the typical behaviour of the posterior distribution under the true generative model.
The performance of Bayesian methods is known to be sensitive to the choice of prior, particularly in high- and inﬁnite-dimensional settings [11, 20], and it is therefore crucial to understand under what conditions and prior calibrations Bayesian inference is reliable.
The frequentist analysis of Bayesian methods has become a standard tool for such tasks in both the statistics and machine learning literatures (e.g. [36, 25, 43, 42, 6, 27, 28]), see the monograph [12].
Precise deﬁnitions and further discussion are provided below.
If the regression function is a generalized additive model, it has been shown that deep neural networks with properly chosen network architecture achieve the optimal minimax rate of convergence [32].
We show here that for this structural constraint, any mean-zero Gaussian process, irrespective of the choice of covariance kernel, will concentrate around the truth at a rate that is strictly slower than the minimax rate by a factor that is polynomially suboptimal in the sample size n .
Thus any posterior GP places a signiﬁcant portion of its posterior probability on functions that are further from the truth in
L 2 -distance than the minimax rate of estimation, which has negative implications for both Bayesian estimation and uncertainty quantiﬁcation.
In contrast, it has recently been shown that deep Gaussian processes [23, 8], which consist of iterations of Gaussian processes and can be viewed as a Bayesian analogue of deep neural networks, are able to attain the minimax rate of contraction for arbitrary compositional classes [10].
As these compositional classes contain generalized additive models as a special case, we have shown that deep GPs provably outperform GPs from a statistical perspective and should thus be preferred.
In related work, it has been established that (not necessarily Bayesian) linear estimators can only attain polynomially suboptimal linear estimation rates over certain spatially inhomogeneous Besov classes [9] or for functions with discontinuities [21].
Agapiou and Wang [1] use the results of [9] to show that GPs also cannot concentrate faster than the linear rate for such Besov classes, see also [31] for some related examples.
Castillo [4] establishes lower bounds for concentration rates in terms of abstract conditions on the ‘concentration function’ of the GP, which can be veriﬁed for certain speciﬁc GP choices.
The lower bound ideas of [9, 21] have also been extended to various spatially inhomogeneous and discontinuous functions settings to demonstrate the theoretical advantages of deep learning over linear estimators [15–17, 34, 35, 38].
Finally, for generalized additive models, [32] shows that (possibly nonlinear) wavelet methods can only achieve suboptimal rates.
Most of the above works exploit the results from [9, 21] showing that minimax rates for linear estimators are slower than full minimax rates in these settings.
Since linear rates are unavailable for generalized additive functions, and seem difﬁcult to prove using existing techniques, we instead use a different and novel proof approach.
We exploit the speciﬁc structure of the GP posterior mean in regression to construct a direct lower bound for its prediction risk.
In an effort to develop the new tools needed to solve such problems, we further provide a second proof in the supplement that involves ﬁrst reducing the regression setting to a simpliﬁed sparse sequence model in which we derive linear rates.
While these do not imply linear rates in the full model we consider, they prove sufﬁcient to construct lower bounds for arbitrary GP posterior means.
Main results 2.1 Problem setup and posterior contraction rates
In the usual multivariate nonparametric regression model we observe n training samples ( X 1 , Y 1 ) , . . .
, ( X n , Y n ) arising as
[Equation]
where the design points X i are either independent and uniformly distributed random variables on [0 , 1] d or equally spaced lattice points in [0 , 1] d .
To simplify technical arguments due to the discretization and provide a clearer exposition, a standard approach in statistical theory is to consider instead the ‘continuous’ analogue of this model [19].
The discrete model (1) is asymptotically equivalent [30] (as n → ∞ ) to observing a realisation
Y =
Y n =
( Y x : x ∈ [0 , 1] d ) of the multidimensional Gaussian white noise model
[Equation]
where W =
( W x : x ∈ [0 , 1] d ) is a d -dimensional Brownian motion.
For large n , the two models thus behave identically from a statistical perspective.
Notation : For a domain D and 1 ≤ p ≤ ∞ , denote by L p ( D ) the space of all measurable functions f :
D → R that have ﬁnite norm (cid:107) f (cid:107)
L p ( D ) :
= ( (cid:82) | f ( u ) | p du )
1 /p if p < ∞ , and are essentially bounded on D if p = ∞ .
If there is no ambiguity, we write L 2 for L 2 [0 , 1] d and (cid:104) f, g (cid:105) 2 = (cid:82) [0 , 1] d f ( u ) g ( u ) du for the corresponding inner product.
Let P f = P nf denote the probability distribution of Y =
Y n arising from (2) with corresponding expectation E f .
We consider the Bayesian setting where we assign to f a (possibly n -dependent) mean-zero Gaussian process prior Π = Π n on L 2 [0 , 1] d with covariance kernel K , i.e. Π is a (Borel) Gaussian probability measure on L 2 [0 , 1] d such that for f ∼ Π , we have
[Equation]
The posterior Π n ( ·| Y ) is then computed as usual using Bayes formula and is again a GP by conjugacy, exactly as in the discrete model (1) [26], see (7) for the exact expression.
We make the following frequentist assumption:
Assumption 1.
There is a true f 0 ∈ L 2 [0 , 1] d generating the data Y ∼ P f
according to (2) .
We study the behaviour of the posterior distribution Π n ( ·| Y ) under Assumption 1, in which case it can be treated as a random probability distribution whose (frequentist) randomness depends on Y .
We next introduce the notion of uniform posterior contraction.
Deﬁnition 1.
We say that the posterior contracts about f 0 at rate ε n → 0 uniformly over a sequence of classes F n ⊂ L 2 [0 , 1] d if as n → ∞
[Equation]
Posterior contraction is often stated in the equivalent form
[Equation]
as n → ∞ , which says that the posterior puts all but a vanishingly small amount of probability on
L 2 -balls of shrinking radius ε n about the true f 0 generating the data [12].
We are typically interested in the size of the smallest such L 2 -ball, i.e. the fastest rate ε n → 0 such that Deﬁnition 1 holds.
Such results not only quantify the typical distance between a point estimator (e.g. posterior mean or median) and the truth, but also the typical spread of the posterior about the truth.
Ideally, most of the posterior probability should concentrate on a ball centered at the true f 0 with radius proportional to the minimax estimation rate, see Section C.1 for more details.
Indeed, since posterior contraction at rate ε n automatically yields a point estimator with convergence rate ε n ([12], Theorem 8.7), the minimax rate provides an information-theoretic lower bound for ε n .
Posterior contraction at a fast rate is a necessary condition for statistically good Bayesian estimation and uncertainty quantiﬁcation.
We use the E f 0 -expectation formulation of contraction to quantify the uniformity over f 0 ∈ F n needed to make our lower bounds precise and meaningful.
Such uniformity captures that the true f 0 is unknown and rules out unrealistic and trivial situations, such as taking a GP with prior mean equal to the true f 0 and arbitrarily small covariance.
Since we consider worst case (uniform) rates in Deﬁnition 1 and it is well-known that the covariance kernel K of the GP determines the characteristics of the corresponding GP [26], including its posterior contraction rates [40], we without loss of generality restrict to mean-zero GPs.
There is by now an extensive literature on proving upper bounds for contraction rates [12], including for GPs [40].
These contraction rates are usually uniform over suitable function classes F n , although the results might not be formulated in this way in the literature.
2.2 Lower bounds for posterior concentration in Gaussian processes regression
While the classical approach in nonparametric statistics is to make smoothness assumptions on the regression function f 0 , recent work in the deep learning literature has studied compositional assumptions.
A special case are generalized additive models of the form
[Equation]
with g 1 , . . .
, g d , h univariate and Lipschitz continuous functions such that all Lipschitz constants are bounded by Λ and (cid:107) g 1 (cid:107) L ∞ [0 , 1] , . . .
, (cid:107) g d (cid:107) L ∞ [0 , 1] , (cid:107) h (cid:107) L ∞ ( R ) ≤ Λ .
The class of all functions of this form is denoted by G (Λ) and we deﬁne G = G (1) .
Generalized additive models are among the most popular structural constraints in function estimation [14].
Equation (13) in [32] and the subsequent discussion shows that the minimax estimation rate R n ( G (Λ)) satisﬁes
[Equation]
for some C, c > 0 .
Up to logarithmic factors, this rate is obtained by carefully calibrated deep GP priors that ﬁrst assign a hyperprior to different compositional structures and subsequently put GP priors on all functions in the function composition [10].
We are now ready to state the main lower bound results of this paper, which we emphasize applies to any GP, irrespective of the choice of covariance kernel (which may itself depend on n ).
Theorem 1.
Let Π n be any sequence of mean-zero Gaussian process priors on L 2 [0 , 1] d .
For any 0 < δ < 1 / 4 and n ≥ N ( d, δ ) large enough, the corresponding posterior distributions satisfy
[Equation]
where C d = [Equation]
d [Equation] 4 d .
In particular, if the posterior distributions contract at rate ε n uniformly over G , that is
[Equation]
ε n | Y ) →as n → ∞ , then ε n > C d n − 2+ d 4+4 d .
For dimension d ≥ 3 , the above rate is strictly slower than the minimax rate n − 1 / 3 by the polynomial factor n d − 2 12+12 d .
As the domain dimension d tends to inﬁnity, our lower bound is approximately n − 1 / 4 , in which case Theorem 1 says the GP posterior will place at least probability 1 / 4 − δ on posterior draws f ∼ Π n ( ·| Y ) having suboptimal reconstruction error (cid:107)
f − f [-
 ∥-]0 (cid:107) L 2 (cid:38) n − 1 / 4 .
In contrast, posteriors based on properly calibrated deep GPs will place almost all their probability on draws with minimax optimal reconstruction error of order n − 1 / 3 [10].
Thus GP posteriors provide poor reconstructions of f 0 ∈ G compared to deep GPs.
Theorem 1 says that no GP can attain a faster rate than n − 2+ d 4+4 d .
It does not say that a given GP even attains this suboptimal rate, and speciﬁc GPs can perform much worse than this bound.
We in fact conjecture that our lower bound can be improved to show that even this rate is not attainable by any
GP, see Section 2.3 for more discussion.
Nonetheless the main message from Theorem 1 is already clear, namely that GPs underperform in this setting by an already signiﬁcant factor that is polynomial in the sample size n .
The last result says that generic posterior draws will be poor reconstructions of f 0 with signiﬁcant posterior probability.
The same holds for the posterior mean.
Theorem 2.
Let Π n be any sequence of mean-zero Gaussian process priors on L 2 [0 , 1] d .
Then the corresponding posterior means ¯ f n = E Π n [ f | Y ] satisfy for n ≥ 2( d + 2)! ,
[Equation]
where C (cid:48) d =
[Equation]
[Table]
Theorem 2 states that the posterior mean ¯ f n will also be a suboptimal reconstruction.
Since the posterior is also Gaussian by conjugacy, and hence centered at ¯ f n , this captures the notion that posteriors draws from the ‘center’ of the posterior will also be poor - this is thus generic behaviour and not an artifact of unusual behaviour in the posterior tails.
Our results also have implications for Bayesian uncertainty quantiﬁcation (UQ) through the use of credible sets, which is one the main motivations for using GPs.
Bayesian credible sets are often used as frequentist conﬁdence sets, a practice which can be rigorously justiﬁed in low dimensional settings ([39], Section 10.2), but is much more subtle in high-dimensional settings like GPs [11].
Whether a resulting credible set contains the truth will depend on the size of the posterior standard deviation relative to the distance between the posterior mean and the truth, which Theorem 2 says is greater than the minimax rate with high-probability.
In the best case scenario, the former is greater than the latter and thus the credible set will provide honest and reliable UQ.
However, in this case the posterior standard deviation must be at least of the order of the contraction rate, and hence the resulting credible sets will be (polynomially) much too large and uninformative.
In the worse case, when the posterior standard deviation is smaller, the credible sets will not contain the truth and provide misleading (useless) UQ.
Note that we are considering UQ for the entire function f 0 - it may still be possible for the GP posterior to provide efﬁcient UQ for certain low-dimensional functionals of interest, which is itself a current topic of research [5, 29].
The proofs are based on a number of inequalities that are easy to state.
Relying on an exponential concentration inequality and Anderson’s inequality, we show that Theorem 1 follows from the lower bound for the posterior mean in Theorem 2 (cf.
Section 4.1).
For Theorem 2 we provide two different proofs.
In the ﬁrst, we lower bound the L 2 -risk of the posterior mean over arbitrary ﬁnite subsets { f 1 , . . .
, f m } ⊂ L 2 [0 , 1] d (cf.
Lemma 7), with the lower bound expressed in terms of expansions of the functions
f i in the basis coming from the Karhunen-Loève expansion of the GP.
We then construct a suitable collection of orthogonal generalized additive models for which the lower bound is maximized independently of the choice of GP basis, as is needed to deal with arbitrary GPs (cf.
Section 4.3).
Given a speciﬁc choice of GP, our lower bound techniques can yield more reﬁned results by tailoring the alternatives f 1 , . . .
, f m to be especially poorly aligned (difﬁcult to learn) for that speciﬁc covariance function, see Section 2.3.
In a second proof of Theorem 2, presented in Section B in the supplement, we reduce the problem of studying orthogonal generalized additive models to the sub-problem of studying the risk of linear estimators in a one-sparse sequence model (cf.
Section B.1).
For the latter, we exactly compute the linear minimax risk in Lemma 9, which, by exploiting the orthogonality, in turn implies the lower bound in Theorem 2.
This second proof sheds additional light on the limitations of GP regression by furnishing new tools to study problems that can be connected to sparse models.
2.3 A sharper lower bound for a speciﬁc class of Gaussian processes
While Theorem 1 holds for arbitrary GPs, we now show that our methods can yield sharper lower bounds in the case of speciﬁc GPs, implying an even stronger suboptimality in the associated reconstruction error.
We illustrate this by considering Gaussian wavelet series priors.
Wavelets have been extensively used in Bayesian nonparametrics to construct priors in function spaces, including applications in inverse problems [41, 2], and image and signal processing [3, 18] among the others; see e.g. [22] for an overview.
Let { ψ γ , γ ∈ Γ } , with Γ = { γ = ( j, k ) :
j = − 1 , 0 , 1 , . . .
, k ∈ I j } , be a compactly supported wavelet basis of L 2 ( R ) restricted (with boundary correction) to [0 , 1] , cf.
[7] or also [13, Chapter 4].
The collection { ψ γ , γ = ( γ 1 , . . . ,
γ d ) ∈ Γ d } , with Γ d
= Γ ×· · ·× Γ ( d times) and ψ γ ( x 1 , . . .
, x d ) := (cid:81) di =1 ψ γ i ( x i ) provides a tensor product wavelet basis of L 2 [0 , 1] d .
Consider a mean-zero Gaussian wavelet series prior on L 2 [0 , 1] d with Karhunen-Loève expansion
[Equation]
for some arbitrary index set I ⊆
Γ d , coefﬁcients λ γ > 0 and ξ γ ∼ iid N (0 , 1) .
Compared to the proof of the general lower bound in Theorem 1, ﬁxing a speciﬁc GP basis allows to consider compositional functions that are especially difﬁcult for that GP to approximate.
For wavelet bases, such approximation theory was developed in [32], which we employ to obtain the next lower bound.
Theorem 3.
Let Π n be any sequence of mean-zero Gaussian wavelet series priors on L 2 [0 , 1] d , deﬁned as in (4) with possibly n -dependent λ γ > 0 .
For any 0 < δ < 1 / 4 and n ≥ N ( d, δ ) large enough, the corresponding posterior distributions satisfy
[Equation]
where C = C ( d, ψ ) > 0 only depends on d and the wavelet basis.
Furthermore, the corresponding posterior means ¯ f n = E Π n [ f | Y ] satisfy for all n
[Equation]
where C (cid:48) =
C (cid:48)
( d, ψ ) > 0 only depends on d and the wavelet basis.
Thus, for the class of Gaussian wavelet series priors, Theorem 3 provides a sharper lower bound than Theorem 1, as n − 12+ d (cid:29) n − 2+ d 4+4 d for all d .
Furthermore, the former rate deteriorates as d grows, showing that the performance of Gaussian wavelet priors suffers from a form of curse of dimensionality.
In particular, when d is large, the fastest rate n − 12+ d achievable by Gaussian wavelet priors is much slower than the minimax rate n − 1 / 3 achieved by deep GPs.
Discussion
In this work, we showed that no mean-zero Gaussian process can attain the minimax estimation rate over a class of generalized additive models.
We proved this by developing new tools for lower bounding posterior contraction rates for GPs.
We provided two different proofs, each of which sheds different light on these phenomena.
It remains open whether even sharper lower bounds can be obtained.
In this context, an important question is whether all GPs suffer from a curse of dimensionality in their posterior contraction rate, which we showed was incurred for a speciﬁc wavelet-based GP.
Our lower bound n − (2+ d ) / (4+4 d ) for arbitrary GPs in Theorem 1 is never slower than n − 1 / 4 , irrespective of the domain dimension d .
While the obtained lower bound already demonstrates the suboptimality of GPs by a polynomial factor in n , say when compared to deep GPs, our stronger lower bound n − 1 / (2+ d ) for Gaussian wavelet series priors becomes arbitrarily slow with increasing d.
We thus proved that for this subclass of GPs, a stronger curse of dimensionality is unavoidable
To avoid the curse of dimensionality, a common two-stage approach is to ﬁrst map the input features to a learned lower dimensional space and then apply Gaussian process regression to these new features.
Our results do not apply to this scenario, in particular when the feature extraction map can be be nonlinear, since the transformed underlying function may no longer be compositional in terms of the new learned features.
We expect similar results to hold for non-uniformly distributed input variables and heteroscedastic noise.
Let σ be a positive function and ρ a density on [0 , 1] d .
If ρ, σ are bounded away from zero and inﬁnity, we conjecture that, up to constants, the same lower bounds can be derived working in the more general nonparametric regression model (1) with, conditionally on X i , independent noise variables w i | X i ∼ N (0 , σ 2 ( X i )) and X i ∼ iid ρ .
Ideally, one wants to develop a broader understanding of which structures in the data-generating distribution or regression function (deep) Gaussian processes can or cannot adapt to.
In order to understand the advantages of deep learning there is an extensive theoretical literature comparing shallow to deep nets.
More reﬁned versions show that for certain approximation rates a minimal network depth is necessary [24].
To gain some insights into the role of depth in deep Gaussian processes, it would be desirable to explore similar limitations if the depth is ﬁxed and a highly structured function needs to be learned.
Proofs
We ﬁrst start with some preliminary facts concerning GPs and the Gaussian white noise model (2).
For Π a GP prior on L 2 [0 , 1] d , we can express its covariance kernel K in (3) in terms of the normalized eigenfunctions { φ k : k ≥ 1 } of the compact, self-adjoint, trace-class operator g
(cid:55)→ (cid:82) [0 , 1] d K ( · , y ) g ( y ) dy on L 2 [0 , 1] d and its associated eigenvalues λ k > 0 , which are summable, via the expansion K ( x, y )
= (cid:80) ∞
k =1 λ k φ k ( x ) φ k ( y ) .
In particular, we can realize the Gaussian random element f ∼ Π through its series expansion
[Equation]
known as the Karhunen-Loève expansion .
The corresponding reproducing kernel Hilbert space (RKHS)
H of Π is
[Equation]
For any orthonormal basis { φ k : k ≥ 1 } of L 2 [0 , 1] d , in particular the Karhunen-Loève basis of the prior (5), the Gaussian white noise model (2) is equivalent to observing the sequence model ( Y k : k ≥ 1) :
[Equation]
for k = 1 , 2 , 3 , . . .
and where w k iid ∼ N (0 , (cid:107) φ k (cid:107) 2 L 2 ) =
N (0 , 1) .
To be precise, the observations ( Y k : k ≥ 1) in the last equation and the original white noise model (2) are equivalent in the sense that each can be perfectly recovered from the other, see Section C.2 in the supplement for more details.
Viewing the prior Π through its series expansion (5), Π can be viewed as a prior on the space of coefﬁcients in the basis expansion of { φ k } leading to the prior distribution f
= ( θ k ) k ∼ ⊗ ∞ k =1 N (0 , λ k ) .
Denoting by P f = P nf the distribution of the sequence representation (6), we have likelihood
P f = ⊗ ∞ k =1 N (
[Equation]
k , 1 /n ) . Using this representation, one can use posterior conjugacy to show that the posterior takes the form f = ∞ (cid:88) k = θ k
[Equation]
see Section C.2 in the supplement for full details.
In particular, the posterior mean is
[Equation]
4.1 Reducing the lower bound for posterior contraction to one for the L 2 -risk of the posterior mean
We will show that a lower bound on the L 2 -risk of the posterior mean implies a corresponding lower bound for the posterior contraction rate, so that it will sufﬁce to consider the former.
The proofs of the following lemmas are deferred to Section A.1 in the supplement due to space constraints.
We start by showing that the squared estimation error of the posterior mean is of the order of its expectation with high probability.
Lemma 4.
Let f 0 ∈
L 2 [0 , 1] d , ¯ f n = E Π [ f | Y ] denote the posterior mean based on a mean-zero
Gaussian process prior Π on L 2 [0 , 1] d and set µ 2 n = E f [-
 ∥-]0 (cid:107) ¯ f n − f 0 (cid:107) 2 L 2 .
Then for n ≥ 1 ,
[Equation]
We next show that a contraction rate for Gaussian process priors implies that the corresponding posterior means converge to the truth at the same rate with frequentist probability tending to one.
Lemma 5.
Let Π n be a sequence of mean-zero Gaussian process priors on L 2 [0 , 1] d .
Then the corresponding posterior means ¯ f n = E Π n [ f | Y ] satisfy
[Equation]
for any sequence γ n and n ≥ 1 .
Combining the last two lemmas allows us to transfer an L 2 -risk bound for the posterior mean to one for posterior contraction.
Lemma 6.
Let f 0 ∈
L 2 [0 , 1] d , Π n be a sequence of mean-zero Gaussian process priors with corresponding posterior means ¯ f n = E Π n [ f | Y ] , and set µ 2 n = E f [-
 ∥-]0 (cid:107) ¯ f n − f 0 (cid:107) 2 L 2 .
Then for n ≥ 1 ,
[Equation]
Furthermore, suppose F n ⊂ L 2 satisfy sup f 0 ∈F n E f [-
-]0 (cid:107) ¯ f n − f
[Equation]
[-¯ f n − f 
 ∥-] 2 L 2 ≥ γ 2 n > 0 for some sequence γ n for which nγ 2 n → ∞ as n → ∞ .
Then for any 0 < δ < 1 / 4 and n such that nγ 2 n ≥ 32 log (cid:16)
[Equation]
In particular, the posteriors cannot contract at rate γ n / 5 , uniformly over F n .
4.2 A general lower bound for the L 2 -risk of the posterior mean
By Lemma 6, it sufﬁces to prove a lower bound for the L 2 -risk of the posterior mean.
We next prove such a general lower bound.
Lemma 7.
Let ¯ f n be an estimator of the form (8) with corresponding orthonormal basis { φ k : k ≥ 1 } .
For functions f 1 , . . .
, f m ∈ L 2 [0 , 1] d ,
[Equation]
Moreover, if f 1 , . .
, f m are orthogonal and max j [Equation] f j [Equation]
2 L 2 ≤ m/n, then
[Equation]
In particular, if cm/n ≤ (cid:107) f j (cid:107) 2 L 2 ≤ m/n for all j and some 0 < c ≤ 1 , then one can take the lower bound to be cm/n.
Proof.
Using the bias-variance decomposition, the expected prediction loss of an estimator of the form (8) is
[Equation]
Setting a k := nλ k / (1 + nλ k ) and T k = 1 m (cid:80) mj
=1 (cid:104) f j , φ k (cid:105) 2 L 2 , the last equation yields
[Equation]
Optimizing over a k shows that the right-hand side is minimized at a ∗ k =
nT k / (1 + nT k ) .
For this choice, we obtain using 1 + nT k ≥ 1 ∨ ( nT k ) , max
[Equation]
proving the ﬁrst claim.
To see the second claim, observe that (cid:107) f j (cid:107) 2 L 2 ≤ m/n and the orthogonality of the f j imply
T k = [Equation] m m
[Equation]
Therefore T k ∧ 1 /n =
T k and using (9) and that ( φ k ) k is a basis, max [Equation] [Equation] (cid:88)
[Equation]
4.3 Construction of local functions and proof of Theorems 1 and
Lemma 8.
Let m =
k d for some integer k ≥ 1 .
Then there exist functions f 1 , . .
. , f m ∈ G having disjoint support and satisfying (cid:107) f j (cid:107) 2 L 2 =
[Equation] [Equation] d [Equation]
m − 2+ dd for j = 1 , . .
, m .
Proof.
For (cid:96) = 1 , . . .
, k, consider the midpoints z (cid:96) :
= ( (cid:96) − 1 / 2) /k of the intervals [( (cid:96) − 1) /k, (cid:96)/k ] .
Overall, there are m = k d points in the grid
[Equation]
Consider the functions,
[Equation]
These functions can be viewed as generalized additive models f a ( x )
=
h ( (cid:80) di
=1 g ai ( x i )) with h ( u ) =
(1 / (2 k ) − u ) + and g ai ( x i ) =
| x i − a i | .
All these functions have Lipschitz constant bounded by 1 .
Moreover, max a ∈ I,i =1 ,...,d (cid:107) g ai (cid:107) L ∞ [0 , 1] ≤ 1 and (cid:107) h (cid:107) L ∞ ( R )
= 1 / (2 k ) ≤ 1 .
Thus, f a ∈ G .
Since | x − a | 1 ≥ | x − a | ∞ , the support of the function f a is contained in the hypercube × d i =
[ a i − 1 / (2 k ) , a i + 1 / (2 k )] .
Since a i is of the form ( (cid:96) − 1 / 2) /k with (cid:96) an integer, the functions f a must have disjoint support and are therefore orthogonal.
It thus remains to compute the L 2 norm of these functions, which by translation are the same as that of f 0 ( x ) =
(1 / (2 k ) − | x | 1 )
+ on R d .
Note that the support of f 0 can be decomposed as supp ( f 0 ) = ∪ (cid:15) ∈{− 1 , 1 } d R (cid:15) , where
[Equation]
Since the L 2 -norm of f 0 on each of these 2 d quadrants is equal, we have
[Equation]
Using repeatedly that (cid:82) M − r
[Equation]
M − r − y
[Equation]
t dy = 1 t +1 ( M − r ) t +1 for 0 ≤ r ≤ M and t > 0 , we obtain
[Equation]
Proof of Theorems 1 and 2.
Let r d = [Equation]
1 2 d +2 (cid:101) and m = k d .
By Lemma 8, there exist orthogonal functions f 1 , . . .
, f m ∈ G with (cid:107) f j (cid:107) 2 L 2 = r d m − 2+ dd .
For such a choice of k and m , cm/n ≤ (cid:107) f j (cid:107) 2 L 2 ≤ m/n holds if and only if
[Equation]
after rearranging the inequalities.
Now the function ϕ ( x ) =
x/ (cid:100) x (cid:101) satisﬁes ϕ ( x ) ≥ kk +1 for all x ≥ k ∈ N , so that the right side is lower bounded by 1/2 for ( r d n ) 1 2 d +2 ≥ 1 , or equivalently n ≥ 1 /r d .
For n ≥ 1 /r d , it therefore holds that cm/n ≤ (cid:107) f j (cid:107) 2 L 2 ≤ m/n with c = (1 / 2) {+
+} d +2 .
Applying Lemma 7 thus gives the risk bound
[Equation]
which proves Theorem 2.
Set
[Equation]
[Equation]
and note that N ( d, δ ) ≥ 1 /r d .
Theorem 1 then follows from the second last display and applying
Lemma 6 with F n = { f 1 , . . .
, f m } ⊂ G and γ 2 n = 2 − (2 d +2)
r d 2 d +2 d n − 2+ d 2+2 d .
Acknowledgments.
We are grateful to three anonymous referees for helpful comments that improved the manuscript.
The research of MG has been supported by the European Research Council under ERC grant agreement No.834275 (GTBB).
The research of JSH has been supported by the
NWO/STAR grant 613.009.034b and the NWO Vidi grant VI.Vidi.192.021.