A DAPTIVITY OF DEEP R E LU NETWORK FOR LEARNING IN B ESOV AND MIXED SMOOTH B ESOV SPACES : OPTIMAL RATE AND CURSE OF DIMENSIONALITY
A BSTRACT
Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior ﬂexibility and adaptivity of deep learning.
To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness.
The Besov space is a considerably general function space including the H ¨ older space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness.
Throughout the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any nonadaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones.
In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space.
We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality.
These results support high adaptivity of deep learning and its superior ability as a feature extractor.
I NTRODUCTION
Deep learning has shown great success in several applications such as computer vision and natural language processing.
As its application range is getting wider, theoretical analysis to reveal the reason why deep learning works so well is also gathering much attention.
To understand deep learning theoretically, several studies have been developed from several aspects such as approximation theory and statistical learning theory.
A remarkable property of neural network is that it has universal approximation capability even if there is only one hidden layer (Cybenko, 1989; Hornik, 1991; Sonoda & Murata, 2017).
Thanks to this property, deep and shallow neural networks can approximate any function with any precision (of course, the meaning of the terminology “any” must be rigorously deﬁned like “any function in L 1 ( R ) ”).
A natural question coming next is its expressive power.
It is shown that the expressive power of deep neural network grows exponentially against the number of layers (Montufar et al., 2014; Bianchini & Scarselli, 2014;
Cohen et al., 2016; Cohen & Shashua, 2016; Poole et al., 2016) where the “expressive power” is deﬁned by several ways.
The expressive power of neural network can be analyzed more precisely by specifying the target function’s property such as smoothness.
Barron (1993; 1994) developed an approximation theory for functions having limited “capacity” that is measured by integrability of their Fourier transform.
An interesting point of the analysis is that the approximation error is not affected by the dimensionality of the input.
This observation matches the experimental observations that deep learning is quite effective also in high dimensional situations.
Another typical approach is to analyze function spaces with smoothness conditions such as the H¨older space.
In particular, deep neural network with the ReLU activation (Nair & Hinton, 2010; Glorot et al., 2011) has been extensively studied recently from the view point of its expressive power and its generalization error.
For example,
Yarotsky (2017) derived the approximation error of the deep network with the ReLU activation for
[Table]
functions in the H¨older space.
Schmidt-Hieber (2018) evaluated the estimation error of regularized least squares estimator performed by deep ReLU network based on this approximation error analysis in a nonparametric regression setting.
Petersen & Voigtlaender (2017) generalized the analysis of
Yarotsky (2017) to the class of piece-wise smooth functions.
Imaizumi & Fukumizu (2018) utilized this analysis to derive the estimation error to estimate the piece-wise smooth function and concluded that deep leaning can outperform linear estimators in that setting.
Although these error analyses are standard from a nonparametric statistics view point and the derived rates are known to be (near) minimax optimal, the analysis is rather limited because they are given mainly based on the H¨older space.
However, there are several other function spaces such as the Sobolev space and the space of ﬁnite total variations.
A comprehensive analysis to deal with such function classes from a uniﬁed view point is required.
In this paper, we give generalization error bounds of deep ReLU networks for a Besov space and its variant with mixed smoothness , which includes the H ¨ older space, the Sobolev space, and the function class with total variation as special cases.
By doing so, (i) we show that deep learningachieves the minimax optimal rate on the Besov space and notably it outperforms any linear estimator such as the kernel ridge regression, and (ii) we show that deep learning can avoid the curse of dimensionality on the mixed smooth Besov space and achieves the minimax optimal rate.
As related work, Mhaskar & Micchelli (1992); Mhaskar (1993); Chui et al. (1994); Mhaskar (1996); Pinkus (1999) also developed an approximation error analysis which essentially leads to analyses for Besov spaces.
However, the ReLU activation is basically excluded and comprehensive analyses for the
Besov space have not been given.
As a summary, the contribution of this paper is listed as follows:
(i) To investigate adaptivity of deep learning, we give an explicit form of approximation and estimation error bounds for deep learning with the ReLU activation where the target functions are in the Besov spaces ( B sp,q ) for s > 0 and 0 < p, q ≤ ∞ with s > d (1 /p − 1 /r )
+ where L r -norm is used for error evaluation.
In particular, deep learning outperforms any linear estimator such as kernel ridge regression if the target function has highly spatial inhomogeneity of its smoothness.
See Table 1 for the overview.
(ii) To investigate the effect of dimensionality, we analyze approximation and estimation problems in so-called the mixed smooth Besov space by ReLU neural network.
It is shown that deep learning with the ReLU activation can ease the curse of dimensionality and achieve the near minimax optimal rate.
The theory is developed on the basis of the sparse grid technique (Smolyak, 1963).
See Table 2 for the overview.
S ET UP OF FUNCTION SPACES
In this section, we deﬁne the function classes for which we develop error bounds.
In particular, we deﬁne the Besov space and its variant with mixed smoothness.
The typical settings in statistical learning theory is to estimate a function with a smoothness condition.
There are several ways to characterize “smoothness.”
Here, we summarize the deﬁnitions of representative functional spaces that are appropriate to deﬁne the smoothness assumption.
Let Ω ⊂ R d be a domain of the functions.
Throughout this paper, we employ Ω =
[0 , 1] d .
For a function f : Ω → R , let (cid:107) f (cid:107) p :
= (cid:107) f (cid:107) L p (Ω) :=
( (cid:82) Ω | f | p d x ) 1 /p for 0 < p < ∞
For p =
[Table]
∞ , we deﬁne (cid:107) f (cid:107) ∞ :
= (cid:107) f (cid:107) L ∞ (Ω) :
= sup x ∈ Ω | f ( x ) | .
For α ∈ R d , let | α | = (cid:80) dj
=1 | α j | .
Let
C 0 (Ω) be the set of continuous functions equipped with L ∞ -norm: C 0 (Ω) :=
{ f : Ω → R | f is continuous and (cid:107) f (cid:107) ∞ < ∞} 2 .
For α ∈ Z d + , we denote by D α f ( x ) =
[Table]
∂
αd
[Table]
d ( x ) 3 .
Deﬁnition 1 (H¨older space ( C β (Ω) )) .
Let β > 0 with β (cid:54)∈ N be the smoothness parameter.
For an m times differentiable function f : R d → R , let the norm of the H¨older space C β (Ω) be (cid:107) f (cid:107) C β := max | α |≤ m (cid:13)(cid:13)
D α f (cid:107) ∞ + max | α | = m sup x,y ∈ Ω | D α f ( x ) − D α f ( y ) | | x − y | β − m , where m = (cid:98) β (cid:99) (the largest integer less than β ).
Then, ( β -)H¨older space C β (Ω) is deﬁned as C β (Ω) =
{ f | (cid:107) f (cid:107) C β < ∞} .
The parameter β > 0 controls the “smoothness” of the function.
Along with the H ¨ older space, the
Sobolev space is also important.
Deﬁnition 2 (Sobolev space ( W kp (Ω)) ) .
Sobolev space ( W kp (Ω)) with a regularity parameter k ∈
N and a parameter 1 ≤ p ≤ ∞ is a set of functions such that the Sobolev norm (cid:107) f (cid:107) W kp :=
( (cid:80) | α |≤ k (cid:107) D α f (cid:107) pp ) 1 p is ﬁnite (where the derivative D is taken in the weak sense).
There are some ways to deﬁne a Sobolev space with fractional order, one of which will be deﬁned by using the notion of interpolation space (DeVore, 1998; Adams & Fournier, 2003), but we don’t pursue this direction here.
Finally, we introduce Besov space which further generalizes the deﬁnition of the Sobolev space.
To deﬁne the Besov space, we introduce the modulus of smoothness.
Deﬁnition 3.
For a function f ∈ L p (Ω) for some p ∈ (0 , ∞ ] , the r -th modulus of smoothness of f is deﬁned by
[Equation]
∈
[Equation]
∆
[Equation]
f
[Equation]
p , where ∆ rh ( f )( x ) =
(cid:40)(cid:80) rj
=0 (cid:0) rj (cid:1) ( − 1) r − j f ( x + jh ) ( x ∈ Ω , x + rh ∈ Ω) , ( otherwise ) .
Based on the modulus of smoothness, the Besov space is deﬁned as in the following deﬁnition.
Deﬁnition 4 (Besov space ( B αp,q (Ω) )) .
For 0 < p, q ≤ ∞ , α > 0 , r := (cid:98) α (cid:99) + 1 , let the seminorm | · | B αp,q be
[Equation]
( q = ∞ )
[Equation]
The norm of the Besov space B αp,q (Ω) can be deﬁned by (cid:107) f (cid:107) B αp,q := (cid:107) f (cid:107) p + | f | B αp,q , and B αp,q (Ω) =
{ f ∈ L p (Ω) | (cid:107) f (cid:107) B αp,q < ∞} .
We
Note that p, q < 1 is also allowed.
In that setting, the Besov space is no longer a Banach space but a quasi-Banach space.
The Besov space plays an important role in several ﬁelds such as nonparametric statistical inference (Kerkyacharian & Picard, 1992; Donoho et al., 1996; Donoho & Johnstone, 1998; Gin´e & Nickl, 2015) and approximation theory (Temlyakov, 1993a).
These spaces are closely related to each other as follows (Triebel, 1983):
• For m ∈ N , B mp, 1 (Ω) (cid:44) → W mp (Ω) (cid:44) → B mp, ∞ (Ω) , and B m 2 , 2 (Ω) =
W m 2 (Ω) 4 .
• For 0 < s < ∞ and s (cid:54)∈ N , C s (Ω) =
B s ∞ , ∞ (Ω) .
• For 0 < s, p, q, r ≤ ∞ with s > δ :=
d (1 /p − 1 /r ) + , it holds that B sp,q (Ω) (cid:44) → B s − δ r,q (Ω) .
In particular, under the same condition, from the deﬁnition of (cid:107) · (cid:107) B sp,q , it holds that
[Equation]
• For 0 < s, p, q ≤ ∞ , if s > d/p , then B sp,q (Ω) (cid:44) → C
[Equation]
Ω) .
[Equation]
Hence, if the smoothness parameter satisﬁes s > d/p , then it is continuously embedded in the set of the continuous functions.
However, if s < d/p , then the elements in the space are no longer continuous.
Moreover, it is known that B 11 , 1 ([0 , 1]) is included in the space of bounded total variation and
B 11 , ∞ ([0 , 1]) includes it (Peetre & Dept, 1976).
Hence, the Besov space also allows spatially inhomogeneous smoothness with spikes and jumps; which makes difference between linear estimators and deep learning (see Sec. 4.1).
It is known that the minimax rate to estimate f o is lower bounded by n − 2 s/ (2 s + d ) , (Kerkyacharian & Picard, 1992; Donoho et al., 1996; Donoho & Johnstone, 1998;
Gin ´ e & Nickl, 2015).
We see that the curse of dimensionality is unavoidable as long as we consider the Besov space.
This is an undesirable property because we easily encounter high dimensional data in several machine learning problems.
Hence, we need another condition to derive approximation and estimation error bounds that are not heavily affected by the dimensionality.
To do so, we introduce the notion of mixed smoothness .
The Besov space with mixed smoothness is deﬁned as follows (Schmeisser, 1987;
Sickel & Ullrich, 2009).
To deﬁne the space, we deﬁne the coordinate difference operator as
[Equation]
for f : R d → R , h ∈ R + , i ∈ [ d ] , and r ≥ 1 .
By applying this difference operator to each coordinate in a subset e ⊂ { 1 , . . .
, d } of coordinates, the mixed differential operator for a step length h ∈ R d is deﬁned as
[Equation]
Then, the mixed modulus of smoothness is deﬁned as
[Equation]
for t ∈ R d + and 0 < p ≤ ∞ .
This quantity measures how the function f is “rough” in a coordinatewise manner.
In contrast to the Besov space (Def. 3), the differentiation is taken over all coordinates in e , which imposes a coordinate-wise smoothness.
Letting 0 < p, q ≤ ∞ , α ∈ R d
++ and r i :=
(cid:98) α
i (cid:99) + 1 , the semi-norm | · | MB α,ep,q based on the mixed smoothness is deﬁned by
[Equation]
By summing up the semi-norm over the choice of e , the (quasi-)norm of the mixed smooth Besov space (abbreviated to m-Besov space) is deﬁned by
[Equation]
and thus MB αp,q (Ω) :
= { f ∈ L p (Ω) | (cid:107) f (cid:107) MB αp,q < ∞} where 0 < p, q ≤ ∞ and α ∈ R d ++ .
In this paper, we assume that α 1 = · · · = α d .
With a slight abuse of notation, we also use the notation
MB αp,q for α > 0 to indicate MB ( α,...,α ) p,q .
It is known that, when p = q , the m-Besov space is characterized as a tensor product space of
B sp,q ([0 , 1]) (Sickel & Ullrich, 2009).
The m-Besov space includes several important models considered in the literature of statistical learning, e.g., the additive model (Meier et al., 2009) and the tensor model (Signoretto et al., 2010).
It is known that an appropriate estimator in these models can avoid curse of dimensionality (Meier et al., 2009; Raskutti et al., 2012; Kanagawa et al., 2016;
Suzuki et al., 2016).
What we will show in this paper supports that this fact is also applied to deep learning from a unifying viewpoint.
The difference between the (normal) Besov space and the m-Besov space can be informally explained as follows.
For regularity condition α i ≤ 2 ( i = 1 , 2) , the m-Besov space consists of functions for which the following derivatives are “bounded”:
[Equation]
That is, the “max” of the orders of derivatives over coordinates needs to be bounded by 2.
On the other hand, the Besov space only ensures the boundedness of the following derivatives:
[Equation]
where the “sum” of the orders needs to be bounded by 2.
This difference directly affects the rate of convergence of approximation accuracy.
Further details about this space and related topics can be found in a comprehensive survey (D ˜ ung et al., 2016).
Relation to Barron class.
Barron (1991; 1993; 1994) showed that, if the Fourier transform of a function f satisﬁes some integrability condition, then we may avoid curse of dimensionality for estimating neural networks with sigmoidal activation functions.
The integrability condition is given by (cid:82) C d (cid:107) ω (cid:107)| ˆ f ( ω ) | d ω < ∞ , where ˆ f is the Fourier transform of a function f .
We call the class of functions satisfying this condition Barron class .
A similar function class is analyzed by Klusowski & Barron (2016) too.
We cannot compare directly the m-Besov space with the Barron class, but they are closely related.
Indeed, if p = q = 2 and s = α 1 = · · · = α d , then m-Besov space MB s 2 , 2 (Ω) is equivalent to the tensor product of Sobolev space (Sickel & Ullrich, 2011) which consists of functions f : Ω → R satisfying (cid:82) C d (cid:81) di
=1 (1 + | ω i | 2 ) s | ˆ f ( ω ) | 2 d ω < ∞ .
Therefore, our analysis gives a (similar but) different characterization of conditions to avoid curse of dimensionality.
A PPROXIMATION ERROR ANALYSIS
In this section, we evaluate how well the functions in the Besov and m-Besov spaces can be approximated by neural networks with the ReLU activation.
Let us denote the ReLU activation by η ( x ) = max { x, 0 } ( x ∈ R ) , and for a vector x , η ( x ) is operated in an element-wise manner.
Deﬁne the neural network with height L , width W , sparsity constraint S and norm constraint B as Φ( L, W, S, B ) :
= { ( W ( L ) η ( · ) + b ( L ) ) ◦ · · · ◦ ( W (1) x
+ b (1) ) | W ( L ) ∈ R 1 × W , b ( L ) ∈ R ,
W (1) ∈ R W × d , b (1) ∈ R W , W ( (cid:96) )
∈ R W × W , b ( (cid:96) )
∈ R W (1 < (cid:96) < L ) , (cid:88) L (cid:96)
[Equation]
( (cid:107)W ( (cid:96) ) (cid:107)
[Equation]
(cid:107) b ( (cid:96)
[Equation]
(cid:107)
[Equation]
max (cid:96) (cid:107)W ( (cid:96)
[Equation]
(cid:107) ∞
[Equation]
(cid:107) b ( (cid:96) )
(cid:107) ∞ ≤ B } , where (cid:107) · (cid:107) 0 is the (cid:96) 0 -norm of the matrix (the number of non-zero elements of the matrix) and (cid:107) · (cid:107) ∞ is the (cid:96) ∞ -norm of the matrix (maximum of the absolute values of the elements).
We want to evaluate how large L, W, S, B should be to approximate f o ∈ MB α p,q (Ω) by an element f ∈ Φ( L, W, S, B ) with precision (cid:15) > 0 measured by L r -norm: min f ∈ Φ (cid:107) f − f o (cid:107) r ≤ (cid:15) .
3. A PPROXIMATION ERROR ANALYSIS FOR B ESOV SPACES
Here, we show how the neural network can approximate a function in the Besov space which is useful to derive the generalization error of deep learning.
Although its derivation is rather standard as considered in Chui et al.
; B¨olcskei et al.
(2017), it should be worth noting that the bound derived here cannot be attained by any non-adaptive method and the generalization error based on the analysis is also unattainable by any linear estimators including the kernel ridge regression.
That explains the high adaptivity of deep neural network and how it outperforms usual linear methods such as kernel methods.
To show the approximation accuracy, a key step is to show that the ReLU neural network can approximate the cardinal B-spline with high accuracy.
Let N ( x ) = 1 ( x ∈ [0 , 1]) , 0 ( otherwise ) , then the cardinal B-spline of order m is deﬁned by taking m + 1 -times convolution of N :
[Equation]
where f ∗ g ( x ) :
= (cid:82) f
( x − t ) g ( t )d t .
It is known that N m is a piece-wise polynomial of order m .
For k = ( k 1 , . . .
, k d ) ∈ Z d + and j = ( j 1 , . . .
, j d ) ∈ Z d + , let M dk,j ( x ) =
(cid:81) di
=1 N m (
2 k i x i − j i ) .
Even for k ∈ Z + , we also use the same notation to express M dk,j ( x )
= (cid:81) di
=1 N m (2 k x i − j i ) .
Here, k controls the spatial “resolution” and j speciﬁes the location on which the basis is put.
Basically, we approximate a function f in a Besov space by a super-position of M mk,j ( x ) , which is closely related to wavelet analysis (Mallat, 1999).
Mhaskar & Micchelli (1992); Chui et al.
(1994) have shown the approximation ability of neural network for a function with bounded modulus of smoothness.
However, the class of the activation functions in their analysis does not include ReLU but they dealt with activation functions satisfying the following conditions,
[Equation]
for k = 2 which excludes ReLU.
Mhaskar (1993) analyzed deep neural network under the same setting but it restricts the smoothness parameter to s = k
+ 1 .
Mhaskar (1996) considered the
Sobolev space W mp with an inﬁnitely many differentiable “bump” function which also excludes
ReLU.
However, approximating the cardinal B-spline by ReLU can be attained by appropriately using the technique developed by Yarotsky (2017) as in the following lemma.
Lemma 1 (Approximation of cardinal B-spline basis by the ReLU activation) .
There exists a constant c ( d,m ) depending only on d and m such that, for all (cid:15) > 0 , there exists a neural network ˇ M ∈
Φ( L 0 , W 0 , S 0 , B 0 ) with L 0 :
= 3+2 (cid:108) log 2 (cid:16) 3 d ∨ m (cid:15)c ( d [Equation] ) (cid:17) + 5 (cid:109) (cid:100) log [Equation]
[Equation]
m ) (cid:101) , W 0 : [Equation] 6 dm ( m +2)+2 d ,
S 0 :
=
L 0 W 20 and B 0 := 2( m + 1) m that satisﬁes
[Equation]
, and ˇ M ( x ) = 0 for all x (cid:54)∈ [0 , m + 1] d .
The proof is in Appendix A. Based on this lemma, we can translate several B-spline approximation results into those of deep neural network approximation.
In particular, combining this lemma and the B-spline interpolant representations of functions in Besov spaces (DeVore & Popov, 1988; DeVore et al., 1993; D˜ung, 2011b), we obtain the optimal approximation error bound for deep neural networks.
Here, let U ( H ) be the unit ball of a quasi-Banach space H , and for a set F of functions, deﬁne the worst case approximation error as
[Equation]
Proposition 1 (Approximation ability for Besov space) .
Suppose that 0 < p, q, r ≤ ∞ and 0 < s < ∞ satisfy the following condition:
[Equation]
Assume that m ∈ N satisﬁes 0 < s [Equation] ( m, m − 1 + 1 /p ) .
Let δ =
d (1 /p − 1 /r ) + and ν = ( s − δ ) / (2 δ ) .
For sufﬁciently large N ∈ N and (cid:15) =
N − s/d − ( ν − 1 + d − 1 )( d/p − s ) + log( N ) − 1 , let
L = 3 + 2 (cid:100) log(cid:18) 3 d ∨ m (cid:15)c ( d,m ) (cid:19) + 5 (cid:101)(cid:100) log 2 ( d ∨ m ) (cid:101) ,
[Equation] [Equation]
S =
[( L − 1) W 20 + 1] N,
[Equation]
d
[Equation]
then it holds that
[Equation]
p
[Equation]
Remark 1.
By Eq.
(1) , the condition (4) indicates that f o ∈ B sp,q satisﬁes f o ∈
L r (Ω) .
If we set p = q = ∞ and r = ∞ , then B sp,q (Ω) =
C s (Ω) which yields the result by Yarotsky (2017) as a special case.
The proof is in Appendix B.
According to the theorem, the approximation error N − s/d can be achieved by the settings L
= O (log( N )) ,
W = O ( N ) and S
= O ( N log( N )) for N ∈ N .
The convergence rate N − s/d is controlled by the smoothness s and the dimensionality d ; as the smoothness s goes up, we have better approximation error, and as the dimensionality d goes up, we have worse error.
An interesting point is that the statement is valid even for p (cid:54) =
r .
In particular, the theorem also supports non-continuous regime ( s < d/p ) in which L ∞ -convergence does no longer hold but instead L r -convergence is guaranteed under the condition s > d
(1 /p − 1 /r ) + .
In that sense, the convergence of the approximation error is guaranteed in considerably general settings.
Pinkus (1999) gave an explicit form of convergence when 1 ≤ p = r for the activation functions satisfying
Eq.
(3) which does not cover ReLU and an important setting p (cid:54) =
r .
Petrushev (1998) considered p = r
= 2 and activation function with Eq.
(3) where s is an integer such that s ≤ k +
+( d − 1) / 2 .
Chui et al.
and B¨olcskei et al.
(2017) dealt with the smooth sigmoidal activation satisfying the condition (3) with k ≥ 2 or a “smoothed version” of the ReLU activation which excludes ReLU; but B¨olcskei et al.
(2017) presented a general strategy for neural-net approximation by using the notion of best M -term approximation.
Mhaskar & Micchelli (1992) gives an approximation bound using the modulus of smoothness, but the smoothness s and the order of sigmoidal function k in (3) is tightly connected and f o is assumed to be continuous which excludes the situation s < d/p .
On the other hand, the above proposition does not require such a tight connection and it explicitly gives the approximation bound for Besov spaces.
Williamson & Bartlett (1992) derived a spline approximation error bound for an element in a Besov space when d = 1 , but the derived bound is only
O ( N − s +(1 /p − 1 /r )
+ ) which is the rate of non-adaptive methods described below, and approximation by a ReLU activation network
is not discussed.
We may also use the analysis of Cohen et al.
(2001) which is based on compactly supported wavelet bases, but the cardinal B-spline is easy to handle through quasi-interpolant representation as performed in the proof of Proposition 1.
It should be noted that the presented approximation accuracy bound is not trivial because it can not be achieved by a non-adaptive method .
Actually, the best N -term approximation error (Kolmorogov width) of the Besov space is lower bounded as
[Equation]
if 1 < p < r ≤ ∞ , 1 ≤ q < ∞ and 1 < s , where S N is any N -dimensional subspace of B sp,q (Romanyuk, 2009; Myronyuk, 2016; Vyb ´ aral, 2008).
That is, any linear/non-linear approximator with ﬁxed N -bases does not achieve the approximation error N − s/d in some parameter settings such as 1 < p < 2 < r .
On the other hand, adaptive methods including deep learning can improve the error rate up to N − s/d which is rate optimal (D˜ung, 2011b).
The difference is signiﬁcant when p < r .
This implies that deep neural network possesses high adaptivity to ﬁnd which part of the function should be intensively approximated.
In other words, deep neural network can properly extracts the feature of the input (which corresponds to construct an appropriate set of bases) to approximate the target function in the most efﬁcient way.
3. A PPROXIMATION ERROR ANALYSIS FOR M -B ESOV SPACE
Here, we deal with m-Besov spaces instead of the ordinary Besov space.
The next theorem gives the approximation error bound to approximate functions in the m-Besov spaces by deep neural network models.
Deﬁne D k,d := (cid:0) 1 + d − 1 k (cid:1) k (cid:16)
+ kd − 1 (cid:17)
d − 1 .
Then, we have the following theorem.
Theorem 1 (Approximation ability for m-Besov space) .
Suppose that 0 < p, q, r ≤ ∞ and s < ∞ satisﬁes s > (1 /p − 1 /r ) + .
Assume that m ∈ N satisﬁes 0 < s < min( m, m − 1 + 1 /p ) .
Let δ =
(1 /p − 1 /r )
+ and ν = ( s − δ ) / (2 δ ) .
For any K ≥ 1 , let K ∗ =
(cid:100) K (1 + 2 δ
α − δ ) (cid:101) .
Then, for
N = (2 + (1 − 2 − ν ) − 1 )2 K D K ∗ ,
d , if we set
L = 3 + 2 (cid:108) log 2 (cid:16) 3 d ∨ m c ( d,m ) (cid:17) +
+ ( s + ( 1 p − s ) + + 1) K ∗ + log([ e ( m + 1)]
d (1 + K ∗ )) (cid:109) (cid:100) log 2 ( d ∨ m ) (cid:101) ,
[Equation]
then it holds that
[Equation]
i
[Equation]
p
[Equation]
r
[Equation]
The proof is given in Appendix C.3.
It holds that N (cid:39) 2 K K ( d − 1) , which implies 2 − K (cid:39)
N − 1 log d − 1 ( N ) if N (cid:29) d (see also the discussion right after Theorem 5 in Appendix C.1 for more details of calculation).
Therefore, when r (cid:29) q , the approximation error can be evaluated as
O ( N − s log s ( d − 1) ( N )) for L =
O (log( N )) ,
W = O ( N ) and S
= O ( N log( N )) for N ∈ N in which the effect of dimensionality d is much milder than that of Proposition 1.
This means that the curse of dimensionality is much eased in the mixed smooth space.
The obtained bound is far from obvious.
Actually, it is better than any linear approximation methods as follows.
Let the linear M -width introduced by Tikhomirov (1960) be λ N ( MB sp,q , L r ) :=
inf L N sup f ∈ U ( MB sp,q ) (cid:107) f − L N ( f ) (cid:107) r , where the inﬁmum is taken over all linear oprators L N with rank N from MB sp,q to L r .
The linear N -width of the m-Besov space has been extensively studied as in the following proposition (see Lemma 5.1 of D ˜ ung
(2011a), and Romanyuk (2001)).
Proposition 2.
Let 1 ≤ p, r ≤ ∞ , 0 < q ≤ ∞ and s > (1 /p − 1 /r )
+ .
Then we have the following asymptotic order of the linear width for the asymptotics N (cid:29) d :
(a) For p ≥ r ,
[Equation]
MB
[Equation] [Equation]
(b) For 1 < p < r < ∞ ,
[Equation]
, L
[Equation] [Equation]
Therefore, the approximation error given in Theorem 1 achieves the optimal linear width ( ( N − 1 log d − 1 ( N )) s ) for several parameter settings of p, q, s .
In particular, when p < r , the bound in Theorem 1 is better than that of Proposition 2.
This is because to prove Theorem 1, we used an adaptive recovery technique instead of a linear recovery method.
This implies that, by constructing a deep neural network accurately, we achieve the same approximation accuracy as the adaptive one which is better than that of linear approximation.
E STIMATION ERROR ANALYSIS
In this section, we connect the approximation theory to generalization error analysis (estimation error analysis).
For the statistical analysis, we assume the following nonparametric regression model:
[Equation]
where x i ∼ P X with density 0 ≤ p ( x ) <
R on [0 , 1] d , and ξ i ∼ N (0 , σ 2 ) .
The data D n = ( x i , y i ) ni =1 is independently identically distributed.
We want to estimate f o from the data.
Here, we consider a regularized learning procedure:
[Equation]
where ¯ f is the clipping of f deﬁned by ¯ f = min { max { f, − F } , F } for F > 0 which is realized by
ReLU units.
Since the sparsity level is controlled by S and the parameter is bounded by B , this estimator can be regarded as a regularized estimator.
In practice, it is hard to exactly compute (cid:98) f .
Thus, we approximately solve the problem by applying sparse regularization such as L 1 -regularization and optimal parameter search through Bayesian optimization.
The generalization error that we present here is an “ideal” bound which is valid if the optimal solution (cid:98) f is computable.
4. E STIMATION ERROR IN B ESOV SPACES
In this subsection, we provide the estimation error rate of deep learning to estimate functions in
Besov spaces.
Theorem 2.
Suppose that 0 < p, q ≤ ∞ and s > d (1 /p − 1 / 2) + .
If f o ∈ B sp,q (Ω) ∩ L ∞ (Ω) and.
(cid:107) f o (cid:107) B sp,q ≤ 1 and (cid:107) f
o (cid:107) ∞ ≤ F for F ≥ 1 , then letting ( W, L, S, B ) be as in Proposition 1 with
N (cid:16) n d 2 s + d , we obtain
[Equation]
where E D n [ · ] indicates the expectation w.r.t.
the training data D n .
The proof is given in Appendix D.
The condition (cid:107) f
o (cid:107) ∞ ≤ F is required to connect the empirical
L 2 -norm 1 n (cid:80) ni
=1 ( (cid:98) f
( x i ) − f o ( x i ))
2 to the population L 2 -norm (cid:107) (cid:98) f − f o (cid:107) 2 L 2 ( P X ) .
It is known that the convergence rate n − 2 s 2 s + d is mini-max optimal (Kerkyacharian & Picard, 1992; Donoho et al., 1996; Donoho & Johnstone, 1998; Gin´e & Nickl, 2015).
Thus, it cannot be improved by any estimator.
Therefore, deep learning can achieve the minimax optimal rate up to log( n ) 3 -order.
The term log( n ) 3 could be improved to log( n ) 2 by using the construction of Petersen & Voigtlaender (2017).
However, we don’t pursue this direction for simplicity.
Here an important remark is that this minimax optimal rate cannot be achieved by any linear estimator .
We call an estimator linear when the estimator depends on ( y i ) ni
=1 linearly (it can be non-linearly dependent on ( x i ) ni =1 ).
Several classical methods such as the kernel ridge regression, the Nadaraya-Watson estimator and the sieve estimator are included in the class of linear estimators (e.g., kernel ridge regression is given as (cid:98) f
( x ) =
k x,X ( k XX + λ I) − 1 Y ).
The following proposition given by Donoho & Johnstone (1998); Zhang et al.
(2002) states that the minimax rate of linear estimators is lower bounded by n −{ 2 s − 2(1 /p − 1 / 2) + } / { 2 s +1 − 2(1 /p − 1 / 2) + } for d = 1 which is larger than the minimax rate n − 2 s [-
-]2 s [-+
-]+1 if p < 2 .
Proposition 3 (Donoho & Johnstone (1998); Zhang et al. (2002)) .
Suppose that d = 1 and the input distribution P X is the uniform distribution on [0 , 1] .
Assume that s > 1 /p , 1 ≤ p, q ≤ ∞ or s = p = q = 1 .
Then,
[Equation]
where v = 2 / ( p ∧ 2) − 1 and (cid:98) f runs over all linear estimators, that is, (cid:98) f depends on ( y i ) ni
=1 linearly.
When p < 2 , the smoothness of the Besov space is somewhat inhomogeneous, that is, a function in the Besov space contains spiky/jump parts and smooth parts (remember that when s = p =
q =for d = 1 , the Besov space is included in the set of functions with bounded total variation).
Here, the setting p < 2 is the regime where there appears difference between non-adaptive methods and deep learning in terms of approximation accuracy (see Eq.
On the other hand, the linear estimator captures only global properties of the function and cannot capture variability of local shapes of the function.
Hence, the linear estimator cannot achieve the minimax optimal rate if the function has spatially inhomogeneous smoothness.
However, deep learning possesses adaptivity to the spatial inhomogeneity.
Imaizumi & Fukumizu (2018) has pointed out that such a discrepancy appears when the target function is non-smooth .
Interestingly, the parameter setting s > 1 /p assumed in Propositionensures smoothness (see Eq.
This means that non-smoothness is not necessarily required to characterize the superiority of deep learning, but non-convexity of the set of target functions is essentially important.
In fact, the gap is coming from the property that the quadratic hull of the model U ( B sp,q ) is strictly larger than the original set (Donoho & Johnstone, 1998).
4. E STIMATION ERROR IN MIXED SMOOTH B ESOV SPACES
Here, we provide the estimation error rate to estimate functions in mixed smooth Besov spaces.
Theorem 3.
Suppose that 0 < p, q ≤ ∞ and s > (1 /p − 1 / 2) + .
Let u =
(1 − 1 /q )
+ for p ≥ 2 and u = (1 / 2 − 1 /q )
+ for p < 2 .
If f o ∈ MB sp,q (Ω) ∩ L ∞ (Ω) and (cid:107)
f
o (cid:107) MB sp,q ≤ 1 and (cid:107) f
o (cid:107) ∞ ≤ F for F ≥ 1 , then letting ( W, L, S, B ) be as in Theorem 1, we obtain
[Equation]
Under the same assumption, if s > u log 2 ( e ) is additionally satisﬁed, we also have
[Equation]
The proof is given in Appendix D.
The risk bound (Theorem 3) indicates that the curse of dimensionality can be eased by assuming the mixed smoothness compared with the ordinary Besov space ( n −
s 2 s + d ).
We show that this is almost minimax optimal in Theorem 4 below.
In the ﬁrst bound, the dimensionality d comes in the exponent of poly log( n ) term.
If u = 0 , then the effect of d can be further eased.
Actually, in this situation ( u = 0 ), the second bound can be rewritten as
[Equation]
where the effect of the dimensionality d completely disappears from the exponent.
This explains partially why deep learning performs well for high dimensional data.
Here, we again remark the adaptivity of deep learning.
Remind that this rate cannot be achieved by the linear estimator for p < 2 when d = 1 by Proposition 3.
Montanelli & Du (2017) has analyzed the mixed smooth
H ¨ older space with s < 2 .
However, our analysis is applicable to the m-Besov space which is more general than the mixed smooth H ¨ older space and the covered range of s, p, q is much larger.
Minimax optimal rate for estimating a function in the m-Besov space Here, we show the minimax optimality of the obtained bound as follows.
Theorem 4.
Assume that 0 < p, q ≤ ∞ and s > (1 /p − 1 / 2) + and P X is the uniform distribution over [0 , 1] d .
Regarding d as a constant, the minimax learning rate in the asymptotics of n → ∞ is lower bounded as follows: There exists a constant (cid:98) C 1 such that
[Equation]
(8) where “inf” is taken over all measurable functions of the observations ( x i , y i ) ni
=1 and the expectation is taken for the sample distribution.
The proof is given in Appendix E. Because of this theorem, our bound given in Theorem 3 achieves the minimax optimal rate in the regime of p < 2 and 1 / 2 − 1 /q > 0 up to log( n ) 3 order.
Even outside of this parameter setting, the discrepancy between our upper bound and the minimax lower bound is just a poly- log oder.
See also Neumann (2000) for some other related spaces and speciﬁc examples such as p = q = 2 .
C ONCLUSION
This paper investigated the learning ability of deep ReLU neural network when the target function is in a Besov space or a mixed smooth Besov space.
Based on the analysis for the Besov space, it was shown that deep learning using the ReLU activation can achieve the minimax optimal rate and outperform the linear method when p < 2 which indicates the spatial inhomogeneity of the shape of the target function.
The analysis for the mixed smooth Besov space showed that deep learning can adaptively avoid the curse of dimensionality.
The bound was derived by sparse grid technique.
All analyses in the paper adopted the cardinal B-spline expansion and the adaptive non-linear approximation technique, which allowed us to show the minimax optimal rate.
The consequences of the analyses partly support the superiority of deep leaning in terms of adaptivity and ability to avoid curse of dimensionality.
From more high level view point, these favorable property is reduced to its high feature extraction ability.
Acknowledgment The author is grateful to Satoshi Hayakawa for showing the correct statement of Proposition 4 by modifying the proof of Lemma 10 of Schmidt-Hieber (2018).
TS was partially supported by MEXT Kakenhi (25730013, 25120012, 26280009, 15H05707 and 18H03201), Japan
Digital Design and JST-CREST.
A P ROOF OF L EMMA
Proof of Lemma 1.
First note that N m ( x ) = 1 m !
(cid:80) m +1 j =
( − 1) j (cid:0) m +1 j (cid:1) ( x − j ) m
+ (see Eq.
(4.28) of
Mhaskar & Micchelli (1992) for example).
Thus, if we can make an approximation of η ( x ) m , then by taking a summation of those basis, we obtain an approximate of N m ( x ) .
It is shown by Yarotsky (2017);
Schmidt-Hieber (2018) that, for D ∈ N and any (cid:15) > 0 , there exists a neural network
φ mult ∈
Φ( L, W, S, B ) with L = (cid:100) log 2 (cid:16) 3 D (cid:15) (cid:17) + 5 (cid:101)(cid:100) log 2 ( D ) (cid:101) , W = 6 d , S = LW 2 and B =such that sup
[Equation]
≤
[Equation]
, and φ mult (0 , . . . , 0)
=
for y ∈ R D such that (cid:81)
Dj =
y j = 0 .
Moreover, for any M > 0 , we can realize the function min { M, max { x, 0 }} by a single-layer neural network φ (0 ,M ) ( x ) :
= η ( x ) − η ( x − M )
(=
min { M, max { x, 0 }} ) .
Thus, for x ∈ R , it holds that
[Equation]
Now, N m ( x ) =
for x (cid:54)∈ [0 , m + 1] gives that
[Equation]
Therefore, letting
[Equation] [Equation]
we have that f ( x ) = 0 for all x ≤ 0 and
[Equation]
where we used (cid:80) m +1 j =0 (cid:0)
m +1 j (cid:1)
=
2 m +1 and Stirling’s approximation m !
≥ √ 2 πm m +1 / 2 e − m in the second inequality.
Hence, we also have that, for all x > m + 1 ,
[Equation]
× φ mult (cid:18) φ (0 ,
[Equation]
jm +1 ) (cid:18) m + 1 − j m +(cid:19) , . . .
, φ (0 , 1 − jm +1 ) (cid:18) m + 1 − j m +(cid:19)(cid:19) =: δ (cid:48) .
It holds that | δ (cid:48) | ≤ (cid:15) (cid:48) .
Since it is possible that δ (cid:48) (cid:54) = 0 , we modify f ( x ) so that f ( x ) =
outside the interval [0 , m + 1] .
Because of this and noting 0 ≤ N m ( x ) ≤ 1 , we see that g ( x ) :=
φ (0 , 1) ( f ( x ) − δ (cid:48) m +1 φ (0 ,m +1) ( x )) yields
[Equation]
Hence, by applying φ mult again, we ﬁnally obtain that sup x
[Equation] [Equation]
| M
[Equation] [Equation]
[Equation]
g ( x
[Equation]
)) | ≤ sup
[Equation]
] d (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
M d
[Equation]
0 ( x ) − d
(cid:89)
j = g (
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
+ sup x
[Equation] [Equation]
1] d (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) d (cid:89)
j = g ( x j ) − φ mult ( g ( x 1 ) , . .
, g ( x d )) (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) ≤ 2 d(cid:15) (cid:48)
+ (cid:15).
We again applying φ (0 , 1) , we obtain that h ( x ) =
φ (0 , 1) ◦ φ mult ( g ( x 1 ) , . .
, g ( x d )) satisﬁes (cid:107) M d 0 , 0 − h (cid:107) L ∞ ( R d ) ≤ 2 d(cid:15) (cid:48)
+ (cid:15) , h ( x ) =
for all x (cid:54)∈ [0 , m +1] d , and (cid:107) h (cid:107) ∞ ≤ 1 .
Finally, by carefully checking the network construction, it is shown that h ∈ Φ( L, W, S, B ) with L = 3 + 2 (cid:100) log (cid:16) 3 d ∨ m (cid:15) (cid:17) + 5 (cid:101)(cid:100) log 2 ( d ∨ m ) (cid:101) , W = 6 dm ( m + 2) + 2 d , S = LW 2 and B = 2( m + 1) m .
Hence, resetting (cid:15) ← 2 d(cid:15) (cid:48)
+ (cid:15) =
(1 + 2 de (2 e ) m √ m ) (cid:15) , h becomes the desired ˇ M .
B P ROOF OF P ROPOSITION
For the order m ∈ N of the cardinal B-spline bases, let J ( k ) =
{− m, − m + 1 , . . .
, 2 k − 1 , 2 k } d and the quasi-norm of the coefﬁcient ( α k,j ) k,j for k ∈ Z + and j ∈ J ( k ) be (cid:107) ( α k,j ) k,j (cid:107) b sp,q . =
 
[Equation]
Lemma 2.
Under one of the conditions (4) in Proposition 1 and the condition 0 < s < min( m, m − 1 + 1 /p ) where m ∈ N is the order of the cardinal B-spline bases, for any f ∈ B sp,q (Ω) , there exists f N that satisﬁes
[Equation]
for N (cid:29) 1 , and has the following form:
[Equation]
M
[Equation] [Equation]
where ( j i ) n k i =
⊂ J ( k ) , K = (cid:100) C 1 log( N ) /d (cid:101) , K ∗ =
(cid:100) log( λN )
ν − 1 (cid:101) + K
+ 1 , n k = (cid:100) λN 2 − ν
( k − K ) (cid:101) ( k = K + 1 , . . .
, K ∗ ) for δ =
d (1 /p − 1 /r ) + and ν = ( s − δ ) / (2 δ ) where the real number constants C 1 > 0 and λ > 0 are chosen to satisfy (cid:80)
Kk =1 (2 k + m ) d
+ (cid:80) K ∗ k
= K +1 n k ≤ N independently to N .
Moreover, we can choose the coefﬁcients ( α k,j ) to satisfy
[Equation]
Proof of Lemma 2.
DeVore & Popov (1988) constructed a linear bounded operator P k having the following form:
[Equation]
where α k,j is constructed in a certain way, where for every f ∈ L p ([0 , 1] d ) with 0 < p ≤ ∞ , it holds
[Equation]
Let
[Equation]
Then, it is shown that for 0 < p, q ≤ ∞ and 0 < s < min( m, m − 1 + 1 /p ) , f belongs to B sp,q if and only if f can be decomposed into
[Equation]
with the convergence condition (cid:107) ( p k
[Equation]
< ∞ ; in particular, (cid:107) f (cid:107) B sp,q (cid:39) (cid:107) ( p k ( f )
[Equation]
∞ k =
[Equation]
(cid:107) b sp ( L p ) .
Here, each p k can be expressed as p k ( x ) = (cid:80) j ∈ J ( k ) α k,j M dk,j ( x ) for a coefﬁcient ( α k,j ) k,j which could be different from ( a k,j ) k,j appearing in Eq.
Hence, f ∈ B sp,q can be decomposed into
[Equation]
[Equation]
with convergence in the sence of L p .
Moreover, it is shown that (cid:107) p k (cid:107) L p (cid:39)
[Equation]
k,j | p ) 1 /p and thus
[Equation]
Based on this decomposition, D ˜ ung (2011b) proposed an optimal adaptive recovery method such that the approximator has the form (10) under the conditions for K, K ∗ , n k given in the statement and satisﬁes the approximation accuracy (9).
This can be proven by applying the proof of Theorem 3.1 in
D ˜ ung (2011b) to the decomposition (13) instead of Eq.
(3.8) of that paper.
See also Theorem 5.4 of
D ˜ ung (2011b).
Moreover, the equivalence (14) gives the norm bound of the coefﬁcient ( α k,j ) .
Proof of Proposition 1.
Basically, we combine Lemma 1 and Lemma 2.
We substitute the approximated cardinal B-spline basis ˇ M into the decomposition of f N (10).
Let the set of indexes ( k, j ) ∈ Z × Z that consists f N given in Eq.
(10) be E N ; i.e., f N = (cid:80) ( k,
j ) ∈
E N α k,j M dk,j .
Accordingly, we set ˇ f := (cid:80) ( k,
j ) ∈ E N α k,j ˇ M dk,j .
For each x ∈ R d , it holds that
[Equation]
where we used the deﬁnition of K ∗ in the last inequality.
Therefore, for each f ∈ U ( B sp,q ([0 , 1] d )) , it holds that
[Equation]
By taking (cid:15) to satisfy
[Equation] [Equation]
N
[Equation]
log( N ) − 1 ), then we obtain the approximation error bound.
Next, we bound the magnitude of the coefﬁcients.
Each coefﬁcient α j,k satisﬁes | α j,k | (cid:46)
[Equation]
p
[Equation]
) + (cid:107) f (cid:107)
[Equation]
q ≤
[Equation]
( d/p
[Equation]
s )
+ (cid:46) N
( ν − 1 + d − 1 )( d/p − s ) + for k ≤ K ∗ .
Finally, the magnitudes of the coefﬁcients hidden in ˇ M dk,j are evaluated.
Remembering that ˇ M mk,j ( x ) =
ˇ M (2 k x 1 − j 1 , . . .
, 2 k x d − j d ) , we see that we just need to bound the quantity 2 k ( k ≤ K ∗ ) .
However, this is bounded by 2 k ≤ N ν −
C P ROOF OF T HEOREMC. P REPARATION : SPARSE GRID
Here, we give technical details behind the approximation bound.
The analysis utilizes the so called sparse grid technique Smolyak (1963) which has been developed in the function approximation theory ﬁeld.
As we have seen in the above, in a typical B-spline approximation scheme, we put the basis functions
M mk,j ( x ) on a “regular grid” for k = 1 , . . .
, K and ( j 1 , . .
j d ) ∈ J ( k ) , and take its superposition as f ( x )
≈ (cid:80) k
=1 ,...,K (cid:80) j ∈ J ( k ) α k,j M mk,j ( x ) , which consists of O (2 Kd ) terms (see Eq.
Hence, the number of parameters O (2 Kd ) is affected by the dimensionality d in an exponential order.
However, to approximate functions with mixed smoothness, we do not need to put the basis on the whole range of the regular grid.
Instead, we just need to put them on a sparse grid which is a subset of the regular grid and has much smaller cardinality than the whole set.
The approximation algorithm utilizing sparse grid is based on Smolyak’s construction (Smolyak, 1963) and its applications to mixed smooth spaces (D˜ung, 1990; 1991; 1992; Temlyakov, 1982; 1993a;b).
D˜ung (2011a) studied an optimal non-adaptive linear sampling recovery method for the mixed smooth Besov space based on the cardinal B-spline bases.
We adopt this method, and combining this with the adaptive technique developed in D ˜ ung (2011b), we give the following approximation bound using a non-linear adaptive method to obtain better convergence for the setting p < r .
Before we state the theorem, we deﬁne an quasi-norm of a set of coefﬁcients α k,j ∈ R for k ∈ Z d + and j ∈ J dm ( k ) :
= {− m, − m + 1 , . .
, 2 k 1 − 1 , 2 k 1 } ×
[Equation]
Theorem 5.
Suppose that 0 < p, q, r ≤ ∞ and s > (1 /p − 1 /r ) + .
Assume that the order m ∈ N of the cardinal B-spline satisﬁes 0 < s < min( m, m − 1 + 1 /p ) .
Let δ =
(1 /p − 1 /r ) + .
Then, for any f ∈ MB sp,q (Ω) and K > 0 , there exists R K ( f ) such that R K ( f ) can be represented as
[Equation]
where K ∗ = (cid:100) K (1 +
δ s − δ ) (cid:101) , ( j ( k ) i ) n k i =
⊂ J dm ( k ) , and n k = (cid:100) 2 K − s − δ 2 δ ( (cid:107) k (cid:107) 1 − K ) (cid:101) , and satisﬁes the following properties:
(i) For p ≥ r , (cid:107) f − R K ( f ) (cid:107)
[Equation]
ii
[Equation]
p
<
r
[Equation]
∞
[Equation]
Moreover, the coefﬁcients ( α k,j ) k,j can be taken to hold (cid:107) ( α k,j ) k,j (cid:107) mb sp,q (cid:46) (cid:107) f (cid:107) MB sp,q .
The proof is given in Appendix C.2.
The total number of cardinal B-spline bases consisting of
R K ( f ) can be evaluated as
[Equation]
Here, D K,d can be evaluated as
[Equation]
Therefore, the total number of bases can be evaluated as
[Equation]
which is much smaller than 2 Kd which is required to approximate functions in the ordinal Besov space (see Lemma 2).
In this proposition, K controls the resolution and as M goes to inﬁnity, the approximation error goes to 0 exponentially fast.
A remarkable point in the proposition is in the construction of R K ( f ) in which the superposition is taken over (cid:107)
k (cid:107) 1 ≤ M instead of (cid:107) k (cid:107) ∞ ≤
K ∗ =
O ( K ) .
Hence, the number of terms appearing in the summation is at most O (2 K K d − 1 ) while the full grid takes O (2 Kd ) terms.
This represents how the mixed smoothness is important to ease the curse of dimensionality.
Several aspects of the m-Besov space such as the optimal N -term approximation error and Kolmogorov widths have been extensively studied in the literature (see a comprehensive survey (D˜ung et al., 2016)).
An analogous result is already given by D˜ung (2011a) in which s > 1 /p is assumed and a linear interpolation method is investigated.
However, our result only requires s > (1 /p − 1 /q )
+ .
This difference comes from a point that our analysis allows nonlinear adaptive interpolation instead of (linear) non-adaptive sampling considered in D ˜ ung (2011a).
Because of this, our bound is better than the optimal rate of linear methods (Galeev, 1996; Romanyuk, 2001) and non-adaptive methods (D˜ung, 1990; 1991; 1992; Temlyakov, 1982; 1993a;b) especially in the regime of p < r (D ˜ ung (1992) also deals with adaptive methods but does not cover p < r for adaptive method).
See Proposition 2 for comparison.
C. P ROOF OF T HEOREM
Proof of Theorem 5.
For k = ( k 1 , . . .
, k d ) ∈
Z d + , let P ( i ) k
i f ( x ) be the function operating P k i deﬁned in (11) to f as a function of x i with other components x j ( j (cid:54) = i ) ﬁxed, and let
[Equation]
Then, p k can be expressed as p k ( x ) = (cid:80) j ∈ J dm ( k ) α k,j M dk,j ( x ) .
Let T ( i ) k
i =
I − P
( i ) k
i and (cid:107) f (cid:107) p,i be the L p -norm of f as a function of x i with other components x j ( j (cid:54) =
i ) ﬁxed (i.e., if p < ∞ , (cid:107) f (cid:107) pp,i = (cid:82)
| f
( x ) | p d x i ), then Eq.
(12) gives
[Equation]
Thus, by applying the same argument again, it also holds
[Equation]
( ∵ the deﬁnition of ∆ r,ih i and Fubini’s theorem ) (cid:46) sup | h i |≤ [Equation] − ki sup |
h j |≤
[Equation]
−
[Equation]
(cid:107)(cid:107) ∆
[Equation]
ih
[Equation]
∆
[Equation] jh [Equation] f ))
[Equation]
,j
[Equation]
p,i , for i (cid:54) =
j .
Thus, applying the same argument recursively, for u ⊂ [ d ] , it holds that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:89) i ∈ u
T ( i ) k
i f (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) p (cid:46) w ur,p ( f, 2 − k ) , for k ∈
Z d + ( u ) .
Therefore, since
[Equation]
= (cid:80) u
⊂ [ d ] ( − 1) | u | (cid:16)(cid:81) i ∈
u T ( i ) k
i (cid:81) i (cid:54)∈ u T ( i ) k
i − 1 (cid:17) f , by letting e = { i | k i > 0 } , we have that (cid:107) p k (cid:107) p (cid:46)
(cid:88) u ⊂ [ d ] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:89) i ∈ u T ( i ) k
i (cid:89) i (cid:54)∈ u T ( i ) k i −  f (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) p (cid:46) (cid:88) u ⊂ [ d ] w ˆ ur,p ( f, 2 − ( k u ) ˆ u ) (cid:46) (cid:88) e ⊂ u w ur,p ( f, 2 − k u ) where k ui :
= k i ( i ∈ u ) and k ui :=
k i −
( i (cid:54)∈ u ) , ˆ u
= { i | k ui ≥ 0 } , and ( k u ) ˆ u is a vector such that ( k u ) ˆ u,i =
k
ui for i ∈
ˆ u and ( k u )
ˆ u,i =
for i (cid:54)∈ ˆ u .
Now let (cid:107) ( p k ) k (cid:107) b αq
[Equation]
L p ) = [Equation] (cid:88)
[Equation]
for p k ∈ L p (Ω) ( k ∈ Z d + ) .
Hence, if we set a k = (cid:80) e ⊂ u w ur,p ( f, 2 − k u ) for k ∈ Z d + and e = { i | k i > 0 } , we have that
[Equation]
On the other hand, following the same line of Theorem 2.1 (ii) of D˜ung (2011a), we also obtain the opposite inequality (cid:107) f (cid:107) MB sp,q (cid:39) (cid:107) ( a k ) (cid:107) b αq
( L p ) (cid:46) (cid:107) (
p k ) k
(cid:107) b αq
L p )
(note that the analogous inequality to Lemma 2.3 of D˜ung (2011a) also holds in our setting by replacing q s with p s and ω er ( f, 2 − k ) p by w er,p ).
Therefore, f ∈ MB αp,q if and only if ( p k )
k ∈ Z d + given by Eq.
(15) satisﬁes (cid:107) ( p k )
k (cid:107) b αq ( L p ) < ∞ and f can be decomposed into f = (cid:80)
k ∈ Z d + p k where convergence is in MB αp,q .
Moreover, it holds that (cid:107) f (cid:107) MB αp,q (cid:39) (cid:107) (
p k ) k (cid:107) b αq ( L p ) .
This can be shown by Theorem 2.1 of D ˜ ung (2011a).
Moreover, by the quasi-norm equivalence (cid:107) p k (cid:107) p (cid:39) 2 −(cid:107) k (cid:107) 1 /p ( (cid:80) j ∈ J dm ( k ) | α k,j | p ) 1 /p , we also have (cid:107) ( α k,j ) k,j (cid:107) mb αp,q (cid:39) (cid:107) f (cid:107) MB αp,q .
If p ≥ r , the assertion can be shown in the same manner as Theorem 3.1 of D˜ung (2011a).
For the setting of p <
r , we need to use an adaptive approximation method.
In the following, we assume p < r .
For a given K , by choosing K ∗ appropriately later, we set
[Equation]
where G k ( p k ) is given as
[Equation]
where ( α k,j i ) | J dm ( k ) | i
= is the sorted coefﬁcients in decreasing order of their absolute value: | α k,j 1 | ≥ | α k,j 2 | ≥ · · · ≥ | α k,j | Jdm ( k ) | | .
Then, it holds that
[Equation]
(cid:107)
[Equation]
n − δ
[Equation]
, where δ :=
(1 /p − 1 /r ) (see the proof of Theorem 3.1 of D˜ung (2011b) and Lemma 5.3 of D˜ung (2011a)).
Moreover, we also have (cid:107) p k (cid:107) r ≤ (cid:107)
p k (cid:107) p 2 δ (cid:107) k (cid:107)for k ∈ Z d + with (cid:107) k (cid:107) 1 > K ∗ .
Here, we deﬁne N as
[Equation]
Let (cid:15) =
( s − δ ) / (2 δ ) , and
K ∗ =
(cid:100) K (1 + 1 /(cid:15) ) (cid:101) , and n k = (cid:100) 2 K − (cid:15) ( (cid:107) k (cid:107) 1 − K ) (cid:101) for k ∈ Z d + with K + 1 ≤ (cid:107) k
(cid:107) 1 ≤ K ∗ .
Then, by Lemma 5.3 of D˜ung (2011a), we have that
[Equation]
In the following, we require an upper bound of (cid:0) k
+ d − 1 d − 1 (cid:1) .
Hence, we evaluate this quantity beforehand.
This can be upper bounded by the Stering’s formula as
[Equation]
Let ξ > 0 be a positive real number satisfying 1 + ξ ≥ K ∗ /K .
We can see that ξ can be chosen as ξ = 1 /(cid:15)
+ o (1) .
Then, we have that
[Equation]
(a) Suppose that q
≤
r
and
r < ∞ .
Then
(cid:107)
[Equation]
−
[Equation]
K
[Equation]
f )
[Equation]
qL
[Table]
[Equation]
(b) Suppose that q > r and r < ∞ .
Then, letting ν = q/r ( > 1) and ν (cid:48)
= 1 / (1 − 1 /ν ) =
q/ ( q − r ) (note that 1 ν + 1 ν (cid:48)
= 1 ), we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]

[Equation]
[Table]
×  
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
K,d ) r (cid:107) f (cid:107) rMB s p,q (c) Suppose that r = ∞ . Then, similarly to the analysis in (b), we can evaluate (cid:107) f
[Equation]
R K ( f ) (cid:107) L r (cid:46) N [Equation] (cid:88) [Equation]
(cid:46) N [Equation] s D ( [Equation]
(cid:107) f (cid:107) MB sp
[Equation]
q
[Equation]
C. P ROOF OF T HEOREM
Let Z d + ( e ) :
= { k ∈
Z d + | k i = 0 , i (cid:54)∈ e } and for k ∈
Z d + ( e ) , we deﬁne 2 − k :
=
(2 − k i 1 , . .
, 2 − k
i | e | ) ∈ R | e | + where { i 1 , . . .
, i | e | } = e .
By deﬁning (cid:107) ( g k ) k (cid:107) b s,eq :=
(cid:16)(cid:80)
k ∈ Z d + ( e ) (2 s (cid:107) k
(cid:107) 1 | g k | )
q (cid:17) 1 /q for a sequence ( g k )
k ∈ Z d + ( e ) , then it holds that
[Equation]
Then, we can prove Theorem 1 based on Theorem 5 as follows.
Proof of Theorem 1.
The result is immediately follows from Theorem 5.
Let the set of indexes of ( k, j ) consisting of R K be E K : R K ( f )
= (cid:80) ( k,
j ) ∈
E K α k,j M dk,j ( x ) .
As in the proof of Proposition 1, we approximate R K ( f ) by a neural network given as
[Equation]
Each coefﬁcient α j,k satisﬁes | α j,
[Equation]
q
[Equation]
2 K ∗ (1 /p − s ) + .
The difference between R K ( f ) and ˇ f can be evaluated as
[Equation]
Therefore, by taking (cid:15) so that (cid:15) ( m + 1) d (1 + K ∗ ) D K ∗ ,d 2 K ∗ (
[Equation]
p
[Equation]
Ks is satisﬁed, it holds that
[Equation]
By the inequality D K ∗ ,d ≤ e K ∗ + d − 1 , it sufﬁces to let (cid:15) ≤
e − K ∗ ( s +(1 /p − s )++1)
[ e ( m +1)]
d (1+ K ∗ ) .
The cardinality of
E ( K ) is bounded as
[Equation]
Since each unit ˇ M dk,j requires width W 0 , the whole width becomes W = NW 0 .
The number of nonzero parameters to construct ˇ M d k,j is bounded by S =
( L − 1) W 2 0 N + N .
Finally, the magnitudes of the coefﬁcients hidden in ˇ M dk,j are evaluated.
Remembering that ˇ M dk,j ( x ) =
ˇ M (2 k 1 x 1 − j 1 , . . .
, 2 k d x d − j d ) , here maximum of 2 k j is bounded by 2 K ∗ (cid:46) N (1+1 /ν ) .
Hence, we obtain the assertion.
Similarly, it holds that | α j,k | (cid:46) N (1+1 /ν ) { 1 ∨ (1 /p − s ) + } .
D P ROOFS OF T HEOREMS 2 AND
Proof of Theorem 2.
We use Proposition 4.
We just need to evaluate the covering number of ˆ F = { ¯ f | f ∈ Ψ( L, W, S, B ) } for ( L, W, S, B ) given in Theorem 1 where ¯ f is the clipped function for a given f .
Note that the covering number of ˆ F is not larger than that of Ψ( L, W, S, B ) .
Hence, it is sufﬁcient to evaluate that of Ψ( L, W, S, B ) .
From Lemma 3, the covering number is obtained as
[Equation]
From Proposition 1, it holds that
[Equation]
Note that
[Equation]
for any f : [0 , 1] d → R because p ( x ) ≤ R .
Therefore, by applying Proposition 4 with δ = 1 /n , we have that
[Equation]
Here, the right hand side is minimized by setting N (cid:16) n d 2 s
[Equation]
d up to log( n ) 3 -order, and then have an upper bound of the RHS as
[Equation]
This gives the assertion.
Proof of Theorem 3.
The proof follows the almost same line as the proof of Theorem 2.
By noting
S =
O (2 K D K,d ) , L = O ( K ) and W =
O (2 K D K,d ) , Lemma 3 gives an upper bound of the covering number as
[Equation]
Letting r = 2 , we have that
[Equation]
where u = (1 − 1 /q ) + for p ≥ 2 and u = (1 / 2 − 1 /q ) + for p < 2 .
Then, by noting that
[Equation]
for any f : [0 , 1] d → R , and by applying Proposition 4 with δ = 1 /n , we have that
[Equation]
Here, we use the following evaluations for D K,d : (a) D K,d (cid:46) K
d − 1 , and (b) D K,d (cid:46) [ e (1 + dK )] K .
(a) For the evaluation, D K,d (cid:46)
K
d − 1 , we have an upper bound of the right hand side of Eq.
(19) as
[Equation]
n , which is minimized by setting K = (cid:100) 1 1+2 s log 2 ( n ) + (2 u − 1)( d − 1) 1+2 s
[Equation]
2 log( n ) (cid:101) up to log log( n ) - order.
In this situation, we have the generalization error bound as
[Equation]
(b) For the evaluation, D K,d (cid:46) [ e
[Equation] [Equation] [Equation]
≤ [Equation] , Eq.
(19) gives an upper bound of
[Equation]
n
[Equation]
Then, the right hand side is minimized by K = (cid:100)
[Equation]
) (cid:101) .
Then, we have that
[Equation]
This gives the assertion.
E M INIMAX OPTIMALITY
Proof of Theorem 4.
First note that since P X is the uniform distribution, it holds that (cid:107) · (cid:107) L 2 ( P X )
= (cid:107) · (cid:107)
L 2 ([0 , 1] d ) .
The (cid:15) -covering number N ( (cid:15), G , L 2 ( P X )) with respect to L 2 ( P X ) for a function class
G is the minimal number of balls with radius (cid:15) measured by L 2 ( P X ) -norm needed to cover the set
G (van der Vaart & Wellner, 1996).
The δ -packing number M ( δ, G , L 2 ( P X )) of a function class
G with respect to L 2 ( P X ) norm is the largest number of functions { f 1 , . . .
, f M } ⊆
G such that (cid:107) f
i − f j (cid:107) L 2 ( P X ) ≥ δ for all i (cid:54) = j . It is easily checked that
[Equation]
For a given δ n > 0 and ε n > 0 , let Q be the δ n packing number M ( δ n , U ( MB sp,q ) , L 2 ( P X )) of
U ( MB sp,q ) and N be the ε n covering number of that.
Raskutti et al.
(2012) utilized the techniques developed by Yang & Barron (1999) to show the following inequality in their proof of Theorem 2(b):
[Equation]
Thus by taking δ n and ε n to satisfy
[Equation]
the minimax rate is lower bounded by δ 2 n 4 .
This can be achieved by properly setting ε n (cid:39) δ n .
Now, for given N with respect to δ n > 0 , M = log( N ) satisﬁes
[Equation]
(Theorem 6.24 of D˜ung et al. (2016)).
Hence, it sufﬁces to take
[Equation]
which gives the assertion.
F A UXILIARY LEMMAS
Let the (cid:15) -covering number with respect to L 2 ( P X ) for a function class
G be N ( (cid:15), G , L 2 ( P X )) as deﬁned in the proof of Theorem 4.
Proposition 4 (Schmidt-Hieber (2018)) .
Let F be a set of functions.
Let (cid:98) f be the least squares estimator in F :
[Equation]
Assume that (cid:107) f o (cid:107) ∞ ≤ F and all f ∈ F satisﬁes (cid:107) f (cid:107) ∞ ≤ F for some F ≥ 1 .
If δ > 0 satisﬁes
N ( δ, F , (cid:107) · (cid:107) ∞ ) ≥ 3 , then it holds that
[Equation]
where C is a universal constant.
Proof of Proposition 4.
This is almost direct consequence of Lemma 10 of Schmidt-Hieber (2018) 5 .
The only difference is the assumption of (cid:107) f (cid:107) ∞ ≤ F for f ∈ F and f = f
o while Lemma 10 of
Schmidt-Hieber (2018) assumed 0 ≤ f ( x ) ≤ F (cid:48) for F (cid:48) > 1 .
However, this can be easily ﬁxed by shifting the function value by + F then the range of f is modiﬁed to [0 , 2 F ] .
Then, our situation is reduced to that of Lemma 10 of Schmidt-Hieber (2018) by substituting F (cid:48) ← 2 F .
Lemma 3 (Covering number evaluation) .
The covering number of Φ( L, W, S, B ) can be bounded by [Equation] S log( δ − 1 L ( B ∨ 1) L − 1 ( W + 1) 2 L ) ≤ 2 SL log(( B ∨ 1)( W + 1))
+ S log( δ − 1 L ) .
Proof of Lemma 3.
Given a network f ∈ Φ( L, W, S, B ) expressed as f ( x ) =
( W ( L ) η
( · ) + b ( L ) ) ◦ · · · ◦ ( W (1) x
+ b (1) ) , let
A k ( f )( x ) =
η ◦ ( W ( k − 1) η ( · ) + b ( k − 1) ) ◦ · · · ◦ ( W (1) x + b (1) ) , and
B k ( f )( x ) =
( W ( L ) η
( · ) + b ( L ) ) ◦ · · · ◦ ( W ( k ) η ( x ) + b ( k ) ) , for k = 2 , . . .
, L .
Corresponding to the last and ﬁrst layer, we deﬁne B L +1 ( f )( x ) = x and
A 1 ( f )( x ) = x .
Then, it is easy to see that f ( x ) =
B k +1 ( f ) ◦ ( W ( k ) · + b ( k ) ) ◦
A k ( f )( x ) .
Now, suppose that a pair of different two networks f, g ∈ Φ( L, W, S, B ) given by f ( x ) =
( W ( L ) η
( · )+ b ( L ) ) ◦· · ·◦ ( W (1) x
+ b (1) ) , g ( x ) =
( W ( L ) (cid:48)
η ( · )+ b ( L ) (cid:48) ) ◦· · ·◦ ( W (1) (cid:48) x
+ b (1) (cid:48) ) , has a parameters with distance δ : (cid:107)W ( (cid:96) )
− W ( (cid:96) ) (cid:48) (cid:107) ∞
≤
δ and (cid:107) b ( (cid:96) ) − b ( (cid:96) ) (cid:48) (cid:107) ∞
≤ δ .
Now, not that (cid:107)A k ( f ) (cid:107) ∞ ≤ max j (cid:107)W [-( k − 1) j, : (cid:107) 
-][Equation]
[Equation] k − 1 ( f ) (cid:107) ∞ + (cid:107) b ( k − 1) (cid:107) ∞ ≤ WB (cid:107)A k − 1 ( f ) (cid:107) ∞ + B ≤ ( B ∨ 1)( W + 1) (cid:107)A k − 1 ( f ) (cid:107) ∞ ≤ ( B ∨ 1) k − 1 ( W + 1) k − 1 , and similarly the Lipshitz continuity of B k ( f ) with respect to (cid:107) · (cid:107) ∞ -norm is bounded as ( BW )
L − k +1 .
Then, it holds that
[Equation]
[Equation]
Thus, for a ﬁxed sparsity pattern (the locations of non-zero parameters), the covering number is bounded by (cid:0) δ/
[ L ( B ∨ 1) L − 1 ( W + 1) L ] (cid:1) − S .
There are the number of conﬁgurations of the sparsity pattern is bounded by (cid:0) ( W +1)
L S
[Equation]
≤ ( W + 1) LS .
Thus, the covering number of the whole space Φ is bounded as
[Equation]
which gives the assertion.