S HALLOW AND D EEP N ETWORKS ARE N EARO PTIMAL A PPROXIMATORS OF K OROBOV F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we analyze the number of neurons and training parameters that a neural network needs to approximate multivariate functions of bounded second mixed derivatives — Korobov functions.
We prove upper bounds on these quantities for shallow and deep neural networks, drastically lessening the curse of dimensionality.
Our bounds hold for general activation functions, including ReLU.
We further prove that these bounds nearly match the minimal number of parameters any continuous function approximator needs to approximate Korobov functions, showing that neural networks are near-optimal function approximators
I NTRODUCTION
Neural networks have known tremendous success in the past years in many applications such as computer vision and pattern detection (Krizhevsky et al., 2017; Silver et al., 2016).
A natural question that arises is how to explain their practical success theoretically.
Neural networks are shown to be universal (Hornik et al., 1989): any Borel-measurable function can be approximated arbitrarily well by a neural network provided the network has a sufﬁcient number of neurons.
Furthermore, universality holds for as low as 1 -hidden-layer neural network with reasonable activation functions.
However, these results do not specify the number of neurons and parameters to train needed for this approximation.
If these numbers are unreasonably high, the universality of neural networks would not explain their practical success.
We are interested in evaluating the number of neurons and training parameters needed to approximate a given function within (cid:15) with a neural network.
An interesting question is how do these numbers scale with (cid:15) and the dimensionality of the problem, that is, the number of variables of the function.
Mhaskar (1996) showed that any function of the Sobolev space of order r and dimension d can be approximated within (cid:15) with a 1 -layer neural network with O ( (cid:15)
− dr ) neurons and an inﬁnitely differentiable activation function.
This bound exhibits the curse of dimensionality: the number of neurons needed for an (cid:15) − approximation scales exponentially in the dimension of the problem d .
Thus, Mhaskar’s bound raises the question of whether this curse is inherent to neural networks.
DeVore et al.
(1989) showed an important result towards answering this question.
They proved that any continuous function approximator (a notion we formalize in Section 5) that approximates all
Sobolev functions of order r and dimension d within (cid:15) , needs at least Θ( (cid:15) − dr ) parameters.
This result bridges the gap with Mhaskar’s bound and conﬁrms that neural networks cannot escape the curse of dimensionality for the Sobolev space.
A main question is therefore for which set of functions can neural networks break this curse of dimensionality.
One way to circumvent the curse of dimensionality is to restrict considerably the considered space of functions and focus on functions with speciﬁc structures adapted to neural networks.
For example,
Mhaskar et al.
(2016) showed that compositional functions with regularity r can be approximated within (cid:15) with deep neural networks with O ( d · (cid:15) − 2 r ) neurons.
Other structural constraints have been considered for compositions of functions (Kohler & Krzy˙zak, 2016), piecewise smooth functions (Petersen & Voigtlaender, 2018; Imaizumi & Fukumizu, 2019), or structures on the data space e.g. lying on a manifold (Mhaskar, 2010; Nakada & Imaizumi, 2019; Schmidt-Hieber, 2019).
Approximation bounds have also been obtained for the function approximation from data under smoothness constraints (Kohler & Krzy˙zak, 2005; Kohler & Mehnert, 2011) and speciﬁcally on mixed smooth
Besov spaces which are known to circumvent the curse of dimensionality (Suzuki, 2018).
An- other example is the class of Sobolev functions of order d/α and dimension d for which Mhaskar’s bound becomes O ( (cid:15) − α ) .
Recently, Montanelli et al.
(2019) considered bandlimited functions and showed that they can be approximated within (cid:15) by deep networks with depth O ((log 1 (cid:15) ) 2 ) and
O ( (cid:15) − 2 (log 1 (cid:15) ) 2 ) neurons.
Weinan et al.
(2019) showed that the closure of the space of 2-layer neural networks with a particular regularity (namely a restriction on the size of the network’s weights) is the
Barron space.
They further show that Barron functions can be approximated within (cid:15) with 2-layer networks with O ( (cid:15) − 2 ) neurons.
Similar line of work restrict the function space with spectral conditions that allows functions to be written as limits of shallow networks (Barron, 1994; Klusowski &
Barron, 2016; 2018).
In this work, we are interested in more general and generic spaces of functions.
Our space of interest is the space of multivariate functions of bounded second mixed derivatives, namely the Korobov space.
This space is included in the Sobolev space but is reasonably large and general.
The Korobov space presents two motivations.
The ﬁrst motivation is theoretical: it is a natural candidate for a large and general space included in the Sobolev space where numerical approximation methods can overcome the curse of dimensionality to some extent (see Section 2.1).
The second motivation is practical: Korobov spaces are useful for solving partial differential equations (Korobov, 1959) and have been used in the context of high-dimensional function approximation (Zenger & Hackbusch, 1991; Zenger, 1991).
Recently, Montanelli & Du (2019) showed that deep neural networks with depth O (log 1 (cid:15) ) and O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)+1 ) neurons can approximate Korobov functions within (cid:15) , lessening the curse of dimensionality for deep neural networks asymptotically in (cid:15) .
While they used deep structures to prove their result, the question of whether shallow neural networks also break the curse of dimensionality for the Korobov space remains open.
In this paper we study deep and shallow neural network’s approximation power for the Korobov space and make the following contributions:
• Representation power of shallow neural networks.
We prove that any Korobov function can be approximated within (cid:15) with a 2-layer neural network with ReLU activation function with O ( (cid:15) − 1 (log 1 (cid:15) )
3( d − 1)+1 ) neurons and O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) training parameters (Theorem 3.1).
We further extend this result to a large class of commonly used activation functions (Theorem 3.4).
Asymptotically in (cid:15) , our bound can be written as O ( (cid:15) − 1 − δ ) for all δ > 0 , and in that sense breaks the curse of dimensionality for shallow neural networks.
• Representation power of deep neural networks.
We prove that any function of the Korobov space can be approximated within (cid:15) with a deep neural network of depth (cid:100) log 2 ( d ) (cid:101) + independent of (cid:15) , with non-linear C 2 activation function, O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) neurons and O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) training parameters (Theorem 4.1).
This result improves over the result in Montanelli & Du (2019) that constructed an approximating neural network with larger depth
O (log 1 (cid:15) log d ) –increasing with the precision (cid:15) – and larger number of neurons O ( (cid:15) − 12 (log 1 (cid:15) )
3( d − 1)+1 ) .
We note however that they used a different activation function, namely ReLU.
• Near-optimality of neural networks as function approximators.
Under continuous function approximators model introduced by DeVore et al.
, we prove that any continuous function approximator needs Θ( (cid:15) − 12 (log 1 (cid:15) )
d − 12 ) parameters to approximate functions of the Korobov space within (cid:15) (Theorem 5.2).
This lower bound nearly matches our established upper bounds on the number of training parameters needed by deep and shallow neural networks to approximate functions of the Korobov space, proving that deep and shallow neural networks are near-optimal function approximators of the Korobov space.
Table 1 summarizes our new bounds and some of the existing bounds on shallow and deep neural network approximation power for the Korobov space, Sobolev space and bandlimited functions.
Our proofs are constructive and give explicit structures to construct such neural networks with ReLU and general activation functions.
Our constructions rely on sparse grid approximation introduced by
Zenger (1991), and studied in detail in Bungartz (1992); Bungartz & Griebel (2004).
Speciﬁcally, we use the sparse grid approach to approximate smooth functions with sums of products then construct neural networks which approximate this structure.
A key difﬁculty is to approximate the product function.
In particular in the case of shallow neural networks, we propose, to the best of our knowledge, the ﬁrst architecture approximating the product function with polynomial number of neurons.
To derive our lower bound on the number of parameters needed to approximate the Korobov space, we construct a linear subspace of the Korobov space, with large Bernstein width.
This subspace is then used to apply a general lower bound on nonlinear approximation derived by DeVore et al.
The rest of the paper is structured as follows.
In Section 2, we deﬁne the notions of interest, formalize our objective and introduce the sparse grids approach which will be key in our constructions.
In Section 3 and 4, we prove our bounds on the number of neurons and training parameters needed by a neural network to approximate functions of the Korobov space.
Section 3 provides bounds for shallow networks and Section 4 for deep networks.
Finally, we formalize in Section 5 the notion of optimal continuous function approximators and prove our novel near-optimality result.
[Table]
[Table] Nb. of
[Table] parameters
Depth
Activation σ
Reference
W r,p (cid:15) − dr (cid:15) − dr
C ∞ , non-poly
Mhaskar (1996)
X 2 , ∞ (cid:15) − 1 (log 1 (cid:15) ) 3( d − 1)
+ (cid:15) − 12 (log 1 (cid:15) )
( d − 1)
ReLU-like
This paper
X 2 , ∞ (cid:15) − 32 (log 1 (cid:15) )
( d − 1)(cid:15)
− 12 (log 1 (cid:15) )
( d − 1)
Sigmoid-like
This paper
W r,p (cid:15) − dr log 1 (cid:15) (cid:15) − dr log 1 (cid:15)
O (log 1 (cid:15) )
ReLU
Yarotsky (2017) Liang & Srikant (2016)
Bandlimitedfunctions (cid:15) − 2 (log 1 (cid:15) )
(cid:15) − 2 (log 1 (cid:15) )
O ((log 1 (cid:15) ) 2 )
ReLU
Montanelli et al.
X 2 , ∞ (cid:15) − 12 (log 1 (cid:15) )
( d − 1) 2 + (cid:15) − 12 (log 1 (cid:15) )
( d − 1)
O (log d · log 1 (cid:15) )
ReLU
Montanelli & Du (2019)
X 2 , ∞ (cid:15) − 12 (log 1 (cid:15) )
( d − 1)(cid:15)
− 12 (log 1 (cid:15) )
( d − 1)(cid:100) log 2 d (cid:101) [Table] [Table]
2 , non-linear
This paper
P RELIMINARIES
We begin by presenting the basic deﬁnitions that will be used in the paper.
In this work, we consider feed-forward neural networks, using a linear output neuron and a non-linear activation function σ : R → R for the other neurons, such as the commonly-used rectiﬁed unit (ReLU) σ ( x ) = max( x, 0) , the sigmoid σ ( x ) =
(1 + e − x ) − 1 or the Heaviside function σ ( x ) = 1 { x ≥ 0 } .
Let d ≥ 1 be the dimension of the input.
We deﬁne a 1 -hidden-layer network with N neurons as
[Equation]
where w k ∈ R d , b k ∈ R for i = 1 , . .
, N are the parameters of the network.
A neural network with several hidden layers is obtained by feeding the outputs of a given layer as inputs to the next layer.
We are interested in the expressive power of neural networks i.e. the ability to approximate a target function f : R d → R with as few neurons as possible, on the unit hyper-cube Ω :=
[0 , 1] d .
Another relevant metric is the number of parameters that needs to be trained to approximate the function, that is, the number of parameters of the approximating network ( u k , w k and b k ) depending on the function to approximate.
We will adopt L ∞ norm as a measure of approximation error.
Let us now deﬁne some notations that are necessary to present the space of functions that we will approximate.
For an integer r , we denote C r the space of one dimensional functions differentiable r times and with continuous derivatives.
In our analysis, we consider functions f with bounded mixed derivatives.
For a multi-index α ∈ N d , we deﬁne the derivative of order α , D α [Equation]
[Equation]
where | α | 1 = (cid:80) d i =1 | α i | .
Two common function spaces in a compact Ω ⊂ R d are the Sobolev spaces W r,p (Ω) of functions having weak partial derivatives up to order r in L p (Ω) and the Korobov spaces X r,p (Ω) of functions vanishing at the boundary and having weak mixed second derivatives up to order r in L p (Ω) .
[Equation]
where ∂ Ω denotes the boundary of Ω , | α | 1 = (cid:80) di
=1 | α i | and | α | ∞ =
sup i =1 ,...,d | α i | are respectively the L 1 and inﬁnity norm.
Note that Korobov spaces X r,p (Ω) are subsets of Sobolev spaces
W r,p (Ω) .
For p = ∞ , the usual norms on these spaces are given by
[Equation]
For simplicity we will write | · | 2 , ∞ for | · | X 2 , ∞ .
We focus our analysis on approximating functions on the Korobov space X 2 , ∞ (Ω) for which the curse of dimensionality is drastically lessened and we show that neural networks are near-optimal.
Intuitively, one of the key differences compared to the Sobolev space is that Korobov functions do not have high frequency oscillations in all directions at a time.
Such functions are known to require an exponential number of neurons Telgarsky (2016) and are one of the main difﬁculties to approximating the Sobolev space which therefore exhibits the curse of dimensionality DeVore et al.
On the contrary, the Korobov space prohibits such behaviour by ensuring that functions can be differentiable on all dimensions together.
Further discussions on Korobov functions and concrete examples are provided in Appendix A.
2. T HE CURSE OF DIMENSIONALITY
In analysing neural networks representation power, we adopt the point of view of asymptotic results in (cid:15) (or equivalently, in the number of neurons), which is a well-established setting in the neural networks representation power literature (Mhaskar, 1996; Bungartz & Griebel, 2004; Yarotsky, 2017;
Montanelli & Du, 2019) and numerical analysis literature (Novak, 2006).
In the rest of the paper, for convenience, we use O notation which hide constants in d .
For each result, full dependencies in d are provided in the appendix.
Previous efforts to quantify the number of neurons needed to approximate large general class of functions within (cid:15) showed that neural networks and most classical functional approximation schemes exhibit the curse of dimensionality.
For example, in the case of Sobolev functions, Mhaskar proved the following approximation bound.
Theorem 2.1 (Mhaskar (1996)) .
Let p, r ≥ 1 , and σ : R → R be an inﬁnitely differentiable activation function, non-polynomial on any interval of R .
Let (cid:15) > 0 sufﬁciently small.
For any f ∈ W r,p , there exists a shallow neural network with one hidden layer, activation function σ , and
O ( (cid:15) − dr ) neurons approximating f within (cid:15) for the inﬁnity norm.
Therefore, the approximation of Sobolev functions by neural networks suffers from the curse of dimensionality since the number of neurons needed grows exponentially with the input space dimension d .
This curse is not due to poor performance of neural networks but rather to the choice of the Sobolev space.
DeVore et al.
(1989) proved that any learning algorithm with continuous parameters needs at least Θ( (cid:15) − d r ) parameters to approximate the Sobolev space W r,p .
This shows that the class of Sobolev functions suffers inherently from the curse of dimensionality and no continuous function approximator can overcome it.
We detail this notion later in Section 5.
The natural question that arises is whether there exists a reasonable and sufﬁciently large class of functions for which there is no inherent curse of dimensionality.
We are interested in this case in the performance of neural networks in approximating this space of functions.
Instead of the Sobolev space, we seek to add more regularity to be able to overcome the curse of dimensionality while having a reasonably large space.
The Korobov space X 2 , ∞ (Ω) —functions with bounded mixed derivatives—is the natural candidate: it is known in the numerical analysis community as a reasonably large space where numerical approximation methods can lessen the curse of dimensionality (Bungartz & Griebel, 2004).
Korobov functions have been introduced for solving partial differential equations (Korobov, 1959; Smolyak, 1963), and have then been used extensively in the context of high-dimensional function approximation (Zenger & Hackbusch, 1991; Bungartz & Griebel, 2004).
This space of functions is included in the Sobolev space, but still reasonably large as the regularity condition concerns only second order derivatives.
Two questions will be of interest.
First, how many neurons and training parameters a neural network needs to approximate any Korobov function within (cid:15) in the L ∞ norm?
Second, how does neural network performance compare to the theoretically best possible function approximators for Korobov space? [Figure]
S PARSE GRIDS AND HIERARCHICAL BASIS
We review in this subsection the notion of sparse grids which will be key in our neural networks constructions to approximate Korobov functions.
Sparse grids were introduced by Zenger Zenger (1991) and have since then been extensively used for high-dimensional function approximation.
They have been proven to lessen the curse of dimensionality for Korobov spaces.
We refer to Bungartz & Griebel (2004) for a thorough review of the topic.
The goal is to deﬁne discrete approximation spaces with basis functions.
Instead of a classical uniform grid partition of the hyper-cube [0 , 1] d involving n d components, where n is the number of partitions in each coordinate, the sparse grid approach uses a smarter partitioning of the cube preserving the approximation accuracy while drastically reducing the number of components of the grid.
The construction involves a 1 -dimensional mother function φ which is used to generate all the functions of the basis.
For example, a simple choice for the building block φ is the standard hat function φ ( x ) :
= (1 − | x | )
+ .
The hat function is not the only possible choice.
In the latter proofs we will specify which mother function is used, in our case either the interpolates of Deslauriers &
Dubuc (1989) (which we deﬁne rigorously later in our proofs) or the hat function φ which can be seen as the Deslaurier-Dubuc interpolet of order 1 .
These more elaborate mother functions enjoy more smoothness while essentially preserving the approximation power.
Assume the mother function has support in [ − k, k ] .
For j = 1 , . . .
, d , it can be used to generate a set of local functions φ l j ,i j : [0 , 1] −→ R for all l j ≥ 1 and 1 ≤ i j ≤ 2 l j − 1 with support (cid:104) [Equation] as follows, φ l j ,i j ( x ) :
= φ (2 l j x − i j ) , x ∈ [0 , 1] .
We then deﬁne a basis of d -dimensional functions by taking the tensor product of these 1 -dimensional functions.
For all l , i ∈ N d with l ≥ 1 and 1 ≤ i ≤ 2 l − 1 where 2 l denotes (2 l 1 , . . .
, 2 l d ) , deﬁne φ l , i ( x ) :
= (cid:81) dj
=1 φ l j ,i j ( x j ) , x ∈ R d .
For a ﬁxed l ∈ N d , we will consider the hierarchical increment space W l which is the subspace spanned by the functions { φ l , i : 1 ≤ i ≤ 2 l − 1 } , as illustrated in Figure 1,
[Equation]
Note that in the hierarchical increment W l , all basis functions have disjoint support.
A key observation is that Korobov functions X 2 ,p (Ω) can be expressed uniquely in this hierarchical basis.
More precisely, there is a unique representation of u ∈ X 2 ,p (Ω) as u ( x ) = (cid:80) l , i v l , i φ l , i ( x ) , where the sum is taken over all multi-indices l ≥ 1 and 1 ≤ i ≤ 2 l − 1 where all components of i are odd.
In particular, all basis functions are linearly independent.
Notice that this sum is inﬁnite, the objective is now to deﬁne a ﬁnite-dimensional subspace of X 2 ,p (Ω) that will serve as an approximation space.
Sparse grids use a carefully chosen subset of the hierarchical basis functions to construct the approximation space V (1) n := (cid:76) | l | 1 ≤ n + d − 1 W l .
For the particular choice of the hat function for φ ,
Bungartz and Griebel Bungartz & Griebel (2004) showed that this choice of approximating space leads to a good approximating error.
Theorem 2.2 (Bungartz & Griebel (2004)) .
Let f ∈ X 2 , ∞ (Ω) and f (1) n be the projection of f on the subspace V (1) n .
We have, (cid:107) f − f (1) n (cid:107) ∞ =
O (cid:0) 2 − 2 n n d − 1 (cid:1) .
Furthermore, if v l , i denotes the coefﬁcient of φ l , i in the decomposition of f (1) n in V (1) n , then we have the upper bound | v l , i | ≤ 2 − d 2 − 2 | l | 1 | f | 2 , ∞ , for all l , i ∈ N d with | l | 1 ≤ n + d − 1 , 1 ≤ i ≤ 2 l − 1 where i has odd components.
T HE R EPRESENTATION P OWER OF S HALLOW N EURAL N ETWORKS
It has recently been shown that deep neural networks, with depth scaling with (cid:15) , lessen the curse of dimensionality on the numbers of neuron needed to approximate the Korobov space Montanelli & [Figure]
· · · x d · · · · · · · · · · · · · ·
[Figure]
Du (2019).
However, to the best of our knowledge, the question of whether shallow neural networks with ﬁxed universal depth — independent of (cid:15) and d — escape the curse of dimensionality as well for the Korobov space remains open.
We settle this question by proving that shallow neural networks also lessen the curse of dimensionality for Korobov space.
Theorem 3.1.
Let (cid:15) > 0 .
For all f ∈ X 2 , ∞ (Ω) , there exists a neural network with 2 layers,
ReLU activation, O ( (cid:15) − 1 (log 1 (cid:15) )
( d − 1)+1 ) neurons, and O ( (cid:15) − 12 (log 1 (cid:15) )
3( d − 1)) training parameters that approximates f within (cid:15) for the inﬁnity norm.
In order to prove Theorem 3.1, we construct the approximating neural network explicitly.
The ﬁrst step is to construct a neural network architecture with two layers and O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons that approximates the product function p : x ∈ [0 , 1] d (cid:55)−→ (cid:81) ni
=1 x i within (cid:15) for all (cid:15) > 0 .
Proposition 3.2.
For all (cid:15) > 0 , there exists a neural network with depth 2 , ReLU activation and
O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons, that approximates the product function p : x ∈ [0 , 1] d −→ (cid:81) di
=1 x i within (cid:15) for the inﬁnity norm.
Sketch of proof The proof builds upon the observation that p ( x ) = exp( (cid:80) di
=1 log x i ) .
We construct an approximating 2 -layer neural network where the ﬁrst layer approximates log x i for 1 ≤ i ≤ d , and the second layer approximates the exponential.
We illustrate the construction of the proof in
Figure 2.
More precisely, ﬁx (cid:15) > 0 . Consider the function h (cid:15) : x ∈ [0 , 1] (cid:55)→ max(log x, log (cid:15) ) .
We approximate h (cid:15) within (cid:15)d with a piece-wise afﬁne function with O ( d 12 (cid:15) − 12 log 1 (cid:15) ) pieces and then represent this piece-wise afﬁne function with a single layer neural network ˆ h (cid:15) with the same number of neurons as the number of pieces (Lemma B.1, Appendix B.1).
This 1 -layer network veriﬁes, therefore, (cid:107) h (cid:15) − ˆ h (cid:15) (cid:107) ∞ ≤ (cid:15)d .
The ﬁrst layer of our ﬁnal network is the union of d copies of the 1 -layer network ˆ h (cid:15) : one for each dimension i , approximating log x i .
Similarly, consider the exponential g : x ∈ R − (cid:55)→ e x .
We construct a 1 -layer neural network ˆ g (cid:15) with O ( (cid:15) − 12 log 1 (cid:15) ) neurons such that (cid:107) g − ˆ g (cid:15) (cid:107) ∞ ≤ (cid:15) .
This will serve as second layer.
Formally, the constructed approximation ˆ p (cid:15) of the product function is ˆ p (cid:15)
=
ˆ g (cid:15) (cid:16)(cid:80) di
=
ˆ h (cid:15) ( x i ) (cid:17) .
This 2 -layer neural network has O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons and veriﬁes (cid:107) ˆ p (cid:15) − p (cid:107) ∞ ≤ (cid:15) .
We use this result to prove Theorem 3.1 and show that we can approximate any Korobov function f ∈ X 2 , ∞ (Ω) within (cid:15) with a 2-layer neural network of O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) neurons.
Consider the sparse grid construction of the approximating space V (1) n using the standard hat function as mother function to create the hierarchical basis W l (introduced in Section 2.2).
Recall that the approximation space is deﬁned as V (1) n := (cid:76) | l | 1 ≤ n + d − 1 W l .
The key idea is to construct a shallow neural network approximating the sparse grid approximation and then use the result of Theorem 2.2 to derive the approximation error.
Let f (1) n be the projection of f on the subspace V (1) n deﬁned in Section 2.2.
f (1) n can be written as f (1) n ( x ) =
(cid:80) ( l , i ) ∈
U (1) n v l , i φ l , i ( x ) , where U (1) n contains the indices ( l , i ) of basis functions present in V (1) n .
We can use Theorem 2.2 and choose n carefully such that f (1) n approximates f within (cid:15) for L ∞ norm.
The goal is now to approximate f (1) n with a shallow neural network.
Note that the basis functions can be written as a product of univariate functions φ l , i = (cid:81)
dj
=1 φ l j ,i j .
We can therefore use a similar structure to the product approximation of Proposition 3.2 to approximate the basis functions.
Speciﬁcally, the ﬁrst layer approximate the d (2 n − 1)
=
O ( (cid:15) − 12 (log 1 (cid:15) )
d − 12 ) terms log φ l j ,i j necessary to construct the basis functions of V (1) n and a second layer to approximate the exponential in order to obtain approximations of the O (2 n n d − 1 ) =
O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) basis functions of V (1) n .
We provide a detailed ﬁgure illustrating the construction, Figure 5 in Appendix B.3.
The shallow network that we constructed in Theorem 3.1 uses the ReLU activation function.
We extend this result to a larger class of activation functions which include commonly used ones.
Deﬁnition 3.3.
A sigmoid-like activation function σ : R → R is a non-decreasing function having ﬁnite limits in ±∞ .
A ReLU-like activation function σ : R → R is a function having a horizontal asymptote in −∞ i.e. σ is bounded in R − , and an afﬁne (non-horizontal) asymptote in + ∞ , i.e. there exists b > 0 such that σ ( x ) − bx is bounded in R + .
The majority of the commonly used activation functions fall into one of the two classes.
Examples of sigmoid-like activations include the Heaviside, logistic, tanh, arctan and softsign activations, while ReLU-like activations include the ReLU, ISRLU, ELU and soft-plus activations.
We extend
Theorem 3.1 to all these activations.
Theorem 3.4.
For any approximation tolerance (cid:15) > 0 , and for any f ∈ X 2 , ∞ (Ω) there exists a neural network with depth 2 and O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) training parameters that approximates f within (cid:15) for the inﬁnity norm, with O (cid:0) (cid:15) − 1 log( 1 (cid:15) (cid:1) 3( d − 1)+1 )
(resp.
O ( (cid:15) − 32 log( 1 (cid:15) )
( d − 1)) ) neurons for a ReLU-like (resp.
sigmoid-like) activation.
We note that these results can further be extended to more general Korobov spaces X r,p .
Indeed, the main dependence of our neural network architectures in the parameters r and p arise from sparse grid approximation.
Bungartz & Griebel (2004) show that results similar to Theorem 2.2 can be extended to various settings with the same sparse grid construction.
For instance, we can use these results combined with our proposed architecture to show that the Korobov space X r, ∞ can be approximated in inﬁnite norm by neural networks with O ( (cid:15) − 1 r (log 1 (cid:15) )
r +1 r ( d − 1) ) training parameters and same number of neurons up to a polynomial factor in (cid:15) .
T HE R EPRESENTATION P OWER OF D EEP N EURAL N ETWORKS
Montanelli and Du Montanelli & Du (2019) used the sparse grid approach to construct deep neural networks with ReLU activation, approximating Korobov functions with O ( (cid:15) − 1 2 (log 1 (cid:15) )
( d − 1)+1 ) neurons, and depth O (log 1 (cid:15) ) for the L ∞ norm.
We show that we can improve this bound for deep neural networks with C 2 non-linear activation functions.
We prove that we only need
O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) neurons and ﬁxed depth, independent of (cid:15) , to approximate the unit ball of the
Korobov space within (cid:15) in the L ∞ norm.
Theorem 4.1.
Let σ ∈ C 2 be a non-linear activation function.
Let (cid:15) > 0 .
For any function f ∈ X 2 , ∞ (Ω) , there exists a neural network of depth (cid:100) log 2 d (cid:101) + 1 , with ReLU activation on the ﬁrst layer and activation function σ for the next layers, O ( (cid:15) − 12 (log 1 (cid:15) )
3( d − 1)) neurons, and
O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) training parameters approximating f within (cid:15) for the inﬁnity norm.
Compared to the bound of shallow networks, Theorem 3.1, the number of neurons for deep networks is lower by a factor O ( √ (cid:15) ) , while the number of training parameters is exactly the same.
This result indicates that deep neural networks are more efﬁcient than shallow neural network in the sense that shallow networks need more “inactive” neurons to reach the same approximation power as deep networks while having the same number of parameters.
This gap in the number of the “inactive” neurons can be consequent in practice, as in common used architectures, we do not know exactly which neurons to train and which neurons to ﬁx.
We also point out that this new bound on the number of parameters and neurons matches the approximation power of sparse grids.
In fact, sparse grids use Θ( (cid:15) − 12 (log 1 (cid:15) )
( d [Figure] parameters (weights of basis functions) to approximate Korobov functions within (cid:15) .
Our construction in Theorem 4. shows that deep neural networks with ﬁxed depth in (cid:15) can fully encode sparse grids approximators.
Neural networks are therefore more powerful function approximators.
In particular, any sparse grid approximation using O ( N ( (cid:15) )) parameters, can be represented exactly by a neural network using O ( N ( (cid:15) )) neurons. [Figure]
[Figure]
{+− 12 (log 1 (cid:15) ) 
+}
( d − 1)) neurons and depth (cid:100) log 2 d (cid:101) + 1 .
The deep approximating neural network (see Figure 3) has a very similar structure to our construction of an approximating shallow network in the Theorem 3.1.
The main difference lies in the approximation of the product function.
Instead of using a 2 -layer neural network, we now make use of a deep network.
The following result of Lin et al.
(Lin et al., 2017, Appendix A) shows that deep neural networks can represent exactly the product function.
Proposition 4.2 (Lin et al.
Let σ be C 2 non linear activation function.
For any approximation error (cid:15) > 0 , there exists a neural network with (cid:100) log 2 d (cid:101) hidden layers and activation σ , using at most 8 d neurons arranged in a binary tree network that approximates the product function (cid:81) di
=1 x i on [0 , 1] d within (cid:15) for the inﬁnity norm.
An important remark is that the structure of the constructed neural network is independent of (cid:15) .
In particular, the depth and number of neurons is independent of the approximation precision (cid:15) , which we refer to as exact approximation.
It is known that an exponential number of neurons is needed in order to exactly approximate the product function with a 1 -layer neural network Lin et al.
(2017), however, the question of whether one could approximate the product with a shallow network and a polynomial number of neurons, remained open.
In Proposition 3.2, we answer positively to this question by constructing an (cid:15) -approximating neural network of depth 2 with ReLU activation and
O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons.
Using the same ideas as in Theorem 3.4, we can generalize this result to obtain an (cid:15) -approximating neural network of depth 2 with O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons for a ReLU-like activation, or O ( d 2 (cid:15) −
[Table]
) neurons for a sigmoid-like activation.
N EURAL N ETWORKS ARE N EAR -O PTIMAL F UNCTION A PPROXIMATORS
In the previous sections, we proved and discussed upper bounds on the number of neurons and training parameters needed by deep and shallow neural networks to approximate the Korobov space
X 2 , ∞ (Ω) .
In this section, we investigate how good is the performance of neural networks as functional approximators.
We prove a lower bound on the number of parameters needed by any continuous function approximator to approximate the Korobov space.
We will show in particular that neural networks, deep and shallow, nearly match this lower bound, making them near-optimal function approximators.
Let us ﬁrst formalize the notion of continuous function approximators, following the framework of DeVore et al.
DeVore et al.
For any Banach space X — one can think of X as a space of functions — and a subset K ⊂ X of elements to approximate, we deﬁne a continuous function approximator with N parameters as a continuous parametrization a : K → R N together with a reconstruction scheme which is a Ndimensional manifold M N : R N → X .
For any element f ∈ K , the approximation given is
M N ( a ( f )) : the parametrization a is derived continuously from the function f and then given as input to the reconstruction manifold that outputs an approximation function in X .
The error of this function approximator is deﬁned as E N,a, M N ( K [Equation] X [Equation]
| [Equation] − M N ( a ( f )) | X .
The best function approximator for space K minimizes this error.
The minimal error for space K is given by
[Equation]
In other terms, a continuous function approximator with N parameters cannot hope to approximate
K better than within E N ( K ) X .
A class of function approximators is a set of function approximators with a given structure.
For example, neural networks with continuous parametrizations are a class of function approximators where the number of parameters corresponds to the number of training parameters i.e, the parameters of the network that depend on the desired function to approximate.
We say that a class of function approximators is optimal for the space of functions K if it matches this minimal error asymptotically in N , within a constant multiplicative factor.
In other words, the number of parameters needed by the class to approximate functions in K within (cid:15) matches asymptotically, within a constant, the least number of parameters N needed to satisfy E N ( K ) X ≤ (cid:15) .
The norm considered in the approximation of the functions of K is the norm associated to the space
X .
DeVore et al. DeVore et al.
(1989) showed that this minimum error E N ( K ) X admits a lower bound characterized by the Bernstein width of the subset K ⊂ X deﬁned as
[Equation]
where the outer sup is taken over all N + 1 dimensional linear sub-spaces of X , and U ( Y ) denotes the unit ball of Y for any linear subspace Y of X .
Theorem 5.1 (DeVore et al.
Let X be a Banach space and let K ⊂ X .
Then, E N ( K ) X ≥ b N ( K ) X .
We prove a lower bound on the least number of parameters any class of continuous function approximators needs to approximate functions of the Korobov space.
Theorem 5.2.
Take X =
L ∞ (Ω) and K = { f ∈ X 2 , ∞ (Ω) : | f | X 2 , ∞ (Ω) ≤ 1 } the unit ball of the
Korobov space.
Then, there exists c > 0 such that E N ( K ) X ≥ cN 2 (log N ) d − 1 .
Equivalently, for (cid:15) > 0 , a continuous function approximator approximating the unit Korobov ball within (cid:15) in L ∞ norm uses at least Θ( (cid:15) − 12 (log 1 (cid:15) )
d − 12 ) parameters.
This lower bound matches within a logarithmic factor the upper bound on the number of training parameters needed by deep and shallow neural networks to approximate the Korobov space within (cid:15) :
O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1))
(Theorem 3.1 and Theorem 4.1).
It exhibits the same exponential dependence in d with base log 1 (cid:15) and the same main dependence on (cid:15) of (cid:15) − 12 .
Note that the upper and lower bound can be rewritten as O ( (cid:15) − 1 / 2 − δ ) for all δ > 0 .
Moreover, our constructions in Theorem 3.1 and Theorem 4.1 are continuous, which comes directly from the continuity of the sparse grid parameters (see bound on v l , i in Theorem 2.2).
Our bounds prove therefore that deep and shallow neural networks are near optimal classes of function approximators for the Korobov space.
Sketch of proof We seek an appropriate subspace X N +1 in order to get lower bound the Bernstein width b N ( K ) X , which in turn provides a lower bound on the approximation error (Theorem 5.1).
To do so, we use the Deslaurier-Dubuc interpolet of degree 2 , φ (2) (see Figure 6) which is C 2 .
Using the sparse grids approach, we construct a hierarchical basis in X 2 , ∞ (Ω) using φ (2) as mother function and deﬁne X N +1 as the approximation space V (1) n .
Here n is chosen such that the dimension of
V (1) n is roughly N + 1 .
The goal is to estimate sup { ρ : ρU ( X N +1 ) ⊂ K } , which will lead to a bound on b N ( K ) X .
To do so, we upper bound the Korobov norm by the L ∞ norm for elements of
X N +1 .
For any function u ∈ X N +1 we can write u = (cid:80) l , i v l , i · φ l , i .
Using a stencil representation of the coefﬁcients v l , i , we are able to obtain an upper bound | u | X 2 , ∞ ≤ Γ d (cid:107) u (cid:107) ∞ where Γ d =
O (2 2 n n d − 1 ) .
Then, b N ( K ) X ≥ 1 / Γ d which yields the desired bound.
Interestingly, the subspace X N +1 our proof uses to show the lower bound is essentially the same as the subspace we use to approximate Koborov functions in our proof of the upper bounds (Theorem 3.1 and 4.1).
The difference is in the choice of the interpolate φ to construct the basis functions, degree 2 for the former (which provides needed regularity for the proof), and 1 for the later.
C ONCLUSION AND D ISCUSSION
We have proved new upper and lower bounds on the number of neurons and training parameters needed by shallow and deep neural networks to approximate functions of the Korobov space.
Our bounds show that shallow and deep networks, not only lessen the curse of dimensionality but are also near-optimal function approximators of the space of Korobov functions.
This work therefore contributes to understanding the practical success of neural networks theoretically.
Our work suggests several extensions.
First, we have constructed throughout the paper explicitly deep and shallow structures nearly reaching the optimal number of parameters needed to approximate Korobov functions.
It would be very interesting to see if the theoretical near-optimality of these structures translates into powerful empirical performance.
While commonly used structures (e.g. Convolution Neural Networks, or Recurrent Neural Networks) are motivated by properties of the data such as symmetries, our structures are motivated by theoretical insights on how to optimally approximate a large general class of functions with a given number of neurons and parameters.
These new structures might prove to be competitive with commonly used structures and constitute new building blocks of neural networks in learning tasks.
Second, our upper bounds (Theorem 3.1 and 4.1) nearly match our lower bound (Theorem 5.2) on the least number of training parameters needed to approximate functions of the Korobov space.
We wonder if it is possible to close the gap between these bounds and prove neural network’s optimality as function approximators.
Using our approach, proving neural network’s optimality would boil down to proving that sparse grids are optimal function approximators by improving our lower bound to match sparse grid number of parameters O ( (cid:15) − 12 (log 1 (cid:15) ) 3( d − 1)) .
Finally, we showed the near-optimality of neural networks among the set of continuous function approximators.
An interesting question would be to explore lower bounds (analog to Theorem 5.2) that we can get when considering larger sets of function approximators.
One possibility would be that the curse of dimensionality vanishes for the Sobolev space when considering discontinuous function approximators.
In this case, it would be insightful to investigate whether some discontinuous neural network construction can break the curse of dimensionality for the Sobolev space leading to stronger approximation results.
The question is then whether neural networks are still near-optimal in these larger sets of function approximators.
A PPENDIX A O N K OROBOV F UNCTIONS
In this section, we further discuss Korobov functions X 2 ,p (Ω) .
Korobov functions enjoy more smoothness than Sobolev functions: smoothness for X 2 ,p (Ω) is measured in terms of mixed derivatives of order two.
Korobov functions
X 2 ,p (Ω) can be differentiated twice in each coordinates simultaneously, while Sobolev W 2 ,p (Ω) functions can only be differentiated twice in total.
For example, in two dimensions, for a function f to be Korobov it is required to have
[Equation]
while for f to be Sobolev it requires only
[Equation]
The former can be seen from | α | ∞ ≤ 2 and the latter from | α | 1 ≤ 2 in the deﬁnition of X r,p (Ω) and W r,p (Ω) .
We now provide intuition on why Korobov functions are easier to approximate.
One of the key difﬁculties in approximating Sobolev functions are possible high frequency oscillations which may require an exponential number of neurons (Telgarsky, 2016).
For instance, consider functions which have similar structure to W ( n...n ) (deﬁned in Subsection 2.2): for any smooth basis function φ with support on the unit cube (see Figure 6 for example), consider the linear function space formed by linear combinations of dilated function φ with support on each cube d -dimensional grid of step 2 − n .
This corresponds exactly to the construction of W ( n...n ) which uses the product of hat function on each dimension as basis function φ.
This function space can have strong oscillations in all directions at a time.
The Korobov space prohibits such behavior by ensuring that functions can be differentiated twice on each dimension, simultaneously.
As a result, functions cannot oscillate in all directions at a time without having large Korobov norm.
We end this paragraph by comparing the Korobov space to the space of bandlimited functions which was shown to avoid the curse of dimensionality (Montanelli et al., 2019).
These are functions for which the support frequency components is restricted to a ﬁxed compact.
Intuitively, approximating these functions can be achieved because the set of frequencies is truncated to a compact, which then allows to sample frequencies and obtain approximation guarantees.
Instead of imposing a hard constraint of cutting high frequencies, the
Korobov space asks for smoothness conditions which do not prohibit high frequencies but rather impose a budget for high frequency oscillations.
We precise this idea in the next example.
A concrete example of Korobov functions is given by an analogous of the function space V (1) n which we used as approximation space in the proof of our results (see Section 2.2).
Similarly to the previous paragraph, one should use a smooth basis function to ensure differentiability.
Recall that
V (1) n is deﬁned as
[Equation]
Intuitively, this approximation space introduces a ”budget” of oscillations for all dimensions through the constraint (cid:80) di
=1 l i ≤ n + d − 1 .
As a result, dilations of the basis function can only occur in a restricted set of directions at a time, which ensures that the Korobov norm stays bounded.
B P ROOFS OF S ECTIONB. A PPROXIMATING THE PRODUCT FUNCTION
In this subsection, we construct a neural network architecture with two layers and O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons that approximates the product function p : x ∈ [0 , 1] d (cid:55)−→ (cid:81) ni
=1 x i within (cid:15) for all (cid:15) > 0 , which proves Proposition 3.2.
We ﬁrst prove a simple lemma to represent univariate piece-wise afﬁne functions by shallow neural networks.
Lemma B.1.
Any one dimensional continuous piece-wise afﬁne function with m pieces is representable exactly by a shallow neural network with ReLU activation, with m neurons on a single layer.
Proof.
This is a simple consequence from Proposition 1 in Yarotsky (2017).
We recall the proof for completeness.
Let x 1 ≤ · · · ≤ x m − 1 be the subdivision of the piece-wise afﬁne function f .
We use a neural network of the form g
[Equation]
where w 0 is the slope of f on the piece ≤ x 1 , w 1 is the slope of f on the piece [ x 1 , x 2 ] ,
[Equation]
for k = 1 , · · · , m − 2 , and w m − 1 = ˜ w − (cid:80) m − 2 k
=
w k where ˜ w is the slope of f on the piece ≥ x m − 1 .
Notice that f and g coincide on all x k for 1 ≤ k ≤ m − 1 .
Furthermore, g has same slope as f on each pieces, therefore, g = f .
We can approximate univariate right continuous functions by piece-wise afﬁne functions, and then use Lemma B.1 to represent them by shallow neural networks.
The following lemma shows that
O ( (cid:15) − 1 ) neurons are sufﬁcient to represent an increasing right-continuous function with a shallow neural network.
Lemma B.2.
Let f : I −→ [ c, d ] be a right-continuous increasing function where I is an interval, and let (cid:15) > 0 .
There exists a shallow neural network with ReLU activation, with (cid:6)
d − c (cid:15) (cid:7) neurons on a single layer, that approximates f within (cid:15) for the inﬁnity norm.
Proof.
Let m = (cid:4) d − c (cid:15) (cid:5) Deﬁne a subdivision of the image interval c ≤ y 1 ≤ . . .
≤ y m ≤ d where y k = c + k(cid:15) for k = 1 , . .
, m .
Note that this subdivision contains exactly (cid:6)
d − c (cid:15) (cid:7) pieces.
Now deﬁne a subdivision of I , x 1 ≤ x 2 ≤ . . .
≤ x m by x k :=
sup { x ∈ I, f ( x ) ≤ y k } , for k = 1 , . . .
, m .
This subdivision stills has (cid:6) d − c (cid:15) (cid:7) pieces.
We now construct our approximation function ˆ f on I as the continuous piece-wise afﬁne function on the subdivision x 1 ≤ . . .
≤ x m such that ˆ f ( x k ) =
y k for all 1 ≤ k ≤ m and ˆ f is constant before x 1 and after x m (see Figure 4).
Let x ∈ I .
• If x ≤ x 1 , because f is increasing and right-continuous, c ≤ f ( x ) ≤ f ( x 1 ) ≤ y
= c
+ (cid:15) .
Therefore | f
( x ) − ˆ f ( x ) | =
| f
( x ) − ( c + (cid:15) )
| ≤ (cid:15) .
• If x k < x ≤ x k +1 , we have y k < f ( x ) ≤ f ( x k +1 )
≤ y k
+1 .
Further note that y k ≤ ˆ f ( x ) ≤ y k
+1 .
Therefore | f
( x ) − ˆ f ( x ) | ≤ y k +
− y k
= (cid:15) .
• If x m < x , then y m < f ( x ) ≤ d .
Again, | f ( x ) − ˆ f ( x ) | = | f
( x ) − y m | ≤ d − y m ≤ (cid:15)
Therefore (cid:107) f
−
ˆ
f
(cid:107) ∞ ≤ (cid:15) .
We can
now use Lemma B. to end the proof.
x y c = yyy...
d = y m (cid:15) x 2 x· · · • • • • • • • • xb = x m of the y axis of step (cid:15) and constructing a linear approximation in the pre-image of each part of the subdivision.
If the function to approximate has some regularity, the number of neurons needed for approximation can be signiﬁcantly reduced.
In the following lemma, we show that O ( (cid:15) − 12 ) neurons are sufﬁcient to approximate a C 2 univariate function with a shallow neural network.
Lemma B.3.
Let f : [ a, b ] −→ [ c, d ] ∈ C 2 , and let (cid:15) > 0 .
There exists a shallow neural network with ReLU activation, with 1 √ 2 (cid:15) min( (cid:82) (cid:112) | f (cid:48)(cid:48) | (1 + µ ( f, (cid:15) )) , ( b − a ) (cid:112) (cid:107) f (cid:48)(cid:48) (cid:107) ∞ ) neurons on a single layer, where µ ( f, (cid:15) )
→ 1 as (cid:15) → 0 , that approximates f within (cid:15) for the inﬁnity norm.
Proof.
See Appendix B.2.1.
We will now use the ideas of Lemma B.2 and Lemma B.3 to approximate a truncated log function, which we will use in the construction of our neural network approximating the product.
Corollary B.4.
Let (cid:15) > 0 sufﬁciently small and δ > 0 .
Consider the truncated logarithm function log : [ δ, 1] −→ R .
There exists a shallow neural network with ReLU activation, with (cid:15) − 12 log 1 δ neurons on a single layer, that approximates f within (cid:15) for the inﬁnity norm.
Proof.
See Appendix B.2.2.
We are now ready to construct a neural network approximating the product function and prove
Proposition 3.2.
The proof builds upon on the observation that (cid:81) di
=1 x i = exp( (cid:80) di
=1 log x i ) .
We construct an approximating 2 -layer neural network where the ﬁrst layer computes log x i for 1 ≤ i ≤ d , and the second layer computes the exponential.
We illustrate the construction of the proof in Figure 2.
Proof of Proposition 3.2 Fix (cid:15) > 0 .
Consider the function h (cid:15) : x ∈ [0 , 1] (cid:55)→ max(log x, log (cid:15) )
∈
[log (cid:15), 0] .
Using Corollary B.4, there exists a neural network ˆ h (cid:15) :
[0 , 1] −→ [log (cid:15), 0] with 1 + (cid:100) d 12 (cid:15) − 12 log 1 (cid:15) (cid:101) neurons on a single layer such that (cid:107) h (cid:15) − ˆ h (cid:15) (cid:107) ∞ ≤ (cid:15)d .
Indeed, one can take the (cid:15) -approximation of h (cid:15) : x ∈ [ (cid:15), 1] (cid:55)→ log x ∈ [log (cid:15), 0] , then extend this function to [0 , (cid:15) ] with a constant equal to log (cid:15) .
The resulting piece-wise afﬁne function has one additional segment corresponding to one additional neuron in the approximating function.
Similarly, consider the exponential g : x ∈ R − (cid:55)→ e x ∈ [0 , 1] .
Because g is right-continuous increasing, we can use Lemma B.3 to construct a neural network ˆ g (cid:15) : R − −→ [0 , 1] with 1 + (cid:108) 1 √ 2 (cid:15) log 1 (cid:15) (cid:109) neurons on a single layer such that (cid:107) g − ˆ g (cid:15) (cid:107) ∞ ≤ (cid:15) .
Indeed, again one can take the (cid:15) -approximation of g (cid:15) : x ∈ [log (cid:15), 0] (cid:55)→ e x ∈ [0 , 1] , then extend this function to ( −∞ , log (cid:15) ] with a constant equal to (cid:15) .
The corresponding neural network has an additional neuron.
We construct our ﬁnal neural network ˆ φ (cid:15) (see Figure 2) as
[Equation]
Note that ˆ φ (cid:15) can be represented as a 2 -layer neutral network: the ﬁrst layer is composed of the union of the 1+ (cid:100) d 12 (cid:15) − 12 log 1 (cid:15) (cid:101) neurons composing each of the 1 -layer neural networks ˆ h i(cid:15) : x ∈ [0 , 1] d (cid:55)→ ˆ h (cid:15) ( x i ) ∈ R for each dimension i ∈ { 1 , . . .
, d } .
The second layer is composed of the 1+ (cid:108) 1 √ 2 (cid:15) log 1 (cid:15) (cid:109) neurons of ˆ g (cid:15) .
Hence, the constructed neural network ˆ φ (cid:15) has O ( d 32 (cid:15) − 12 log 1 (cid:15) ) neurons.
Let us now analyze the approximation error.
Let x ∈ [0 , 1] d .
For the sake of brevity, denote ˆ y = (cid:80) di
=
ˆ h (cid:15) ( x i ) and y = (cid:80) di =1 log( x i ) .
We have, |
[Equation]
( y ) | ≤ (cid:15) + d (cid:89) i
= x i · | exp(ˆ y − y ) − 1 | , where we used the fact that | ˆ φ (cid:15) ( x ) − exp(ˆ y ) | = | ˆ g (cid:15) (ˆ y ) − g (ˆ y ) | ≤ (cid:107) ˆ g (cid:15) − g (cid:107) ∞ ≤ (cid:15) .
First suppose that x ≥ (cid:15) .
In this case, for all i ∈ { 1 , . . .
, d } we have | ˆ h (cid:15) ( x i ) − log( x i ) | =
| ˆ h (cid:15) ( x i ) − h (cid:15) ( x i ) | ≤ (cid:15)d .
Then, | ˆ y − y | ≤ (cid:15) .
Consequently, | ˆ φ (cid:15) ( x ) − p ( x ) | ≤ (cid:15) +max( | e (cid:15) − 1 | , | e − (cid:15) − 1 | ) ≤ 3 (cid:15) , for (cid:15) > 0 sufﬁciently small.
Without loss of generality now suppose x 1 ≤ (cid:15) .
Then ˆ y ≤ h (cid:15) ( x 1 ) ≤ log (cid:15) , so by deﬁnition of ˆ g (cid:15) , we have 0 ≤ ˆ φ (cid:15) ( x )
= ˆ g (cid:15) (ˆ y ) ≤ exp(log (cid:15) ) =
(cid:15) .
Also, 0 ≤ p ( x ) ≤ (cid:15) so ﬁnally | ˆ φ (cid:15) ( x ) − p ( x ) | ≤ (cid:15) .
Remark B.5.
Note that using Lemma B.2 instead of Lemma B.3 to construct approximating shallow networks for log and exp would yield approximation functions
ˆ h (cid:15) with O ( (cid:6) d(cid:15) log 1 (cid:15) (cid:7) ) neurons and ˆ g (cid:15) with O ( (cid:6) 1 (cid:15) (cid:7) ) neurons.
Therefore, the corresponding neural network would approximate the product p with O ( d 2 (cid:15) − 1 log 1 (cid:15) ) neurons.
B. M ISSING PROOFS OF S ECTION B. B.2. P ROOF OF L EMMA B.
Proof.
Similarly as the proof of Lemma B.2, the goal is to approximate f by a piece-wise afﬁne function ˆ f deﬁned on a subdivision x 0 = a ≤ x 1 ≤ . . .
≤ x m ≤ x m +1 =
b such that f and ˆ f coincide on x 0 , . . .
, x m +1 .
We ﬁrst analyse the error induced by a linear approximation of the function on each piece.
Let x ∈ [ u, v ] for u, v ∈
I .
Using the mean value theorem, there exists α x ∈ [ u, x ] such that f ( x ) − f ( u ) =
f (cid:48)
( α x )( x − u ) and β x ∈ [ x, v ] such that f ( v ) − f ( x ) = f
[Equation] [Equation] v − x ) .
Combining these two equalities, we get,
[Equation]
Hence,
[Equation]
We now apply this result to bound the approximation error on each pieces of the subdivision.
Let k ∈ [ m ] .
Recall ˆ f is linear on the subdivision [ x k , x k +1 ] and ˆ f ( x k ) =
f
( x k ) and ˆ f ( x k +1 ) =
f
( x k +1 ) .
Hence, for all x ∈ [ x k , x k +1 ] , ˆ f ( x ) =
f
( x k ) + ( x − x k ) f
( x k +1 )
− f ( x k ) x
k +1 − x k .
Using Equation equation 1 with u = x k and v = x k +1 , we get,
[Equation]
Therefore, using a regular subdivision with step (cid:113)
[Equation]
∞ yields an (cid:15) -approximation of f with (cid:24)[-( b − a ) √ (cid:107) f (cid:48)(cid:48) (cid:107) ∞ √ 
-]
[Equation] pieces.
We now show that for any µ > 0 , there exists an (cid:15) -approximation of f with at most (cid:82)
[Equation]
pieces.
To do so, we use the fact that the upper Riemann sum for √ f (cid:48)(cid:48) converges to the integral since √ f (cid:48)(cid:48) is continuous on [ a, b ] .
First deﬁne a partition a = X 0 ≤ X K
=
b of [ a, b ] such that the upper Riemann sum R ( √ f (cid:48)(cid:48) ) on this subdivision satisﬁes R ( √ f (cid:48)(cid:48) ) ≤ (1 + µ/ 2) (cid:82) b a √ f (cid:48)(cid:48) .
Now deﬁne on each interval I k of the partition a regular subdivision with step (cid:113)
[Equation]
Ik as before.
Finally, consider the subdivision union of all these subdivisions, and construct the approximation ˆ f on this ﬁnal subdivision.
By construction, (cid:107) f − ˆ f (cid:107) ∞ ≤ (cid:15) because the inequality holds on each piece of the subdivision.
Further, the number of pieces is
[Equation]
for (cid:15) > 0 small enough.
Using Lemma B.1 we can complete the proof.
B.2. P ROOF OF C OROLLARY B.
Proof.
In view of Lemma B.3, the goal is to show that we can remove the dependence of µ ( f, (cid:15) ) in δ .
This essentially comes from the fact that the upper Riemann sum behaves well for approximating log .
Consider the subdivision x 0 :
=
δ ≤ x 1 ≤ . . .
≤ x m ≤ x m +1 :
= 1 with m = (cid:4) 1 ˜ (cid:15) log 1 δ (cid:5) where ˜ (cid:15) :
= log(1 + √ 2 (cid:15) ) , such that x k = e log δ + k ˜
(cid:15) , for k = 0 , . . .
, m − 1 .
Denote ˆ f
the corresponding piece-wise afﬁne approximation.
Similarly to the proof of Lemma B.3, for k = 0 , . . .
, m − 1 , (cid:107) log
[Equation]
) ≤ (cid:15).
The proof follows.
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] B. P ROOF OF T HEOREM 3.1: A PPROXIMATING THE K OROBOV SPACE X
In this subsection, we prove Theorem 3.1 and show that we can approximate any Korobov function f ∈ X 2 , ∞ (Ω) within (cid:15) with a 2 -layer neural network of O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) neurons.
We illustrate the construction in Figure 5.
Our proof combines the constructed network approximating the product function and a decomposition of f as a sum of separable functions, i.e. a decomposition of the form
[Equation]
Consider the sparse grid construction of the approximating space V (1) n using the standard hat function as mother function to create the hierarchical basis W l (introduced in Section 2.2).
We recall that the approximation space is deﬁned as V (1) n
[Equation]
We will construct a neural network approximating the sparse grid approximation and then use the result of Theorem 2.2 to derive the approximation error.
Figure 5 gives an illustration of the construction.
Let f (1) n be the projection of f on the subspace V (1) n .
f (1) n can be written as
[Equation]
where U (1) n contains the indices ( l , i ) of basis functions present in V (1) n i.e.
[Equation]
all
[Equation]
Throughout the proof, we explicitly construct a neural network that uses this decomposition to approximate f (1) n .
We then use Theorem 2.2 and choose n carefully such that f (1) n approximates f within (cid:15) for L ∞ norm.
Note that the basis functions can be written as a product of univariate functions φ l , i = (cid:81)
dj
=1 φ l j ,i j .
We can therefore use the product approximation of Proposition 3.
to approximate the basis functions.
Speciﬁcally, we will use one layer to approximate the terms log φ l j ,i j and a second layer to approximate the exponential.
We now present in detail the construction of the ﬁrst layer.
First, recall that φ l j ,i j is a piece-wise afﬁne function with subdivision {+
+} ≤ i j − 1 2 l j ≤ i j 2 l j ≤ i j +1 2 l j ≤ 1 .
Deﬁne the error term ˜ (cid:15) :
= (cid:15) 2 | f | 2 , ∞ .
We consider a symmetric subdivision of the interval (cid:104) i j − 1+˜ (cid:15) 2 lj ,
[Equation]
We deﬁne it as follows:
[Equation]
where m = (cid:106) 1 (cid:15) 0 log 1˜ (cid:15) (cid:107) and (cid:15) 0 :
=
log(1 + (cid:112) 2˜ (cid:15)/d ) , such that
[Equation]
Note that with this deﬁnition, the terms log(2 l j x k − i j ) form a regular sequence with step (cid:15) 0 .
We now construct the piece-wise afﬁne function ˆ g l j ,i j on the subdivision x 0 ≤ · · · ≤ x 2 m +2 which coincides with log φ l j ,i j on x 0 , · · · , x 2 m +2 and is constant on [0 , x 0 ] and [ x 2 m +2 , 1] .
By Lemma B.1, this function can be represented by a 1 -layer neural network with as much neurons as the number of pieces of ˆ g , i.e. at most 2 (cid:113)
d ˜ (cid:15) log 1˜ (cid:15) neurons for (cid:15) sufﬁciently small.
A similar proof to that of Corollary B.4 shows that ˆ g approximates max(log φ l j ,i j , log(˜ (cid:15)/ 3)) within ˜ (cid:15)/ (3 d ) for the inﬁnity norm.
We use this construction to compute in parallel, ˜ (cid:15)/ (3 d ) -approximations of max(log φ l j ,i j ( x j ) , log ˜ (cid:15) ) for all 1 ≤ j ≤ d , and 1 ≤ l j ≤ n , 1 ≤ i j ≤ 2 l j where i j is odd.
These are exactly the 1 -dimensional functions that we will need, in order to compute the d -dimensional function basis of the approximation space V (1) n .
There are d (2 n − 1) such univariate functions, therefore our ﬁrst layer contains at most 2 n +1 d (cid:113) 3 d ˜ (cid:15) log 1˜ (cid:15) neurons.
We now turn to the second layer.
The result of the ﬁrst two layers will be ˜ (cid:15)/ 3 -approximations of φ l , i for all ( l , i )
∈ U (1) n .
Recall that U (1) n contains the indices for the functions forming a basis of the approximation space V (1) n .
To do so, for each indexes ( l , i )
∈ U (1) n we construct a 1 -layer neural network approximating the function exp , which will compute an approximation of exp(ˆ g l 1 ,i 1 + · · · + ˆ g l d ,i d ) .
The approximation of exp is constructed in the same way as for Lemma B.3.
Consider a regular subdivision of the interval [log(˜ (cid:15)/ 3) , 0] with step (cid:112) 2(˜ (cid:15)/ 3) , i.e. x 0 :
= log(˜ (cid:15)/ 3) ≤ x 1 ≤ · · · ≤ x m ≤ x m +1 = 0 where m = (cid:106)(cid:113) 32˜ (cid:15) log 3˜ (cid:15) (cid:107) , such that x k = log ˜ (cid:15) + k √ 2˜ (cid:15), 0 ≤ k ≤ m.
Construct the piece-wise afﬁne function ˆ h on the subdivision x 0 ≤ · · · ≤ x m +1 which coincides with exp on x 0 , · · · , x m +1 and is constant on ( −∞ , x 0 ] .
Lemma B.3 shows that ˆ h approximates exp on R − within ˜ (cid:15) for the inﬁnity norm.
Again, Lemma B.1 gives a representation of ˆ h as a 1 -layer neural network with as many neurons as pieces in ˆ h i.e. 1 + (cid:108)(cid:113) 32˜ (cid:15) log 3˜ (cid:15) (cid:109) .
The second layer is the union of 1 -layer neural networks approximating exp within ˜ (cid:15)/ 3 , for each indexes ( l , i )
∈ U (1) n .
Therefore, the second layer contains (cid:12)(cid:12)(cid:12) U (1) n (cid:12)(cid:12)(cid:12) (cid:16) 1 + (cid:108)(cid:113) 32˜ (cid:15) log 3˜ (cid:15) (cid:109)(cid:17) neurons.
As shown in Bungartz &
Griebel (2004),
[Equation]
Therefore, the second layer has O (cid:16) 2 n n d − 1 ( d − 1)!
˜ (cid:15) − 12 log 1˜ (cid:15) (cid:17) neurons.
Finally, the output layer computes the weighted sum of the basis functions to approximate f (1) n .
Denote by ˆ f (1) n the corresponding function of the constructed neural network (see Figure 5), i.e.
[Equation]
Let us analyze the approximation error of our neural network.
The proof of Proposition 3.2 shows that the output of the two ﬁrst layers h (cid:16)(cid:80) dj
=1 ˆ g ( · j ) (cid:17) approximates φ l , i within ˜ (cid:15) .
Therefore, we obtain (cid:107) f (1) n − ˆ f (1) n (cid:107) ∞ ≤ ˜ (cid:15) (cid:80) ( l , i ) ∈
U (1) n | v l , i | .
We now use approximation bounds from Theorem 2.
on f (1) n .
[Equation]
where
[Equation]
Let us now take n (cid:15)
=
min (cid:110) n
[Equation]
Then, using the above inequality shows that the neural network ˆ f (1) n (cid:15) approximates f within (cid:15) for the inﬁnity norm.
We will now estimate the number of neurons in each layer of this network.
Note that n (cid:15) ∼
[Equation]
We can use the above estimates to show that the constructed neural network has at most N 1 (resp. N 2 ) neurons on the ﬁrst (resp. second) layer where
[Equation]
[Equation] [Equation]
d [Equation]
[Equation]
f
[Equation]
[Equation] (cid:15) (cid:18) log(cid:15) (cid:19) d [Equation] N
[Equation]
[Equation]
[Equation]
( d
[Equation]
[Equation]
d
[Equation]
32 · | f |
[Equation]
(cid:15) (cid:18) log(cid:15) (cid:19) 3( d − 1)+ .
This proves the bound the number of neurons.
Finally, to prove the bound on the number of training parameters of the network, notice that the only parameters of the network that depend on the function f are the parameters corresponding to the weighs v l,i of the sparse grid decomposition.
This number is | U (1) n (cid:15) | = O (
[Equation]
n (cid:15) n d − 1 (cid:15)
[Equation]
B. P ROOF OF T HEOREM 3.4: G ENERALIZATION TO GENERAL ACTIVATION FUNCTIONS
We start by formalizing the intuition that a sigmoid-like (resp.
ReLU-like) function is a function that resembles the Heaviside (resp.
ReLU) function by zooming out along the x (resp.
x and y ) axis.
Lemma B.6.
Let σ be a sigmoid-like activation with limit a (resp.
b ) in −∞ (resp.
+ ∞ ).
For any δ > 0 and error tolerance (cid:15) > 0 , there exists a scaling M > 0 such that x (cid:55)→ σ ( Mx ) b − a − a approximates the Heaviside function within (cid:15) outside of ( − δ, δ ) for the inﬁnity norm.
Furthermore, this function has values in [0 , 1] .
Let σ be a ReLU-like activation with asymptote b · x + c in + ∞ .
For any δ > 0 and error tolerance (cid:15) > 0 , there exists a scaling M > 0 such that x (cid:55)→ σ ( Mx ) Mb approximates the ReLU function within (cid:15) for the inﬁnity norm.
Proof.
Let δ, (cid:15) > 0 and σ a sigmoid-like activation with limit a (resp.
b ) in −∞ (resp.
+ ∞ ).
There exists x 0 > 0 sufﬁciently large such that ( b − a ) | σ ( x ) − a | ≤ (cid:15) for x ≤ − x 0 and ( b − a ) | σ ( x ) − b | ≤ (cid:15) for x ≥ x 0 .
It now sufﬁces to take M := x 0 /δ to obtain the desired result.
Now let σ be a ReLU-like activation with oblique asymptote bx in + ∞ where b > 0 .
Let M such that | σ | ≤ Mb(cid:15) for x ≤ 0 and | σ ( x ) − bx | ≤ Mb(cid:15) for x ≥ 0 .
One can check that | σ ( Mx ) Mb | ≤ (cid:15) for x ≤ 0 , and | σ ( Mx )
Mb − x | ≤ (cid:15) for x ≥ 0 .
Using this approximation, we reduce the analysis of sigmoid-like (resp.
ReLU-like) activations to the case of a Heaviside (resp ReLU) activation to prove the desired theorem.
Proof of Theorem 3.4 We start by the class of ReLU-like activations.
Let σ be a ReLU-like activation function.
Lemma B.6 shows that one can approximate arbitrarily well the ReLU activation with a linear map σ .
Take the neural network approximator ˆ f of a target function f given by Theorem 3.1.
At each node, we can add the linear map corresponding to x (cid:55)→ σ ( Mx ) Mb with no additional neuron nor parameter.
Because the approximation is continuous, we can take M > 0 arbitrarily large in order to approximate ˆ f with arbitrary precision on the compact [0 , 1] d .
The same argument holds for sigmoid-like activation functions in order to reduce the problem to
Heaviside activation functions.
Although quadratic approximations for univariate functions similar to Lemma B.3 are not valid for general sigmoid-like activations – in particular the Heaviside — we can obtain an analog to Lemma B.2 as Lemma B.7 given in the Appendix B.4.1.
This results is an increased number of neurons.
In order to approximate a target function f ∈ X 2 , ∞ (Ω) , we use the same structure as the neural network constructed for ReLU activations and use the same notations as in the proof of Theorem 3.1.
The ﬁrst difference lies in the approximation of log φ l j ,i j in the ﬁrst layer.
Instead of using Corollary B.4, we use Lemma B.7.
Therefore, 12 d ˜ (cid:15) log 3˜ (cid:15) neurons are needed to compute a ˜ (cid:15)/ (3 d ) -approximation of max(log φ l j ,i j , log(˜ (cid:15)/ 3)) .
The second difference is in the approximation of the exponential in the second layer.
Again, we use Lemma B.7 to construct a ˜ (cid:15)/ 3 -approximation of the exponential on R − with 6˜ (cid:15) neurons for the second layer.
As a result, the ﬁrst layer contains at most 2 n +2 3 d 2 ˜ (cid:15) log 1˜ (cid:15) neurons for (cid:15) sufﬁciently small, and the second layer contains (cid:12)(cid:12)(cid:12) U (1) n (cid:12)(cid:12)(cid:12) 6˜ (cid:15) neurons.
Using the same estimates as in the proof of Theorem 3.1 shows that the constructed neural network has at most N 1 (resp.
N 2 ) neurons on the ﬁrst (resp.
second) layer where
[Equation]
This ends the proof.
B.4. P ROOF OF L EMMA B.
Lemma B.7.
Let σ be a sigmoid-like activation.
Let f : I −→ [ c, d ] be a right-continuous increasing function where I is an interval, and let (cid:15) > 0 .
There exists a shallow neural network with activation σ , with at most 2 d − c (cid:15) neurons on a single layer, that approximates f within (cid:15) for the inﬁnity norm.
Proof.
The proof is analog to that of Lemma B.2.
Let m = (cid:98) d − c (cid:15) (cid:99) .
We deﬁne a regular subdivision of the image interval c ≤ y 1 ≤ . . .
≤ y m ≤ d where y k = c + k(cid:15) for k = 1 , . .
, m , then using the monotony of f , we can deﬁne a subdivision of I , x 1 ≤ . . .
≤ x m such that x k :=
sup { x ∈ I, f ( x ) ≤ y k } .
Let us ﬁrst construct an approximation neural network ˆ f with the Heaviside activation.
Consider
[Equation]
Let x ∈ [ c, d ] and k such that x ∈ [ x k , x k +1 ] .
We have by monotony y k ≤ f ( x ) ≤ y k +
and y
k
= y
+ ( k − 1) (cid:15) ≤ ˆ f ( x ) ≤ y 1 + k(cid:15) =
y k +1 .
Hence, ˆ f approximates f within (cid:15) in inﬁnity norm.
Let δ < min i
=1 ,...,m ( x i +1 − x i ) / 4 and σ a general sigmoid-like activation with limits a in −∞ and b in + ∞ .
Take M given by Lemma B.7 such that σ ( Mx ) b − a − a approximates the Heaviside function within 1 /m outside of ( − δ, δ ) and has values in [0 , 1] .
Using the same arguments as above, the function
[Equation]
approximates f within 2 (cid:15) for the inﬁnity norm.
The proof follows.
C P ROOFS OF S ECTIONC. P ROOF OF T HEOREM 4.1: A PPROXIMATING K OROBOV FUNCTIONS WITH DEEP NEURAL NETWORKS
Let (cid:15) > 0 .
We construct a similar structure to the network deﬁned in Theorem 3.1 by using the sparse grid approximation of Subsection 2.2.
For a given n , let f (1) n be the projection of f in the approximation space V (1) n (deﬁned in Subsection 2.2) and U (1) n (deﬁned in equation 2) the set of indices ( l , i ) of basis functions present in V (1) n .
Recall f (1) n can be uniquely decomposed as
[Equation]
where φ l , i = (cid:81) dj
=1 φ l j ,i j are the basis functions deﬁned in Subsection 2.2.
In the ﬁrst layer, we compute exactly the piece-wise linear hat functions φ l j ,i j , then in the next set of layers, we use the product-approximating neural network given by Proposition 4.2 to compute the basis functions φ l , i = (cid:81) dj
=1 φ l j ,i j
(see Figure 3).
The output layer computes the weighted sum (cid:80) ( l , i )
∈ U (1) n v l , i φ l , i ( x ) and outputs f (1) n .
Because the approximation has arbitrary precision, we can chose the network of Proposition 4.2 such that the resulting network ˆ f veriﬁes (cid:107) ˆ f − f (1) n (cid:107) ∞ ≤ (cid:15)/ 2 .
More precisely, as φ l j ,i j is piece-wise linear with four pieces, we can compute it exactly with four neurons with ReLU activation on a single layer (Lemma B.1).
Our ﬁrst layer is composed of the union of all these ReLU neurons, for the d (2 n − 1) indices l j , i j such that 1 ≤ j ≤ d , 1 ≤ l j ≤ n , 1 ≤ i j ≤ 2 l j and i j is odd.
Therefore, it contains at most d 2 n +2 neurons with ReLU activation.
The second set of layers is composed of the union of product-approximating neural networks to compute φ l ,
i for all ( l , i ) ∈ U (1) n .
This set of layers contains (cid:100) log 2 d (cid:101) layers with activation σ and at most | U (1) n | · 8 d neurons.
The output of these two sets of layers is an approximation of the basis functions φ l , i with arbitrary precision.
Consequently, the ﬁnal output of the complete neural network is an approximation of f (1) n with arbitrary precision.
Similarly to the proof of Theorem 3.1, we can chose the smallest n such that (cid:107) f − f (1) n (cid:107) ∞ ≤ (cid:15)/ 2 (see equation 3 for details).
Finally, the network has depth at most log 2 d + 2 and N neurons where
[Equation]
The parameters of the network depending on the function are exactly the coefﬁcients v l , i of the sparse grid approximation.
Hence, the network has O ( (cid:15) − 12 (log 1 (cid:15) )
( d − 1)) training parameters.
D P ROOFS OF S ECTIOND. P ROOF OF T HEOREM 5.2: N EAR -O PTIMALITY OF N EURAL N ETWORKS FOR K OROBOV FUNCTIONS
Our goal is to deﬁne an appropriate subspace X N +1 in order to get a good lower bound on the
Bernstein width b N ( K ) X , deﬁned in equation 5, which in turn provides a lower bound on the approximation error (Theorem 5.1).
To do so, we introduce the Deslaurier-Dubuc interpolet φ ( L ) :
R → R .
The construction of this function uses an interpolating scheme on binary rationals.
First, φ ( L ) is deﬁned on all integers φ ( L ) ( k ) = 1 k =0 for k ∈ Z .
Then, we deﬁne the function on half integers 2 k + by ﬁtting a polynomial of degree 2 L − 1 interpolating the standard hat function φ at k − L + 1 , · · · , k + L .
Iteratively, we deﬁne the interpolet on binary rationals of the form 2 k +1
j +1 from the value of the interpolet on rationals with denominator 2 j .
Speciﬁcally, let P ( L ) j,k be the unique polynomial of degree 2 L interpolating φ ( L ) at points k (cid:48)
2 j for k (cid:48) ∈ { k − L + 1 , · · · , k + L } .
We set
[Equation]
−−−    −−−    −−−    φ (1) φ (2) φ (3)
For example, for L = 2 we get φ (2) ( 2 k +1 2 j +1 ) :
= 9 16 φ (2) ( k 2 j ) + 9 16 φ (2) ( k +1 2 j ) − 1 16 φ (2) ( k − 2 2 j ) − 116 φ (2) ( k +32 j ) .
This process deﬁnes the interpolet on binary rationals.
We then extend the function to the real line by continuity.
See Figure 6 for an illustration.
Deslauriers & Dubuc (1989) proved that the regularity of the interpolet is an increasing function of L the degree of interpolation.
We now prove results in the case L = 2 .
Lemma D.1.
The interpolet of degree 2 , φ (2) is C 2 and has support Supp (cid:0) φ (2) (cid:1) =
[ − 3 , 3] .
Proof.
See Appendix D.2.1.
We will now use the interpolet φ (2) to construct the subspace X N +1 .
Using the sparse grids approach, we can construct a hierarchical basis in X 2 , ∞ (Ω) using the interpolet φ (2) as mother function.
In the remaining of the proof, we will write φ instead of φ (2) for simplicity, and use the notations and deﬁnitions related to sparse grids, introduced in Subsection 2.2.
Because E N ( K ) is decreasing in N , it sufﬁces to show the result for N n when we deﬁne our space
X N +1 to be exactly the approximating space V (1) n of sparse grids X N n +1 :
= V (1) n .
The following equation establishes the relation between n and N n .
In the following, for simplicity, we will write N instead of N n .
[Equation]
First, let us give some properties about the subspace X N +1 .
Proposition D.2.
Let u ∈ X N +1 and write it in decomposed form u = (cid:80) l , i v l , i · φ l , i , where the sum is taken over all multi-indices corresponding to basis functions of X N +1 .
The coefﬁcients v l , i can be computed in the following way.
[Equation]
where I l j ,i j u = u
( i j 2 j ) − 916 u
( i j − 1 2 j ) − 916 u
( i j +12 j ) + 116 u ( i j − 3 2 j ) + 116 u ( i j +32 j ) .
Here, I l , i denotes the d -dimensional stencil which gives a linear combination of values of u at 5 d nodal points.
Proof.
See Appendix D.2.2.
Note that the stencil representation of the coefﬁcients gives directly | v l , i | ≤ 5 d (cid:107) u (cid:107) ∞ .
We are now ready to make our estimates.
The goal is to compute sup { ρ : ρU ( X N +1 ) ⊂ K } , which will lead to a bound on b N ( K ) X .
To do so, it sufﬁces to upper bound the Korobov norm by the L ∞ norm for elements of X N +1 .
In fact, if Γ d > 0 satisﬁes
Now take u ∈ X N +1 and let us write u = (cid:80) l , i v l , i · φ l , i .
Note that basis functions in the same hierarchical class W l are almost disjoint.
More precisely, at each point x ∈ Ω , at most 3 d basis functions φ l , i have non-zero contribution to u ( x ) .
Therefore, for any 0 ≤ α ≤ 2 · 1 , (cid:107)
[Equation]
α u
[Equation]
Finally, denoting by C d the constant 60 d ( d − 1)!
| φ | X 2 , ∞ (Ω) , we get for n sufﬁciently large [Equation]
Furthermore, recall N =
[Equation]
[Equation] [Equation] 1 + O (cid:0) 1 n (cid:1)(cid:1) .
Therefore, n ∼ log N log2 , and 2 n ∼ ( d − 1)!(log 2)
d − 1 ·
N (log N ) d − 1 .
Finally we obtain for some constant c d > 0 ,
[Equation]
We conclude by analyzing the minimum number of parameters in order to get an (cid:15) -approximation of the Korobov unit ball K .
Deﬁne n (cid:15) :
= min { n : 2 C d 1 2 2 n · n d − 1 ≤ (cid:15) } .
This yields n (cid:15) ∼ 1 2log2 log 1 (cid:15) , and 2 n ∼ (cid:113) 2 C d (2 log 2) d −1 √ (cid:15) · (log 1 (cid:15) )
d − 12 .
The number of needed parameters to obtain an (cid:15) approximation K is therefore
[Equation]
for some constant ˜ C d > 0 .
Interestingly, the subspace X N +1 our proof uses to show the lower bound is essentially the same as the subspace we use to approximate Koborov functions in our proof of the upper bounds (Theorem 3.1, 4.1).
The difference is in the choice of the interpolate φ to construct the basis functions, degree 1 for the former, and 2 for the later.
D. M ISSING PROOFS OF S UBSECTION D. D.2. P ROOF OF L EMMA D.
Lemma D.3.
The interpolet of degree 2 , φ (2) is C 2 and has support Supp (cid:0) φ (2) (cid:1) =
[ − 3 , 3] .
Proof.
To analyze the regularity of the interpolet, we introduce the trigonometric polynomial
[Equation]
We can write P (2) ( θ ) as
[Equation]
· S ( θ ) , where S ( θ ) = 14 − 116 ( e iθ + e − iθ ) is a trigonometric polynomial of degree 1 .
Deslaurier and Dubuc (Deslauriers & Dubuc, 1989, Theorem 7.11) showed that the interpolet φ (2) has regularity (cid:106) − log r log2 (cid:107) where r is the spectral radius of the matrix B :=
[ s j − 2 k ] − 1 ≤ j,k ≤ 1 where s j are the coefﬁcients of the trigonometric polynomial S ( θ ) .
In our case, matrix B writes
B = (cid:32) − 1 / 16 − 1 /  1 /  − 1 / 16 − 1 /(cid:33) , and has spectral radius r = 1 / 4 .
Therefore, the regularity of φ (2) is at least (cid:106) − log r log2 (cid:107)
= 2 .
For the support, one can check that more generally Supp ( φ ( L ) ) =
[ − 2 L + 1 , 2 L − 1] .
D.2. P ROOF OF P ROPOSITION D.
Proposition D.4.
Let u ∈ X N +1 and write it in decomposed form u = (cid:80) l , i v l , i · φ l , i , where the sum is taken over all multi-indices corresponding to basis functions of X N +1 .
The coefﬁcients v l , i can be computed in the following way.
[Equation]
where I l j ,i j u = u
( i j 2 j ) − 916 u ( i j − 1 2 j ) − 916 u
( i j +12 j ) + 116 u ( i j − 3 2 j ) + 116 u ( i j +32 j ) .
Here, I l , i denotes the d -dimensional stencil which gives a linear combination of values of u at 5 d nodal points.
Proof.
We start by looking at the 1 -dimensional case.
One can check that I l,i φ ˜ l, ˜ i = 1 ˜ l = l, ˜ i
= i .
Indeed, if ˜ l > l of if ˜ l = l and ˜ i (cid:54) =
i , then φ ˜ l, ˜ i will be zero at the nodal values of I l,i .
Further, if ˜ l < l the iterative construction of φ gives directly.
[Equation]
Therefore, we obtain I l,i φ ˜ l, ˜ i = 0 .
Finally, I l,i u
= (cid:80) ˜ l, ˜
i v
˜ l. ˜
i · I l,i φ ˜ l, ˜ i
= v l,i · I l,i φ l,i = v l,i .
This proves the stencil representation of v l , i for dimension 1 .
Finally, using the tensor product approach of the stencil operator I l , i we can generalize the formula to general dimensions.