Approximation with CNNs in Sobolev Space: with Applications to Classiﬁcation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We derive a novel approximation error bound with explicit prefactor for Sobolevregular functions using deep convolutional neural networks (CNNs).
The boundis non-asymptotic in terms of the network depth and ﬁlter lengths, in a ratherﬂexible way.
For Sobolev-regular functions which can be embedded into the
Hölder space, the prefactor of our error bound depends on the ambient dimensionpolynomially instead of exponentially as in most existing results, which is ofindependent interest.
We also establish a new approximation result when thetarget function is supported on an approximate lower-dimensional manifold.
Weapply our results to establish non-asymptotic excess risk bounds for classiﬁcationusing CNNs with convex surrogate losses, including the cross-entropy loss, thehinge loss (SVM), the logistic loss, the exponential loss and the least squares loss.
We show that the classiﬁcation methods with CNNs can circumvent the curse ofdimensionality if input data is supported on a neighborhood of a low-dimensionalmanifold.
Introduction
Classiﬁcation methods with hypothesis spaces speciﬁed through deep neural networks have achievedremarkable successes in a variety of machine learning tasks [36].
In particular, convolutional neuralnetworks (CNNs) have demonstrated outstanding performance in many applications, includingcomputer vision [35], natural language processing [58], and sequence analysis in bioinformatics[1, 66].
However, to the best of our knowledge, there has only been limited study on the approximationwith CNNs and systematic studies on the properties of various classiﬁcation methods using CNNs interms of the non-asymptotic bounds for the excess risks.
1.1 Related work
There has been much effort devoted to understanding the approximation properties of deep neuralnetworks in recent years.
Many interesting results have been obtained concerning the approximationpower of deep neural networks for multivariate functions; some examples of important recent worksinclude [60, 61, 40, 50, 54, 55, 44, 13, 28].
These works focused on the approximation power for
ReLU activated feedforward neural networks (FNNs) on different kinds of smooth functions.
The approximation power of CNNs has been studied in limited works.
[2] studied the approximationpower of the composited CNNs, where the network consists of fully-connected CNNs followed by afully-connected layer.
They proved that under suitable conditions, convolution neural networks caninherit the universal approximation property of its last fully connected layers.
[49] showed that fortranslation equivariant functions, all upper and lower approximation bounds are equivalent between
FNNs and CNNs with the same order of network length and size.
[47] showed that ResNet-type
CNNs, a special CNN architecture with skip-layer connections (or identity connections betweeninconsecutive layers), can replicate the learning ability of FNNs having block-sparse structures.
The
ResNet-type CNN in [47], as claimed, can be dense, and its width, channel size, and ﬁlter size areconstant with respect to sample size rather than the diverging one in [49].
In addition, a series of work studied the approximation power of deep CNNs with Toeplitz typeconvolutional matrices [67, 65, 64].
The universality of such CNNs was established in [65] for targetfunctions restricted to Sobolev space W β, 2 ( R d ) with β ≥ d/ 2 + 2 .
The approximation error is oforder
O
( L − 1 / 2 − 1 /d )
(up to logarithms), where L is the number of layers of such CNN.
But the CNNdeﬁned in [65] has width increasing linearly with respect to depth.
To overcome the difﬁculty intheoretical analysis of such CNNs, [64] introduced a downsampling operator to reduce the widths andthe approximation power of downsampled CNNs for a special class of functions ( β -Hölder continuousridge function) was studied.
The approximate rate is shown to be O ( N − β ) by downsampled CNNswith a uniform ﬁlter length s
= O (4 N + 6) and width W
= O ( N ) for positive integer N .
Further, itwas shown that a downsampled CNN can compute the same function as a FNN does, with the totalnumber of free parameters of downsampled CNN being at most 8 times of that of FNN.
For the CNNsconsidered by [67, 16, 64] and [65], each convolutional layer has only one ﬁlter with size s and theweight matrix is of a special Toeplitz type with shape ( d in + s )
× d in and no fully-connected layeris allowed in hidden layers.
The paper [27] studied the approximation properties of convolutionalarchitectures for the target functions deﬁned on inﬁnite time domains tailored to temporal sequencedata.
To mitigate the curse of dimensionality and show the advantages of CNNs in classiﬁcation problems,several works studied the approximation as well as the statistical learning theories under somelow-dimensional assumptions on the target function or the data distribution.
In [32, 33], the posteriorprobability function is assumed to be Hölder smooth satisfying hierarchical max-pooling model.
In [34], the target function is assumed to be spatially rotation invariant.
In [18] and [38], inputdata is assumed to be supported on spheres and manifolds, respectively.
Under different structuralassumptions on the target function, similar results have also been proved for FNNs [7, 52, 53].
Ifdata is assumed to be supported on low-dimensional manifolds, the curse of dimensionality can alsobe mitigated for FNNs [51, 44, 12, 28].
1.2 Our contributions
We study the approximation power of CNNs for functions in Sobolev spaces, including the specialcase of Hölder class and apply our new approximation results to establish non-asymptotic errorbounds for several important classiﬁcation methods using CNNs.
Our main contributions are asfollows.
(i)
We derive a novel error bound for approximating Sobolev-regular functions using deep CNNs,with the error bound explicitly expressed in terms the network parameters and model parameters.
The error bound depends on the network depth and ﬁlter lengths in a rather ﬂexible way.
Forfunctions which can be embedded into Hölder space, the prefactor of our error bound depends
on the ambient dimension polynomially instead of exponentially as in existing results.
We also
establish a novel approximation result when the target function is supported on an approximate
lower-dimensional manifold, which shows that CNNs are capable of mitigating the curse ofdimensionality under such a distributional condition.
(ii)
As an application of our CNN approximation results, we establish non-asymptotic bounds forthe stochastic and approximation errors of classiﬁcation methods with a general class of convex
surrogate losses in Sobolev space using CNNs.
Our main results are also applicable to otherproblems that use CNNs approximations.
(iii)
We apply our general results
to establish the non-asymptotic excess risk bounds for classiﬁcation using CNNs with convex surrogate losses in Sobolev space, including the cross-entropy loss, the hinge loss (SVM), the logistic loss, the exponential loss, and the least squares loss.
We show that classiﬁcation methods with CNNs can circumvent the curse of dimensionality if the input data is supported on a neighborhood of a low-dimensional manifold embedded in the ambient space.
[Table]
In Table 1, we summarize and compare some recent and most related work on CNN approximationresults for various function spaces.
The statistical convergence properties of the excess risk of the empirical risk minimizer (ERM) forthe misclassiﬁcation 0-1 loss have been studied extensively under various conditions.
Different typesof hypothesis spaces have been used in these approaches, including the linear space, the reproducingkernel Hilbert space and the class of tree-based models, see, for example, [9, 56, 42, 6, 29, 8] and[63].
However, there are limited studies worked on deep binary classiﬁcations by FNNs [46, 25, 30],
ResNet [26, 45] and CNNs [32, 33, 38].
Approximation power of CNNs2.1 Convolutional neural networks
Convolutional networks are a special type of structured sparse feedforward neural network (FNN) thatuse convolution in place of general matrix multiplication in at least one of their layers [21].
There aredifferent formulations of CNNs in the literature [2, 67, 47, 37, 64, 65, 33, 32, 38].
In this paper, weconsider ReLU activated downsampled CNNs with bias vectors in the convolutional layers deﬁnedin [64].
We also do not require the norm of network parameters (weight and bias) to be uniformlybounded.
We consider a general CNN function f CNN :
X → R , where X ⊂ R d .
For simplicity, we take
X = (0 , 1) d .
Suppose f CNN has L number of hidden layers, then it can be expressed as
[Equation]
where ◦ denotes the functional composition.
The A i ’s are either convolutional operators or downsampling operators.
For convolutional layers, A i ( x ) =
σ ( W ci x + b ci ) , where W ci ∈ R d i × d i − 1 is thestructured sparse Toeplitz type weight matrix induced by the convolutional ﬁlter { w ( i ) j } s ( i ) j =0 with ﬁlterlength and s ( i ) ∈ N + , b
ci ∈ R d i is a bias vector, and σ is the rectiﬁed linear unit (ReLU) activationfunction applying to each component of the input vector.
For downsampling layers, for any x ∈ R d i − 1 , A i ( x ) =
D i ( x ) =
( x jm i ) (cid:98) d i − 1 /m
i (cid:99) j
= , where D i :
R d i × d i − 1 is the downsampling operator with scaling parameter m i ≤
d i − 1 in the i -th layer.
Weintroduce the convolutional operation and downsampling operation [64] in details in Appendix B.
We focus on the function class consisting of deep CNNs, denoted by F CNN which is deﬁned as
F CNN =
{ f CNN deﬁned in (1) over all possible choiceof A i , i = 1 , . . .
, L + 1 } .
For a general function class F and any f ∈ F , deﬁne the (cid:107) · (cid:107) ∞ metric on F by sup x ∈X (cid:107) f
( x ) (cid:107) ∞ .
Then let B := sup f ∈F CNN (cid:107) f (cid:107) ∞ denote the bound of the functions in F CNN .
Let L be th number ofhidden layers and S be the total number of parameters for networks in F CNN and let s min and s maxbe the minimum and maximum ﬁlter length over convolutional layers respectively.
2.2 Approximation in Sobolev space
We use the Sobolev and Hölder classes for the target functions in our approximation results sincethey are well-established formulations for describing smooth functions in the literature.
Indeed, earlier works on approximation theory of neural networks have been developed for Sobolev functions[60, 65, 24] and for Hölder functions [33, 52].
Let N 0 be the set of non-negative integers and β ∈ N 0 .
The Sobolev class of functions W β,p ( X )with X = (0 , 1) d is deﬁned as
[Equation]
all
[Equation]
where D α
[Equation]
with α
[Equation]
=
[Equation]
di =1 α i and ≤ p ≤ ∞ .
For f ∈
W β,p ( X ) and≤ m ≤ β , we deﬁne the norm
[Equation]
[Equation]
D α f (cid:107) pL p ( X ) (cid:1) 1 /p for 1 ≤ p < ∞ , and deﬁne (cid:107) f (cid:107)
[Equation]
[Equation]
In the remainder of the paper, for any positive integers M, N ∈ N + , let F CNN be the class of CNNsdeﬁned in (2) with depth L , size S and ﬁlter length speciﬁed as follows:
[Equation]
where
[Equation]
denotes the maximum incremental width (number of neurons) for consecutive layers in the networkand (cid:100) a (cid:101) denotes the smallest integer no less than a .
Theorem 2.1.
Assume that f ∈ W β,p ( X ) with 1 ≤ β ∈ N 0 , 1 ≤ p ≤ ∞ and (cid:107) f (cid:107) W β,p ( X ) ≤ B 0 for
B 0 > 0 .
For any M, N ∈ N + , there exists a function f CNN ∈ F CNN deﬁned in (2) with depth L andﬁlter lengths and size S speciﬁed in (4), (5) and (6), such that for m = 0 , 1 ,
[Equation]
C
[Equation]
d, β, p )( NM ) − 2( β − m
[Equation]
/d , where C 0 ( d, β, p ) = 37 · 2 2 β +2 d/p B 20 ( β + 1) 3 × { π − d/ 2 Γ( d/ 2 + 1) } 2 /p
+
(1 + 2 √ d ) d d 4 β .
HereΓ( · ) is the gamma function.
Our result is new for approximation of Sobolev-regular functions in W β,p ( X ) for a positive integerβ ∈ N + with respect to the Sobolev norms (cid:107) · (cid:107) W m,p ( X ) for m = 0 , 1 in terms of explicitly deﬁnedapproximation error prefactor, clearly deﬁned and ﬂexible network parameters, as well as an fastapproximation rate comparable with [22].
A toy example is provided in Appendix D to illustrate theapproximation power of CNNs and examine how the approximation error varies according to theﬁlter size and depth of the networks.
In [24], approximation result on β -smooth functions by networks with squared ReLU activationfunction is also provided under the norm (cid:107) · (cid:107) W m,p ( X ) for positive integer m < β .
The paper[22]analyzed the approximation rates of ReLU DNN for Sobolev-regular functions with respect to theweaker Sobolev norms (cid:107) · (cid:107) W s,p ( X ) for s ∈ [0 , 1] , which can be seen as a generalization of the resultin [60].
Approximation results on Sobolev-regular functions W β,p ( X ) by networks with generalactivation functions are derived in [23], where the error is with respect to (cid:107) · (cid:107) W
m,p ( X ) for someinteger m < β .
By the general Sobolev inequality, see Theorem 6 in Chapter 5 of [15], W k,p ( X ) with kp > d can beembedded into a Hölder class
H β
( X , B 0 ) (with β = k − (cid:98) dp (cid:99) − 1 ) deﬁned as
[Equation]
X ,
[Equation]
where β > 0 and (cid:98)
β (cid:99) ∈ N 0 denotes the largest integer strictly smaller than β .
Because of the extraregularity on the Hölder class, a much improved prefactor C 0 in the error bound of Theorem 2.1 canbe obtained.
Theorem 2.2.
Let f ∈ H β
( X , B 0 ) be deﬁned in (8) and let X ∈ R
d be a random vector whose probability distribution is supported on X = (0 , 1) d and absolutely continuous with respect to the
Lebesgue measure.
For any M, N ∈ N + , there exists a function f CNN ∈ F CNN deﬁned in (2) with depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6), such that
[Equation]
where C 0 ( d, β ) = 18 B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 .
Here a ∨ b := max { a, b } for a, b ∈ R .
The convergence rate ( NM ) − 2 β/d in Theorems 2.1 and 2.2 with respect to the network depth andﬁlter lengths speciﬁed by M and N , is in line with the nearly optimal rate of ReLU FNNs on smoothfunctions under Sobolev norm in [24] and on Hölder smooth functions under L ∞ norm in [28].
Theprefactor 18 B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 of approximation error depends on the dimension d polynomially,different from the exponential dependence in many existing neural network approximation resultsmentioned in Section 1.1.
2.3 Approximation with a lower-dimensional support
In many modern machine learning problems, the ambient dimension d of the input data could bevery large, which results in an extremely slow convergence rate.
This fact is known as the curseof dimensionality .
Fortunately, many types of data have a low-dimensional latent structure, that is,although the ambient dimension d is large, the distribution of the data is approximately supported ona low-dimensional subset of R d , in which case the approximation error bound can be substantiallyimproved.
We establish an approximation result for CNNs in Sobolev spaces under a low-dimensionalsupport assumption.
Assumption 2.3.
The distribution of X is supported on M ρ , a ρ -neighborhood of M ⊂ X , where
M is a compact d M -dimensional Riemannian submanifold and M ρ = { x ∈ X : inf {(cid:107) x − y (cid:107) 2 :y ∈ M} ≤ ρ } for ρ ∈ (0 , 1) .
In real-world applications, data are hardly observed to locate on an exact manifold , instead they couldbe more realistically viewed as consisting of a latent part supported on a low-dimensional manifold
M plus noises.
Therefore, Assumption 2.3 is more realistic than the exact manifold assumption[51, 12, 38].
Deﬁne
[Equation]
In our error bound results below, it is the d ε that will affect the convergence rate.
Since it is often thecase that d M (cid:28) d and therefore d ε (cid:28) d, the manifold assumption will lead to a better convergencerate than those in Theorems 2.1 and 2.2.
In addition, we require ρ ≤ ρ ε .
Theorem 2.4.
Suppose that Assumption 2.3 holds, f ∈ H β
( X , B 0 ) and the distribution of Xis absolutely continuous with respect to the Lebesgue measure.
Let d ε and ρ ε be deﬁned in (9)and (10), respectively.
Assume that ρ ≤ ρ ε .
Then, for any M, N ∈ N + , there exists a CNNfunction f CNN ∈ F CNN with depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6) with
W = 38 2 ( (cid:98) β
(cid:99) + 1)
d 2 (cid:98) β (cid:99) +
ε
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 such that
[Equation]
where C ( d, β ) =
(18 + C 2 ) B 0 (1 − ε ) β ( β + 1) 2 d 1 / 2 d 3 β/ 2 ε .
Since the intrinsic dimension d ε can be much smaller than the ambient dimension d , the rate( NM ) − 2 β/d ε in Theorem 2.4 is greatly improved compared with the rate ( NM ) − 2 β/d in Theorem2.2 and the curse of dimensionality is mitigated.
The result here is of independent interest and can beuseful in other problems that involve the use of CNNs.
Excess risk in classiﬁcation
In this section, we present the application of the approximation results Theorems 2.1 and 2.4 to theerror analysis of classiﬁcation with CNNs.
Consider a binary classiﬁcation problem with a predictor X ∈ X ⊂ R d and its binary label
Y ∈ { 1 , − 1 } .
We are interested in learning a classiﬁer h :
X → { 1 , − 1 } from a class of functions,or a hypothesis space, denoted by H .
Let the joint distribution of ( X, Y ) be P .
The goal is to ﬁnd aclassiﬁer that minimizes the misclassiﬁcation error or the 0-1 risk: R ∗ ( h ) =
P (cid:8) h ( X ) (cid:54) =
Y (cid:9) , h ∈ H .
Denote the misclassiﬁcation risk minimizer at the population level by h 0 = argmin
h measurable R ∗ ( h ) .
For any h ∈ H , the excess risk of h is R ∗ ( h ) − R ∗ ( h 0 ) , the difference between the misclassiﬁcationerrors of h and h 0 .
Since the probability measure P is unknown in practice, the classiﬁer h will be learned based on arandom sample S =
{ ( X i , Y i ) } ni =1 from P , where n is the sample size.
The empirical risk minimizer(ERM) is deﬁned by
[Equation]
However, the empirical risk function based on the 0 - 1 loss is non-continuous and non-convex, thusthis minimization problem is typically computationally intractable.
Rather than minimizing the non-smooth 0-1 loss, many popular methods adopt a proper convexloss function to train classiﬁers with computational efﬁciency that can be done in polynomial time.
Moreover, proper surrogate convex loss functions have been shown to be consistent with the 0-1 lossfunction for binary classiﬁcation problems [62, 5].
3.1 Convex surrogate loss functions
Let φ be a given convex univariate loss function φ : R → [0 , ∞ ) .
We consider the risk function withrespect to the loss φ
[Equation] [Equation]
where for simplicity of notation and without causing confusion we omit the superscript φ in R ,similarly for ˆ f n and f 0 deﬁned below.
For a given random sample S = {
( X i , Y i ) } ni =1 , we denotethe ERM over F CNN with a given loss φ by
[Equation]
Based on the ERM ˆ f n , a classiﬁer ˆ h n ( x ) :
= sign( ˆ f n ( x )) for x ∈ X can be deﬁned.
As shown in[62, 5], for a properly chosen φ , ˆ f n can help reduce the excess risk R (ˆ h n ) − R 0 .
Speciﬁcally, deﬁnethe measurable minimizer of R in (12) as
[Equation]
and the corresponding minimal φ -risk as R 0 = R ( f 0 ) .
Then for a proper φ , we have ψ ( R (ˆ h n ) −
R 0 ) ≤ R ( ˆ f n ) − R 0 , where ψ : [ − 1 , 1] → [0 , ∞ ) is a nonnegative continuous function, invertible on[0 , 1] , and achieves its minimum at 0 with ψ (0)
= 0 .
A variety of classiﬁcation methods are basedon this tactic.
Generally, classiﬁcation-calibrated φ considered in this paper is non-increasing andconvex; details are given in the Supplementary Materials.
Deﬁne the conditional probability
[Equation]
Let the conditional φ -risk of f given X = x be denoted by R f ( x ) :
= E { φ ( Y f ( X ))
| X = x } .
SVM and cross entropy, two important examples of φ , and the corresponding f 0 , R ( f 0 ) , ψ and itsinverse on [0 , 1] in Table 2 in the supplementary materials.
For the form of ψ , Theorem 34 in [11]shows that if φ is convex, φ (cid:48)(cid:48) (0) > 0 exists and φ (cid:48) < 0 , then ψ ( u ) =
u 2 .
[Table] [Table]
[Table] [Table]
Non-asymptotic error bounds
For the ERM ˆ f n deﬁned in (13) we ﬁrst state a basic inequality for bounding the excess risk of ˆ f n .
Lemma 4.1.
For any loss φ and any random sample S = {
( X i , Y i ) } ni =1 , the excess φ -risk of the
ERM ˆ f n satisﬁes R ( ˆ f n ) − R ( f 0 ) ≤ StoErr + AppErr , where
[Equation]
Therefore, the excess risk of ˆ f n is bounded by the sum of two terms: the stochastic error and theapproximation error.
For a given loss function φ , the upper bound no longer depends on the ERMˆ f n , but the function class F and the random sample S .
The stochastic error in (16) depends on thecomplexity of F and the approximation error in (16) depends on the approximation power of theclass F for f 0 .
4.1 Stochastic error
We bound the stochastic error in terms of the pseudo-dimension of the downsampled CNNs deﬁned in(2); particularly, we further bound its pseudo-dimension and express the bound in terms of quantitiesrelated to CNNs.
Theorem 4.2.
[Stochastic error bound] Suppose that φ is convex and non-increasing.
For any
M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with B ≥ 0 , depth L and size S and letφ B := sup | a |≤B φ ( a ) .
Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ , the stochastic errorin (16) satisﬁes
[Equation]
where C 0 > 0 is a universal constant.
The quantity φ B for common loss functions are presented in Table 2.
It worth noting that the errorbound here does not require the norm of the CNN parameters (of weight and bias) to be uniformlybounded.
In comparison, those stochastic error bounds based on the covering number generallyassume an uniformly bounded norm on CNN parameters, which may hinder the approximation powerof network since most approximation results need the norm of network parameters to tend to inﬁnityas the approximation error tends to zero.
4.2 Approximation error
We derive an upper bound of the approximation error by relating inf f ∈F CNN R ( f ) − R ( f 0 ) toinf f ∈F CNN (cid:107) f − f 0 (cid:107) under proper conditions on φ and η , where R is deﬁned in (12).
The target function f 0 may be non-smooth or unbounded, which poses extra difﬁculty.
Most of the existing studies on the approximation properties of networks assume smooth and bounded target functions.
Asshown in Table 2, the target function f 0 for SVM is non-continuous; for the cross entropy loss, the lossis unbounded on [ − 0 .
For logistic loss and exponential loss, we have f 0 ( x ) ∈ { + ∞ , −∞}once η ( x ) ∈ { 0 , 1 } .
We overcome these difﬁculties by imposing mild conditions on the conditionalprobability η and the loss function φ , and use the truncation technique to analyze the approximation.
Assumption 4.3.
(a)
The conditional probability η ( x ) =
P { Y = 1 | X = x } is continuous on thesupport of X and the probability measure of X is absolutely continuous with respect to the Lebesguemeasure.
(b) The loss function φ : R → [0 , ∞ ) is classiﬁcation-calibrated, convex, non-increasingand continuously differentiable on its support.
Assumption 4.3 (a) and (b) are regular conditions ensuring that the target function f 0 is continuous and the loss φ is Lipschitz on bounded intervals.
The absolute continuity assumption of theprobability measure of X with respect to the Lebesgue measure is reasonable for the approximatelow-dimensional manifold case but incompatible with the exact low-dimensional manifold condition.
This assumption is for deriving better L p approximation error bound, without which the error boundin term of the L ∞ norm can still be established, but at the price of a wider neural network and a largerprefactor.
It can be veriﬁed that commonly used loss functions, such as the cross entropy, the logisticand the exponential loss functions, satisfy Assumption 4.3 (b).
To deal with the approximation of unbounded target function, we truncate the target f 0 by a constant
T , where T may depend on n .
Let f 0 ,T be the truncated version of f 0 deﬁned asf 0 ,T ( x ) =
(cid:26) f 0 ( x ) , if | f 0 ( x ) | ≤ T,
T sign( f 0 ( x )) ,
[Equation]
Denote the error of the loss function φ due to truncation by
[Equation]
where Ran ( f 0 ) is the range of the target function f 0 .
This error decreases as T increases.
Then, theapproximation error can be decomposed into two terms that are easier to deal with.
Lemma 4.4.
[ φ -approximation error] Suppose that Assumption 4.3 holds and T ≤ B .
Then, theφ -approximation error inf f ∈F CNN R ( f ) − R ( f 0 ) with respect to the loss function φ satisﬁes
[Equation]
where B φ is deﬁned as the Lipschitz constant of φ on the interval [ −B , B ] .
We list B φ and ∆ φ ( T ) for hinge and cross entropy loss functions in Table 2.
A detailed tableincluding other loss functions are given in the supplementary materials.
Theorem 4.5.
[Approximation error bound] Suppose that Assumption 4.3 holds and f 0 ∈
H β ([0 , 1] d , B 0 ) .
For any M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with T ≤ B ,depth L and ﬁlter lengths speciﬁed in (4) and (5).
Then, the approximation error deﬁned in (16)satisﬁes AppErr ≤ C ( d, β )( NM ) − 2 β/d + ∆ φ ( T ) , where C ( d, β ) =
18 B φ B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 .
There is a trade-off since T ≤ B and ∆ φ ( T ) deceases in T but C ( d, β ) increases in B φ (or B ).
4.3 Non-asymptotic excess risk bound
4.2 and 4.5, we obtain the excess error bound.
Theorem 4.6.
[Non-asymptotic excess φ -risk bound] Suppose that Assumption 4.3 holds and f 0 ∈
H β ([0 , 1] d , B 0 ) .
For any M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with T ≤ B ,depth L and ﬁlter lengths speciﬁed in (4) and (5).
Then, for any δ ∈ (0 , 1) , with probability at least1 − δ , the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
18) with
[Equation]
where C ( d, β ) =
18 B φ B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 and C 0 > 0 is a universal constant.
The upper bound of the excess φ -risk R ( ˆ f n ) − R ( f 0 ) in Theorem 4.6 is a sum of two error terms, thestochastic error and the approximation error.
To achieve the optimal error rate, we need to balancethe trade-off between them.
On one hand, the bound for the stochastic error StoError increases withthe size and the depth of F CNN .
On the other hand, the bound for the approximation error AppErrordecreases with the depth and the upper bound of ﬁlter lengths of F CNN .
4.4 Circumventing the curse of dimensionality
We state a theorem that provides a non-asymptotic excess risk bound under the approximate lowdimensional manifold assumption.
Theorem 4.7 (Circumventing the curse of dimensionality) .
Suppose that Assumptions 2.3, 4.3hold, and f 0 ∈ H β
([0 , 1] d , B 0 ) .
For any M, N ∈ N + , let F CNN be the class of CNNs deﬁnedin (2) with T ≤ B , depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6) with W =38 2 ( β + 1) 4 d 2 β +2 ε
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 .
Suppose ρ in Assumption 2.3 satisﬁes ρ ≤ ρ ε , , where ρ ε isdeﬁned in (10).
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ , the ERM ˆ f n deﬁned in (13)satisﬁes
[Equation]
(19) with
StoErr
[Equation]
where C 1 , C 2 > 0 are universal constants.
For a high-dimensional input X with a large d , d ε satisﬁes d M ≤ d ε < d for ε ∈ (0 , 1) .
For thestochastic error StoErr ∗ , comparing with the bound (18) in Theorem 4.6, we see that StoErr ∗ =
StoErr , that is, the stochastic error does not change under the approximate low-dimensional manifoldassumption.
For the approximation error AppErr ∗ , we see that the convergence rate ( NM ) − 2 β/d εonly depends on d ε , which leads to a much faster convergence rate.
Examples
In this section, we illustrate the applications of Theorems 4.6 and 4.7 to obtaining non-asymptoticerror bounds for the excess risk in classiﬁcation.
We apply the general excess risk bounds establishedin these theorems to several important classiﬁcation methods with CNNs when a speciﬁc form of φ isgiven.
We present the results for the cross entropy loss below.
The non-asymptotic error bounds forthe hinge, the logistic, the exponential and the least squares losses are given in Appendix C in thesupplementary material.
For the cross entropy loss φ ( a ) =
− log(0 .
5 + a ) , the Lipschitz constant B φ on [ − 0 . 5 , 0 . 5] is notbounded.
As the minimizer of φ -risk η − 0 .
5 is bounded, we can choose T = B
= 0 .
5 − n β/ (2 d +4 β ) .
Then ∆ φ ( T ) ≤ n − β/ (2 d +4 β ) and B φ = n β/ (2 d +4 β ) .
Denote ξ n = n d/ (2 d +4 β ) .
Take N = 1 in (7) and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the classof CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 (cid:98) ξ n (cid:99)(cid:100) log 2 (8 (cid:98) ξ n (cid:99) ) (cid:101) /
( s min − 1) , ﬁlterlengths 2 ≤ s min ≤ s max ≤ 9 × 38
( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗38 4 (
(cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99)
+4 (cid:98) ξ n (cid:99)(cid:100) log 2 (8 (cid:98) ξ n (cid:99) ) (cid:101) /
( s min − 1) .
Theorem 4.6 implies that, with probabilityat least 1 − exp( − ξ 2 n ) , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C =
O
( B 0 ( β + 1) 8 d 3 β +3 / ( s min − 1)) is a constant independent of n .
Let T =
B
= 0 .
5 − (log n ) − 1 .
Using a modiﬁed cross entropy loss φ ( a ) = max {− log(0 .
5 +a ) , τ } with τ = − log(1 − (log n ) − 1 ) , the excess φ -risk of the ERM ˆ f n can be improved to
O ( n − β/ ( d +2 β ) (log n ) 2 ) .
If the approximate low-dimensional manifold Assumption 2.3 holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in Assumption 2.3 satisﬁes ρ ≤ ρ ε , where ρ ε is deﬁned in (10), then Theorem 4.7 implies that the rate of convergence can be improved:
[Equation]
where C is a constant independent of n .
The above discussion shows that deep binary classiﬁcations with CNNs are adaptive to the lowdimensional structure of the data and the smoothness of the target.
Moreover, the prefactor Cdepends on the ambient dimension d polynomially, which improves the prefactors depending on dexponentially in the existing excess risk bounds [46, 25, 30, 26, 45, 32, 33, 38].
Moreover, the ﬁlterlengths here are more ﬂexible comparing with those requiring certain ﬁlter lengths in [32, 33, 49].
Conclusion
In this work, we derive new approximation error bounds with explicit prefactor in terms of networkparameters for Sobolev-regular functions and Hölder smooth functions using deep convolutionalnetworks.
New approximation result when the target function is supported on an approximatelower-dimensional manifold is established.
Different from existing results, the prefactor of our errorbound depends on the ambient dimension polynomially instead of exponentially for Hölder smoothfunctions.
The new approximation results are applied to establish non-asymptotic excess risk boundsfor a class of classiﬁcation methods using CNNs.
An important limitation of our work is that we only considered binary classiﬁcation problems inapplying our approximation results using deep convolutional networks.
It would be interesting toapply the results in this work to other settings involving CNN approximation, such as the multiclassclassiﬁcation and regression problems.
In addition, our work only partially explains the empiricalsuccesses of CNNs in practice.
For image data, the assumption of approximate lower-dimensionalsupport does not capture all the structural information.
For example, spatial invariance is likely to beexpected in some problems such as image classiﬁcation.
We believe that if such properties are takeninto account, the theoretical bounds can be further improved.
We hope to study this in the future.
Appendix
In the appendix, we include detailed descriptions on convex surrogate losses,convolutional neuralnetworks, non-asymptotic error bounds for commonly used loss functions, and prove Theorems2.1,2.2, 2.4, 4.4, 4.6, and 4.7, as well as the error bounds for cross entropy, SVM, logistic, exponentialand least squares loss examples.
A toy example on the numerical performance of CNN approximationis presented in Appendix D.
A Convex surrogate loss
We next give a brief review of the convex surrogate loss functions and discuss in details on theconnection between the excess risk with respect to the φ -loss and that of 0-1 loss [62, 5].
Let φ be a given convex univariate function φ : R → [0 , ∞ ) .
Instead of minimizing the excess risk Rover H , we consider minimizing the risk with respect to the loss φ ( φ -risk)
[Equation]
over a certain class of functions F , where φ : R → [0 , ∞ ) is some generic loss function.
For thespecial case when H = { h : h ( x ) = sign( f ( x )) , f ∈ F} and φ ( · ) is a step function, i.e., φ ( x ) =
1if x < 0 and φ ( x ) =
if x ≥ 0 , then to minimize R over H , it sufﬁces to minimize R over F .
Given
S = { ( X i , Y i ) } ni =1 , let R n ( f ) :
= (cid:80) ni
=1 φ ( Y i f
( X i ))
/n be the empirical risk of f w.r.t the φ -loss,and deﬁne the empirical risk estimator (ERM) as
[Equation]
Based on the ERM ˆ f n , a classiﬁcation rule or a classiﬁer ˆ h n ( x ) :
= sign( ˆ f n ( x )) for x ∈ X can be induced which aims at minimizing the 0-1 risk.
As shown in [62] and [5], for a properly chosen φ , ˆ f n can indeed help reduce the excess risk R (ˆ h n ) − R 0 .
More precisely, let R 0 :=
inf f measurable R ( f ) , then for a proper φ , we have
[Equation]
where ψ : [ − 1 , 1] → [0 , ∞ ) is a nonnegative continuous function, invertible on [0 , 1] , and achievesits minimum at 0 with ψ (0)
= 0 .
A wide variety of popular classiﬁcation methods are based onthis tactic.
For instance, when φ ( a ) = log { 1 + exp( − a ) } , it is the logistic regression [20]; whenφ ( a ) = max { 1 − a, 0 } , it becomes the SVM [14];
when φ ( a ) = exp( − a ) is the exponential loss, it isthe AdaBoost algorithm
[19]; when φ ( a ) =
(1 − a ) 2 , it is the least squares method for classiﬁcation,and so on.
Deﬁne the measurable minimizer of R as
[Equation]
and the corresponding minimal φ -risk as R 0 = R ( f 0 ) .
Clearly, the optimal f 0 depends on theloss function φ .
If φ is not properly chosen, the resulting f 0 can be poor and thus the classiﬁerh 0 ( x ) = sign( f 0 ( x ))
based on f 0 can be invalid.
To study the basic conditions imposed on φ , so asto produce a valid classiﬁer towards minimizing the 0-1 loss, we need a thorough understanding ofthe risk R .
Let the conditional φ -risk of f given X = x be denoted by R x ( f ) :
= E { φ ( Y f ( X ))
| X = x } .
Recall that η ( x ) =
P { Y = 1 | X = x } as deﬁned in (15).
We have
R
[Equation]
f
[Equation]
x
[Equation]
x
[Equation]
f
[Equation]
x )) , x ∈ X .
For a good classiﬁer, if η ( x ) > 1 / 2 , it is naturally expected that f 0 ( x ) > 0 and h 0 ( x ) =sign( f 0 ( x ))
=
1 (correct sign); thus to encourage f 0 ( x ) > 0 , we should at least requireφ ( f 0 ( x )) <
φ ( − f 0 ( x )) to minimize the conditional risk R x .
Similarly, if η ( x ) < 1 / 2 , we expect that the contrary happens.
A rigorous deﬁnition for such an ideal φ is given in Deﬁnition 1 of[5].
Deﬁnition A.1 (Classiﬁcation-calibrated) .
For η ∈ [0 , 1] and a ∈ R , deﬁne H ( η, a )
= ηφ ( a ) + (1 − η ) φ ( − a ) .
Then, we say that φ is classiﬁcation-calibrated if for any η (cid:54) = 1 / 2 , inf a ∈ R H ( η, a ) < inf a ∈ R : a (2 η − 1) ≤ 0 H
( η, a ) .
With a classiﬁcation-calibrated loss φ , it is guaranteed that the “incorrect" label sign( f ( X )) in thesense that it is inconsistent with the Bayes estimator sign(2 η ( X ) − 1) ) results in a strictly larger lossunder φ .
It was shown by [5] that, the surrogate loss function φ is able to produce the optimal Bayesclassiﬁer.
For ease of reference, we state this result as Lemma A.2 below.
Lemma A.2 (Theorem 1 of [5]) .
For any nonnegative loss function φ : R → [0 , ∞ ) , any measurable f : X → R , its induced classiﬁer
h f = sign( f ) :
X → {± 1 } , and any probability measure P of ( X, Y ) on X × {± 1 } ,
[Equation]
where ψ : [ − 1 , 1] → [0 , ∞ ) is the Fenchel-Legendre biconjugate of ˜ ψ : [ − 1 , 1] → R , and
[Equation]
Besides, if φ is classiﬁcation-calibrated, then for any sequence { a m } in [0 , 1] , ψ ( a m ) → 0 , if andonly if a m → 0 .
Lemma A.2 has some important implications.
First, for any nonnegative φ , ψ is simply the functionalconvex hull (the greatest convex minorant) of ˜ ψ .
Both ψ and ˜ ψ are continuous on [ − 1 , 1] , and ψ isnonnegative that attains its minimum at 0 with ψ (0)
= 0 .
Second, if the loss φ is nonnegative andclassiﬁcation-calibrated, then ψ ( θ ) > 0 for all θ ∈ (0 , 1] and ψ is invertible on [0 , 1] .
In this case, wehave R ( h f ) − R 0 ≤ ψ − 1 ( R ( f ) − R 0 ) .
Third, if φ is nonnegative convex and classiﬁcation-calibrated,then ψ ( θ ) = φ (0) − inf a ∈ R H ((1 + θ ) / 2 , a ) , which gives an easy way to compute the function ψ .
Note that for the formulation of ψ , Theorem 34 in [11] shows that if φ is convex, φ (cid:48)(cid:48) (0) > 0 existsand φ (cid:48) < 0 , then ψ ( θ )
=
θ 2 .
Next, we present several examples of φ , and the corresponding f 0 , R ( f 0 ) , ψ and its inverse on [0 , 1]in Table 3.
[Table]
Note
[Table]
) is written as η for notational simplicity.
We also list the Lipschitz constant B φ and ∆ φ ( T ) for commonly-used SVM, cross entropy, leastsquares, exponential and logistic loss functions in Table 4.
[Table]
B Convolutional neural networks
As indicated by their name
[Table]
CNNs employ a mathematical operation called convolution.
Convolutionalnetworks are a specialized type of structured sparse feedforward neural network (FNN) that use convolution in place of general matrix multiplication in at least one of their layers [21].
There aredifferent formulations of CNNs in the literature [2, 67, 47, 37, 64, 65].
For a general neural network, let L denote the number of layers and ( σ 1 , . . .
, σ L ) denote the activationfunctions, such as the rectiﬁed linear unit (ReLU) and max pooling function.
Besides, let d i denotethe width (the number of neurons or computational units) of the i -th layer, i = 1 , . . .
, L + 1 , and
W = max { d 1 , . . .
, d L } denote the maximum width among layers.
In our present classiﬁcationproblem, d 0 = d (the dimension of the input X ) and d L +1 = 1 (the dimension of the output Y ).
In the following, two common types of neural networks including FNNs or MLPs and CNNs withdownsampling [64] are shown.
• The architecture of a MLP f MLP : R d → R can be expressed as a composition of a series
of linear transformations:
for any x ∈ R d ,
[Equation]
where W i ∈ R d i +1 ×
d i is a fully connected weight matrix, and b i ∈ R
d i +1 is a bias vector in
the i -th linear transformation.
Usually, the activation functions ( σ 1 , . . .
, σ L ) are the ReLU
activation, where ReLU ( x ) = max { 0 , x } for any x ∈ R (deﬁned for each component of x
if x is a vector).
• The formulation of a downsampled CNN f CNN : R d → R is essentially a specially structured
architectures of MLP, and here we use the formulation deﬁned in [67, 64, 65]
[Equation]
23) where ◦ denotes the functional composition and A i is a linear operation, i = 1 , . . .
, L + 1 .
The A i ’s are either convolutional operators or downsampling operators.
For convolutional layers,
[Equation]
where σ is the rectiﬁed linear unit (ReLU) activation function applying to each componentof the input vector, W ci ∈ R d i × d i − 1 is the structured sparse Toeplitz type weight matrixinduced by the convolutional ﬁlter { w ( i ) j } s ( i )
j =0 with ﬁlter length s ( i ) ∈
N involving onlys ( i ) +1 free parameters.
The corresponding ( d i −
+ s ( i ) )
× d i − 1 Toeplitz type weight matrix
W ci is given explicitly by
[Equation]
  
[Equation] [Equation]
w
[Equation]
i
[Equation]
w ( i )  · · · · · · ... w ( i ) s ( i ) w ( i ) s ( i ) −· · · w ( i ) · · ·  w ( i ) s ( i ) ·
· · w ( i )w ( i )0 · · · .. .
· · · w ( i ) s ( i ) · · · w ( i )w ( i ) · · ·  w ( i ) s ( i ) · · · w ( i )...
[Equation]
[Equation]

  
[Equation]
It can be seen that by convolutional operations here, the network has linearly increasingwidths d i =
d i −
+ s ( i ) for convolutional layers.
On the contrary, the downsampling layersdecrease the width of the networks.
For downsampling layers,
[Equation]
where D i : R d i − 1 → R [ d i − 1 /m i ] is the downsampling operator with a scaling parameter m i ≤
d i − 1 and W D ∈ R
d
i − 1 ×
[ d i − 1 /m i ] is corresponding structured sparse weight matrix induced by D i in the i -th layer.
Here the downsampling operator [64] is given by
[Equation]
which takes a real vector x ∈ R d i − 1 as input and outputs a subvector of x with length
R [ d i − 1 /m i ] where [ a ] denotes the integer part of a ∈ R .
It can be seen that by downsamplingoperation D i here, the network width is scaled down by about m i times.
For the functionclass of the downsampled CNNs formulated as in (23), we let L be th number of hiddenlayers and S be the total number of parameters for networks in F CNN and let W be themaximum ﬁlter length of convolutional layers.
A downsampled CNN f CNN is called uniform if the length of its ﬁlters { s ( i ) } in theconvolutional layers are the same.
[64] showed that any multilayer perceptron can berepresented by a uniform downsampled CNN with parameters no more than 8 times as thatof the MLP.
C Non-asymptotic error bounds for commonly used loss functions
In this section, we show more examples of the non-asymptotic error bounds for commonly used lossfunctions including the hinge loss, the logistic loss, the exponential loss and the least square loss.
C.1 SVM: the hinge loss
For SVM, the loss function φ ( a ) =
max { 1 − a, 0 } is not differentiable at a = 1 and thus theφ -risk minimizer f 0 ( x )
=
sign(2 η ( x ) − 1) may not be continuous even though η is continuous, asf 0 ( x ) is discontinuous when η ( x ) = 1 / 2 .
To tackle this problem, we additionally impose the lownoise condition on η [41, 56], i.e., there exist c noise > 0 and q ∈ [0 , ∞ ] such that for any t > 0 , 630 P ( | 2 η ( X ) − 1 | ≤ t ) ≤ c noise t q , where the constant q is called the noise exponent .
Suppose that Assumptions 4.3 holds f 0 ∈ H β ([0 , 1] d , B 0 ) .
Denote ζ n = n d/ { 2 d +4 β ( q +1) } .
Let F CNN be the class of CNNs deﬁned in (2) with B ≥ 1 , depth L ≤ 378 · 38 2 ( (cid:98)
β (cid:99) +1) 6 d 2 (cid:98) β (cid:99)
+2 (cid:98) ζ n (cid:99)(cid:100) log(8 ζ n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) +1) 4 d 2 (cid:98) β (cid:99)
+2and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38 4 ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) ζ n (cid:99)(cid:100) log(8 ζ n ) (cid:101) /
( s min − 1) .
Theorem4.6 implies that, with probability at least 1 − exp( − ζ 2 n ) , the excess φ -risk of the ERM ˆ f n deﬁned in(13) satisﬁes
[Equation]
where C =
O ( c noise 4 q B q
+ ( β + 1) 2 q +8 d ( q +1)(3 β +3) ) .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
The excess risk bound depends on q , the noise exponent.
When q = 0 (high noise), the convergencerate is n − β/
( d +2 β ) ; when q = + ∞ (no noise), the rate will be signiﬁcantly improved to n − 1 / 2 .
Asimilar result can be found in Theorem 3.3 of [30].
Suppose Assumption 2.3 holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in thisassumption satisﬁes ρ ≤ ρ ε , then Theorem 4.7 implies that
[Equation]
where C is a constant independent of n .
C.2 The logistic loss
For the logistic loss function φ ( a ) = log { 1+exp( − a ) } , the minimizer of φ -risk log { η ( x ) / 1 − η ( x ) }is unbounded with the Lipschitz constant B φ = 1 / { exp( −B ) + 1 } ≤ 1 on [ −B , B ] , and φ B =log(1 + exp( B )) ≤ B + ∞ and the truncation error ∆ φ ( T )
=
log { 1 + exp( − T ) } .
A feasible choiceis T = B = log( n ) where n is the sample size, in which case ∆ φ ( T ) = log { 1 + 1 /n } ≤ 1 /n .
Suppose Assumptions 4.3 (in main context) holds and f 0 ∈ H β ([0 , 1] d , B 0 ) .
Take N = 1 in (7)and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β
(cid:99) +1)
6 d 2 (cid:98) β (cid:99) +2 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99)
+
[Table] [Table]
[Table]
Theorem 4.6 implies that, with probability at least 1 − exp {− n d/
( d +2 β ) } , the excess φ -risk of the
ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
, or simply
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) +1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
In addition, suppose the approximate low-dimensional manifold Assumption 2.3 also holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in Assumption 2.3 satisﬁes ρ ≤
C 2 ( NM )
[Equation] [Equation]
β (cid:99) +1) 2 d 1 / 2 d 3 (cid:98) β (cid:99) ε (
[Equation]
d/d ε +1 − ε ) − 1 (1 − ε ) 1 − β , where d ε = O ( d M log( d/ε ) /ε 2 ) is an integer satisfying d M ≤ d ε (cid:28) d.
Then Theorem 4.7 shows that the rate of convergence can be improved to
[Equation]
where C ( d, d ε , ε, β, B 0 , s min )
=
O ( B 0 (1 − ε ) − β
( (cid:98) β (cid:99) + 1) 8 d 1 / 2 d 3 (cid:98) β (cid:99) +3 ε / ( s min − 1)) is a constantindependent of n .
C. The exponential loss
For the exponential loss function φ ( a ) = exp( − a ) , the minimizer of φ -risk log { η ( x ) / 1 − η ( x ) } / 2is unbounded, the truncation error ∆ φ ( T ) =
exp( − T ) , the Lipschitz constant B φ = exp( B ) andφ B = exp( B ) .
A feasible choice is T = B
=
( β/ { 2 d + 4 β } ) log( n ) where n is the sample size.
Suppose Assumptions 4.3 (in main context) holds and f 0 ∈ H β ([0 , 1] d , B 0 ) .
Take N = 1 in (7)and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β
(cid:99) +1)
6 d 2 (cid:98) β (cid:99) +2 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99)
+
[Equation] [Equation]
+2 and size S ≤ 8 W L ≤ 42 ∗
[Equation]
9 2 ∗ 38 4 (
[Equation]
β (cid:99)
[Equation] [Equation]
[Equation]
+4 (cid:98) n
[Equation]
/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min −1) .
Theorem 4.6 implies that, with probability at least 1 − exp {− n d/
d +2 β ) } , the excess φ -risk ofthe
ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
or simply
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) + 1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
In addition, suppose the approximate low-dimensional manifold Assumption 2.3 also holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in Assumption 2.3 satisﬁes ρ ≤ C 2 ( NM ) − 2 β/d ε ( (cid:98) β
(cid:99) + 1) 2 d 1 / 2 d 2 (cid:98) β
(cid:99)
ε ( (cid:112) d/d ε +1 − ε ) − 1 (1 − ε ) 1 − β , where d ε =
O ( d M log( d/ε ) /ε 2 ) is an integer satisfying d M ≤ d ε (cid:28) d.
Then Theorem 4.7 shows that the rate of convergence can be improved to
[Equation]
where C ( d, d ε , ε, β, B 0 , s min )
=
O ( B 0 (1 − ε ) − β
( (cid:98) β (cid:99) + 1) 8 d 1 / 2 d 3 (cid:98) β (cid:99) +3 ε / ( s min − 1)) is a constantindependent of n .
To improve the rate of convergence, one can use a modiﬁed exponential loss function, i.e. φ ( a ) =max { exp( − a ) , τ } for some small τ > 0 .
The minimum of modiﬁed φ can be achieved at a ∗ ( τ )
=− log( τ ) , and the minimizer f 0 ( x ) will be a truncated version of log( η/ (1 − η )) / 2 (truncated bya ∗ ( τ ) =
− log( τ ) ).
In light of this, under the modiﬁed exponential loss with τ = exp( − T ) , the corresponding measurable minimizer deﬁned in (14) is f 0 ,T , the truncated version of the minimizerf 0 under the original exponential loss; see Table 4.
Then ∆ φ ( T ) = 0 by its deﬁnition since theinﬁmum of the modiﬁed loss can be achieved within [ − T, T ] .
By choosing T = B = log log n andτ = exp( − T ) , with the modiﬁed exponential loss, Theorem 4.6 implies that with probability at least1 − exp {− n d/
d +2 β ) } , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) +1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
C.
The least squares loss
For the least squares loss φ ( a ) = (1 − a ) 2 , we ﬁrst prove that the approximation error is of a specialform.
For any f ∈ F CNN
[Equation]
Ans we have
[Equation]
Besides, note that for least squares f 0 = 2 η − 1 and (cid:107) f 0 (cid:107) ∞ = 1 , thus taking the truncation T = 1 issufﬁcient for the error control analysis.
Let F CNN be a class of CNNs deﬁned in (2) with B ≥ T = 1 .
By Theorem 2.2, there exists a function f implemented by CNN with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 M (cid:100) log(8 M ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) + and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38 4 ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 N (cid:100) log(8 N ) (cid:101) /
( s min − 1) such that
[Equation]
and based on Lemma 4.1 and Theorem 4.2, the empirical φ -risk minimizer ˆ f n deﬁned in (13) satisﬁes,for any δ > 0 , with probability at least 1 − δ ,
[Equation]
Let F CNN be the class of CNNs deﬁned in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β
(cid:99) +1)
6 d 2 (cid:98) β (cid:99) +2 (cid:98) n d/ (2 d +8 β ) (cid:99)(cid:100) log(8 n d/ (2 d +8 β ) )
(cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤9 × 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W
[Equation]
≤ 42 ∗
[Equation]
∗ 9 2 ∗ 38
( (cid:98) β (cid:99) +1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) n d/ (2 d +8 β ) (cid:99)(cid:100) log(8 n d/ (2 d +8 β ) ) (cid:101) /
( s min − 1) , B ≥ B 0 and δ = exp {− n d/ ( d +4 β ) } .
Byplunging in φ B =
( B +1) 2 and above values, we have with probability at least 1 − exp {− n d/ ( d +4 β ) } ,
R ( ˆ f n ) − R ( f 0 ) ≤ (log n ) 2 n − β/ (3 d/ 8+2 β ) × (cid:110) c
( β + 1)
[Equation]
B + 1) 2 d 3 (cid:98) β (cid:99) +3 / ( s min − 1) + 18 2 B 2 ( (cid:98) β
(cid:99) + 1)
d 2 (cid:98) β (cid:99)
+( β ∨ 1) (cid:111) , or simply
[Equation]
where C ( d, β, B , s min )
=
O ( B 2 ( (cid:98) β (cid:99) + 1) 8 d 3 (cid:98) β (cid:99) +3 /
( s min − 1)) is a constant independent of n .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
The excess risk bound under the approximate low-dimensional manifold assumption can be obtainedin a similar way.
C.5 Excess misclassiﬁcation errors
In this subsection, we summarize the misclassiﬁcation error bounds obtained in existing results andcompare them with ours.
When the hypothesis space is taken to be the class of measurable functions itself, for the excessmisclassiﬁcation error with the 0-1 loss, the convergence rates of order O ( n − 1 / 2 ) can be attainedusing oracle inequalities [41, 59].
Under Tsybakov noise condition of exponent θ and α -Höldersmooth condition on the decision boundaries, [56] proved that the minimax lower bound of the excessmisclassiﬁcation rate is O ( n − α
( θ +1) /
{ α ( θ +2)+( d − 1) θ } )
over all measurable classiﬁers.
With largeenough α and θ , the convergence rate can be close to O ( n − 1 ) arbitrarily.
For classiﬁcations using deep neural networks with the SVM hinge loss function, [30] showedthat the minimax optimal convergence rate is O ( n − α
( θ +1) /
{ α ( θ +2)+( d − 1) θ } ) under the Tsybakovnoise condition of exponent θ and α -Hölder smooth condition on the decision boundaries and
O ( n − α
( θ +1) /
{ α ( θ +2)+ d } ) under the Tsybakov noise condition of exponent θ and α -Hölder smoothcondition on the conditional class probability η ( x ) =
P ( Y = 1 | X = x ) .
For classiﬁcations using deep convolutional networks with the p -norm hinge loss function φ ( u ) =max { 1 − u, 0 } p with p ≥ 1 (it is hinge loss when p = 1 ), [18] derived the approximation error rateand the excess misclassiﬁcation rate for a target function in the Sobolev space W β,p ( S d − 1 ) with inputdata supported on the sphere S d − 1 in R d under the variancing power condition [57].
Two quantitiesincluding γ = max { 1 , ( d + 3 + β ) / 2( d − 1) } and τ for variancing power condition are involvedin the convergence rate.
Details can be found in [18].
Based on Table II in [18]
[Table]
we list the excessmisclassiﬁcation rates and compare them in the table below
[Table] [Table]
This
[Table]
D A toy example for CNN approximation
In this subsection, we use a toy example to illustrate the approximation power of CNNs.
We examinehow the approximation error varies according to the ﬁlter size and depth of the CNNs.
We consider a target function f 0 deﬁned as
[Equation]
where x = ( x 1 , x 2 ) ∈ [0 , 1] 2 is the two-dimensional input.
The 2D heatmap and 3D surfacevisualizations for the target function f 0 are presented in Figure 1.
To construct CNNs that approximates f 0 , we train the CNNs on noiseless data, which is generatedby taking values of the target function in [0 , 1] 2 .
Speciﬁcally, we choose 100 × 100 lattice points{ X i }
10 , 000 i = uniformly on the square [0 , 1] 2 and calculate the corresponding target function values{ Y i =
f 0 ( X i ) } 10 , 000 i = to get the training data { X i , Y i } 10 , 000 i = for function approximation.
We implement the training in Python via Pytorch and use Adam [31] as the optimization algorithmwith default learning rate 0.01 and default β = (0 . 9 , 0 . 99) (coefﬁcients used for computing runningaverages of gradients and their squares).
During the training process, we set the batch size as 2 , 500 ,i.e. the 1 / 4 of the sample size.
The maximum epoch of training is set as 1,000 and an early stoppingrule is applied where we stop the iteration if the training losses for the last 200 consecutive epochs donot achieve a new minimum.
We investigate the approximation power of CNNs with different shapes, which are speciﬁed throughthe ﬁlter length and the depth.
For ease of demonstration, we consider CNNs with L hidden layerswith each hidden layer being a convolutional layer with a uniform ﬁlter length s. We set the last layeras a fully connected layer with one-dimensional scalar output.
We train 12 CNNs that approximate the target function for L = 1 , 2 , 3 and s = 20 , 50 , 100 , 200 , thenwe calculate the empirical L 1 and L 2 distances measuring the approximation error according to
[Equation] [Equation]
where ˆ f denotes the approximate function by CNN.
Due to the randomness of the optimizationalgorithms, we train each CNN for several times and take the best one as the ﬁnal approximator onthe target function.
The summary statistics of empirical L 1 and L 2 distances between the CNNs to the target functionon the training data are presented in Table 6.
A visualization of CNN approximations on the targetfunction by heatmaps are presented in Figure 2.
[Table]
E Proofs
In this section, we prove Theorems 2.1,2.2, 2.4, 4.4, 4.6, 4.7, as well as the non-asymptotic errorbound results for the cross entropy, the SVM, the logistic, the exponential and the least squares losses.
E.1 Proof of Theorem 2.1
As in [22], this approximation result in term of (cid:107) · (cid:107) W β − 1 ,p norm is proved for target function f ∈ W β,p ( X ) , which can be seen as an generalization of Theorem 1 in [60] where L ∞ error bounds for f ∈ W β, ∞ ( X ) is derived.
Besides, our results improves the prefactor in d of the network width in Theorem 4.1 in [22].
The main idea of our proof is to approximate the averaged Taylor expansion of function f in Sobolev functions.
The deﬁnition and properties of averaged Taylor expansion can be found in subsection E.13 of this supplementary material.
By Lemma E.6, let B be a ball with proper radius in Ω such that Ω is star-shaped with respect to B , for 1 ≤ p ≤ ∞ we have
[Equation]
where C k,β,γ > 0 is a constant, r Ω is the diameter of Ω ⊆ X , γ is the chunkiness parameter of Ωdeﬁned in deﬁnition E.4 and T β f is the averaged Taylor expansion over B deﬁned as in (33):
[Equation]
where
[Equation]
and φ is an arbitrary cut-off function supported in ¯ B being inﬁnitely differentiable.
This reminder term could be well controlled when the approximation to Taylor expansion in implemented in a fairly small local region.
Then we can focus on the approximation of the Taylorexpansion locally.
The proof is divided into several parts:
• Partition X into small cubes ∪ θ Q θ , and construct a network ψ that approximately maps each x ∈ Q θ to a ﬁxed point x θ ∈ Q θ .
Hence, ψ approximately discretize X .
• For any multi-index α , construct a network φ α that approximates the coefﬁcient of theaveraged Taylor expansion over a ball B in Q θ , that is x ∈ Q θ
(cid:55)→ c α ( x θ ) , where Q θ =(cid:80) α ≤ β c α ( x θ ) x α by Lemma E.2.
Once X is discretized, the approximation is reduced to adata ﬁtting problem.
• Construct a network P α ( x ) to approximate the polynomial x α := x α 1 1 . . .
x α d d where x =( x 1 , . . .
, x d ) (cid:62) ∈ R d and α =
( α 1 , . .
. , α d ) (cid:62) ∈ N d 0 .
In particular, we can construct a
network φ × ( · , · ) approximating the product function of two scalar inputs.
• Lastly, our construction of neural network can be written in the
[Equation]
Proof.
Before proving the theorem, we ﬁrstly show that each ReLU activated multilayer-perceptroncan be computed by a downsampled CNN f CNN deﬁned as in (23).
The proof here follows Theorem
in [64] where we construct downsampled CNN exactly computing a fully-connected layer.
Due tothe simplicity of the formulation of ReLU activated multilayer-perceptron, we focus on showing thateach fully-connected layer can be reproduced by a down sampled CNN with proper choice of ﬁlters.
Without loss of generality, let
[Equation]

[Equation]
be the weight matrix and b = ( b 1 , . . .
, b d out ) (cid:62) ∈
R d out be the bias vector of a fully-connect layer
A F ( · ) : R d in → R d out , where w ij , i = 1 , . . .
, d out ; j = 1 , . . .
, d in are weights.
Then the output of the fully-connect layer with respect to x ∈ R d in is
[Equation]
To construct a downsampled CNN to represent the fully-connected layer, we deﬁne a sequence w = ( w 0 , . .
. , w d in × d out ) ∈ R d in × d out with length d in × d out by stacking the reversed row vectors of the weight matrix W as
[Equation]
or
[Equation]
in , . .
w 11 , w 2 d in , . .
w 21 , . .
. , w d out d in , . . .
, w d out 1 ) , where w ij are weights of the matrix W .
By Lemma 2 in [64], there exists a sequence of ﬁlter { w ( i ) } Ii =1 of equal ﬁlter length s for some s ∈ [2 , d in × d out ] and I ≤ (cid:100)
( d in × d out − 1) /
( s − 1) (cid:101) such that w has convolutional factorization w ( I ) ∗ w ( I − 1) ∗ · · · ∗ w (1) .
We let { w ( i ) } Ii =1 be the ﬁlters of the I Toeplitz type convolutional matrices T (1) , . . .
, T ( I ) of the convolutional operators
[Equation]
for i = 1 , . . .
, I .
Next we construct the bias vectors { b ( i ) } Ii =1 according to Lemma 3 in [64].
Let
[Equation]
with B denoting the inﬁnity norm of the input x of the fully-connected layer, and let
[Equation]
for i = 2 , . . .
, I − 1 .
Then we have
[Equation]
Finally, we set
[Equation]
where θ ∈ R d in + s ( I − 1) s an arbitrary vector satisfying D
I ( θ ) =
b with D I : R d in + sI → R d out denoting the downsampling operator and b is the bias vector of the target fully-connected layer.
Then we can deﬁned the downsampled CNN by
[Equation]
and by Lemma 1 in [64] we have
[Equation]
And the the total number of free parameters in computing the fully-connected layer A F is no more than
[Equation]
) (cid:101) ≤ 8 d in × d out , which is 8 times of the total parameters of the fully-connected layer.
The depth of f CNN is I + 1 ≤(cid:100)
( d in × d out − 1) /
( s − 1) (cid:101)
+ 1 .
If we take s =
d in × d out , then f CNN is a 2-layer convolutionalnetwork.
By induction, we know that for any MLP with depth L , width ( d 0 , d 1 , . . .
, d L +1 )
andsize S , there exists a downsampled CNN as deﬁned in (23) that exactly computes the MLP.
Thedownsampled CNN has size no more than 8 S , depth no more than L + (cid:80) Li =
(cid:100)
d i × d i +1 /
( s ( i ) − 1) (cid:101)where s ( i ) is the ﬁlter length of the 2 i -th layer of downsampled CNN and the
i − 1 -th layers aredownsampling operations.
More speciﬁcally, if take the maximum possible ﬁlter size s ( i ) =
d i × d i +1for each construction of the fully-connected layer above, the downsampled CNN that computed
the
MLP has depth no more than 2 L , size no more than 8 S and maximum ﬁlter length no more than
W = max { d 0 d 1 , . . .
, d L d L +1 } .
In general, the maximum ﬁlter length s
= O (1) , and the constructeddownsampled CNN has size no more than 8 S , depth no more than L + (cid:80) Li
=0 (cid:100)
d i × d i +1 /
( s ( i ) − 1) (cid:101) ≤
L (1 + W 2 / ( s min − 1)) ≤ 2 L W 2 / ( s min − 1) where s min is the minimum of the ﬁlter lengths overconvolutional layers.
Now we prove the theorem.
Without loss of generality, we assume the Sobolev norm of f is 1, i.e.(cid:107)
f (cid:107) W β,p ( X ) ≤ 1 .
The reason is that we can always approximate f/B 0 ﬁrstly by a network φ withapproximation error (cid:15) , then the scaled network B 0 φ will approximate f with error no more than (cid:15)B 0 .
Besides, it is a trivial case when the Sobolev norm of f is 0.
Firstly, when β > 1 , we divide the proofinto three steps as follows.
Step 1: Discretization.
Given N, M ∈ N + , set K = (cid:98) N 1 /d (cid:99) 2 (cid:98) M 2 /d (cid:99) and δ ∈ (0 , 1 / (3 K )] , for each θ =
( θ 1 , . .
. , θ d ) ∈ { 0 , 1 , . .
, K − 1 } d , we deﬁne
[Equation]
Let Ω( X , K, δ ) =
X\ ∪ θ Q θ be the triﬂing region or the complement of ∪ θ Q θ .
By the deﬁnition of
Q θ , the region X is approximately divided into hypercubes.
By Lemma E.7, there exists a ReLU network ψ 1 with width 4 (cid:98) N 1 /d (cid:99) + 3 and depth 4 M + 5 such that
[Equation]
and the Lipschitz constant of ψ 1 is no more than 2 MK − 2 δ − 2 .
We deﬁne ψ ( x ) :
= ( ψ 1 ( x 1 ) , . . .
, ψ 1 ( x d )) , x = ( x 1 , . . .
, x d ) ∈ R d .
Then we have ψ ( x ) = θ/K :=
( θ 1 /K, . .
, θ d /K ) (cid:62) for x ∈ Q θ and ψ is a ReLU network withwidth d (4 (cid:98) N 1 /d (cid:99) + 3) and depth 4 M + 5 .
Step 2: Local Approximation of averaged Taylor expansion.
We consider the local approximation of f ∈ W β,p ( X ) on the subset Q θ for each θ by its averaged Taylor expansion T β f ( x ) = (cid:82) B θ
T
βy f ( x ) φ ( y ) dy over the ball B θ where
T βy f ( x ) =
[Equation]
(cid:107) α (cid:107) 1 ≤ β 1 α !
D
α f ( y )( x − y )
α , the ball B θ = { y ∈ X : (cid:107) y − ( θ +(1 − δ ) / 2) /K
(cid:107) 2 ≤ (1 − δ ) / (2 K ) } support in Q θ , and φ is an cut-off function supported in ¯ B deﬁned in Deﬁnition E.1.
Note that
Q θ has diameter r Q θ = √ d (1 − δ ) /K
and it is star-shaped with respect to the ball B θ with radius (1 − δ ) /
(2 K ) .
By Lemma E.6, we have error bound for the remainder of the averaged Taylor expansion,
[Equation]
Besides, by Lemma E.2, the averaged Taylor expansion is actually a polynomial T β f ( x ) = (cid:80) (cid:107) α
[Equation]
1 ≤ β c θα x α for x ∈ Q θ with | c θα | ≤ C ( β, d ) for all α where
[Equation]
Next we seek to approximate the polynomial T β f ( x ) = (cid:80) (cid:107) α (cid:107) 1 ≤ β c θα x α locally by deep neural networks.
We ﬁrstly approximate the coefﬁcients of averaged Taylor expansion.
Since θ ∈ { 0 , 1 , . .
, K − 1 } d is one-to-one correspondence to i θ := (cid:80)
dj
=1 θ j K j − 1 ∈ { 0 , 1 . .
, K d − 1 } , we deﬁne
[Equation] [Equation]
then ψ 0 ( x )
=
d (cid:88)
j
=
[Equation]
=
[Equation]
, if x ∈ Q θ ,
[Equation]
where ψ 0 ( x ) has width d (4 (cid:98) N
1 /d (cid:99) + 3) and depth 4 M + 5 .
For any α ∈ N d 0 satisfying (cid:107) α (cid:107) 1 ≤ β and each i = i θ ∈ { 0 , 1 , . . .
, K d − 1 } , we denote ξ α,i :=
( c θα + C ( β, d )) / (2 C ( β, d )) ∈ [0 , 1] .
Since K d ≤ N 2 M 2 , by Lemma E.8, there exists a ReLU network ϕ α with width 16( β + 2)( N + 1) (cid:100) log 2 (8 N ) (cid:101) and depth 5( M + 2) (cid:100) log 2 (4 M ) (cid:101) such that
[Equation]
for all i ∈ { 0 , 1 , . . .
, K d − 1 } .
We deﬁne φ α
[Equation]
Then φ α can be implemented by a network with width 16 d ( β + 2)( N + 1) (cid:100) log 2 (8 N ) (cid:101) ≤ 32 d ( β +2) N (cid:100) log 2 (8 N ) (cid:101) and depth 5( M + 2) (cid:100) log 2 (4 M ) (cid:101) + 4 M + 5 ≤ 15 M (cid:100) log 2 (8 M ) (cid:101) .
And we have forany θ ∈ { 0 , 1 , . .
, K − 1 } d , if x ∈ Q θ ,
[Equation]
Step 3: Local Approximation of the polynomial.
Let ϕ ( t )
=
min { max { t, 0 } , 1 } = σ ( t ) − σ ( t − 1) for t ∈ R where σ ( · ) is the ReLU activationfunction.
With a slightly abuse of the notation, we extend its deﬁnition to R d coordinate-wisely, i.e.,for x = ( x 1 , . . .
, x d ) (cid:62) ∈
X ⊆ R d , we let ϕ ( x ) =
ϕ
( x 1 , . . .
, x d ) :=
( ϕ ( x 1 ) , . . .
, ϕ ( x d ))
(cid:62) denotethe extension of φ to R d .
By Lemma E.9, there exists a ReLU network with width 9 N + 1 and depth4 βM + 8 such that for any t 1 , t 2 ∈ [ a, b ] ,
[Equation]
with (cid:107) φ × (cid:107) W 1 , ∞ (( a,b ) 2 ) ≤ 12( b − a ) 2 .
By Lemma E.10, for any α ∈ N d 0 with (cid:107) α (cid:107) 1 ≤ β ∈ N + , thereexists a ReLU network P α with width 9( N + 1) + β and depth 7( β + 1) 2 M and (cid:107) P α (cid:107) W 1 , ∞ ( X ) ≤ 18such that for any x ∈ X
[Equation]
Step 4: Approximation of f on ∪ θ ∈{ 0 , 1 ,...,K − 1 } d Q θ .
For any x ∈ Q θ , θ ∈ { 0 , 1 , . .
, K − 1 } d , we can now approximate the averaged Taylor expansion off ( x ) by combined sub-networks.
Motivated by this, we deﬁne
[Equation]
where (cid:107) φ 0 (cid:107) W 1 , ∞ ( X ) ≤ 48 C ( β, d ) 2 ( β + 1) d β .
Then φ 0 is our constructed neural network approximating the target function f on X .
Recall that width and depth of ϕ is (2 d, 1) , width and depth of ψis ( d (4 (cid:98) N
1 /d (cid:99) + 3) , 4 M + 5) , width and depth of P α is (9( N + 1) + β, 7( β + 1) 2 M ) , width anddepth of φ α is width ( 16 d ( β + 2)( N + 1) (cid:100) log 2 (8 N ) (cid:101) ,
( M + 2) (cid:100) log 2 (4 M ) (cid:101)
+ 4 M + 5 ) and widthand depth of φ × is (9 N + 1 , 2 βM + 4) .
Hence, by our construction, φ 0 can be implemented by aneural network with width 38( β + 1) 2 d β +1 N (cid:100) log 2 (8 N ) (cid:101) and depth 21( β + 1) 2 M (cid:100) log 2 (8 M ) (cid:101) .
For θ ∈ { 0 , 1 , . . .
, K − 1 } d , thanks to Lemma E.6, we have error bound for the remainder of the averaged Taylor expansion,
[Equation]
where C β,d,r = 3 βd β − 1 Γ( d/ 2 + 1) π − d/ 2 (1 + 2 √ d ) d .
Then for any θ ∈ { 0 , 1 , . . .
, K − 1 } d , theapproximation error of φ 0 on f over Q θ can be upper bounded,
[Equation]
We denote E α = (cid:107) c θα x α − φ × (cid:0) φ α ( x ) , P α ( ϕ ( x )) (cid:1) (cid:107) W 1 , ∞ ( Q θ ) for each α ∈ N d 0 with (cid:107) α (cid:107) 1 ≤ β .
Using the inequality | t 1 t 2 − φ × ( t 3 , t 4 ) | ≤ | t 1 t 2 − t 3 t 2 | + | t 3 t 2 − t 3 t 4 | + | t 3 t 4 − φ × ( t 3 , t 4 ) | ≤| t 2 || t 1 − t 3 | + | t 3 || t 2 − t 4 | + | t 3 t 4 − φ × ( t 3 , t 4 ) | for any t 1 , t 2 , t 3 , t 4 ∈ R , and by (24), (25) and (26),for (cid:107) α (cid:107) 1 ≤ β
we have
[Equation] [Equation]
[Equation]
[Equation]
d )
[Equation]
N
[Equation]
≤ (12 C ( β, d )( β + 1) + 24 C ( β, d ) 2 )( NM )
[Equation]
where the last inequality holds since p ≥ 1 and (cid:107) f (cid:107) W β,p (Ω) ≤ B 0 .
Besides, observe that the numberof terms in the summation can be bounded by
[Equation]
Recall that K = (cid:98) N
1 /d (cid:99) 2 (cid:98) M 2 /d (cid:99) , combining (27), (28) and (29) and θ ∈ { 0 , 1 , . . .
, K − 1 } d , wehave
[Equation]
Recall that X =
Ω
( X , K, δ )
∪ ( ∪ θ ∈{ 0 , 1 ,...,K − 1 } d Q θ ) and the Lebesgue measure of Ω( X , K, δ ) canbe upper bounded by δdK where δ ∈ (0 , 1 / (3 K )] can be arbitrarily small.
Then the approximationerror (cid:107) f ( x ) − φ 0 ( x ) (cid:107) W
[Equation]
p ( X ) can be further bounded as follows.
[Equation]
where (cid:107) f − φ 0 (cid:107) pW 1 ,p (Ω( X ,K,δ )) can be arbitrarily small since (cid:107) f (cid:107) pW 1 , ∞ ( X ) and (cid:107) φ 0 (cid:107) pW 1 , ∞ ( X ) arebounded, and δ ∈ (0 , 1(3 K )] can be arbitrarily small.
When p = ∞ , the target function f ∈ W β, ∞ ( X ) actually is Hölder smooth with order β and constant
B 0 = (cid:107) f (cid:107) W β, ∞ ( X ) .
Then by Lemma A.8 in [48], for any x, x 0 ∈ X , we have
[Equation]
where s = (cid:98) β (cid:99) .
This reminder term could be well controlled when the approximation to Taylor expansion in implemented in a fairly small local region.
By similar construction of approximators [28], it is proved that there exists a function φ 0 implemented by a neural network with width 38( (cid:98) β (cid:99) + 1) 2 d (cid:98) β (cid:99)
+1 N (cid:100) log 2 (8 N ) (cid:101) and depth
( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101) such that
[Equation]
for any x ∈ ∪ θ ∈{ 0 , 1 ,...,K − 1 } d Q θ .
If the probability measure of X is absolutely continuous with respect to the Lebesgue measure, we have
[Equation]
As we have shown at the beginning of the proof, the multilayer-perceptron φ 0 here can be computed by a downsampled CNN with ﬁlter lengths
≤ s min ≤ s max ≤ W = 38 2 ( (cid:98) β
(cid:99) +1) 4 d {+2 (cid:98) β (cid:99) +
+}
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 and depth L ≤ 42( (cid:98)
β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101)W / ( s min − 1) and size
S ≤ 8 W L .
We know that there exists a f CNN ∈ F CNN such that the approximation results hold.
This completes the proof.
E.2 Proof of Theorem 2.
Proof.
Let K ∈ N + and τ ∈ (0 , 1 /K ) , deﬁne a region Ω( X , K, τ ) of X as Ω( X , K, ε ) = ∪ di =1 { x = [ x 1 , x 2 , ..., x d ] T : x i ∈ ∪ K − 1 k =1 ( k/K − τ, k/K ) } .
By Theorem 3.3 of [28], for any M, N ∈ N + , there exists a function f ∗ ∈ F MLP with (cid:107) f ∗ (cid:107) ∞
≤ B , depth 21( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101) and width 38( (cid:98) β (cid:99) + 1) 2 d (cid:98)
β
(cid:99) +1 N (cid:100) log 2 (8 N ) (cid:101) such that
[Equation]
for any x ∈ X\ Ω( X , K, τ )
where K = (cid:98) N 1 /d (cid:99) 2 (cid:98)
M 1 /d (cid:99) 2 and τ is an arbitrary number in (0 , 13 K ] and (cid:98) a (cid:99) denotes the largest integer smaller than a .
Note that the Lebesgue measure of Ω( X , K, τ ) is no more than dKτ which can be arbitrarily small if τ is arbitrarily small.
Since P is absolutely continuous with respect to Lebesgue measure, then we have
[Equation]
By the proof of Theorem 2.1, any MLP can be computed exactly by a downsampled CNN.
Thusthere exists a function f implemented by CNN with its depth no more than L = 42( (cid:98) β (cid:99) +1) 2 M (cid:100) log 2 (8 M ) (cid:101)W / ( s min − 1) where W = 38 2 ( (cid:98) β (cid:99)
+ 1) 4 d 2 (cid:98) β (cid:99) +2 N 2 (cid:100) log 2 (8 N ) (cid:101) 2 and its ﬁlter lengths
≤ s min ≤ s max ≤ W = 38 2 ( (cid:98) β
(cid:99) + 1)
d 2 (cid:98) β (cid:99) +2 N 2 (cid:100) log 2 (8 N ) (cid:101) 2 such that
E (cid:107) f
( X ) − f 0 ,T ( X ) (cid:107) 2 ≤ 18 B 0 ( (cid:98) β (cid:99) + 1)
d (cid:98) β (cid:99) +( β ∨ 1) / 2 ( NM ) − 2 β/d .
E.3 Proof of Theorem 2.4
Proof.
The data can be projected to a low-dimensional space and then use CNN to approximate the low-dimensional function where the idea is similar to that of Theorem 1.2 in [55].
Based on
Theorem 3.1 in [3], there exists a linear projector A ∈ R d δ × d that maps a low-dimensional manifold in a high-dimensional space to a low-dimensional space nearly preserving the distance.
Speciﬁcally, there exists a matrix A ∈ R d δ × d such that AA T =
( d/d δ )
I d δ where I d δ is an identity matrix of size d δ × d δ , and
[Equation]
for any x 1 , x 2 ∈ M .
And it is easy to check
[Equation]
Note that for any z ∈ A ( M ) , there exists a unique x ∈ M such that Ax = z .
To prove this, let x
[Equation]
M be another point on M satisfying Ax (cid:48)
=
z , then (1 − δ ) (cid:107) x − x (cid:48) (cid:107) 2 ≤ (cid:107)
Ax − Ax (cid:48) (cid:107) 2 ≤ (1 + δ ) (cid:107) x − x (cid:48) (cid:107) 2 implies that (cid:107) x − x (cid:48) (cid:107) 2 = 0 .
Then for any z ∈ A ( M ) , deﬁne x z = SL ( { x ∈ M :
Ax = z } ) where SL ( · ) is a set function which returns a unique element of a set.
Note that if Ax = z where x ∈ M and z ∈ A ( M ) , then x = x z by our argument since { x ∈ M :
Ax = z } is a set with only one element when z ∈ A ( M ) .
And we can see that SL : A ( M ) → M is a differentiable function with the norm of its derivative locates in [1 / (1 + δ ) , 1 / (1 − δ )] , since
[Equation]
, for any z 1 , z 2 ∈ A ( M ) ⊆
E where E :=
[ − (cid:112) d/d δ , (cid:112) d/d δ ] d δ .
For the high-dimensional function f 0 : [0 , 1] d → R 1 , we deﬁne its low-dimensional representation ˜ f 0 : R d δ → R 1 by ˜ f
( z ) =
f
( x z ) , for any
z ∈ A ( M ) ⊆
R d δ .
Recall that f 0 ∈
H β ( X , B 0 ) , then ˜ f 0 ∈ H β ( A ( M ) , B 0 / (1 − δ ) β ) .
Note that M is compact and A is a linear mapping, then by the extended version of Whitney’ extension theorem in [17], there exists a function ˜ F 0 ∈ H
β ( E, B 0 / (1 − δ ) β ) such that ˜ F 0 ( z ) =
˜ f
0 ( z ) for any z ∈ A ( M ) .
With E =
[ − (cid:112) d/d δ , (cid:112) d/d δ ] d δ , by Theorem 2.1, for any N, M ∈ N + , there exists a function ˜ f n : R d δ → R 1 implemented by a downsampled CNN with ﬁlter lengths 2 ≤ s min ≤ s max ≤ W = 38 2 ( (cid:98)
β
(cid:99) + 1)
d 2 (cid:98) β (cid:99) +
δ
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 and depth L =
( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101)W / ( s min − 1) such that
[Equation]
for all z ∈ E \ Ω( E ) where Ω( E ) is a subset of E with an arbitrarily small Lebesgue measure as wellas Ω := { x ∈ M ρ : Ax ∈ Ω( E ) } does.
If we deﬁne f ∗ n = ˜ f n ◦ A which is f ∗ n ( x ) = ˜ f n ( Ax ) for any x ∈ X , then f ∗ n is also a downsampled
CNN with the same parameter as ˜ f n .
For any x ∈ M ρ \ Ω and z
=
Ax , there exists a ˜ x ∈ M suchthat (cid:107) x − ˜ x (cid:107) 2 ≤ ρ , then
[Table]
[Equation]
where C 2 > 0 is a constant not depending on any parameter.
The last inequality follows from
[Equation]
[Equation]
d/d δ +1 − δ } −
[Equation]
1 − δ ) 1 − β .
Since the probabilitymeasure ν of X is absolutely continuous with respect to the Lebesgue measure, we have
[Equation]
where d δ = O ( d M log( d/δ ) /δ 2 ) is assumed to satisfy d δ (cid:28) d .
This completes the proof of Theorem2.4.
E.4 Proof of Theorem 4.2
The proof follows Theorem 11.8 and Corollary 3.19 in [43].
Recall that ( X, Y ) has joint distribution
P , we let P n denote the empirical distribution of the sample S =
{ ( X i , Y i ) } ni =1 and let ( ˜ X, ˜ Y ) ∼ P n .
Note that for any f ∈ F CNN we have (cid:107) f (cid:107) ∞ ≤ B , then 0 ≤ φ ( yf ( x )) ≤ φ B for any ( x, y ) ∼ P .
For any f ∈ F CNN and t ≥ 0 , we denote by c ( f, t ) the classiﬁer deﬁned by c ( f, t ) : ( x, y )
(cid:55)→( φ ( Y f ( X )) > t ) .
Correspondingly, we deﬁne the risk of c ( f, t ) by
R ( c ( f, t ))
=
P [ c ( f, t )( X, Y ) = 1] = P [ φ ( Y f ( X )) > t ] , and similarly deﬁne its empirical risk by R n ( c ( f, t ))
=
P n [ φ ( ˜ Y f
( ˜ X ))
> t ] .
Then we write
[Equation]
And this implies that
[Equation]
φ B (cid:35) .
F CNN =
{ ( x, y )
(cid:55)→ φ ( yf ( x )) :
f ∈ F CNN } .
Now we have with probability at least 1 − δ over thechoice of an i.i.d.
sample S of size n that
[Equation]
Next, we bound the pseudo-dimension of F CNN by that of F CNN , and this leads to the ﬁnal stochastic error bound in terms of the parameters of F CNN .
For a sample S = {
( X i , Y i ) } ni =1 with size n , if there exist t 1 , . . . , t n ∈ [0 , φ B ] such that for each ( b 1 , . . .
, b n ) ∈ { 0 , 1 } n there exists a f ∈ F such that 1 ( φ ( y i f
( x i ))
> t i )
=
b i for i = 1 , . . . , n , then since φ is convexity and non-increasing, we also have 1 ( f ( x i ) >
φ − 1 ( t i ))
=
1 − b i if y i
=
and 1 ( f ( x i ) > − φ − 1 ( t i ))
=
b i if y i
=
− 1 .
Thus { x i } ni =1 is shattered by F CNN and the threshold values y 1 φ − 1 ( t 1 ) , . . .
, y n φ − 1 ( t n ) witness the shattering.
This implies Pdim ( F CNN ) ≤ Pdim ( F CNN ) .
Besides, for piece-wise linear neural networks
F CNN , it is proved in Theorem 6 of [4] that
[Equation]
for some universal constant C > 0 where S and L are the size and depth of F CNN respectively.
Combine above inequities, we have with probability at least 1 − δ over the choice of an i.i.d.
sample
S of size n thatsup
[Equation]
for some universal constant C > 0 .
E.5 Proof of Lemma 4.
Proof.
To deal with the approximation of unbounded target function, we consider truncating thetarget function f 0 by a constant T , where T may depend on n .
Let f 0 ,T be the truncated version off 0 deﬁned as
[Equation]
Then, the approximation error can be decomposed into two terms that are easier to deal with.
Denotethe error of the loss function φ due to truncation by
[Equation]
where Ran ( f 0 ) is the range of the target function f 0 .
This error decreases as the threshold T increases,however, a bigger T leads to bigger B and B φ , thus a bigger stochastic error.
Firstly, we prove that under Assumption 4.3(a) and (b), f 0 ,T , the truncated version of the targetfunction f 0 , is bounded by T and continuous on X .
Recall that for any a ∈ R and η ∈ [0 , 1] ,the conditional risk function H ( η, a ) =
ηφ ( a ) + (1 − η ) φ ( − a ) .
Given η ∈ [0 , 1] , the functiondH ( η, a ) /da
= ηφ (cid:48)
( a ) − (1 − η ) φ (cid:48) ( − a ) is continuous in both η and a .
Thus under Assumption 4.3(
b) the solution a ( η ) =
arg min a ∈ ¯ R H ( η, α )
is continuous with respect to η , and | a ( η ) | < ∞ whenη ∈ (0 , 1) .
By Assumption 4.3 (a), η ( x ) is continuous on { x ∈ X : η ( x ) ≤ 1 − δ } for any δ ∈ (0 , 1) ,thus f 0 ( x ) = f 0 ( η ( x )) is continuous and η ( x ) φ (cid:48)
( f 0 ( x )) − (1 − η ( x )) φ (cid:48) ( − f 0 ( x ))
=
on for anyx ∈ X .
And f 0 ( x ) is continuous on { x ∈ X : | f 0 ( η ( x ))
| ≤ T } for any T ≥ 0 .
For any T > 0 , theset { x ∈ X : | f 0 ( x ) | ≤ T } is a compact set.
Thus f 0 ,T , the truncated version of target function isalso continuous on X .
Recall that for any f , the φ -risk is deﬁned by R ( f ) =
E φ ( Y f ( X )) , then for f 0 ,T , the truncatedversion of target function satisﬁes
R
[Equation]
f
[Equation] [Equation] [Equation]
X
[Equation]
} =
[Equation]
[Equation]
X
[Equation] [Equation]
X
[Equation]
=
E (cid:104) η
( X ) { φ ( f 0 ,T ( X ))
− φ ( f 0 ( X )) }
+ (1 − η ( X )) { φ ( − f 0 ,T ( X )) − φ ( − f
( X )) } (cid:105) .
For x ∈ X , we deﬁne g ( x ) =
η ( x ) { φ ( f 0 ,T ( x )) − φ ( f 0 ( x )) }
+ (1 − η ( x )) { φ ( − f 0 ,T ( x )) −φ ( − f 0 ( x )) } .
Recall that φ is non-increasing and we have:
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
Thus for any x ∈ X , we have g ( x ) ≤
φ ( T ) − inf a ∈ Ran ( f 0 ) φ ( a ) and R ( f 0 ,T ) − R ( f 0 ) =
E { g ( X ) } ≤φ ( T ) − inf a ∈ Ran ( f 0 ) φ ( a ) , where Ran ( f 0 ) is the range of the target function f 0 .
[Equation]
then left thing to do is to prove
[Equation]
With f 0 ∈ H β ([0 , 1] d , B 0 )
[Equation]
φ is B φ -Lipschitz
[Equation]
and it is a straightforward proof that
[Equation]
E.6 Proof of Theorem 4.
The main focus of the proof is to deal with the truncation.
If f 0 ∈ H β ([0 , 1] d , B 0 ) is uniformly bounded by T ≤ B , by Theorem 2.2, there exists a ReLU neural network φ such that
[Equation]
[Equation]
We let φ 0 ( x ) = σ ( φ ( x ) + B ) − σ ( φ ( x ) − B ) − B be the truncated network where σ denotes the
ReLU activation function.
Then
[Equation]
If f 0 ∈ H β ([0 , 1] d , B 0 ) is not uniformly bounded by by T , then the set { x : | f 0 ( x ) | ≤ T } does not equal to the whole domain X .
Actually { x : | f 0 ( x ) | ≤ T } can be written as a union of connected subsets since f 0 is Hölder smooth on its domain X
= [0 , 1] d , and on interior of each connected subset, the function f 0 keeps its Hölder smoothness β .
By [17], the truncated target function f 0 ,T on { x ∈ X : | f 0 ( x ) | ≤ T } can be extended to X to be β -Hölder and bound T + c for some small constant c > 0 .
Let ˜ f 0 ,T be the extended function on X , then by Theorem 2.2, there exists a ReLU neural network φ such that
[Equation]
We let φ 0 ( x ) = σ ( φ ( x ) + B ) − σ ( φ ( x ) − B ) − B be the truncated network where σ denotes the
ReLU activation function.
Then
[Equation]
E.7 Proof of Theorem 4.6
Combining Lemma 4.1, Theorem 4.2 and 4.5, Theorem 4.6 is easily proved.
For the proof of Theorem 4.7, we need the following lemma, which can be proved using the techniquein [55].
We refer to [55] for the detailed description of the technique.
E.8 Proof of Theorem 4.7
Proof.
The proof is similar to that of Theorem 4.6, except how we deal with inf f ∈F CNN E (cid:107) f ( X ) −f 0 ,T ( X ) (cid:107) 2 .
Combining Lemma 4.4 and using Theorem 2.4, we have
[Equation]
This completes the proof of Theorem 4.7.
E.9 Proof of the cross entropy loss example
[-the cross entropy loss example in Section 
-]
{+in Section 
+}
For the cross entropy loss function φ ( a ) =
− log { 0 .
5 + a } , with the choice T = B
= 0 .
5 −n β/ (2 d +4 β ) , we have B ≤ 0 . 5 , ∆ φ ( T ) ≤ n − β/ (2 d +4 β ) and B φ = n β/ (2 d +4 β ) .
Suppose Assumptions2.3, 4.3 hold and f 0 ∈ H β ([0 , 1] d , B 0 ) .
Take N = 1 in (7) and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) withdepth L ≤ 378 · 38 2 ( (cid:98)
β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38
( (cid:98) β (cid:99) +1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) .
Theorem 4.6 implies that, with probability at least 1 − exp {− n d/
( d +2 β ) } , the excess φ -risk of the
ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
or simply
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B 0 ( β + 1) 8 d 3 β +3 / ( s min − 1)) is a constant independent of n .
Theexcess risk bound under the approximate low-dimensional manifold assumption can be obtained in asimilar way.
Under the modiﬁed cross entropy loss φ ( a ) = max {− log { 0 .
5 + a } , τ } with τ = − log { 0 .
5 + T } ,the corresponding measurable minimizer deﬁned in (14) is f 0 ,T and ∆ φ ( T )
= 0 by deﬁnition sincethe inﬁmum of the modiﬁed cross entropy loss can be achieved within [ − T, T ] ⊂ [ − 0 .
Bychoosing T =
B = 0 .
5 − (log n ) − 1 and τ = − log { 1 − (log n ) − 1 } , with the modiﬁed cross entropyloss, Theorem 4.6 implies that with probability at least 1 − exp {− n d/ ( d +2 β ) } , the excess φ -risk ofthe
ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) + 1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
E.10 Proof of the SVM example
Proof.
Based on Lemma 4.1, our proof focus on controlling the approximation error inf f ∈F CNN R ( f ) − R ( f 0 ) .
Recall that f 0 ( x ) =
sign(2 η ( x ) − 1) , we would ﬁrstly approximate Hölder smooth function η by CNNs.
Similar to the proof in Theorem 4.6, for any N, M ∈ N + , there exists a function η ∗ ∈ F CNN with bound B = 1 , depth
L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 M (cid:100) log(8 M ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98)
β (cid:99) +2 N (cid:98) log(8 N ) (cid:99) and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 38 4 ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 M (cid:100) log(8 M ) (cid:101) N 2 (cid:100) log(8 N ) (cid:101) 2 / ( s min − 1) such that | η ∗ ( x )
− η ( x ) | ≤ 18 B 0 ( (cid:98) β (cid:99) + 1) 2 d (cid:98)
β (cid:99) +( β ∨ 1) / 2 ( NM ) − 2 β/d , for any x ∈ X\ Ω where Ω is a set with arbitrarily small Lebesgue measure.
Let ε n = 18 B
( (cid:98) β (cid:99) +1)
2 d (cid:98) β (cid:99) +( β ∨ 1) /
( NM )
− 2 β/d .
We construct the neural network f CNN by adding one layer to η ∗ as
[Equation]
, where σ is the ReLU activation function.
Then we can see that (cid:107) f
CNN (cid:107) ∞ ≤ 1 and f CNN ( x ) = 1 ifη ∗ ( x ) ≥ ε n +1 / 2 , f CNN ( x ) = 2( η ∗ ( x ) − 1 / 2) /ε
n − 1 if 1 / 2+ ε n > η ∗ ( x ) ≥ 1 / 2 and f CNN ( x ) =− 1 if η ∗ ( x ) < 1 / 2 .
Let Ω η,ε = { x ∈ X : | 2 η ( x ) − 1 | > ε } for (cid:15) > 0 .
Then for x ∈ Ω η, 4 ε n , wehave | f CNN ( x ) − f 0 ( x ) | = 0 , since η ∗ ( x ) − 1 / 2 = ( η ( x ) − 1 / 2) − ( η ∗ ( x ) − η ( x )) >
ε n whenη ( x ) − 1 / 2 > 2 ε n and η ∗ ( x ) − 1 / 2 < − ε n when η ( x ) − 1 / 2 < − 2 ε n .
Then
[Equation]
Setting the parameters in η ∗ by N = 1 and M = (cid:98) n d/ { d +4 β ( q +1) } (cid:99) and size (number of parameters)
[Equation]
s min − 1) , and let δ =exp( − n d/
( d +2 β ( q +1)) ) , then by Theorem 4.6, with probability at least 1 − exp( − n d/ ( d +2 β ( q +1)) ) ,
[Equation]
The excess risk bound under the approximate low-dimensional manifold assumption can be obtainedin a similar way.
E.11 Proof of the logistic loss example
The proof is similar to that of the cross entropy loss example.
We omitted the proof here.
E.12 Proof of the exponential loss example
The proof is similar to that of the cross entropy loss example.
We omitted the proof here.
E.13 Supporting deﬁnitions and Lemmas
For ease of reference, we collect several deﬁnitions and existing results that we used in our proofs.
Firstly, we present the deﬁnition and properties of the averaged Taylor expansion in chapter 4 of [10].
Deﬁnition E.1 (Averaged Taylor expansion) .
Let β ∈ N , 1 ≤ p ≤ ∞ and f ∈ W β − 1 ,p , and letΩ ∈ R d , x 0 ∈ Ω , r > 0 , B :=
B r ( x 0 ) = { x ∈ R d : (cid:107) x − x 0 (cid:107) 2 < r } with its closure ¯ B compact inΩ .
The corresponding Taylor polynomial of order β of f averaged over B is deﬁned as
[Equation]
where
[Equation]
and φ is an arbitrary cut-off function supported in ¯ B being inﬁnitely differentiable, i.e., φ ∈ C ∞ ( R d )with supp φ = ¯ B and (cid:82) R
d φ ( x ) dx = 1 .
For example, let
[Equation]
else
[Equation]
and let c = (cid:82) R d ψ ( x ) dx ( c > 0) , then φ ( x ) = ψ ( x ) /c is a cut-off function and (cid:107) φ (cid:107) L ∞ ( B ) :
=max | φ ( x ) | ≤ c ( d ) · r − d where c ( d ) > 0 is a constant depending only on d .
Besides,
[Equation]
else
[Equation]
is a another example of cut-off function, in which case φ puts constant weight on the ball ddimensional B with radius r .
The averaged Taylor polynomial deﬁned above is an integral of the traditional Taylor expansionweighted by the cut-off function over a region.
Following the equations (4.1.5)-(4.1.8) in [10], weshow the below lemma that the averaged Taylor polynomial of order β is indeed a polynomial ofdegree less than β in x .
Lemma E.2.
Let β ∈ N , 1 ≤ p ≤ ∞ and f ∈ W β − 1 ,p (Ω) , and let x 0 ∈ Ω , r > 0 , R ≥ 1 such that for the ball B :=
B r ( x 0 ) with its closure ¯ B compact in Ω and (cid:107) y (cid:107) ∞
≤ R for all y ∈ B .
Then
the
Taylor polynomial of order β of f averaged over B can be written as
[Equation]
for x ∈ Ω , where
[Equation]
for all α satisfying (cid:107) α (cid:107) 1 ≤ β − 1 .
Proof.
Recall that for x, y ∈ R d and α =
( α 1 , . .
. , α d ) ∈ N d 0 , if we write
[Equation]
where γ = ( γ 1 , . . . , γ d ) and θ = ( θ 1 , . .
, θ d ) are d -tuples of nonnegative integers, and a ( γ, θ ) ∈
Rare the constants satisfying | a ( γ, θ ) | ≤ (cid:0) αγ (cid:1) =
α ! / ( γ ! θ !)
in multi-index notation.
Then
T
[Equation]
Then we consider bounding the coefﬁcient of x γ .
Note that
[Equation]
where the second inequality holds since (cid:107) y (cid:107) ∞
≤ R, ∀ y ∈ B and the use of Hölder’s inequality with1 /p
+ 1 /q
= 1 , the third inequality holds again by Hölder’s inequality, and the last two lines holdsince (cid:107) φ (cid:107) L 1 ( B )
= 1 and (cid:107) φ ( x ) (cid:107) L ∞ ( B ) ≤ π − d/ 2 Γ( d/ 2 + 1) · r − d .
Lastly, the coefﬁcient c γ of x γcan be upper bounded by
[Equation]
Note that
[Equation]
Thus we have
[Equation]
Next, we present the main approximation result of averaged Taylor expansion on Sobolev functions by
Bramble-Hilbert (Lemma 4.3.8 in [10]) after introduce some prerequisite deﬁnitions and conceptions.
Deﬁnition E.
(Star-shaped set) .
Let Ω , B ⊂ R d .
Ω is star-shaped with respect to B if for all x ∈ Ω ,the closed convex hull of { x } ∪ B is a subset of Ω .
Deﬁnition E.4 (Chunkiness parameter) .
Suppose Ω ⊂ R d has diameter r Ω and is star-shaped withrespect to a ball B .
Let r max = sup { r ∈ R + : Ω is star-shaped with respect to a ball of radius r } .
Then the chunkiness parameter of Ω is deﬁned by γ = r Ω /r max .
Lemma E.5 (Proposition 4.2.8 in [10]) .
Let f ∈ W β,p (Ω) and T β be its averaged Taylor expansion deﬁned in (33) over the ball B with radius r , then the remainder R β f :=
f − T β f satisﬁes
[Equation]
where C x is the convex hull of { x } ∪ B , z
= x + s ( y − x ) , k α ( x, z ) =
(1 /α !)
( x − z ) α k ( x, z ) and | k (
[Equation]
Lemma E.6 (Bramble-Hilbert) .
Let B be a ball in Ω such that Ω is star-shaped with respect to B and such that its radius r > r max / 2 , where r max is deﬁned in Deﬁnition E.4.
Let T β
f
be the Taylor polynomial of order β of f averaged over B where f ∈ W β,p (Ω) and p ≥ 1 .
Then
[Equation]
where r Ω is the diameter of Ω .
The proof can be found in Lemma 4.3.8 in [10].
Lemma E.7 (Proposition 4.3 in [39]) .
For any N, M, d ∈ N + and δ ∈ (0 , 3 K ] with K = (cid:98) N 1 /d (cid:99) 2 (cid:98) M
/d
(cid:99) , there exists a one-dimensional function φ implemented by a ReLU FNN with width 4 (cid:98) N 1 /d (cid:99) + 3 and depth 4 M + 5 such that
[Equation]
for
[Equation]
[Equation]
Lemma E.8 (Proposition 4.4 in [39]) .
Given any N, M, s ∈ N + and ξ i ∈ [0 , 1] for i = 0 , 1 , . . .
, N 2 L 2 − 1 , there exists a function φ implemented by a ReLU FNN with width 16 s ( N + 1) (cid:100) log 2 (8 N ) (cid:101) and depth (5 M + 2) (cid:100) log 2 (4 M ) (cid:101) such that | φ ( i ) − ξ i | ≤ N − 2 s M − 2 s , for i = 0 , 1 , . . .
, N 2 M 2 − 1 , and 0 ≤ φ ( x ) ≤ 1 for any x ∈ R .
The next lemma demonstrate that the production function and polynomials can be approximated by
ReLU neural networks.
The basic idea is ﬁrstly to approximate the square function using “sawtooth"functions then the production function, which is ﬁrstly raised in [60].
A general polynomial can befurther approximated combining the approximated square function and production function.
Thefollowing two lemmas are more general results than those in [60].
Lemma E.9 (Lemma 3.4 in [24]) .
For any N, M ∈ N + , and a, b ∈ R with a < b , there exists a function φ implemented by a ReLU FNN with width 9 N + 1 and depth 2 M such that (cid:107) φ (cid:107) W 1 , ∞
(( a,b ) 2 ) ≤ 12( b − a ) 2 and (cid:107) φ ( x, y ) − xy (cid:107) W 1 , ∞
(( a,b ) 2 ) ≤ 6( b − a ) 2 N − M for any x, y ∈
( a, b ) .
Lemma E.10 (Theorem 4.1 in [39], Proposition 3.6.
in [24]) .
Assume P ( x ) =
x α = x
α 1 1 x α 2 2 · · · x α d d for α ∈ N d with (cid:107) α (cid:107) 1 ≤ k ∈ N + .
For any N, M ∈ N + , there exists a function φ implemented by a
ReLU FNN with width 9( N + 1) + k − 1 and depth 14 k 2 M such that (cid:107) φ (cid:107) W 1 , ∞ ( X ) ≤ 18 and (cid:107) φ ( x ) − P ( x ) (cid:107) W 1 , ∞ ( X ) ≤ 10 k ( N + 1) − 7 kM .