Approximation with CNNs in Sobolev Space: with Applications to Classiﬁcation
Abstract
We derive a novel approximation error bound with an explicit prefactor for Sobolevregular functions using deep convolutional neural networks (CNNs).
The bound is non-asymptotic in terms of the network depth and ﬁlter lengths, in a rather ﬂexible way.
For Sobolev-regular functions which can be embedded into the Hölder space, the prefactor of our error bound depends on the ambient dimension polynomially instead of exponentially as in most existing results, which is of independent interest.
We also establish a new approximation result when the target function is supported on an approximate lower-dimensional manifold.
We apply our results to establish non-asymptotic excess risk bounds for classiﬁcation using CNNs with convex surrogate losses, including the cross-entropy loss, the hinge loss, the logistic loss, the exponential loss and the least squares loss.
We show that the classiﬁcation methods with CNNs can circumvent the curse of dimensionality if input data is supported on a neighborhood of a low-dimensional manifold.
Introduction
Classiﬁcation methods with hypothesis spaces speciﬁed through deep neural networks have achieved remarkable successes in a variety of machine learning tasks [36].
In particular, convolutional neural networks (CNNs) have demonstrated outstanding performance in many applications, including computer vision [35], natural language processing [57], and sequence analysis in bioinformatics [1, 65].
However, to the best of our knowledge, there has only been limited study on the approximation with CNNs and systematic studies on the properties of various classiﬁcation methods using CNNs in terms of the non-asymptotic bounds for the excess risks.
1.1 Related work
There has been much effort devoted to understanding the approximation properties of deep neural networks in recent years.
Many interesting results have been obtained concerning the approximation power of deep neural networks for multivariate functions; some examples of important recent works include [59, 60, 40, 49, 53, 54, 44, 13, 28].
These works focused on the approximation power for
ReLU activated feedforward neural networks (FNNs) on different kinds of smooth functions.
The approximation power of CNNs has been studied in limited works.
[2] studied the approximation power of the composited CNNs, where the network consists of fully-connected CNNs followed by a fully-connected layer.
They proved that under suitable conditions, convolution neural networks can inherit the universal approximation property of its last fully connected layers.
[48] showed that for translation equivariant functions, all upper and lower approximation bounds are equivalent between
FNNs and CNNs with the same order of network length and size.
[47] showed that ResNet-type
CNNs, a special CNN architecture with skip-layer connections (or identity connections between inconsecutive layers), can replicate the learning ability of FNNs having block-sparse structures.
The
ResNet-type CNN in [47], as claimed, can be dense, and its width, channel size, and ﬁlter size are constant with respect to sample size rather than the diverging one in [48].
In addition, a series of papers studied the approximation power of deep CNNs with Toeplitz type convolutional matrices [66, 64, 63].
The universality of such CNNs was established in [64] for target functions restricted to the Sobolev space W β, 2 ( R d ) with β ≥ d/ 2 + 2 .
The approximation error is of the order
O
( L − 1 / 2 − 1 /d )
(up to a logarithmic factor), where L is the number of layers of such CNNs.
But the CNN deﬁned in [64] has width increasing linearly with respect to depth.
To overcome the difﬁculty in theoretical analysis of such CNNs, [63] introduced a downsampling operator to reduce the widths and the approximation power of downsampled CNNs for a special class of functions ( β -Hölder continuous ridge function) was studied.
The approximate rate is shown to be O ( N − β ) by downsampled CNNs with a uniform ﬁlter length s
= O (4 N + 6) and width W
= O ( N ) for positive integer N .
Further, it was shown that a downsampled CNN can compute the same function as a FNN does, with the total number of free parameters of downsampled CNN being at most 8 times of that of FNN.
For the CNNs considered by [66, 16, 63] and [64], each convolutional layer has only one ﬁlter with size s and the weight matrix is of a special Toeplitz type with shape ( d in + s )
× d in and no fully-connected layer is allowed in hidden layers.
Approximation properties of convolutional architectures for target functions deﬁned on inﬁnite time domains tailored to temporal sequence data were studied in [27].
To mitigate the curse of dimensionality and show the advantages of CNNs in classiﬁcation problems, several works studied the approximation as well as the statistical learning theories under some low-dimensional assumptions on the target function or the data distribution.
In [32, 33], the posterior probability function is assumed to be Hölder smooth satisfying hierarchical max-pooling model.
In [34], the target function is assumed to be spatially rotation invariant.
In [18] and [38], input data is assumed to be supported on spheres and manifolds, respectively.
Under different structural assumptions on the target function, similar results have also been proved for FNNs [7, 51, 52].
If data is assumed to be supported on low-dimensional manifolds, the curse of dimensionality can also be mitigated for FNNs [50, 44, 12, 28].
1.2 Our contributions
We study the approximation power of CNNs for functions in Sobolev spaces, including the special case of Hölder class and apply our new approximation results to establish non-asymptotic error bounds for several important classiﬁcation methods using CNNs.
Our main contributions are as follows.
(i)
We derive a novel error bound for approximating Sobolev-regular functions using deep CNNs, with the error bound explicitly expressed in terms the network parameters and model parameters.
The error bound depends on the network depth and ﬁlter lengths in a rather ﬂexible way.
For functions which can be embedded into Hölder space, the prefactor of our error bound depends on the ambient dimension polynomially instead of exponentially as in existing results.
We also establish a novel approximation result when the target function is supported on an approximate lower-dimensional manifold, which shows that CNNs are capable of mitigating the curse of dimensionality under such a distributional condition.
(ii)
As an application of our CNN approximation results, we establish non-asymptotic bounds for the stochastic and approximation errors of classiﬁcation methods with a general class of convex surrogate losses in Sobolev space using CNNs.
Our main results are also applicable to other problems that use CNNs approximations.
(iii)
We apply our general results to establish the non-asymptotic excess risk bounds for classiﬁcation using CNNs with convex surrogate losses in Sobolev space, including the cross-entropy loss, the hinge loss (SVM), the logistic loss, the exponential loss, and the least squares loss.
We show that classiﬁcation methods with CNNs can circumvent the curse of dimensionality if the input data is supported on a neighborhood of a low-dimensional manifold embedded in the ambient space.
[Table]
This paper
[Table] In Table [Table] we summarize and compare some recent and most related work on CNN approximation results for various function spaces
[Table]
The statistical convergence properties of the excess risk of the empirical risk minimizer (ERM) for the misclassiﬁcation 0-1 loss have been studied extensively under various conditions.
Different types of hypothesis spaces have been used in these approaches, including the linear space, the reproducing kernel Hilbert space and the class of tree-based models, see, for example, [9, 55, 42, 6, 29, 8] and [62].
However, there are limited studies worked on deep binary classiﬁcations by FNNs [46, 25, 30],
ResNet [26, 45] and CNNs [32, 33, 38].
Approximation power of CNNs 2.1 Convolutional neural networks
Convolutional networks are a special type of structured sparse feedforward neural network (FNN) that use convolution in place of general matrix multiplication in at least one of their layers [21].
There are different formulations of CNNs in the literature [2, 66, 47, 37, 63, 64, 33, 32, 38].
In this paper, we consider ReLU activated downsampled CNNs with bias vectors in the convolutional layers deﬁned in [63].
We do not require the norm of network parameters (weight and bias) to be uniformly bounded.
We consider a general CNN function f CNN :
X → R , where X ⊂ R d .
For simplicity, we take
X = (0 , 1) d .
Suppose f CNN has L number of hidden layers, then it can be expressed as
[Equation]
where ◦ denotes the functional composition.
The A i ’s are either convolutional operators or downsampling operators.
For convolutional layers, A i ( x ) =
σ ( W ci x + b ci ) , where W ci ∈ R d i × d i − 1 is the structured sparse Toeplitz type weight matrix induced by the convolutional ﬁlter { w ( i ) j } s ( i ) j =0 with ﬁlter length and s ( i ) ∈ N + , b
ci ∈ R d i is a bias vector, and σ is the rectiﬁed linear unit (ReLU) activation function applying to each component of the input vector.
For downsampling layers, for any x ∈ R d i − 1 , A i ( x ) =
D i ( x ) =
( x jm i ) (cid:98) d i − 1 /m
i (cid:99) j
= , where D i :
R d i × d i − 1 is the downsampling operator with scaling parameter m i ≤
d i − 1 in the i -th layer.
We introduce the convolutional operation and downsampling operation [63] in details in Appendix B.
We focus on the function class consisting of deep CNNs, denoted by F CNN which is deﬁned as
F CNN =
{ f CNN deﬁned in (1) over all possible choice of A i , i = 1 , . . .
, L + 1 } .
For a general function class F and any f ∈ F , deﬁne the (cid:107) · (cid:107) ∞ metric on F by sup x ∈X (cid:107) f
( x ) (cid:107) ∞ .
Then let B := sup f ∈F CNN (cid:107) f (cid:107) ∞ denote the bound of the functions in F CNN .
Let L be the number of hidden layers and S be the total number of parameters for networks in F CNN and let s min and s max be the minimum and maximum ﬁlter length over convolutional layers respectively.
2.2 Approximation in Sobolev space
We use the Sobolev and Hölder classes for the target functions in our approximation results since they are well-established formulations for describing smooth functions in the literature.
Indeed, earlier works on approximation theory of neural networks have been developed for Sobolev functions [59, 64, 24] and for Hölder functions [33, 51].
Let N 0 be the set of non-negative integers and β ∈ N 0 .
The Sobolev class of functions W β,p ( X ) with X = (0 , 1) d is deﬁned as
[Equation]
where≤ p ≤ ∞ , D α = ∂ α 1 · · · ∂ α d with α =
( α 1 , . .
. , α d ) (cid:62) ∈
N d 0 , and (cid:107) α (cid:107)= (cid:80) di =1 α i .
For f ∈
W β,p ( X ) and≤ m ≤ β , we deﬁne the norm (cid:107) f (cid:107) W m,p ( X ) :
{+= (cid:0) (cid:80) 
+}
≤(cid:107) α (cid:107) 1 ≤ m (cid:107) D α f (cid:107) pL p ( X ) (cid:1) 1 /p for 1 ≤ p < ∞ , and deﬁne (cid:107) f (cid:107) W m, ∞ ( X ) := max 0 ≤(cid:107)
α (cid:107) 1 ≤ m (cid:107) D α f (cid:107) L ∞ ( X ) .
In the remainder of the paper, for any positive integers M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with depth L , size S and ﬁlter length speciﬁed as follows: [Equation] [Equation]
[Equation]
(6) where
[Equation]
(7) denotes the maximum incremental width (number of neurons) for consecutive layers in the network and (cid:100) a (cid:101) denotes the smallest integer no less than a .
Theorem 2.1.
Assume that f ∈ W β,p ( X ) with 1 ≤ β ∈ N 0 , 1 ≤ p ≤ ∞ and (cid:107) f (cid:107) W β,p ( X ) ≤ B 0 for
B 0 > 0 .
For any M, N ∈ N + , there exists a function f CNN ∈ F CNN deﬁned in (2) with depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6), such that for m = 0 , 1 ,
[Equation]
where C 0 ( d, β, p ) = 37 · 2 2 β +2 d/p B 20 ( β + 1) 3 × { π − d/ 2 Γ( d/ 2 + 1) } 2 /p
+
(1 + 2 √ d ) d d 4 β .
Here Γ( · ) is the gamma function.
Our result is new for approximating Sobolev-regular functions in W β,p ( X ) for a positive integer β ∈ N + with respect to the Sobolev norms (cid:107) · (cid:107) W m,p ( X ) for m = 0 , 1 in terms of explicitly deﬁned approximation error prefactor, clearly deﬁned and ﬂexible network parameters, as well as an fast approximation rate comparable with [22].
A toy example is provided in Appendix D to illustrate the approximation power of CNNs and examine how the approximation error varies according to the ﬁlter size and depth of the networks.
In [24], an approximation result on β -smooth functions by networks with squared ReLU activation function is also provided under the norm (cid:107) · (cid:107) W m,p ( X ) for positive integer m < β .
In [22], the approximation rates of ReLU DNN for Sobolev-regular functions with respect to the weaker Sobolev norms (cid:107) · (cid:107) W s,p ( X ) for s ∈ [0 , 1] were analyzed, which can be seen as a generalization of the result in [59].
Approximation results on Sobolev-regular functions W β,p ( X ) by networks with general activation functions are derived in [23], where the error is with respect to (cid:107) · (cid:107) W
m,p ( X ) for some integer m < β .
By the general Sobolev inequality, see Theorem 6 in Chapter 5 of [15], W k,p ( X ) with kp > d can be embedded into a Hölder class
H β ( X , B 0 ) (with β = k − (cid:98) d/p (cid:99) − 1 ) deﬁned as
[Equation]
where β > 0 and (cid:98)
β (cid:99) ∈ N 0 denotes the largest integer strictly smaller than β .
Because of the extra regularity on the Hölder class, a much improved prefactor C 0 in the error bound of Theorem 2.1 can be obtained.
Theorem 2.2.
Let f ∈ H β
( X , B 0 ) be deﬁned in (8) and let X ∈ R
d be a random vector whose probability distribution is supported on X = (0 , 1) d and absolutely continuous with respect to the
Lebesgue measure.
For any M, N ∈ N + , there exists a function f CNN ∈ F CNN deﬁned in (2) with depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6), such that
[Equation]
where C 0 ( d, β ) = 18 B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 .
Here a ∨ b := max { a, b } for a, b ∈ R .
The convergence rate ( NM ) − 2 β/d in Theorems 2.1 and 2.2 with respect to the network depth and ﬁlter lengths speciﬁed by M and N , is in line with the nearly optimal rate of ReLU FNNs on smooth functions under Sobolev norm in [24] and on Hölder smooth functions under L ∞ norm in [28].
The prefactor 18 B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 of approximation error depends on the dimension d polynomially, different from the exponential dependence in many existing neural network approximation results mentioned in Section 1.1.
2.3 Approximation with a lower-dimensional support
In many modern machine learning problems, the ambient dimension d of the input data could be very large, which results in an extremely slow convergence rate.
This fact is known as the curse of dimensionality .
Fortunately, many types of data have a low-dimensional latent structure, that is, although the ambient dimension d is large, the distribution of the data is approximately supported on a low-dimensional subset of R d , in which case the approximation error bound can be substantially improved.
We establish an approximation result for CNNs in Sobolev spaces under a low-dimensional support assumption.
Assumption 2.3.
The distribution of X is supported on M ρ , a ρ -neighborhood of M ⊂ X , where
M is a compact d M -dimensional Riemannian submanifold and M ρ = { x ∈ X : inf {(cid:107) x − y (cid:107) 2 : y ∈ M} ≤ ρ } for ρ ∈ (0 , 1) .
In real-world applications, data are hardly observed to locate on an exact manifold , instead they could be more realistically viewed as consisting of a latent part supported on a low-dimensional manifold
M plus noises.
Therefore, Assumption 2.3 is more realistic than the exact manifold assumption [50, 12, 38].
Deﬁne
[Equation]
In our error bound results below, it is the d ε that will affect the convergence rate.
Since it is often the case that d M (cid:28) d and therefore d ε (cid:28) d, the manifold assumption will lead to a better convergence rate than those in Theorems 2.1 and 2.2.
In addition, we require ρ ≤ ρ ε .
Theorem 2.4.
Suppose that Assumption 2.3 holds, f ∈ H β
( X , B 0 ) and the distribution of X is absolutely continuous with respect to the Lebesgue measure.
Let d ε and ρ ε be deﬁned in (9) and (10), respectively.
Assume that ρ ≤ ρ ε .
Then, for any M, N ∈ N + , there exists a CNN function f CNN ∈ F CNN with depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6) with
W = 38 2 ( (cid:98) β
(cid:99) + 1)
d 2 (cid:98) β (cid:99) +
ε
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 such that
[Equation]
where C ( d, β ) = (18 + C 2 ) B 0 (1 − ε ) − β ( β + 1) 2 d 1 / 2 d 3 β/ 2 ε .
Since the intrinsic dimension d ε can be much smaller than the ambient dimension d , the rate ( NM ) − 2 β/d ε in Theorem 2.4 is greatly improved compared with the rate ( NM ) − 2 β/d in Theorem 2.2 and the curse of dimensionality is mitigated.
The result here is of independent interest and can be useful in other problems that involve the use of CNNs.
Excess risk in classiﬁcation
In this section, we present the application of the approximation results Theorems 2.1 and 2.4 to the error analysis of classiﬁcation with CNNs.
Consider a binary classiﬁcation problem with a predictor X ∈ X ⊂ R d and its binary label
Y ∈ { 1 , − 1 } .
We are interested in learning a classiﬁer h :
X → { 1 , − 1 } from a class of functions, or a hypothesis space, denoted by H .
Let the joint distribution of ( X, Y ) be P .
The goal is to ﬁnd a classiﬁer that minimizes the misclassiﬁcation error or the 0-1 risk: R ∗ ( h ) =
P (cid:8) h ( X ) (cid:54) =
Y (cid:9) , h ∈ H .
Denote the misclassiﬁcation risk minimizer at the population level by h 0 = argmin
h measurable R ∗ ( h ) .
For any h ∈ H , the excess risk of h is R ∗ ( h ) − R ∗ ( h 0 ) , the difference between the misclassiﬁcation errors of h and h 0 .
Since the probability measure P is unknown in practice, the classiﬁer h will be learned based on a random sample S =
{ ( X i , Y i ) } ni =1 from P , where n is the sample size.
The empirical risk minimizer (ERM) is deﬁned by
[Equation]
[Equation] However, the empirical risk function based on the 0 - 1 loss is non-continuous and non-convex, thus this minimization problem is typically computationally intractable.
Rather than minimizing the non-smooth 0-1 loss, many popular methods adopt a proper convex loss function to train classiﬁers with computational efﬁciency that can be done in polynomial time.
Moreover, proper surrogate convex loss functions have been shown to be consistent with the 0-1 loss function for binary classiﬁcation problems [61, 5].
3.1 Convex surrogate loss functions
Let φ be a given convex univariate loss function φ : R → [0 , ∞ ) .
We consider the risk function with respect to the loss φ
[Equation] [Equation]
where for simplicity of notation and without causing confusion we omit the superscript φ in R , similarly for ˆ f n and f 0 deﬁned below.
For a given random sample S = {
( X i , Y i ) } ni =1 , we denote the ERM over F CNN with a given loss φ by [Equation]
[Equation]
Based on the ERM ˆ f n , a classiﬁer ˆ h n ( x ) :
= sign( ˆ f n ( x )) for x ∈ X can be deﬁned.
As shown in [61, 5], for a properly chosen φ , ˆ f n can help reduce the excess risk R (ˆ h n ) − R 0 .
Speciﬁcally, deﬁne the measurable minimizer of R in (12) as
[Equation]
and the corresponding minimal φ -risk as R 0 = R ( f 0 ) .
Then for a proper φ , we have ψ ( R (ˆ h n ) −
R 0 ) ≤ R ( ˆ f n ) − R 0 , where ψ : [ − 1 , 1] → [0 , ∞ ) is a nonnegative continuous function, invertible on [0 , 1] , and achieves its minimum at 0 with ψ (0)
= 0 .
A variety of classiﬁcation methods are based on this tactic.
Generally, classiﬁcation-calibrated φ considered in this paper is non-increasing and convex; details are given in the Supplementary Materials.
Deﬁne the conditional probability
[Equation]
Let the conditional φ -risk of f given X = x be denoted by R f ( x ) :
= E { φ ( Y f ( X ))
| X = x } .
SVM and cross entropy, two important examples of φ , and the corresponding f 0 , R ( f 0 ) , ψ and its inverse on [0 , 1] in Table 2 in the appendix.
For the form of ψ , Theorem 34 in [11] shows that if φ is convex, φ (cid:48)(cid:48) (0) > 0 exists and φ (cid:48) < 0 , then ψ ( u ) =
u 2 .
[Table]
φ ( a ) max { 1 − a, 0 } − log { 0 . 5 + a } f 0 ( x ) sign(2 η − 1) η − 0 . R f 0 ( x ) 1 − | 2 η − 1 | − η log( η ) − (1 − η ) log(1 − η ) ψ ( θ ) | θ | θφ B B +− log
[Equation] [Equation]
Note: The conditional probability η ( x ) deﬁned in (15) is written as η for notational simplicity. "
**" stands for "does not exist"
Non-asymptotic error bounds
For the ERM ˆ f n deﬁned in (13), we ﬁrst state a basic inequality for bounding the excess risk of ˆ f n .
Lemma 4.1.
For any loss φ and any random sample S = {
( X i , Y i ) } ni =1 , the excess φ -risk of the
ERM ˆ f n satisﬁes R ( ˆ f n ) − R ( f [Equation] ≤ StoErr + AppErr , where
[Equation]
Therefore, the excess risk of ˆ f n is bounded by the sum of two terms: the stochastic error and the approximation error.
For a given loss function φ , the upper bound no longer depends on the ERM ˆ f n , but the function class F and the random sample S .
The stochastic error in (16) depends on the complexity of F and the approximation error in (16) depends on the approximation power of the class F for f 0 .
4.1 Stochastic error
We bound the stochastic error in terms of the pseudo-dimension of the downsampled CNNs deﬁned in (2); particularly, we further bound its pseudo-dimension and express the bound in terms of quantities related to CNNs.
Theorem 4.2.
[Stochastic error bound] Suppose that φ is convex and non-increasing.
For any
M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with B ≥ 0 , depth L and size S and let φ B := sup | a |≤B φ ( a ) .
Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ , the stochastic error in (16) satisﬁes
[Equation]
where C 0 > 0 is a universal constant.
The quantity φ B for common loss functions are presented in Table 2.
It worth noting that the error bound here does not require the norm of the CNN parameters (of weight and bias) to be uniformly bounded.
In comparison, those stochastic error bounds based on the covering number generally assume an uniformly bounded norm on CNN parameters, which may hinder the approximation power of network since most approximation results need the norm of network parameters to tend to inﬁnity as the approximation error tends to zero.
4.2 Approximation error
We derive an upper bound of the approximation error by relating inf f ∈F CNN R ( f ) − R ( f 0 ) to inf f ∈F CNN (cid:107) f − f 0 (cid:107) under proper conditions on φ and η , where R is deﬁned in (12).
The target function f 0 may be non-smooth or unbounded, which poses extra difﬁculty.
Most of the existing studies on the approximation properties of networks assume smooth and bounded target functions.
As shown in Table 2, the target function f 0 for SVM is non-continuous; for the cross entropy loss, the loss is unbounded on [ − 0 .
For logistic loss and exponential loss, we have f 0 ( x ) ∈ { + ∞ , −∞} once η ( x ) ∈ { 0 , 1 } .
We overcome these difﬁculties by imposing mild conditions on the conditional probability η and the loss function φ , and use the truncation technique to analyze the approximation.
Assumption 4.3.
(a)
The conditional probability η ( x ) =
P { Y = 1 | X = x } is continuous on the support of X and the probability measure of X is absolutely continuous with respect to the Lebesgue measure.
(b) The loss function φ : R → [0 , ∞ ) is classiﬁcation-calibrated, convex, non-increasing and continuously differentiable on its support.
Assumption 4.3 (a) and (b) are regular conditions ensuring that the target function f 0 is continuous and the loss φ is Lipschitz on bounded intervals.
The absolute continuity assumption of the probability measure of X with respect to the Lebesgue measure is reasonable for the approximate low-dimensional manifold case but incompatible with the exact low-dimensional manifold condition.
This assumption is for deriving better L p approximation error bound, without which the error bound in term of the L ∞ norm can still be established, but at the price of a wider neural network and a larger prefactor.
It can be veriﬁed that commonly used loss functions, such as the cross entropy, the logistic and the exponential loss functions, satisfy Assumption 4.3 (b).
To deal with the approximation of unbounded target function, we truncate the target f 0 by a constant
T , where T may depend on n .
Let f 0 ,T be the truncated version of f 0 deﬁned as
[Equation]
Denote the error of the loss function φ due to truncation by
[Equation]
where Ran ( f 0 ) is the range of the target function f 0 .
This error decreases as T increases.
Then, the approximation error can be decomposed into two terms that are easier to deal with.
Lemma 4.4.
[ φ -approximation error] Suppose that Assumption 4.3 holds and T ≤ B .
Then, the φ -approximation error inf f ∈F CNN R ( f ) − R ( f 0 ) with respect to the loss function φ satisﬁes
[Equation]
where B φ is deﬁned as the Lipschitz constant of φ on the interval [ −B , B ] .
We list B φ and ∆ φ ( T ) for hinge and cross entropy loss functions in Table 2.
A detailed table including other loss functions are given in the appendix.
Theorem 4.5.
[Approximation error bound] Suppose that Assumption 4.3 holds and f 0 ∈
H β ([0 , 1] d , B 0 ) .
For any M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with T ≤ B , depth L and ﬁlter lengths speciﬁed in (4) and (5).
Then, the approximation error deﬁned in (16) satisﬁes AppErr ≤ C ( d, β )( NM ) − 2 β/d + ∆ φ ( T ) , where C ( d, β ) =
18 B φ B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 .
There is a trade-off since T ≤ B and ∆ φ ( T ) deceases in T but C ( d, β ) increases in B φ (or B ).
4.3 Non-asymptotic excess risk bound
Combining Theorems 4.2 and 4.5, we obtain the excess error bound.
Theorem 4.6.
[Non-asymptotic excess φ -risk bound] Suppose that Assumption 4.3 holds and f 0 ∈
H β ([0 , 1] d , B 0 ) .
For any M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with T ≤ B , depth L and ﬁlter lengths speciﬁed in (4) and (5).
Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ , the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
with
[Equation]
where C ( d, β ) =
18 B φ B 0 ( β + 1) 2 d β +( β ∨ 1) / 2 and C 0 > 0 is a universal constant.
The upper bound of the excess φ -risk R ( ˆ f n ) − R ( f 0 ) in Theorem 4.6 is a sum of two error terms, the stochastic error and the approximation error.
To achieve the optimal error rate, we need to balance the trade-off between them.
On one hand, the bound for the stochastic error StoError increases with the size and the depth of F CNN .
On the other hand, the bound for the approximation error AppError decreases with the depth and the upper bound of ﬁlter lengths of F CNN .
4.4 Circumventing the curse of dimensionality
We state a theorem that provides a non-asymptotic excess risk bound under the approximate lowdimensional manifold assumption.
Theorem 4.7 (Circumventing the curse of dimensionality) .
Suppose that Assumptions 2.3, 4. hold, and f 0 ∈
H β ([0 , 1] d , B 0 ) .
For any M, N ∈ N + , let F CNN be the class of CNNs deﬁned in (2) with T ≤ B , depth L and ﬁlter lengths and size S speciﬁed in (4), (5) and (6) with W = 38 2 ( β + 1) 4 d 2 β +2 ε
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 .
Suppose ρ in Assumption 2.3 satisﬁes ρ ≤ ρ ε , , where ρ ε is deﬁned in (10).
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ , the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
[Equation] with
StoErr
[Equation]
where C 1 , C 2 > 0 are universal constants.
For a high-dimensional input X with a large d , d ε satisﬁes d M ≤ d ε < d for ε ∈ (0 , 1) .
For the stochastic error StoErr ∗ , comparing with the bound (18) in Theorem 4.6, we see that StoErr ∗ =
StoErr , that is, the stochastic error does not change under the approximate low-dimensional manifold assumption.
For the approximation error AppErr ∗ , we see that the convergence rate ( NM ) − 2 β/d ε only depends on d ε , which leads to a much faster convergence rate.
Examples
In this section, we illustrate the applications of Theorems 4.6 and 4.7 to obtaining non-asymptotic error bounds for the excess risk in classiﬁcation.
We apply the general excess risk bounds established in these theorems to several important classiﬁcation methods with CNNs when a speciﬁc form of φ is given.
We present the results for the cross entropy loss below.
The non-asymptotic error bounds for the hinge, the logistic, the exponential and the least squares losses are given in Appendix C.
For the cross entropy loss φ ( a ) =
− log(0 .
5 + a ) , the Lipschitz constant B φ on [ − 0 . 5 , 0 . 5] is not bounded.
As the minimizer of φ -risk η − 0 .
5 is bounded, we can choose T = B
= 0 .
5 − n β/ (2 d +4 β ) .
Then ∆ φ ( T ) ≤ n − β/ (2 d +4 β ) and B φ = n β/ (2 d +4 β ) .
Denote ξ n = n d/ (2 d +4 β ) .
Take N = 1 in (7) and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 (cid:98) ξ n (cid:99)(cid:100) log 2 (8 (cid:98) ξ n (cid:99) ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 [-
-]2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 {+∗ 38 
+}
( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99)
+4 (cid:98) ξ n (cid:99)(cid:100) log 2 (8 (cid:98) ξ n (cid:99) ) (cid:101) /
( s min − 1) .
Theorem 4.6 implies that, with probability at least 1 − exp( − ξ 2 n ) , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C =
O
( B 0 ( β + 1) 8 d 3 β +3 / ( s min − 1)) is a constant independent of n .
Let T =
B
= 0 .
5 − (log n ) − 1 .
Using a modiﬁed cross entropy loss φ ( a ) = max {− log(0 .
5 + a ) , τ } with τ = − log(1 − (log n ) − 1 ) , the excess φ -risk of the ERM ˆ f n can be improved to
O ( n − β/ ( d +2 β ) (log n ) 2 ) .
If the approximate low-dimensional manifold Assumption 2.3 holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in Assumption 2.3 satisﬁes ρ ≤ ρ ε , where ρ ε is deﬁned in (10), then Theorem 4.7 implies that the rate of convergence can be improved:
[Equation]
where C is a constant independent of n .
The above discussion shows that deep binary classiﬁcations with CNNs are adaptive to the low dimensional structure of the data and the smoothness of the target.
Moreover, the prefactor C depends on the ambient dimension d polynomially, which improves the prefactors depending on d exponentially in the existing excess risk bounds [46, 25, 30, 26, 45, 32, 33, 38].
Moreover, the ﬁlter lengths here are more ﬂexible comparing with those requiring certain ﬁlter lengths in [32, 33, 48].
Conclusion
In this work, we derive new approximation error bounds with explicit prefactor in terms of network parameters for Sobolev-regular functions and Hölder smooth functions using deep convolutional networks.
New approximation result when the target function is supported on an approximate lower-dimensional manifold is established.
Different from existing results, the prefactor of our error bound depends on the ambient dimension polynomially instead of exponentially for Hölder smooth functions.
The new approximation results are applied to establish non-asymptotic excess risk bounds for a class of classiﬁcation methods using CNNs.
An important limitation of our work is that we only considered binary classiﬁcation problems in applying our approximation results using deep convolutional networks.
It would be interesting to apply the results in this work to other settings involving CNN approximation, such as the multiclass classiﬁcation and regression problems.
In addition, our work only partially explains the empirical successes of CNNs in practice.
For image data, the assumption of approximate lower-dimensional support does not capture all the structural information.
For example, spatial invariance is likely to be expected in some problems such as image classiﬁcation.
We believe that if such properties are taken into account, the theoretical bounds can be further improved.
We hope to study this in the future.
Acknowledgements
The authors wish to thank three anonymous reviewers for their comments and suggestions that helped improve the paper signiﬁcantly.
Y. Jiao is supported in part by the National Science Foundation of China under Grantand the research fund of KLATASDSMOE. Y. Lin is supported by the Hong Kong Research Grants Council (Grant Nos. 14306219 and 14306620), the National Natural Science Foundation of China (Grant No. 11961028) and Direct Grants for Research, The Chinese University of Hong Kong. J.
Huang is partially supported by the research grant P0042888 from The Hong Kong Polytechnic
University. Appendix In the appendix, we include detailed descriptions on convex surrogate losses,convolutional neural networks, non-asymptotic error bounds for commonly used loss functions, and prove Theorems 2.1,2.2, 2.4, 4.4, 4.6, and 4.7, as well as the error bounds for cross entropy, SVM, logistic, exponential and least squares loss examples.
A toy example on the numerical performance of CNN approximation is presented in Appendix D.
A Convex surrogate loss
We next give a brief review of the convex surrogate loss functions and discuss in details on the connection between the excess risk with respect to the φ -loss and that of 0-1 loss [61, 5].
Let φ be a given convex univariate function φ : R → [0 , ∞ ) .
Instead of minimizing the excess risk R over H , we consider minimizing the risk with respect to the loss φ ( φ -risk)
[Equation]
over a certain class of functions F , where φ : R → [0 , ∞ ) is some generic loss function.
For the special case when H = { h : h ( x ) = sign( f ( x )) , f ∈ F} and φ ( · ) is a step function, i.e., φ ( x ) =
if x < 0 and φ ( x ) = [-
-]0 if x ≥ 0 , then to minimize R over H , it sufﬁces to minimize R over F .
Given
S = { ( X i , Y i ) } ni =1 , let R n ( f ) :
= (cid:80) ni
=1 φ ( Y i f
( X i ))
/n be the empirical risk of f w.r.t the φ -loss, and deﬁne the empirical risk estimator (ERM) as
[Equation]
Based on the ERM ˆ f n , a classiﬁcation rule or a classiﬁer ˆ h n ( x ) :
= sign( ˆ f n ( x )) for x ∈ X can be induced which aims at minimizing the 0-1 risk.
As shown in [61] and [5], for a properly chosen φ , ˆ f n can indeed help reduce the excess risk R (ˆ h n ) − R 0 .
More precisely, let R 0 :=
inf f measurable R ( f ) , then for a proper φ , we have
[Equation]
where ψ : [ − 1 , 1] → [0 , ∞ ) is a nonnegative continuous function, invertible on [0 , 1] , and achieves its minimum at 0 with ψ (0)
= 0 .
A wide variety of popular classiﬁcation methods are based on this tactic.
For instance, when φ ( a ) = log { 1 + exp( − a ) } , it is the logistic regression [20]; when φ ( a ) = max { 1 − a, 0 } , it becomes the SVM [14];
when φ ( a ) = exp( − a ) is the exponential loss, it is the AdaBoost algorithm [19]; when φ ( a ) =
(1 − a ) 2 , it is the least squares method for classiﬁcation, and so on.
Deﬁne the measurable minimizer of R as
[Equation]
and the corresponding minimal φ -risk as R 0 = R ( f 0 ) .
Clearly, the optimal f 0 depends on the loss function φ .
If φ is not properly chosen, the resulting f 0 can be poor and thus the classiﬁer h 0 ( x ) = sign( f 0 ( x ))
based on f 0 can be invalid.
To study the basic conditions imposed on φ , so as to produce a valid classiﬁer towards minimizing the 0-1 loss, we need a thorough understanding of the risk R .
Let the conditional φ -risk of f given X = x be denoted by R x ( f ) :
= E { φ ( Y f ( X ))
| X = x } .
Recall that η ( x ) =
P { Y = 1 | X = x } as deﬁned in (15).
We have
[Equation]
} φ ( − f ( x )) , x ∈ X .
For a good classiﬁer, if η ( x ) > 1 / 2 , it is naturally expected that f 0 ( x ) > 0 and h 0 ( x ) = sign( f 0 ( x ))
=
1 (correct sign); thus to encourage f 0 ( x ) > 0 , we should at least require φ ( f 0 ( x )) <
φ ( − f 0 ( x )) to minimize the conditional risk R x .
Similarly, if η ( x ) < 1 / 2 , we expect that the contrary happens.
A rigorous deﬁnition for such an ideal φ is given in Deﬁnition 1 of [5].
Deﬁnition A.1 (Classiﬁcation-calibrated) .
For η ∈ [0 , 1] and a ∈ R , deﬁne H ( η, a )
= ηφ ( a ) + (1 − η ) φ ( − a ) .
Then, we say that φ is classiﬁcation-calibrated if for any η (cid:54) = 1 / 2 ,
[Equation]
With a classiﬁcation-calibrated loss φ , it is guaranteed that the “incorrect" label sign( f ( X )) in the sense that it is inconsistent with the Bayes estimator sign(2 η ( X ) − 1) ) results in a strictly larger loss under φ .
It was shown by [5] that, the surrogate loss function φ is able to produce the optimal Bayes classiﬁer.
For ease of reference, we state this result as Lemma A.2 below.
Lemma A.2 (Theorem 1 of [5]) .
For any nonnegative loss function φ : R → [0 , ∞ ) , any measurable f : X → R , its induced classiﬁer
h f = sign( f ) :
X → {± 1 } , and any probability measure P of ( X, Y ) on X × {± 1 } ,
[Equation]
where ψ : [ − 1 , 1] → [0 , ∞ ) is the Fenchel-Legendre biconjugate of ˜ ψ : [ − 1 , 1] → R , and
[Equation]
Besides, if φ is classiﬁcation-calibrated, then for any sequence { a m } in [0 , 1] , ψ ( a m ) → 0 , if and only if a m → 0 .
Lemma A.2 has some important implications.
First, for any nonnegative φ , ψ is simply the functional convex hull (the greatest convex minorant) of ˜ ψ .
Both ψ and ˜ ψ are continuous on [ − 1 , 1] , and ψ is nonnegative that attains its minimum at 0 with ψ (0)
= 0 .
Second, if the loss φ is nonnegative and classiﬁcation-calibrated, then ψ ( θ ) > 0 for all θ ∈ (0 , 1] and ψ is invertible on [0 , 1] .
In this case, we have R ( h f ) − R 0 ≤ ψ − 1 ( R ( f ) − R 0 ) .
Third, if φ is nonnegative convex and classiﬁcation-calibrated, then ψ ( θ ) = φ (0) − inf a ∈ R H ((1 + θ ) / 2 , a ) , which gives an easy way to compute the function ψ .
Note that for the formulation of ψ , Theorem 34 in [11] shows that if φ is convex, φ (cid:48)(cid:48) (0) > 0 exists and φ (cid:48) < 0 , then ψ ( θ ) = θ 2 .
Next, we present several examples of φ , and the corresponding f 0 , R ( f 0 ) , ψ and its inverse on [0 , 1] in Table 3.
[Table]
We also list the Lipschitz constant B φ and ∆ φ ( T ) for commonly-used SVM, cross entropy, least squares, exponential and logistic loss functions in Table 4. , B ] for 1 ≤ T ≤ B ; and cross entropy loss function restricted to [ −B , B ] for T ≤ B ≤ 0 .
[Table]
B Convolutional neural networks
As indicated by their name, CNNs employ a mathematical operation called convolution.
Convolutional networks are a specialized type of structured sparse feedforward neural network (FNN) that use convolution in place of general matrix multiplication in at least one of their layers [21].
There are different formulations of CNNs in the literature [2, 66, 47, 37, 63, 64].
For a general neural network, let L denote the number of layers and ( σ 1 , . . .
, σ L ) denote the activation functions, such as the rectiﬁed linear unit (ReLU) and max pooling function.
Besides, let d i denote the width (the number of neurons or computational units) of the i -th layer, i = 1 , . . .
, L + 1 , and
W = max { d 1 , . . .
, d L } denote the maximum width among layers.
In our present classiﬁcation problem, d 0 = d (the dimension of the input X ) and d L +1 = 1 (the dimension of the output Y ).
In the following, two common types of neural networks including FNNs or MLPs and CNNs with downsampling [63] are shown.
• The architecture of a MLP f MLP : R d → R can be expressed as a composition of a series of linear transformations: for any x ∈ R d ,
[Equation]
where W i ∈ R d i +1 ×
d i is a fully connected weight matrix, and b i ∈ R
d i +1 is a bias vector in the i -th linear transformation.
Usually, the activation functions ( σ 1 , . . .
, σ L ) are the ReLU activation, where ReLU ( x ) = max { 0 , x } for any x ∈ R (deﬁned for each component of x if x is a vector).
• The formulation of a downsampled CNN f CNN : R d → R is essentially a specially structured architectures of MLP, and here we use the formulation deﬁned in [66, 63, 64]
[Equation]
where ◦ denotes the functional composition and A i is a linear operation, i = 1 , . .
, L + 1 .
The A i ’s are either convolutional operators or downsampling operators.
For convolutional layers,
[Equation]
where σ is the rectiﬁed linear unit (ReLU) activation function applying to each component of the input vector, W ci ∈ R d i × d i − 1 is the structured sparse Toeplitz type weight matrix induced by the convolutional ﬁlter { w ( i ) j } s ( i )
j =0 with ﬁlter length s ( i ) ∈
N involving only s ( i ) +1 free parameters.
The corresponding ( d i −
+ s ( i ) )
× d i − 1 Toeplitz type weight matrix
W ci is given explicitly by
W ci =     w ( i )   · · · · · · w ( i )w ( i )  · · · · · · ... w ( i ) s
( i ) w ( i ) s
( i ) −· · · w ( i ) · · ·  w ( i ) s ( i ) · · · w ( i )w ( i )0 · · · .. .
· · · w ( i ) s ( i ) · · · w ( i )w ( i ) · · ·  w ( i ) s ( i ) · · · w ( i )... · · · w ( i ) s ( i ) 
   ∈ R ( d i − 1 + s ( i ) )
× d i − 1 .
It can be seen that by convolutional operations here, the network has linearly increasing widths d i =
d i −
+ s ( i ) for convolutional layers.
On the contrary, the downsampling layers decrease the width of the networks.
For downsampling layers,
[Equation]
where D i : R d i − 1 → R [ d i − 1 /m i ] is the downsampling operator with a scaling parameter m i ≤
d i − 1 and W D ∈ R
d
i − 1 ×
[ d i − 1 /m i ] is corresponding structured sparse weight matrix induced by D i in the i -th layer.
Here the downsampling operator [63] is given by
[Equation]
which takes a real vector x ∈ R d i − 1 as input and outputs a subvector of x with length
R [ d i − 1 /m i ] where [ a ] denotes the integer part of a ∈ R .
It can be seen that by downsampling operation D i here, the network width is scaled down by about m i times.
For the function class of the downsampled CNNs formulated as in (23), we let L be th number of hidden layers and S be the total number of parameters for networks in F CNN and let W be the maximum ﬁlter length of convolutional layers.
A downsampled CNN f CNN is called uniform if the length of its ﬁlters { s ( i ) } in the convolutional layers are the same.
[63] showed that any multilayer perceptron can be represented by a uniform downsampled CNN with parameters no more than 8 times as that of the MLP.
C Non-asymptotic error bounds for commonly used loss functions
In this section, we show more examples of the non-asymptotic error bounds for commonly used loss functions including the hinge loss, the logistic loss, the exponential loss and the least square loss.
C.1 SVM: the hinge loss
For SVM, the loss function φ ( a ) =
max { 1 − a, 0 } is not differentiable at a = 1 and thus the φ -risk minimizer f 0 ( x ) =
sign(2 η ( x ) − 1) may not be continuous even though η is continuous, as f 0 ( x ) is discontinuous when η ( x ) = 1 / 2 .
To tackle this problem, we additionally impose the low noise condition on η [41, 55], i.e., there exist c noise > 0 and q ∈ [0 , ∞ ] such that for any t > 0 , P ( | 2 η ( X ) − 1 | ≤ t ) ≤ c noise t q , where the constant q is called the noise exponent .
Suppose that Assumptions 4.3 holds f 0 ∈ H β ([0 , 1] d , B 0 ) .
Denote ζ n = n d/ { 2 d +4 β ( q +1) } .
Let F CNN be the class of CNNs deﬁned in (2) with B ≥ 1 , depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1)
6 d 2 (cid:98) β (cid:99) +2 (cid:98) ζ n (cid:99)(cid:100) log(8 ζ n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) +1) 4 d 2 (cid:98) β (cid:99) + and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38 4 ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) ζ n (cid:99)(cid:100) log(8 ζ n ) (cid:101) /
( s min − 1) .
Theorem 4.6 implies that, with probability at least 1 − exp( − ζ 2 n ) , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C =
O ( c noise 4 q B q
+ ( β + 1) 2 q +8 d ( q +1)(3 β +3) ) .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
The excess risk bound depends on q , the noise exponent.
When q = 0 (high noise), the convergence rate is n − β/
( d +2 β ) ; when q = + ∞ (no noise), the rate will be signiﬁcantly improved to n − 1 / 2 .
A similar result can be found in Theorem 3.3 of [30].
Suppose Assumption 2.3 holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in this assumption satisﬁes ρ ≤ ρ ε , then Theorem 4.7 implies that
[Equation]
where C is a constant independent of n .
C.2 The logistic loss
For the logistic loss function φ ( a ) = log { 1+exp( − a ) } , the minimizer of φ -risk log { η ( x ) / 1 − η ( x ) } is unbounded with the Lipschitz constant B φ = 1 / { exp( −B ) + 1 } ≤ 1 on [ −B , B ] , and φ B = log(1 + exp( B )) ≤ B + ∞ and the truncation error ∆ φ ( T )
=
log { 1 + exp( − T ) } .
A feasible choice is T = B = log( n ) where n is the sample size, in which case ∆ φ ( T ) = log { 1 + 1 /n } ≤ 1 /n .
Suppose Assumptions 4.3 (in main context) holds and f 0 ∈ H β ([0 , 1] d , B 0 ) .
Take N = 1 in (7) and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) {++ 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38 
+}
( (cid:98) β (cid:99) +1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) .
Theorem 4.6 implies that, with probability at least 1 − exp {− n d/
( d +2 β ) } , the excess φ -risk of the
ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
, or simply
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) +1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
In addition, suppose the approximate low-dimensional manifold Assumption 2.3 also holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in Assumption 2.3 satisﬁes ρ ≤
C [Equation] NM )
[Equation] [Equation]
[Equation]
d/d ε +1 − ε ) − 1 (1 − ε ) 1 − β , where d ε = O ( d M log( d/ε ) /ε 2 ) is an integer satisfying d M ≤ d ε (cid:28) d.
Then Theorem 4.7 shows that the rate of convergence can be improved to
[Equation]
where C ( d, d ε , ε, β, B 0 , s min )
=
O ( B 0 (1 − ε ) − β
( (cid:98) β (cid:99) + 1) 8 d 1 / 2 d 3 (cid:98) β (cid:99) +3 ε / ( s min − 1)) is a constant independent of n .
C. The exponential loss
For the exponential loss function φ ( a ) = exp( − a ) , the minimizer of φ -risk log { η ( x ) / 1 − η ( x ) } /is unbounded, the truncation error ∆ φ ( T ) =
exp( − T ) , the Lipschitz constant B φ = exp( B ) and φ B = exp( B ) .
A feasible choice is T = B
=
( β/ { 2 d + 4 β } ) log( n ) where n is the sample size.
Suppose Assumptions 4.3 (in main context) holds and f 0 ∈ H β ([0 , 1] d , B 0 ) .
Take N = 1 in (7) and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) + {+1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 
+}
[Equation]
[Equation]
log
[Equation]
) (cid:101) /
( s
[Equation]
− 1) .
Theorem 4.6 implies that, with probability at least 1 − exp {− n d/
d +2 β ) } , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
or simply
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) + 1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
In addition, suppose the approximate low-dimensional manifold Assumption 2.3 also holds and for any ε ∈ (0 , 1) , the radius of the neighborhood ρ in Assumption 2.3 satisﬁes ρ ≤ C 2 ( NM ) − 2 β/d ε ( (cid:98) β
(cid:99) + 1) 2 d 1 / 2 d 2 (cid:98) β
(cid:99)
ε ( (cid:112) d/d ε +1 − ε ) − 1 (1 − ε ) 1 − β , where d ε =
O ( d M log( d/ε ) /ε 2 ) is an integer satisfying d M ≤ d ε (cid:28) d.
Then Theorem 4.7 shows that the rate of convergence can be improved to
[Equation]
where C ( d, d ε , ε, β, B 0 , s min )
=
O ( B 0 (1 − ε ) − β
( (cid:98) β (cid:99) + 1) 8 d 1 / 2 d 3 (cid:98) β (cid:99) +3 ε / ( s min − 1)) is a constant independent of n .
To improve the rate of convergence, one can use a modiﬁed exponential loss function, i.e. φ ( a ) = max { exp( − a ) , τ } for some small τ > 0 .
The minimum of modiﬁed φ can be achieved at a ∗ ( τ )
= − log( τ ) , and the minimizer f 0 ( x ) will be a truncated version of log( η/ (1 − η )) / 2 (truncated by a ∗ ( τ ) = − log( τ ) ).
In light of this, under the modiﬁed exponential loss with τ = exp( − T ) , the corresponding measurable minimizer deﬁned in (14) is f 0 ,T , the truncated version of the minimizer f 0 under the original exponential loss; see Table 4.
Then ∆ φ ( T ) = 0 by its deﬁnition since the inﬁmum of the modiﬁed loss can be achieved within [ − T, T ] .
By choosing T = B = log log n and τ = exp( − T ) , with the modiﬁed exponential loss, Theorem 4.6 implies that with probability at least 1 − exp {− n d/ ( d +2 β ) } , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) +1) 8 d 3 (cid:98) β (cid:99)
+3 /
( s min − 1)) is a constant independent of n .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
C. The least squares loss
For the least squares loss φ ( a ) = (1 − a ) 2 , we ﬁrst prove that the approximation error is of a special form.
For any f ∈ F CNN
[Equation]
Ans we have
[Equation]
Besides, note that for least squares f 0 = 2 η − 1 and (cid:107) f 0 (cid:107) ∞ = 1 , thus taking the truncation T = 1 is sufﬁcient for the error control analysis.
Let F CNN be a class of CNNs deﬁned in (2) with B ≥ T = 1 .
By Theorem 2.2, there exists a function f implemented by CNN with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 M (cid:100) log(8 M ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) + and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38 4 ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 N (cid:100) log(8 N ) (cid:101) /
( s min − 1) such that
[Equation]
and based on Lemma 4.1 and Theorem 4.2, the empirical φ -risk minimizer ˆ f n deﬁned in (13) satisﬁes, for any δ > 0 , with probability at least 1 − δ ,
[Equation]
Let F CNN be the class of CNNs deﬁned in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99)
+2 (cid:98) n d/ (2 d +8 β ) (cid:99)(cid:100) log(8 n d/ (2 d +8 β ) )
(cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ {+8 ∗ 9 2 ∗ 38 
+}
[-∗ 9 2 ∗ 38 
-] ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) n d/ (2 d +8 β ) (cid:99)(cid:100) log(8 n d/ (2 d +8 β ) ) (cid:101) /
( s min − 1) , B ≥ B 0 and δ = exp {− n d/ ( d +4 β ) } .
By plunging in φ B = ( B +1) 2 and above values, we have with probability at least 1 − exp {− n d/ ( d +4 β ) } ,
R ( ˆ f n ) − R ( f 0 ) ≤ (log n ) 2 n − β/ (3 d/ 8+2 β ) × (cid:110) c
( β + 1) 8 ( B + 1) 2 d 3 (cid:98) β (cid:99) +3 / ( s min − 1) + 18 2 B 2 (
[Equation]
[-B + 1) 2 d 3 (cid:98) β (cid:99) +3 / ( s min − 1) + 18 2 B 2 ( (cid:98) β (cid:99) + 1) 
 d 2 (cid:98) β (cid:99) +( β ∨ 1) (cid:111) ,-] or simply
[Equation]
where C ( d, β, B , s min )
=
O ( B 2 ( (cid:98) β (cid:99) + 1) 8 d 3 (cid:98) β (cid:99) +3 /
( s min − 1)) is a constant independent of n .
For the induced classiﬁer ˆ h n = sign( ˆ f n ) , the excess misclassiﬁcation error satisﬁes
[Equation]
The excess risk bound under the approximate low-dimensional manifold assumption can be obtained in a similar way.
C.5 Excess misclassiﬁcation errors
In this subsection, we summarize the misclassiﬁcation error bounds obtained in existing results and compare them with ours.
When the hypothesis space is taken to be the class of measurable functions itself, for the excess misclassiﬁcation error with the 0-1 loss, the convergence rates of order O ( n − 1 / 2 ) can be attained using oracle inequalities [41, 58].
Under Tsybakov noise condition of exponent θ and α -Hölder smooth condition on the decision boundaries, [55] proved that the minimax lower bound of the excess misclassiﬁcation rate is O ( n − α
( θ +1) /
{ α ( θ +2)+( d − 1) θ } )
over all measurable classiﬁers.
With large enough α and θ , the convergence rate can be close to O ( n − 1 ) arbitrarily.
For classiﬁcations using deep neural networks with the SVM hinge loss function, [30] showed that the minimax optimal convergence rate is O ( n − α
( θ +1) /
{ α ( θ +2)+( d − 1) θ } ) under the Tsybakov noise condition of exponent θ and α -Hölder smooth condition on the decision boundaries and
O ( n − α
( θ +1) /
{ α ( θ +2)+ d } ) under the Tsybakov noise condition of exponent θ and α -Hölder smooth condition on the conditional class probability η ( x ) =
P ( Y = 1 | X = x ) .
For classiﬁcations using deep convolutional networks with the p -norm hinge loss function φ ( u ) = max { 1 − u, 0 } p with p ≥ 1 (it is hinge loss when p = 1 ), [18] derived the approximation error rate and the excess misclassiﬁcation rate for a target function in the Sobolev space W β,p ( S d − 1 ) with input data supported on the sphere S d − 1 in R d under the variancing power condition [56].
Two quantities including γ = max { 1 , ( d + 3 + β ) / 2( d − 1) } and τ for variancing power condition are involved in the convergence rate.
Details can be found in [18].
Based on Table II in [18], we list the excess misclassiﬁcation rates and compare them in the table below
[Table] [Table]
This
[Table]
D A toy example for CNN approximation
In this subsection [Table] we use a toy example to illustrate the approximation power of CNNs.
We examine how the approximation error varies according to the ﬁlter size and depth of the CNNs.
We consider a target function f 0 deﬁned as
[Equation]
where x = ( x 1 , x 2 ) ∈ [0 , 1] 2 is the two-dimensional input.
The 2D heatmap and 3D surface visualizations for the target function f 0 are presented in Figure 1.
To construct CNNs that approximates f 0 , we train the CNNs on noiseless data, which is generated by taking values of the target function in [0 , 1] 2 .
Speciﬁcally, we choose 100 × 100 lattice points { X i }
10 , 000 i = uniformly on the square [0 , 1] 2 and calculate the corresponding target function values { Y i =
f 0 ( X i ) } 10 , 000 i = to get the training data { X i , Y i } 10 , 000 i = for function approximation.
We implement the training in Python via Pytorch and use Adam [31] as the optimization algorithm with default learning rate 0.01 and default β = (0 . 9 , 0 . 99) (coefﬁcients used for computing running averages of gradients and their squares).
During the training process, we set the batch size as 2 , 500 , i.e. the 1 / 4 of the sample size.
The maximum epoch of training is set as 1,000 and an early stopping rule is applied where we stop the iteration if the training losses for the last 200 consecutive epochs do not achieve a new minimum.
We investigate the approximation power of CNNs with different shapes, which are speciﬁed through the ﬁlter length and the depth.
For ease of demonstration, we consider CNNs with L hidden layers with each hidden layer being a convolutional layer with a uniform ﬁlter length s. We set the last layer as a fully connected layer with one-dimensional scalar output.
We train 12 CNNs that approximate the target function for L = 1 , 2 , 3 and s = 20 , 50 , 100 , 200 , then we calculate the empirical L 1 and L 2 distances measuring the approximation error according to
[Equation]
where ˆ f denotes the approximate function by CNN.
Due to the randomness of the optimization algorithms, we train each CNN for several times and take the best one as the ﬁnal approximator on the target function.
The summary statistics of empirical L 1 and L 2 distances between the CNNs to the target function on the training data are presented in Table 6.
We can see that the numerical results generally support our theory in the sense that the approximation error shrinks by a proper rate with respect to the ﬁlter length and depth of CNN.
However, we would like to mention that the optimization procedure using
Adam is not fully tuned and optimization errors may slightly perturb the results, especially when the network is deep and wide.
A visualization of CNN approximations on the target function by heatmaps are presented in Figure 2.
[Table]
E Proofs
In this section, we prove Theorems 2.1,2.2, 2.4, 4.4, 4.6, 4.7, as well as the non-asymptotic error bound results for the cross entropy, the SVM, the logistic, the exponential and the least squares losses.
E.1 Proof of Theorem 2.
As in [22], this approximation result in term of (cid:107) · (cid:107) W β − 1 ,p norm is proved for target function f ∈ W β,p ( X ) , which can be seen as an generalization of Theorem 1 in [59] where L ∞ error bounds for f ∈ W β, ∞ ( X ) is derived.
Besides, our results improves the prefactor in d of the network width in Theorem 4.1 in [22].
The main idea of our proof is to approximate the averaged Taylor expansion of function f in Sobolev functions.
The deﬁnition and properties of averaged Taylor expansion can be found in subsection E.13 of this supplementary material.
By Lemma E.6, let B be a ball with proper radius in Ω such that Ω is star-shaped with respect to B , for 1 ≤ p ≤ ∞ we have [Equation] W
[Equation]
Ω
[Equation] − m Ω [Equation] where C k,β,γ > 0 is a constant, r Ω is the diameter of Ω ⊆ X , γ is the chunkiness parameter of Ω deﬁned in deﬁnition E.4 and T β f is the averaged Taylor expansion over B deﬁned as in (32):
[Equation]
where
[Equation]
and φ is an arbitrary cut-off function supported in ¯ B being inﬁnitely differentiable.
This reminder term could be well controlled when the approximation to Taylor expansion in implemented in a fairly small local region.
Then we can focus on the approximation of the Taylor expansion locally.
The proof is divided into four parts:
(i) Partition X into small cubes ∪ θ Q θ , and construct a network ψ that approximately maps each x ∈ Q θ to a ﬁxed point x θ ∈ Q θ .
Hence, ψ approximately discretize X .
(ii)
For any multi-index α , construct a network φ α that approximates the coefﬁcient of the averaged Taylor expansion over a ball B in Q θ , that is x ∈ Q
θ (cid:55)→ c α ( x θ ) , where Q θ = (cid:80) α ≤ β c α ( x θ ) x α by Lemma E.2.
Once X is discretized, the approximation is reduced to a data ﬁtting problem.
(iii) Construct a network P α ( x ) to approximate the polynomial x α := x α 1 1 . . .
x α d d where x = ( x 1 , . . .
, x d ) (cid:62) ∈ R d and α =
( α 1 , . .
. , α d ) (cid:62) ∈ N d 0 .
In particular, we can construct a network φ × ( · , · ) approximating the product function of two scalar inputs.
(iv) Lastly, our construction of neural network can be written in the form,
[Equation]
Proof.
Before proving the theorem, we ﬁrstly show that each ReLU activated multilayer-perceptron can be computed by a downsampled CNN f CNN deﬁned as in (23).
The proof here follows [-Theorem
-]Theorem 2 in [63] where we construct downsampled CNN exactly computing a fully-connected layer.
Due to the simplicity of the formulation of ReLU activated multilayer-perceptron, we focus on showing that each fully-connected layer can be reproduced by a down sampled CNN with proper choice of ﬁlters.
Without loss of generality, let
[Equation]

[Equation]
be the weight matrix and b = ( b 1 , . . .
, b d out ) (cid:62) ∈
R d out be the bias vector of a fully-connect layer
A F ( · ) : R d in → R d out , where w ij , i = 1 , . . .
, d out ; j = 1 , . . .
, d in are weights.
Then the output of the fully-connect layer with respect to x ∈ R d in is
A F ( x ) = σ ( Wx + b ) .
To construct a downsampled CNN to represent the fully-connected layer, we deﬁne a sequence w = ( w 0 , . .
. , w d in × d out ) ∈ R d in × d out with length d in × d out by stacking the reversed row vectors of the weight matrix W as
[Equation]
or
[Equation]
, w d out 1 ) , where w ij are weights of the matrix W .
By Lemma 2 in [63], there exists a sequence of ﬁlter { w ( i ) } Ii =1 of equal ﬁlter length s for some s ∈ [2 , d in × d out ] and I ≤ (cid:100)
( d in × d out − 1) /
( s − 1) (cid:101) such that w has convolutional factorization w ( I ) ∗ w ( I − 1) ∗ · · · ∗ w (1) .
We let { w ( i ) } Ii =1 be the ﬁlters of the I Toeplitz type convolutional matrices T (1) , . . .
, T ( I ) of the convolutional operators
[Equation]
for i = 1 , . . .
, I .
Next we construct the bias vectors { b ( i ) } Ii =1 according to Lemma 3 in [63].
Let
[Equation]
with B denoting the inﬁnity norm of the input x of the fully-connected layer, and let
[Equation]
for i = 2 , . . .
, I − 1 .
Then we have
[Equation]
Finally, we set
[Equation]
where θ ∈ R d in + s ( I − 1) s an arbitrary vector satisfying D
I ( θ ) =
b with D I : R d in + sI → R d out denoting the downsampling operator and b is the bias vector of the target fully-connected layer.
Then we can deﬁned the downsampled CNN by
[Equation]
and by Lemma 1 in [63] we have
[Equation]
And the the total number of free parameters in computing the fully-connected layer A F is no more than
[Equation]
) (cid:101) ≤ 8 d in × d out , which is 8 times of the total parameters of the fully-connected layer.
The depth of f CNN is I + 1 ≤ (cid:100)
( d in × d out − 1) /
( s − 1) (cid:101)
+ 1 .
If we take s =
d in × d out , then f CNN is a 2-layer convolutional network.
By induction, we know that for any MLP with depth L , width ( d 0 , d 1 , . . .
, d L +1 ) and size S , there exists a downsampled CNN as deﬁned in (23) that exactly computes the MLP.
The downsampled CNN has size no more than 8 S , depth no more than L + (cid:80) Li =
(cid:100)
d i × d i +1 /
( s ( i ) − 1) (cid:101) where s ( i ) is the ﬁlter length of the 2 i -th layer of downsampled CNN and the
i − 1 -th layers are downsampling operations.
More speciﬁcally, if take the maximum possible ﬁlter size s ( i ) =
d i × d i + for each construction of the fully-connected layer above, the downsampled CNN that computed the
MLP has depth no more than 2 L , size no more than 8 S and maximum ﬁlter length no more than
W = max { d 0 d 1 , . . .
, d L d L +1 } .
In general, the maximum ﬁlter length s
= O (1) , and the constructed downsampled CNN has size no more than 8 S , depth no more than L + (cid:80) Li {+=
+}
(cid:100)
d i × d i +1 /
( s ( i ) − 1) (cid:101) ≤
L (1 + W 2 / ( s min − 1)) ≤ 2 L W 2 / ( s min − 1) where s min is the minimum of the ﬁlter lengths over convolutional layers.
Now we prove the theorem following the four parts described above. Part (i) (domain discretization) uses an idea similar to that in the proof of Theorem 3.3 in [28]. We target for approximation on Sobolev functions that the inequalities will be obtained for Sobolev norms instead of L 1 or L 2 norms, and the local averaged Taylor’s expansion instead of Taylor’s expansion of the target function will be approximated.
Without loss of generality, we assume the Sobolev norm of f is 1, i.e. (cid:107) f (cid:107) W β,p ( X ) ≤ 1 .
The reason is that we can always approximate f/B 0 ﬁrstly by a network φ with approximation error (cid:15) , then the scaled network B 0 φ will approximate f with error no more than (cid:15)B 0 .
Besides, it is a trivial case when the Sobolev norm of f is 0.
Next, we prove the four parts of the sketch as follows.
Part (i): Discretization.
We ﬁrst divide the domain X approximately into hypercubes as in [54, 28]. We present below the sketch, and details can be found in proof of Theorem 3.3 in [28]. (i) The
domain
is approximately
divided
into
hypercubes
{ Q θ } indexed by θ
Given
N, M ∈ N + , set K = (cid:98) N 1 /d (cid:99) 2 (cid:98) M
2 /d (cid:99) and δ ∈ (0 , 1 / (3 K )] , for each θ =
( θ 1 , . .
. , θ d ) ∈ { 0 , 1 , . .
, K − 1 } d , we deﬁne
[Equation]
Let Ω( X , K, δ ) =
X\ ∪ θ Q θ be the triﬂing region or the complement of ∪ θ Q θ .
(ii)
For each θ , there exists a ReLU network maps all x ∈ Q θ to a common value depending on θ .
To be speciﬁc, by Lemma E.7 and proof in Theorem 3.3 in [28], there exists a ReLU network ψ ( x ) with width d (4 (cid:98)
N 1 /d (cid:99) + 3) and depth 4 M + 5 such that for any θ = ( θ 1 , . . . , θ d ) ∈ { 0 , 1 , . . , K − 1 } d , we have
[Equation]
for all x = ( x 1 , . . .
, x d ) (cid:62) ∈ Q θ .
Part (ii): Local Approximation of averaged Taylor expansion coefﬁcients.
We consider the local approximation of f ∈ W β,p ( X ) on the subset Q θ for each θ by its averaged Taylor expansion T β f ( x ) = (cid:82) B θ
T
βy f ( x ) φ ( y ) dy over the ball B θ where
T βy f ( x ) =
(cid:80) (cid:107) α (cid:107) 1 ≤ β 1 α !
D
α f ( y )( x − y )
α , the ball B θ = { y ∈ X : (cid:107) y − ( θ +(1 − δ ) / 2) /K
(cid:107) 2 ≤ (1 − δ ) / (2 K ) } support in Q θ , and φ is an cut-off function supported in ¯ B deﬁned in Deﬁnition E.1.
Note that
Q θ has diameter r Q θ = √ d (1 − δ ) /K
and it is star-shaped with respect to the ball B θ with radius (1 − δ ) /
(2 K ) .
By Lemma E.6, we have error bound for the remainder of the averaged Taylor expansion,
[Equation]
Besides, by Lemma E.2, the averaged Taylor expansion is actually a polynomial T β f ( x ) = (cid:80) [Equation] α
[Equation]
1 ≤ β c θα x α for x ∈ Q θ with | c θα | ≤ C ( β, d ) for all α where
[Equation]
Next we seek to approximate the polynomial T β f ( x ) = (cid:80) (cid:107) α (cid:107) 1 ≤ β c θα x α locally by deep neural networks.
In this part of proof, we ﬁrstly construct network to approximate the coefﬁcients of averaged Taylor expansion.
For each hypercube Q θ , we locate the input x ∈ Q θ by mapping multi-dimensional x to an integer i θ for each θ .
Given the fact that θ ∈ { 0 , 1 , . . .
, K − 1 } d is one-to-one correspondence to i θ := (cid:80)
dj
=1 θ j K j − 1 ∈ { 0 , 1 . .
, K d − 1 } , and based on the locator ψ in part (i), it is proved that in [39, 28], there exists a network ψ 0 :
R d → R with width d (4 (cid:98) N 1 /d (cid:99) + 3) and depth 4 M + 5 such that for each θ ∈ { 0 , 1 , . .
, K − 1 } d , ψ 0 ( x ) =
(cid:80) dj
=1 θ j {+K j − 
+}
=
i θ for all x ∈ Q θ .
For any α ∈ N d 0 satisfying (cid:107) α (cid:107) 1 ≤ β and each i = i θ ∈ { 0 , 1 , . . .
, K d − 1 } , we denote ξ α,i :=
( c θα + C ( β, d )) / (2 C ( β, d )) ∈ [0 , 1] .
Since K d ≤ N 2 M 2 , by Lemma E.8, there exists a ReLU network ϕ α with width 16( β + 2)( N + 1) (cid:100) log 2 (8 N ) (cid:101) and depth 5( M + 2) (cid:100) log 2 (4 M ) (cid:101) such that | ϕ α ( i ) − ξ α,i | ≤ ( NM ) − 2 β − 4 , for all i θ ∈ { 0 , 1 , . .
, K d − 1 } .
We deﬁne
[Equation]
Then φ α can be implemented by a network with width 16 d ( β + 2)( N + 1) (cid:100) log 2 (8 N ) (cid:101) ≤ 32 d ( β + 2) N (cid:100) log 2 (8 N ) (cid:101) and depth 5( M + 2) (cid:100) log 2 (4 M ) (cid:101) + 4 M + 5 ≤ 15 M (cid:100) log 2 (8 M ) (cid:101) such that
[Equation] for any θ ∈ { 0 , 1 , . . .
, K − 1 } d and x ∈ Q θ .
Part (iii): Local Approximation of the polynomial.
In this part, we present two lemmas showing that the multiplication operator and polynomials can be approximated by proper networks under (cid:107) · (cid:107) W 1 , ∞ norm.
For multiplication operator, by Lemma E.9, there exists a ReLU network with width 9 N + 1 and depth 4 βM + 8 such that for any t 1 , t 2 ∈ [ a, b ] ,
[Equation]
[Equation] [Equation]
≤ [Equation] b [Equation]
2 N − 2 βM
[Equation]
(25) with (cid:107) φ × (cid:107) W 1 , ∞ (( a,b ) 2 ) ≤ 12( b − a ) 2 .
For polynomials, by Lemma E.10, for any α ∈ N d 0 with (cid:107) α (cid:107) 1 ≤ β ∈ N + , there exists a ReLU network P α with width 9( N + 1) + β and depth 7( β + 1) 2 M and (cid:107) P α (cid:107) W 1 , ∞ ( X ) ≤ 18 such that for any x ∈⊆ [0 , 1] d
[Equation]
Note that above approximation result holds for x ∈ [0 , 1] d , we then provide below network construction for truncation operator of the input onto [0 , 1] d .
Let ϕ 0 ( t )
=
min { max { t, 0 } , 1 } = σ ( t ) − σ ( t − 1) for t ∈ R , where σ ( · ) is the ReLU activation function.
And let ϕ ( x ) =
ϕ
( x 1 , . . .
, x d ) :=
( ϕ 0 ( x 1 ) , . . .
, ϕ 0 ( x d )) (cid:62) denote the extension of ϕ 0 to R d .
Then ϕ truncates the input x ∈ R d onto [0 , 1] d .
Part (iv): Approximation of f on ∪ θ ∈{ 0 , 1 ,...,K − 1 } d Q θ .
For any x ∈ Q θ , θ ∈ { 0 , 1 , . .
, K − 1 } d , we can now approximate the averaged Taylor expansion of f ( x ) by combined sub-networks.
Motivated by this, we deﬁne
[Equation]
where (cid:107) φ 0 (cid:107) W 1 , ∞ ( X ) ≤ 48 C ( β, d ) 2 ( β + 1) d β .
Then φ 0 is our constructed neural network approximating the target function f on X .
Recall that width and depth of ϕ is (2 d, 1) , width and depth of ψ is ( d (4 (cid:98) N 1 /d (cid:99) + 3) , 4 M + 5) , width and depth of P α is (9( N + 1) + β, 7( β + 1) 2 M ) , width and depth of φ α is width ( 16 d ( β + 2)( N + 1) (cid:100) log 2 (8 N ) (cid:101) , [-
 (-]5( M + 2) (cid:100) log 2 (4 M ) (cid:101) + 4 M + 5 ) and width and depth of φ × is (9 N + 1 , 2 βM + 4) .
Hence, by our construction, φ 0 can be implemented by a neural network with width 38( β + 1) 2 d β +1 N (cid:100) log 2 (8 N ) (cid:101) and depth 21( β + 1) 2 M (cid:100) log 2 (8 M ) (cid:101) .
For θ ∈ { 0 , 1 , . . .
, K − 1 } d , thanks to Lemma E.6, we have error bound for the remainder of the averaged Taylor expansion,
[Equation]
where C β,d,r = 3 βd β − 1 Γ( d/ 2 + 1) π − d/ 2 (1 + 2 √ d ) d .
Then for any θ ∈ { 0 , 1 , . . .
, K − 1 } d , the approximation error of φ 0 on f over Q θ can be upper bounded, [Equation] [Equation] We denote E α =
[Equation]
c θα [Equation] × [Equation]
φ α
[Equation] x [Equation] , P
[Equation]
x
[Equation]
[Equation]
W 1 , ∞ ( Q θ ) for each α ∈ N d 0 with (cid:107) α (cid:107) 1 ≤ β .
Using the inequality | t 1 t 2 − φ × ( t 3 , t 4 ) | ≤ | t 1 t 2 − t 3 t 2 | + | t 3 t 2 − t 3 t 4 | + | t 3 t 4 − φ × ( t 3 , t 4 ) | ≤ | t 2 || t 1 − t 3 | + | t 3 || t 2 − t 4 | + | t 3 t 4 − φ × ( t 3 , t 4 ) | for any t 1 , t 2 , t 3 , t 4 ∈ R , and by (24), (25) and (26), for (cid:107) α (cid:107) 1 ≤ β we have
[Equation]
where the last inequality holds since p ≥ 1 and (cid:107) f (cid:107) W β,p (Ω) ≤ B 0 .
Moreover, as in the proof of Theorem 3.3 in [28], the number of terms |{ α : α ∈ N d 0 , (cid:107) α (cid:107) 1 ≤ β }| in the summation can be bounded by ( β + 1) d β .
Recall that K = (cid:98) N
1 /d (cid:99) 2 (cid:98) M 2 /d (cid:99) [Equation] combining (27) and (28) [Equation]
we have (cid:107) [Equation]
[Equation]
for θ ∈ { 0 , 1 , . . .
, K − 1 } d .
Note that X
= Ω( X , K, δ ) ∪ ( ∪ θ ∈{ 0 , 1 ,...,K − 1 } d Q θ ) and the Lebesgue measure of Ω( X , K, δ ) can be upper bounded by δdK since δ ∈ (0 , 1 / (3 K )] can be arbitrarily small.
Then the approximation error (cid:107) f
( x ) − φ 0 ( x ) (cid:107) W 1 ,p ( X ) can be further bounded as follows.
[Equation]
where (cid:107) f − φ 0 (cid:107) pW 1 ,p (Ω( X ,K,δ )) can be arbitrarily small since (cid:107) f (cid:107) pW 1 , ∞ ( X ) and (cid:107) φ 0 (cid:107) pW 1 , ∞ ( X ) are bounded, and δ ∈ (0 , 1(3 K )] can be arbitrarily small.
When p = ∞ , the target function f ∈ W β, ∞ ( X ) actually is Hölder smooth with order β and constant B 0 = (cid:107) f (cid:107) W β, ∞ ( X ) .
In this case, it is shown in Theorem 3.3 of [28] that there exists a function φ 0 implemented by a neural network with width 38( (cid:98) β (cid:99) + 1) 2 d (cid:98) β (cid:99) +1 N (cid:100) log 2 (8 N ) (cid:101) and depth
( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101) such that
[Equation]
for any x ∈ ∪ θ ∈{ 0 , 1 ,...,K − 1 } d Q θ .
If the probability measure of X is absolutely continuous with respect to the Lebesgue measure, we also have
[Equation]
As we have shown at the beginning of the proof, the multilayer-perceptron φ 0 here can be computed by a downsampled CNN with ﬁlter lengths
≤ s min ≤ s max ≤ W = 38 2 ( (cid:98) β
(cid:99) {++ 1) 
+}
d 2 (cid:98) β (cid:99) +2 N 2 (cid:100) log 2 (8 N ) (cid:101) 2 and depth L ≤ 42( (cid:98)
β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101)W / ( s min − 1) and size
S ≤ 8 W L .
We know that there exists a f CNN ∈ F CNN such that the approximation results hold.
This completes the proof.
E.2 Proof of Theorem 2.
Proof.
Let K ∈ N + and τ ∈ (0 , 1 /K ) , deﬁne a region Ω( X , K, τ ) of X as Ω( X , K, ε ) = ∪ di =1 { x = [ x 1 , x 2 , ..., x d ] T : x i ∈ ∪ K − 1 k =1 ( k/K − τ, k/K ) } .
By Theorem 3.3 of [28], for any M, N ∈ N + , there exists a function f ∗ ∈ F MLP with (cid:107) f ∗ (cid:107) ∞
≤ B , depth 21( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101) and width 38( (cid:98) β (cid:99) + 1) 2 d (cid:98)
β
(cid:99) +1 N (cid:100) log 2 (8 N ) (cid:101) such that
[Equation]
for any x ∈ X\ Ω( X , K, τ )
where K = (cid:98) N 1 /d (cid:99) 2 (cid:98)
M 1 /d (cid:99) 2 and τ is an arbitrary number in (0 , 13 K ] and (cid:98) a (cid:99) denotes the largest integer smaller than a .
Note that the Lebesgue measure of Ω( X , K, τ ) is no more than dKτ which can be arbitrarily small if τ is arbitrarily small.
Since P is absolutely continuous with respect to Lebesgue measure, then we have
[Equation]
By the proof of Theorem 2.1, any MLP can be computed exactly by a downsampled CNN.
Thus there exists a function f implemented by CNN with its depth no more than L = 42( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101)W / ( s min − 1) where W = 38 2 ( (cid:98) β (cid:99)
+ 1) 4 d 2 (cid:98) β (cid:99) +2 N 2 (cid:100) log 2 (8 N ) (cid:101) 2 and its ﬁlter lengths [-
-][Equation]
s min [Equation] s max ≤ [Equation]
[-(cid:99) + 1) 
-]= [Equation]
[Equation] β [Equation] +2 N 2 (cid:100) log 2 (8 N ) (cid:101) 2 such that
[-E (cid:107) f ( X ) − f 0 ,T ( X ) (cid:107) 2 ≤ 18 B 0 ( (cid:98) β (cid:99) + 1) 
 d (cid:98) β (cid:99) +( β ∨ 1) / 2 ( NM ) − 2 β/d .-][Equation]
E.3 Proof of Theorem 2.
Proof.
The idea of the proof is to project the data into a low-dimensional space using a shallow CNN and then approximate the low-dimensional function using another CNN.
The idea is similar to that of Theorem 6.1 in [28] and Theorem 1.2 in [54] and the existence of such a projection follows from Theorem 3.1 in [3]. We present a proof sketch of the projection and details can be completed following the proof of Theorem
1 in [28]. (i) There exists a linear projector A ∈ R d δ × d that maps a low-dimensional manifold (embedded in a high-dimensional space) to a low-dimensional space almost preserving the distance, i.e. there exists a matrix A ∈ R d δ × d such that AA T =
( d/d δ )
I d δ where I d δ is an identity matrix of size d δ × d δ , and (1 − δ ) (cid:107) x 1 − x 2 (cid:107) 2 ≤ (cid:107)
Ax 1 − Ax 2 (cid:107) 2 ≤ (1 + δ ) (cid:107) x 1 − x 2 (cid:107) 2 , for any x 1 , x 2 ∈ M .
(ii)
For the linear map A and ρ -neighborhood M ρ , we have A ( M ρ ) ⊆ A ( X ) ⊆
E :=
[ − (cid:113) dd δ , (cid:113) dd δ ] d δ .
(iii)
For any z ∈ A ( M ) , there exists a unique x ∈ M such that Ax = z . Then for any z ∈ A ( M ) , deﬁne x z = SL ( { x ∈ M :
Ax = z } ) where SL ( · ) is a set function which returns a unique element of a set.
Then SL : A ( M ) → M is a differentiable function with the norm of its derivative locates in [1 / (1 + δ ) , 1 / (1 − δ )] , i.e. 11+ δ (cid:107) z 1 − z 2 (cid:107) 2 ≤ (cid:107) x
z 1 − x z 2 (cid:107) 2 ≤ 11 − δ (cid:107) z 1 − z 2 (cid:107) 2 , for any z 1 , z 2 ∈ A ( M ) .
(iv) For the function f 0 : [0 , 1] d → R 1 , deﬁne its low-dimensional representation ˜ f 0 : R d δ → Rby ˜ f [-
-]0 ( z )
=
f 0 ( x z ) , for any z ∈ A ( M ) ⊆
R d δ .
Then ˜ f 0 ∈ H
β ( A ( M ) , B 0 / (1 − δ ) β ) since f 0 ∈ H β ( X , B 0 ) .
(v) By the extended version of Whitney’ extension theorem in [17], there exists a function
˜ F 0 ∈ H β ( E, B 0 / (1 − δ ) β ) such that ˜ F 0 ( z ) =
˜ f
0 ( z ) for any z ∈ A ( M ) , given that M is compact and A is a linear mapping.
Now on E =
[ − (cid:112) d/d δ , (cid:112) d/d δ ] d δ , we consider approximate ˜ F 0 by CNN. By Theorem 2.1, for any N, M ∈ N + , there exists a function ˜ f n : R d δ → R 1 implemented by a downsampled CNN with ﬁlter lengths 2 ≤ s min ≤ s max ≤ W = 38 2 ( (cid:98) β (cid:99) + 1) [-
-]{+4 d 2 (cid:98) β (cid:99) +
+}
[-d 2 (cid:98) β (cid:99) +
-]
δ
N 2 (cid:100) log 2 (8 N ) (cid:101) 2 and depth L =
( (cid:98) β (cid:99) + 1) 2 M (cid:100) log 2 (8 M ) (cid:101)W / ( s min − 1) such that
[Equation]
[Equation] for all z ∈ E \ Ω( E ) where Ω( E ) is a subset of E with an arbitrarily small Lebesgue measure as well as Ω [Equation] M ρ : Ax ∈ Ω( E ) } does.
If we deﬁne f ∗ n = ˜ f n ◦ A , i.e. f ∗ n ( x ) = ˜ f n ( Ax ) for x ∈ X , then f ∗ n is also a downsampled CNN with one more layer than ˜ f n since linear map A can be realized by one ReLU layer.
Note that for any x ∈ M ρ \ Ω and z
=
Ax , there exists a ˜ x ∈ M such that (cid:107) x − ˜ x (cid:107) 2 ≤ ρ , and
[Equation]
where C 2 > 0 is a constant not depending on any parameter and the last inequality follows from ρ ≤ C 2 ( NM )
− 2 β/d δ ( (cid:98) β (cid:99) +1) 2 d 1 / 2 d (3 β +1) / 2 δ { (cid:112) d/d δ +1 − δ } − 1 (1 − δ ) 1 − β .
Given the probability measure ν of X is absolutely continuous with respect to the Lebesgue measure, we have [Equation]
[Equation]
where d δ = O ( d M log( d/δ ) /δ 2 ) is assumed to satisfy d δ (cid:28) d .
This completes the proof of Theorem 2.4.
E.4 Proof of Theorem 4.
The proof follows Theorem 11.8 and Corollary 3.19 in [43].
Recall that ( X, Y ) has joint distribution
P , we let P n denote the empirical distribution of the sample S =
{ ( X i , Y i ) } ni =1 and let ( ˜ X, ˜ Y ) ∼ P n .
Note that for any f ∈ F CNN we have (cid:107) f (cid:107) ∞ ≤ B , then 0 ≤ φ ( yf ( x )) ≤ φ B for any ( x, y ) ∼ P .
For any f ∈ F CNN and t ≥ 0 , we denote by c ( f, t ) the classiﬁer deﬁned by c ( f, t ) : ( x, y )
(cid:55)→( φ ( Y f ( X )) > t ) .
Correspondingly, we deﬁne the risk of c ( f, t ) by
[Equation]
>
[Equation]
] , and similarly deﬁne its empirical risk by R n ( c ( f, t ))
=
P n [ φ ( ˜ Y f
( ˜ X ))
> t ] .
Then we write
[Equation]
And this implies that
[Equation]
The right-hand side can be bounded using a standard generalization bound for classiﬁcation (Corollary 3.19 in [43]) in terms of the VC-dimension of the class of functions { c ( f, t ) : f ∈ F CNN , t ∈ [0 , φ B ] } , which, by deﬁnition of the pseudo-dimension, is Pdim ( F CNN ) the pseudo-dimension of function class
F CNN =
{ ( x, y )
(cid:55)→ φ ( yf ( x )) :
f ∈ F CNN } .
Now we have with probability at least 1 − δ over the choice of an i.i.d.
sample S of size n that
[Equation]
Next, we bound the pseudo-dimension of F CNN by that of F CNN , and this leads to the ﬁnal stochastic error bound in terms of the parameters of F CNN .
For a sample S = {
( X i , Y i ) } ni =1 with size n , if there exist t 1 , . . . , t n ∈ [0 , φ B ] such that for each ( b 1 , . . .
, b n ) ∈ { 0 , 1 } n there exists a f ∈ F such that 1 ( φ ( y i f
( x i ))
> t i )
=
b i for i = 1 , . . . , n , then since φ is convexity and non-increasing, we also have 1 ( f ( x i ) >
φ − 1 ( t i ))
=
1 − b i if y i
=
and 1 ( f ( x i ) > − φ − 1 ( t i ))
=
b i if y i
=
− 1 .
Thus { x i } ni =1 is shattered by F CNN and the threshold values y 1 φ − 1 ( t 1 ) , . . .
, y n φ − 1 ( t n ) witness the shattering.
This implies Pdim ( F CNN ) ≤ Pdim ( F CNN ) .
Besides, for piece-wise linear neural networks
F CNN , it is proved in Theorem 6 of [4] that
[Equation]
for some universal constant C > 0 where S and L are the size and depth of F CNN respectively.
Combine above inequities, we have with probability at least 1 − δ over the choice of an i.i.d.
sample
S of size n that
[Equation]
for some universal constant C > 0 .
E.5 Proof of Lemma 4.
Proof.
To deal with the approximation of unbounded target function, we consider truncating the target function f 0 by a constant T , where T may depend on n .
Let f 0 ,T be the truncated version of f 0 deﬁned as
[Equation]
Then, the approximation error can be decomposed into two terms that are easier to deal with.
Denote the error of the loss function φ due to truncation by
[Equation]
where Ran ( f 0 ) is the range of the target function f 0 .
This error decreases as the threshold T increases, however, a bigger T leads to bigger B and B φ , thus a bigger stochastic error.
Firstly, we prove that under Assumption 4.3(a) and (b), f 0 ,T , the truncated version of the target function f 0 , is bounded by T and continuous on X .
Recall that for any a ∈ R and η ∈ [0 , 1] , the conditional risk function H ( η, a ) =
ηφ ( a ) + (1 − η ) φ ( − a ) .
Given η ∈ [0 , 1] , the function dH ( η, a ) /da
= ηφ (cid:48)
( a ) − (1 − η ) φ (cid:48) ( − a ) is continuous in both η and a .
Thus under Assumption 4.
(b) the solution a ( η ) =
arg min a ∈ ¯ R H ( η, α ) is continuous with respect to η , and | a ( η ) | < ∞ when η ∈ (0 , 1) .
By Assumption 4.3 (a), η ( x ) is continuous on { x ∈ X : η ( x ) ≤ 1 − δ } for any δ ∈ (0 , 1) , thus f 0 ( x ) = f 0 ( η ( x )) is continuous and η ( x ) φ (cid:48)
( f 0 ( x )) − (1 − η ( x )) φ (cid:48) ( − f 0 ( x ))
=
on for any x ∈ X .
And f 0 ( x ) is continuous on { x ∈ X : | f 0 ( η ( x ))
| ≤ T } for any T ≥ 0 .
For any T > 0 , the set { x ∈ X : | f 0 ( x ) | ≤ T } is a compact set.
Thus f 0 ,T , the truncated version of target function is also continuous on X .
Recall that for any f , the φ -risk is deﬁned by R ( f ) =
E φ ( Y f ( X )) , then for f 0 ,T , the truncated version of target function satisﬁes
[Equation]
For x ∈ X , we deﬁne g ( x ) = η ( x ) { φ ( f 0 ,T ( x )) − φ ( f
[Equation]
} +
[Equation]
η
[-+ (1 − η ( X )) { φ ( − f 0 ,T ( X )) − φ ( − f 
 ( X )) } (cid:105) . For x ∈ X , we deﬁne g ( x ) = η ( x ) { φ ( f 0 ,T ( x )) − φ ( f 0 ( x )) }-][Equation]
) { φ ( − f [Equation] T ( x )) − φ ( − f 0 ( x )) } .
Recall that φ is non-increasing and we have:
{+(a) If − T ≤ f 0 ( x ) ≤ T , then f 0 ( x ) = f 0 ,T ( x ) and g ( x ) = 0 ; (b) If f 0 ( x ) > T , then f 0 ,T ( x ) = T and φ ( − f 0 ,T ( x )) − φ ( − f 
 ( x )) ≤ 0 , and g ( x ) =+}
η ( x ) { φ ( f 0 ,T ( x )) − φ ( f 0 ( x )) } + (1 − η ( x )) { φ ( − f 0 ,T ( x )) − φ ( − f 0 ( x )) } ≤ η ( x ) { φ ( f 0 ,T ( x )) − φ ( f 0 ( x )) } = ( x ) { φ ( T ) − φ ( f 0 ( x )) } ≤ φ ( T ) − φ ( f 0 ( x )); (c) If f 0 ( x ) < − T , then f 0 ,T ( x ) = − T and φ ( f 0 ,T ( x )) − φ ( f 0 ( x )) ≤ 0 , and g ( x ) =
η ( x ) { φ
( f 0 ,T ( x )) − φ f ( x )) +
(1 − η x ) { φ ( − f T x − φ
− f 0 ( x )) } ≤ (1 − η ( x )) { φ ( − f T ( x )) − φ ( − f x )) }
= − η ( x
φ ( T ) − φ ( − f
x ) ) } ≤ φ ( T ) − φ
( − f 0 ( x ))
Thus for any x ∈ X , we have g ( x ) ≤
φ ( T ) − inf a ∈ Ran ( f 0 ) φ ( a ) and R ( f 0 ,T ) − R ( f 0 ) =
E { g ( X ) } ≤ φ ( T ) − inf a ∈ Ran ( f 0 ) φ ( a ) , where Ran ( f 0 ) is the range of the target function f 0 .
[Equation]
then left thing to do is to prove
[Equation]
With f 0 ∈ H β ([0 , 1] d , B 0 ) , φ is B φ -Lipschitz, and it is a straightforward proof that
[Equation]
E.6 Proof of Theorem 4.
The main focus of the proof is to deal with the truncation.
If f 0 ∈ H β ([0 , 1] d , B 0 ) is uniformly bounded by T ≤ B , by Theorem 2.
[Equation]
there exists a ReLU neural network φ such that
[Equation]
We let φ 0 ( x ) = σ ( φ ( x ) + B ) − σ ( φ ( x ) − B ) − B be the truncated network where σ denotes the
ReLU activation function.
Then
[Equation]
If f 0 ∈ H β ([0 , 1] d
[Equation]
B 0 ) is not uniformly bounded by T [Equation] then the set { x : | f 0 ( x ) | ≤ T } does not equal to the whole domain X .
Actually { x : | f 0 ( x ) | ≤ T } can be written as a union of connected subsets since f 0 is Hölder smooth on its domain X
= [0 , 1] d , and in the interior of each connected subset, the function f 0 keeps its Hölder smoothness β .
By [17], the truncated target function f 0 ,T on { x ∈ X : | f 0 ( x ) | ≤ T } can be extended to X to be β -Hölder and bound T + c for some small constant c > 0 .
Let ˜ f 0 ,T be the extended function on X , then by Theorem 2.2, there exists a ReLU neural network φ such that
[Equation]
We let φ 0 ( x ) = σ ( φ ( x ) + B ) − σ ( φ ( x ) − B ) − B be the truncated network where σ denotes the
ReLU activation function.
Then
[Equation]
E.7 Proof of Theorem 4.
Combining Lemma 4.1, Theorem 4.2 and 4.5, Theorem 4.6 is easily proved.
For the proof of Theorem 4.7, we need the following lemma, which can be proved using the technique in [54].
We refer to [54] for the detailed description of the technique.
E.8 Proof of Theorem 4.
Proof.
The proof is similar to that of Theorem 4.6, except how we deal with inf f ∈F CNN E (cid:107) f ( X ) − f 0 ,T ( X ) (cid:107) 2 .
Combining Lemma 4.4 and using Theorem 2.4, we have
[Equation]
This completes the proof of Theorem 4.7.
E.9 Proof of the cross entropy loss example in Section
[-the cross entropy loss example in Section 
-]
For the cross entropy loss function φ ( a ) =
− log { 0 .
5 + a } , with the choice T = B
= 0 .
5 − n β/ (2 d +4 β ) , we have B ≤ 0 . 5 , ∆ φ ( T ) ≤ n − β/ (2 d +4 β ) and B φ = n β/ (2 d +4 β ) .
Suppose Assumptions 2.3, 4.3 hold and f 0 ∈ H β ([0 , 1] d , B 0 ) .
Take N = 1 in (7) and M = (cid:98) ξ (cid:99) in (4) and let F CNN be the class of CNNs in (2) with depth L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99)
+2 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 9 × 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98) β (cid:99) +2 and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 9 2 ∗ 38
( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 (cid:98) n d/ { 2 d +4 β } (cid:99)(cid:100) log(8 n ) (cid:101) /
( s min − 1) .
Theorem 4.6 implies that, with probability at least 1 − exp {− n d/
( d +2 β ) } , the excess φ -risk of the
ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
or simply
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B 0 ( β + 1) 8 d 3 β +3 / ( s min − 1)) is a constant independent of n .
The excess risk bound under the approximate low-dimensional manifold assumption can be obtained in a similar way.
Under the modiﬁed cross entropy loss φ ( a ) = max {− log { 0 .
5 + a } , τ } with τ = − log { 0 .
5 + T } , the corresponding measurable minimizer deﬁned in (14) is f 0 ,T and ∆ φ ( T )
= 0 by deﬁnition since the inﬁmum of the modiﬁed cross entropy loss can be achieved within [ − T, T ] ⊂ [ − 0 .
By choosing T = B = 0 .
5 − (log n ) − 1 and τ = − log { 1 − (log n ) − 1 } , with the modiﬁed cross entropy loss, Theorem 4.6 implies that with probability at least 1 − exp {− n d/ ( d +2 β ) } , the excess φ -risk of the ERM ˆ f n deﬁned in (13) satisﬁes
[Equation]
where C ( d, β, B 0 , s min )
=
O ( B
( (cid:98) β (cid:99) + 1) 8 d [Equation] β [Equation]
+3 / ( s min − 1)) is a constant independent of n .
E.10 Proof of the SVM example
Proof.
Based on Lemma 4.1, our proof focus on controlling the approximation error inf f ∈F CNN R ( f ) − R ( f 0 ) .
Recall that f 0 ( x ) =
sign(2 η ( x ) − 1) , we would ﬁrstly approximate Hölder smooth function η by CNNs.
Similar to the proof in Theorem 4.6, for any N, M ∈ N + , there exists a function η ∗ ∈ F CNN with bound B = 1 , depth
L ≤ 378 · 38 2 ( (cid:98) β (cid:99) + 1) 6 d 2 (cid:98) β (cid:99) +2 M (cid:100) log(8 M ) (cid:101) /
( s min − 1) , ﬁlter lengths 2 ≤ s min ≤ s max ≤ 38 2 ( (cid:98) β (cid:99) + 1) 4 d 2 (cid:98)
β (cid:99) +2 N (cid:98) log(8 N ) (cid:99) and size S ≤ 8 W L ≤ 42 ∗ 8 ∗ 38 4 ( (cid:98) β (cid:99) + 1) 10 d 4 (cid:98) β (cid:99) +4 M (cid:100) log(8 M ) (cid:101) N 2 (cid:100) log(8 N ) (cid:101) 2 / ( s min − 1) such that [Equation]
+ [Equation] [Equation] NM
[Equation] − 2 β/ [Equation] , for any x ∈ X\ Ω where Ω is a set with arbitrarily small Lebesgue measure.
Let ε n = 18 B
( (cid:98) β (cid:99) + 1) 2 d (cid:98) β (cid:99) +( β ∨ 1) / [-
-]2 ( NM ) − 2 β/d .
We construct the neural network f CNN by adding one layer to η ∗ as
[Equation]
where σ is the ReLU activation function.
Then we can see that (cid:107) f
CNN (cid:107) ∞ ≤ 1 and f CNN ( x ) = 1 if η ∗ ( x ) ≥ ε n +1 / 2 , f CNN ( x ) = 2( η ∗ ( x ) − 1 / 2) /ε
n − 1 if 1 / 2+ ε n > η ∗ ( x ) ≥ 1 / 2 and f CNN ( x ) = − 1 if η ∗ ( x ) < 1 / 2 .
Let Ω η,ε = { x ∈ X : | 2 η ( x ) − 1 | > ε } for (cid:15) > 0 .
Then for x ∈ Ω η, 4 ε n , we have | f CNN ( x ) − f 0 ( x ) | = 0 , since η ∗ ( x ) − 1 / 2 = ( η ( x ) − 1 / 2) − ( η ∗ ( x ) − η ( x )) >
ε n when η ( x ) − 1 / 2 > 2 ε n and η ∗ ( x ) − 1 / 2 < − ε n when η ( x ) − 1 / 2 < − 2 ε n .
Then
[Equation]
Setting the parameters in η ∗ by N = 1 and M = (cid:98) n d/ { d +4 β ( q +1) } (cid:99) and size (number of parameters) [Equation] + [Equation]
log
[Equation]
n ) (cid:101) /
( s min − 1) , and let δ = exp( − n d/
d +2 β ( q +1)) ) , then by Theorem 4.6, with probability at least 1 − exp( − n d/ ( d +2 β ( q +1)) ) ,
[Equation]
The excess risk bound under the approximate low-dimensional manifold assumption can be obtained in a similar way.
E.11 Proof of the logistic loss example
The proof is similar to that of the cross entropy loss example.
We omitted the proof here.
E.12 Proof of the exponential loss example
The proof is similar to that of the cross entropy loss example.
We omitted the proof here.
E.13 Supporting deﬁnitions and Lemmas
For ease of reference, we collect several deﬁnitions and existing results that we used in our proofs.
Firstly, we present the deﬁnition and properties of the averaged Taylor expansion in chapter 4 of [10].
Deﬁnition E.1 (Averaged Taylor expansion) .
Let β ∈ N , 1 ≤ p ≤ ∞ and f ∈ W β − 1 ,p , and let Ω ∈ R d , x 0 ∈ Ω , r > 0 , B :=
B r ( x 0 ) = { x ∈ R d : (cid:107) x − x 0 (cid:107) 2 < r } with its closure ¯ B compact in Ω .
The corresponding Taylor polynomial of order β of f averaged over B is deﬁned as
[Equation]
32) where
[Equation]
and φ is an arbitrary cut-off function supported in ¯ B being inﬁnitely differentiable, i.e., φ ∈ C ∞ ( R d ) with supp φ = ¯ B and (cid:82) R
d φ ( x ) dx = 1 .
For example, let ψ ( x ) =
{+(cid:26) exp {− (1 − ( | x − x 
 |+}
/r ) 2 ) − 1 } , if | x − x 0 | < r, , else , and let c = (cid:82) R d ψ ( x ) dx ( c > 0) , then φ ( x ) = ψ ( x ) /c is a cut-off function and (cid:107) φ (cid:107) L ∞ ( B ) :=
max | φ ( x ) | ≤ c ( d ) · r − d where c ( d ) > 0 is a constant depending only on d .
Besides,
[Equation]
, else , is a another example of cut-off function, in which case φ puts constant weight on the ball ddimensional B with radius r .
The averaged Taylor polynomial deﬁned above is an integral of the traditional Taylor expansion weighted by the cut-off function over a region.
Following the equations (4.1.5)-(4.1.8) in [10], we show the below lemma that the averaged Taylor polynomial of order β is indeed a polynomial of degree less than β in x .
Lemma E.2.
Let β ∈ N , 1 ≤ p ≤ ∞ and f ∈ W β − 1 ,p (Ω) , and let x 0 ∈ Ω , r > 0 , R ≥ 1 such that for the ball B :=
B r ( x 0 ) with its closure ¯ B compact in Ω and (cid:107) y (cid:107) ∞
≤ R for all y ∈ B .
Then
the
Taylor polynomial of order β of f averaged over B can be written as
[Equation]
x ∈ Ω [Equation] where
[Equation]
for all α satisfying (cid:107) α (cid:107) 1 ≤ β − 1 .
Proof.
Recall that for x, y ∈ R d and α =
( α 1 , . .
. , α d ) ∈ N d 0 , if we write
[Equation]
where γ = ( γ 1 , . . . , γ d ) and θ = ( θ 1 , . .
, θ d ) are d -tuples of nonnegative integers, and a ( γ, θ ) ∈ R are the constants satisfying | a ( γ, θ ) | ≤ (cid:0) αγ (cid:1) =
α ! / ( γ ! θ !)
in multi-index notation.
Then
T β f ( x ) = (cid:90)
[Equation]
Then we consider bounding the coefﬁcient of x γ .
Note that
[Equation]
where the second inequality holds since (cid:107) y (cid:107) ∞
≤ R, ∀ y ∈ B and the use of Hölder’s inequality with 1 /p
+ 1 /q
= 1 , the third inequality holds again by Hölder’s inequality, and the last two lines hold since (cid:107) φ (cid:107) L 1 ( B )
= 1 and (cid:107) φ ( x ) (cid:107) L ∞ ( B ) ≤ π − d/ 2 Γ( d/ 2 + 1) · r − d .
Lastly, the coefﬁcient c γ of x γ can be upper bounded by
[Equation]
Note that
[Equation]
≤
[Equation]
Thus we have
[Equation]
Next, we present the main approximation result of averaged Taylor expansion on Sobolev functions by
Bramble-Hilbert (Lemma 4.3.8 in [10]) after introduce some prerequisite deﬁnitions and conceptions.
Deﬁnition E.
(Star-shaped set) .
Let Ω , B ⊂ R d .
Ω is star-shaped with respect to B if for all x ∈ Ω , the closed convex hull of { x } ∪ B is a subset of Ω .
Deﬁnition E.4 (Chunkiness parameter) .
Suppose Ω ⊂ R d has diameter r Ω and is star-shaped with respect to a ball B .
Let r max = sup { r ∈ R + : Ω is star-shaped with respect to a ball of radius r } .
Then the chunkiness parameter of Ω is deﬁned by γ = r Ω /r max .
Lemma E.5 (Proposition 4.2.8 in [10]) .
Let f ∈ W β,p (Ω) and T β be its averaged Taylor expansion deﬁned in (32) over the ball B with radius r , then the remainder R β f :=
f − T β f satisﬁes
[Equation]
where C x is the convex hull of { x } ∪ B , z
= x + s ( y − x ) , k α ( x, z ) =
(1 /α !)
( x − z ) α k ( x, z ) and
[Equation]
Lemma E.6 (Bramble-Hilbert) .
Let B be a ball in Ω such that Ω is star-shaped with respect to B and such that its radius r > r max / 2 , where r max is deﬁned in Deﬁnition E.4.
Let T β
f
be the Taylor polynomial of order β of f averaged over B where f ∈ W β,p (Ω) and p ≥ 1 .
Then
[Equation]
where r Ω is the diameter of Ω .
The proof can be found in Lemma 4.3.8 in [10].
Lemma E.7 (Proposition 4.3 in [39]) .
For any N, M, d ∈ N + and δ ∈ (0 , 3 K ] with K = (cid:98) N 1 /d (cid:99) 2 (cid:98) M
/d
(cid:99) , there exists a one-dimensional function φ implemented by a ReLU FNN with width 4 (cid:98) N 1 /d (cid:99) + 3 and depth 4 M + 5 such that
[Equation]
Lemma E.8 (Proposition 4.4 in [39]) .
Given any N, M, s ∈ N + and ξ i ∈ [0 , 1] for i = 0 , 1 , . . .
, N 2 L 2 − 1 , there exists a function φ implemented by a ReLU FNN with width 16 s ( N + 1) (cid:100) log 2 (8 N ) (cid:101) and depth (5 M + 2) (cid:100) log 2 (4 M ) (cid:101) such that | φ ( i ) − ξ i | ≤ N − 2 s M − 2 s , for i = 0 , 1 , . . .
, N 2 M 2 − 1 , and 0 ≤ φ ( x ) ≤ 1 for any x ∈ R .
The next lemma demonstrate that the production function and polynomials can be approximated by
ReLU neural networks.
The basic idea is ﬁrstly to approximate the square function using “sawtooth" functions then the production function, which is ﬁrstly raised in [59].
A general polynomial can be further approximated combining the approximated square function and production function.
The following two lemmas are more general results than those in [59].
Lemma E.9 (Lemma 3.4 in [24]) .
For any N, M ∈ N + , and a, b ∈ R with a < b , there exists a function φ implemented by a ReLU FNN with width 9 N + 1 and depth 2 M such that (cid:107) φ (cid:107) W 1 , ∞
(( a,b ) 2 ) ≤ 12( b − a ) 2 and (cid:107) φ ( x, y ) − xy (cid:107) W 1 , ∞
(( a,b ) 2 ) ≤ 6( b − a ) 2 N − M for any x, y ∈
( a, b ) .
Lemma E.10 (Proposition 4.1 in [39], Proposition 3.6.
in [24]) .
Assume P ( x ) =
x α = x
α 1 1 x α 2 2 · · · x α d d for α ∈ N d with (cid:107) α (cid:107) 1 ≤ k ∈ N + .
For any N, M ∈ N + , there exists a function φ implemented by a ReLU FNN with width 9( N + 1) + k − 1 and depth 14 k 2 M such that (cid:107) φ (cid:107) W 1 , ∞ ( X ) ≤ 18 and (cid:107) φ ( x ) − P ( x ) (cid:107) W 1 , ∞ ( X ) ≤ 10 k ( N + 1) − 7 kM for any x ∈ [0 , 1] d .