Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint
Abstract
Overparameterized neural networks enjoy great representation power on complex data, and more importantly yield sufficiently smooth output, which is crucial to their generalization and robustness.
Most existing function approximation theories suggest that with sufficiently many parameters, neural networks can well approximate certain classes of functions in terms of the function value.
The neural network themselves, however, can be highly nonsmooth.
To bridge this gap, we take convolutional residual networks (ConvResNets) as an example, and prove that large ConvResNets can not only approximate a target function in terms of function value, but also exhibit sufficient first-order smoothness.
Moreover, we extend our theory to approximating functions supported on a low-dimensional manifold.
Our theory partially justifies the benefits of using deep and wide networks in practice.
Numerical experiments on adversarial robust image classification are provided to support our theory.
Introduction
Deep neural networks of enormous sizes have achieved remarkable success in various applications.
Some well-known examples include ViT-Huge of 632 million parameters (Dosovitskiy et al., 2020), and the gigantic GPT-3 of 175 billion parameters (Brown et al., 2020).
In addition to outstanding testing accuracy, there has been evidence that large neural networks favor smoothness and yield good robustness (Madry et al., 2017; Bubeck and Sellke, 2021).
Among vast literature on explaining the success of neural networks universal approximation theories analyze how well neural networks can represent complex data models (see literature in related works in Appendix A).
These works focus on approximating a target function in terms of its function value (i e., in function L ∞ norm).
However, other important properties, espcifically the smoothness of the neural networks, are less investigated.
A few early results provide asymptotic results on two-layer networks with smooth activation for approximating both function value and derivatives (Hornik et al., 1990; Cardaliaguet and Euvrard, 1992).
Recently, Gühring et al.
Hon and Yang (2021) established nonasymptotic approximation theory of feedforward networks in terms of Sobolev norms.
In real-world applications, on the other hand, practitioners empirically demonstrated a close tie between the smoothness of a trained neural network to its adversarial robustness (Gu and Rigazio, 2014; Hein and Andriushchenko, 2017; Weng et al., 2018; Miyato et al., 2018).
The intuition behind is relatively clear.
Consider, for instance, adding some adversarial perturbation to an input.
A network of small (local) Lipschitz constant produces less deviation to the original output, and therefore, is often resilient to adversarial attackes.
On the contrary, a network that is vulnerable to adversarial attacks usually has a large Lipschitz constant.
Over the years, many computational methods are proposed and extensively tested in experiments for promoting network smoothness (Goodfellow et al., 2014; Madry et al., 2017; Miyato et al., 2018; Zhang et al., 2019).
Apart from these explicit training methodologies, the size of a network is also recognized as a critical factor to its generalization and robustness (Zagoruyko and Komodakis, 2016;
Madry et al., 2017; Wu et al., 2020).
Yet, theoretical understanding is largely missing.
In this paper, we investigate universal approximation ability of neural networks with smoothness guarantees.
We consider the convolutional residual networks (ConvResNet, see a description in Section D.2) with ReLU activation as an example.
We measure the approximation error of ConvResNet in terms of not only the function value, but also higher order smoothness.
Specifically, suppose given a target function f belonging to a Sobolev space in a D -dimensional hypercube.
We provide an approximation error estimate in terms of Sobolev norm as a function of the size of ConvResNet.
We also extend our theory to functions supported on a d -dimensional Riemannian manifold ( d ≪ D ).
We summarize our main results in the following informal theorem.
Theorem 1 (informal) .
Consider a ConvResNet architecture with (cid:102) M residual blocks and each convolutional filter having at most (cid:101) J channels.
Let α ≥ 2 and 1 ≤ p ≤ ∞ be positive integers.
Then
• (Euclidean) for any target function in a Sobolev space W α,p ((0 , 1) D )
with Sobolev
norm ∥ f ∥ W α,p ((0 , 1) D ) ≤ 1 , there exists (cid:101) f yielded by the ConvResNet architecture , such that
[Equation]
with the constant depending on D, α , p ; • (Manifold) given M ⊂ R D a d -dimensional Riemannian manifold
satisfying mild regularity conditions, for any target function in a Sobolev space W α, ∞ ( M ) with ∥ f
∥
W α, ∞ ( M ) ≤ 1 , there exists (cid:101) f yielded by the ConvResNet architecture, such that
[Equation]
for k
[Equation]
with the constant depending on α, p, M .
Our theory restricts to s ≤ 1 , since only first-order weak derivatives exist for ReLU networks.
Moreover, setting s = 0 or s = 1 is of particular interest, as s = 0 recovers the function value approximation guarantee and s = 1 extends the guarantee to first-order derivatives.
As can be seen, to achieve the same function value approximation error, s = 1 requires a larger network, but enjoys good smoothness.
This can partially explain that larger networks are often more robust.
We refer readers to Corollary 1 for more discussion.
Our manifold setting is motivated by the fact that many high-dimensional data in real applications often have low-dimensional structures (Tenenbaum et al., 2000; Roweis and Saul, 2000; Coifman et al., 2005; Allard et al., 2012;
Pope et al., 2021).
Our results show ConvResNet can adapt to data geometric structures and does not suffer from the curse of ambient dimensionality.
Theorem 1 implies that as the number of residual blocks increases or each filter having more channels,
ConvResNet gives better approximation of the target function.
In order to achieve an ϵ -error, we may set (cid:102) M (cid:101)
J
= O
( ϵ − Dα − s )
( O ( ϵ − dα − s ) for the manifold case), while there is no scaling restriction between (cid:102) M and (cid:101) J .
See an explicit configuration of ConvResNet architecture depending on (cid:102) M and (cid:101) J in Theorem 2 and Theorem 5.
Convolutional residual networks
In this paper, we consider convolutional residual networks (ConvResNet) which consists of several residual blocks and a fully connected layer.
Denote the composition of residual blocks by Q (see (5) in Appendix D.2), and the weight matrix and bias in the fully connected layer by W and b , respectively.
We define the class of ConvResNets as
[Equation] [Equation]
[Equation] [Equation] x ) has M residual blocks .
The number of filters per block is bounded by L ; filter size is bounded by K ; ∥ W ∥ ∞ ∨ | b | ≤ κ 2 , the number of channels is bounded by J ; max m,l ∥W ( l )
m ∥ ∞ ∨ ∥ B ( l ) m ∥ ∞ ≤ κ 1 (cid:9) .
In (1), W ( l ) m , B ( l ) m denote the convolutional filter and bias in the l -th layer of the m -th residual block.
Details on the network architecture are deferred to Appendix D.2.
Approximation in Euclidean Space
Consider a Sobolev function class defined on a unit hypercube (0 , 1) D .
We aim to use convolutional residual networks for approximating functions in the target class in terms of the W s,p norm (see
Appendix D.1 for definitions of Sobolev spaces and Sovolev norms).
Here p is a positive integer and s can vary in [0 , 1] ; in particular, s = 0 corresponds to function value approximation, and s =resembles the result Section 1.
We formally define our target function class as a Sobolev norm ball.
Assumption 1.
Let α ≥ 2 , 1 ≤ p ≤ + ∞ be integers.
Assume the target function f satisfies
[Equation]
We set the norm ball of radius 1 for the sake of simplicity, while the results in the sequel hold for any constant radius.
We also let α ≥ 2 for techincal convenience.
In the following theorem, we show that
ConvResNets can approximate any functions in a Sobolev norm ball in terms of W s,p norm ( s ≤ 1 ).
Theorem 2.
For any positive integers K ∈ [2 , D ] , (cid:102) M , and (cid:101) J > 0 , we choose
[-L = O (log( (cid:102) M (cid:101) J )) , J = O ( (cid:101) J ) , κ 
 = O (( (cid:102) M (cid:101) J ) 1 /D ) , κ 
-][Equation]
Then given s ∈ [0 , 1] , the ConvResNet architecture C ( M, L, J, K, κ 1 , κ 2 ) can approximate any function f satisfying Assumption 1, i.e., there exists (cid:101) f ∈ C ( M, L, J, K, κ 1 , κ 2 ) with
[Equation]
for some constant C 1 depending on D, α, p .
Theorem 2 says that the approximation power of ConvResNet amplifies as its width and depth increase.
This property is validated by experiments on adversarial robust image classification, see
Appendix C for details.
To better interpret the result, we choose s = 1 and p = ∞ , which corresponds to simultaneously approximating function value and first-order derivatives.
Corollary 1.
In the setup of Theorem 2, taking s = 1 and p = ∞ , the ConvResNet architecture
C ( M, L, J, K, κ 1 , κ 2 ) can approximate any f satisfying Assumption 1 up to first-order, i.e., there exists (cid:101) f ∈ C ( M, L, J, K, κ 1 , κ 2 ) with
[Equation]
and sup i
[Equation]
where the constant C 2 depends on D and α .
In particular, we have Lipschitz continuity bound
[Equation]
Theorem 2 and Corollary 1 have rich implications.
Large network for smooth approximation .
Taking s = 0 in Theorem 2 recovers function approximation in terms of L ∞ norm.
The approximation error scales as O (( (cid:102) M (cid:101) J ) − αD ) .
A quick comparison to Corollary 1 indicates that in order to additionally capture the first-order information of a target function, large network is needed to achieve the same function value error bound.
Arbitrary width and depth .
Gühring et al.
Hon and Yang (2021) provide approximation guarantees of feedforward networks in terms of W s,p norm.
Despite different network architectures, we remark that our theory covers general networks with arbitrary width and depth.
More specifically, for a given approximation error ϵ , Gühring et al.
(2020) set the network depth and width as O (log 1 /ϵ ) and O ( ϵ − D/ ( α − s ) ) , respectively.
Yet in our result, we only need to ensure (cid:102) M (cid:101)
J
= O
( ϵ − D/ ( α − s ) ) , which does not require any scaling relation between (cid:102) M and (cid:101) J .
Theorem 2 can be used as a tool to analyze the empirical residual error.
Specifically, assume the response in the data set contains bounded zero–mean noise, we have the following probability bound on the upper bound of the empirical residual error (see a proof in Appendix H)
Theorem 3.
Let { ( x i , y i ) } ni
=1 be a given data set where x i ’s are i.i.d.
samples from some distribution defined on [0 , 1] D and y i
= f ( x i ) + ξ i with i.i.d.
noise ξ i ’s satisfying E [ ξ i ] = 0 and | ξ i | ≤ σ for all i = 1 , ..., n .
Assume f satisfy Assumption 1 with p = + ∞ .
For 0 < ε < min { σ, 1 } , let
C = C ( M, L, J, K, κ 1 , κ 2 ) be the network architecture in Theorem 2 with (cid:102) M (cid:101)
J = ( ε/C 1 )
− D/α =
O ( ε − D/α ) .
We have
[Equation]
Theorem 3 implies that with high probability, larger network architectures ensure the existence of a network that has small empirical residual error as well as certain smoothness, i.e., a bounded
Lipschitz constant which is close to that of the underlying function.
Our result is an upper bound counterpart of Bubeck and Sellke (2021, Theorem 3), in which a high probability lower bound of the
Lipschitz constant is derived.
Theorem 2 is also closely related to adversarial robustness (see Appendix B for the definition), which is commonly used to measure the smoothness of a network.
Consider the supervised learning scenario in which noisy or noiseless response is generated by a ground truth function satisfying Assumption 1.
Corollary 1 indicates the existence of a properly large ConvResNet capable of smoothly approximating the data model.
The network’s Lipschitz continuity closely relates to adversarial risk, for which we have the following theorem (see a proof in Appendix I)
Theorem 4.
Let ρ be a data distribution defined on [0 , 1] D × [ − R, R ] for some constant R and l ( · , · ) be a loss function with Lipschitz constant L Lip .
Denote the population risk minimizer by f :
[Equation]
Assume f satisfies Assumption 1 with p = + ∞ .
For 0 < ε < 1 , let C ( M, L, J, K, κ 1 , κ 2 ) be the network architecture in Theorem 2 with (cid:102) M (cid:101)
J = (cid:16) εC 1 (cid:17) − D/α
= O ( ε − D/α ) .
Then there exists (cid:101) f ∈ C ( M, L, J, K, κ 1 , κ 2 ) so that ∥
[Equation]
and
[Equation]
In Theorem 4, the difference between the adversarial risk and population risk depends on the Lipschitz constant of the network (cid:101) f , the Lipscthiz constant of the loss function and the adversarial parameter δ .
It implies that large networks can give rise to smooth functions with a small adversarial risk
Approximation on Manifold
Theorem 2 indicates a curse of data dimensionality: When data dimension D is large, such as image data, Theorem 2 converges extremely slowly and becomes less attractive.
Motivated by applications, we model data as a low-dimensional Riemannian manifold M and extend our approximation theory to functions defined on M .
We will show that ConvResNet is adaptable to manifold structures.
We first impose some mild regularity conditions.
Assumption 2.
M is a d -dimensional compact Riemannian manifold isometrically embedded in R D .
There exists a constant B > 0 such that for any x ∈ M , we have ∥ x ∥ ∞ ≤ B .
Assumption 3.
The reach of M is τ > 0 .
See Appendix J for the definition of reach and some other concepts related to manifolds.
Similar to
Section 3, we consider a Sobolev norm ball on M as target function class.
Assumption 4.
Let α ≥ 2 be an integer.
Assume the target function f satisfies
[Equation]
We now present a counterpart of Theorem 2, showing an efficient approximation of functions on M .
Theorem 5.
For any positive integers K ∈ [2 , D ] , (cid:102) M , and (cid:101) J > 0 , we choose
[Equation]
=
[-+ D, J = O ( D (cid:101) J ) , κ 
 = O (( (cid:102) M (cid:101) J ) 1 /d ) , κ 
 = O (( (cid:102) M (cid:101) J ) 1 /d ) , M =-][Equation]
M ) [Equation]
Then given k ∈ { 0 , 1 } , the ConvResNet architecture C ( M, L, J, K, κ 1 , κ 2 ) can approximate any function f satisfying Assumption 4, i.e., there exists (cid:101) f ∈ C ( M, L, J, K, κ 1 , κ 2 ) with
[Equation]
where constant C 3 depends on d, α, B, τ , and the surface area of M .
As can be seen, the approximation error decays at a rate only depending on intrinsic data dimension d , which is a significant improvement over Theorem 2 given d ≪ D .
We also note that the size of
ConvResNet has a weak dependence on D , yet it is inevitable due to the residual connection preserves input dimensionality.
Conclusion
We provide universal approximation theories of Convolutional Residual Networks in terms of Sobolev norms.
Our theory applies to Sobolev function spaces defined on a high-dimensional hypercube or low-dimensional Riemannian manifold.
We demonstrate that deep and wide ConvResNets can provide approximation with good first-order smoothness properties.
This partially justifies why using large networks in practice often leads to better performance and robustness. Acknowledgment
The work of Hao Liu is partially supported by HKBU 162784 and HKBU 179356. The work of Wenjing Liao is partially supported by NSF CAREER 2145167. The work of Wenjing Liao and Tuo
Zhao is partially supported by DMS 2012652.
Appendix A Related work
Approximation theories of feedforward neural network have been studied for a long time, most of which dedicate to function value approximation.
The earliest literature dates back to late 1980s.
For example, Irie and Miyake (1988); Funahashi (1989); Cybenko (1989); Hornik (1991); Chui and Li (1992); Leshno et al.
(1993) investigated the approximation power of two-layer feedforward neural networks with sigmoidal activation for square integrable functions and established some asymptotic results, where the number of neurons goes to infinity.
Barron (1993); Mhaskar (1996) established nonasymptotic results for the so-called “Barron” function space.
For multi-layer feedforward neural networks with ReLU activation, Yarotsky (2017) analyzed the approximation of Sobolev W α, ∞ functions in a D -dimensional hypercube, and proved nonasymptotic results that given a pre-specified approximation error ϵ , the depth and width of neural networks need to be at most of the order
O ( ϵ − D/α ) and O (log(1 /ϵ )) , respectively.
More recently, Suzuki (2019); Suzuki and Nitanda (2019);
Liu et al.
(2021) extended to more general function classes such as Besov spaces.
Approximation theories for convolutional networks are established by Zhou (2020b,a); Petersen and Voigtlaender (2020).
In Zhou (2020b), the authors consider CNN with ReLU activation whose width increases linearly from the first layer to the last.
They show that such a CNN can approximate functions in Sobolev W α, 2 space with arbitrary accuracy for integer α ≥ 2 + D/ 2 .
To have a better control on the width of the network, the authors of Zhou (2020a) studied downsampled CNNs, and show that the downsampled CNN can approximate Lipschitz ridge functions with an arbitrary accuracy.
In Petersen and Voigtlaender (2020), the authors show that any approximation bounds of
FNN can be achieved by CNNs.
The results in Oono and Suzuki (2019); Liu et al.
(2021) dedicate to convolutional residual networks.
In Oono and Suzuki (2019), the authors show that ConvResNets is able to approximate Hölder functions with an arbitrary accuracy.
Theoretical results on approximating or learning functions on low-dimensional manifold can be found in Shaham et al.
(2018); Chui and Mhaskar (2018);
Schmidt-Hieber (2019); Chen et al.
(2019a,b, 2020); Nakada and Imaizumi (2019); Cloninger and Klock (2020);
Shen et al.
; Montanelli and
Yang (2020); Liu et al.
These works show that when the target function is defined on or around a low-dimensional manifold, to achieve an approximation error ϵ , the network size mainly depends on the intrinsic dimension and weakly depends on the ambient dimension.
B Adversarial risk
The adversarial risk (Uesato et al., 2018; Zhao et al., 2021) is defined as
Definition 1 (Adversarial risk) .
Given a data distribution ρ , and a loss function l ( · , · ) , for a positive constant δ > 0 , we define the adversarial risk of a network (cid:101) f as
[Equation]
where B δ ( x ) is the Euclidean ball with radius δ centered at x .
In the case δ = 0 , the adversarial risk R ( (cid:101) f, 0) reduces to the population risk
E ( x ,y ) ∈ supp(
[Equation]
) (cid:104)
[Equation]
(cid:16) (cid:101) f
( x ) ,
[Equation]
(cid:17)(cid:105) .
C Numerical Experiments
We verify our theory by numerical experiments.
Due to the complex structure of convolutional residual networks, directly estimating the Lipschitz constant is rather difficult.
We instead testing the adversarial robustness as an indication of the network smoothness.
We consider the TRADES model which uses a data driven smoothness regularization and encourages model smoothness.
By keeping the same clean testing accuracy, we can compare model smoothness through the robust testing accuracy.
We follow the setup in TRADES (Zhang et al., 2019), and report the performance of WideResNet (Zagoruyko and Komodakis, 2016) with different widening factor (WF) and number of convolutional layers per residual block (we term as “depth” in the sequel).
We use the CIFAR-10 data set.
Hyperparameters in training are set as follows: perturbation diameter ϵ = 0 .
031 under the ℓ ∞ norm, step size for generating perturbation 0 .
007 , number of iterations 10 , learning rate 0 .
1 , batch size b = 128 and run 76 epochs on the training dataset.
We run the White-box attacks by applying PGD attack with 20 iterations (PGD-20) and the step size is 0 .
We report the robust accuracy A rob and the natural accuracy A nat on the test data set.
The training objective is
[Equation]
where L is the cross entropy loss, R is the KL-divergence, x is the clean input, (cid:101) x is the adversarial input, y is the label, λ is the tuning parameter controlling the strength of the regularizer, and D denotes the training dataset { x i , y i } ni =1 .
For a fair comparison, we tune λ such that networks of different sizes achieve approximately the same natural accuracy.
This can be understood as achieving approximately the same L ∞ approximation error to the data model.
As can be seen in Table 1, A nat of different models about matches the performance in Zhang et al.
(2019), indicating the network has been sufficiently trained.
By comparing the robust accuracy A rob , we observe that wider and deeper WideResNet attains better robustness.
When fixing the depth, a wider network can achieve a higher robust accuracy.
Similarly, when fixing the widening factor, a deeper network can achieve a higher robust accuracy.
[Table]
D Notations and definitions
Notations : We use lower case letters to denote scalars, bold lower case letters to denote vectors, upper case letters to denote matrices, and calligraphic letters to denote tensors and sets.
For x = [ x 1 , ..., x D ] ⊤ , v = [ v 1 , ..., v D ] ⊤ , we denote x v = x v 1 1 · · · x v D D (if well-defined) and | v | = (cid:80) Di =1 | v i | .
Let α =
[ α 1 , ..., α D ] ⊤ ∈ N D be a multi-index and f be a function, we denote D α f = ∂ | α | f ∂x α 11 ··· ∂x αDD .
Let Ω be a subset in R D , we denote Ω as its closure and ch(Ω) as its convex hull.
We use B r ( c ) to denote the closed Euclidean ball with radius r and centered at c .
We use ∥·∥ ∞ to denotes the entrywise maximum norm, i.e., when the input argument is a vector, it returns the vector ℓ ∞ norm; when the input is a matrix or a tensor, it returns the maximum magnitude of its entries, e.g., for a 3 -dimensional tensor W , ∥W∥ ∞ =
max j,k,l |W j,k,l | .
D.1 Sobolev Functions
We focus on studying neural networks for approximating Sobolev functions.
We provide a formal definition of Sobolev functions in both Euclidean spaces and on manifolds.
We begin with Sobolev functions in Euclidean spaces (Brezis and Brézis, 2011, Chapter 8).
Definition 2 (Sobolev spaces) .
Let α ≥ 0 , 1 ≤ p ≤ ∞ be integers, and domain Ω ⊂ R D .
We define
Sobolev space W α,p (Ω) as
[Equation]
where α is a multi-index.
For f ∈ W α,p (Ω) , we define its Sobolev norm as
[Equation]
In the special case of p = ∞ , the Sobolev norm can be rewritten as ∥ f
∥
W α, ∞ (Ω) = max | α |≤ α ∥
D α f ∥ L ∞ (Ω) .
In this case, ∥ f ∥ W 0 , ∞ < ∞ implies the function value is bounded, and
∥ f ∥ W 1 , ∞ < ∞ implies both the function value and its gradient are bounded.
Our later approximation theories will provide error estimate in terms of Sobolev norms.
To allow more flexibility, we define fractional Sobolev norms, which can be viewed as a generalization of
Sobolev norms to non-integer α .
The fractional Sobolev functions are defined as follows.
Definition 3 (Sobolev–Slobodeckij spaces (Slobodeckij, 1958)) .
For 0 < s < 1 and 1 ≤ p ≤ ∞ , we define W s,p (Ω) as
[Equation]
with
[Equation]
for 1 ≤ p < ∞ and
[Equation]
We restrict our attention to s < 1 for simplicity, as we focus on approximation guarantees up to first-order continuity.
Next, we extend Sobolev spaces to Riemannian manifolds.
We provide a brief introduction to manifold; a more detailed description can be found in Appendix J. Roughly speaking, a Riemannian manifold M is a collection of local neighborhoods, each of which is diffeomorphic to a lowdimensional Euclidean space.
These local neighborhoods are termed charts, and a collection of which is an atlas.
We provide a formal definition.
Definition 4 (Atlas) .
A smooth atlas for a d -dimensional manifold M ⊂ R D is a collection of charts { ( U α , ϕ α ) } α ∈A , which verifies (cid:83) α ∈A U
α = M and ϕ α : U α (cid:55)→ R d being diffeomorphic and pairwise compatible, i.e.,
[Equation]
are both smooth for any α, β ∈ A .
An atlas is called finite if it contains finitely many charts.
To define Sobolev spaces on a manifold M , we shall consider function regularity on each chart, as charts are geometrically “akin” to a Eulidean space through the chart mapping ϕ α .
One caveat, however, is that the chart mapping ϕ
α can be arbitrarily rescaled, which results in potential unboundedness.
We therefore, fix an atlas on M to mitigate this issue.
We are ready to define Sobolev spaces on a manifold (Driver, 2003, Definition 48.17).
Definition 5 (Sobolev spaces on manifold) .
Let M be a compact Riemannian manifold of dimension d .
Let { ( U i , ϕ i ) } C M i
=1 be a finite atlas on M and { ρ i } C M i =1 be a partition of unity on M such that supp( ρ i ) ⊂
U i .
For integers k ≥ 0 and 1 ≤ p ≤ ∞ , a function f : M → R is in the Sobolev space
W k,p ( M ) if
[Equation]
Since M is compact, a finite altas exists on M .
Besides, we introduce the partition of unity ρ i to follow the standard definition in Tu (2010, Definition 13.4).
The existence of a smooth partition of unity is shown in Appendix J. From Definition 5, we observe that a Sobolev function on M is locally
Sobolev on each chart.
D.2 Convolutional Residual Networks
We consider one-sided stride-one convolution in our network.
Let W =
W j,k,l } ∈ R C ′
× K × C be a filter where C ′ is the output channel size, K is the filter size and C is the input channel size.
For
Z ∈ R D × C , the convolution of W with Z gives Y = W ∗ Z ∈ R D × C ′ with
Y i
[Equation]
j = [Equation] where we set Z i + k − 1 ,l = 0 for i + k − 1 > D .
See a graphical demonstration in Figure 1(a).
In this paper, we study convolutional residual networks (ConvResNets) equipped with the rectified linear unit ( ReLU ) activation function ( ReLU( z ) = max( z, 0) ).
The ConvResNet we consider consists consecutively of a padding layer, several residual blocks, and finally a fully connected output layer.
Given an input vector x ∈ R D , the network first applies a padding operator P : R D
→ R D × C for some integer C ≥ 1 such that
[Equation]
Then the matrix Z is passed through M residual blocks.
To ease the notation, we denote the input matrix to the m -th block as Z m and its output as Z m +
(Consequently, Z 1 = Z ).
In the m -th block, let W m =
{W (1) m , ..., W ( L m ) m } and B m =
{ B (1) m , ..., B ( L m ) m } be a collection of filters and biases of proper sizes.
The m -th residual block maps its input matrix Z m from R D × C to R D × C by the operator
[Equation]
where id is the identity mapping (also known as the shortcut connection) and
[Equation]
with ReLU applied entrywise.
We denote the mapping from input x to the output of the M -th residual block as
[Equation]
[Figure]
Given (5), a ConvResNet applies an additional fully connected layer to Q and outputs
[Equation]
where W ∈ R D × C and b ∈ R are a weight matrix and a bias, respectively, and ⊗ denotes sum of entrywise product, i.e., W ⊗ Q ( x ) =
(cid:80) i,j W i,j [ Q ( x )] i,j .
E Proof Sketch
We highlight key steps in establishing Theorem 2 and 5 in this section.
Full proofs are deferred to
Appendix G and K, respectively.
E.1 Proof Sketch of Theorem
The main idea consists of two stages: 1) Approximating target function f in terms of W s,p norm using a sum of averaged Taylor polynomials; 2) Implementing the sum of averaged Taylor polynomials by a given width and depth ConvResNet up to a certain error.
In stage 1), we rely on tools from the finite element anaylsis to quantify approximation error.
In stage 2), we first represent polynomials using convolutional networks, and then assemble them according to the specified width and depth as a ConvResNet.
We dive into the following four steps.
Step 1:
Decompose f using a partition of unity.
Given the network size parameter (cid:102) M and (cid:101) J , we define a partition of unity { ϕ j } N D j =1 on (0 , 1) D for an integer N =
O
(( (cid:102) M (cid:101) J ) 1 /D ) , so that each ϕ
j is supported on a small hypercube of edge length 43 N .
The function f is decomposed into f = (cid:80) N D j =1 f j with f j = fϕ j .
See Figure 2(a) for an illustration.
Step 2: Averaged Taylor polynomial approximation.
Each f j is a Sobolev function, which may not have classical derivatives but weak derivatives.
Similar to approximating differentiable functions by Taylor polynomials, we approximate f j by an averaged Taylor polynomial (cid:98)
f j , which is defined in an integral form and indeed is a polynomial.
The approximation error of averaged Taylor polynomial is similar to that of using Taylor polynomial, and can be found in Lemma 2.
Step 3: Network implementation.
As shown in Lemma 16 and 5, CNN can approximate multiplication and compositions of muliplications well.
Since a polynomial is a sum of compositions of multiplication, each (cid:98) f i can be approximated by a sum of O (1) CNNs, and therefore (cid:80) N D i =1 (cid:98)
f i is approximated by a sum of O ( N D ) CNNs, each of which has width of O (1) .
We prove in Lemmathat such a sum can be realized by a sum of (cid:102) M CNNs with width (cid:101) J .
The new sum can be realized by a ConvResNet with (cid:102) M residual blocks (Lemma 8), where each summand corresponds to a residual block and the sum is realized using skip-layer connections.
Step 4: Error estimation.
To estimate the approximation error of (cid:101) f , we decompose the error as
[Equation]
On the right-hand side of (6), the second term is the approximation error of averaged Taylor polynomial, whose upper bound is given by Lemma 4.
The first term is the network implementation error.
We derive an upper bound of it in Lemma 6.
In the proof of Lemma 6, we first derive an upper bound with respect to the W k,p norm for k = 0 , 1 .
The case k
= 0 corresponds to the error of function value approximation, and the case k = 1 corresponds to the error of first order weak derivative approximation.
Note that each (cid:98) f
j is a polynomial, and each (cid:101)
f j consists of compositions of (cid:101) × , the network approximation of multiplication × .
The error indeed is the approximation error of compositions of (cid:101) × .
We first derive the W k, ∞ approximation error of (cid:101) ×
and then show that compositions of (cid:101) × have W k,p approximation errors of the same order.
After the upper bounds of W 0 ,p and W 1 ,p errors are derived, these upper bounds are generalized to W s,p errors using an argument on interpolation spaces, which is discussed in Appendix L.2.
Combining the upper bounds of both terms in (6) gives rise to the total approximation error as a function of N .
Utilizing the relation (cid:102) M (cid:101)
J = O ( N D )
, we can further express the approximation error in terms of number of blocks and width of the ConvResNet.
E.2 Proof Sketch of Theorem
We exploit the geometric nature of manifold M and Sobolev functions on it to prove Theorem 5.
By an explicit construction of a finite atlas on M based on the curvature condition in Assumption 3, we first restrict ourselves to a single chart on M .
Recall Definition 5 that a Sobolev function f on M is locally Sobolev on a chart.
We are thus, able to locally approximate f on each chart by the results in
[Figure]
Theorem 2.
However, the main challenge stems from combining these local approximations to obtain a global guarantee.
This requires to determine which charts a given input belongs to.
We develop a chart determination sub-network for approximating indicator functions of charts, nonetheless, its
Lipschitz continuity is troublesome due to the sharp jump on the boundary of a chart.
We resolve such an issue by carefully constructing a partition of unity vanishing at a neighborhood of the boundary of charts.
We provide more details in the following four steps.
Step 1:
Decompose f using an atlas and partition of unity of M .
We first construct an atlas and a partition of unity of M so that each function in the partition of unity is compactly supported in a chart (Lemma 9).
To construct an atlas of M , we use a set of D -dimensional Euclidean balls { B r/ 2 ( c i ) } C M i =1 with centers { c i } C M i =1 ⊂ M and radius r/ 2 satisfying 0 < r < τ/ 4 to cover M .
Since M is compact, C M is finite.
The collection of intersections between each ball and M , denoted by { (cid:101) U i } C M i =1 with (cid:101) U i
=
B r/ 2 ( c i ) ∩ M , forms an open cover of M .
It is guaranteed that there exists a
C ∞ partition of unity { ρ i } M i =1 so that ρ i is supported in (cid:101)
U i (Lemma 19).
We then double the radius and denote U i
=
B r
( c i ) ∩ M .
The collection { U i } C M i =1 is also an open cover of M .
Since (cid:101) U i
⊂ U i ,
ρ i is compactly supported in U i and the distance between the support of ρ i and ∂U i is at least r/ 2 .
For each U i , an orthogonal projection φ i with proper scaling and shifting, which projects any x ∈ U i to a tangent plane, is constructed so that φ i ( U i ) ⊂ (0 , 1) d .
See the proof of Lemma 9 for details.
With this construction, we illustrate U i and ρ i in Figure 2(b).
We then focus on the atlas { U i , φ i } C M
i
= and partition of unity { ρ i } M i =1 .
We decompose f as f = (cid:80) C M
i
=
( f i ◦ φ − 1 i ) ◦
φ i with f i =
fρ i .
Step 2: Averaged Taylor polynomial approximation.
In the decomposition in Step 1, each f i ◦ φ − 1 i is a Sobolev function compactly supported in φ i ( U i ) ⊂ (0 , 1) d .
Extend f
i ◦ φ − 1 i to (0 , 1) d by 0.
The extended function has the same smoothness as f i ◦ φ − 1 i , and can be approximated by a sum of local averaged Taylor polynomials (cid:80) N d i =1 (cid:98)
f i,j , as what has been done in the proof of Theorem 2.
Step 3: Network implementation.
Each polynomial (cid:98) f i,j can be approximated by a CNN (cid:101)
f i,j .
Since we are only interested in the value of (cid:101) f i,j ◦ φ i ( x ) when x ∈ U i , we need to determine the chart it belongs to.
We accomplish this by introducing a chart determination function 1 i ( x ) = 1 [0 ,r 2 ] ◦ d 2 i ( x ) , where 1 [0 ,r 2 ] ( a ) is a step function which outputs 1 when a ∈ [0 , r 2 ] and outputs 0 otherwise, d 2 i ( x ) computes the squared Euclidean distance between x and c i .
The squared distance function d 2 i can be approximated by a CNN with high accuracy.
To approximate the step function 1 [0 ,r 2 ] , we construct a
CNN which outputs 1 on [0 , r 2 − ∆] , 0 on [ r 2 , ∞ ) and is linear on [ r 2 − ∆ , r 2 ] for some small ∆ .
The
CNN approximation of 1 i , denoted by (cid:101)
i , is illustrated in Figure 3(a).
Our network approximation of f is constructed as
[Equation]
where (cid:101) × denotes the CNN approximation of multiplication.
By Lemma 7 and 8, (cid:101) f can be realized by a ConvResNet with (cid:102) M blocks and width of O ( (cid:101) J ) as long as (cid:102) M (cid:101)
J
= O ( N d ) .
Step 4: Error estimation.
We decompose the error into two parts: 1) the error between f and its averaged Taylor polynomial approximation, and 2) the error between the averaged Taylor polynomial and its network approximation, see (39) in Appendix K.
The first part can be bounded using Lemma 4.
The second part is characterized by the approximation error of (cid:101) × for multiplication, of (cid:102)
f i,j for averaged Taylor polynomials, and of (cid:101)
i for chart determination 1 i .
The first two errors can be bounded using techniques similar to those in the proof of Theorem 2.
[Figure]
For the approximation error of (cid:101)
i , bounding its W 1 , ∞ norm is the most challenging task.
To derive an upper bound, one needs to bound | ( (cid:101)
f i,j ◦ φ i ) × ( ∂ ( (cid:101)
i ◦ φ − 1 i ) /∂z l ) | for l = 1 , ..., d and z ∈ φ i ( U i ) .
In our network construction, (cid:101) 1 i ◦ φ − 1 i is linear on a narrow band, denoted by Ω i, 2 , with width of
O (∆) .
Its weak derivative on the narrow band is of O (1 / ∆) , which blows up as ∆ → 0 and causes problems.
To eliminate the effect of ∆ , we show that the value of (cid:101)
f i,j ◦ φ i is small enough so that its product with ∂ ( (cid:101) 1 i ◦ φ − 1 i ) /∂z
l does not blow up as ∆ → 0 .
Specifically, thanks to the fact that f i is compactly supported on U i , we have f i ◦ φ − 1 is compactly supported on φ i ( U i ) .
Therefore there exists another band Ω i, 1 adjacent to φ i ( ∂U i ) so that f i ◦ φ − 1 = 0 on Ω i, 1 .
We choose ∆ small enough so that Ω i, 2 ⊂ Ω i, 1 , and (cid:98) f i,j and all of its first order weak derivatives vanish on Ω i, 2 , see
Figure 3(a) and (b) for illustrations.
Note that (cid:101)
f i,j is an approximation of (cid:98) f
i,j .
We can show that (cid:101) f
i = 0 on φ i ( ∂U i ) , and all of its first order weak derivatives on Ω i, 2 are in the same order of other error terms.
Since the width of Ω i, 2 is of O (∆) , by Taylor’s theorem, | (cid:101)
f i,j ◦ φ i | is bounded by a linear function of ∆ on Ω i, 2 .
With such a construction and proper choice of ∆ , the resulting upper bound is in the same order of those of other terms.
See Lemma 11 for details.
Combining all of the error bounds, we can express the error in terms of N .
Substituting the relation (cid:102) M (cid:101)
J
= O
( N d ) proves Theorem 5.
F Convolutional neural networks and multi-layer perceptions
Our proofs are based on approximation theories of convolutional neural networks (CNN) and their relations to multi-layer perceptions (MLP).
In this section, we introduce related notations and definitions.
For the convenience of notation, we use · to denote ⊗ , the sum of entrywise product.
We consider CNNs in the form of
[Equation]
where Conv W , B ( Z ) is defined in (4), W is the weight matrix of the fully connected layer, W , B are sets of filters and biases, respectively.
We define the class of CNNs as
F CNN ( L, J, K, κ 1 , κ 2 ) = (cid:8) f | f ( x ) in the form (7) with L layers.
Each convolutional layer has filter size bounded by K .
The number of channels of each layer is bounded by J.
[Equation]
For MLP, we consider the following form
[Equation]
where W 1 , . .
, W L and b 1 , . . .
, b L are weight matrices and bias vectors of proper sizes, respectively.
The class of MLP is defined as
[Equation]
x ) in the form [Equation] 8) with L -layers and width bounded by J
[Equation]
In some cases it is necessary to enforce the output of the MLP to be bounded.
We define such a class as
[Equation]
In some case we do not need the constraint on the output, we denote such MLP class as
F MLP ( L, J, κ ) .
G Proof of Theorem
Before we prove Theorem 2, we define the Sobolev semi-norm:
Definition 6.
For any integers 0 ≤ k ≤ α , 1 ≤ p < ∞ and function f ∈ W α,p (Ω) , we define its
Sobolev semi-norm as
[Equation]
Now we prove Theorem 2.
Proof of Theorem 2.
We prove Theorem 2 in four steps.
Step 1: Decompose (0 , 1) D using locally supported functions.
We define ψ ( x ) =
  | x | < 1 ,
[Equation]
< | x | , 2 − | x | 1 ≤ | x | ≤and ϕ m ( x ) =
D (cid:89) k
= ψ (cid:16) 3 N (cid:16) x k − m k N (cid:17)(cid:17) with m =
( m 1 , m 2 , ..., m D ) ∈ { 0 , ..., N } D .
We have (cid:80) m ϕ m = 1 on (0 , 1) D and ϕ m is supported on B 23 N, ∥·∥ ∞ ( m N ) ⊂
B 1 /N, ∥·∥ ∞ ( m N ) .
We denote S N = { 0 , 1 , ..., N } D .
The following lemma shows that each ψ (cid:0) 3 N (cid:0) x k − m k N (cid:1)(cid:1) can be realized by a CNN (see a proof in Appendix L.3).
Lemma 1.
There exists a CNN architecture F CNN ( L, J, K, κ 1 , κ 2 ) such that for any N, m , such an architecture yields a CNN (cid:101) ψ with
[Equation]
Such an architecture has
[Equation]
= O [Equation] N ) . Further more, the weight matrix in the fully connected layer of F CNN has nonzero entries only in the first row.
We then decompose f as
[Equation]
Step 2: Approximate each ϕ
m f using averaged Taylor polynomials.
On each B 1 /N, ∥·∥ ∞ ( m N ) , we approximate ϕ m f by an averaged Taylor polynomial.
The averaged Taylor polynomial is defined as follows:
Definition 7 (Averaged Taylor polynomials) .
Let α > 0 , 1 ≤ p ≤ + ∞ be integers and f ∈
W α − 1 ,p (Ω) .
For x 0 ∈ Ω , r > 0 such that B r, ∥·∥ ( x 0 ) is compact in Ω , the corresponding Taylor polynomial of order α of f averaged over B r, ∥·∥ ( x 0 ) is defined as
[Equation]
with
[Equation]
and ϕ being arbitrary cut-off function satisfying ϕ ∈ C ∞ c ( R D ) with ϕ ( x ) ≥ 0 for all x ∈ R D , supp( ϕ ) =
B r, ∥·∥ ( x 0 ) and
[Equation]
where C ∞ c ( R D ) denotes the space of infinitely differentiable functions on R D with compact support.
Under proper assumptions, the averaged Taylor polynomial can approximate f and its partial derivatives well.
We first define the star-shaped sets and chunkiness parameter, which are used in the error estimation result.
Definition 8 (Star-shaped sets, Definition 4.2.2 of Brenner et al.
Let Ω , (cid:101) Ω ⊂ R D .
Then Ω is called star-shaped with respect to (cid:101) Ω if for all x ∈ Ω , we have
[Equation]
Definition 9 (Chunkiness parameter, Definition 4.2.16 of Brenner et al.
Let Ω ⊂ R D be bounded.
Define
R = (cid:8) r > 0 : there exists x ∈ Ω such that Ω is star-shaped with respect to B r, |·| ( x ) (cid:9) .
For R ̸ = ∅ , we define r
[Equation]
and
[Equation]
where γ is called the chunkiness parameter of Ω .
The following lemma gives an error estimation of averaged Taylor polynomials:
Lemma 2 (Bramble-Hilbert, Lemma 4.3.8 of Brenner et al.
Let Ω ⊂ R D be open and bounded, x ∈ Ω and r > 0 such that Ω is star-shaped with respect to B r, ∥·∥ ( x 0 ) and r > 12 r ∗ max , with r ∗ max defined in Definition 9.
Let n > 0 , 1 ≤ p ≤ + ∞ be integers and γ be the chunkiness parameter of Ω .
Then we have
[Equation]
for k = 0 , 1 , ..., α , where h = diam(Ω) and C is a constant depending on D, α, γ .
Lemma 3 below shows that Q α f can be written as a weighted sum of polynomials.
Lemma 3 (Lemma B.9 of Gühring et al.
Let α > 0 , 1 ≤ p ≤ + ∞ be integers and f ∈ W α − 1 ,p (Ω) .
Let x 0 ∈ Ω , r > 0 such that B r, ∥·∥ ( x 0 ) is compact in Ω , and there exists (cid:101) r >with B r, ∥·∥ ( x 0 )
⊂ B (cid:101) r, ∥·∥ ∞ (0) .
Then the averaged Taylor polynomial Q α x 0 ( f ) can be written as
[Equation]
11) for x ∈ Ω .
There exists a constant C depending on α, D, (cid:101) r such that
[Equation]
for all | v | ≤ α − 1 .
Using averaged Taylor polynomials, we approximate ϕ m f by
[Equation]
|
[Equation] ≤ α [Equation] c
[Equation]
|
[Equation]
≤ α
[Equation]
c
[Equation]
[Equation]
Define
[Equation]
where c m , v ’s are the coefficients in (11).
Then (cid:98) f is an approximation of f .
The following lemma gives an upper bound on the approximation error
Lemma 4 (Lemma C.4 of Gühring et al.
Let α ≥ 2 be an integer and 1 ≤ p ≤ ∞ .
For any s ∈ [0 , 1] and f ∈ W α,p ((0 , 1) D ) , one has
[Equation]
where C is a constant depending on α, p, D .
Furthermore, the coefficients in (cid:98) f satisfies
[Equation]
for some constant C 1 depending on D, α, p .
Step 3: Network approximation Note that (cid:98) f is a sum of functions in the form of ϕ m x v with weights c m , v ’s.
We next approximate each ϕ m x v by a CNN.
Lemma 5.
For any 0 < ε < 1 , x ∈ (0 , 1) D , N > 0 , m ∈ { 0 , 1 , ..., N } D , | v | < α , there exists a
CNN architecture F CNN ( L, J, K, κ, κ ) that yields a CNN (cid:101) g with ∥ (cid:101) g m , v ( x ) − ϕ m
[Equation]
for k = 0 , 1 , where C 2 is a constant depending on α, k .
Such an architecture has
[Equation]
The constants hidden in O depends on α, k .
Further more, the weight matrix in the fully connected layer of F CNN has nonzero entries only in the first row.
Lemma 5 is proved in Appendix L.4.
By Lemma 5, each ϕ m x v can be approximated by a CNN.
Denote the network approximation of ϕ m x v by (cid:101) g m , v ( x ) .
We approximate (cid:98) f by (cid:101)
f defined as
[Equation]
The following lemma gives an upper bound of the approximation error of (cid:101) f
(see a proof in Appendix
L.6).
Lemma 6.
Let α ≥ 2 and 1 ≤ p ≤ ∞ be integers.
For any f ∈ W α,p ((0 , 1) D ) , let ϕ m Q α m /N
f ( x ) be the averaged Taylor approximation of ϕ m f defined in (12).
For any 0 < η < 1 , let (cid:101) g m , v be the
CNN approximation of ϕ m Q α m /N
f ( x ) constructed in Lemma 5 with accuracy η .
For 0 ≤ s ≤ 1 , we have
[Equation]
where c m , v ’s are coefficients defined in (12), C 3 is a constant depending on D, α, s, p .
Note the (cid:101) f is the sum of no more than N D
( D + 1) α − 1 CNNs of which the width is of J
= O ( D ) .
The following lemma shows that under appropriate conditions, the sum of n 0 CNNs with width in the same order can be realized by the sum of n 1 CNNs with a proper width (see a proof in Appendix
L.8):
Lemma 7.
Let { f i } n
i
=1 be a set of CNNs with architecture F CNN ( L 0 , J 0 , K 0 , κ 0 , κ 0 ) .
For any integers 1 ≤ n ≤ n 0 and (cid:101) J satisfying n (cid:101) J
= O ( n 0 J 0 ) and (cid:101) J ≥ J 0 , there exists a CNN architecture
F CNN ( L, J, K, κ, κ ) that gives a set of CNNs { g i } ni =1 such that n (cid:88) i = g [Equation] ( x [Equation]
=
[Equation] (cid:88) i = f [Equation]
( x ) .
Such an architecture has
[Equation]
Furthermore, the fully connected layer of f has nonzero elements only in the first row.
By Lemma 7, for any (cid:102) M, (cid:101) J satisfying (cid:102) M (cid:101) J
= O ( N D )
, there exists a CNN architecture
F CNN ( L, J, K, κ, κ ) that gives rise to { g i } ni =1 with (cid:101) f
= (cid:102) M (cid:88)
i = g i , where
L = O (cid:18) logη (cid:19) , J = O
( (cid:101) J ) , κ = 3 N.
The following lemma shows that the sum of CNNs can be realized by a ConvResNet:
Lemma 8 (Lemma 18 in Liu et al.
Let F CNN ( L, J, K, κ 1 , κ 2 ) be any CNN architecture from R D to R .
Assume the weight matrix in the fully connected layer of F CNN ( L, J, K, κ 1 , κ 2 ) has nonzero entries only in the first row.
Let M be a positive integer.
There exists a ConvResNet architecture C ( M, L, J, κ 1 , κ 2 (1 ∨ κ − 1 1 )) such that for any { f i ( x ) } Mi =1 ⊂ F CNN ( L, J, K, κ 1 , κ 2 ) , there exists (cid:101) f ∈ C ( M, L, J, κ 1 , κ 2 (1 ∨ κ − 1 1 )) with (cid:101) f
( x ) =
M (cid:88) i =
f i ( x ) .
By Lemma 8, there exits a ConvResNet architecture C ( M, L, J, K, κ 1 , κ 2 ) with
L =
O (log 1 /η ) , J =
O ( (cid:101) J ) , κ 1 = O
(3 N ) , κ
= O (3 N ) , M =
O ( (cid:102) M ) (18) and (cid:101) J, (cid:102) M satisfying (cid:102) M (cid:101) J
= O ( N D ) , (19) that yields a ConvResNet realizing (cid:101) f .
Step 4: Error estimation.
We compute
[Equation]
N
[Equation]
where C 4 , C 5 are two constants depending on D, α, s, p, R .
In the second inequality, we use Lemma 4 and 6 for the first and second term, respectively.
In the third inequality, we set η = N − α to balance the two terms.
Using the relation (19), we have
[Equation]
Substituting (21) into (20) gives rise to
[Equation]
[Equation]
for some constant C 6 depending on D, α, s, p, R .
Substituting (21) into (18) and (19) gives rise to the network architecture
[Equation]
H Proof of Theorem
Proof of Theorem 3.
By Theorem 2 and the choice of (cid:102) M (cid:101) J , there exits (cid:101) f ∈ C so that ∥ (cid:101) f − f ∥ ∞ ≤ ε and
[Equation]
) which implies
[Equation]
We have
[Equation]
≤ ε 2 + σ
[Equation]
Denote X i =
1 n ( (cid:101) f ( x i ) − y i )
2 − E (cid:104) ( (cid:101) f
( x i ) − y i ) 2 (cid:105) .
We have
[Equation]
26) and
[Equation]
By Bernstein inequality, we deduce
P
[Equation]
[Equation]
Therefore
[Equation]
Setting t = ε 2 gives rise to
[Equation]
I Proof of Theorem
Proof of Theorem 4.
By Theorem 2 and the choice of (cid:102) M (cid:101) J , there exits (cid:101) f ∈ C so that ∥ (cid:101) f − f ∥ ∞ ≤ ε and
[Equation]
Since ∥ f ∥ W α, ∞ ≤ 1 , we have
[Equation]
We have
[Equation]
J A Brief Introduction to Manifold
We introduce some concepts and quantities that characterize a low-dimensional Riemannian manifold. (Some are restatements of the main text for completeness.)
These concepts and quantities are used in our theorems and proofs.
We refer readers to Lee (2006);
Tu (2010) for more details.
Let M be a d -dimensional manifold embedded in R D with d ≤ D .
The first concept related to manifolds is chart, which defines a local coordinate neighborhood of a manifold.
Definition 10 (Chart) .
A chart on M is a pair ( U, ϕ ) where U ⊂ M is open and ϕ : U → R d , is a homeomorphism (i.e., bijective, ϕ and ϕ − 1 are both continuous).
In a chart ( U, ϕ ) , U is called a coordinate neighborhood and ϕ is a coordinate system on U .
A collection of charts which covers M is called an atlas of M .
Definition 11 ( C k Atlas) .
A C k atlas for M is a collection of charts { ( U α , ϕ α ) } α ∈A which satisfies (cid:83) α ∈A
U
α = M , and are pairwise C k compatible, i.e.,
[Equation]
are both C k for any α, β ∈ A .
An atlas is called finite if it contains finitely many charts.
With the concept of atlas, we then define smooth manifolds:
Definition 12 (Smooth Manifold) .
A smooth manifold is a manifold M together with a C ∞ atlas.
Simple examples of smooth manifold include the Euclidean space, the torus and the unit sphere.
C s functions on a smooth manifold M are defined as follows:
Definition 13 ( C s functions on M ) .
Let M be a smooth manifold and f : M → R be a function on M .
We say f is a C s function defined on M , if for every chart ( U, ϕ ) on M , the function f ◦ ϕ − 1 : ϕ
( U ) → R is a C s function.
We next define the C ∞ partition of unity which is an important tool for the study of functions on manifolds.
Definition 14 (Partition of Unity) .
A C ∞ partition of unity on a manifold M is a collection of C ∞ functions { ρ α } α ∈A with ρ α : M → [0 , 1] such that for any x ∈ M ,
there is a neighbourhood of x where only a finite number of the functions in { ρ α } α ∈A are nonzero, and 2.
(cid:88)
α ∈A ρ α ( x ) = 1 .
An open cover of M is called locally finite if every x ∈ M has a neighbourhood that intersects with a finite number of sets in the cover.
For a locally finite cover of a smooth manifold M , there always exists a C ∞ partition of unity subordinate to the cover (Spivak, 1973, Chapter 2, Theorem 15).
Proposition 1 (Existence of a C ∞ partition of unity) .
Let { U α } α ∈A be a locally finite cover of a smooth manifold M .
There is a C ∞ partition of unity { ρ α } ∞ α =1 such that supp( ρ α ) ⊂
U α .
Let { ( U α , ϕ α ) } α ∈A be a C ∞ atlas of M .
Proposition 1 guarantees the existence of a partition of unity { ρ α } α ∈A such that ρ α is supported on U α .
The reach of M introduced by Federer (Federer, 1959) is an important quantity defined below.
Let d ( x , M ) = inf y ∈M ∥
x − y ∥ 2 be the distance from x to M .
Definition 15 (Reach (Federer, 1959; Niyogi et al., 2008)) .
Define the set
G = { x ∈ R D : ∃ distinct p , q ∈ M such that
[Equation]
The closure of G is called the medial axis of M .
The reach of M is defined as
[Equation]
K Proof of Theorem
Proof of Theorem 5.
We prove Theorem 5 in three steps.
Step 1: Decomposition of f
• Construct an atlas on M .
According to Assumption 2, M is bounded.
Therefore, for any given 0 < r < τ/ 2 , we can find a finite collection of points c i } C M i =1 ⊂ M such that
[Equation]
Denote U i
=
B r
( c i ) ∩ M .
Then { U i } C M i =1 form an open cover of M and each U i is diffeomorphic to an open subset of R d .
The total number of partitions if bounded by C M ≤ (cid:108) SA( M ) r d
[Equation]
(cid:109) , where
SA( M ) is the surface area of M and T d is the average number of U
i ’s that contain a given point on
M .
On each U i , we define a transformation ϕ
i that projects any x ∈ U i to T c
i ( M ) , the tangent space of M at c i .
Let V
i ∈
R D ×
d be an orthogonal matrix whose columns form an orthonomal basis of
T c i ( M ) .
Define
[Equation]
34) where a i ∈ R is a scaling factor and b ) i ∈
R
d is a shifting vector that ensure φ i ( U i ) ⊆
[0 , 1] d .
Then { ( U i , φ i ) } C M i =1 form an atlas of M .
• Decomposition of f by a partition of unity.
The following lemma shows that under proper assumption, there exists a partition of unity { ρ i } C M i =1 subordinate to { ( U i , φ i ) } C M i =1 (see Appendix M.1 for a proof).
Lemma 9.
Let { ( U i , φ i ) } C M i =1 be the atlas of M defined above with r <
τ/ 4 .
There exist a finite number C M and a C ∞ partition of unity { ρ i } C M i =1 satisfying
(i) supp( ρ i ) is compact in U i .
(ii) (cid:80) C M i
=
ρ
i
( x ) = 1 for any x ∈ M .
(iii)
There exists a constant c > 0 depending on r such that for any i , we have
[Equation]
Here C M depends on the surface area of M and the average number of U
i ’s that contain a given point on M .
Let { ρ i } C M i =1 be the partition of unity from Lemma 9.
Since for each i , φ i is a bijection from U i to a subset of [0 , 1] d , φ − 1 exists and is a linear operator.
We decompose f as f =
[Equation]
i = f
i with f i = ( fρ i ) .
Here each f i is compactly supported on U i and each f i ◦ φ − 1 i is compactly supportedin φ
i ( U i ) ⊆
[0 , 1] d .
We extend f i ◦ φ − 1 i by 0 on [0 , 1] d \ φ
i ( U i ) .
The extended function is in W α,k
([0 , 1] d ) .
To simplify the notation, we still use f i ◦ φ − 1 i to denote the extended function.
For each i , we use averaged Taylor polynomials to approximate f i ◦ φ − 1 on [0 , 1] d as in (13): f i [Equation] φ − 1 i [Equation]
(cid:98) f i = (cid:88)
[Equation]
Step 2: Network approximation
• Approximate (cid:98) f i by CNNs.
Since each (cid:98) f i is the averaged Taylor polynomial approximation of f i ◦ φ − 1 i , by Lemma 5, it can be approximated by a sum of ( d + 1) α − 1 N d CNNs.
Denote the approximation accuracy by η as in Lemma 5, each CNN has depth O (log(1 /η )) , width O (1) , all weight parameters are of O ( N ) .
• Chart determination For any input x , to determine the chart it belongs to, we are going to construct an indicator function.
With our construction of charts, we have x ∈ U i if and only if ∥ x − c i ∥ 22 ≤ r 2 .
Define the indicator function
[Equation]
≤ [Equation] 2 , otherwise,
and the squared distance function
[Equation]
[Equation]
where we used the expression x = [ x 1 , ..., x D ] ⊤ and c i
=
[ c i, 1 , ..., c i,D ] ⊤ .
The compositioni = 1 [0 ,r 2 ] ◦ d 2 i outputs 1 if x ∈ U i and outputs 0 otherwise.
We are going to construct a CNN to approximate 1 i .
In (35), the function d 2 i is a sum of D square functions.
By Lemma 16, For any 0 < θ < 1 / 2 , x ∈ [ − B, B ] , and K ≥ 2 , there is a CNN architecture F CNN ( L, J, K, κ, κ ) that yields a CNN, denoted by (cid:101) d 2 , such that
[Equation]
Such a network has
[Equation]
Furthermore, one has
[Equation]
for some absolute constant C 7 .
We approximate d i by
[Equation]
According to Lemma 7, (cid:101)
d
i can be realized by a CNN with O (cid:0) log 1 θ (cid:1) layers, O ( D ) width and all weight parameters of O (1) .
The approximation error is bounded as
[Equation]
(cid:101)
[Equation]
Dθ. The following Lemma shows that 1 [0 ,r 2 ] can be approximated by a CNN:
Lemma 10 (Lemma 9 of Liu et al.
For any 0 < θ < 1 and ∆ ≥ 8 B 2 Dθ , there exists a
CNN (cid:101) 1 ∆ approximating 1 [0 ,ω 2 ] with (cid:101) 1 ∆ ( x ) =
  1 , if a ≤ (1 − 2 − w )( r 2 − 4 B 2 Dθ ) , 0 , if a ≥ r 2 − 4 B 2 Dθ, 2 w (( r 2 − 4 B 2 Dθ ) − 1 a − 1) , otherwise for x ∈ M , where w = (cid:6) log( r 2 / ∆) (cid:7) such that (1 − 2 − k )( ω 2 − 4 B 2 Dθ ) ≥ ω 2 − ∆+4 B 2 Dθ .
Such a CNN has (cid:6) log( r 2 / ∆) (cid:7) + D layers, 2 channels.
All weight parameters are of O (1) .
Let 1 ∆ be the CNN defined in Lemma 10.
We have
[Equation]
for some constant C 8 depending on r .
The function 1 i is approximated by
[Equation]
Combining (37) and (36) gives rise to
[Equation]
Step 3: Error analysis.
Our network approximation of f is
[Equation]
i with
[Equation]
where (cid:101) g m , v is the CNN approximation of ϕ m z v for z ∈ [0 , 1] d as in (16).
We decompose the error as ∥ (cid:101) f
[Equation]
The second term can be bounded using Lemma 4.
We next focus on the first term
[Equation]
[Equation]
[Equation]
−
[Equation]
(cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
[Equation]
(cid:3)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
=
∥
[Equation]
with
[Equation]
Denote the W 1 , ∞ error of (cid:101) × by δ .
We first derive an upper bound for A 1 .
We can show that ∥ (cid:101) g m , v ∥ ∞ ≤ α + d (see (66)) and ∥ (cid:101) 1 i ◦ φ − 1 i ∥
L ∞ = 1 .
Therefore by Lemma
[Equation]
we have for k =[-
-]
[Equation]
and for k [-= 
-]=|
A 1 |
[Equation]
for some constants C 9 , C 10 , C ′ depending on r, α, d , where we used Lemma 4 and (62) in the last inequality.
Combining (44) and (45) gives rise to
[Equation]
for k = 0 , 1 and a constant C 11 depending on d, α, r .
Before we derive upper bounds for A 2 and A 3 , we define some sets which will be used in our following proof.
Define the set
[Equation]
where c is the constant from Lemma 9.
Denote Ω i, 1 = φ i ( (cid:101)
Ω i, 1 ) .
According to Lemma 9, we have
[Equation]
Since φ i is a bijection, both Ω i, 1 and (cid:101)
Ω i, 1 have two disjoint boundaries.
Denote the two boundaries of Ω i, 1 by λ i, 1 , 1 and λ i, 1 , 2 .
We define the thickness of Ω i, 1 as
[Equation]
Since each φ i is a bijection, there exists a constant c 1 depending on c and the atlas such that χ
i, 1 ≥ cfor all i ’s. Again since ϕ i is a linear bijection, its inverse exists and is linear, and there exists a constant c 2 such that
[Equation]
We will choose θ and ∆ small enough such that
[Equation]
Define the region
[Equation]
According to (47) [Equation] (48) and the definition of Ω i, 1 , we have Ω i, 2 ⊂ Ω i, 1 .
For any z ∈ φ i ( U i ) \ Ω i, 2 , denote z ∗ = argmin [Equation]
z ∈ φ i ( ∂U i ) ∥
z − (cid:101) z
∥ 2 .
We have
[Equation]
Therefore ∥
[Equation]
when ∆ ≤ r 2 and (cid:101) 1 i ◦ φ − 1 ( z ) = 1 , ∂ (cid:101) [Equation] z j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) φ i ( U i ) \ Ω i,=for j = 1 , ..., d , where we used the notation z =
[ z 1 , ..., z d ] ⊤ .
Note that each (cid:101) g m , v and ϕ m z v is supported on B 1 /N, ∥·∥ ∞ ( m /N ) , a hyper cube with edge length 2 /N .
We will choose N large enough such that
[Equation]
Such a choice of N ensures that along any directions of z j for j = 1 , ..., d , there are at leasthypercubes that entirely locate inside Ω i, 2 .
Since any z ∈ [0 , 1] d is only covered by 2 hypercubes along each coordinate direction, we have { c i, m , v : there exits z ∈ φ i ( ∂U i ) such that z ∈ B 1 /N, ∥·∥ ∞ ( m /N ) } =(50) and (cid:101)
f i ◦ φ
i ( z ) = 0 for any z ∈ φ i ( ∂U i ) .
See Figure 4 for an illustration.
[Figure]
We have the following lemma on the bound of ∥ A 2 ∥ W k, ∞ ( φ i ( U i ))
(see Appendix M.2 for a proof):
Lemma 11.
Let A 2 be defined as in (42).
Assume ∆ ≤ r 2 .
We have
[Equation]
(51) for k = 1 , 2 .
The term A 3 can be bounded using Lemma 6:
[Equation]
for some constant C 13 depending on d, α, R .
Substituting (46) [Equation] (51) and (52) into (40) gives rise to
[Equation]
The second term in (39) can be bounded by Lemma 4 as
[Equation]
Substituting (53) and (54) into (39) gives rise to ∥
[Equation]
Setting
[Equation]
we have
[Equation]
for k = 0 , 1 and a constant C 15 depending on d, α, τ and the surface area of M .
• Network size We analyze the network size for each (cid:101)
f i :
• (cid:101)
i : The chart dermination network is the composition of (cid:101)
d i and (cid:101) 1 ∆ , where (cid:101) d
i has O (log 1 θ ) =
O (log N + log D ) layers and O ( D ) width, (cid:101) 1 ∆ has O (log 1 δ ) + D =
O (log N ) + D layers and O (1) width.
In both subnetworks, all parameters are of O (1) .
By Lemma 13, the chart dermination network has O (log N + log D ) + D layers, O ( D ) width and all weight parameters are of O (1) .
• (cid:101) × : The multiplication network has O (log 1 δ ) =
O (log N ) layers, O (1) width.
All weight parameters are bounded by 2( α + d + 1) .
• φ i : the projection φ i can be realized by a single layer with width d .
All parameters are of O (1) .
• (cid:101) g
i, m , v : By Lemma 5, each (cid:101) g i, m , v has O (log N ) layers and O ( d ) width.
All parameters are of O ( N ) .
• c i, m , v : By Lemma 4 with p = ∞ , each c i, m , v is of O (1) .
By Lemma 13, each c i, m , v ( (cid:101) g m , v ◦
φ i ( x )) (cid:101) × (cid:101)
i ( x ) is a CNN with O (log N +log D )
+ D layers, O ( D ) width and all parameters of O ( N ) .
According to (38), (cid:101) f can be written as a sum of C M N d ( d + 1) α
CNNs
[Equation]
By Lemma 7, for any (cid:102) M, (cid:101) J satisfying (cid:102) M (cid:101) J
= O ( N d ) , there exists a CNN architecture
F CNN ( L, J, K, κ, κ ) that gives rise to { g i } ni =1 with (cid:101) f
= (cid:102) M (cid:88) i =
g i and
L =
O (log N + log D ) + D, J
= O ( D (cid:101) J ) , κ = O ( N ) .
By Lemma 8, there exits a ConvResNet architecture C ( M, L, J, K, κ 1 , κ 2 ) with
L = O (log N ) + D, J
=
O
( D (cid:101) J ) , κ 1 = κ
= O ( N ) ,
M = O ( (cid:102) M ) (57) and (cid:101) J, (cid:102) M satisfying (cid:102) M (cid:101) [Equation]
[Equation] d ) , (58) that yields a ConvResNet realizing (cid:101) f .
Setting N =
O (( (cid:102) M (cid:101) J ) 1 /d ) in (55) and (57) gives rise to
[Equation]
(59) and the network size
[Equation]
L Definitions, Lemmas and their proofs used in Section G L.1 Existing lemmas on CNNs
Lemma 12 shows that any MLP can be realized by a CNN.
Lemma 12 (Theorem 1 in Oono and Suzuki (2019)) .
Let D be the dimension of the input.
Let L, J be positive integers and κ > 0 .
For any 2 ≤ K ′ ≤ D , any MLP architectures F MLP ( L, J, κ ) can be realized by a CNN architecture F CNN ( L ′ , J ′ , K ′ , κ ′ 1 , κ ′ 2 ) with
L ′ =
L + D, J ′ =
4 J, κ ′ 1 = κ ′
= κ.
Specifically, any ¯ f MLP ∈
F MLP ( L, J, κ ) can be realized by a CNN ¯ f CNN ∈
F CNN ( L ′ , J ′ , K ′ , κ ′ 1 , κ ′ 2 ) .
Furthermore, the weight matrix in the fully connected layer of ¯ f CNN has nonzero entries only in the first row.
Lemma 13 shows that the composition of two CNNs can be realized by a CNN.
Lemma 13 (Lemma 13 in Liu et al.
Let F CNN1 ( L 1 , J 1 , K 1 , κ 1 , κ 1 ) be a CNN architecture from R D
→ R and F CNN2 ( L 2 , J 2 , K 2 , κ 2 , κ 2 ) be a CNN architecture from R → R .
Assume the weight matrix in the fully connected layer of F CNN1 ( L 1 , J 1 , K 1 , κ 1 , κ 1 ) and
F CNN2 ( L 2 , J 2 , K 2 , κ 2 , κ 2 ) has nonzero entries only in the first row.
Then there exists a CNN architecture F CNN ( L, J, K, κ, κ ) from R D
→ R with
L = L 1 + L 2 , J = max( J 1 , J 2 ) , K = max( K 1 , K 2 ) , κ = max( κ 1 , κ 2 ) such that for any f 1 ∈ F CNN ( L 1 , J 1 , K 1 , κ 1 , κ 1 ) and f 2 ∈ F CNN ( L 2 , J 2 , K 2 , κ 2 , κ 2 ) , there exists f ∈ F CNN ( L, J, K, κ, κ ) such that f ( x ) =
f 2 ◦ f 1 ( x ) .
Furthermore, the weight matrix in the fully connected layer of F CNN ( L, J, K, κ, κ ) has nonzero entries only in the first row.
L.2 Interpolation spaces
Definition 16 (Interpolation spaces) .
Let ( B 0 , B 1 ) be an interpolation couple.
For any u ∈ B 1 , define
[Equation]
and the norm
[Equation]
for 1 ≤ p < ∞
[Equation]
for p = ∞ .
Then the interpolation space ( B 0 , B 1 ) θ,p is defined by
[Equation]
The following lemma shows that the fractional Sobolev space is an interpolation space:
Lemma 14 (Theorem 14.2.3 of Brenner et al.
Let Ω ∈ R D be an Lipschitz domain.
Then for any 0 < s < 1 and 1 ≤ p ≤ ∞ , we have
[Equation]
The following lemma shows that the norm of the interpolation space of ( B 0 , B 1 ) θ,p can be bounded using ∥ · ∥ B 0 and ∥ · ∥ B 1 :
Lemma 15.
Let ( B 0 , B 1 ) be an interpolation couple.
Moreover, let 0 < θ < 1 and 1 ≤ p ≤ ∞ .
Then there exists a constant C depending on θ and p such that for all u ∈ B 1 , we have
[Equation]
In particular, when p = ∞ , we have C = 1 .
L.3 Proof of Lemma
Proof of Lemma 1.
Note that ψ ( x ) can be realized by a two-layer MLP
[Equation]
with
[Equation]
According to Lemma 12, for any 2 ≤ K , such an MLP can be realized by a CNN in
F CNN (2 , 16 , 2 , 2 , 2) .
According to the expression of the right-hand-side of (9), we have (cid:101) ψ m,N ( x ) ∈
F CNN (2 , 16 , 2 , 3 N, 3 N )
To prove (10), the case k = 0 follows by the definition of ψ .
For k = 1 , we have
[Equation]
L.4 Proof of Lemma
Proof of Lemma 5.
For any given m and v , ϕ m x v is a product of at most α + D quantities each of which can be realized by a CNN.
The following lemma shows that the multiplication operator × can be well approximated by a CNN (see a proof in Appendix L.5):
Lemma 16.
For any 0 < η < 1 / 2 , x, y ∈ [ − B, B ] , and K ≥ 2 , there is a CNN architecture
F CNN ( L, J, K, κ, κ ) that yields a CNN, denoted by (cid:101) × ( · , · ) , such that
[Equation]
Such a network has
[Equation]
Furthermore, one has
[Equation]
for some absolute constant C .
For simplicity, we denote ψ m k
( x ) = ψ (cid:0) 3 N (cid:0) x k − m k N (cid:1)(cid:1) for k = 1 , ..., D .
Then we construct (cid:101) g m , v ( x ) as
[Equation]
where (cid:101) p v ( x ) is the network approximation of x v defined by
[Equation]
The structure of (cid:101) g m , v is visualized in Figure 5.
Here (cid:101) g m , v consists of no more than α + D −compositions of (cid:101) × and 2 D additional channels.
These additional channels are used to pass the information x + and x − .
By applying Lemma 13 α + D − 2 times, we have (cid:101) g m , v ∈ F CNN ( L, J, K, κ, κ ) with
L = O (cid:18)
D log 1 ε (cid:19) , J = O ( D ) , κ = 3 N.
We next prove (14) and (15).
First note that we can express
[Equation]
for some n ≤ α + D , where each h i can be realized by one layer and satisfies
[Equation]
To prove (14) and (15) [Equation] it is enough to show
[Equation]
for any 1 ≤ n ≤ α + D − 1 , where { c n } α + D − 1 n = and C 16 are constants depending on D and α .
For n = 1 , we have
[Equation]
By Lemma 16 with B = α + D + 1 , we have for k = 0 ,
[Equation]
For k = 1 , by Lemma 16, we deduce
[Equation]
where C ′ is a constant depending on D .
We set c 1 = 3 C ′ .
Furthermore,
[Equation]
where C 17 , C 18 are constants depending on D, α .
Therefore, the inequalities (60) and (62) hold for n = 1 .
For (61), if g 1 ( x ) = 0 , then h 1 ( x ) = 0 .
By Lemma 16, (cid:101) g 1 ( x ) = 0 .
Assume (60)–(62) hold for any 1 ≤ n ≤ t for some integer t satisfying 1 ≤ t ≤ α + D − 2 , i [Equation] e., for any 1 ≤ n ≤ t , we have
[Equation]
We also deduce that
[Equation]
For n = t + 1 , we have
[Equation]
Consider the first term in (67).
For k = 0 , we have
[Equation]
For k = 1 , we have
[Equation]
where (63) with k = 1 is used in the last inequality, C ′ is a constant depending on D .
For the second term in (67), we first consider k = 0 :
[Equation]
where (63) with k = 0 is used.
For k = 1 , we have
[Equation]
where C 20 , C 21 are constants depending on D and α .
In (71), (63) with k = 0 and k = 1 are used in the second inequality.
Combining (68)–(71) and setting c t +1 = 3 C 20 c t + C 21 gives rise to
[Equation]
Therefore, (60) holds for n = t + 1 .
To prove (61), note that if g t +1 = 0 , then either h t +1 = 0 , or g t = 0 .
By our induction assumption, when g t = 0 , we have (cid:101) g t = 0 .
Since (cid:101) g t
+
=
(cid:101) ×
( (cid:101) g t , h t +1 ) , by Lemma 5, we have (cid:101) g t +1 = 0 and (61) holds for n = t + 1 .
For (62), we deduce
[Equation]
≤ C 22 N [Equation] where C 22 is a constant depending on D and α .
Therefore, (60)–(62) hold for n = t + 1 .
By mathematical induction, (60)–(62) hold for any 1 ≤ n ≤ D + α + 1 , and (14) and (15) are proved.
L.5 Proof of Lemma
Proof of Lemma 16.
The proof of Lemma 16 is based on the following lemma.
Lemma 17 (Proposition C.2 in Gühring et al.
For any 0 < η < 1 / 2 , x, y ∈ [ − B, B ] .
There is an MLP, denoted by (cid:101) × ( · , · ) , such that
[Equation]
Such a network has O (cid:16) log 1 η (cid:17) layers and parameters.
The width of each layer is bounded by 6 and all parameters are bounded by 2 .
Furthermore, we have
[Equation]
for some absolute constant C .
Combing Lemma 17 and 12, for any ε > 0 , K ≥ 2 , there exits a CNN (cid:101) × ∈ F CNN ( L, J, K, κ, κ ) such that for any | x | ≤ B, | y | ≤ B , we have
[Equation]
23 B, where C 23 is an absolute constant.
Such an architecture has
[Equation]
O
[Equation]
J
[Equation]
[Equation]
κ = 1 .
L.6 Proof of Lemma
Proof of Lemma 6.
Denote Ω m ,N = B 1 N , ∥·∥ ∞ (cid:0) m N (cid:1) .
We have
[Equation]
(cid:88)
[Equation] [Table]
(cid:88)
[Table]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
[Table]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) p
[Equation]
[Table]
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:88)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
where the first equality follows from (12), the last inequality holds since (0 , 1) D ⊂ ∪ (cid:101) m ∈S N Ω (cid:101) m ,N .
For each (cid:101) m , we have
[Equation]
where C 21 is the constant in Lemma 3, ¯ f is the extension of f to R D from Stein (1970, Theorem
VI.3.1.5), which satisfies
[Equation]
[Equation]
for some constant C 25 depending on D, p, α .
We next derive an upper bound of the summand of (73).
We first deduce that
[Equation]
[Equation] where (cid:12)(cid:12) Ω (cid:101) m ,N ∩ (0 , 1) D (cid:12)(cid:12) denotes the volume of Ω (cid:101) m ,N ∩ (0 , 1) D , C 26 , C 27 are constants depending on D [Equation] α and p .
We used Lemma 5 in the last inequality.
Substituting (75) into (73) gives rise to
[Equation]
where C 28 = C 24 C 27 ( D + 1) α − 1 .
By Hölder’s inequality
[Equation]
we have
[Equation]
where q = 1 / (1 − 1 /p ) .
Substituting (76), (77) into (72) gives rise to ∥
[Equation]
where C 29 is a constant depending on D, α, p .
In the above, we used (74) in the last inequality.
Lemma 6 is proved for s = 0 and s = 1 .
For any 0 < s < 1 and 1 ≤ p ≤ ∞ , by Lemma 15, we have
[Equation]
for some constant C 30 depending on D, α, s, p .
The proof is finished.
L.7 Lemma 18 and its proof
Lemma 18.
Let { f i } ni =1 be a set of CNNs with architecture F CNN ( L 0 , J 0 , K 0 , κ 0 , κ 0 ) .
Then there for any integer 1 ≤ w ≤ n , there exists a CNN architecture F CNN ( L w , J w , K w , κ w , κ w ) that gives rise to a CNN g w such that
[Equation]
Such an architecture has
[Equation]
κ = κ 0 . Furthermore, the fully connected layer of f has nonzero elements only in the first row.
Proof of Lemma 18.
The idea of the proof is similar to Liu et al.
(2021, proof of Lemma 14).
Following the proof of Liu et al.
(2021, Lemma 14), we can show that there exist a set of filters W and biases B such that
[Equation]
where Conv W , B has depth bounded by L 0 , number of channels bounded by wj 0 and all weight parameters bounded by κ 0 .
We write g w as g w = W 1 · Conv W , B , where W 1 is given as
W 1 = (cid:20)− − 1 · · · −    · · · (cid:21) .
The proof is finished.
L.8 Proof of Lemma
Proof of Lemma 7.
For any given (cid:101) J , let c be the smallest integer such that (cid:101) J ≤ cJ 0 .
Then we set
J = cJ 0 and n = ⌈ n 0 /c ⌉ .
By Lemma 18, there exists a CNN architecture F CNN ( L, J, K, κ, κ ) with
L = O
( L 0 ) , J = cJ 0 , K = K 0 , κ = κ 0 .
Such an architecture gives rise to CNNs { g j } ⌈ n 0 /c
⌉ j
= such that g j = min [Equation] cj,n
[Equation]
(cid:88) i =
[Equation]
[Equation]
[Equation]
f i .
The lemma is proved.
M Proof of lemmas in Appendix K M.1 Proof of Lemma
Proof of Lemma 9.
Following the construction in Step 1 of the proof of Theorem 5, for (cid:101) r
= r/ 2 < τ/ 8 , there exists a collection of points atlas of M denoted by { (cid:101)
U
i , (cid:101) φ i } (cid:101) C M
i
=1 , where (cid:101) U
i =
B (cid:101) r
( (cid:101) c i ) for some (cid:101) c i ∈ M , and (cid:101) φ i is defined according to (34).
By Conway and Sloane (1988, Chapter
Equation (1)), the number of charts is bounded by (cid:101)
[Equation]
r/ [Equation] (cid:39) . The following lemma shows that for any locally finite cover of a smooth manifold, a C ∞ partition of unity always exists:
Lemma 19 (Chapter 2 Theorem 15 of Spivak (1973)) .
Let { U α } α ∈A be a locally finite cover of a smooth manifold M .
There is a C ∞ partition of unity { ρ α } ∞ α =1 such that supp( ρ α ) ⊂
U α .
Let { ρ i } (cid:101) C M
i
=1 be the partition of unity in Lemma 19 with respect to { (cid:101) U i } C M
i
=1 .
We set C M = (cid:101) C M and define U i =
B r ( (cid:101) c i ) and φ i according to (34).
Since (cid:101) r < r , (cid:101) U i ⊂
U
i , we have (cid:101) U i
⊂ U i and
[Equation]
Therefore { U i } C M i =1 is an open cover of M and { U i , φ i } C M
i
=1 is an atlas of M .
Since supp( ρ i ) ⊆
(cid:102)
U
i , we have supp( ρ i ) ⊂
U i and
[Equation]
The lemma is proved.
Proof of Lemma 11.
We deduce
[Equation]
for k = 0 , 1 , where the last equality holds since
[Equation]
on φ i ( U i ) \ Ω i, 2 .
According to (50), (cid:101)
f i ◦ φ − 1 i ( z )
= (cid:98) f
i ◦ φ − 1 i ( z ) = 0 for z ∈
φ
i ( ∂U i ) .
For any z ∈ Ω i, 2 , let
[Equation]
argmin [Equation] According to (49), we have ∥ z − z ∗ ∥ 2 ≤ ∆ / ( c 2 r ) .
By Lemma 6 with some small η > 0 and for s = k = 0 , 1 , we have
[Equation]
where C 31 is a constant depending on d, α, R .
Since ∥ (cid:98) f i ∥ W 1 , ∞ (Ω i, 2 ) = 0 , we have max j (cid:12)(cid:12)(cid:12) ∂
[Equation]
f i ∂z j [Equation] ≤
C
Nη for any z ∈ Ω i, 2 .
Therefore |
[Equation]
79) for any z ∈ Ω i, 2 .
Using | (cid:101) 1 i ◦ φ − 1 i ( z ) − 1 i ◦ φ − 1 i ( z ) | ≤ 1 , we bound A 2 as | A
[Equation]
i
[Equation]
i
[Equation]
i
[Equation] [Equation]
[Equation]
for k = [-
 and|-]0 and | A 2 | W 1 , ∞ ( φ i ( U i ))
= (cid:12)(cid:12)(cid:12) (cid:101)
f i ◦ φ − 1 i
( z ) × (cid:16)(cid:101)
i ◦ φ [Equation] z ) [Equation]
i ◦ [Equation]
W 1 , ∞ (Ω i, 2 ) [Equation] i [Equation]
[Equation]
i
[Equation]
for k = 1 , where C 12 is a constant depending on α, R, τ .
In the first inequality of (81), we used (79), the inequality
[Equation]
by (37) and the fact 1 i ◦ φ − 1 i ( z )
= 1 for z ∈ Ω i, 2 , and the inequality
[Equation]
by (78).
Combining (80) and (81) gives rise to
[Equation]
for k = 0 , 1 .