Original: A PPROXIMATION AND NON - PARAMETRIC ESTIMATION OF FUNCTIONS OVER HIGH - DIMENSIONAL SPHERES VIA DEEP R E LU NETWORKS
Revised: A PPROXIMATION AND NON - PARAMETRIC ESTIMATION OF FUNCTIONS OVER HIGH - DIMENSIONAL SPHERES VIA DEEP R E LU NETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We develop a new approximation and statistical estimation analysis of deep feedforward neural networks (FNNs) with the Rectified Linear Unit (ReLU) activation.
Revised: We develop a new approximation and statistical estimation analysis of deep feedforward neural networks (FNNs) with the Rectified Linear Unit (ReLU) activation.

Original: The functions of interests for the approximation and estimation are assumed to be from Sobolev spaces defined over the d -dimensional unit sphere with smoothness index r > 0 .
Revised: The functions of interests for the approximation and estimation are assumed to be from Sobolev spaces defined over the d -dimensional unit sphere with smoothness index r > 0 .

Original: In the regime where r is in the constant order (i.e., r = O (1) ), it is shown that at most d d active parameters are required for getting d − C approximation rate for some constant C > 0 .
Revised: In the regime where r is of the constant order (i.e., r = O (1) ), it is shown that at most d d active parameters are required for getting d − C approximation rate for some constant C > 0 .

Original: In the regime where the index r grows in the order of d (i.e., r = O ( d ) ) asymptotically, we prove the approximation error decays in the rate d − d β with 0 < β < 1 up to some constant factor independent of d .
Revised: In the regime where the index r grows in the order of d (i.e., r = O ( d ) ) asymptotically, we prove the approximation error decays in the rate d − d β with 0 < β < 1 up to some constant factor independent of d .

Original: The required number of active parameters in the networks for the approximation increases polynomially in d as d → ∞ .
Revised: The required number of active parameters in the networks for the approximation increases polynomially in d as d → ∞ .

Original: It is also shown that bound on the excess risk has a d d factor, when r = O (1) , whereas it has d O (1) factor, when r = O ( d ) .
Revised: It is also shown that bound on the excess risk has a d d factor, when r = O (1) , whereas it has d O (1) factor, when r = O ( d ) .

Original: We emphasize our findings by making comparisons to the results on  approximation and estimation errors of deep ReLU FNN when functions are from Sobolev spaces defined over d -dimensional cube.
Revised: We emphasize our findings by making comparisons to the results on the approximation and estimation errors of deep ReLU FNN when functions are from Sobolev spaces defined over d -dimensional cube.

Original: Here, we show that with the current state-of-the-art result, d d factor remain both in the approximation and estimation errors, regardless of the order of r .
Revised: In this case, we show that with the current state-of-the-art result, d d factor remain both in the approximation and estimation errors, regardless of the order of r .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks have demonstrated tremendous success in the tasks of image classification (Krizhevsky et al., 2012;
Revised: Neural networks have demonstrated tremendous success in the tasks of image classification (Krizhevsky et al., 2012;

Original: Long et al., 2015), pattern recognition (Silver et al., 2016), natural language processing (Graves et al., 2013; Bahdanau et al., 2015; Young et al., 2018), etc.
Revised: Long et al., 2015), pattern recognition (Silver et al., 2016), natural language processing (Graves et al., 2013; Bahdanau et al., 2015; Young et al., 2018), etc.

Original: The datasets used in these real world applications frequently lie in high-dimensional spaces (Wainwright, 2019),
Revised: The datasets used in these real world applications frequently lie in high-dimensional spaces (Wainwright, 2019).

Original: In this paper, we try to understand the fundamental limits of neural network in the high-dimensional regime through the lens of its approximation power and its generalization error.
Revised: In this paper, we try to understand the fundamental limits of neural network in the high-dimensional regime through the lens of its approximation power and its generalization error.

Original: Both approximation power and generalization error of neural network can be analyzed through specifying the target function’s property such as its smoothness index r > 0 and the input space
Revised: Both approximation power and generalization error of neural network can be analyzed through specifying the target function’s property such as its smoothness index r > 0 and its input space

Original: X .
Revised: X .

Original: In particular, deep feed-forward neural networks (FNNs) with Rectified Linear Unit (ReLU) have been extensively studied when they are used for approximating and estimating functions from general function class such as Sobolev class defined on d -dimensional cube (i.e., X := C d ), denoted as W rp ( C d ) for 1 ≤ p ≤ ∞ .
Revised: In particular, deep feed-forward neural networks (FNNs) with Rectified Linear Unit (ReLU) have been extensively studied when they are used for approximating and estimating functions from general function class such as Sobolev class defined on d -dimensional cube (i.e., X := C d ), denoted as W rp ( C d ) for 1 ≤ p ≤ ∞ .

Original: However, in practice, signals on spherical surface (i.e.,
Revised: However, in practice, signals on spherical surface (i.e.,

Original: X := S d − 1 = { x ∈ R d : ∥ x ∥ 2 = 1 } ) rather than in Euclidean spaces often arise in various situations, such as astrophysics (Starck et al., 2006; Wiaux et al., 2005), computer vision (Brechb¨uhler et al., 1995), and medical imaging (Yu et al., 2007).
Revised: X := S d − 1 = { x ∈ R d : ∥ x ∥ 2 = 1 } ) rather than on Euclidean spaces often arise in various fields, such as astrophysics (Starck et al., 2006; Wiaux et al., 2005), computer vision (Brechb¨uhler et al., 1995), and medical imaging (Yu et al., 2007).

Original: Motivated from this, we focus our attention in the cases where deep ReLU FNNs are used for function approximators and estimators, when functions are assumed to be from the Sobolev spaces defined over S d − 1 ; that is f ∈ W r ∞ ( S d − 1 ) .
Revised: Motivated from this, we focus our attention in the cases where deep ReLU FNNs are used for function approximators and estimators, when functions are assumed to be from the Sobolev spaces defined over S d − 1 ; that is f ∈ W r ∞ ( S d − 1 ) .

Original: Under this setting, our analysis focuses on how the input dimension d explicitly affects the approximation and estimation rates of f ∈ W r ∞ ( S d − 1 ) .
Revised: Under this setting, our analysis focuses on how the input dimension d explicitly affects the approximation and estimation rates of f ∈ W r ∞ ( S d − 1 ) .

Original: And, at the same time, we show how the scalability of deep ReLU FNNs grows in the high-dimensional regime.
Revised: At the same time, we show how the scalability of deep ReLU FNNs grows in the high-dimensional regime.

Original: Here, the scalability is mainly measured through the three metrics: (1) the width denotedas W , (2) the depth, denoted as L , and (3) the number of active parameters, denoted as N of the network, (Anthony & Bartlett, 1999). It should be emphasized that we find there exists an
Revised: Here, the scalability is mainly measured through the three metrics: (1) the width denoted

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: O

Original: [Table]
Revised: [Table]

Original: O

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: O [Equation] d 2 ) [Equation]
Revised: as W , (2) the depth, denoted as L , and (3) the number of active parameters, denoted as N of the network, (Anthony & Bartlett, 1999).

Original: [Table] [Equation] interaction with smoothness index r > 0 and dimension d , whereas we cannot find one for the case when f ∈ W r ∞ ( C d ) .
Revised: It should be emphasized that we find there exists an interaction with smoothness index r > 0 and dimension d , whereas we cannot find one for the case when f ∈ W r ∞ ( C d ) .

Original: We further summarize our detailed findings in the following subsection.
Revised: We further summarize our detailed findings in the following Subsection.

Original: 1. P APER R OAD MAP AND C ONTRIBUTIONS
Revised: 1. P APER R OAD MAP AND C ONTRIBUTIONS

Original: In Theorem 3.1, we provide an approximation bound of deep ReLU FNN (i.e., ˜ f ) for approximating the target functions in Sobolev spaces defined over sphere (i.e., f ∈ W r ∞ ( S d − 1 ) ).
Revised: In Theorem 3.1, we provide an approximation bound of deep ReLU FNN (i.e., ˜ f ) for approximating the target functions in Sobolev spaces defined over sphere (i.e., f ∈ W r ∞ ( S d − 1 ) ).

Original: Notably, in the bound, we track the explicit dependence on data dimension d allowing it tend to infinity.
Revised: Notably, in the bound, we track the explicit dependence on data dimension d allowing it tend to infinity.

Original: This tracking enables how the three components of network architecture, width ( W ), depth ( L ), and the number of active parameters ( N ), should change as d increases, for obtaining the good approximation error rate.
Revised: This tracking enables how the three components of network architecture, width ( W ), depth ( L ), and the number of active parameters ( N ), should change as d increases, for obtaining the good approximation error rate.

Original: As a Corollary of Theorem 3.1, we show how the order of function smoothness r can have the effect on the scale of network in terms of d .
Revised: As a Corollary of Theorem 3.1, we show how the order of function smoothness r can have the effect on the scale of network in terms of d .

Original: Specifically, when the function smoothness r = O (1) , we show that the constructed network, ˜ f , requires W =
Revised: Specifically, when the function smoothness r = O (1) , we show that the constructed network, ˜ f , requires W =

Original: O ( d d ) , L = O ( d γ log 2 d ) for 0 < γ < 1 , and
Revised: O ( d d ) , L = O ( d γ log 2 d ) for 0 < γ < 1 , and

Original: N = O ( d d +1 ) for obtaining d −O (1) approximation error up to some constant factors independent with d .
Revised: N = O ( d d +1 ) for obtaining d −O (1) approximation error up to some constant factors independent with d .

Original: Furthermore, when r = O ( d ) , we show that only W = O
Revised: Furthermore, when r = O ( d ) , we show that only W = O

Original: ( d α ) , L =
Revised: ( d α ) , L =

Original: O ( d γ log 2 d ) , and
Revised: O ( d γ log 2 d ) , and

Original: at most N = O
Revised: at most N = O

Original: ( d 2 ) are required for obtaining the sharp approximation rate O
Revised: ( d 2 ) are required for obtaining the sharp approximation rate O

Original: ( d − d β ) for 0 <
Revised: ( d − d β ) for 0 <

Original: α, β < 1 .
Revised: α, β < 1 .

Original: See Corollary 3.3 for the detailed statement of the result.
Revised: See Corollary 3.3 for the detailed statement of the result.

Original: Our result implies that for approximating f ∈ W r ∞ ( S d − 1 ) , the larger the smoothness index r is, the narrower the width of the network should be enough, while the depth of the network can be fixed.
Revised: Our result implies that for approximating f ∈ W r ∞ ( S d − 1 ) , the larger the smoothness index r is, the narrower the width of the network should be enough, while the depth of the network can be fixed.

Original: Moreover, when r is in the same order of d , the network can avoid the curse of dimensionality requiring only O ( d 2 ) number of active parameters.
Revised: Moreover, when r is in the same order of d , the network can avoid the curse of dimensionality requiring only O ( d 2 ) number of active parameters.

Original: It is interesting to note that the function smoothness index can affect the design of the network, specifically on width, while it has little effect on the design of depth.
Revised: It is interesting to note that the function smoothness index can affect the design of the network, specifically on width, while it has little effect on the design of depth.

Original: Admittedly, the condition r =
Revised: Admittedly, the condition r =

Original: O ( d ) is restrictive in a sense that it makes the function space W r ∞ ( S d − 1 ) small.
Revised: O ( d ) is restrictive in a sense that it makes the function space W r ∞ ( S d − 1 ) small.

Revised: Nonetheless, it contains some interesting examples: that is, reproducing kernel Hilbert spaces (RKHS) generated by C ∞ kernels such as Gaussian kernels.

Original: Nonetheless, to the best of our knowledge, this finding is not observed in the current approximation theory of neural network literature when f ∈ W r ∞ ( C d ) where C d denotes some d -dimensional cubes, and ˜ f is a deep ReLU FNN.
Revised: Additionally, to the best of our knowledge, this finding is not observed in the current approximation theory of neural network literature when f ∈ W r ∞ ( C d ) where C d denotes some d -dimensional cubes, and ˜ f is a deep ReLU FNN.

Original: Out of the long list of literature to be introduced shortly, we choose the result from Schmidt-Hieber (2020) for the comparison as it also has the explicit dependence on d in their approximation bound.
Revised: Out of the long list of literature to be introduced shortly, we choose the result from Schmidt-Hieber (2020) for the comparison as it also has the explicit dependence on d in their approximation bound.

Original: From their result, it can be seen that the curse cannot be avoided, even when r = O ( d ) .
Revised: From their result, it can be seen that the curse cannot be avoided, even when r = O ( d ) .

Original: The width of their constructed network is lower bounded by Ω( r d ∨ e d ) and the number of active parameters is upper-bounded by O (( r + d ) d ) .
Revised: The width of their constructed network is lower bounded by Ω( r d ∨ e d ) and the number of active parameters is upper-bounded by O (( r + d ) d ) .

Original: 1 Note that the bounds on both components grow exponentially in d as r increases.
Revised: 1 Note that the bounds on both components grow exponentially in d as r increases.

Original: See subsection 3.1 for the detailed comparisons.
Revised: See Subsection 3.1 for the detailed comparisons.

Original: We further make the comparisons between estimating functions f ∈ W r ∞ ( S d − 1 ) (Theorems 4.3) versus f ∈ W r ∞ ( C d ) (Theorems 4.4) via deep ReLU FNNs under the non-parametric regression framework.
Revised: We further make the comparisons between estimating functions f ∈ W r ∞ ( S d − 1 ) (Theorems 4.3) versus f ∈ W r ∞ ( C d ) (Theorems 4.4) via deep ReLU FNNs under the non-parametric regression framework.

Original: Given n noisy samples, the two Theorems suggest the specific orders of W , L and N in terms of n , d and r , for which they give the tightest bound on excess risk of respective function estimator from Proposition 4.2.Under the high-dimensional setting where n =
Revised: Given n noisy samples, the two Theorems suggest the specific orders of W , L and N in terms of n , d and r , for which they give the tightest bound on excess risk of respective function estimator from Proposition 4.2.

Original: O ( d q ) for 0 < q < 1 , when r = O (1) , it is shown that the excess risk upper-bounds of  both function estimators have d d in the constant factors.
Revised: When r = O (1) , it is shown that the excess risk upper-bounds of on width both function estimators have d d in the constant factors.

Original: In contrast, when r = O ( d ) , estimating functions f ∈ W r ∞ ( S d − 1 ) has at most d O (1) factor in the bound, whereas the bound for function estimator of f ∈ W r ∞ ( C d ) has d d .
Revised: In contrast, when r = O ( d ) , estimating functions f ∈ W r ∞ ( S d − 1 ) has at most d O (1) factor in the bound, whereas the bound for function estimator of f ∈ W r ∞ ( C d ) has d d .

Original: See Table 1 and Subsection 4.2 for detailed comparisons.
Revised: See Table 1 and Subsection 4.2 for detailed comparisons.

Original: 1. R ELATED WORKS
Revised: 1. R ELATED WORKS

Original: In this subsection, to aid readers have more clear understandings on the contributions of our paper, we provide the list of relevant works with comparisons on how these works are different from ours.
Revised: In this Subsection, to aid readers have more clear understandings on the contributions of our paper, we provide the list of relevant works with comparisons on how these works are different from ours.

Original: Approximation of f ∈ W r ∞ ( S d − 1 ) via deep CNN.
Revised: Approximation of f ∈ W r ∞ ( S d − 1 ) via deep CNN.

Original: For the approximation theory of f ∈
Revised: For the approximation theory of f ∈

Original: W r ∞ ( S d − 1 ) , we must refer readers Fang et al.
Revised: W r ∞ ( S d − 1 ) , we must refer readers Fang et al.

Original: (2020) and Feng et al.
Revised: (2020) and Feng et al.

Original: But in their works, the convolutional neural network (CNN) is used for the function approximator under fixed d setting.Approximation of
Revised: But in their works, the convolutional neural network (CNN) is used for the function approximator under fixed d setting.

Original:  ∈ W r ∞ ( C d ) via deep ReLU FNN.Approximation
Revised: Approximation of f ∈ W r ∞ ( C d ) via deep ReLU FNN.

Original:  theory of deep ReLU FNN for functions f ∈ W r ∞ ( C d ) has a lengthy history in the literature.
Revised: Approximation theory of deep ReLU FNN for functions f ∈ W r ∞ ( C d ) has a lengthy history in the literature.

Original: Representatively, Mhaskar (1996) showed that f can be approximated uniformly within ε -approximation accuracy with a 1 -layer neural network of O ( ε − d/r ) neurons and an infinitely differentiable activation function.
Revised: Representatively, Mhaskar (1996) showed that f can be approximated uniformly within ε -approximation accuracy with a 1 -layer neural network of O ( ε − d/r ) neurons and an infinitely differentiable activation function.

Original: Later, for deep
Revised: Later, for deep

Original: ReLU networks, Yarotsky (2017) showed that the number of active parameters ( N ) in networks is bounded by O ( ε − d/r log (cid:0) 1 ε (cid:1) ) , and the depth has the order O (log( 1 ε )) .
Revised: ReLU networks, Yarotsky (2017) showed that the number of active parameters ( N ) in networks is bounded by O ( ε − d/r log (cid:0) 1 ε (cid:1) ) , and the depth has the order O (log( 1 ε )) .

Original: He further proved that N is lower-bounded by the order O ( ε − d/r ) , which is backed up by the result in DeVore et al.
Revised: He further proved that N is lower-bounded by the order O ( ε − d/r ) , which is backed up by the result in DeVore et al.

Original: For f ∈ W rp ( C d ) with 1 ≤ p ≤ ∞ , Petersen & Voigtlaender (2018) showed that there exists a deep
Revised: For f ∈ W rp ( C d ) with 1 ≤ p ≤ ∞ , Petersen & Voigtlaender (2018) showed that there exists a deep

Original: ReLU network with bounded and quantized weight parameters, with O ( ε − d/r ) network size, and with ε -independent depth for achieving the ε -accuracy in the L p norm.
Revised: ReLU network with bounded and quantized weight parameters, with O ( ε − d/r ) network size, and with ε -independent depth for achieving the ε -accuracy in the L p norm.

Original: For approximating functions f ∈ W r ∞ ( C d ) , Schmidt-Hieber (2020) proved that a network of size O ( ε − d/r ) with bounded weight parameters achieves ε -approximation error in the L ∞ norm.
Revised: For approximating functions f ∈ W r ∞ ( C d ) , Schmidt-Hieber (2020) proved that a network of size O ( ε − d/r ) with bounded weight parameters achieves ε -approximation error in the L ∞ norm.

Original: Function spaces with special structures.
Revised: Function spaces with special structures.

Original: The result of Yarotsky (2017) implies that deep ReLU net cannot escape the curse of dimensionality for approximating f ∈ W r ∞ ( C d ) .
Revised: The result of Yarotsky (2017) implies that deep ReLU net cannot escape the curse of dimensionality for approximating f ∈ W r ∞ ( C d ) .

Original: Many papers have demonstrated that the effects of dimension can be either avoided or lessened by considering function spaces different from Sobolev spaces, but defined over C d .
Revised: Many papers have demonstrated that the effects of dimension can be either avoided or lessened by considering function spaces different from Sobolev spaces, but defined over C d .

Original: Just to name a few, Mhaskar et al.
Revised: Just to name a few, Mhaskar et al.

Original: (2016) studied that a function with a compositional structure with regularity r can be approximated by neural network with O ( ε − 2 /r ) neurons within ε accuracy.
Revised: (2016) studied that a function with a compositional structure with regularity r can be approximated by neural network with O ( ε − 2 /r ) neurons within ε accuracy.

Original: Suzuki (2018) proved the deep ReLU network with O ( ε − 1 /r ) neurons can avoid the curse for approximating functions in mixed smooth
Revised: Suzuki (2018) proved the deep ReLU network with O ( ε − 1 /r ) neurons can avoid the curse for approximating functions in mixed smooth

Original: Besov spaces.
Revised: Besov spaces.

Original: Chen et al. (2019a) showed the network size scales as O (
Revised: Chen et al. (2019a) showed the network size scales as O (

Original: ε − D/r ) for approximating
Revised: ε − D/r ) for approximating

Original: C r functions, when they are defined on a Riemannian manifold isometrically embedded in R d with manifold dimension D with D ≪ d .
Revised: C r functions, when they are defined on a Riemannian manifold isometrically embedded in R d with manifold dimension D with D ≪ d .

Original: Montanelli & Du (2019) and Blanchard & Bennouna (2022) showed respectively the deep and shallow ReLU network break the curse for Korobov spaces.
Revised: Montanelli & Du (2019) and Blanchard & Bennouna (2022) showed respectively the deep and shallow ReLU network break the curse for Korobov spaces.

Original: Estimation rates of excess risk under non-parametric framework.
Revised: Estimation rates of excess risk under non-parametric framework.

Original: Many researchers also have tried to tackle how the neural networks avoid the curse by considering specially designed function spaces under the non-parametric regression framework.
Revised: Many researchers also have tried to tackle how the neural networks avoid the curse by considering specially designed function spaces under the non-parametric regression framework.

Original: We only provide the incomplete list of them.
Revised: We only provide the incomplete list of them.

Original: Such structures include additive ridge functions (Fang & Cheng, 2022), composite function spaces with hierarchical structures (Schmidt-Hieber, 2020; Han et al., 2022), mixed-Besov spaces (Suzuki, 2018), H ¨ older spaces defined over a lower-dimensional manifold embedded in R d (Chen et al., 2019b).
Revised: Such structures include additive ridge functions (Fang & Cheng, 2022), composite function spaces with hierarchical structures (Schmidt-Hieber, 2020; Han et al., 2022), mixed-Besov spaces (Suzuki, 2018), H¨older spaces defined over a lower-dimensional manifold embedded in R d (Chen et al., 2019b).

Original: They all showed the function estimators with neural network architectures can lessen the curse by showing the excess risks of the estimators are bounded by O ( n − 2 r/ (2 r + D ′ ) ) , where n denotes the size of a noisy dataset, and D ′
Revised: They all showed the function estimators with neural network architectures can lessen the curse by showing the excess risks of the estimators are bounded by O ( n − 2 r/ (2 r + D ′ ) ) , where n denotes the size of a noisy dataset, and D ′

Original: ≪
Revised: ≪

Original: d is an intrinsic dimension uniquely determined through the characteristics of function spaces, when they are compared with the minimax risk
Revised: d is an intrinsic dimension uniquely determined through the characteristics of function spaces, when they are compared with the minimax risk

Original: O ( n − 2 r/ (2 r + d ) )
Revised: O ( n − 2 r/ (2 r + d ) )

Original: (Donoho & Johnstone, 1998) for f ∈ W r ∞ ( C d ) .
Revised: (Donoho & Johnstone, 1998) for f ∈ W r ∞ ( C d ) .

Original: Comparisons.
Revised: Comparisons.

Original: The aforementioned works mainly focused on the approximation and estimation of functions defined on C d , not S d − 1 , for the fixed d .
Revised: The aforementioned works mainly focused on the approximation and estimation of functions defined on C d , not S d − 1 , for the fixed d .

Original: Moreover, the introduced papers on approximation theory, except the work of Schmidt-Hieber (2020), hide the dependence on d in the Big- O notation of N in ε -accuracy, even for papers where they consider the function spaces with special structures.
Revised: Moreover, the introduced papers on approximation theory, except the work of Schmidt-Hieber (2020), hide the dependence on d in the Big- O notation of N in ε -accuracy, even for papers where they consider the function spaces with special structures.

Original: Thus, it is not clear how the d affects the approximation bound and the scale of the provided network architecture.
Revised: Thus, it is not clear how the d affects the approximation bound and the scale of the provided network architecture.

Original: Introduced papers on estimation rate for excess risk also follow the same philosophy with papers on approximation theory, as they work on the fixed d setting.
Revised: Introduced papers on estimation rate for excess risk also follow the same philosophy with papers on approximation theory, as they work on the fixed d setting.

Original: In contrast, we work on the S d − 1 input space, track the explicit dependence on d in the error bound, and describe how d affects the scale of deep ReLU FNN as d → ∞ with its interactions with function smoothness r > 0 .
Revised: In contrast, we work on the S d − 1 input space, track the explicit dependence on d in the error bound, and describe how d affects the scale of deep ReLU FNN as d → ∞ with its interactions with function smoothness r > 0 .

Original: Our paper focuses on tracking the dependence on d in the constant factor hidden in the Big- O notations both in approximation and estimation error rates, rather than paying attentions on reducing the exponential dependence of d with base ε in N or with base n in excess risk bound.
Revised: Our paper focuses on tracking the dependence on d in the constant factor hidden in the Big- O notations both in approximation and estimation error rates, rather than paying attentions on reducing the exponential dependence of d with base ε in N or with base n in excess risk bound.

Original:  P RELIMINARY D EFINITIONS
Revised:  P RELIMINARY D EFINITIONS

Original: In this section, we provide the mathematical definitions of deep ReLU FNN and Sobolev function spaces on unit sphere.
Revised: In this Section, we provide the mathematical definitions of deep ReLU FNN and Sobolev function spaces on unit sphere.

Original: 2. D EFINITION OF D EEP R E LU NETWORK
Revised: 2. D EFINITION OF D EEP R E LU NETWORK

Original: For defining the deep ReLU network mathematically, we adopt the notation used in Schmidt-Hieber (2020).
Revised: For defining the deep ReLU network mathematically, we adopt the notation used in Schmidt-Hieber (2020).

Original: For v = ( v 1 , . . .
Revised: For v = ( v 1 , . . .

Original: , v r ) ∈ R r , let σ v : R r
Revised: , v r ) ∈ R r , let σ v : R r

Original: → R r be the shifted ReLU (Rectified Linear Units) activation function as σ v (( y 1 , . . .
Revised: → R r be the shifted ReLU (Rectified Linear Units) activation function as σ v (( y 1 , . . .

Original: , y r ) ⊤ ) :
Revised: , y r ) ⊤ ) :

Original: = σ (( y 1 − v 1 , . . .
Revised: = σ (( y 1 − v 1 , . . .

Original: , y r − v r ) ⊤ ) , where σ ( x ) = max ( x, 0) .
Revised: , y r − v r ) ⊤ ) , where σ ( x ) = max ( x, 0) .

Original: With this notation, the network architecture ( L, p ) consists of a positive integer L , called the number of hidden layers , and a width vector p :
Revised: With this notation, the network architecture ( L, p ) consists of a positive integer L , called the number of hidden layers , and a width vector p :

Original: = ( p 0 , . .
Revised: = ( p 0 , . .

Original: , p L +1 ) ∈ N L +2 .
Revised: , p L +1 ) ∈ N L +2 .

Original: A deep ReLU network with architecture ( L, p ) considered in this work is then any function of the form
Revised: A deep ReLU network with architecture ( L, p ) considered in this work is then any function of the form

Original: [Equation]
Revised: [Equation]

Original: where W i ∈
Revised: where W i ∈

Original: R p
Revised: R p

Original: i +

Revised: i +

Original: × p
Revised: × p

Original: i is a weight matrix with p 

Revised: i is a weight matrix with p

Original: = d , p L +1 = 1 and v i ∈
Revised: = d , p L +1 = 1 and v i ∈

Original: R p i is a shift vector.
Revised: R p i is a shift vector.

Original: Network functions are built by alternating matrix-vector multiplications with the action of the nonlinear activation function σ .
Revised: Network functions are built by alternating matrix-vector multiplications with the action of the nonlinear activation function σ .

Original: Let ∥ W j ∥ 0 and | v j | 0 be the number of nonzero entries of W j and v j in the j th hidden layer.
Revised: Let ∥ W j ∥ 0 and | v j | 0 be the number of nonzero entries of W j and v j in the j th hidden layer.

Original: The final form of neural network we consider in this paper is given by
Revised: The final form of neural network we consider in this paper is given by

Original: [Equation]
Revised: [Equation]

Original:  The main advantage of using this notation comes from its convenience for tracking the construction process of network ˜ f
Revised: the form [Equation] The main advantage of using this notation comes from its convenience for tracking the construction process of network ˜ f

Original: for approximating f ∈ W r ∞ ( S d − 1 ) .
Revised: for approximating f ∈ W r ∞ ( S d − 1 ) .

Original: See Section B.2 in the Appendix.
Revised: See Section D.2 in the Appendix.

Original: Now, we define the Sobolev spaces over sphere in the next subsection.
Revised: Now, we define the Sobolev spaces over sphere in the next Subsection.

Original: 2. D EFINITION OF S OBOLEV S PACES OVER S PHERE
Revised: 2. D EFINITION OF S OBOLEV S PACES OVER S PHERE

Original: For 1 ≤ p ≤ ∞ , we denote L p ( S d − 1 ) =
Revised: For 1 ≤ p ≤ ∞ , we denote L p ( S d − 1 ) =

Original: L p ( S d − 1 , ρ X ) as the L p -function space defined with respect to the normalized Lebesgue measure ρ X on S d − 1 , with norm ∥ g ∥ p := (cid:0) (cid:82) S d − 1 | g ( x ) | p ρ X ( d x ) (cid:1) 1 /p .
Revised: L p ( S d − 1 , ρ X ) as the L p -function space defined with respect to the normalized Lebesgue measure ρ X on S d − 1 , with norm ∥ g ∥ p := (cid:0) (cid:82) S d − 1 | g ( x ) | p ρ X ( d x ) (cid:1) 1 /p .

Original: Let H dk be the space of homogeneous harmonic polynomials of total degree k ∈ Z + restricted on
Revised: Let H dk be the space of homogeneous harmonic polynomials of total degree k ∈ Z + restricted on

Original: S d − 1 ⊂ R d .
Revised: S d − 1 ⊂ R d .

Original: In Dai & Xu (2013); Efthimiou & Frye (2014), its dimension for k ∈ N is found to be
Revised: In Dai & Xu (2013); Efthimiou & Frye (2014), its dimension for k ∈ N is found to be

Original: [Equation]
Revised: [Equation]

Original: Note that L 2 ( S d − 1 ) is a Hilbert space with inner product ⟨ f, g ⟩ L 2 ( S d − 1 ) :
Revised: Note that L 2 ( S d − 1 ) is a Hilbert space with inner product ⟨ f, g ⟩ L 2 ( S d − 1 ) :

Original: = (cid:82) S d − 1 f ( x ) g ( x ) ρ X ( x ) for f, g ∈ L 2 ( S d − 1 ) .
Revised: = (cid:82) S d − 1 f ( x ) g ( x ) ρ X ( x ) for f, g ∈ L 2 ( S d − 1 ) .

Original: The spaces H dk , for k ∈ Z + , of spherical harmonics are mutually orthogonal with respect to the inner product of L 2 ( S d − 1 ) .
Revised: The spaces H dk , for k ∈ Z + , of spherical harmonics are mutually orthogonal with respect to the inner product of L 2 ( S d − 1 ) .

Original: Since the space of spherical polynomials is dense in L 2 ( S d − 1 ) , every f ∈ L 2 ( S d − 1 ) has a spherical harmonic expansion 
Revised: Since the space of spherical polynomials is dense in L 2 ( S d − 1 ) , every f ∈ L 2 ( S d − 1 ) has a spherical harmonic expansion f

Original: [Equation]
Revised: = (cid:80) ∞

Original: {+k =
 Proj k ( f ) = (cid:80) ∞ k =0 (cid:80) N ( k,d ) ℓ = (cid:98) f k,ℓ Y k,ℓ+} converging in the L 2 ( S d − 1 ) norm.
Revised: {+k =
 Proj k ( f ) = (cid:80) ∞ k =0 (cid:80) N ( k,d ) ℓ = (cid:98) f k,ℓ Y k,ℓ+} converging in the L 2 ( S d − 1 ) norm.

Original: Hereafter, (cid:8) Y k,ℓ (cid:9) N ( k,d ) ℓ = denotes an orthonormal basis of H dk , (cid:98) f k,ℓ is the Fourier coefficients of f given by 
Revised: Hereafter, (cid:8) Y k,ℓ (cid:9) N ( k,d ) ℓ = denotes an orthonormal basis of H dk , (cid:98) f k,ℓ is the Fourier coefficients of f given by (cid:98) f k,ℓ := ⟨ f, Y k,ℓ ⟩ L 2 ( S d − 1 ) :

Original: [Equation]
Revised: = (cid:82) S d − 1 f ( x ) Y k,ℓ ( x )

Original:  and Proj k ( f ) denotes the orthogonal projection of L 2 ( S d − 1 ) onto H dk , which has an integral representation 
Revised: ρ X ( d x ) , and Proj k ( f ) denotes the orthogonal projection of L 2 ( S d − 1 ) onto H dk , which has an integral representation Proj k ( f )( x ) :

Original: Proj
Revised: = (cid:82) S d − 1 f ( y ) Z k ( x , y ) ρ X ( d y ) , ∀ x ∈ S d − 1 , where Z k ( x , y ) : = (cid:80) N

Original: [Equation]
Revised: ( k,d ) ℓ =

Original: where
Revised: Y k,ℓ ( x ) Y k,ℓ ( y ) , ∀ x , y ∈ S

Original: [Equation]
Revised: d − 1 .

Original: We know that Z k ( x , y ) is a reproducing kernel of H dk , independent of the choice of (cid:8) Y k,ℓ (cid:9)
Revised: We know that Z k ( x , y ) is a reproducing kernel of H dk , independent of the choice of (cid:8) Y k,ℓ (cid:9)

Original: N ( k,d ) ℓ = [Equation] and with λ G = d − 22 , 
Revised: N ( k,d ) ℓ = , and with λ G = d − 22 , Z k ( x , y ) :

Original: [Equation]
Revised: = N + λ G λ G G λ G k (cid:0) ⟨ x , y ⟩ (cid:1) , ∀ x , y ∈ S

Original:  where G λ G k is the Gegenbauer polynomial of degree k with parameter λ G > − 12 , see for instance Dai & Xu (2013).
Revised: d − 1 where G λ G k is the Gegenbauer polynomial of degree k with parameter λ G > − 12 , see for instance Dai & Xu (2013).

Original: Denote u := ⟨ x , y ⟩ , the exact expression of G λ G k (cid:0) u (cid:1) is given in terms of the Gamma function by
Revised: Denote u := ⟨ x , y ⟩ , the exact expression of G λ G k (cid:0) u (cid:1) is given in terms of the Gamma function by

Original: [Equation]
Revised: [Equation]

Original: The space of H dk of spherical harmonics can also be characterized as eigenfunction spaces of the
Revised: The space of H dk of spherical harmonics can also be characterized as eigenfunction spaces of the

Original: Laplace-Beltrami operator ∆ S d − 1 on S d − 1 .
Revised: Laplace-Beltrami operator ∆ S d − 1 on S d − 1 .

Original: Indeed, 
Revised: Indeed, H dk = (cid:26)

Original: [Equation]
Revised: [Equation]

Original: where λ k = k ( k + d − 2) and C 2 (cid:0) S d − 1 (cid:1) denotes the space of all twice continuously differentiable functions on S d − 1 .
Revised: where λ k = k ( k + d − 2) and C 2 (cid:0) S d − 1 (cid:1) denotes the space of all twice continuously differentiable functions on S d − 1 .

Original: In fact, with the identity operator I , we may define the fractional power of (cid:0) − ∆ S d − 

Revised: In fact, with the identity operator I , we may define the fractional power of (cid:0) − ∆ S d −

Original: + I (cid:1) α of the operator (cid:0) − ∆ S d − 

Revised: + I (cid:1) α of the operator (cid:0) − ∆ S d −

Original: + I (cid:1) in a distributional sense for α ∈ R :
Revised: + I (cid:1) in a distributional sense for α ∈ R :

Original: [Equation]
Revised: Proj k (cid:0)(cid:0) − ∆ S d − 1 + I (cid:1) α f (cid:1) = (cid:0) 1 + λ k (cid:1) α Proj k (cid:0) f (cid:1) .

Original: Now, we define the Sobolev space W rp ( S d − 1 ) to be the subspace of L p ( S d − 1 ) for 1 ≤ p ≤ ∞ , r > 0 , with the finite norm
Revised: Now, we define the Sobolev space W rp ( S d − 1 ) to be the subspace of L p ( S d − 1 ) for 1 ≤ p ≤ ∞ , r > 0 , with the finite norm

Original: [Equation]
Revised: [Equation]

Revised: In this paper, we consider the case p = ∞ (i.e., f ∈ W r ∞ ( S d − 1 ) ), which is essentially the H ¨ older space.

Original: The sphere S d − 1 is a smooth Riemannian manifold without boundary.
Revised: The sphere S d − 1 is a smooth Riemannian manifold without boundary.

Original: Its nice Laplace-Beltrami operator (i.e., ∆ S d − 1 ) acting as a Hessian operator of functions on the sphere gives the natural definition of Sobolev spaces W r ∞ ( S d − 1 ) in (7); that is, the Sobolev space is a collection of continuous functions defined on sphere S d − 1 whose generalized (distributional) derivatives up to order r are essentially bounded.
Revised: Its nice LaplaceBeltrami operator (i.e., ∆ S d − 1 ) acting as a Hessian operator of functions on the sphere gives the natural definition of Sobolev spaces W r ∞ ( S d − 1 ) in (5); that is, the Sobolev space is a collection of continuous functions defined on sphere S d − 1 whose generalized (distributional) derivatives up to order r are essentially bounded.

Original: See Equations (16) in Hesse (2006), (3.4) in Fang et al.
Revised: See Equations (16) in Hesse (2006), (3.4) in Fang et al.

Original: (2020), (16) in Feng et al.
Revised: (2020), (16) in Feng et al.

Original: (2021), (5.1.9) in Freeden et al.
Revised: (2021), (5.1.9) in Freeden et al.

Original: (1998) for more detailed treatments on W r ∞ ( S d − 1 ) .
Revised: (1998) for more detailed treatments on W r ∞ ( S d − 1 ) .

Original: Readers can also refer the definition of W r ∞ ( C d ) in the Appendix A, when C d = [0 , 1] d , for comparison with W r ∞ ( S d − 1 ) and later use in Subsection 3.1.
Revised: Readers can also refer the definition of W r ∞ ( C d ) in the Appendix A, when C d = [0 , 1] d , for comparison with W r ∞ ( S d − 1 ) and later use in Subsection 3.1.

Original:  A PPROXIMATION ERROR
Revised:  A PPROXIMATION ERROR

Original: Now, we present our Theorem on approximating functions f ∈ W r ∞ ( S d − 1 ) via F ( L, p , N ) in (2).
Revised: Now, we present our Theorem on approximating functions f ∈ W r ∞ ( S d − 1 ) via F ( L, p , N ) in (2).

Original: Theorem 3.1 Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .
Revised: Theorem 3.1 Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .

Original: For any function f ∈
Revised: For any function f ∈

Original: W r ∞ ( S d − 1 ) with r > 0 , there exists a network
Revised: W r ∞ ( S d − 1 ) with r > 0 , there exists a network

Original: [Equation]
Revised: [Equation]

Original: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · [Equation] m + 3) + 2 N + 4) + 1 such that
Revised: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that

Original: [Equation]
Revised: [Equation]

Original: where C ′′ η is a constant dependent on η , and independent on d, r, N, M or f .
Revised: where C ′′ η is a constant dependent on η , and independent on d, r, N, M or f .

Original: The proof of Theorem 3.1 is lengthy and technical.
Revised: The proof of Theorem 3.1 is lengthy and technical.

Original: We provide the detailed proof ideas with technical remarks for the Lemmas and Proposition used for the proof of Theorem 3.1 in the Appendix B.
Revised: We provide the detailed proof ideas with technical remarks for the Lemmas and Proposition used for the proof of Theorem 3.1 in the Appendix D.

Original: The detailed technical proofs of those Lemmas and Proposition are provided in the Appendix C.
Revised: The detailed technical proofs of those Lemmas and Proposition are provided in the Appendix E.

Original: Here, for conciseness, we provide some important remarks on the Theorem and a simple proof sketch, where it starts with a simple triangle inequality: 
Revised: Here, for conciseness, we provide some important remarks on the Theorem and a simple proof sketch, where it starts with a simple triangle inequality: [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Three error terms in (9) correspond to the bounds on three terms of right-hand side in the inequality (10).
Revised: Three error terms in (7) correspond to the bounds on three terms of right-hand side in the inequality (8).

Original: We want to emphasize that the constant C ′′ η > 0 in (9) is independent of d .
Revised: We want to emphasize that the constant C ′′ η > 0 in (7) is independent of d .

Original: Furthermore, we track how the bound is explicitly dependent on d allowing it to tend to infinity.
Revised: Furthermore, we track how the bound is explicitly dependent on d allowing it to tend to infinity.

Original: For first term, note that any f ∈ W r ∞ ( S d − 1 ) is approximated by a weighted sum of Proj k
Revised: For first term, note that any f ∈ W r ∞ ( S d − 1 ) is approximated by a weighted sum of Proj k

Original: ( f ) for 0 ≤ k ≤ 2 N , denoted as L N ( f ) .
Revised: ( f ) for 0 ≤ k ≤ 2 N , denoted as L N ( f ) .

Original: The corresponding approximation error is small for large enough
Revised: The corresponding approximation error is small for large enough

Original: N and r .
Revised: N and r .

Original: Here, importantly, we set the N = ⌈
Revised: Here, importantly, we set the N = ⌈

Original: d α ⌉ for 0 < α < 1 , so that the input dimension d grows faster than N .
Revised: d α ⌉ for 0 < α < 1 , so that the input dimension d grows faster than N .

Original: For second term, notice that the definition of L N ( f ) is involved with the integral over the sphere, and the key for approximating the function is to discretize this integral by M random samples
Revised: For second term, notice that the definition of L N ( f ) is involved with the integral over the sphere, and the key for approximating the function is to discretize this integral by M random samples

Original: y
Revised: y

Original: = { y 1 , . . .
Revised: = { y 1 , . . .

Original: , y M } independently drawn from ρ X .
Revised: , y M } independently drawn from ρ X .

Original: The discretized version of L N ( f ) is denoted as (cid:98) L y N,M ( f ) .
Revised: The discretized version of L N ( f ) is denoted as (cid:98) L y N,M ( f ) .

Original: As observed in the error bound, the higher degree N the L N ( f ) has, the more sampled points M the approximation requires.
Revised: As observed in the error bound, the higher degree N the L N ( f ) has, the more sampled points M the approximation requires.

Original: However, the requirement is ameliorated as r increases.
Revised: However, the requirement is ameliorated as r increases.

Original: Similar effect can be observed in the constant factor in d .
Revised: Similar effect can be observed in the constant factor in d .

Original: The higher the data dimension d is, the more the sampled point M is required for good approximation, but the requirement is alleviated as the smoothness index r increases.
Revised: For the fixed smoothness index r , the higher the data dimension d is, the more the sampled point M is required for good approximation, but the requirement is alleviated as the smoothness index r increases.

Original: If r increases up to order d , the factor 2 decays exponentially fast as d → ∞ , eventually letting M ≥ 1 to be any integer.
Revised: If r increases up to order d , the factordecays exponentially fast as d → ∞ , eventually letting M ≥ 1 to be any integer.

Original: This phenomena is further investigated in the Corollary 3.3.
Revised: This phenomena is further investigated in the Corollary 3.3.

Original: The last term corresponds to the error of the neural network ˜ f approximating (cid:98) L y N,M ( f ) .
Revised: The last term corresponds to the error of the neural network ˜ f approximating (cid:98) L y N,M ( f ) .

Original: For any point x ∈ S d − 1 , the evaluated function value (cid:98) L y N,M ( f )( x ) is simply a weighted average of ξ N,r ( ⟨ x , y i ⟩ ) , for the sampled y =
Revised: For any point x ∈ S d − 1 , the evaluated function value (cid:98) L y N,M ( f )( x ) is simply a weighted average of ξ N,r ( ⟨ x , y i ⟩ ) , for the sampled y =

Original: { y 1 , . . .
Revised: { y 1 , . . .

Original: , y M } .
Revised: , y M } .

Original: Here, ξ N,r ( ⟨ x , y i ⟩ ) is a linear combination of
Revised: Here, ξ N,r ( ⟨ x , y i ⟩ ) is a linear combination of

Original: G λ G k (cid:0) ⟨ x , y i ⟩ (cid:1) in (6) for 0 ≤ k ≤ 2 N .
Revised: G λ G k (cid:0) ⟨ x , y i ⟩ (cid:1) in (4) for 0 ≤ k ≤ 2 N .

Original: Thus, it is sum of univariate polynomials of degree up to 2 N .
Revised: Thus, it is sum of univariate polynomials of degree up to 2 N .

Original: We construct sub-networks approximating ξ N,r ( ⟨ x , y i ⟩ ) for each i ∈
Revised: We construct sub-networks approximating ξ N,r ( ⟨ x , y i ⟩ ) for each i ∈

Original: [ M ] .
Revised: [ M ] .

Original: This explains the width of ˜ f is proportional to NM .
Revised: This explains the width of ˜ f is proportional to NM .

Original: The corresponding error bound is dependent on d 2 N , where it comes from the applications of Stirling’s formula on the coefficient factors in G λ G k (cid:0) ⟨ x , y i ⟩ (cid:1) .
Revised: The corresponding error bound is dependent on d 2 N , where it comes from the applications of Stirling’s formula on the coefficient factors in G λ G k (cid:0) ⟨ x , y i ⟩ (cid:1) .

Original: The error, (cid:0) log 2 (2 N ) (cid:1) 2 2 − 2 m , comes from approximating ⟨ x , y i ⟩ k for 0 ≤ k ≤ 2 N via neural networks.
Revised: The error, (cid:0) log 2 (2 N ) (cid:1) 2 2 − 2 m , comes from approximating ⟨ x , y i ⟩ k for 0 ≤ k ≤ 2 N via neural networks.

Original: The larger the m is, the deeper the network becomes as L
Revised: The larger the m is, the deeper the network becomes as L

Original: = O ( m ) , and the error gets smaller.
Revised: = O ( m ) , and the error gets smaller.

Original: 3. C OMPARISON WITH S CHMIDT -H IEBER (2020)
Revised: 3. C OMPARISON WITH S CHMIDT -H IEBER (2020)

Original: In this subsection, we compare the result from Theorem 3.1 with the result from Schmidt-Hieber (2020), where they consider the approximation of f ∈ W r ∞ ([0 , 1] d ) via deep ReLU FNN.
Revised: In this Subsection, we compare the result from Theorem 3.1 with the result from Schmidt-Hieber (2020), where they consider the approximation of f ∈ W r ∞ ([0 , 1] d ) via deep ReLU FNN.

Original: The
Revised: The

Original: Theorem is stated as follows:
Revised: Theorem is stated as follows:

Original: Theorem 3.2 [Theorem 5 of Schmidt-Hieber (2020)]
Revised: Theorem 3.2 [Theorem 5 of Schmidt-Hieber (2020)]

Original: For any function f ∈ W r ∞ ([0 , 1] d ) and let
Revised: For any function f ∈ W r ∞ ([0 , 1] d ) and let

Original: K > 0 be the radius of H¨older ball.
Revised: K > 0 be the radius of H¨older ball.

Original: Then, for any integers m ≥ 1 and N H ≥ ( r +1)
Revised: Then, for any integers m ≥ 1 and N H ≥ ( r +1)

Original: d ∨ ( K +1)
Revised: d ∨ ( K +1)

Original: e d , there exists a network
Revised: e d , there exists a network

Original: [Equation]
Revised: [Equation]

Original:  with depth L = 8 + ( m + 5) (cid:0) 1 + ⌈ log 2 ( d ∨ r ) ⌉ (cid:1) and the number of parameters N H ≤ 141(1 + d + r ) 3+ d N H ( m + 6) , such that
Revised: (9) with depth L = 8 + ( m + 5) (cid:0) 1 + ⌈ log 2 ( d ∨ r ) ⌉ (cid:1) and the number of parameters N H ≤ 141(1 + d + r ) 3+ d N H ( m + 6) , such that

Original: [Equation]
Revised: [Equation]

Original: To avoid the confusion with the notations used in Theorem 3.1, we put the superscript H to a parameter that determines width of the network (i.e., N H ), to the total number of parameters in the network (i.e., N H ), and to the network class (i.e., F H ).
Revised: To avoid the confusion with the notations used in Theorem 3.1, we put the superscript H to a parameter that determines width of the network (i.e., N H ), to the total number of parameters in the network (i.e., N H ), and to the network class (i.e., F H ).

Original: It is interesting to note that the exponential growth of the network size in d is observed in the construction of F H , whereas there exists a flexibility in F  dependent on the choice of M .
Revised: It is interesting to note that the exponential growth of the network size in d is observed in the construction of F H , whereas there exists a flexibility in F , dependent on the choice of M .

Original: Specifically, the width of the network in F H is exponentially dependent on d as N H =
Revised: Specifically, the width of the network in F H is exponentially dependent on d as N H =

Original: Ω( r d ∨ e d ) , whereas the width of the network in F is dependent on two parameters N = o ( d ) and any integers M ≥ 1 .
Revised: Ω( r d ∨ e d ) , whereas the width of the network in F is dependent on two parameters N = o ( d ) and any integers M ≥ 1 .

Original: For the total number of network parameters, we have N H
Revised: For the total number of network parameters, we have N H

Original: =
Revised: =

Original: O
Revised: O

Original: (( d + r ) d ) , whereas N = O ( Md + Nmd ) .
Revised: (( d + r ) d ) , whereas N = O ( Md + Nmd ) .

Original: Analogously, the bound on the approximation error of ˜ f H in (12) is dependent on d exponentially, but this exponential dependence in d can be avoided in the error bound of ˜ f
Revised: Analogously, the bound on the approximation error of ˜ f H in (10) is dependent on d exponentially, but this exponential dependence in d can be avoided in the error bound of ˜ f

Original: in (9) under two scenarios: (I) r
Revised: in (7) under two scenarios: (I) r

Original: = O ( d ) and any integer M ≥ 1 or (II) r
Revised: = O ( d ) and any integer M ≥ 1 or (II) r

Original: = O (1) and M =
Revised: = O (1) and M =

Original: O ( d d ) .
Revised: O ( d d ) .

Original: In the Corollary presented in the next subsection, we further specify the two scenarios, and describe how the approximation error bound in each scenario converges to 0 in terms of d .
Revised: In the Corollary presented in the next Subsection, we further specify the two scenarios, and describe how the approximation error bound in each scenario converges to 0 in terms of d .

Original: 3. F AST A PPROXIMATION ERROR IN TERMS OF d
Revised: 3. F AST A PPROXIMATION ERROR IN TERMS OF d

Original: Corollary 3.3 Let 0 < α, β, γ < 1 with γ > max { α, β } and N ∈ N with 1 ≤ N ≤ d α + 1 .
Revised: Corollary 3.3 Let 0 < α, β, γ < 1 with γ > max { α, β } and N ∈ N with 1 ≤ N ≤ d α + 1 .

Original: For any f ∈ W r ∞ ( S d − 1 ) with r > 0 , we have: 
Revised: For any f ∈ W r ∞ ( S d − 1 ) with r > 0 , we have: [Equation]

Original: (I) For 3 d − 

Revised: I) For [Equation] d −

Original: [Equation]
Revised: [Equation]

Original: with some constant C 1 ≥ 0 independent of d , there exists a network˜
Revised: with some constant C 1 ≥ 0 independent of d , there exists a network

Original: f ( I ) ∈ F ( L, ( d, 66 N, 66 N, . .
Revised: [Equation]

Original: . , 66 N, 1) , N ) with depth L = O ( d γ log 2 d ) and the number of active parameters N = O (cid:0) d max { α + γ, 1 } (cid:1) , such that (cid:13)(cid:13)(cid:13) f − ˜ f
Revised: with depth L = O ( d γ log 2 d ) and the number of active parameters N = O (cid:0) d max { α + γ, 1 } (cid:1) , such that (cid:13)(cid:13)(cid:13) f − ˜ f

Original: ( I ) (cid:13)(cid:13)(cid:13) ∞ ≤ C ′
Revised: ( I ) (cid:13)(cid:13)(cid:13) ∞ ≤ C ′

Original: η,α,β,γ ∥ f
Revised: η,α,β,γ ∥ f

Original: ∥ W r ∞ ( S d − 1 )
Revised: ∥ W r ∞ ( S d − 1 )

Original: d − d β , where C ′ η,α,β,γ is a constant depending only on C 1 , η, α, β and γ .
Revised: d − d β , where C ′ η,α,β,γ is a constant depending only on C 1 , η, α, β and γ .

Original: (II) For r = O (1) and M = 
Revised: (II) For r = O (1) and M = O (cid:16)

Original: O (cid:16) 9 d d 94 d (cid:17) , there exists a network ˜ f ( II ) ∈ F (cid:0) L, (cid:0) d, 22 NM, . .
Revised: 9 d d 94 d (cid:17) , there exists a network [Equation]

Original: , 22 NM, 1 (cid:1) , N (cid:1) with depth L = O ( d γ log 2 d ) and the number of active parameters N = O (cid:16) 9 d d 134 d (cid:17) such that (cid:13)(cid:13)(cid:13) f − ˜ f
Revised: with depth L = O ( d γ log 2 d ) and the number of active parameters N = O (cid:16) 9 d d 134 d (cid:17) such that (cid:13)(cid:13)(cid:13) f − ˜ f

Original: ( II ) (cid:13)(cid:13)(cid:13) ∞ ≤ C ′
Revised: ( II ) (cid:13)(cid:13)(cid:13) ∞ ≤ C ′

Original: η,α,β,γ ∥ f
Revised: η,α,β,γ ∥ f

Original: ∥ W r ∞ ( S d − 1 ) d − αr , where C ′ η,α,β,γ is a constant depending only on η, α, β and γ .
Revised: ∥ W r ∞ ( S d − 1 ) d − αr , where C ′ η,α,β,γ is a constant depending only on η, α, β and γ .

Original: The detailed proof on Corollary 3.3 is deferred in the Appendix C.6.
Revised: The detailed proof on Corollary 3.3 is deferred in the Appendix E.6.

Original: The approximation error in scenario (I) decays at a rate d − d β for 0 < β < 1 , while the required number of active parameters
Revised: The approximation error in scenario (I) decays at a rate d − d β for 0 < β < 1 , while the required number of active parameters

Original: N is at most O ( d 2 ) .
Revised: N is at most O ( d 2 ) .

Original: Here, the construction of network ˜ f
Revised: Here, the construction of network ˜ f

Original: ( I ) is independent with the choice of M , and we simply choose M = 3 .
Revised: ( I ) is independent with the choice of M , and we simply choose M = 3 .

Original: In scenario (II), since r = O (1) and 0 < α < 1 , the approximation error decays to 0 at d −O (1) rate, which can be slower than d − d β for β close to 1 .
Revised: In scenario (II), since r = O (1) and 0 < α < 1 , the approximation error decays to 0 at d −O (1) rate, which can be slower than d − d β for β close to 1 .

Original: The width of ˜ f ( II ) grows exponentially in d requiring M =
Revised: The width of ˜ f ( II ) grows exponentially in d requiring M =

Original: O ( d d ) .
Revised: O ( d d ) .

Original: Interestingly, in both scenarios, the depth L has the same order in d as O ( d γ log 2 d ) for 0 < γ < 1 .
Revised: Interestingly, in both scenarios, the depth L has the same order in d as O ( d γ log 2 d ) for 0 < γ < 1 .

Revised: Remark 3.4 As suggested by one of the reviewers, we further compare our results in Corollary 3.

Revised: (I) with the CNN architecture with downsampling operation suggested in Fang et al.

Revised: (2020) for approximating f ∈ W r ∞ ( S d − 1 ) , and (II) with Lu et al.

Original:   S TATISTICAL RISK BOUND
Revised: ; Jiao et al. (2021) where they consider the problem of approximating f ∈ W r ∞ ( C d ) vie deep ReLU FNN. Due to the limited space, we defer the detailed remarks on the comparisons in the Appendix C.  S TATISTICAL RISK BOUND

Original: Let X := S d − 1 and Y ⊂ R be the measureable feature space and output space.
Revised: Let X := S d − 1 and Y ⊂ R be the measureable feature space and output space.

Original: We denote ρ as a joint probability measure on the product space Z :=
Revised: We denote ρ as a joint probability measure on the product space Z :=

Original: X × Y , and let ρ X be the marginal distribution of the feature space X .
Revised: X × Y , and let ρ X be the marginal distribution of the feature space X .

Original: We assume that the noisy data set D := { ( x i , y i ) } ni =1 are generated from the non-parametric regression model
Revised: We assume that the noisy data set D := { ( x i , y i ) } ni =1 are generated from the non-parametric regression model

Original: [Equation]
Revised: [Equation]

Original: ) where the noise ε i is assumed to be centered sub-gaussian random variable and E ( ε i | x i )
Revised: where the noise ε i is assumed to be centered sub-gaussian random variable and E ( ε i | x i )

Original: = 0 .
Revised: = 0 .

Original: Our goal is to estimate the regression function f ρ ( x ) with the given noisy data set D .
Revised: Our goal is to estimate the regression function f ρ ( x ) with the given noisy data set D .

Original: Specifically, it is assumed that the regression function belongs to Sobolev space on d -dimensional sphere; that is f ρ ∈ W r ∞ ( S d − 1 ) .
Revised: Specifically, it is assumed that the regression function belongs to Sobolev space on d -dimensional sphere; that is f ρ ∈ W r ∞ ( S d − 1 ) .

Original: It is easy to see regression function f ρ :=
Revised: It is easy to see regression function f ρ :=

Original: E ( y | x ) is a minimizer of the following population risk E ( f ) defined as:
Revised: E ( y | x ) is a minimizer of the following population risk E ( f ) defined as:

Original: [Equation]
Revised: [Equation]

Original: However, since the joint distribution ρ is unknown, we cannot find f ρ directly.
Revised: However, since the joint distribution ρ is unknown, we cannot find f ρ directly.

Original: Instead, we solve a following empirical risk minimization problem induced from the dataset D :
Revised: Instead, we solve a following empirical risk minimization problem induced from the dataset D :

Original: [Equation]
Revised: [Equation]

Original: Note that the function estimator is taken from the feedforward neural network hypothesis space
Revised: Note that the function estimator is taken from the feedforward neural network hypothesis space

Original: F ( L, p , N ) 3 defined in (2),
Revised: F ( L, p , N ) 3 defined in (2),

Original: and we denote the empirical minimizer of (14) as (cid:98) f n .
Revised: and we denote the empirical minimizer of (12) as (cid:98) f n .

Original: It is assumed that | y | ≤ B almost everywhere and we have | f ρ ( x ) | ≤ B .
Revised: It is assumed that | y | ≤ B almost everywhere and we have | f ρ ( x ) | ≤ B .

Original: We project the output function f : S d − 1 → R onto the interval [ − B, B ] by a projection operator
Revised: We project the output function f : S d − 1 → R onto the interval [ − B, B ] by a projection operator

Original: [Equation]
Revised: [Equation]

Original: We consider the clipped estimator π B (cid:98) f n for recovering the regression function f ρ .
Revised: We consider the clipped estimator π B (cid:98) f n for recovering the regression function f ρ .

Original: Note that the clipped estimator has been widely used in statistical learning papers Suzuki (2018); Fang & Cheng (2022); Oono & Suzuki (2019).
Revised: Note that the clipped estimator has been widely used in statistical learning papers Suzuki (2018); Fang & Cheng (2022); Oono & Suzuki (2019).

Original: The quality of π B (cid:98) f n is measured through the difference between two expected risks (i.e., excess risk) defined as E (cid:0) π B (cid:98) f n (cid:1) − E (cid:0) f ρ (cid:1) .
Revised: The quality of π B (cid:98) f n is measured through the difference between two expected risks (i.e., excess risk) defined as E (cid:0) π B (cid:98) f n (cid:1) − E (cid:0) f ρ (cid:1) .

Original: 4. U PPER - BOUND ON EXCESS RISK
Revised: 4. U PPER - BOUND ON EXCESS RISK

Original: In this Subsection, we provide the upper-bound on the excess risk of the clipped estimator π B ( (cid:98) f n ) with respect to the pseudo-dimension (i.e., Pdim ( F ) ) and the approximation error (i.e., ∥ f − f ρ ∥ ∞ ).
Revised: In this Subsection, we provide the upper-bound on the excess risk of the clipped estimator π B ( (cid:98) f n ) with respect to the pseudo-dimension (i.e., Pdim ( F ) ) and the approximation error (i.e., ∥ f − f ρ ∥ ∞ ).

Original: Before presenting the bound, the definition of Pdim ( F ) is presented.
Revised: Before presenting the bound, the definition of Pdim ( F ) is presented.

Original: Definition 4.1 Denote by Pdim ( F ) , the pseudo-dimension of F , which is the largest integer ℓ , for which there exists ( ξ 1 , . .
Revised: Definition 4.1 Denote by Pdim ( F ) , the pseudo-dimension of F , which is the largest integer ℓ , for which there exists ( ξ 1 , . .

Original: . , ξ ℓ , η 1 , . .
Revised: . , ξ ℓ , η 1 , . .

Original: . , η ℓ ) ∈
Revised: . , η ℓ ) ∈

Original: X ℓ × R ℓ such that for any ( a 1 , . .
Revised: X ℓ × R ℓ such that for any ( a 1 , . .

Original: . , a ℓ ) ∈ { 0 , 1 } ℓ
Revised: . , a ℓ ) ∈ { 0 , 1 } ℓ

Original: , there is some f ∈ F satisfying
Revised: , there is some f ∈ F satisfying

Original: [Equation]
Revised: [Equation]

Original: For more comprehensive exploration on Pdim ( F ) can be found in references Anthony & Bartlett (1999); Bartlett et al.
Revised: For more comprehensive exploration on Pdim ( F ) can be found in references Anthony & Bartlett (1999); Bartlett et al.

Original: We provide the first theorem on the excess risk.
Revised: We provide the first theorem on the excess risk.

Original: Proposition 4.2 Set δ ∈ (0 , 1) .
Revised: Proposition 4.2 Set δ ∈ (0 , 1) .

Original: Then, with probability at least 1 − δ , we have
Revised: Then, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where C B,δ,f is an absolute constant dependent on B, δ, f independent on n, r, d .
Revised: where C B,δ,f is an absolute constant dependent on B, δ, f independent on n, r, d .

Original: A detailed proof of Proposition 4.2 is deferred in the Appendix.
Revised: A detailed proof of Proposition 4.2 is deferred in the Appendix.

Original: The excess risk E (cid:0) π B (cid:98) [Equation]
Revised: The excess risk E (cid:0) π B (cid:98) f n (cid:1) − E (cid:0) f

Original:  is a random quantity over the estimator (cid:98) f n and the statement in the Theorem holds with probability at least 1 − δ .
Revised: ρ (cid:1) is a random quantity over the estimator (cid:98) f n and the statement in the Theorem holds with probability at least 1 − δ .

Original: The failure probability δ ∈ (0 , 1) is hidden in the constant C B,δ,f logarithmically, i.e., log( 1 δ ) .
Revised: The failure probability δ ∈ (0 , 1) is hidden in the constant C B,δ,f logarithmically, i.e., log( 1 δ ) .

Original: In the bound, it should be noted that there is a trade-off between the “approximation error” (i.e., ∥ f − f ρ ∥ ∞ ) term and the combinatorial “complexity measure” term of a neural network class F (i.e., Pdim ( F ) · log( n ) /n ); that is, the richer the network hypothesis space F becomes, the finer the approximation result we get.
Revised: In the bound, it should be noted that there is a trade-off between the “approximation error” (i.e., ∥ f − f ρ ∥ ∞ ) term and the combinatorial “complexity measure” term of a neural network class F (i.e., Pdim ( F ) · log( n ) /n ); that is, the richer the network hypothesis space F becomes, the finer the approximation result we get.

Original: Nonetheless, the arbitrary increase in the hypothesis space F eventually leads the increase of the bound in excess risk.
Revised: Nonetheless, the arbitrary increase in the hypothesis space F eventually leads the increase of the bound in excess risk.

Original: In the following subsection, we will show how the specifications (i.e., the choices of ( L, p , N ) ) of the network architecture affect the tension between these two terms.
Revised: In the following Subsection, we will show how the specifications (i.e., the choices of ( L, p , N ) ) of the network architecture affect the tension between these two terms.

Original: 4. C ONVERGENCE R ATE OF E XCESS R ISK
Revised: 4. C ONVERGENCE R ATE OF E XCESS R ISK

Original: Now we are ready to formally state bounds on the excess risks of π M (cid:98) f n when f ρ ∈ W r ∞ ( S d − 1 ) (i.e., Theorem 4.3) and f ρ ∈ W r ∞ ([0 , 1] d ) (i.e., Theorem 4.4), respectively.
Revised: Now we are ready to formally state bounds on the excess risks of π M (cid:98) f n when f ρ ∈ W r ∞ ( S d − 1 ) (i.e., Theorem 4.3) and f ρ ∈ W r ∞ ([0 , 1] d ) (i.e., Theorem 4.4), respectively.

Original: Theorem 4.3 Suppose f ρ ∈ W r ∞ ( S d − 1 ) with r > 0 .
Revised: Theorem 4.3 Suppose f ρ ∈ W r ∞ ( S d − 1 ) with r > 0 .

Original: A network (cid:98) f n from (8) with choices N = ⌈ n 2 3 d +4 r ⌉ , M = ⌈ n 3 d 3 d +4 r ⌉ , and m = ⌈ r 3 d +4 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:
Revised: A network (cid:98) f n from (6) with choices N = ⌈ n 2 3 d +4 r ⌉ , M = ⌈ n 3 d 3 d +4 r ⌉ , and m = ⌈ r 3 d +4 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:

Original: [Equation]
Revised: [Equation]

Original: where C B,η,δ,f depends on B , η , δ , f and independent on d, r and n .
Revised: where C B,η,δ,f depends on B , η , δ , f and independent on d, r and n .

Original: Theorem 4.4 Suppose f ρ ∈ W r ∞ ([0 , 1] d ) with r > 0 .
Revised: Theorem 4.4 Suppose f ρ ∈ W r ∞ ([0 , 1] d ) with r > 0 .

Original: A network (cid:98) f n from (11) with choices N H = ⌈ n d 2 d + r ⌉ , and m H = ⌈
Revised: A network (cid:98) f n from (9) with choices N H = ⌈ n d 2 d + r ⌉ , and m H = ⌈

Original: d + r d +2 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:
Revised: d + r d +2 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:

Original: [Equation]
Revised: [Equation]

Original: where C B,η,δ,K depends on B , η , δ , K and independent on d, r and n .
Revised: where C B,η,δ,K depends on B , η , δ , K and independent on d, r and n .

Original: Detailed proofs on Theorems 4.3 and 4.4 are deferred in the Appendix D.2 and D.3.
Revised: Detailed proofs on Theorems 4.3 and 4.4 are deferred in the Appendix F.2 and F.3.

Original: Both proofs are simple applications of Proposition 4.2 with results from Theorem 3.1 and 3.2.
Revised: Both proofs are simple applications of Proposition 4.2 with results from Theorem 3.1 and 3.2.

Original: For both cases,
Revised: For both cases,

Original: Pdim ( F ) can be easily computed from Lemma E.1 in the Appendix.
Revised: Pdim ( F ) can be easily computed from Lemma H.1 in the Appendix.

Original: The parameters that determine the network architectures, N, M, m and N H , m H in two Theorems are chosen in a way that the bound in (16) is tight in terms of sample size n .
Revised: The parameters that determine the network architectures, N, M, m and N H , m H in two Theorems are chosen in a way that the bound in (14) is tight in terms of sample size n .

Original: Constant factors C B,η,δ,f and C B,η,δ,K are dependent on δ ∈ (0 , 1) as log( 1 δ ) .
Revised: Constant factors C B,η,δ,f and C B,η,δ,K are dependent on δ ∈ (0 , 1) as log( 1 δ ) .

Original: The bound in Theorem 4.3, O d ( n − 2 r 2 r +1 .
Revised: The bound in Theorem 4.3, O d ( n − 2 r 2 r +1 .

Original: 5 d ) , is sub-optimal in a minimax sense for estimating functions f ρ ∈ W r ∞ ( S d − 1 ) , where O d hides the constant factor in d .
Revised: 5 d ) , is sub-optimalin a minimax sense for estimating functions f ρ ∈ W r ∞ ( S d − 1 ) , where O d hides the constant factor in d .

Original: The extra 0 .
Revised: The extra 0 .

Original: 5 d factor in the denominator of exponent comes from the Sobolev embedding
Revised: 5 d factor in the denominator of exponent comes from the Sobolev embedding

Original: Lemma (Lemma B.1.3) and discretization Lemma (Lemma B.1.4).
Revised: Lemma (Lemma D.1.3) and discretization Lemma (Lemma D.1.4).

Original: For the constant factor in d , when r = O (1) , the exponential dependence on d can be observed.
Revised: For the constant factor in d , when r = O (1) , the exponential dependence on d can be observed.

Original: However, when r = O ( d ) , the excess bound in (17) reduces to E (cid:0) π M (cid:98) f n (cid:1) −E (cid:0)
Revised: However, when r = O ( d ) , the excess bound in (15) reduces to E (cid:0) π M (cid:98) f n (cid:1) −E (cid:0)

Original: f ρ (cid:1) ≤ C B,η,δ,f · max (cid:8) (log 2 ( n )) 4 , d 4 N (cid:9) · n − 2 r 2 r +1 .
Revised: f ρ (cid:1) ≤ C B,η,δ,f · max (cid:8) (log 2 ( n )) 4 , d 4 N (cid:9) · n − 2 r 2 r +1 .

Original: 5 d, with N = ⌈ n 2 3 d +4 r ⌉ .Specifically, in high-dimensional setting where n = O
Revised: 5 d .

Original: ( d q ) for 0 < q < 1 , the constant in d becomes d d 2q 3 d +4 r . Then, as d, r → ∞ , the constant d 4 N becomes d O (1) .
Revised: With a choice of N = ⌈ n 2 3 d +4 r ⌉ , as d, r → ∞ , the constant d 4 N becomes d O (1) .

Original: In contrast, in (62) for estimating functions f ρ ∈ W r ∞ ([0 , 1] d ) , the rate n − 2 r 2 r + d is minimax optimal, but we cannot observe the interactions between r and d as we observe in (17).
Revised: In contrast, in (16) for estimating functions f ρ ∈ W r ∞ ([0 , 1] d ) , the rate n − 2 r 2 r + d is minimax optimal, but we cannot observe the interactions between r and d as we observe in (15).

Original: Remark 4.5 From the technical point of view, the result in Theorem 4.3 should be compared with the results in the existing literature, i.e., Schmidt-Hieber (2020); Chen et al. (2019b); Suzuki (2018), in a sense that our result doesn’t require the boundedness of the weight parameters in the network construction.
Revised: Remark 4.5 From the technical point of view, the result in Theorem 4.3 should be compared with the results in the existing literature, i.e., Schmidt-Hieber (2020); Chen et al. (2019b); Suzuki (2018), in a sense that our result doesn’t require the boundedness of the weight parameters in the network construction.

Original: The detailed readings of their proofs reveal that they require the bound on the uniform covering number of F and it can be bounded by the Lipschitzness of the network output with respect to the weight parameters.
Revised: The detailed readings of their proofs reveal that they require the bound on the uniform covering number of F and it can be bounded by the Lipschitzness of the network output with respect to the weight parameters.

Original: Naturally, for the discretizations of the parameter space, the boundedness assumption is required.
Revised: Naturally, for the discretizations of the parameter space, the boundedness assumption is required.

Original: In contrast, in our result, due from the Bartlett et al.
Revised: In contrast, in our result, due from the Bartlett et al.

Original: (See
Revised: (See

Original: Lemma E.1), bounding the complexity measure Pdim ( F ) doesn’t require the parameter boundedness assumption.
Revised: Lemma H.1), bounding the complexity measure Pdim ( F ) doesn’t require the parameter boundedness assumption.

Original:  A N O PEN Q UESTION
Revised:  A N O PEN Q UESTION

Original: In this paper, we prove when r = O ( d ) , deep ReLU FNNs only require at most N =
Revised: In this paper, we prove when r = O ( d ) , deep ReLU FNNs only require at most N =

Original: O ( d 2 ) parameters to get a sharp approximation rate.
Revised: O ( d 2 ) parameters to get a sharp approximation rate.

Original: However, this condition seems restrictive, and needs further investigation whether it is a necessary and sufficient condition to avoid the curse of dimensionality for approximating f ∈ W r ∞ ( S d − 1 ) .
Revised: However, this condition seems restrictive, and needs further investigation whether it is a necessary and sufficient condition to avoid the curse of dimensionality for approximating f ∈ W r ∞ ( S d − 1 ) .

Original: To answer this question, it is essential to study the lower bound of N with a similar approximation error as stated in Theorem 3.1, and see if it has the matching order with the upper-bound we get in d .
Revised: To answer this question, it is essential to study the lower bound of N with a similar approximation error as stated in Theorem 3.1, and see if it has the matching order with the upper-bound we get in d .

Original: We conjecture obtaining this result is possible by combining the ideas of using VC-dimension of deep ReLU FNNs (Bartlett et al., 2019;
Revised: We conjecture obtaining this result is possible by combining the ideas of using VC-dimension of deep ReLU FNNs (Bartlett et al., 2019;

Original: Yarotsky, 2017) and of constructing the packing set on the sphere through the spherical cap (Hesse, 2006), while tracking the d -dependency in the constant factor carefully.
Revised: Yarotsky, 2017) and of constructing the packing set on the sphere through the spherical cap (Hesse, 2006), while tracking the d -dependency in the constant factor carefully.

Original: We leave this for the future research.
Revised: We leave this for the future research.

Original: A d d - DEPENDENT CONSTANT IN N FOR APPROXIMATING f ∈ W r ∞ ([0 , 1] d )
Revised: A d d - DEPENDENT CONSTANT IN N FOR APPROXIMATING f ∈ W r ∞ ([0 , 1] d )

Original: First, we define the function space W r ∞ ([0 , 1] d ) on the d -dimensional unit cube.
Revised: First, we define the function space W r ∞ ([0 , 1] d ) on the d -dimensional unit cube.

Original: For r = n + σ where n ∈ N 0 and σ ∈ (0 , 1] , a function has H¨older smoothness index r if all partial derivatives up to order n exist and are bounded and the partial derivatives of order n are σ H¨older.
Revised: For r = n + σ where n ∈ N 0 and σ ∈ (0 , 1] , a function has H¨older smoothness index r if all partial derivatives up to order n exist and are bounded and the partial derivatives of order n are σ H¨older.

Original: Formally, the ball of r -H¨older functions with radius Q is then defined as
Revised: Formally, the ball of r -H¨older functions with radius Q is then defined as

Original: [Equation]
Revised: [Equation]

Original: where ∂ α f := ∂
Revised: where ∂ α f := ∂

Original: | α | ∂ α 1 ...∂
Revised: | α | ∂ α 1 ...∂

Original: αd f for the multi-index notation, α :=
Revised: αd f for the multi-index notation, α :=

Original: ( α 1 , . . . , α d ) .
Revised: ( α 1 , . . . , α d ) .

Original: The fundamental ideas for approximating functions f ∈ W r ∞ ([0 , 1] d ) in the existing literature rely on a local Taylor approximation technique.
Revised: The fundamental ideas for approximating functions f ∈ W r ∞ ([0 , 1] d ) in the existing literature rely on a local Taylor approximation technique.

Original: The technique dicretizes d -dimensional input cube into a sub-cube set whose size is ( K + 1) d where ( K + 1) is the grid size of each coordinate.
Revised: The technique dicretizes d -dimensional input cube into a sub-cube set whose size is ( K + 1) d where ( K + 1) is the grid size of each coordinate.

Original: For any x in the input cube, the function f is approximated by using the closest 2 d grid points to x via Taylor expansion of f up to the degree ⌊ r ⌋ , where we denote the largest integer less than or equal to u >as ⌊ u ⌋ .
Revised: For any x in the input cube, the function f is approximated by using the closest 2 d grid points to x via Taylor expansion of f up to the degree ⌊ r ⌋ , where we denote the largest integer less than or equal to u >as ⌊ u ⌋ .

Original: Therefore, the total number of active parameters for the net is at least more than the total number of coefficients of partial derivatives ∂ α f :=
Revised: Therefore, the total number of active parameters for the net is at least more than the total number of coefficients of partial derivatives ∂ α f :=

Original: [Equation]
Revised: [Equation]

Revised: This yields the lower bound on the active parameters for the network via local Taylor approximation as ( K + 1) d · (cid:80) ⌊ r ⌋ i =0 (cid:0)

Revised: d

Revised: + i − 1 d (cid:1) .

Revised: B S UBOPTIMAL CONVERGENCE RATE OF EXCESS RISK FOR f ∈ W r ∞ ( S d − 1 )

Revised: Since S d − 1 ⊂ C d , in light of Whitney’s extension theorem, it is obvious that the convergence rate of excess risk for f ∈ W r ∞ ( S d − 1 ) should achieve the minimax optimal rate n − 2 r 2 r + d same as for estimating f ∈ W r ∞ ( C d ) .

Revised: However, from this perspective, it is not possible to track the explicit dependence on d in prefactor of the rate.

Revised: From Proposition 4 . 2 , we track this dependence with the combination of our own approximation result.

Revised: First of all, in order to achieve the minimax learning rate in n (e.g., sample size); that is, n − 2 r 2 r + d , we need to achieve the optimal approximation rate, known as O ( N − rd ) , where N denotes the number of active parameters in the network.

Revised: This can be easily checked in Schmidt-Hieber (2020)’s result.

Revised: But this cannot be achieved in our Theorem 3 . 1 , the result of approximation theorem.

Revised: The main reason arises from the employment of Sobolev embedding theorem (Proposition D.1.3) and of concentration inequality from Smale & Zhou (2007)

Revised: (Lemma E.2.1) used for bounding the term ∥ L N ( f ) − (cid:98) L y N,M ( f ) ∥ ∞ .

Revised: Proposition D.1.3 specifies ∥ L N ( f ) − (cid:98)

Revised: L y N,M ( f ) ∥ ∞ ≤

Revised: C d ∥

Revised: L N ( f ) − (cid:98)

Revised: L y N,M ( f ) ∥

Revised: W s 2 ( S d − 1 ) for some s > (3 d − 2) / 4 and Lemma D.1.4 specifies the relations ∥

Revised: L N ( f ) − (cid:98)

Revised: L y N,M ( f ) ∥

Revised: W s 2 ( S d − 1 ) ≤ C ′ d,r ∥ f

Revised: ∥ W r

Revised: [Equation]

Revised: In this process, there are some extra factors multiplied leading the rate sub-optimal, reflected in the term (2 N + 1) 3 d − 4 r 4 .

Revised: It would be an interesting research direction, if we can develop a nice mathematical framework eliminating this extra factor.

Revised: C F URTHER COMPARISONS WITH EXISTING LITERATURE C. C OMPARISONS OF C OROLLARY 3 . 3 . WITH THE CNN APPROXIMATOR FOR THE APPROXIMATION OF f ∈ W r ∞ ( S d − 1 )

Revised: We consider the case when the approximator is deep ReLU CNN followed by downsampling operations and very few fully-connected layers.

Revised: This is exactly same architecture considered in the paper

Revised: Fang et al.

Revised: (2020), and by applying our result (Lemma D.1.4), we get a following theorem.

Revised: Theorem C.1.

Revised: Let 2 ≤ S ≤ d , 0 < α < 1 , and B, N, M ∈ N with 1 ≤ N ≤ d α + 1 .

Revised: Let

Revised: J ≥ ⌈

Revised: Md − 1 S − 1 ⌉ , D 1 = (2 B + 3)

Revised: ⌊ ( d + JS ) /d ⌋ , and D 2 =

Revised: ⌊ ( d + JS ) /d ⌋ .

Revised: Then, for any function f ∈

Revised: Schmidt-Hieber (Thm 5)

Revised: Lu et al.

Revised: [Table]

Revised: Jiao et al.

Revised: (Corol. 3.1)

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: O

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: W r ∞ ( S d − 1 ) with r > 0 , there exists a network

Revised: ˜ f CNN ∈ H J, D 1 , D 2 ,S with the number of parameters

Revised: N ≤ J (3 S + 2) + M + 2 B + 4 such that

Revised: [Equation]

Original: This yields the lower bound on the active parameters for the network via local Taylor approximation as ( K
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: where C ′′ η,α is a constant dependent on η, α , and independent on d, r, N, M or f .

Revised: The only part we need to pay attention to is bounding the term ∥ (cid:98) L y N,M ( f ) − ˜ f CNN ∥ ∞ , and track the explicit dependence on d in the bound, and its proof is deferred in Appendix G.

Revised: We consider the case r

Revised: = O ( d ) and any integer M ≥ 1 .

Revised: In this case, our result for deep ReLU

Revised: FNN shows that O ( d − d β ) can be achieved for the approximation, with at most O ( d 2 ) active parameters.

Revised: In light of the result from Zhou (2020), we also should expect the same results for CNN with downsampling operation.

Revised: In order to get an approximation rate O ( d − d β ) for some 0 < β < 1 , controlling the first two term in (1.1) is same with that of our proof in the Appendix E.6.

Revised: We only need to pay attention to the last term.

Revised: Since 1 ≤ N ≤ d α + 1 , for some 0 ≤ α < 1 , we have r

Revised: [Equation]

Revised: for some constant C > 0 independent with d > 0 and r > 0 .

Original:  B R OADMAPS FOR PROOF OF T HEOREM 3.
Revised: The rate d − d β is obtainable only when B = O ( d d ) . Then, the number of parameters N ≤ J (3 S + 2) + M + 2 B + 4 is bounded by O ( d d ) . This is an unsatisfatory result.

Revised: However, we firmly believe this result can be improved, and leave it as an open question for the future work.

Revised: C. C OMPARISONS OF C OROLLARY 3 . 3 . WITH THE RECENT APPROXIMATION RESULTS FOR f ∈ W r ∞ ( C d ) VIA DEEP R E LU FNN

Revised: After the publication of Schmidt-Hieber (2020), a series of works further studied the approximation and estimation of f ∈ W r ∞ ( C d ) via deep ReLU FNNs.

Revised: Specifically, Ghorbani et al.

Revised: (2020) pointed out the additive function studied in Schmidt-Hieber (2020) has an exponential dependence in d in the prefactor of the estimation error, requiring n ≿ d d sample sizes for good convergence rate.

Revised: Later, Shen et al.

Revised: (2021) and Lu et al.

Revised: (2021) tracked the explicit dependence on d in the approximation error as well as in the architectural components, (i.e., width ( W ) and depth ( L )), for approximating f ∈ C ( C d ) (i.e., Lipschitz continuous functions on C d ) and f ∈ W r ∞ ( C d ) , respectively.

Revised: Specifically, Lu et al.

Revised: (2021) improved the prefactor O (8 r ( r + 1) d ) when it is compared with

Revised: O (6 d ( r + 1) d ) from Schmidt-Hieber (2020) for the approximation error.

Revised: Most recently, Jiao et al.

Revised: (2021) further improved the prefactor O (( ⌊ r ⌋ + 1) 2 d ⌊ r ⌋ +( r ∨ 1) / 2 ) for approximating f ∈ W r ∞ ( C d ) .

Revised: In Table 2, comparisons of depths, widths, and approximation rates of the constructed deep ReLU networks in Schmidt-Hieber (2020); Lu et al.

Revised: ; Jiao et al.

Revised: (2021) are summarized.

Revised: Since N is not tracked in the works of Lu et al.

Revised: ; Jiao et al.

Revised: (2021), direct comparisons of their results with our results for the approximation of f ∈ W r ∞ ( S d − 1 ) can be challenging.

Revised: However, it can be roughly checked that the result in Jiao et al.

Revised: (2021) doesn’t escape the curse for all r > 0 .

Revised: Specifically, for r = O (1) , the constructed network has the exponential dependence in d for its width.

Revised: For the case r = O ( d ) , achieving the approximation rate O ( d − d β ) requires O ( d d ) width in the suggested network.

Original: In this section of the Appendix, we provide the definitions of L N ( f ) and (cid:98) L y N,M ( f ) along with the overall picture for the proof of Theorem 3.1.
Revised: In this section of the Appendix, we provide the definitions of L N ( f ) and (cid:98) L y N,M ( f ) along with the overall picture for the proof of Theorem 3.1.

Original: Recall we have the following decomposition:
Revised: Recall we have the following decomposition:

Original: [Equation]
Revised: [Equation]

Original: In Subsection B.1, we provide the idea for bounding (I) and (II) .
Revised: In Subsection D.1, we provide the idea for bounding (I) and (II) .

Original: In Subsection B.2, the construction of neural network ˜ f for approximating (cid:98) L y N,M ( f ) is described.
Revised: In Subsection D.2, the construction of neural network ˜ f for approximating (cid:98) L y N,M ( f ) is described.

Original: In this section, no proofs on
Revised: In this section, no proofs on

Original: Propositions and Lemmas are included, but only key ideas for the proofs and technical comparisons with other literature are provided.
Revised: Propositions and Lemmas are included, but only key ideas for the proofs and technical comparisons with other literature are provided.

Original: All the detailed proofs of technical statements in this section are deferred in the Appendix C. 1 .
Revised: All the detailed proofs of technical statements in this section are deferred in the Appendix C. 1 .

Original: B. E RROR BOUNDS FOR (I) AND (II)
Revised: D. E RROR BOUNDS FOR (I) AND (II)

Original: A function f ∈ W r ∞ ( S d − 1 ) is approximated by a linear scheme L N defined as follows.
Revised: A function f ∈ W r ∞ ( S d − 1 ) is approximated by a linear scheme L N defined as follows.

Original: Definition B.1.1 Given a C ∞ ([0 , ∞ ]) function η with η ( t ) = 1 for 0 ≤ t ≤ 1 and η ( t ) = [-
-] for t ≥ 2 , we define a sequence of linear operator L N , N ∈ N , on L p ( S d − 1 ) with 1 ≤ p ≤ ∞ by L N( f )( x ) :=
Revised: Definition D.1.1 Given a C ∞ ([0 , ∞ ]) function η with η ( t ) = 1 for 0 ≤ t ≤ 1 and η ( t ) = [-
-]0 for t ≥ 2 , we define a sequence of linear operator L N , N ∈ N , on L p ( S d − 1 ) with 1 ≤ p ≤ ∞ by L N

Original: [Equation]
Revised: [Equation]

Original: where with λ G = d − 22 , ℓ N,d is a kernel given by
Revised: where with λ G = d − 22 , ℓ N,d is a kernel given by

Original: [Equation]
Revised: [Equation]

Original: It can be found in Dai & Xu (2013) (Chapter 4 ) that L N is near best, achieving the order of best approximation for f ∈ W r p ( S d − 1 ) .
Revised: It can be found in Dai & Xu (2013) (Chapter 4 ) that L N is near best, achieving the order of best approximation for f ∈ W rp ( S d − 1 ) .

Original: Lemma B.1.2 (Lemma 1 in Fang et al.
Revised: Lemma D.1.2 (Lemma 1 in Fang et al.

Original: For N ∈ N , 1 ≤ p ≤ ∞ , r > 0 , and f ∈
Revised: For N ∈ N , 1 ≤ p ≤ ∞ , r > 0 , and f ∈

Original: W rp ( S d − 1 ) , there holds
Revised: W rp ( S d − 1 ) , there holds

Original: [Equation]
Revised: [Equation]

Original: (22) where C η is a constant depending only on the function η in defining L N .
Revised: (21) where C η is a constant depending only on the function η in defining L N .

Original: Note that (cid:0) − ∆ S d − 1 + I (cid:1) r/ 2 is a self-adjoint operator.
Revised: Note that (cid:0) − ∆ S d − 1 + I (cid:1) r/ 2 is a self-adjoint operator.

Original: For x ∈ S d − 1 , recalling the definition of
Revised: For x ∈ S d − 1 , recalling the definition of

Original: L N ( f ) , we have
Revised: L N ( f ) , we have

Original: [Equation]
Revised: [Equation]

Original:  Hereafter, we denote F r = (cid:0) − ∆ S d − 

Revised: [Equation] Hereafter, we denote F r = (cid:0) − ∆ S d −

Original: + I (cid:1) r/ 2 f and ξ N,r ( ⟨ x , ·⟩ ) =
Revised: + I (cid:1) r/ 2 f and ξ N,r ( ⟨ x , ·⟩ ) =

Original: (cid:0) − ∆ S d − 1 + I (cid:1) − r/ 2 ℓ N,d ( ⟨ x , ·⟩ ) .
Revised: (cid:0) − ∆ S d − 1 + I (cid:1) − r/ 2 ℓ N,d ( ⟨ x , ·⟩ ) .

Original: By the fractional power of the operator (cid:0) − ∆ S d − 1 + I (cid:1) − r/ 2 in a distributional sense, ξ N,r ( · ) is a polynomial of degree at most 2 N written as:
Revised: By the fractional power of the operator (cid:0) − ∆ S d − 1 + I (cid:1) − r/ 2 in a distributional sense, ξ N,r ( · ) is a polynomial of degree at most 2 N written as:

Original: [Equation]
Revised: [Equation]

Original: The fractional power of ( − ∆ S d − 1 + I ) caused by the regularity f ∈ W r ∞ ( S d − 1 ) enables rdependent error bound for discretizing L N ( f ) : the larger the regularity r becomes, the smaller the bound for approximation error gets.
Revised: The fractional power of ( − ∆ S d − 1 + I ) caused by the regularity f ∈ W r ∞ ( S d − 1 ) enables rdependent error bound for discretizing L N ( f ) : the larger the regularity r becomes, the smaller the bound for approximation error gets.

Original: Following Fang et al.
Revised: Following Fang et al.

Original: (2020), the key idea for a constructing neural network that approximates
Revised: (2020), the key idea for a constructing neural network that approximates

Original: L N ( f ) is to discretize the integral form (23) by M random samples
Revised: L N ( f ) is to discretize the integral form (22) by M random samples

Original: y
Revised: y

Original: = { y 1 , . . .
Revised: = { y 1 , . . .

Original: , y M } independently drawn from ρ X .
Revised: , y M } independently drawn from ρ X .

Original: We write the discretized version of (23) as :
Revised: We write the discretized version of (22) as :

Original: [Equation]
Revised: [Equation]

Original: Before estimating the distance between L N ( f ) and (cid:98) L y N,M ( f ) , we need a Sobolev embedding property.
Revised: Before estimating the distance between L N ( f ) and (cid:98) L y N,M ( f ) , we need a Sobolev embedding property.

Original: Proposition B.1.3 For d ≥ 5 , 1 ≤ p ≤ ∞ , and s ≥ 3 d − 2 4 , the Sobolev space W sp ( S d − 1 ) is continuously embedded into C ( S d − 1 ) , the space of continuous functions on S d − 1 , which implies
Revised: Proposition D.1.3 For d ≥ 5 , 1 ≤ p ≤ ∞ , and s ≥ 3 d − 2 4 , the Sobolev space W sp ( S d − 1 ) is continuously embedded into C ( S d − 1 ) , the space of continuous functions on S d − 1 , which implies

Original: [Equation]
Revised: [Equation]

Original: where c 0 is an absolute constant independent of r , d , s , and f .
Revised: where c 0 is an absolute constant independent of r , d , s , and f .

Original: Proposition B.1.3 is motivated from Eq.
Revised: Proposition D.1.3 is motivated from Eq.

Original: (14) in Hesse (2006), where they proved ∥ f ∥ ∞ ≤
Revised: (14) in Hesse (2006), where they proved ∥ f ∥ ∞ ≤

Original: C s,d ∥ f ∥ W sp ( S d − 1 ) , f ∈ W sp ( S d − 1 ) for s ≥ d − 1 2 .
Revised: C s,d ∥ f ∥ W sp ( S d − 1 ) , f ∈ W sp ( S d − 1 ) for s ≥ d − 12 .

Original: The constant obtained in Hesse (2006) is
Revised: The constant obtained in Hesse (2006) is

Original: C s,d := (cid:0) 1 ω d (cid:80) ∞ k =
Revised: C s,d := (cid:0) 1 ω d (cid:80) ∞ k =

Original: N ( k,d ) ( k + d − 22 ) 2 s (cid:1) 1 / 2 , where ω d is the surface of d -dimensional sphere.
Revised: N ( k,d ) ( k + d − 22 ) 2 s (cid:1) 1 / 2 , where ω d is the surface of d -dimensional sphere.

Original: For large enough d , (1 /ω d ) 1 / 2 grows in the order of O (cid:0)(cid:0) d 2 πe (cid:1) d/ 4 (cid:1) .
Revised: For large enough d , (1 /ω d ) 1 / 2 grows in the order of O (cid:0)(cid:0) d 2 πe (cid:1) d/ 4 (cid:1) .

Original: Then, by choosing s ≥ 3 d − 2 4 , (1 /ω d ) 1 /can be absorbed into the infinite sum making the constant C s,d converge in an asymptotic regime of d .
Revised: Then, by choosing s ≥ 3 d − 2 4 , (1 /ω d ) 1 /can be absorbed into the infinite sum making the constant C s,d converge in an asymptotic regime of d .

Original: It should be noted that the threshold on smoothness index ( i.e., s ≥ 3 d − 2 4 ) is larger than that from Hesse (2006) ( i.e., s ≥ d − 12 ) , where they consider the fixed d .
Revised: It should be noted that the threshold on smoothness index ( i.e., s ≥ 3 d − 2 4 ) is larger than that from Hesse (2006) ( i.e., s ≥ d − 12 ) , where they consider the fixed d .

Original: See Appendix C.1 for the proof.
Revised: See Appendix E.1 for the proof.

Original: Next, we state the discretization lemma which provides a probabilistic bound on the difference L N ( f ) − (cid:98) L y N,M ( f ) .
Revised: Next, we state the discretization lemma which provides a probabilistic bound on the difference L N ( f ) − (cid:98) L y N,M ( f ) .

Original: Lemma B.1.4 Let r ≤ 3 d − 2and 0 < α < 1 .
Revised: Lemma D.1.4 Let r ≤ 3 d − 2and 0 < α < 1 .

Original: If f ∈ W r ∞ ( S d − 1 ) , then for any M ∈ N and 1 ≤ N ≤
Revised: If f ∈ W r ∞ ( S d − 1 ) , then for any M ∈ N and 1 ≤ N ≤

Original: d
Revised: d

Original: α + 1 , there exist y =
Revised: α + 1 , there exist y =

Original: { y 1 , y 2 , . . .
Revised: { y 1 , y 2 , . . .

Original: , y M } ⊂ S d − 1 such that
Revised: , y M } ⊂ S d − 1 such that

Original: [Equation]
Revised: [Equation]

Original: where C ′′ > 0 is a constant depending on α but independent of r , f , N , M , and d .
Revised: where C ′′ > 0 is a constant depending on α but independent of r , f , N , M , and d .

Original: Lemma B.1.4 is motivated by Lemma 2 in Fang et al.
Revised: Lemma D.1.4 is motivated by Lemma 2 in Fang et al.

Original: The main framework of the proof is based on the Sobolev embedding property in Proposition B.1.3 and the concentration inequality for random variables with values in a Hilbert space, which can be found in Smale & Zhou (2007).
Revised: The main framework of the proof is based on the Sobolev embedding property in Proposition D.1.3 and the concentration inequality for random variables with values in a Hilbert space, which can be found in Smale & Zhou (2007).

Original: For the application of the concentration inequality, the random variable ξ ( y i ) :
Revised: For the application of the concentration inequality, the random variable ξ ( y i ) :

Original: = F r ( y i ) ξ N,r ( ⟨ x , y i ⟩ ) in (25) needs to be bounded in ∥ · ∥ W s 2 ( S d − 1 ) norm for s ≥ 3 d − 2 4 .
Revised: = F r ( y i ) ξ N,r ( ⟨ x , y i ⟩ ) in (24) needs to be bounded in ∥ · ∥ W s 2 ( S d − 1 ) norm for s ≥ 3 d − 2 4 .

Original: See Appendix C.2 for the proof of the Lemma.
Revised: See Appendix E.2 for the proof of the Lemma.

Original: When compared with the technical proof of Lemma 2 from Fang et al.
Revised: When compared with the technical proof of Lemma 2 from Fang et al.

Original: (2020), the most notable difference comes from tracking the explicit dependency on d in the constant factor.
Revised: (2020), the most notable difference comes from tracking the explicit dependency on d in the constant factor.

Original: Specifically, under the fixed d setting, Fang et al.
Revised: Specifically, under the fixed d setting, Fang et al.

Original: (2020) did not explicitly express how the constant c s,r,d (see the statement in their Lemma) depends on d. However, in our paper, since the main focus is how the approximation error behaves under d → ∞ , we need to keep tracking on how d explicitly affects the bound.
Revised: (2020) did not explicitly express how the constant c s,r,d (see the statement in their Lemma) depends on d. However, in our paper, since the main focus is how the approximation error behaves under d → ∞ , we need to keep tracking on how d explicitly affects the bound.

Original: The result of Proposition 4.1 in our paper serves as an important role for this tracking.
Revised: The result of Proposition 4.1 in our paper serves as an important role for this tracking.

Original: Note that the constant c 0 is independent of s, d, r, f in the bound of Proposition 4.1, and we obtain the bound decays at the rate (cid:0) 6 πe (cid:1) d/ 4 .
Revised: Note that the constant c 0 is independent of s, d, r, f in the bound of Proposition 4.1, and we obtain the bound decays at the rate (cid:0) 6 πe (cid:1) d/ 4 .

Original: However, in Fang et al.
Revised: However, in Fang et al.

Original: , they utilized the result from Hesse (2006); that is, ∥ f ∥ ∞ ≤ C s,d ∥ f ∥ W sp ( S d − 1 ) , f ∈ W sp ( S d − 1 ) for s ≥ d − 12 .
Revised: , they utilized the result from Hesse (2006); that is, ∥ f ∥ ∞ ≤ C s,d ∥ f ∥ W sp ( S d − 1 ) , f ∈ W sp ( S d − 1 ) for s ≥ d − 12 .

Original: Here, note that the constant
Revised: Here, note that the constant

Original: C s,d is a function of d, and since they work under the fixed d setting, they didn’t pay much attention to the dependency.
Revised: C s,d is a function of d, and since they work under the fixed d setting, they didn’t pay much attention to the dependency.

Original: Of course, since we are in an asymptotic setting, we use the Stirling’s formula to see behaviors of N ( k, d ) as d → ∞ , whereas Fang et al.
Revised: Of course, since we are in an asymptotic setting, we use the Stirling’s formula to see behaviors of N ( k, d ) as d → ∞ , whereas Fang et al.

Original: (2020) just used a simple calculation
Revised: (2020) just used a simple calculation

Original: N ( k, d ) ≤ c ′ d k d − 2 , for some c ′ d dependent on d.
Revised: N ( k, d ) ≤ c ′ d k d − 2 , for some c ′ d dependent on d.

Original: B. C ONSTRUCTION OF D EEP R E LU N ETWORKS : E RROR BOUND FOR (III)
Revised: D. C ONSTRUCTION OF D EEP R E LU N ETWORKS : E RROR BOUND FOR (III)

Original: In this section, several useful tools for the construction of neural network for approximating function (cid:98) L y N,M ( f ) are introduced.
Revised: In this section, several useful tools for the construction of neural network for approximating function (cid:98) L y N,M ( f ) are introduced.

Original: Then, the full proof of our main theorem is presented.
Revised: Then, the full proof of our main theorem is presented.

Original: The first key lemma is from Yarotsky (2017) wherein the neural network that approximates the quadratic function x 2 for x ∈ [0 , 1] is constructed.
Revised: The first key lemma is from Yarotsky (2017) wherein the neural network that approximates the quadratic function x 2 for x ∈ [0 , 1] is constructed.

Original: Lemma B.2.1 (Proposition 2 in Yarotsky (2017))
Revised: Lemma D.2.1 (Proposition 2 in Yarotsky (2017))

Original: For any positive integer m ≥ 1 , there exists a deep ReLU network
Revised: For any positive integer m ≥ 1 , there exists a deep ReLU network

Original: [Equation]
Revised: [Equation]

Original: such that ˜ f m ∈ [0 , 1] and (cid:12)(cid:12)(cid:12) ˜ f m ( x ) − x 2 (cid:12)(cid:12)(cid:12) ≤ 2 − 2 m − 2 , for all x ∈ [0 , 1] .
Revised: such that ˜ f m ∈ [0 , 1] and (cid:12)(cid:12)(cid:12) ˜ f m ( x ) − x 2 (cid:12)(cid:12)(cid:12) ≤ 2 − 2 m − 2 , for all x ∈ [0 , 1] .

Original: The main idea of Lemma B.2.1 is to approximate the quadratic function via ˜ f m ( x ) :
Revised: The main idea of Lemma D.2.1 is to approximate the quadratic function via ˜ f m ( x ) :

Original: = x − (cid:80) ms =1 g s ( x ) 2 2 s .
Revised: = x − (cid:80) ms =1 g s ( x ) 2 2 s .

Original: Here, g s ( x ) is a s -compositions of sawtooth functions defined as
Revised: Here, g s ( x ) is a s -compositions of sawtooth functions defined as

Original: [Equation]
Revised: [Equation]

Original: Note that g ( x ) can be implemented by a single layer ReLU network.
Revised: Note that g ( x ) can be implemented by a single layer ReLU network.

Original: Then, we can easily construct a ReLU network ˜ f m , which belongs to F ( m, (1 , 5 , . . .
Revised: Then, we can easily construct a ReLU network ˜ f m , which belongs to F ( m, (1 , 5 , . . .

Original: Next lemma states that we can construct a neural network that can implement the multiplication operator.
Revised: Next lemma states that we can construct a neural network that can implement the multiplication operator.

Original: Lemma [-B.2.
-] For any positive integer m ≥ 1 , there exists a deep ReLU network
Revised: Lemma [-B.2.
-]D.2.2 For any positive integer m ≥ 1 , there exists a deep ReLU network

Original: [Equation]
Revised: [Equation]

Original: . , 10 , 1 (cid:1)(cid:1) , such that Mult m ( x, y ) ∈ [0 , 1] and | Mult m
Revised: , such that Mult m ( x, y ) ∈ [0 , 1] and | [Equation]

Original: ( x, y ) − xy | ≤ 2 − 2 m − 1 , for all x, y ∈ [0 , 1] .
Revised: [Equation] , for all x, y ∈ [0 , 1] .

Original: Moreover, Mult m ( x, 0) =
Revised: Moreover, Mult m ( x, 0) =

Original: Mult m (0 , y ) = 0 .
Revised: Mult m (0 , y ) = 0 .

Original: The key idea for constructing Mult m ( x, y ) is to invoke the identity xy = 14 (( x + y ) 2 − ( x − y ) 2 ) .
Revised: The key idea for constructing Mult m ( x, y ) is to invoke the identity xy = 14 (( x + y ) 2 − ( x − y ) 2 ) .

Original: The first two hidden layers in the network are used to compute | x + y 2 | ∈ [0 , 1] and | x − y 2 | ∈ [0 , 1] via | x | = σ ( x ) + σ ( − x ) .
Revised: The first two hidden layers in the network are used to compute | x + y 2 | ∈ [0 , 1] and | x − y 2 | ∈ [0 , 1] via | x | = σ ( x ) + σ ( − x ) .

Original: Given the values | x + y 2 | and | x − y 2 | as inputs, ˜ f m in Lemma B.2.1 is used for approximating 1 4 ( x + y ) 2 and 1 4 ( x − y ) 2 in the identity.
Revised: Given the values | x + y 2 | and | x − y 2 | as inputs, ˜ f m in Lemma D.2.1 is used for approximating 14 ( x + y ) 2 and 14 ( x − y ) 2 in the identity.

Original: See Appendix C.3 for the detailed proof.
Revised: See Appendix E.3 for the detailed proof.

Original: The final key ingredient is to construct a deep ReLU network that approximates univariate polynomial functions of degree k ∈ N , that is x k for x ∈ [0 , 1] .
Revised: The final key ingredient is to construct a deep ReLU network that approximates univariate polynomial functions of degree k ∈ N , that is x k for x ∈ [0 , 1] .

Original: Lemma B.2.3 For any positive integer m ≥ 1 , N ≥ 2 and for P = ⌈ log 2 ( N ) ⌉ , there exists a deep
Revised: Lemma D.2.3 For any positive integer m ≥ 1 , N ≥ 2 and for P = ⌈ log 2 ( N ) ⌉ , there exists a deep

Original: ReLU network
Revised: ReLU network

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , with the depth L = m +( m +4) (cid:0) ⌈ log 2 ( N ) ⌉− 1 (cid:1) and the number of parameters N ≤ 202 N · ( m +3) such that Poly { N } m ( x ) ∈ [0 , 1] 2 P and (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: jm
Revised: P

Original: [Equation]
Revised: [Equation]

Original: x
Revised: , N

Original: [Equation]
Revised: [Equation]

Original: − x j [Equation]
Revised: , with the depth L = m +( m +4) (cid:0) ⌈ log 2 ( N ) ⌉− 1 (cid:1) and the number of parameters N ≤ 202 N · ( m +3) such that Poly { N } m ( x ) ∈ [0 , 1] 2 P and (cid:12)(cid:12)

Original:  ≤ P 2 · 2 − 2 m −for all j ∈ { 1 , . .
Revised: Poly jm ( x ) − x j (cid:12)(cid:12) ≤ P 2 · 2 − 2 m −for all j ∈ { 1 , . .

Original: . , 2 p } for all x ∈ [0 , 1] .
Revised: . , 2 p } for all x ∈ [0 , 1] .

Original: Note that the network Poly { N } m ( x ) :
Revised: Note that the network Poly { N } m ( x ) :

Original: = { Poly 1 m ( x ) , . . .
Revised: = { Poly 1 m ( x ) , . . .

Original: , Poly 2 P m ( x ) } with P = ⌈ log 2 ( N ) ⌉ provides approximations to monomials x j of degree up to 2 N for x ∈ [0 , 1] at its final output.
Revised: , Poly 2 P m ( x ) } with P = ⌈ log 2 ( N ) ⌉ provides approximations to monomials x j of degree up to 2 N for x ∈ [0 , 1] at its final output.

Original: The key idea for the construction is to employ a tree structure; that is, the width of the network at (( m +1)+( m +4) · j ) th hidden layer is doubled from that at (( m +1)+( m +4) · ( j − 1))
Revised: The key idea for the construction is to employ a tree structure; that is, the width of the network at (( m +1)+( m +4) · j ) th hidden layer is doubled from that at (( m +1)+( m +4) · ( j − 1))

Original: th hidden layer for j ∈ { 1 , . .
Revised: th hidden layer for j ∈ { 1 , . .

Original: . , p − 1 } as
Revised: . , p − 1 } as

Original: [Equation]

Original: → (cid:8) Poly 1 m ( x ) , . .

Original: , Poly

Original: [Equation]

Original: m

Original: [Equation]
Revised: [Equation]

Original: Mult m

Original: [Equation]

Original: m

Original: [Equation] [Equation] [Equation] The first 2 j − 1 input values in (( m + 1) + ( m + 4) · j ) th hidden layer is exactly copied from input values at the (( m + 1) + ( m + 4) · ( j − 1))
Revised: The first 2 j − 1 input values in (( m + 1) + ( m + 4) · j ) th hidden layer is exactly copied from input values at the (( m + 1) + ( m + 4) · ( j − 1))

Original: th hidden layer.
Revised: th hidden layer.

Original: The remaining 2 j − 1 input values in (( m +1)+( m +4) · j ) th hidden layer approximates monomials { x 2 j − 1 +1 , . . .
Revised: The remaining 2 j − 1 input values in (( m +1)+( m +4) · j ) th hidden layer approximates monomials { x 2 j − 1 +1 , . . .

Original: , x 2 j } through ˜ f m and
Revised: , x 2 j } through ˜ f m and

Original: Mult m operations in Lemmas B.2.1 and B.2.2.
Revised: Mult m operations in Lemmas D.2.1 and D.2.2.

Original: The approximation error can be obtained via proof by induction.
Revised: The approximation error can be obtained via proof by induction.

Original: Readers can find the detailed proof in the Appendix C.4 with the exact descriptions on the construction of Poly { N } m .
Revised: Readers can find the detailed proof in the Appendix E.4 with the exact descriptions on the construction of Poly { N } m .

Original: Finally, we are ready to state Proposition B.2.4 on the construction of network ˜ f which approximates (cid:98) L y N,M ( f ) .
Revised: Finally, we are ready to state Proposition D.2.4 on the construction of network ˜ f which approximates (cid:98) L y N,M ( f ) .

Original: Proposition B.2.4 Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .
Revised: Proposition D.2.4 Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .

Original: For any function f ∈ W r ∞ ( S d − 1 ) with r > 0 , define (cid:98) L y N,M ( f ) in (25) . 
Revised: For any function f ∈ W r ∞ ( S d − 1 ) with r > 0 , define (cid:98) L y N,M ( f ) in (24) . Then, there exists a network ˜ f [Equation]

Original: Then, there exists a network ˜ f ∈ F (cid:0) L, (cid:0) d, 22 NM, . .
Revised: F [Equation] L, (cid:0) [Equation]

Original: , 22 NM, 1 (cid:1) , N (cid:1) with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that
Revised: , 22 NM, . . . [Equation] 22 NM [Equation]

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that

Original: [Equation]
Revised: [Equation]

Original:  where C ′ η is a positive constant depending on η and α , but not on d, r, m, N, M or f .
Revised: 26) where C ′ η is a positive constant depending on η and α , but not on d, r, m, N, M or f .

Original: A detailed proof for Proposition B.2.4 is deferred in the Appendix C.5.
Revised: A detailed proof for Proposition D.2.4 is deferred in the Appendix E.5.

Original: Given the input data x ∈ S d − 1 , recall the definition of (cid:98) L y N,M ( f )( x ) in (25).
Revised: Given the input data x ∈ S d − 1 , recall the definition of (cid:98) L y N,M ( f )( x ) in (24).

Original: The crux of the whole construction procedure of our network is to build the sub-network which approximates ξ N,r ( ⟨ x , y i ⟩ ) for each i ∈
Revised: The crux of the whole construction procedure of our network is to build the sub-network which approximates ξ N,r ( ⟨ x , y i ⟩ ) for each i ∈

Original: [ M ] .
Revised: [ M ] .

Original: The key observation is that ξ N,r ( ⟨ x , y i ⟩ ) is the weighted sum of univariate polynomials of degree up to 2 N .
Revised: The key observation is that ξ N,r ( ⟨ x , y i ⟩ ) is the weighted sum of univariate polynomials of degree up to 2 N .

Original: Let u i =
Revised: Let u i =

Original: ⟨ x , y i ⟩ .
Revised: ⟨ x , y i ⟩ .

Original: With the properly defined constant α i,q (see its definition in the Appendix C.5), ξ N,r ( ⟨ x , y i ⟩ ) can be re-written as ξ N,r ( u i ) :
Revised: With the properly defined constant α i,q (see its definition in the Appendix E.5), ξ N,r ( ⟨ x , y i ⟩ ) can be re-written as ξ N,r ( u i ) :

Original: = (cid:80) 

Revised: = (cid:80)

Original: Nq =

Revised: Nq =

Original: α i,q | u i | q .
Revised: α i,q | u i | q .

Original: Since | u i | ∈ [0 , 1] , with the help of network constructed in Lemma B.2.3 with P = ⌈ log 2 (2 N ) ⌉ , the sub-network that approximates ξ N,r ( u i ) is easily constructed.
Revised: Since | u i | ∈ [0 , 1] , with the help of network constructed in Lemma D.2.3 with P = ⌈ log 2 (2 N ) ⌉ , the sub-network that approximates ξ N,r ( u i ) is easily constructed.

Original: Recall this is enabled through the reproducing property of the kernel of H dk for 0 ≤ K ≤ 2 N .
Revised: Recall this is enabled through the reproducing property of the kernel of H d k for 0 ≤ K ≤ 2 N .

Original: C P ROOFS OF S TATEMENTS IN A PPENDIX B AND C OROLLARY 3. C. P ROOF OF P ROPOSITION B.1. 
Revised: E P ROOFS OF S TATEMENTS IN A PPENDIX B AND C OROLLARY 3. E. P ROOF OF P ROPOSITION D.1.

Original: Proposition B.1.3 For d ≥ 5 , 1 ≤ p ≤ ∞ , and s ≥ 3 d − 2 4 , the Sobolev space W sp ( S d − 1 ) is continuously embedded into C ( S d − 1 ) , the space of continuous functions on S d − 1 , which implies
Revised: Proposition D.1.3 For d ≥ 5 , 1 ≤ p ≤ ∞ , and s ≥ 3 d − 2 4 , the Sobolev space W sp ( S d − 1 ) is continuously embedded into C ( S d − 1 ) , the space of continuous functions on S d − 1 , which implies

Original: [Equation]
Revised: [Equation]

Original: where c 0 is an absolute constant independent of r , d , s , and f .
Revised: where c 0 is an absolute constant independent of r , d , s , and f .

Original: Proof.
Revised: Proof.

Original: For f ∈ W sp ( S d − 1 ) , by Sobolev embedding Lemma (see Hesse (2006) Eq.
Revised: For f ∈ W sp ( S d − 1 ) , by Sobolev embedding Lemma (see Hesse (2006) Eq.

Original: 14 , p. 420 )
Revised: 14 , p. 420 )

Original: , the infinity norm can be bounded by the Sobolev norm as
Revised: , the infinity norm can be bounded by the Sobolev norm as

Original: [Equation]
Revised: [Equation]

Original: 28) where the constant C s,d is defined with its square as
Revised: where the constant C s,d is defined with its square as

Original: [Equation]
Revised: [Equation]

Original: (29) with ω d = 2 π d 2 / Γ (cid:0) d 2 (cid:1) .
Revised: [Equation] with ω d = 2 π d 2 / Γ (cid:0) d 2 (cid:1) .

Original: Recalling (3), it is easy to see that by Stirling’s formula, for large d ,
Revised: Recalling (3) [Equation] it is easy to see that by Stirling’s formula, for large d ,

Original: N ( k, d ) =
Revised: N ( k, d ) =

Original: ( k + d − 22 )
Revised: ( k + d − 22 )

Original: d − 2 (cid:0) 1 + O (cid:0) 1 d (cid:1)(cid:1) .
Revised: d − 2 (cid:0) 1 + O (cid:0) 1 d (cid:1)(cid:1) .

Original: Also, we haveΓ (cid:18) d (cid:19) = d Γ (cid:18) d 2 +(cid:19) =(cid:114) π d (cid:18) d 2 e (cid:19) d 2 (cid:18) 1 + O (cid:18)d (cid:19)(cid:19) .
Revised: Also, we have

Original: [Equation]
Revised: [Equation]

Original: When s > d − 12 , we have
Revised: When s > d − [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Observe that d ≥ 5 , we have d − 22 − 1 ≥ d 12 .
Revised: Observe that d ≥ 5 , we have d − 22 − 1 ≥ d 12 .

Original: Thus, when s ≥ 3 d − 2 4 , we have 2 s + 1 − d ≥ d/ 2 and thereby (29) is bounded as
Revised: Thus, when s ≥ 3 d − 2 4 , we have 2 s + 1 − d ≥ d/ 2 and thereby (28) is bounded as

Original: [Equation]
Revised: [Equation]

Revised: √ π d √ d (cid:18)πe (cid:19) d 2 (cid:18) 1 + O (cid:18)d (cid:19)(cid:19) .

Original: Then, there exists an absolute constant c 0 such that
Revised: Then, there exists an absolute constant c 0 such that

Original: [Equation]
Revised: [Equation]

Original: This yields the claim.
Revised: This yields the claim.

Original: C. P ROOF OF L EMMA B.1. 
Revised: E. P ROOF OF L EMMA D.1.

Original: Lemma B.1.4 Let 0 < r ≤ 3 d − 2and 0 < α < 1 .
Revised: Lemma D.1.4 Let 0 < r ≤ 3 d − 2and 0 < α < 1 .

Original: If f ∈ W r ∞ ( S d − 1 ) , then for any M ∈ N and 1 ≤ N ≤
Revised: If f ∈ W r ∞ ( S d − 1 ) , then for any M ∈ N and 1 ≤ N ≤

Original: d
Revised: d

Original: α + 1 , there exist y =
Revised: α + 1 , there exist y =

Original: { y 1 , y 2 , . . .
Revised: { y 1 , y 2 , . . .

Original: , y M } ⊂ S d − 1 such that
Revised: , y M } ⊂ S d − 1 such that

Original: [Equation]
Revised: [Equation]

Original: where C ′′ > 0 is a constant depending on α but independent of r , f , N , M , and d .
Revised: where C ′′ > 0 is a constant depending on α but independent of r , f , N , M , and d .

Original: Proof.
Revised: Proof.

Original: We recall the following probability inequality for random variables with values in a Hilbert space which can be found in Smale & Zhou (2007).
Revised: We recall the following probability inequality for random variables with values in a Hilbert space which can be found in Smale & Zhou (2007).

Original: Lemma C.2.1 Let ( H, ∥ · ∥ ) be a Hilbert space and ξ be a random variable on ( Y, ρ X ) with values in H .
Revised: Lemma E.2.1 Let ( H, ∥ · ∥ ) be a Hilbert space and ξ be a random variable on ( Y, ρ X ) with values in H .

Original: Assume ∥
Revised: Assume ∥

Original: ξ ∥ ≤ M < ∞ almost surely.
Revised: ξ ∥ ≤ M < ∞ almost surely.

Original: Denote σ 2 ( ξ ) =
Revised: Denote σ 2 ( ξ ) =

Original: E ( ∥ ξ ∥ 2 ) .
Revised: E ( ∥ ξ ∥ 2 ) .

Original: Let { y i } Mi =1 be independent samples from ρ X .
Revised: Let { y i } Mi =1 be independent samples from ρ X .

Original: Then for any 0 < δ < 1 , we have with probability at least 1 − δ , 
Revised: Then for any 0 < δ < 1 , we have with probability at least 1 − δ , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) M M

Original: [Equation]
Revised: [Equation]

Original: Let us define the random variable ξ on ( S d − 1 , ρ X ) with values in H given by
Revised: Let us define the random variable ξ on ( S d − 1 , ρ X ) with values in H given by

Original: [Equation]
Revised: [Equation]

Original: To bound the norm ∥ ξ ∥ =
Revised: To bound the norm ∥ ξ ∥ =

Original: ∥ ξ ( y ) ∥
Revised: ∥ ξ ( y ) ∥

Original: 2 W s 2 , we set s = [Equation] and recall the norm of W s 2 ( S d − 1 ) given with p = 2 and for y ∈ S
Revised: 2 W s 2 , we set s = 3 d − 2and recall the norm of W s 2 ( S d − 1 ) given with p = 2 and for y ∈ S

Original: d − 1 ,
Revised: d − 1 ,

Original: [Equation]
Revised: [Equation]

Original: Recall λ k = k ( k + d − 2) .
Revised: Recall λ k = k ( k + d − 2) .

Original: Then, for 0 ≤ k ≤ 2 N , d ≥ 3 , we have k 2 < 1 + λ k ≤ dk 2 .
Revised: Then, for 0 ≤ k ≤ 2 N , d ≥ 3 , we have k 2 < 1 + λ k ≤ dk 2 .

Original: We find (1 + λ k ) s − r ≤ d s − r k 2( s − r ) by s = 3 d − 2≥ r ( ∵ s − r ≥ 0) .
Revised: We find (1 + λ k ) s − r ≤ d s − r k 2( s − r ) by s = 3 d − 2≥ r ( ∵ s − r ≥ 0) .

Original: Also note that 0 ≤ η ( t ) ≤for t ∈ [0 , 2] .
Revised: Also note that 0 ≤ η ( t ) ≤for t ∈ [0 , 2] .

Original: Employing Stirling’s formula d ! = √ 2 πd (cid:0) de (cid:1) d (cid:0) 

Revised: Employing Stirling’s formula d ! = √ 2 πd (cid:0) de (cid:1) d (cid:0)

Original: + O (1 /d ) (cid:1) in the expression (3) for N ( k, d ) yields N ( k, d )
Revised: + O (1 /d ) (cid:1) in the expression (3) for N ( k, d ) yields N ( k, d )

Original: ≤ Cd k for 0 ≤ k ≤ 2 N and some constant C depending on α but independent of d .
Revised: ≤ Cd k for 0 ≤ k ≤ 2 N and some constant C depending on α but independent of d .

Original: By using the identity Z k ( y, y ) =
Revised: By using the identity Z k ( y, y ) =

Original: N ( k, d ) (see Corollary 1.2.7.
Revised: N ( k, d ) (see Corollary 1.2.7.

Original: in Dai & Xu (2013)), ∥ ξ ∥ 2 W s 2 ( S d − 1 ) can be bounded as 
Revised: in Dai & Xu (2013)), ∥ ξ ∥ 2 W s 2 ( S d − 1 ) can be bounded as F r (

Original: [Equation]
Revised: [Equation]

Original: while the term (cid:80) 2 Nk =1 k

Original: [Equation]
Revised: [Equation]

Original:  ) with s − r ≥ 0 can be bounded as
Revised: while the term (cid:80) 2 Nk =1 k 2( s − r ) with s − r ≥ 0 can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Combining this with the definitions of the norm ∥ f ∥ W r ∞ ( S d − 1 ) , we know that ∥ ξ ( y ) ∥ 2 W s 2 can be bounded as
Revised: Combining this with the definitions of the norm ∥ f ∥ W r ∞ ( S d − 1 ) , we know that ∥ ξ ( y ) ∥ 2 W s 2 can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where C ′ is a constant depending on α but independent of r , s , f , N , and d .
Revised: where C ′ is a constant depending on α but independent of r , s , f , N , and d .

Original: Thus the random variable ξ satisfies the condition ∥
Revised: Thus the random variable ξ satisfies the condition ∥

Original: ξ ∥ ≤ M < ∞ in Lemma C.2.1 with M =
Revised: ξ ∥ ≤ M < ∞ in Lemma E.2.1 with M =

Original: C ′ ∥
Revised: C ′ ∥

Original: f ∥ W r ∞ ( S d− 1 )
Revised: f ∥ W r ∞ [Equation] S d

Revised: [Equation]

Original:  d N + s − r 2 (2 N + 1) ( s − r ) + 12 .
Revised: 1 ) d N + s − r 2 (2 N + 1) ( s − r )+ 12 .

Original: So by Lemma C.2.1, with δ = 12 and σ 2 ( ξ ) ≤ M 2 , we know from the positive measure of the sample set that there exists a set of points y
Revised: So by Lemma E.2.1, with δ = 12 and σ 2 ( ξ ) ≤ M 2 , we know from the positive measure of the sample set that there exists a set of points y

Original: =
Revised: =

Original: { y i } Mi =1 ∈
Revised: { y i } Mi =1 ∈

Original: S d − 1 such that
Revised: S d − 1 such that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since s = 3 d − 2 4 , combining the result from Proposition B.1.3 with (34) yields
Revised: Since s = 3 d − 2 4 , combining the result from Proposition D.1.3 with (33) yields

Original: [Equation]
Revised: [Equation]

Original: where C ′′ > 0 is a constant depending on α but independent of r , f , N , M , and d .
Revised: where C ′′ > 0 is a constant depending on α but independent of r , f , N , M , and d .

Original: C. P ROOF OF L EMMA B.2.
Revised: E

Revised: P ROOF OF L EMMA D.2.

Original: Lemma [-B.2.
-] For any positive integer m ≥ 1 , there exists a deep ReLU network
Revised: Lemma [-B.2.
-]D.2.2 For any positive integer m ≥ 1 , there exists a deep ReLU network

Original: [Equation]
Revised: [Equation]

Original:  such that Mult m ( x, y ) ∈ [0 , 1] and 
Revised: , such that Mult m ( x, y ) ∈ [0 , 1] and | Mult m ( x, y ) − xy | ≤ 2 − 2 m

Original: [Equation]
Revised: [Equation]

Original:  for all x, y ∈ [0 , 1] .
Revised: 1 , for all x, y ∈ [0 , 1] .

Original: Moreover, Mult m ( x, 0) =
Revised: Moreover, Mult m ( x, 0) =

Original: Mult m (0 , y ) = 0 .
Revised: Mult m (0 , y ) = 0 .

Original: Proof.
Revised: Proof.

Original: Given input ( x, y ) , the network Mult m ( x, y ) computes in the first hidden layer ( x, y ) → (cid:26) σ (cid:18) x
Revised: Given input ( x, y ) , the network Mult m ( x, y ) computes in the first hidden layer [Equation] [Table] [Equation]

Original: + y (cid:19) , σ (cid:18) − (cid:18) x + y (cid:19)(cid:19) , σ (cid:18) x − y (cid:19) , σ (cid:18) − (cid:18) x − y (cid:19)(cid:19)(cid:27) .
Revised: [Table] [Equation]

Original: By using the equality | x | = σ ( x )+ σ ( − x ) for x ∈ [0 , 
Revised: By using the equality | x | = σ ( x )+ σ ( − x ) for x ∈ [0 , 1] [Table]

Revised: the network computes in the second hidden layer

Original: [Table]
Revised: [Table]

Revised: = (cid:12)(cid:12) x − y 2 (cid:12)(cid:12) .

Original: We apply the network ˜ f
Revised: We apply the network ˜ f

Original: m on the two components respectively.
Revised: m on the two components respectively.

Original: This gives a network of ( m + 2) hidden layers with width vector (2 , 10 , . . .
Revised: This gives a network of ( m + 2) hidden layers with width vector (2 , 10 , . . .

Original: , 10 , 2) that computes
Revised: , 10 , 2) that computes

Original: [Table]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The network Mult m computes (35) in the ( m + 3) th hidden layer.
Revised: The network Mult m computes (34) in the ( m + 3) th hidden layer.

Original: Since ˜ f m ∈ [0 , 1] , σ (cid:0) ˜ f m ( x ) (cid:1) =
Revised: Since ˜ f m ∈ [0 , 1] , σ (cid:0) ˜ f m ( x ) (cid:1) =

Original: ˜
Revised: ˜

Original: f m ( x ) .
Revised: f m ( x ) .

Original: In the output layer, the network value is computed as
Revised: In the output layer, the network value is computed as

Original: [Equation]
Revised: [Equation]

Original: [Equation] Since ˜ f m is an increasing function in argument, Mult m ( x, y ) ≥ 0 , and since ˜ f m ∈ [0 , 1] ,
Revised: Since ˜ f m is an increasing function in argument, Mult m ( x, y ) ≥ 0 , and since ˜ f m ∈ [0 , 1] ,

Original: Mult m ( x, y ) ≤ 1 .
Revised: Mult m ( x, y ) ≤ 1 .

Original: By identity, xy = (cid:12)(cid:12) x
Revised: By identity, xy = (cid:12)(cid:12) x

Original: + y
Revised: + y

Original: [Equation]
Revised: [Equation]

Original: 2 − 
Revised: 2 − (cid:12)(cid:12) x − y

Original: [Equation]
Revised: [Equation]

Original: x − y 2 (cid:12)(cid:12) 2 , and Lemma B.2.1, the error is computed as follows:
Revised: 2 , and Lemma D.2.1, the error is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: [Table] ≤ 2 − 2 m − [Table] If either x = 0 or y = 0 , by the definition of (36), we have Mult m ( x, 0)
Revised: If either x = 0 or y = 0 , by the definition of (35), we have Mult m ( x, 0)

Original: =
Revised: =

Original: Mult m (0 , y ) = 0 .
Revised: Mult m (0 , y ) = 0 .

Original: C. P ROOF OF L EMMA B.2. 
Revised: E. P ROOF OF L EMMA D.2.

Original: Lemma B.2.3 For any positive integer m ≥ 1 , N ≥ 2 and for P = ⌈ log 2 ( N ) ⌉ , there exists a deep
Revised: Lemma D.2.3 For any positive integer m ≥ 1 , N ≥ 2 and for P = ⌈ log 2 ( N ) ⌉ , there exists a deep

Original: ReLU network
Revised: ReLU network

Original: Poly { N } m ∈ F (cid:0) L, (cid:0) 1 , 11 N, . .
Revised: Poly { N } m ∈ F (cid:0) L, (cid:0) 1 , 11 N, . .

Original: . , 11 N, 2 P (cid:1) , N (cid:1) , with the depth L = m +( m +4) (cid:0) ⌈ log 2 ( N ) ⌉− 1 (cid:1) and the number of parameters N ≤ 202 N · ( m +3) such that Poly { N } m ( x ) ∈
Revised: . , 11 N, 2 P (cid:1) , N (cid:1) , with the depth L = m +( m +4) (cid:0) ⌈ log 2 ( N ) ⌉− 1 (cid:1) and the number of parameters N ≤ 202 N · ( m +3) such that Poly { N } m ( x ) ∈

Original: [0 , 1] 2 P and (cid:12)(cid:12)
Revised: [0 , 1] 2 P and (cid:12)(cid:12)

Original: Poly jm ( x ) − x j (cid:12)(cid:12) ≤ P 2 · 2 − 2 m −for all j ∈ { 1 , . .
Revised: Poly jm ( x ) − x j (cid:12)(cid:12) ≤ P 2 · [Equation] 2 m −for all j ∈ { 1 , . .

Original: . , 2 P } for all x ∈ [0 , 1] .
Revised: . , 2 P } for all x ∈ [0 , 1] .

Original: Proof.
Revised: Proof.

Original: Let us describe the construction of the network Poly { N } m .
Revised: Let us describe the construction of the network Poly { N } m .

Original: With the application of
Revised: With the application of

Original: Lemma B.2.1, in the ( m + 1) th hidden layer, the network computes
Revised: Lemma D.2.1, in the ( m + 1) th hidden layer, the network computes

Original: [Equation]
Revised: [Equation]

Original: with the width p = (1 , 5 , . . .
Revised: with the width p = (1 , 5 , . . .

Original: For approximating x 3 , the network Mult m is applied on the pair ( σ ( x ) , σ ( ˜ f m ( x ))) , and for approximating x 4 , the network ˜ f m is applied on the σ ( ˜ f m ( x )) .
Revised: For approximating x 3 , the network Mult m is applied on the pair ( σ ( x ) , σ ( ˜ f m ( x ))) , and for approximating x 4 , the network ˜ f m is applied on the σ ( ˜ f m ( x )) .

Original: Therefore, in the { ( m + 1) + ( m + 4) } th hidden layer, the network Poly { N } m computes
Revised: Therefore, in the { ( m + 1) + ( m + 4) } th hidden layer, the network Poly { N } m computes

Original: [Equation]
Revised: [Equation]

Original: Note that each component in the hidden layer is in [0 , 1] by Lemmas B.2.1 and B.2.2.
Revised: Note that each component in the hidden layer is in [0 , 1] by Lemmas D.2.1 and D.2.2.

Original: This procedure is continued until a following vector is in the final output layer, x →
Revised: This procedure is continued until a following vector is in the final output layer, x →

Original: [Equation]
Revised: [Equation]

Original: The resulting network is referred as Poly { N } m and has m + ( m + 4) (cid:0) ⌈ log 2 ( N ) ⌉ − 1 (cid:1) hidden layers.
Revised: The resulting network is referred as Poly { N } m and has m + ( m + 4) (cid:0) ⌈ log 2 ( N ) ⌉ − 1 (cid:1) hidden layers.

Original: Recall P = ⌈ log 2 ( N ) ⌉ .
Revised: Recall P = ⌈ log 2 ( N ) ⌉ .

Original: By the construction procedure of the network, we can compute the upper bound of maximum width as
Revised: By the construction procedure of the network, we can compute the upper bound of maximum width as,

Original: [Equation]
Revised: [Equation]

Original: where we use ⌈ log 2 ( N ) ⌉ ≤ log 2 ( N )+1 in the second inequality.
Revised: where we use ⌈ log 2 ( N ) ⌉ ≤ log 2 ( N )+1 in the second inequality.

Original: Now, we need to count the number of active parameters in the network.
Revised: Now, we need to count the number of active parameters in the network.

Original: For k ∈ { 1 , . . .
Revised: For k ∈ { 1 , . . .

Original: , ⌈ log 2 ( N ) ⌉} , we compute the upper bound on the total number of active parameters in-between following hidden layers:
Revised: , ⌈ log 2 ( N ) ⌉} , we compute the upper bound on the total number of active parameters in-between following hidden layers:

Original: [Equation]
Revised: [Equation]

Original: Think of a network which takes the hidden layer in the left hand side of (39) as an input, and gives the hidden layer in the right hand side of (39) as an output.
Revised: Think of a network which takes the hidden layer in the left hand side of (38) as an input, and gives the hidden layer in the right hand side of (38) as an output.

Original: It is easy to count the number of active parameters in input, hidden, and output layers, separately as follows: 
Revised: It is easy to count the number of active parameters in input, hidden, and output layers, separately as follows:  

Original: [Equation]

Original: 
Revised: Input layer

Original: [Equation]
Revised: [Equation]

Original: Hidden layers
Revised: Hidden layers

Original: [Equation]
Revised: [Equation]

Original: Output layer
Revised: Output layer

Original: [Equation]
Revised: [Equation]

Original: Since the k runs over { 1 , . .
Revised: Since the k runs over { 1 , . .

Original: . , ⌈ log 2 ( N ) ⌉} , the total number of active parameters can be bounded as:
Revised: . , ⌈ log 2 ( N ) ⌉} , the total number of active parameters can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: +(cid:19)(cid:18) 101 · 2 k − 1 − 75 (cid:19) + (cid:18) 14 · 2 k − 1 − 6 (cid:19)(cid:27) ≤ ( m + 2)[-·⌈ log 2 ( N ) ⌉ (cid:88) k = 
 k − 1 + 14-] · ⌈ log 2 ( N ) ⌉ (cid:88) k
Revised: ≤ ( m + 2)[-·⌈ log 2 ( N ) ⌉ (cid:88) k = 
 k − 1 + 14-] · [Equation]

Original: = 2 k −≤ 202 N · ( m + 3) .
Revised: ≤ 202 N · ( m + 3) [Equation]

Original: The approximation error is proved via induction on the number of iterated multiplications P = ⌈ log 2 ( N ) ⌉ .
Revised: The approximation error is proved via induction on the number of iterated multiplications P = ⌈ log 2 ( N ) ⌉ .

Original: For P = 1 , that is N = 2 , we have
Revised: For P = 1 , that is N = 2 , we have

Original: [Equation]
Revised: [Equation]

Original: by Lemma B.2.1.
Revised: by Lemma D.2.1.

Original: For the convenience of notation, denote ˜ x a : =
Revised: For the convenience of notation, denote ˜ x a :=

Original: Poly a m ( x ) for some positive integer a .
Revised: Poly am ( x ) for some positive integer a .

Original: For P = k − 1 , assume a following holds
Revised: For P = k − 1 , assume a following holds

Original: [Equation]
Revised: [Equation]

Original: for j ∈ { 1 , . . .
Revised: for j ∈ { 1 , . . .

Original: , 2 k − 1 } .
Revised: , 2 k − 1 } .

Original: Then, for P =
Revised: Then, for P =

Original: k , we want to prove
Revised: k , we want to prove

Original: [Equation]
Revised: [Equation]

Original: for j ∈ { 1 , . . 
Revised: for j ∈ { 1 , . . .

Original: [Equation]

Original:  k } [Equation]
Revised: , 2 k } .

Original: By the construction of neural network and induction assumption, for j ∈ { 1 , . . .
Revised: By the construction of neural network and induction assumption, for j ∈ { 1 , . . .

Original: , 2 k − 1 } , we have (cid:12)(cid:12) x
Revised: , 2 k − 1 } , we have (cid:12)(cid:12) x

Original: j − ˜ x j (cid:12)(cid:12) ≤ 3 k − 2 · 2 − 2 m − 1 ≤ 3 k − 1 · 2 − 2 m − 1 .
Revised: j − ˜ x j (cid:12)(cid:12) ≤ 3 k − 2 · 2 − 2 m − 1 ≤ 3 k − 1 · 2 − 2 m − 1 .

Original: For any j ∈ { 2 k − 1 + 1 , . . .
Revised: For any j ∈ { 2 k − 1 + 1 , . . .

Original: , 2 k } , find any a, b ∈ { 1 , . .
Revised: , 2 k } , find any a, b ∈ { 1 , . .

Original: , 2 k − 1 } such that j = a + b .
Revised: , 2 k − 1 } such that j = a + b .

Original: Then, for x ∈ [0 , 1] ,
Revised: Then, for x ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: By using the fact log 2 (3) < 2 , we can deduce 3 k − 1 < P 2 and conclude the proof.
Revised: By using the fact log 2 (3) < 2 , we can deduce 3 k − 1 < P 2 and conclude the proof.

Original: C. P ROOF OF P ROPOSITION B.2. 
Revised: E. P ROOF OF P ROPOSITION D.2.

Original: Proposition B.2.4 Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .
Revised: Proposition D.2.4 Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .

Original: For any function f ∈ W r ∞ ( S d − 1 ) with r > 0 , define (cid:98) L y N,M ( f ) in (25) .
Revised: For any function f ∈ W r ∞ ( S d − 1 ) with r > 0 , define (cid:98) L y N,M ( f ) in (24) .

Original: Then, there exists a network˜ f
Revised: Then, there exists a network

Original: [Equation]
Revised: [Equation]

Original: , N

Original: [Equation]
Revised: [Equation]

Original: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that
Revised: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that

Original: [Equation]
Revised: [Equation]

Original: where C ′ η is a positive constant depending on η and α , but not on d, r, m, N, M or f .
Revised: where C ′ η is a positive constant depending on η and α , but not on d, r, m, N, M or f .

Original: Proof.
Revised: Proof.

Original: We adopt the shorthand notation denoting [ n ] := { 1 , 2 , . . . , n } and [ n ] 0 :
Revised: We adopt the shorthand notation denoting [ n ] := { 1 , 2 , . . . , n } and [ n ] 0 :

Original: = { 0 , 1 , . .
Revised: = { 0 , 1 , . .

Original: . , n } for n ∈ N in the proof.
Revised: . , n } for n ∈ N in the proof.

Original: Given the input data x ∈ S d − 1 , recall the definition of (cid:98) L y N,M ( f )( x ) in (25).
Revised: Given the input data x ∈ S d − 1 , recall the definition of (cid:98) L y N,M ( f )( x ) in (24).

Original: The crux of the whole construction procedure is to build the the sub-network which approximates ξ N,r ( ⟨ x , y i ⟩ ) for each i ∈
Revised: The crux of the whole construction procedure is to build the the sub-network which approximates ξ N,r ( ⟨ x , y i ⟩ ) for each i ∈

Original: [ M ] .
Revised: [ M ] .

Original: First, observe that, by (6) and (24), ξ N,r ( u i ) can be written as: 
Revised: First, observe that, by (4) and (23), ξ N,r ( u i ) can be written as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 41) for i ∈
Revised: for i ∈

Original: [ M ] .
Revised: [ M ] .

Original: The key observation is that Eq.
Revised: The key observation is that Eq.

Original: (41) is the weighted sum of univariate polynomials of degree up to 2 N .
Revised: (40) is the weighted sum of univariate polynomials of degree up to 2 N .

Original: We define a constant c k,ℓ,η,λ k ,r,d as
Revised: We define a constant c k,ℓ,η,λ k ,r,d as

Original: [Equation]
Revised: [Equation]

Original: For i ∈ { 1 , . . .
Revised: For i ∈ { 1 , . . .

Original: , M } , set α
Revised: , M } , set α

Original: i,q as
Revised: i,q as

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: u i

Original:  otherwise, (43) where for each q ∈ { 0 , . . . 
Revised: [Equation] q is odd, [Equation] otherwise, (42) where for each q ∈ { 0 , . . . , 2 N } ,

Original: , 2 N } , the set
Revised: the set

Original: A q is given by A q := { ( k, ℓ ) ∈ [2 N ] 0 ×
Revised: A q is given by A q := { ( k, ℓ ) ∈ [2 N ] 0 ×

Original: [ ⌊ k/ 2 ⌋ ] 0 : k − 2 ℓ = q } .
Revised: [ ⌊ k/ 2 ⌋ ] 0 : k − 2 ℓ = q } .

Original: Then, (41) can be re-written as ξ N,r ( u i ) :
Revised: Then, (40) can be re-written as ξ N,r ( u i ) :

Original: = (cid:80) 

Revised: = (cid:80)

Original: Nq =

Revised: Nq =

Original: α i,q | u i | q .
Revised: α i,q | u i | q .

Original: The Network Construction.
Revised: The Network Construction.

Original: Now, we are ready for the construction of ˜ f .
Revised: Now, we are ready for the construction of ˜ f .

Original: Through
Revised: Through

Original: Lemma B.1.4, we know that there exists y = { y 1 , . . .
Revised: Lemma D.1.4, we know that there exists y = { y 1 , . . .

Original: , y M } that satisfies the bound (B.1.4).
Revised: , y M } that satisfies the bound (D.1.4).

Original: Then, for each i ∈
Revised: Then, for each i ∈

Original: [ M ] , we put y i ∈ S
Revised: [ M ] , we put y i ∈ S

Original: d − 1 as a weight vector that connects input x to the (2 i − 1) th and (2 i ) th nodes in the first hidden layer.
Revised: d − 1 as a weight vector that connects input x to the (2 i − 1) th and (2 i ) th nodes in the first hidden layer.

Original: Through this, ˜ f computes in its first hidden layer x → (cid:8) σ (cid:0) ⟨ x , y 1 ⟩ (cid:1) , σ (cid:0) − ⟨ x , y 1 ⟩ (cid:1) , . . .
Revised: Through this, ˜ f computes in its first hidden layer x → (cid:8) σ (cid:0) ⟨ x , y 1 ⟩ (cid:1) , σ (cid:0) − ⟨ x , y 1 ⟩ (cid:1) , . . .

Original: , σ (cid:0) ⟨ x , y M ⟩ (cid:1) , σ (cid:0) − ⟨ x , y M ⟩ (cid:1)(cid:9) ∈ [0 , 1] 2 M .
Revised: , σ (cid:0) ⟨ x , y M ⟩ (cid:1) , σ (cid:0) − ⟨ x , y M ⟩ (cid:1)(cid:9) ∈ [0 , 1] 2 M .

Original: Then, by the identity | x | = σ ( x ) + σ ( − x ) for x ∈ R , the network computes in its second hidden layer x → (cid:8) σ (cid:0) | u 1 | (cid:1) , σ (cid:0) | u 2 | (cid:1) , . . .
Revised: Then, by the identity | x | = σ ( x ) + σ ( − x ) for x ∈ R , the network computes in its second hidden layer x → (cid:8) σ (cid:0) | u 1 | (cid:1) , σ (cid:0) | u 2 | (cid:1) , . . .

Original: , σ (cid:0) | u M | (cid:1)(cid:9) ∈ [0 , 1] M , where u i := ⟨ x , y i ⟩ ∈ [ − 1 , 1] for i ∈ [ M ] .
Revised: , σ (cid:0) | u M | (cid:1)(cid:9) ∈ [0 , 1] M , where u i := ⟨ x , y i ⟩ ∈ [ − 1 , 1] for i ∈ [ M ] .

Original: Since σ ( | u i | ) = | u i | ∈
Revised: Since σ ( | u i | ) = | u i | ∈

Original: [0 , 1] , Poly { 2N } m with P = ⌈ log 2 (2 N ) ⌉ is applicable for each {| u i |} Mi =1 , and it generates Poly qm ( | u i | ) with q at most 4 N .
Revised: [0 , 1] , Poly { 2N } m with P = ⌈ log 2 (2 N ) ⌉ is applicable for each {| u i |} Mi =1 , and it generates Poly qm ( | u i | ) with q at most 4 N .

Original: Set B max := max i =1 ,...,M (cid:12)(cid:12)(cid:12)(cid:80) 

Revised: Set B max := max i =1 ,...,M (cid:12)(cid:12)(cid:12)(cid:80)

Original: Nq =

Revised: Nq =

Original: α i,q · Poly qm ( | u i | ) (cid:12)(cid:12)(cid:12) .
Revised: α i,q · Poly qm ( | u i | ) (cid:12)(cid:12)(cid:12) .

Original: Using the definition of the constant α i,q , the network ˜ f computes in the ( m + 4) ⌈ log 2 (2 N )
Revised: Using the definition of the constant α i,q , the network ˜ f computes in the ( m + 4) ⌈ log 2 (2 N )

Original: ⌉ th hidden layer { σ ( (cid:80) 

Revised: ⌉ th hidden layer { σ ( (cid:80)

Original: Nq =

Revised: Nq =

Original: α 1 ,q Poly qm ( | u 1 | ) + 2 B max ) , . . .
Revised: α 1 ,q Poly qm ( | u 1 | ) + 2 B max ) , . . .

Original: , σ ( (cid:80) 

Revised: , σ ( (cid:80)

Original: Nq =

Revised: Nq =

Original: α M,q Poly qm ( | u M | ) + 2 B max ) } ∈ R M .
Revised: α M,q Poly qm ( | u M | ) + 2 B max ) } ∈ R M .

Original: By the definition of B max , it is easy to see each component in the hidden layer is positive.
Revised: By the definition of B max , it is easy to see each component in the hidden layer is positive.

Original: Set the weight of output layer as { 1 M F r ( y i ) } Mi =1 .
Revised: Set the weight of output layer as { 1 M F r ( y i ) } M i =1 .

Original: Define L ( | u i | ) :
Revised: Define L ( | u i | ) :

Original: = (cid:80) 

Revised: = (cid:80)

Original: Nq =

Revised: Nq =

Original: α i,q · Poly qm ( | u i | ) + 2 · B max .
Revised: α i,q · Poly qm ( | u i | ) + 2 · B max .

Original: Then, given the data y =
Revised: Then, given the data y =

Original: { y 1 , . .
Revised: { y 1 , . .

Original: , y M } , the network ˜ f computes its final output as ˜ f
Revised: , y M } , the network ˜ f computes its final output as ˜ f

Original: ( x ) = 1 M (cid:80) Mj
Revised: ( x ) = 1 M (cid:80) Mj

Original: =1 F r ( y j ) · (cid:0) L ( |⟨ x , y j ⟩| ) − 2 B max (cid:1) :
Revised: =1 F r ( y j ) · (cid:0) L ( |⟨ x , y j ⟩| ) − 2 B max (cid:1) :

Original: = 1 M (cid:80) Mi
Revised: = 1 M (cid:80) Mi

Original: =1 F r ( y i ) ·
Revised: =1 F r ( y i ) ·

Original: L (cid:0) ξ N,r (cid:1)(cid:0) ⟨ x , y i ⟩ (cid:1) .
Revised: L (cid:0) ξ N,r (cid:1)(cid:0) ⟨ x , y i ⟩ (cid:1) .

Original: The Width and Number of Active Parameters of ˜ f .
Revised: The Width and Number of Active Parameters of ˜ f .

Original: By the construction of network ˜ f and the result of Lemma B.2.3, it is easy to see the maximum width of the network is 22 NM .
Revised: By the construction of network ˜ f and the result of Lemma D.2.3, it is easy to see the maximum width of the network is 22 NM .

Original: Now, we work oncounting the numberof active parameters in the network as  
Revised: Now, we work on the number in the network as  

Original: From Input to 2 nd hidden layer : 2 Md + 2 M.
Revised: From Input to 2 nd hidden layer : [Equation] 2 M.

Original: From 2 nd to (cid:0) ( m + 4) ⌈ log 2 (2 N ) ⌉ − 1 (cid:1) th hidden layer : 404 NM · ( m + 3) .
Revised: From 2 nd to (cid:0) ( m + 4) ⌈ log 2 (2 N ) ⌉ − 1 (cid:1) th hidden layer : 404 NM · ( m + 3) .

Original: From (cid:0) ( m + 4) ⌈ log 2 (2 N ) ⌉ − 1 (cid:1) th hidden layer to output layer : (2 N + [Equation]
Revised: From (cid:0) ( m + 4) ⌈ log 2 (2 N ) ⌉ − 1 (cid:1) th hidden layer to output layer : (2 N + 1) M + M + 1 .

Original: Summing up the total number yields the desired result.
Revised: Summing up the total number yields the desired result.

Original: Approximation Error Computation.
Revised: Approximation Error Computation.

Original: A remaining thing is to calculate the approximation error:
Revised: A remaining thing is to calculate the approximation error:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Recall the definition of α i,q in (43).
Revised: Recall the definition of α i,q in (42).

Original: Using Stirling’s Formula, Γ( n +1) = √ 2 πn (cid:0) ne (cid:1) n (1+ O (1 /n )) , we observe the behavior of Gegenbauer coefficient in (6) where λ G = d
Revised: Using Stirling’s Formula, Γ( n +1) = √ 2 πn (cid:0) ne (cid:1) n (1+ O (1 /n )) , we observe the behavior of Gegenbauer coefficient in (4) where λ G = d

Original: − 

Revised: −

Original: ≫ d α +1 ≥ N , and find that it can be bounded as C · λ k − ℓ G · 2 k − 2 ℓ (1 + O (1 /d )) , where C > 0 is a constant independent of d .
Revised: ≫ d α +1 ≥ N , and find that it can be bounded as C · λ k − ℓ G · 2 k − 2 ℓ (1 + O (1 /d )) , where C > 0 is a constant independent of d .

Original: For k ∈ { 0 , 1 , . . .
Revised: For k ∈ { 0 , 1 , . . .

Original: , 2 N } , combining the facts (1 + λ k ) − r 2 < 1 , η ( · ) ≤ 1 , k + λ G λ G ≤ 2 for k ≤ 2 N ≤ 2( d α + 1) with λ G = d − 22 yields[Equation]
Revised: , 2 N } , combining the facts (1 + λ k ) − r 2 < 1 , η ( · ) ≤ 1 , k + λ G λ G ≤ 2 for k ≤ 2 N ≤ 2( d α + 1) with λ G = d − 22 yields

Original: [Equation]
Revised: [Equation]

Original: where C ′ η > 0 is a constant dependent on α and η .
Revised: where C ′ η > 0 is a constant dependent on α and η .

Original: Recall L (cid:0) ξ N,r (cid:1)(cid:0) ⟨ x , y j ⟩ (cid:1) :
Revised: Recall L (cid:0) ξ N,r (cid:1)(cid:0) ⟨ x , y j ⟩ (cid:1) :

Original: = (cid:80) 

Revised: = (cid:80)

Original: Nq =

Revised: Nq =

Original: α j,q ·
Revised: α j,q ·

Original: Poly qm ( |⟨ x , y j ⟩| ) and note that (cid:80) 

Revised: Poly qm ( |⟨ x , y j ⟩| ) and note that (cid:80)

Original: Nq =0 | α j,q | = (cid:80) 2 Nk
Revised: Nq =0 | α j,q | = (cid:80) 2 Nk

Original: =0 (cid:80)
Revised: =0 (cid:80)

Original: ⌊ k 2 ⌋ ℓ =0 (cid:12)(cid:12) c
Revised: ⌊ k 2 ⌋ ℓ =0 (cid:12)(cid:12) c

Original: k,ℓ,η,λ k ,r,d (cid:12)(cid:12) .
Revised: k,ℓ,η,λ k ,r,d (cid:12)(cid:12) .

Original: Then, we have 
Revised: Then, we have (cid:13)(cid:13) ξ N,r − L (cid:0) ξ N, [Equation] (cid:1)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: where we used the result from Lemma B.2.3 and (45) in the second inequality.
Revised: ≤ (cid:18) [Equation] N (cid:88) k

Original: Using (cid:80) ⌊ k 2 ⌋ ℓ = 
Revised: = ⌊ [Equation] [Equation] (cid:88)

Original: [Equation]
Revised: [Equation]

Original:  ≤ 2 in the last inequality yields the claim.
Revised: [Equation] [Equation] where we used the result from Lemma D.2.3 and (44) in the second inequality. Using (cid:80) ⌊ k 2 ⌋ ℓ = 1 (2 d ) ℓ ≤ 2 in the last inequality yields the claim.

Revised: E. P ROOF OF C OROLLARY 3.

Original: Corollary 3.3 Let 0 < α, β, γ < 1 with γ > max { α, β } and N ∈ N with 1 ≤ N ≤ d α + 1 .
Revised: Corollary 3.3 Let 0 < α, β, γ < 1 with γ > max { α, β } and N ∈ N with 1 ≤ N ≤ d α + 1 .

Original: For any f ∈ W r ∞ ( S d − 1 ) with r > 0 , we have :
Revised: For any f ∈ W r ∞ ( S d − 1 ) with r > 0 , we have :

Original: (I) For 3 d − 2− C 1 ≤ r ≤ 3 d − 2with some constant C 1 ≥ 0 independent of d , there exists a network ˜ f ( I ) ∈ F ( L, ( d, 66 N, 66 N, . . . , 66 N, 1) , N ) with depth L = O ( d γ log 2 d ) and the number of active parameters N = O (cid:0) d max { α + γ, 1 } (cid:1) , such that (cid:13)(cid:13)(cid:13) f − ˜ f{+
+}
Revised: (I) For 3 d − {+
+}

Original: ( I ) (cid:13)(cid:13)(cid:13) ∞ ≤ C ′

Original: η,α,β,γ ∥ f

Original: ∥ W r ∞ ( S d − 1 )

Original: d − d β , where C ′ η,α,β,γ is a constant depending only on C 1 , η, α, β, γ .

Original: (II) For r = O (1) and M = O (cid:16)

Original: [Equation]
Revised: [Equation]

Original:  d94 d [Equation] , there exists a network
Revised: 2with some constant C 1 ≥ 0 independent of d , there exists a network

Original: [Equation]
Revised: [Equation]

Original: with depth L
Revised: with depth L

Original: = O ( d γ log 2 d ) and the number of active parameters N = O (cid:16) 9 d d 134 d (cid:17) such that (cid:13)(cid:13)(cid:13)f
Revised: = O ( d γ log 2 d ) and the number of active parameters N = O (cid:0) d max { α + γ, 1 } (cid:1) , such that (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: f
Revised: ( I )

Original: [Equation]
Revised: [Equation]

Revised: C

Original: [Equation]
Revised: [Equation]

Original: αr , where C ′ η,α,β,γ is a constant depending only on  η, α, β, γ 
Revised: η,α,β,γ ∥ f ∥ W r ∞ ( S d − 1 ) d − d β , where C ′ η,α,β,γ is a constant depending only on C 1 , η, α, β, γ . (II) For r = O (1) and M = O (cid:16) 9 d d 94 d (cid:17) , there exists a network

Original: [Equation]
Revised: [Equation]

Revised: with depth L

Revised: = O ( d γ log 2 d ) and the number of active parameters N = O (cid:16)

Revised: 9 d d 134 d such that where C ′ η,α,β is a constant depending only on η, α, β,

Revised: γ

Original: Proof.
Revised: Proof.

Original: By the results of Theorem 3.1, for 1 ≤ N ≤ d α + 1 , we have the following inequality on the approximation error
Revised: By the results of Theorem 3.1, for 1 ≤ N ≤ d α + 1 , we have the following inequality on the approximation error

Original: [Equation]
Revised: [Equation]

Original:  where C ′′ η is a constant dependent on η , and independent on d, r, N, M or f .
Revised: ) where C ′′ η is a constant dependent on η , and independent on d, r, N, M or f .

Original: We divide the proof into two cases.
Revised: We divide the proof into two cases.

Original: (I) r
Revised: (I) r

Original: = O ( d ) and any integer M ≥
Revised: = O ( d ) and any integer M ≥

Original: For the first term in (46), since N = ⌈ d α ⌉ , we know that N − r
Revised: For the first term in (45), since N = ⌈ d α ⌉ , we know that N − r

Original: =
Revised: =

Original: O ( d − αr )
Revised: O ( d − αr )

Original: =
Revised: =

Original: O ( d − d β ) with any 0 < β < 1 .
Revised: O ( d − d β ) with any 0 < β < 1 .

Original: This is due to the assumption that 3 d − 2− C 1 ≤ r ≤ 3 d − 2 4 , which implies d = O ( r ) and d β = o ( r ) .
Revised: This is due to the assumption that 3 d − 2− C 1 ≤ r ≤ 3 d − 2 4 , which implies d = O ( r ) and d β = o ( r ) .

Original: For the second term in (46), since N = ⌈
Revised: For the second term in (45), since N = ⌈

Original: d α ⌉ with 0 < α < 1 , we know that it is bounded by
Revised: d α ⌉ with 0 < α < 1 , we know that it is bounded by

Original: [Equation]
Revised: [Equation]

Original: As 3 d − 2− C 1 ≤ r ≤ 3 d − 2 4 , we know the term on the right hand side of (47) can be written as(cid:0) 6 πe (cid:1)
Revised: As 3 d − 2− C 1 ≤ r ≤ 3 d − 2 4 , we know the term on the right hand side of (46) can be written as

Original: d 4 d d α + O (1) 3 O (1) / √ M .
Revised: [Equation] α + O (1) 3 O (1) / √ M .

Original: To show that the bound is of order O ( d − d β ) , we multiply the bound by d d β , take the logarithm, and find that for any 0 < α, β < 1 , 
Revised: To show that the bound is of order O ( d − d β ) , we multiply the bound by d d β , take the logarithm, and find that for any 0 < α, β < 1 , log [Equation]

Original: [Equation]
Revised: [Equation]

Original: as d → ∞ .
Revised: as d → ∞ .

Original: Hence, there exists a constant C α,β > 0 depending only on C 1 , α , β such that
Revised: Hence, there exists a constant C α,β > 0 depending only on C 1 , α , β such that

Original: [Equation]
Revised: [Equation]

Original: for any fixed M ∈ N .
Revised: for any fixed M ∈ N .

Original: In our proof, we simply choose M = 3 .
Revised: In our proof, we simply choose M = 3 .

Original: For the third term in (46), take m = ⌈
Revised: For the third term in (45), take m = ⌈

Original: d γ ⌉ with max { α, β } < γ < 1 , then there exists a constant C α,β,γ depending on α, β, γ such that
Revised: d γ ⌉ with max { α, β } < γ < 1 , then there exists a constant C α,β,γ depending on α, β, γ such that

Original: [Equation]
Revised: [Equation]

Original: where log 2 (2 d α + 2) ≤ log 2 (4 d α ) < 2 + log 2 ( d ) is used in the first inequality, and the last inequality follows from the same argument as above, of multiplying with d d β and taking the logarithm.
Revised: where log 2 (2 d α + 2) ≤ log 2 (4 d α ) < 2 + log 2 ( d ) is used in the first inequality, and the last inequality follows from the same argument as above, of multiplying with d d β and taking the logarithm.

Original: Combining all the analysis above, we have
Revised: Combining all the analysis above, we have

Original: [Equation]
Revised: [Equation]

Original: where C ′ η,α,β,γ > 0 is a constant dependent on η, α, β, γ , and C 1 .
Revised: where C ′ η,α,β,γ > 0 is a constant dependent on η, α, β, γ , and C 1 .

Original: Recall from Proposition B.2.4, ˜ f is a network with depth L =
Revised: Recall from Proposition D.2.4, ˜ f is a network with depth L =

Original: ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4)
Revised: ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4)

Original: + 1 .
Revised: + 1 .

Original: By simply plugging-in m = ⌈ d γ ⌉ ,
Revised: By simply plugging-in m = ⌈ d γ ⌉ ,

Original: N = ⌈
Revised: N = ⌈

Original: d α ⌉ and M = 3 , we have L = O
Revised: d α ⌉ and M = 3 , we have L = O

Original: ( d γ log 2 ( d )) and N = O (cid:0) d max { α + γ, 1 } (cid:1) .
Revised: ( d γ log 2 ( d )) and N = O (cid:0) d max { α + γ, 1 } (cid:1) .

Original: (II) r
Revised: (II) r

Original: = O (1) and M =
Revised: = O (1) and M =

Original: O ( d d ) .
Revised: O ( d d ) .

Original: For the first term in (46), since N = ⌈ d α ⌉ , we know that N − r
Revised: For the first term in (45), since N = ⌈ d α ⌉ , we know that N − r

Original: =
Revised: =

Original: O ( d − αr ) .
Revised: O ( d − αr ) .

Original: For the second term in (46), since N = ⌈
Revised: For the second term in (45), since N = ⌈

Original: d α ⌉ with 0 < α < 1 , we know that it is bounded by 
Revised: d α ⌉ with 0 < α < 1 , we know that it is bounded by (cid:0)

Original: [Equation]
Revised: [Equation]

Original: Take M =
Revised: Take M =

Original: O (9 d d 94 d ) , multiply the bound (48) by d d β , take the logarithm, and find that for any 0 < α, β < 1 ,
Revised: O (9 d d 94 d ) , multiply the bound (47) by d d β , take the logarithm, and find that for any 0 < α, β < 1 ,

Original: [Equation]
Revised: [Equation]

Original:  as d → ∞ .
Revised: , as d → ∞ .

Original: Hence, there exists a constant C α,β > 0 depending only on C 1 , α , β such that
Revised: Hence, there exists a constant C α,β > 0 depending only on C 1 , α , β such that

Original: [Equation]
Revised: [Equation]

Original: for M =
Revised: for M =

Original: O (9 d d 94 d ) .
Revised: O (9 d d 94 d ) .

Original: For the third term in (46), take m = ⌈
Revised: For the third term in (45), take m = ⌈

Original: d γ ⌉ with max { α, β } < γ < 1 , then there exists a constant C α,β,γ depending on α, β, γ such that
Revised: d γ ⌉ with max { α, β } < γ < 1 , then there exists a constant C α,β,γ depending on α, β, γ such that

Original: [Equation]
Revised: [Equation]

Original: where log 2 (2 d α + 2) ≤ log 2 (4 d α ) < 2 + log 2 ( d ) is used in the first inequality, and the last inequality follows from the same argument as above, of multiplying with d d β and taking the logarithm.
Revised: where log 2 (2 d α + 2) ≤ log 2 (4 d α ) < 2 + log 2 ( d ) is used in the first inequality, and the last inequality follows from the same argument as above, of multiplying with d d β and taking the logarithm.

Original: Combining all the analysis above, we have
Revised: Combining all the analysis above, we have

Original: [Equation]
Revised: [Equation]

Original: where C ′ η,α,β,γ > 0 is a constant dependent on η, α, β, γ , and C 1 .
Revised: where C ′ η,α,β,γ > 0 is a constant dependent on η, α, β, γ , and C 1 .

Original: Recall from Proposition B.2.4, ˜ f is a network with depth L =
Revised: Recall from Proposition D.2.4, ˜ f is a network with depth L =

Original: ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4)
Revised: ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4)

Original: + 1 .
Revised: + 1 .

Original: By simply plugging-in m = ⌈ d γ ⌉ ,
Revised: By simply plugging-in m = ⌈ d γ ⌉ ,

Original: N = ⌈
Revised: N = ⌈

Original: d α ⌉ and M =
Revised: d α ⌉ and M =

Original: O (9 d d 94 d ) , we have L
Revised: O (9 d d 94 d ) , we have L

Original: = O
Revised: = O

Original: ( d γ log 2 ( d )) and N = O (cid:16) 9 d d 134 d (cid:17) .
Revised: ( d γ log 2 ( d )) and N = O (cid:16) 9 d d 134 d (cid:17) .

Original: D P ROOFS OF P ROPOSITION 4.2, T HEOREM 4.3 AND T HEOREM 4. D. P ROOF OF P ROPOSITION 4. 
Revised: F P ROOFS OF P ROPOSITION 4.2, T HEOREM 4.3 AND T HEOREM 4. F. P ROOF OF P ROPOSITION 4.

Original: Proposition 4.2 Set δ ∈ (0 , 1) .
Revised: Proposition 4.2 Set δ ∈ (0 , 1) .

Original: Then, with probability at least 1 − δ , we have
Revised: Then, with probability at least 1 − δ , we have

Original: E (cid:0) π B (cid:98) f n (cid:1) − E (cid:0) f

Original: ρ (cid:1) ≤ C B,δ,f · (cid:18) Pdim ( F ) · log( n )

Original: [Equation]
Revised: [Equation]

Original: where C B,δ,f is an absolute constant dependent on B, δ, f independent on n, r, d .
Revised: where C B,δ,f is an absolute constant dependent on B, δ, f independent on n, r, d .

Original: Proof.
Revised: Proof.

Original: Since (cid:98) f n is an empirical risk minimizer in (14) [Equation] we have E D ( (cid:98) f n ) ≤ E D ( f ) for any fixed f ∈ F and E D ( π B (cid:98) f n ) ≤ E D ( (cid:98) f n ) .
Revised: Since (cid:98) f n is an empirical risk minimizer in (12), we have E D ( (cid:98) f n ) ≤ E D ( f ) for any fixed f ∈ F and E D ( π B (cid:98) f n ) ≤ E D ( (cid:98) f n ) .

Original: Then, we have a following decomposition:
Revised: Then, we have a following decomposition:

Original: [Equation]
Revised: [Equation]

Original: Let F B :=
Revised: Let F B :=

Original: { π B f : ∀ f ∈ F} and define two quantities:
Revised: { π B f : ∀ f ∈ F} and define two quantities:

Original: [Equation]
Revised: [Equation]

Original: Step 1 : Control S 1 ( n, F B ) .
Revised: Step 1 : Control S 1 ( n, F B ) .

Original: The following concentration inequality is needed for controlling the term.
Revised: The following concentration inequality is needed for controlling the term.

Original: Lemma 4.1.1 [Theorem 11.4 of Gy¨orfi et al.
Revised: Lemma F.1.1 [Theorem 11.4 of Gy¨orfi et al.

Original: Assume | y | ≤ B almost surely and B ≥ 1 .
Revised: Assume | y | ≤ B almost surely and B ≥ 1 .

Original: Let α, β > 0 and 0 < ε ≤ 1 / 2 .
Revised: Let α, β > 0 and 0 < ε ≤ 1 / 2 .

Original: If F ′ is a set of functions f : R d → [ − B, B ] , then for any f ∈ F ′ , we have
Revised: If F ′ is a set of functions f : R d → [ − B, B ] , then for any f ∈ F ′ , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.1.2 [Theorem 6 of Haussler (2018)] Let B > 0 and F ′ be a set of functions f : X → [ − B, B ] .
Revised: Lemma F.1.2 [Theorem 6 of Haussler (2018)] Let B > 0 and F ′ be a set of functions f : X → [ − B, B ] .

Original: Then for any ε ∈ (0 , B ] , there holds
Revised: Then for any ε ∈ (0 , B ] , there holds

Original: [Equation]
Revised: [Equation]

Original: Recall a classical relation between ε -packing number and ε -covering number that asserts
Revised: Recall a classical relation between ε -packing number and ε -covering number that asserts

Original: [Equation]
Revised: [Equation]

Original:  for any ε > 0 .
Revised: 51) for any ε > 0 .

Original: Combining  (51),(52), the facts log x < x , ∀ x > [Equation] and Pdim ( F B ) ≤ Pdim ( F ) (See Maiorov & Ratsaby (1999), page 297), we have the upper-bound on N ( ε, F B , ∥ · ∥ L 1 ( D ) )) as follows:
Revised: Combining (50), (51), the facts log x < x , ∀ x > 0 , and Pdim ( F B ) ≤ Pdim ( F ) (See Maiorov & Ratsaby (1999), page 297), we have the upper-bound on N ( ε, F B , ∥ · ∥ L 1 ( D ) )) as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, taking ε = 12 , β = 1 n in Lemma 4.1.1, using the upper-bound on covering number in (53) yields the lower bound for the confidence level in Lemma (4.1.1) as follows:
Revised: Then, taking ε = 12 , β = 1 n in Lemma F.1.1, using the upper-bound on covering number in (52) yields the lower bound for the confidence level in Lemma (F.1.1) as follows:

Original: [Equation]
Revised: [Equation]

Original: where C B > 0 is some absolute constants dependent on B .
Revised: where C B > 0 is some absolute constants dependent on B .

Original: Choosing α in (54) such that
Revised: Choosing α in (53) such that

Original: [Equation]
Revised: [Equation]

Original: n ) n with a properly chosen C B,δ > 0 absolute constant dependent on B and δ yields the probability of following event is at least 1 − δ 2 :
Revised: with a properly chosen C B,δ > 0 absolute constant dependent on B and δ yields the probability of following event is at least 1 − δ 2 :

Original: [Equation]
Revised: [Equation]

Original: Step 2 : Control S 2 ( n, F ) .
Revised: Step 2 : Control S 2 ( n, F ) .

Original: Define a random variable η on Z = X
Revised: Define a random variable η on Z = X

Original: × Y to be
Revised: × Y to be

Original: [Equation]
Revised: [Equation]

Original: Since | η ( z ) | ≤ (3 B + ∥ f ∥ ∞ ) 2 , then | η ( z ) − E [ η ( z )] | ≤ 2(3 B + ∥ f ∥ ∞ ) 2 .
Revised: Since | η ( z ) | ≤ (3 B + ∥ f ∥ ∞ ) 2 , then | η ( z ) − E [ η ( z )] | ≤ 2(3 B + ∥ f ∥ ∞ ) 2 .

Original: It is also easy to see σ 2 ≤ E (cid:2)
Revised: It is also easy to see σ 2 ≤ E (cid:2)

Original: η 2 (cid:3) ≤ (cid:0) 3 B + ∥ f ∥ ∞ (cid:1) 2 ∥ f − f ρ ∥ 2 ∞ .
Revised: η 2 (cid:3) ≤ (cid:0) 3 B + ∥ f ∥ ∞ (cid:1) 2 ∥ f − f ρ ∥ 2 ∞ .

Original: Then, by the one-side Bernstein’s inequality (see
Revised: Then, by the one-side Bernstein’s inequality (see

Original: Lemma E.2), we have
Revised: Lemma H.2), we have

Original: [Equation]
Revised: [Equation]

Original: Taking 
Revised: Taking δ 2 = exp

Original: [Equation]
Revised: [Equation]

Original: and solving the quadratic equation with respect to ε yield the following inequalities with some absoluteconstant C ′′ 0 > 0 :
Revised: solving the quadratic equation with respect to ε yield the following inequalities with some absolute

Original: [Equation]
Revised: [Equation]

Original: where in the first inequality, the facts √ a + b ≤ √ a + √ b for a, b > 0 is used, and C B,f,δ is a constant dependent on C, B and f .
Revised: where in the first inequality, the facts √ a + b ≤ √ a + √ b for a, b > 0 is used, and C B,f,δ is a constant dependent on C, B and f .

Original: Then, with probability at least 1 − δ 2 , we have
Revised: Then, with probability at least 1 − δ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Step 3 : Combining Everything.
Revised: Step 3 : Combining Everything.

Original: Note E (cid:0) f (cid:1) − E (cid:0)
Revised: Note E (cid:0) f (cid:1) − E (cid:0)

Original: f ρ (cid:1) =
Revised: f ρ (cid:1) =

Original: ∥
Revised: ∥

Original: f − f ρ ∥ 2 ρ X ≤ ∥ f − f ρ ∥ 2 ∞ .
Revised: f − f [Equation] ∥ 2 ρ X ≤ ∥ f − f ρ ∥ 2 ∞ .

Original: Then, plugging the (55) and (56) in (50) yields the claim.
Revised: Then, plugging the (54) and (55) in (49) yields the claim.

Original: D. P ROOF OF T HEOREM 4. 
Revised: F. P ROOF OF T HEOREM 4.

Original: Theorem 4.3 Suppose f ρ ∈ W r ∞ ( S d − 1 ) with r > 0 .
Revised: Theorem 4.3 Suppose f ρ ∈ W r ∞ ( S d − 1 ) with r > 0 .

Original: A network (cid:98) f n from (8) with choices N = ⌈ n 2 3 d +4 r ⌉ , M = ⌈ n 3 d 3 d +4 r ⌉ , and m = ⌈ r 3 d +4 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:
Revised: A network (cid:98) f n from (6) with choices N = ⌈ n 2 3 d +4 r ⌉ , M = ⌈ n 3 d 3 d +4 r ⌉ , and m = ⌈ r 3 d +4 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:

Original: [Equation]
Revised: [Equation]

Original: where C B,η,δ,f depends on B , η , δ , f and independent on d, r and n 
Revised: where C B,η,δ,f depends on B , η , δ , f and independent on d, r and n .

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .
Revised: Let 0 < α < 1 , m, N, M ∈ N with 1 ≤ N ≤ d α + 1 .

Original: Then, for f ρ ∈ W r ∞ ( S d − 1 ) , recall from Theorem B.2.4 that there exists a network 
Revised: Then, for f ρ ∈ W r ∞ ( S d − 1 ) , recall from Theorem D.2.4 that there exists a network [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that the corresponding network’s approximation error is bounded as:
Revised: with depth L = ( m + 4) ⌈ log 2 (2 N ) ⌉ and number of parameters N ≤ M (2 d + 404 N · ( m + 3) + 2 N + 4) + 1 such that the corresponding network’s approximation error is bounded as:

Original: [Equation]
Revised: [Equation]

Original: where C ′′ η is a constant dependent on η , and independent on d, r, N, M and f .
Revised: where C ′′ η is a constant dependent on η , and independent on d, r, N, M and f .

Original: Since the network width is 22 NM , the total number of units across the L -hidden layers (i.e., U ) of ˜ f is bounded as
Revised: Since the network width is 22 NM , the total number of units across the L -hidden layers (i.e., U ) of ˜ f is bounded as

Original: [Equation]
Revised: U ≤ 22 NM · ( m + 4) ⌈ log 2 (2 N ) ⌉ .

Original: If Nm = o ( d ) , it is easy to see N ≤ O ( Md ) .
Revised: If Nm = o ( d ) , it is easy to see N ≤ O ( Md ) .

Original: Recall from the result of Lemma E.1, the pseudodimension of function class F in (58) is bounded as follows: for some universal constants C > 0 :
Revised: Recall from the result of Lemma H.1, the pseudodimension of function class F in (57) is bounded as follows: for some universal constants C > 0 :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Plug the (58), (59) in (48) from Proposition 4.2.

Original: [Equation]
Revised: [Equation]

Original: where C B,η,δ,f depends on B , η , δ , f and independent on d, r and n .
Revised: where C B,η,δ,f depends on B , η , δ , f and independent on d [Equation] r and n .

Original: Then, under the regime 1 ≤ N ≤ d α + 1 for some 0 < α < 1and n ≪ d , choices of m = ⌈
Revised: Then, under the regime 1 ≤ N ≤ d α + 1 for some 0 < α < 1 , choices of m = ⌈

Original: [Equation]
Revised: [Equation]

Original: andM = ⌈ n
Revised: and

Original: [Equation]
Revised: [Equation]

Original: d +4 r ⌉ make the fraction of the first term in (61) simple as follows: 
Revised: make the fraction of the first term in (60) simple as follows: [Equation] Then

Original: [Equation]
Revised: [Equation]

Original: Then, with the same choices of m
Revised: with the same choices of m

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: M as above
Revised: M as above

Original: [Equation]
Revised: [Equation]

Original: we obtain the bound on the excess risk as :
Revised: we obtain the bound on the excess risk as :

Original: [Equation]
Revised: [Equation]

Original: This conlcudes the proof
Revised: This conlcudes the proof

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF T HEOREM 4. 
Revised: F. P ROOF OF T HEOREM 4.

Original: Theorem 4.4 Suppose f ρ ∈ W r ∞ ([0 , 1] d ) with r > 0 .
Revised: Theorem 4.4 Suppose f ρ ∈ W r ∞ ([0 , 1] d ) with r > 0 .

Original: A network (cid:98) f n from (11) with choices N H = ⌈ n d 2 d + r ⌉
Revised: A network (cid:98) f n from (9) with choices N H = ⌈ n d 2 d + r ⌉

Original: [Equation]
Revised: [Equation]

Original: and m H =
Revised: and m H =

Original: ⌈ d + r d +2 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:
Revised: ⌈ d + r d +2 r log 2 ( n ) ⌉ yield the bound on the excess risk with probability at least 1 − δ as follows:

Original: [Equation]
Revised: [Equation]

Original: where C B,η,δ,K depends on B , η , δ , K and independent on d, r and n .
Revised: where C B,η,δ,K depends on B , η , δ , K and independent on d, r and n .

Original: Proof.
Revised: Proof.

Original: From Theorem 5 of Schmidt-Hieber (2020), for any function f ρ ∈ C rd ([0 , 1] d , K ) and any integers m ≥ 1 and N ≥ ( r + 1) d ∨ ( K + 1) e d , there exists a network
Revised: From Theorem 5 of Schmidt-Hieber (2020), for any function f ρ ∈ C rd ([0 , 1] d , K ) and any integers m ≥ 1 and N ≥ ( r + 1) d ∨ ( K + 1) e d , there exists a network

Original: [Equation]
Revised: [Equation]

Original: with depth L = 8 + ( m + 5) (cid:0) 1 + ⌈ log 2 ( d ∨ r ) ⌉ (cid:1) and the number of parameters N ≤ 141(1 + d + r ) 3+ d N ( m + 6) , such that[Equation]
Revised: with depth L = 8 + ( m + 5) (cid:0) 1 + ⌈ log 2 ( d ∨ r ) ⌉ (cid:1) and the number of parameters N ≤ 141(1 + d + r ) 3+ d N ( m + 6) , such that

Original: [Equation]
Revised: [Equation]

Original: Then, similarly with the proof in Theorem 4.3, by the result of Lemma E.1, the pseudo-dimension of F in (63) can be bounded as
Revised: Then, similarly with the proof in Theorem 4.3, by the result of Lemma H.1, the pseudo-dimension of F in (62) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: for some universal constants C > 0 .
Revised: for some universal constants C > 0 .

Original: Plug the (64) and (65) in (49) from Proposition 4.2.
Revised: Plug the (63) and (64) in (48) from Proposition 4.2.

Original: Then, we obtain the bound on the excess risk as follows:
Revised: Then, we obtain the bound on the excess risk as follows:

Original: [Equation]
Revised: [Equation]

Original: where C B,δ,K depends on B , δ , K and independent on d, r and n .
Revised: where C B,δ,K depends on B , δ , K and independent on d, r and n .

Original: Note that we use ( a
Revised: Note that we use ( a

Original: + b ) 2 ≤ 2 a 

Revised: + b ) 2 ≤ 2 a

Original: + 2 b 2 for all a, b ∈ R for getting the bound on (cid:13)(cid:13)(cid:13)
Revised: + 2 b 2 for all a, b ∈ R for getting the bound on (cid:13)(cid:13)(cid:13)

Original: ˜ f
Revised: ˜ f

Original: − f ρ (cid:13)(cid:13)(cid:13) 2 ∞ .
Revised: − f ρ (cid:13)(cid:13)(cid:13) 2 ∞ .

Original: We choose the N = ⌈ n d 2 r + d ⌉ and m = ⌈
Revised: We choose the N = ⌈ n d 2 r + d ⌉ and m = ⌈

Original: d + r 2 r + d log 2 ( n ) ⌉ .
Revised: d + r 2 r + d log 2 ( n ) ⌉ .

Original: Then, a fraction of the first term in (66) can be bounded as: 
Revised: Then, a fraction of the first term in (65) can be bounded as: log

Original: [Equation]
Revised: [Equation]

Original: where n ≪ d is used in the second inequality.

Original: Then, we obtain the bound on the excess risk as :
Revised: Then, we obtain the bound on the excess risk as :

Original: [Equation]
Revised: [Equation]

Revised: ≤ C B

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: · max

Revised: [Equation]

Original: This conlcudes the proof.
Revised: This conlcudes the proof.

Revised: G P ROOF OF APPROXIMATION RESULT FOR CNN: F ANG ET AL . (2020)

Original: E U SEFUL L EMMAS
Revised: Theorem C.1. Let 2 ≤ S ≤ d , 0 < α < 1 , and B, N, M ∈ N with 1 ≤ N ≤ d α + 1 . Let J ≥ ⌈

Revised: Md − 1 S − 1 ⌉ , D 1 = (2 B + 3)

Revised: ⌊ ( d + JS ) /d ⌋ , and D 2 =

Revised: ⌊ ( d + JS ) /d ⌋ .

Revised: Then, for any function f ∈

Revised: W r ∞ ( S d − 1 ) with r > 0 , there exists a network

Revised: ˜ f CNN ∈ H J, D 1 , D 2 ,S with the number of parameters

Revised: N ≤ J (3 S + 2) + M + 2 B + 4 such that

Revised: [Equation]

Revised: [Equation]

Revised: where C ′′ η,α is a constant dependent on η, α , and independent on d, r, N, M or f .

Revised: Proof.

Revised: By the inequality (5.9) in the paper Fang et al.

Revised: (2020), we have

Revised: [Equation]

Revised: [Equation]

Revised: where L t ( ξ N,r )( t ) is an univariate function for t ∈ [ − 1 , 1] defined in Lemma 7 in Fang et al.

Revised: Now, we work on controlling ∥ ξ N,r − L t ( ξ N,r ) ∥

Revised: C [ − 1 , 1] .

Revised: [Equation]

Revised: In the first inequality, we use Lemma 7 in Fang et al.

Revised: (2020), where ω ( ξ N,r , 1 /B ) is a modulus continuity of ξ N,r given by

Revised: [Equation]

Revised: In the second inequality, we use the definition of modulus continuity of ξ N,r .

Revised: In the third inequality, since ξ N,r is an algebraic polynomial of degree at most 2 N , by Markov’s inequality, we have ∥ ξ ′ N,r ∥ C [ − 1 , 1] ≤ (2 N ) 2 ∥ ξ N,r ∥ C [ − 1 , 1] .

Original: {+In the fourth inequality, we have the bound ∥ ξ N,r ∥ C [ − 1 , 1] ≤ (cid:80) 
+}
Revised: {+In the fourth inequality, we have the bound ∥ ξ N,r ∥ C [ − 1 , 1] ≤ (cid:80) 
+}

Revised: Nk

Revised: =1 k − r N ( k, d ) by Corollary 1.2.7 of Dai & Xu (2013).

Original: {+Employing Stirling’s formula d ! = √ 2 πd (cid:0) de (cid:1) d (cid:0) 
+}
Revised: {+Employing Stirling’s formula d ! = √ 2 πd (cid:0) de (cid:1) d (cid:0) 
+}

Revised: + O (1 /d ) (cid:1) in N ( k, d ) yields N ( k, d )

Revised: ≤ C α d k for 0 ≤ k ≤ 2 N and some constant C α depending on α but independent of d .

Revised: In the last inequality, we used the following inequality:

Revised: [Equation]

Revised: Now, we combine our result from Lemma D. 1 . 2 and Lemma D. 1 . 4 , and conclude the bound in (66).

Revised: H U SEFUL L EMMAS

Original: Lemma E.1 [Theorem 6 of Bartlett et al.
Revised: Lemma H.1 [Theorem 6 of Bartlett et al.

Original: ] Consider the function class F computed by a feed-forward neural network architecture with N parameters and U computation units arranged across L layers.
Revised: ] Consider the function class F computed by a feed-forward neural network architecture with N parameters and U computation units arranged across L layers.

Original: Suppose that all non-ouput units have piecewise-polynomial activation functions with p + 1 pieces and degree no more than d , and the output unit has the identity function as its activation function.
Revised: Suppose that all non-ouput units have piecewise-polynomial activation functions with p + 1 pieces and degree no more than d , and the output unit has the identity function as its activation function.

Original: Then the VC-dimension and pseudo-dimension of class F is upper bounded by
Revised: Then the VC-dimension and pseudo-dimension of class F is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: ) , Pdim

Original: [Equation]
Revised: [Equation]

Original:  with some universal constants C > 0 .
Revised: , with some universal constants C > 0 .

Original: Lemma E.2 [Theorem 2.8.4 of Vershynin (2018)] Let η be a random variable on a probability space Z with mean E ( η ) = µ , variance σ 2 ( η ) =
Revised: Lemma H.2 [Theorem 2.8.4 of Vershynin (2018)] Let η be a random variable on a probability space Z with mean E ( η ) = µ , variance σ 2 ( η ) =

Original: σ 2 , and satisfying | η ( z ) − E ( η ) | ≤ B η for almost z ∈ Z .
Revised: σ 2 , and satisfying | η ( z ) − E ( η ) | ≤ B η for almost z ∈ Z .

Original: Then, for any ε > 0 , 
Revised: Then, for any ε > 0 , P

Original: [Equation]
Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: − nε

Revised: [Equation]