Original: L EARNABILITY OF CONVOLUTIONAL NEURAL NETWORKS FOR INFINITE DIMENSIONAL INPUT VIA MIXED AND ANISOTROPIC SMOOTHNESSAnonymous authors
Revised: L EARNABILITY OF CONVOLUTIONAL NEURAL NETWORKS FOR INFINITE DIMENSIONAL INPUT VIA MIXED AND ANISOTROPIC SMOOTHNESS

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Among a wide range of success of deep learning, convolutional neural networks have been extensively utilized in several tasks such as speech recognition, image processing, and natural language processing, which require inputs with large dimensions.
Revised: Among a wide range of success of deep learning, convolutional neural networks have been extensively utilized in several tasks such as speech recognition, image processing, and natural language processing, which require inputs with large dimensions.

Original: Several studies have investigated function estimation capability of deep learning, but most of them have assumed that the dimensionality of the input is much smaller than the sample size.
Revised: Several studies have investigated function estimation capability of deep learning, but most of them have assumed that the dimensionality of the input is much smaller than the sample size.

Original: However, for typical data in applications such as those handled by the convolutional neural networks described above, the dimensionality of inputs is relatively high or even inﬁnite.
Revised: However, for typical data in applications such as those handled by the convolutional neural networks described above, the dimensionality of inputs is relatively high or even inﬁnite.

Original: In this paper, we investigate the approximation and estimation errors of the (dilated) convolutional neural networks when the input is inﬁnite dimensional.
Revised: In this paper, we investigate the approximation and estimation errors of the (dilated) convolutional neural networks when the input is inﬁnite dimensional.

Original: Although the approximation and estimation errors of neural networks are affected by the curse of dimensionality in the existing analyses for typical function spaces such as the H ¨ older and Besov spaces, we show that, by considering anisotropic smoothness, they can alleviate exponential dependency on the dimensionality but they only depend on the smoothness of the target functions.
Revised: Although the approximation and estimation errors of neural networks are affected by the curse of dimensionality in the existing analyses for typical function spaces such as the H ¨ older and Besov spaces, we show that, by considering anisotropic smoothness, they can alleviate exponential dependency on the dimensionality but they only depend on the smoothness of the target functions.

Original: Our theoretical analysis supports the great practical success of convolutional networks.
Revised: Our theoretical analysis supports the great practical success of convolutional networks.

Original: Furthermore, we show that the dilated convolution is advantageous when the smoothness of the target function has a sparse structure.
Revised: Furthermore, we show that the dilated convolution is advantageous when the smoothness of the target function has a sparse structure.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has shown high performance in several tasks such as image recognition, speech recognition, and natural language processing.
Revised: Deep learning has shown high performance in several tasks such as image recognition, speech recognition, and natural language processing.

Original: In particular, convolutional neural networks (CNNs) and dilated CNNs have been quite effective in tasks involving high-dimensional data (van den Oord et al., 2016;
Revised: In particular, convolutional neural networks (CNNs) and dilated CNNs have been quite effective in tasks involving high-dimensional data (van den Oord et al., 2016;

Original: He et al., 2016; Simonyan & Zisserman, 2015; Yoon, 2014).
Revised: He et al., 2016; Simonyan & Zisserman, 2015; Yoon, 2014).

Original: However, many aspects of its theoretical nature are still unclear while related theoretical studies have attracted much attention.
Revised: However, many aspects of its theoretical nature are still unclear while related theoretical studies have attracted much attention.

Original: Aside from the analysis of CNNs, one of the most fundamental issues in deep learning theories is its function approximation and estimation capabilities.
Revised: Aside from the analysis of CNNs, one of the most fundamental issues in deep learning theories is its function approximation and estimation capabilities.

Original: For example, it is well known that any continuous function with compact support can be approximated with arbitrary accuracy by a twolayer fully connected neural network (Cybenko, 1989; Hornik, 1991).
Revised: For example, it is well known that any continuous function with compact support can be approximated with arbitrary accuracy by a twolayer fully connected neural network (Cybenko, 1989; Hornik, 1991).

Original: Moreover, the representation ability of deep learning to approximate a function in some function classes such as H¨older classes has also been extensively analyzed (Mhaskar & Micchelli, 1992; Mhaskar, 1993; Chui et al., 1994;
Revised: Moreover, the representation ability of deep learning to approximate a function in some function classes such as H¨older classes has also been extensively analyzed (Mhaskar & Micchelli, 1992; Mhaskar, 1993; Chui et al., 1994;

Original: Mhaskar, 1996; Pinkus, 1999; Yarotsky, 2017; Petersen & Voigtlaender, 2017).
Revised: Mhaskar, 1996; Pinkus, 1999; Yarotsky, 2017; Petersen & Voigtlaender, 2017).

Original: In addition to the approximation ability, the estimation ability of deep learning for estimating a function by a ﬁnite sample has also been extensively studied.
Revised: In addition to the approximation ability, the estimation ability of deep learning for estimating a function by a ﬁnite sample has also been extensively studied.

Original: For example, Schmidt-Hieber (2020) derived the estimation error bound of deep learning with ReLU activation (Nair & Hinton, 2010; Glorot et al., 2011) to estimate functions in the H¨older space and showed the rate of convergence achieves the (near) minimax optimal rate.
Revised: For example, Schmidt-Hieber (2020) derived the estimation error bound of deep learning with ReLU activation (Nair & Hinton, 2010; Glorot et al., 2011) to estimate functions in the H¨older space and showed the rate of convergence achieves the (near) minimax optimal rate.

Original: Suzuki (2019) derived approximation and estimation error rates of deep learning with ReLU activation for the Besov spaces, which were also shown to be (near) minimax optimal.
Revised: Suzuki (2019) derived approximation and estimation error rates of deep learning with ReLU activation for the Besov spaces, which were also shown to be (near) minimax optimal.

Original: Although the derived rates of convergence are near optimal, these studies assumed that the dimensionality of inputs is ﬁxed and much smaller than the sample size.
Revised: Although the derived rates of convergence are near optimal, these studies assumed that the dimensionality of inputs is ﬁxed and much smaller than the sample size.

Original: Indeed, the derived rates suffer from the curse of dimensionality.
Revised: Indeed, the derived rates suffer from the curse of dimensionality.

Original: However, in practice, we often encounter settings where the input dimensionality is larger than the sample size or even inﬁnite.
Revised: However, in practice, we often encounter settings where the input dimensionality is larger than the sample size or even inﬁnite.

Original: For example, in image recognition and natural language processing, the dimensionality of inputs (images or texts) is very large, and they could be seen as almost inﬁnite dimensional.
Revised: For example, in image recognition and natural language processing, the dimensionality of inputs (images or texts) is very large, and they could be seen as almost inﬁnite dimensional.

Original: To address this issue, some researches considered a setting where the dimensionality of the support of the data distribution is low dimensional.
Revised: To address this issue, some researches considered a setting where the dimensionality of the support of the data distribution is low dimensional.

Original: Chen et al.
Revised: Chen et al.

Original: (2019b;a) considered a setting where data can be embedded in a low dimensional sub-manifold and derived the approximation error of functions that depends merely on the dimensionality of the sub-manifold instead of that of the entire space.
Revised: (2019b;a) considered a setting where data can be embedded in a low dimensional sub-manifold and derived the approximation error of functions that depends merely on the dimensionality of the sub-manifold instead of that of the entire space.

Original: Nakada & Imaizumi (2020) also considered a similar setting, and showed that the estimation error is characterized by the Minkowski dimension of the support of the data distribution.
Revised: Nakada & Imaizumi (2020) also considered a similar setting, and showed that the estimation error is characterized by the Minkowski dimension of the support of the data distribution.

Original: Suzuki (2019) showed that, even if the data cannot be embedded in a low dimensional manifold, anisotropic smoothness of the target function can mitigate the curse of dimensionality.
Revised: Suzuki (2019) showed that, even if the data cannot be embedded in a low dimensional manifold, anisotropic smoothness of the target function can mitigate the curse of dimensionality.

Original: Although these studies revealed that deep learning can avoid curse of dimensionality by utilizing some low dimensional structures of data and the target functions, it still remains unclear how deep learning performs for very high dimensional settings including an inﬁnite dimensional setting.
Revised: Although these studies revealed that deep learning can avoid curse of dimensionality by utilizing some low dimensional structures of data and the target functions, it still remains unclear how deep learning performs for very high dimensional settings including an inﬁnite dimensional setting.

Original: See Table 1 for a summarized comparison to existing studies.
Revised: See Table 1 for a summarized comparison to existing studies.

Original: In terms of inﬁnite dimensional inputs, there have been already several studies on approximation and estimation errors for non-deep-learning methods.
Revised: In terms of inﬁnite dimensional inputs, there have been already several studies on approximation and estimation errors for non-deep-learning methods.

Original: For example, so called hyperbolic cross approximation has been considered to approximate a function in a tensor product space with support on [0 , 1] ∞ (D˜ung & Griebel, 2016) and a polynomial order approximation is possible for functions with mixed smoothness, that is, speciﬁc summability properties of the smoothness indices are fulﬁlled.
Revised: For example, so called hyperbolic cross approximation has been considered to approximate a function in a tensor product space with support on [0 , 1] ∞ (D ˜ ung & Griebel, 2016) and a polynomial order approximation is possible for functions with mixed smoothness, that is, speciﬁc summability properties of the smoothness indices are fulﬁlled.

Original: Ingster & Stepanova (2011) analyzed a Gaussian white noise model with an inﬁnite dimensional input and showed that the estimation accuracy for signals on inﬁnite dimensional anisotropic Sobolev spaces depends on the reciprocal sum of the smoothness per axis (see also Ingster & Stepanova (2006); Ingster & Suslina (2007); Ingster & Stepanova (2009)).
Revised: Ingster & Stepanova (2011) analyzed a Gaussian white noise model with an inﬁnite dimensional input and showed that the estimation accuracy for signals on inﬁnite dimensional anisotropic Sobolev spaces depends on the reciprocal sum of the smoothness per axis (see also Ingster & Stepanova (2006); Ingster & Suslina (2007); Ingster & Stepanova (2009)).

Original: Oliva et al.
Revised: Oliva et al.

Original: (2013; 2015) proposed methods to estimate a map where the input and output are functions or distributions, and derive the rate of convergence.
Revised: (2013; 2015) proposed methods to estimate a map where the input and output are functions or distributions, and derive the rate of convergence.

Original: Ferraty et al.
Revised: Ferraty et al.

Original: (2007) analyzed the Nadaraya-Watson estimator when the inputs are functions, derived the convergence rate of the estimator, and gave the asymptotic conﬁdence band in the context of functional data analysis (see Ling & Vieu (2018) as a comprehensive survey of the nonlinear functional data analysis literature).
Revised: (2007) analyzed the Nadaraya-Watson estimator when the inputs are functions, derived the convergence rate of the estimator, and gave the asymptotic conﬁdence band in the context of functional data analysis (see Ling & Vieu (2018) as a comprehensive survey of the nonlinear functional data analysis literature).

Original: However, these researches are not for the deep learning and the beneﬁt of deep learning for such situation has not been well characterized in the literature.
Revised: However, these researches are not for the deep learning and the beneﬁt of deep learning for such situation has not been well characterized in the literature.

Original: In this study, we analyze the approximation and estimation accuracy in a setting where the input is inﬁnite dimensional, and derive their convergence rates.
Revised: In this study, we analyze the approximation and estimation accuracy in a setting where the input is inﬁnite dimensional, and derive their convergence rates.

Original: We assume that the true function has mixed and anisotropic smoothness, that is, the function has different smoothness toward different coordinate similarly to D ˜ ung & Griebel (2016); Ingster & Stepanova (2011).
Revised: We assume that the true function has mixed and anisotropic smoothness, that is, the function has different smoothness toward different coordinate similarly to D ˜ ung & Griebel (2016); Ingster & Stepanova (2011).

Original: The intuition behind this setting is as follows:
Revised: The intuition behind this setting is as follows:

Original: Considering a function which takes an image as an input, an image can be decomposed into different frequency components and usually a function of images has less sensitivity on the high frequency components and more dependent on the low frequency components, which can be formulated as non-uniform smoothness toward each coordinate direction.
Revised: Considering a function which takes an image as an input, an image can be decomposed into different frequency components and usually a function of images has less sensitivity on the high frequency components and more dependent on the low frequency components, which can be formulated as non-uniform smoothness toward each coordinate direction.

Original: By considering such a setting, we can show that the rate of convergence can avoid the curse of dimensionality and be of polynomial order.
Revised: By considering such a setting, we can show that the rate of convergence can avoid the curse of dimensionality and be of polynomial order.

Original: Our contribution can be summarized as follows:
Revised: Our contribution can be summarized as follows:

Original: We consider a learning problem in which the target function to be approximated or estimated can take an inﬁnite dimensional input and has mixed or anisotropic smoothness.
Revised: We consider a learning problem in which the target function to be approximated or estimated can take an inﬁnite dimensional input and has mixed or anisotropic smoothness.

Original: Then, we show that deep learning by fully connected neural networks can achieve approximation and estimation errors that depend only on smoothness of the target function andare independent of the dimensionality.
Revised: We show that deep learning by fully connected neural networks can achieve approximation and estimation errors dependent only on smoothness of the target function and independent of the dimension.

Original: We also consider a setting where the smoothness of the target function has a sparse structure, and then we show that dilated CNNs can ﬁnd appropriate variables and improve the rate of convergence.
Revised: We also consider a setting where the smoothness of the target function has a sparse structure, and then we show that dilated CNNs can ﬁnd appropriate variables and improve the rate of convergence.

Original: This indicates that CNNs can capture a long range dependence among the input.
Revised: This indicates that CNNs can capture a long range dependence among the input.

Original: These results show that even when the dimension d of the data is very large compared to the number of observations n , or even when the input is inﬁnite dimensional, it is possible to derive a polynomial order estimation error bound that depends only on the smoothness of the function class.
Revised: These results show that even when the dimension d of the data is very large compared to the number of observations n , or even when the input is inﬁnite dimensional, it is possible to derive a polynomial order estimation error bound that depends only on the smoothness of the function class.

Original: This analysis partially explains the great success of CNNs in various applications with high dimensional inputs.
Revised: This analysis partially explains the great success of CNNs in various applications with high dimensional inputs.

Original:  P ROBLEM S ETTING AND NOTATIONS
Revised:  P ROBLEM S ETTING AND NOTATIONS

Original: In this section, we prepare the notations and introduce the problem setting.
Revised: In this section, we prepare the notations and introduce the problem setting.

Original: Throughout this paper, we use the following notations.
Revised: Throughout this paper, we use the following notations.

Original: Let R > 0 :
Revised: Let R > 0 :

Original: = { s ∈ R : s > 0 } , and for a set D , let D ∞ :
Revised: = { s ∈ R : s > 0 } , and for a set D , let D ∞ :

Original: = { ( s 1 , . . .
Revised: = { ( s 1 , . . .

Original: , s i , . . . )
Revised: , s i , . . . )

Original: : s i ∈ D } (for example, R ∞ :
Revised: : s i ∈ D } (for example, R ∞ :

Original: = { ( s i ) ∞ i =1 : s i ∈ R
Revised: = { ( s i ) ∞ i =1 : s i ∈ R

Original: ( ∀ i = 1 , 2 , . . . ) } ).
Revised: ( ∀ i = 1 , 2 , . . . ) } ).

Original: For
Revised: For

Original: [Table]
Revised: [Table]

Original: s ∈ R ∞ , let supp( s ) =
Revised: s ∈ R ∞ , let supp( s ) =

Original: { i ∈ N : s i (cid:54) = 0 } .
Revised: { i ∈ N : s i (cid:54) = 0 } .

Original: Let N ∞ 0 :
Revised: Let N ∞ 0 :

Original: = { l ∈ ( N ∪ { 0 } ) ∞ : supp( l ) < ∞} and deﬁne Z ∞ 0 and R ∞ 0 in the same way.
Revised: = { l ∈ ( N ∪ { 0 } ) ∞ : supp( l ) < ∞} and deﬁne Z ∞ 0 and R ∞ 0 in the same way.

Original: Furthermore, for s ∈ R ∞ 0 , let 2 s := 2 (cid:80) ∞ i =1 s i .
Revised: Furthermore, for s ∈ R ∞ 0 , let 2 s := 2 (cid:80) ∞ i =1 s i .

Original: For L ∈ N , let [ L ] = { 1 , . . .
Revised: For L ∈ N , let [ L ] = { 1 , . . .

Original: , L } .
Revised: , L } .

Original: For a ∈ R , let (cid:98) a (cid:99) be the largest integer less than or equal to a .
Revised: For a ∈ R , let (cid:98) a (cid:99) be the largest integer less than or equal to a .

Original: 2. R EGRESSION PROBLEM WITH INFINITE DIMENSIONAL PREDICTOR
Revised: 2. R EGRESSION PROBLEM WITH INFINITE DIMENSIONAL PREDICTOR

Original: In this paper, we consider a regression problem where the predictor (input) is inﬁnite dimensional.
Revised: In this paper, we consider a regression problem where the predictor (input) is inﬁnite dimensional.

Original: Let λ be the uniform probability measure on ([0 , 1] , B ([0 , 1]))
Revised: Let λ be the uniform probability measure on ([0 , 1] , B ([0 , 1]))

Original: where B ([0 , 1]) is the Borel σ -ﬁeld on [0 , 1] , and let λ ∞ be the product measure of λ on ([0 , 1] ∞ , B ([0 , 1] ∞ )) where B ([0 , 1] ∞ ) is the product σ -algebra generated by the cylindric sets ∩ j ≤ d { x ∈ [0 , 1] ∞ : x j ∈ B j } for d = 1 , 2 , . . .
Revised: where B ([0 , 1]) is the Borel σ -ﬁeld on [0 , 1] , and let λ ∞ be the product measure of λ on ([0 , 1] ∞ , B ([0 , 1] ∞ )) where B ([0 , 1] ∞ ) is the product σ -algebra generated by the cylindric sets ∩ j ≤ d { x ∈ [0 , 1] ∞ : x j ∈ B j } for d = 1 , 2 , . . .

Original: and B j ∈ B ([0 , 1]) .
Revised: and B j ∈ B ([0 , 1]) .

Original: Let P X be a probability measure deﬁned on the measurable space ([0 , 1] ∞ , B ([0 , 1] ∞ )) that is absolutely continuous to λ ∞ and its Radon-Nikodym derivative satisﬁes (cid:107) d P X d λ ∞ (cid:107) L ∞ ([0 , 1] ∞ ) <
Revised: Let P X be a probability measure deﬁned on the measurable space ([0 , 1] ∞ , B ([0 , 1] ∞ )) that is absolutely continuous to λ ∞ and its Radon-Nikodym derivative satisﬁes (cid:107) d P X d λ ∞ (cid:107) L ∞ ([0 , 1] ∞ ) <

Original: ∞ 1 .
Revised: ∞ 1 .

Original: Then, suppose that there exists a true function f o : [0 , 1] ∞ → R , and consider the following nonparametric regression problem with an inﬁnite dimensional input:
Revised: Then, suppose that there exists a true function f o : [0 , 1] ∞ → R , and consider the following nonparametric regression problem with an inﬁnite dimensional input:

Original: [Equation]
Revised: [Equation]

Original: where X is a random variable taking its value on [0 , 1] ∞ and obeys the distribution P X introduced above, and ξ is a observation noise generated from N (0 , σ 2 ) (a normal distribution with mean 0 and variance σ 2 > 0 ).
Revised: where X is a random variable taking its value on [0 , 1] ∞ and obeys the distribution P X introduced above, and ξ is a observation noise generated from N (0 , σ 2 ) (a normal distribution with mean 0 and variance σ 2 > 0 ).

Original: Let P be the joint distribution of X and Y obeying the regression model.
Revised: Let P be the joint distribution of X and Y obeying the regression model.

Original: What we investigate in the following is (i) how efﬁciently we can approximate the true function f o by a neural network, and (ii) how accurately deep learning can estimate the true function f o from n observations D n =
Revised: What we investigate in the following is (i) how efﬁciently we can approximate the true function f o by a neural network, and (ii) how accurately deep learning can estimate the true function f o from n observations D n =

Original: ( X i , y i ) ni
Revised: ( X i , y i ) ni

Original: =1 where ( X i , y i ) ni
Revised: =1 where ( X i , y i ) ni

Original: =1 are i.i.d.
Revised: =1 are i.i.d.

Original: observations from the model.
Revised: observations from the model.

Original: As a performance measure, we employ the mean squared error (cid:107) f − f o (cid:107) 2 P X := E P
Revised: As a performance measure, we employ the mean squared error (cid:107) f − f o (cid:107) 2 P X := E P

Original: [( f ( X ) − f o ( X )) 2 ] , which can be seen as the excess risk of the predictive error E ( X,Y ) ∼ P
Revised: [( f ( X ) − f o ( X )) 2 ] , which can be seen as the excess risk of the predictive error E ( X,Y ) ∼ P

Original: [( f ( X ) − Y ) 2 ] associated with the squared loss (i.e., (cid:107) f − f o (cid:107) 2 P X
Revised: [( f ( X ) − Y ) 2 ] associated with the squared loss (i.e., (cid:107) f − f o (cid:107) 2 P X

Original: =
Revised: =

Original: E
Revised: E

Original: ( X,Y ) ∼ P [( f ( X ) − Y ) 2 ] − E ( X,Y ) ∼ P [( f o ( X ) − Y ) 2 ] =
Revised: ( X,Y ) ∼ P [( f ( X ) − Y ) 2 ] − E ( X,Y ) ∼ P [( f o ( X ) − Y ) 2 ] =

Original: E ( X,Y ) ∼ P [( f ( X ) − Y ) 2 ] − inf f :measurable E ( X,Y ) ∼ P [( f ( X ) − Y ) 2 ] ).
Revised: E ( X,Y ) ∼ P [( f ( X ) − Y ) 2 ] − inf f :measurable E ( X,Y ) ∼ P [( f ( X ) − Y ) 2 ] ).

Original: 2. M IXED AND ANISOTROPIC SMOOTHNESS ON INFINITE DIMENSIONAL VARIABLES
Revised: 2. M IXED AND ANISOTROPIC SMOOTHNESS ON INFINITE DIMENSIONAL VARIABLES

Original: Here, we introduce a function class in which we suppose the true function f o is included.
Revised: Here, we introduce a function class in which we suppose the true function f o is included.

Original: For a given l ∈ Z ∞ 0 , deﬁne ψ l i : [0 , 1] → R as ψ l i ( x ) =
Revised: For a given l ∈ Z ∞ 0 , deﬁne ψ l i : [0 , 1] → R as ψ l i ( x ) =

Original:   √ 2 cos(2 π | l i | x ) ( l i < 0) , √ 2 sin(2 π | l i | x ) ( l i > 0) , ( l i = 0) , for x ∈ [0 , 1] , and deﬁne ψ l ( X ) :
Revised:   √ 2 cos(2 π | l i | x ) ( l i < 0) , √ 2 sin(2 π | l i | x ) ( l i > 0) , ( l i = 0) , for x ∈ [0 , 1] , and deﬁne ψ l ( X ) :

Original: = (cid:81) ∞
Revised: = (cid:81) ∞

Original: i =1 ψ l i ( x i ) for X
Revised: i =1 ψ l i ( x i ) for X

Original: = ( x i ) ∞
Revised: = ( x i ) ∞

Original: i =1 ∈ [0 , 1] ∞ .
Revised: i =1 ∈ [0 , 1] ∞ .

Original: Let L 2 ([0 , 1] ∞ ) :
Revised: Let L 2 ([0 , 1] ∞ ) :

Original: = { f : [0 , 1] ∞ → R : (cid:82) [0 , 1] ∞ f 2 ( x )d λ ∞ ( x ) < ∞} equipped with the inner product (cid:104) f, g (cid:105) :
Revised: = { f : [0 , 1] ∞ → R : (cid:82) [0 , 1] ∞ f 2 ( x )d λ ∞ ( x ) < ∞} equipped with the inner product (cid:104) f, g (cid:105) :

Original: = (cid:82) [0 , 1] ∞ f
Revised: = (cid:82) [0 , 1] ∞ f

Original: ( x ) g ( x )d λ ∞ ( x ) for f, g ∈ L 2 ([0 , 1] ∞ ) .
Revised: ( x ) g ( x )d λ ∞ ( x ) for f, g ∈ L 2 ([0 , 1] ∞ ) .

Original: Then, ( ψ l ) l ∈ Z ∞ 0 forms a complete orthonormal system of L 2 ([0 , 1] ∞ ) , that is, f ∈ L 2 ([0 , 1] ∞ ) can be expanded as f ( X )
Revised: Then, ( ψ l ) l ∈ Z ∞ 0 forms a complete orthonormal system of L 2 ([0 , 1] ∞ ) , that is, f ∈ L 2 ([0 , 1] ∞ ) can be expanded as f ( X )

Original: = (cid:80) l ∈ Z ∞ 

Revised: = (cid:80) l ∈ Z ∞

Original: (cid:104) f, ψ l (cid:105) ψ l ( X ) (see Ingster & Stepanova (2011) for example).
Revised: (cid:104) f, ψ l (cid:105) ψ l ( X ) (see Ingster & Stepanova (2011) for example).

Original: For s ∈ N ∞ 0 , let δ s ( f ) :
Revised: For s ∈ N ∞ 0 , let δ s ( f ) :

Original: R ∞
Revised: R ∞

Original: → R be
Revised: → R be

Original: [Equation]
Revised: [Equation]

Original: which can be seen as the frequency component of f of frequency | l i | (cid:39) 2 s i toward each coordinate.
Revised: which can be seen as the frequency component of f of frequency | l i | (cid:39) 2 s i toward each coordinate.

Original: We also deﬁne (cid:107) f (cid:107) p :=
Revised: We also deﬁne (cid:107) f (cid:107) p :=

Original: (cid:16)(cid:82)
Revised: (cid:16)(cid:82)

Original: [0 , 1] ∞ | f | p d λ ∞ (cid:17) 1 /p for p ≥ 1 .
Revised: [0 , 1] ∞ | f | p d λ ∞ (cid:17) 1 /p for p ≥ 1 .

Original: Then, we deﬁne a function space with a general smoothness conﬁguration as follows.
Revised: Then, we deﬁne a function space with a general smoothness conﬁguration as follows.

Original: Deﬁnition 1 (Function class with γ -smoothness) .
Revised: Deﬁnition 1 (Function class with γ -smoothness) .

Original: For a given γ : N ∞ 

Revised: For a given γ : N ∞

Original: → R > 0 which is monotonically non-decreasing with respect to each coordinate.
Revised: → R > 0 which is monotonically non-decreasing with respect to each coordinate.

Original: For p ≥ 1 , θ ≥ 1 , we deﬁne the γ -smooth space as
Revised: For p ≥ 1 , θ ≥ 1 , we deﬁne the γ -smooth space as

Original: [Equation]
Revised: [Equation]

Original: equipped with the norm (cid:107) f
Revised: equipped with the norm (cid:107) f

Original: [Equation]
Revised: [Equation]

Original: In the following, F γp,θ ([0 , 1] ∞ ) is abbreviated to F γp,θ , and its unit ball is denoted by U ( F γp,θ ) .
Revised: In the following, F γp,θ ([0 , 1] ∞ ) is abbreviated to F γp,θ , and its unit ball is denoted by U ( F γp,θ ) .

Original: Remind that δ s ( f ) represents the frequency component associated with the frequency (2 s i )
Revised: Remind that δ s ( f ) represents the frequency component associated with the frequency (2 s i )

Original: ∞
Revised: ∞

Original: i =1 , and then the norm of the γ -smooth space imposes weight 2 θγ ( s ) on each frequency component associated with s .
Revised: i =1 , and then the norm of the γ -smooth space imposes weight 2 θγ ( s ) on each frequency component associated with s .

Original: In that sense, γ ( s ) controls the weight of each frequency component and accordingly a function in the space can have different smoothness toward different coordinates.
Revised: In that sense, γ ( s ) controls the weight of each frequency component and accordingly a function in the space can have different smoothness toward different coordinates.

Original: As a special case of γ ( s ) , we investigate the following ones in this paper.
Revised: As a special case of γ ( s ) , we investigate the following ones in this paper.

Original: We can see that a ﬁnite dimensional analysis can be easily reduced to a special case of the inﬁnite dimensional analysis (see Appendix A).
Revised: We can see that a ﬁnite dimensional analysis can be easily reduced to a special case of the inﬁnite dimensional analysis (see Appendix A).

Original: In that sense, our analysis generalizes existing ﬁnite dimensional analyses.
Revised: In that sense, our analysis generalizes existing ﬁnite dimensional analyses.

Original: Deﬁnition 2 (Mixed smoothness and anisotropic smoothness) .
Revised: Deﬁnition 2 (Mixed smoothness and anisotropic smoothness) .

Original: Given a monotonically nondecreasing sequence a = ( a i ) ∞ i =1 ∈ R ∞ > 0 , we deﬁne the mixed smoothness as (mixed smoothness)γ [Equation] = [Equation]
Revised: Given a monotonically nondecreasing sequence a = ( a i ) ∞ i =1 ∈ R ∞ > 0 , we deﬁne the mixed smoothness as (mixed smoothness)

Original: [Equation]
Revised: [Equation]

Original: , where (cid:104) a, s (cid:105) :
Revised: , where (cid:104) a, s (cid:105) :

Original: = (cid:80) ∞ i =
Revised: = (cid:80) ∞ i =

Original: 1 a
Revised: 1 a

Original: i s i 2 , and deﬁne the anisotropic smoothness as (anisotropic smoothness)
Revised: i s i 2 , and deﬁne the anisotropic smoothness as (anisotropic smoothness)

Original: [Equation]
Revised: [Equation]

Original: Each component a i of a = ( a i ) ∞ i =1 represents the smoothness of the function with respect to the variable x i .
Revised: Each component a i of a = ( a i ) ∞ i =1 represents the smoothness of the function with respect to the variable x i .

Original: Since we assumed ( a i ) ∞
Revised: Since we assumed ( a i ) ∞

Original: i =1 is monotonically non-decreasing, a function in the space has higher smoothness toward the coordinate x
Revised: i =1 is monotonically non-decreasing, a function in the space has higher smoothness toward the coordinate x

Original: i with higher index i .
Revised: i with higher index i .

Original: In other words, the function f in the space is less sensitive to the variable x
Revised: In other words, the function f in the space is less sensitive to the variable x

Original: i with a larger index i .
Revised: i with a larger index i .

Original: For example, in computer vision tasks, we may suppose x i with a large index i corresponds to a higher frequency component of the input image, and then the function is less sensitive to such high frequency components and more sensitive to a low-frequency “global” information.
Revised: For example, in computer vision tasks, we may suppose x i with a large index i corresponds to a higher frequency component of the input image, and then the function is less sensitive to such high frequency components and more sensitive to a low-frequency “global” information.

Original: This can be seen as an inﬁnite dimensional variant of the mixed smooth Besov space (Schmeisser, 1987; Sickel & Ullrich, 2009) and the anisotropic Besov space (Nikol’skii, 1975; Vybiral, 2006; Triebel, 2011) (see Appendix C for detailed discussions).
Revised: This can be seen as an inﬁnite dimensional variant of the mixed smooth Besov space (Schmeisser, 1987; Sickel & Ullrich, 2009) and the anisotropic Besov space (Nikol’skii, 1975; Vybiral, 2006; Triebel, 2011) (see Appendix C for detailed discussions).

Original: In our theoretical analysis, we will assume that the true target function f o is included in the γ -smooth function space.
Revised: In our theoretical analysis, we will assume that the true target function f o is included in the γ -smooth function space.

Original: Assumption 3.
Revised: Assumption 3.

Original: The target function satisﬁes f o ∈ U ( F γp,θ ) with p ≥ 1 and θ ≥ 1 , and (cid:107) f o (cid:107) ∞ ≤
Revised: The target function satisﬁes f o ∈ U ( F γp,θ ) with p ≥ 1 and θ ≥ 1 , and (cid:107) f o (cid:107) ∞ ≤

Original: B f for a ﬁxed constant B f > 0 , where the smoothness γ is either the mixed smoothness or the anisotropic smoothness.
Revised: B f for a ﬁxed constant B f > 0 , where the smoothness γ is either the mixed smoothness or the anisotropic smoothness.

Original:  R ELATION TO EXISTING WORK
Revised:  R ELATION TO EXISTING WORK

Original: A function space with the mixed smoothness in a ﬁnite dimensional setting can be found in
Revised: A function space with the mixed smoothness in a ﬁnite dimensional setting can be found in

Original: Schmeisser (1987); Sickel & Ullrich (2009), in which the mixed smooth Besov space is deﬁned.
Revised: Schmeisser (1987); Sickel & Ullrich (2009), in which the mixed smooth Besov space is deﬁned.

Original: The approximation and estimation errors of deep neural networks for the mixed smooth Besov space were analyzed by Suzuki (2019) for a special setting of a 1 = · · · = a d , and an approximation error analysis for a 1 = · · · = a d = 2 was given by Montanelli & Du (2019) using a sparse-grid technique.
Revised: The approximation and estimation errors of deep neural networks for the mixed smooth Besov space were analyzed by Suzuki (2019) for a special setting of a 1 = · · · = a d , and an approximation error analysis for a 1 = · · · = a d = 2 was given by Montanelli & Du (2019) using a sparse-grid technique.

Original: The mathematical properties of the anisotropic Besov space with ﬁnite dimensional input were analyzed in Nikol’skii (1975); Vybiral (2006); Triebel (2011).
Revised: The mathematical properties of the anisotropic Besov space with ﬁnite dimensional input were analyzed in Nikol’skii (1975); Vybiral (2006); Triebel (2011).

Original: The statistical analysis on the anisotropic
Revised: The statistical analysis on the anisotropic

Original: Besov space can be dated back to Ibragimov & Khas’minskii (1984) and they derived the minimax optimal rate for density estimation where the density is in an anisotropic Besov space.
Revised: Besov space can be dated back to Ibragimov & Khas’minskii (1984) and they derived the minimax optimal rate for density estimation where the density is in an anisotropic Besov space.

Original: Nyssbaum (1983; 1987) also analyzed a nonparametric regression problem on an anisotropic Besov space.
Revised: Nyssbaum (1983; 1987) also analyzed a nonparametric regression problem on an anisotropic Besov space.

Original: The approximation and estimation error bounds by deep neural networks for composition functions in anisotropic Besove spaces and superiority of deep learning compared to the kernel methods are shown by Suzuki & Nitanda (2021).
Revised: The approximation and estimation error bounds by deep neural networks for composition functions in anisotropic Besove spaces and superiority of deep learning compared to the kernel methods are shown by Suzuki & Nitanda (2021).

Original: However, all of these studies are about ﬁnite dimensional input and it is far from trivial to generalize it to the inﬁnite dimensional setting.
Revised: However, all of these studies are about ﬁnite dimensional input and it is far from trivial to generalize it to the inﬁnite dimensional setting.

Original: Our analysis for the γ -smooth function space is closely related to Ingster & Stepanova (2011) in which the anisotropic Sobolev space deﬁned by F c = W
Revised: Our analysis for the γ -smooth function space is closely related to Ingster & Stepanova (2011) in which the anisotropic Sobolev space deﬁned by F c = W

Original: [Equation]
Revised: [Equation]

Original: ∞ (cid:9) is analyzed.
Revised: ∞ (cid:9) is analyzed.

Original: They also derived a similar convergence rate to ours for non-deep learning estimator for a Gaussian white noise model.
Revised: They also derived a similar convergence rate to ours for non-deep learning estimator for a Gaussian white noise model.

Original: In the literature of the functional data analysis , the NadarayaWatson estimator for functional input has been extensively studied (Ferraty et al.
Revised: In the literature of the functional data analysis , the NadarayaWatson estimator for functional input has been extensively studied (Ferraty et al.

Original: and Ling & Vieu (2018) for a comprehensive survey).
Revised: and Ling & Vieu (2018) for a comprehensive survey).

Original: If we apply the bound given in the literature to our setting, the learning rate can be 1 / poly- log( n ) which is much slower than our analysis.
Revised: If we apply the bound given in the literature to our setting, the learning rate can be 1 / poly- log( n ) which is much slower than our analysis.

Original: This is because their analysis does not make use of γ -smoothness.
Revised: This is because their analysis does not make use of γ -smoothness.

Original: See Appendix C for more details.
Revised: See Appendix C for more details.

Original: Kohler & Langer (2020) analyzed CNNs in a setting where the target function has a hierarchical max-pooling structure each layer of which is sufﬁciently smooth.
Revised: Kohler & Langer (2020) analyzed CNNs in a setting where the target function has a hierarchical max-pooling structure each layer of which is sufﬁciently smooth.

Original: On the other hand, our γ -smooth function class imposes smoothness more directly on the target function.
Revised: On the other hand, our γ -smooth function class imposes smoothness more directly on the target function.

Original: Liu et al.
Revised: Liu et al.

Original: (2021) analyzed learning ability of CNNs with a ResNet structure in a classiﬁcation task where the data are distributed on a low-dimensional manifold and established a rate which only depends on the dimensionality of the low dimensional manifold.
Revised: (2021) analyzed learning ability of CNNs with a ResNet structure in a classiﬁcation task where the data are distributed on a low-dimensional manifold and established a rate which only depends on the dimensionality of the low dimensional manifold.

Original: However, the input should be distributed on a low dimensional manifold, while our analysis allows its support to be inﬁnite dimensional.
Revised: However, the input should be distributed on a low dimensional manifold, while our analysis allows its support to be inﬁnite dimensional.

Original: Estimation errors on a low dimensional structure also have been studied in Yang & Dunson (2016);
Revised: Estimation errors on a low dimensional structure also have been studied in Yang & Dunson (2016);

Original: Bickel & Li (2007);
Revised: Bickel & Li (2007);

Original: Nakada & Imaizumi (2020); Schmidt-Hieber (2019); Chen et al. (2019b).
Revised: Nakada & Imaizumi (2020); Schmidt-Hieber (2019); Chen et al. (2019b).

Original:  D EFINITION OF A DILATED CONVOLUTIONAL NEURAL NETWORK
Revised:  D EFINITION OF A DILATED CONVOLUTIONAL NEURAL NETWORK

Original: In this section, we introduce the neural network model that we investigate in this paper.
Revised: In this section, we introduce the neural network model that we investigate in this paper.

Original: Let L ∈
Revised: Let L ∈

Original: N be the depth of the network and d i ( i = 1 , . . .
Revised: N be the depth of the network and d i ( i = 1 , . . .

Original: , L + 1) be the width of the i -th layer in the network where we set d L +1 = 1 .
Revised: , L + 1) be the width of the i -th layer in the network where we set d L +1 = 1 .

Original: Then, the fully connected neural network (FNN) can be given by ( A L η ( · ) + b L ) ◦ · · · ◦ ( A i η ( · ) + b i ) ◦ · · · ◦ ( A 1 x + b 1 ) where A i ∈ R d i +1 ×
Revised: Then, the fully connected neural network (FNN) can be given by ( A L η ( · ) + b L ) ◦ · · · ◦ ( A i η ( · ) + b i ) ◦ · · · ◦ ( A 1 x + b 1 ) where A i ∈ R d i +1 ×

Original: d i , b i ∈ R d
Revised: d i , b i ∈ R d

Original: i +1 and η ( x ) = max { x, 0 } is the ReLU activation function that is applied element-wise.
Revised: i +1 and η ( x ) = max { x, 0 } is the ReLU activation function that is applied element-wise.

Original: The set of FNN with depth L ∈ N , maximum width W ∈ N , norm bound B > 0 , and sparsity level S ∈ N is deﬁned by
Revised: The set of FNN with depth L ∈ N , maximum width W ∈ N , norm bound B > 0 , and sparsity level S ∈ N is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) ∞ is the maximum absolute value among the elements of a vector or matrix 3 , and (cid:107) ·
Revised: where (cid:107) · (cid:107) ∞ is the maximum absolute value among the elements of a vector or matrix 3 , and (cid:107) ·

Original: (cid:107)is the number of non-zero elements of a vector or matrix.
Revised: (cid:107)is the number of non-zero elements of a vector or matrix.

Original: Next, we deﬁne the (dilated) CNNs.
Revised: Next, we deﬁne the (dilated) CNNs.

Original: Let C ∈ N be the number of channels and R C ×∞ :
Revised: Let C ∈ N be the number of channels and R C ×∞ :

Original: = (cid:8) ( x 1 , . . .
Revised: = (cid:8) ( x 1 , . . .

Original: x i , . . . )
Revised: x i , . . . )

Original: : x i ∈ R C (cid:9) .
Revised: : x i ∈ R C (cid:9) .

Original: Suppose that w ∈ R C × W (cid:48) is a ﬁlter with a width W (cid:48) ∈ N , channel size C ∈ N and an interval h ∈ N , then deﬁne the dilated convolution w (cid:63) h X (cid:48)
Revised: Suppose that w ∈ R C × W (cid:48) is a ﬁlter with a width W (cid:48) ∈ N , channel size C ∈ N and an interval h ∈ N , then deﬁne the dilated convolution w (cid:63) h X (cid:48)

Original: ∈ R ∞ for an inﬁnitesequence of vectors X (cid:48)
Revised: ∈ R ∞ for an inﬁnitesequence of vectors X (cid:48)

Original: =
Revised: =

Original: ( x (cid:48)
Revised: ( x (cid:48)

Original: i,
Revised: i,j

Original: [Equation]
Revised: [Equation]

Original: , ∞

Original:  i [Equation] [Equation] as ( 
Revised: C, ∞ i =1 ,j =1 ∈ R C ×∞ as ( w (cid:63) h X (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: (cid:63)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  k .
Revised: j − 1)+ k .

Original: When h = 1 , it is called a normal convolution.
Revised: When h = 1 , it is called a normal convolution.

Original: Moreover, given a ﬁlter F ∈ R C (cid:48) × C × W (cid:48) with ( C (cid:48) ) - multiple channel outputs, we deﬁne its corresponding convlution Conv h,F : R C ×∞ → R C (cid:48) ×∞ as
Revised: Moreover, given a ﬁlter F ∈ R C (cid:48) × C × W (cid:48) with ( C (cid:48) ) - multiple channel outputs, we deﬁne its corresponding convlution Conv h,F : R C ×∞ → R C (cid:48) ×∞ as

Original: [Equation]
Revised: [Equation]

Original: Then, the dilated CNN can be deﬁned as follows (its illustration can be found in Figure 1).
Revised: Then, the dilated CNN can be deﬁned as follows (its illustration can be found in Figure 1).

Original: Deﬁnition 4 (Dilated CNN) .
Revised: Deﬁnition 4 (Dilated CNN) .

Original: For a given L (cid:48) , W (cid:48) ∈ N , suppose that we are given ﬁlters F l ∈
Revised: For a given L (cid:48) , W (cid:48) ∈ N , suppose that we are given ﬁlters F l ∈

Original: R C l +1 × C l × W (cid:48) with the number of channels C l ∈
Revised: R C l +1 × C l × W (cid:48) with the number of channels C l ∈

Original: N ( l ∈
Revised: N ( l ∈

Original: [ L (cid:48) ]) with
Revised: [ L (cid:48) ]) with

Original: C 1 = 1 and an FNN g FNN ∈ Φ( L, W, B, S ) , then a neural network given by f ( X ) = (cid:16)
Revised: C 1 = 1 and an FNN g FNN ∈ Φ( L, W, B, S ) , then a neural network given by f ( X ) = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17) 1 is called a dilated CNN 4 , where g FNN is assumed to be applied in an element-wise manner to the inﬁnite sequence.
Revised: 1 is called a dilated CNN 4 , where g FNN is assumed to be applied in an element-wise manner to the inﬁnite sequence.

Original: The set of dilated CNNs with the same number of channels C l = C (cid:48) (2 ≤ ∀ l ≤ L (cid:48) ) in all layers but C 1 = 1 is denoted by
Revised: The set of dilated CNNs with the same number of channels C l = C (cid:48) (2 ≤ ∀ l ≤ L (cid:48) ) in all layers but C 1 = 1 is denoted by

Original: [Equation]
Revised: [Equation]

Original: For simplicity, the set of dilated CNNs is abbreviated to P when there is no ambiguity about the parameter conﬁguration.
Revised: For simplicity, the set of dilated CNNs is abbreviated to P when there is no ambiguity about the parameter conﬁguration.

Original: When L (cid:48)
Revised: When L (cid:48)

Original: = 1 , it coincides with a set of regular CNNs.
Revised: = 1 , it coincides with a set of regular CNNs.

Original: In our analysis, it is sufﬁcient to consider an dilated CNN with a constant number of channels throughout all layers ( C l = C ( ∀ l ∈
Revised: In our analysis, it is sufﬁcient to consider an dilated CNN with a constant number of channels throughout all layers ( C l = C ( ∀ l ∈

Original: [ L (cid:48) ]) ).
Revised: [ L (cid:48) ]) ).

Original: To evaluate the estimation accuracy, it is important to assume the functions in the set is bounded in terms of the L ∞ -norm.
Revised: To evaluate the estimation accuracy, it is important to assume the functions in the set is bounded in terms of the L ∞ -norm.

Original: For that purpose, we consider an dilated CNN clipped by a bound B f > 0 deﬁned as ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, S, B ) :
Revised: For that purpose, we consider an dilated CNN clipped by a bound B f > 0 deﬁned as ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, S, B ) :

Original: = (cid:8) ¯ f ( X ) = ( − B f ∨ ( B f ∧ f ( X ))) :
Revised: = (cid:8) ¯ f ( X ) = ( − B f ∨ ( B f ∧ f ( X ))) :

Original: f ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, S, B ) (cid:9) .
Revised: f ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, S, B ) (cid:9) .

Original: Remark 5.
Revised: Remark 5.

Original: In the deﬁnition of the dilated CNN, we do not impose ReLU activation.
Revised: In the deﬁnition of the dilated CNN, we do not impose ReLU activation.

Original: However, since
Revised: However, since

Original: ReLU activation can realize a linear function for a bounded input and thus our analysis can be straightforwardly applied even if there is nonlinear ReLU activation.
Revised: ReLU activation can realize a linear function for a bounded input and thus our analysis can be straightforwardly applied even if there is nonlinear ReLU activation.

Original: Moreover, this paper mainly focuses on 1D-convolution, but it can be generalized to 2D-convolution .
Revised: Moreover, this paper mainly focuses on 1D-convolution, but it can be generalized to 2D-convolution .

Original: See Appendix I for the detailed discussions in which it is shown that γ -smoothness over a wavelet decomposition of an input image achieves the same rate of convergence as in 1D-convolution.
Revised: See Appendix I for the detailed discussions in which it is shown that γ -smoothness over a wavelet decomposition of an input image achieves the same rate of convergence as in 1D-convolution.

Original:  A PPROXIMATION AND ESTIMATION ERRORS OF DEEP LEARNING
Revised:  A PPROXIMATION AND ESTIMATION ERRORS OF DEEP LEARNING

Original: In this section, we give our main result about the approximation and estimation errors of FNNs and dilated CNNs when the true function f
Revised: In this section, we give our main result about the approximation and estimation errors of FNNs and dilated CNNs when the true function f

Original: o is in the γ -smooth function class.
Revised: o is in the γ -smooth function class.

Original: 5. A PPROXIMATION E RROR A NALYSIS BY F ULLY C ONNECTED N EURAL N ETWORKS
Revised: 5. A PPROXIMATION E RROR A NALYSIS BY F ULLY C ONNECTED N EURAL N ETWORKS

Original: Here, we present the approximation error analysis of FNNs for a general smoothness γ not restricted to the mixed/anisotropic smoothness.
Revised: Here, we present the approximation error analysis of FNNs for a general smoothness γ not restricted to the mixed/anisotropic smoothness.

Original: For a given T > 0 and the smoothness γ : N ∞ 

Revised: For a given T > 0 and the smoothness γ : N ∞

Original: → R > 0 , deﬁne
Revised: → R > 0 , deﬁne

Original: I ( T
Revised: I ( T,

Original: [Equation]
Revised: [Equation]

Original: and then the following quantities play an important role in our approximation error analysis.
Revised: and then the following quantities play an important role in our approximation error analysis.

Original: Deﬁnition 6 (Axial complexity and frequency direction complexity) .
Revised: Deﬁnition 6 (Axial complexity and frequency direction complexity) .

Original: The axial complexity is deﬁned by d max ( T, γ ) :
Revised: The axial complexity is deﬁned by d max ( T, γ ) :

Original: =
Revised: =

Original: | I ( T, γ ) | .
Revised: | I ( T, γ ) | .

Original: Moreover, the frequency direction complexity is deﬁned by f max ( T, γ ) :
Revised: Moreover, the frequency direction complexity is deﬁned by f max ( T, γ ) :

Original: = max s ∈ N ∞ 0 : γ ( s ) ≤ T max i ∈ N
Revised: = max s ∈ N ∞ 0 : γ ( s ) ≤ T max i ∈ N

Original: s i .
Revised: s i .

Original: The axial complexity is used to evaluate how many components need to be extracted from a given inﬁnite-dimensional sequence X ∈ R ∞ to achieve a particular approximation error, and the frequency complexity characterizes up to which frequency we require to approximate a target function with a particular error.
Revised: The axial complexity is used to evaluate how many components need to be extracted from a given inﬁnite-dimensional sequence X ∈ R ∞ to achieve a particular approximation error, and the frequency complexity characterizes up to which frequency we require to approximate a target function with a particular error.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where ( x ) + := max { x, 0 } .
Revised: where ( x ) + := max { x, 0 } .

Original: Then, a general approximation error theory for FNNs can be obtained as follows.
Revised: Then, a general approximation error theory for FNNs can be obtained as follows.

Original: Theorem 7 (Approximation error for the γ -smooth space by FNNs) .
Revised: Theorem 7 (Approximation error for the γ -smooth space by FNNs) .

Original: Assume that γ, γ (cid:48) : N ∞ 

Revised: Assume that γ, γ (cid:48) : N ∞

Original: → R > 0 satisfy
Revised: → R > 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: and the target function f ∈ F γp,θ ( p ≥ 1 , θ ≥ 1) to be approximated satisﬁes (cid:107) f (cid:107) ∞ ≤ B f for a constant B f ∈ R > 0 .
Revised: and the target function f ∈ F γp,θ ( p ≥ 1 , θ ≥ 1) to be approximated satisﬁes (cid:107) f (cid:107) ∞ ≤ B f for a constant B f ∈ R > 0 .

Original: For arbitrary T > 0 , we let a tuple ( d max , f max , G ) be 
Revised: For arbitrary T > 0 , we let a tuple ( d max , f max , G ) be ( d max , f max , [Equation] d

Original: [Equation]
Revised: [Equation]

Original:  (2 < θ ) , and with some positive constants K, K (cid:48) depending only on B f , we let
Revised: ) (2 < θ ) , and with some positive constants K, K (cid:48) depending only on B f , we let

Original: [Equation]
Revised: [Equation]

Original: Then, there exists an FNN ˆ R T ∈
Revised: Then, there exists an FNN ˆ R T ∈

Original: Φ( L, W, S, B ) with d max -dimensional input that takes ( x i ) i ∈ I ( T,γ ) ∈ [0 , 1] d max as an input such that f (cid:48) :
Revised: Φ( L, W, S, B ) with d max -dimensional input that takes ( x i ) i ∈ I ( T,γ ) ∈ [0 , 1] d max as an input such that f (cid:48) :

Original: [0 , 1] ∞
Revised: [0 , 1] ∞

Original: → R given by f (cid:48) ( X ) :
Revised: → R given by f (cid:48) ( X ) :

Original: = ˆ R T (cid:0) ( x i ) i ∈ I ( T,γ ) (cid:1) for X
Revised: = ˆ R T (cid:0) ( x i ) i ∈ I ( T,γ ) (cid:1) for X

Original: =
Revised: =

Original: ( x i ) ∞
Revised: ( x i ) ∞

Original: i =1 ∈ [0 , 1] ∞ satisﬁes
Revised: i =1 ∈ [0 , 1] ∞ satisﬁes

Original: [Equation]
Revised: [Equation]

Revised:  

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (2 <
Revised: 2 <

Original: [Table]
Revised: [Table]

Original: According to this theorem, the derived approximation error can be achieved by FNNs if the required d max components of the input X is extracted.
Revised: According to this theorem, the derived approximation error can be achieved by FNNs if the required d max components of the input X is extracted.

Original: This theorem clariﬁes how the decay rate of the frequency components of the target function affects the approximation accuracy.
Revised: This theorem clariﬁes how the decay rate of the frequency components of the target function affects the approximation accuracy.

Original: Since the approximation accuracy is determined by ( d max , f max , G ) , it is not directly affected by the dimensionality but is characterized merely by the smoothness parameter γ .
Revised: Since the approximation accuracy is determined by ( d max , f max , G ) , it is not directly affected by the dimensionality but is characterized merely by the smoothness parameter γ .

Original: Intuitively, T > 0 controls the approximation accuracy and simultaneously controls up to which frequency is used for the approximation.
Revised: Intuitively, T > 0 controls the approximation accuracy and simultaneously controls up to which frequency is used for the approximation.

Original: Speciﬁcally, the difﬁculty of the approximation is determined by the number of bases required that is characterized by the number of s ∈ N ∞ 0 with γ ( s ) < T , and the maximum frequency required for the approximation is also important for the analysis.
Revised: Speciﬁcally, the difﬁculty of the approximation is determined by the number of bases required that is characterized by the number of s ∈ N ∞ 0 with γ ( s ) < T , and the maximum frequency required for the approximation is also important for the analysis.

Original: The bound is proven by evaluating an approximation error of a trigonometric polynomial approximation of f ∈ F γp,θ and showing that we can construct a neural network that approximates a trigonometric polynomial with a certain accuracy.
Revised: The bound is proven by evaluating an approximation error of a trigonometric polynomial approximation of f ∈ F γp,θ and showing that we can construct a neural network that approximates a trigonometric polynomial with a certain accuracy.

Original: 5. S MOOTHNESS WITH POLYNOMIAL ORDER INCREASE
Revised: 5. S MOOTHNESS WITH POLYNOMIAL ORDER INCREASE

Original: Here, we derive a concrete convergence rate for CNNs in a setting where γ is mixed or anisotropic smoothness and the smoothness parameter a = ( a i ) ∞
Revised: Here, we derive a concrete convergence rate for CNNs in a setting where γ is mixed or anisotropic smoothness and the smoothness parameter a = ( a i ) ∞

Original: i =1 is polynomially increasing.
Revised: i =1 is polynomially increasing.

Original: In this setting, we just need to use only one layer CNN.
Revised: In this setting, we just need to use only one layer CNN.

Original: Deeper CNN layers will be used in the next section (Section 5.3) to perform adaptive feature extraction from wide range of inputs.
Revised: Deeper CNN layers will be used in the next section (Section 5.3) to perform adaptive feature extraction from wide range of inputs.

Original: Assumption 8.
Revised: Assumption 8.

Original: There exists 0 < q < ∞ such that the smoothness parameter a = ( a i ) ∞
Revised: There exists 0 < q < ∞ such that the smoothness parameter a = ( a i ) ∞

Original: i =1 satisﬁes a i = Ω( i q ) .
Revised: i =1 satisﬁes a i = Ω( i q ) .

Original: We also assume a 1 < a 2 for the mixed smoothness setting.
Revised: We also assume a 1 < a 2 for the mixed smoothness setting.

Original: This assumption impose that the target function should be sufﬁciently smooth with respect to higher order indices.
Revised: This assumption impose that the target function should be sufﬁciently smooth with respect to higher order indices.

Original: Under this setting, we show the approximation and estimation errors as follows.
Revised: Under this setting, we show the approximation and estimation errors as follows.

Original: First, the approximation error by the CNNs can be evaluated as follows.
Revised: First, the approximation error by the CNNs can be evaluated as follows.

Original: Theorem 9 (Approximation error bound under smoothness with polynomial order increase) .
Revised: Theorem 9 (Approximation error bound under smoothness with polynomial order increase) .

Original: Suppose that Assumptions 3 and 8 hold, then we have the following approximation error bounds:
Revised: Suppose that Assumptions 3 and 8 hold, then we have the following approximation error bounds:

Original: Mixed smoothness ( γ ( s ) =
Revised: Mixed smoothness ( γ ( s ) =

Original: (cid:104) a, s (cid:105) )
Revised: (cid:104) a, s (cid:105) )

Original: : Suppose that v/a 1 < 1 .
Revised: : Suppose that v/a 1 < 1 .

Original: Then, for arbitrary T > 0 , there exists a conﬁguration of the network structure, L (cid:48)
Revised: Then, for arbitrary T > 0 , there exists a conﬁguration of the network structure

Original: = 1 , B (cid:48) 
Revised: , L (cid:48) = B

Original: = 1 , W (cid:48) ∼ T

Original: [Equation]
Revised: [Equation]

Original: ∼ T
Revised: , W

Original: [Equation]
Revised: [Equation]

Original: and
Revised: T

Original: [Equation]
Revised: [Equation]

Original: such that there exists an dilated CNN f (cid:48) ∈ P( L
Revised: such that there exists an dilated CNN f (cid:48) [Equation] P

Original: [Equation]
Revised: [Equation]

Revised: L

Revised: [Equation]

Original: , B
Revised: , B

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , L
Revised: , L

Original: [Equation]
Revised: [Equation]

Original: T ) , W 1 ( T ) , S 1 ( T ) , B 1 ( T )) satisfying the following approximation error:
Revised: T ) , W 1 ( T ) , S 1 ( T ) , B 1 ( T )) satisfying the following approximation error:

Original: [Equation]
Revised: [Equation]

Original: Anisotropic smoothness ( γ ( s ) =
Revised: Anisotropic smoothness ( γ ( s ) =

Original: max i { a i s i } ) : Let ˜ a :=
Revised: max i { a i s i } ) : Let ˜ a :=

Original: ( (cid:80) ∞ i =1 a − 1 i ) − 1 and suppose 0 < ˜ a and v < ˜ a , 
Revised: ( (cid:80) ∞ i =1 a − 1 i ) − 1 and suppose 0 < ˜ a and v < ˜ a , then

Original: then there exists a network structure setting L (cid:48)
Revised: there exists a network structure setting L (cid:48)

Original: = 1 , B (cid:48)
Revised: = 1 , B (cid:48)

Original: = 1 , W (cid:48) ∼ T 1 q , C (cid:48) ∼ T 1 q and
Revised: = 1 , W (cid:48) ∼ T 1 q , C (cid:48) ∼ T 1 q and

Original: L 2 ( T ) ∼ max (cid:8) T 2 q , T 2 (cid:9) , W 2 ( T ) ∼ T 1 q 2 T/ ˜ a , S 2 ( T ) ∼ T 2 q max (cid:8) T 2 q , T 2 (cid:9) 2 T/ ˜ a , B 2 ( T ) ∼ ( √ 2) T 1 q , such that there exists an dilated CNN f (cid:48) ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L 2 ( T ) , W 2 ( T ) , S 2 ( T ) , B 2 ( T )) satisfying the following approximation error: (cid:107) [Equation] [Equation] a
Revised: L 2 ( T ) ∼ max (cid:8) T 2 q , T 2 (cid:9) , W 2 ( T ) ∼ T 1 q 2 T/ ˜ a , S 2 ( T ) ∼ T 2 q max (cid:8) T 2 q , T 2 (cid:9) 2 T/ ˜ a , B 2 ( T ) ∼ ( √ 2) T 1 q , such that there exists an dilated CNN f (cid:48) ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L 2 ( T ) , W 2 ( T ) , S 2 ( T ) , B 2 ( T )) satisfying the following approximation error: (cid:107) f

Original: [Equation]
Revised: [Equation]

Original: T .
Revised: T .

Original: The proof can be found in Appendix E. From this theorem, we can see that the number of layers, the width, the number of parameters, and the size of the parameters are both determined by T and the smoothness parameter a .
Revised: The proof can be found in Appendix E. From this theorem, we can see that the number of layers, the width, the number of parameters, and the size of the parameters are both determined by T and the smoothness parameter a .

Original: Moreover, in Theorem 7, the approximation error was derived assuming that the appropriate index set I ( T, γ ) was provided.
Revised: Moreover, in Theorem 7, the approximation error was derived assuming that the appropriate index set I ( T, γ ) was provided.

Original: On the other hand, in Theorem 9, we do not make such an assumption because the CNNs can automatically extract the required index
Revised: On the other hand, in Theorem 9, we do not make such an assumption because the CNNs can automatically extract the required index

Original: I ( T, γ ) .
Revised: I ( T, γ ) .

Original: Next, we consider the estimation error of these models in the regression problem (Eq.
Revised: Next, we consider the estimation error of these models in the regression problem (Eq.

Original: Suppose that we are given n observations D n =
Revised: Suppose that we are given n observations D n =

Original: ( X i , y i ) ni
Revised: ( X i , y i ) ni

Original: =1 following the model (1).
Revised: =1 following the model (1).

Original: We consider the empirical risk minimization estimator (ERM estimator) in the model ¯ P that is given by any minimizer of the empirical risk:
Revised: We consider the empirical risk minimization estimator (ERM estimator) in the model ¯ P that is given by any minimizer of the empirical risk:

Original: [Equation]
Revised: [Equation]

Original: As we have stated above, we employ the mean squared error (cid:107) ˆ f − f o (cid:107) 2 P X as a performance measure.
Revised: As we have stated above, we employ the mean squared error (cid:107) ˆ f − f o (cid:107) 2 P X as a performance measure.

Original: Since ˆ f depends on the training data D n , we take expectation with respect to D n : E P n [ (cid:107) ˆ f − f o (cid:107) 2 P X ] :
Revised: Since ˆ f depends on the training data D n , we take expectation with respect to D n : E P n [ (cid:107) ˆ f − f o (cid:107) 2 P X ] :

Original: =
Revised: =

Original: E ( X i ,y i ) ni
Revised: E ( X i ,y i ) ni

Original: =1 ∼ P n [ (cid:107) ˆ f − f o (cid:107) 2 P X ] .
Revised: =1 ∼ P n [ (cid:107) ˆ f − f o (cid:107) 2 P X ] .

Original: Then, the following theorem holds.
Revised: Then, the following theorem holds.

Original: Theorem 10 (Estimation error under smoothness with polynomial order increase) .
Revised: Theorem 10 (Estimation error under smoothness with polynomial order increase) .

Original: Suppose that
Revised: Suppose that

Original: Assumptions 3 and 8 hold, then we have the following estimation error bounds:
Revised: Assumptions 3 and 8 hold, then we have the following estimation error bounds:

Original: Mixed smoothness ( γ ( s ) =
Revised: Mixed smoothness ( γ ( s ) =

Original: (cid:104) a, s (cid:105) )
Revised: (cid:104) a, s (cid:105) )

Original: : If v/a 1 < 1 , then by setting the network structure as L (cid:48) = 1 , B (cid:48)
Revised: : If v/a 1 < 1 , then by setting the network structure as L (cid:48) = 1 , B (cid:48)

Original: = 1 , W (cid:48) ∼ (log n ) 1 q , C (cid:48) ∼ (log n ) 1 q
Revised: = 1 , W (cid:48) ∼ (log n ) 1 q , C (cid:48) ∼ (log n ) 1 q

Original: and ( L, W, S, B ) =
Revised: and ( L, W, S, B ) =

Original: ( L 1 ( T ) , W 1 ( T ) , S 1 ( T ) , B 1 ( T )) for T = a [-
 [Equation]-] a
Revised: ( L 1 ( T ) , W 1 ( T ) , S 1 ( T ) , B 1 ( T )) for T = a [-
 [Equation]-]1 2( a

Original: [Equation]
Revised: [Equation]

Original: v )+1 log 2 ( n ) , the ERM estimator ˆ f in ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, S, B ) achievesE
Revised: v )+1 log 2 ( n ) , the ERM estimator ˆ f in ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, S, B ) achieves

Original: [Equation]
Revised: [Equation]

Original: Anisotropic smoothness ( γ ( s ) =
Revised: Anisotropic smoothness ( γ ( s ) =

Original: max i { a i s i } ) :
Revised: max i { a i s i } ) :

Original: Under the same setting, if v < ˜ a , by setting the network structure as L (cid:48) = 1 , B (cid:48)
Revised: Under the same setting, if v < ˜ a , by setting the network structure as L (cid:48) = 1 , B (cid:48)

Original: = 1 , W (cid:48) ∼ (log n ) 1 q , C (cid:48) ∼ (log n ) 1 q
Revised: = 1 , W (cid:48) ∼ (log n ) 1 q , C (cid:48) ∼ (log n ) 1 q

Original: and ( L, W, S, B ) =
Revised: and ( L, W, S, B ) =

Original: ( L 2 ( T ) , W 2 ( T ) , S 2 ( T ) , B 2 ( T )) for T = ˜ a 2(˜ a − v )+1 log 2 ( n ) , the ERM estimator ˆ f in ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, S, B ) achieves
Revised: ( L 2 ( T ) , W 2 ( T ) , S 2 ( T ) , B 2 ( T )) for T = ˜ a 2(˜ a − v )+1 log 2 ( n ) , the ERM estimator ˆ f in ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, S, B ) achieves

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in
Revised: The proof can be found in

Original: Appendix F.
Revised: Appendix F.

Original: This theorem shows that even if the dimension of the input data is inﬁnite, for a function with a particular smoothness, CNNs can achieve a dimensionindependent convergence rate which is a polynomial order, that is, it can avoid the curse of dimensionality by utilizing the increasing smoothness.
Revised: This theorem shows that even if the dimension of the input data is inﬁnite, for a function with a particular smoothness, CNNs can achieve a dimensionindependent convergence rate which is a polynomial order, that is, it can avoid the curse of dimensionality by utilizing the increasing smoothness.

Original: We can see that the derived convergence rate is a direct extension of ﬁnite dimensional one.
Revised: We can see that the derived convergence rate is a direct extension of ﬁnite dimensional one.

Original: Actually, if v = 0 , the rate for the anisotropic smoothness matches that of the ﬁnite dimensional one (Suzuki & Nitanda, 2021) up to poly-log order which is known as minimax optimal.
Revised: Actually, if v = 0 , the rate for the anisotropic smoothness matches that of the ﬁnite dimensional one (Suzuki & Nitanda, 2021) up to poly-log order which is known as minimax optimal.

Original: Therefore, CNNs can achieve the optimal rate up to poly-log order at least when v = 0 .
Revised: Therefore, CNNs can achieve the optimal rate up to poly-log order at least when v = 0 .

Original: As for the mixed smoothness, a ﬁnite dimensional version was analyzed (Suzuki, 2019) and a similar rate was derived.
Revised: As for the mixed smoothness, a ﬁnite dimensional version was analyzed (Suzuki, 2019) and a similar rate was derived.

Original: However, our analysis assumes a 1 < a 2 and a i = Ω( i q ) and thus obtained completely dimensionality independent bound while the bound by Suzuki (2019) depends on d in the exponent of the poly-log order.
Revised: However, our analysis assumes a 1 < a 2 and a i = Ω( i q ) and thus obtained completely dimensionality independent bound while the bound by Suzuki (2019) depends on d in the exponent of the poly-log order.

Original: 5. S MOOTHNESS WITH SPARSITY
Revised: 5. S MOOTHNESS WITH SPARSITY

Original: Next, we relax the assumption a i = Ω( i q ) and consider a situation where there is a kind of sparse structure in a .
Revised: Next, we relax the assumption a i = Ω( i q ) and consider a situation where there is a kind of sparse structure in a .

Original: As we have seen in the previous section, under the assumption that the coordinates with large indices are not important, polynomial-order convergence rate depending only on the smoothness can be achieved by the ordinary CNNs.
Revised: As we have seen in the previous section, under the assumption that the coordinates with large indices are not important, polynomial-order convergence rate depending only on the smoothness can be achieved by the ordinary CNNs.

Original: In this section, we show that similar rates can be achieved by using dilated CNNs even when a does not satisfy the polynomial order increase if a has sparsity .
Revised: In this section, we show that similar rates can be achieved by using dilated CNNs even when a does not satisfy the polynomial order increase if a has sparsity .

Original: For that purpose, we ﬁrst deﬁne the sparsity of the smoothness.
Revised: For that purpose, we ﬁrst deﬁne the sparsity of the smoothness.

Original: Deﬁnition 

Revised: Deﬁnition

Original: (Weak (cid:96) q -norm of smoothness) .
Revised: (Weak (cid:96) q -norm of smoothness) .

Original: Given a = ( a i ) ∞ i =1 ∈ R ∞ > 0 which is not necessarily monotonically increasing, consider the sorted sequence 0 < a i 1 ≤ a i 2 ≤ · · · in the ascending order.
Revised: Given a = ( a i ) ∞ i =1 ∈ R ∞ > 0 which is not necessarily monotonically increasing, consider the sorted sequence 0 < a i 1 ≤ a i 2 ≤ · · · in the ascending order.

Original: Then, deﬁne its weak (cid:96) q -norm for 0 < q < ∞ as (cid:107) a (cid:107) wl q :=
Revised: Then, deﬁne its weak (cid:96) q -norm for 0 < q < ∞ as (cid:107) a (cid:107) wl q :=

Original: sup j j q a − 1 i j .
Revised: sup j j q a − 1 i j .

Original: This kind of sparsity inducing norm were introduced and discussed previously in Donoho (1993);
Revised: This kind of sparsity inducing norm were introduced and discussed previously in Donoho (1993);

Original: Donoho et al.
Revised: Donoho et al.

Original: ; Yang & Barron (1999) to quantify sparsity of coefﬁcients of basis expansions.
Revised: ; Yang & Barron (1999) to quantify sparsity of coefﬁcients of basis expansions.

Original: We notice that, if (cid:107) a (cid:107) wl q is small, almost all a i s are very large and there are only few indices that are small, which means sparseness.
Revised: We notice that, if (cid:107) a (cid:107) wl q is small, almost all a i s are very large and there are only few indices that are small, which means sparseness.

Original: If the smoothness parameter a has a small weak (cid:96) q -norm, then we can say that the functions with such smoothness has a small number of important coordinate directions.
Revised: If the smoothness parameter a has a small weak (cid:96) q -norm, then we can say that the functions with such smoothness has a small number of important coordinate directions.

Original: Therefore, it is expected that we can approximate such a function efﬁciently by a neural network.
Revised: Therefore, it is expected that we can approximate such a function efﬁciently by a neural network.

Original: In this section, we analyze the approximation and estimation errors under the condition of sparse smoothness.
Revised: In this section, we analyze the approximation and estimation errors under the condition of sparse smoothness.

Original: Assumption 12.
Revised: Assumption 12.

Original: a = ( a i ) ∞
Revised: a = ( a i ) ∞

Original: i =1 satisﬁes (cid:107) a (cid:107) wl q ≤ 1 for 0 < q < ∞ and a i = Ω(log i ) .
Revised: i =1 satisﬁes (cid:107) a (cid:107) wl q ≤ 1 for 0 < q < ∞ and a i = Ω(log i ) .

Original: Note that this assumption relaxes the condition a i = Ω( i q ) in Assumption 8 to a i = Ω(log i ) .
Revised: Note that this assumption relaxes the condition a i = Ω( i q ) in Assumption 8 to a i = Ω(log i ) .

Original: Instead, it imposes the sparsity (cid:107) a (cid:107) wl q ≤ 1 .
Revised: Instead, it imposes the sparsity (cid:107) a (cid:107) wl q ≤ 1 .

Original: Under this assumption, we obtain the following approximation error bound.
Revised: Under this assumption, we obtain the following approximation error bound.

Original: Theorem 13 (Approximation error bound for sparse smoothness) .
Revised: Theorem 13 (Approximation error bound for sparse smoothness) .

Original: Suppose that Assumptions 3 and 12 hold, then we have the following approximation error bounds for any T > 1 :
Revised: Suppose that Assumptions 3 and 12 hold, then we have the following approximation error bounds for any T > 1 :

Original: Mixed smoothness ( γ ( s ) =
Revised: Mixed smoothness ( γ ( s ) =

Original: (cid:104) a, s (cid:105) )
Revised: (cid:104) a, s (cid:105) )

Original: : Suppose that v/a i 1 < 1 , then there exist a set of network structure parameters satisfying
Revised: : Suppose that v/a i 1 < 1 , then there exist a set of network structure parameters satisfying

Original: [Equation]
Revised: [Equation]

Original: such that there exists an dilated 
Revised: such that there exists an dilated CNN

Original: [Equation]
Revised: [Equation]

Original: B 1 ( T )) satis-
Revised: 1 ( T )) satis-

Original: fying (cid:107) f (cid:48)
Revised: fying (cid:107) f (cid:48)

Original: − f o (cid:107) 2 (cid:46) 2 − (1 − 1 ai 1 ) T .
Revised: − f o (cid:107) 2 (cid:46) 2 − (1 − 1 ai 1 ) T .

Original: Anisotropic smoothness ( γ ( s ) =
Revised: Anisotropic smoothness ( γ ( s ) =

Original: max i { a i s i } ) : Suppose that 0 < ˜ a and v < ˜ a , then there exist a set of network structure parameters satisfying L (cid:48) ∼ T, B (cid:48) ∼ 1 , W (cid:48)
Revised: max i { a i s i } ) : Suppose that 0 < ˜ a and v < ˜ a , then there exist a set of network structure parameters satisfying L (cid:48) ∼ T, B (cid:48) ∼ 1 , W (cid:48)

Original: = 3 , C (cid:48) ∼ T 1 q such that there exists an dilated CNN f (cid:48) ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L 2 ( T ) , W 2 ( T ) , S 2 ( T ) , B 2 ( T )) satisfying
Revised: = 3 , C (cid:48) ∼ T 1 q such that there exists an dilated CNN f (cid:48) ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L 2 ( T ) , W 2 ( T ) , S 2 ( T ) , B 2 ( T )) satisfying

Original: (cid:107) f
Revised: (cid:107) f

Original: (cid:48) − f o (cid:107) 

Revised: (cid:48) − f o (cid:107)

Original: (cid:46) 2 − (1 − v/ ˜ a ) T .
Revised: (cid:46) 2 − (1 − v/ ˜ a ) T .

Original: We can see that this approximation error bound gives the same bound as Theorem 9 under a relaxed condition Assumption 12 with sparse smoothness.
Revised: We can see that this approximation error bound gives the same bound as Theorem 9 under a relaxed condition Assumption 12 with sparse smoothness.

Original: The only difference is the setting of the convolution part ( L (cid:48) , W (cid:48) , C ) and other parts are same as Theorem 9.
Revised: The only difference is the setting of the convolution part ( L (cid:48) , W (cid:48) , C ) and other parts are same as Theorem 9.

Original: This difference is required to ﬁnd the important indices that are relatively non-smooth compared with other indices.
Revised: This difference is required to ﬁnd the important indices that are relatively non-smooth compared with other indices.

Original: Thanks to the structure of dilated convolution, it can ﬁnd such indices from a long range of index set: { i ∈ N : i =
Revised: Thanks to the structure of dilated convolution, it can ﬁnd such indices from a long range of index set: { i ∈ N : i =

Original: O (3 L (cid:48) ) }
Revised: O (3 L (cid:48) ) }

Original: (see Figure 1 for illustration of feature extraction by CNNs).
Revised: (see Figure 1 for illustration of feature extraction by CNNs).

Original: Accordingly, we also have the following estimation error bound.
Revised: Accordingly, we also have the following estimation error bound.

Original: Theorem 14 (Estimation error for sparse smoothness) .
Revised: Theorem 14 (Estimation error for sparse smoothness) .

Original: Suppose that Assumptions 3 and 12 hold, then by setting L (cid:48) ∼ log n, B (cid:48) ∼ 1 , W (cid:48)
Revised: Suppose that Assumptions 3 and 12 hold, then by setting L (cid:48) ∼ log n, B (cid:48) ∼ 1 , W (cid:48)

Original: = 3 , C (cid:48) ∼ (log n ) 1 q
Revised: = 3 , C (cid:48) ∼ (log n ) 1 q

Original: and ( L, W, S, B ) as in Theorem 10, the ERM estimator ˆ f in the class of dilated CNNs can achieve the same convergence rate of the estimation error as Theorem 10.
Revised: and ( L, W, S, B ) as in Theorem 10, the ERM estimator ˆ f in the class of dilated CNNs can achieve the same convergence rate of the estimation error as Theorem 10.

Original: The proof can be found in Appendix G. These theorems show that for polynomially increasing smoothness, ordinary CNNs can perform optimal coordinate selection, while for sparse smoothness, dilated CNNs play an important role in coordinate selection.
Revised: The proof can be found in Appendix G. These theorems show that for polynomially increasing smoothness, ordinary CNNs can perform optimal coordinate selection, while for sparse smoothness, dilated CNNs play an important role in coordinate selection.

Original: This theorem shows that, when extracting data with long-term dependence, convergence rates that avoid dependence on dimensionality can be achieved by using dilated CNNs.
Revised: This theorem shows that, when extracting data with long-term dependence, convergence rates that avoid dependence on dimensionality can be achieved by using dilated CNNs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this study, we gave a condition on the smoothness of the function space as one of the situations in which the curse of dimensionality can be avoided when the input is ultra-high dimensional ( n (cid:28) d ) or inﬁnite dimensional ( d = ∞ ).
Revised: In this study, we gave a condition on the smoothness of the function space as one of the situations in which the curse of dimensionality can be avoided when the input is ultra-high dimensional ( n (cid:28) d ) or inﬁnite dimensional ( d = ∞ ).

Original: This study showed that the smoothness of the target function plays an essential role in characterizing the estimation error bound.
Revised: This study showed that the smoothness of the target function plays an essential role in characterizing the estimation error bound.

Original: Especially, when the smoothness parameter ( a i ) ∞
Revised: Especially, when the smoothness parameter ( a i ) ∞

Original: i =1 grows up as the index i increase, we can obtain a polynomial order convergence rate even if the input is inﬁnite dimensional.
Revised: i =1 grows up as the index i increase, we can obtain a polynomial order convergence rate even if the input is inﬁnite dimensional.

Original: Future plans include, for example, considering a situation where the smoothness depends on each input location, and extending the deﬁnition of F γp,θ so that it captures more realistic situations. 
Revised: Future plans include, for example, considering a situation where the smoothness depends on each input location, and extending the deﬁnition of F γp,θ so that it captures more realistic situations. A CKNOWLEDGMENT

Revised: This study was partially supported by JSPS KAKENHI (18H03201), Japan Digital Design and JST

Revised: CREST.

Original: N OTATION LISTS
Revised: N OTATION LISTS

Original: [Table]
Revised: [Table]

Original: accompanied with an FNN in Ψ( L, W, S, B )
Revised: accompanied with an FNN in Ψ( L, W, S, B )

Original: I ( T, γ ) the set of features contributing a frequency component s with
Revised: I ( T, γ ) the set of features contributing a frequency component s with

Original: [Equation]
Revised: [Equation]

Original: A C ONNECTION TO FINITE DIMENSIONAL INPUT SETTING
Revised: A C ONNECTION TO FINITE DIMENSIONAL INPUT SETTING

Original: We can easily see that the analysis in our paper can be directly applied to a setting where the input is ﬁnite dimensional (say, d -dimensional).
Revised: We can easily see that the analysis in our paper can be directly applied to a setting where the input is ﬁnite dimensional (say, d -dimensional).

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: , and suppose that γ ( s ) : N ∞ → R > 0 ∪ {∞} satisﬁes γ ( s ) < ∞ ( s ∈ J d ) and γ ( s ) =
Revised: , and suppose that γ ( s ) : N ∞ → R > 0 ∪ {∞} satisﬁes γ ( s ) < ∞ ( s ∈ J d ) and γ ( s ) =

Original: ∞ ( s / ∈ J d ) .
Revised: ∞ ( s / ∈ J d ) .

Original: If we set
Revised: If we set

Original: F γp [Equation]
Revised: F γp,θ,d ([0 , 1] ∞ ) :

Original: [Equation] = (cid:26) 
Revised: = (cid:26) f = (cid:88) l

Original: [Equation]
Revised: [Equation]

Original:  (cid:88) [Equation]
Revised: (cid:104) f, ψ l (cid:105) ψ l : (cid:18) (cid:88) s

Original: [Equation]
Revised: [Equation]

Original: ψ
Revised: d

Original: [Equation]
Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Original: <
Revised: <

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( f [Equation] = 0 ( [Equation]{+
+}
Revised: ( f ) = {+
+}

Original:  s [Equation] ∈ J d ) (cid:27) , then by the condition δ s ( f ) = 

Revised: ( ∀ s / ∈ J d ) (cid:27) , then by the condition δ s ( f ) =

Original: ( ∀ s / ∈ J d ) , f ∈ F γ p,θ,d ([0 , 1] ∞ ) is independent of ( x i ) i>d : f ( x 1 , . . .
Revised: ( ∀ s / ∈ J d ) , f ∈ F γ p,θ,d ([0 , 1] ∞ ) is independent of ( x i ) i>d : f ( x [Equation]

Original: , x d , x d +1 , . . .
Revised: , x d , x d +1 , . . .

Original: , x i , . . . )
Revised: , x i , . . . )

Original: =
Revised: =

Original: f ( x 1 , . . .
Revised: f ( x 1 , . . .

Original: , x d , x (cid:48)
Revised: , x d , x (cid:48)

Original: d +1 , . . .
Revised: d +1 , . . .

Original: , x (cid:48) i . . . ) , for any ( x i ) ∞ i =1 ∈ [0 , 1] ∞ and x (cid:48)
Revised: , x (cid:48) i . . . ) , for any ( x i ) ∞ i =1 ∈ [0 , 1] ∞ and x (cid:48)

Original: i ∈ [0 , 1] ( i =
Revised: i ∈ [0 , 1] ( i =

Original: d + 1 , d + 2 , . . . ) .
Revised: d + 1 , d + 2 , . . . ) .

Original: Then, for f ∈ F γp,θ,d ([0 , 1] ∞ ) , we may deﬁne f
Revised: Then, for f ∈ F γp,θ,d ([0 , 1] ∞ ) , we may deﬁne f

Original: d ( x 1 , . . .
Revised: d ( x 1 , . . .

Original: x d ) :
Revised: x d ) :

Original: = f
Revised: = f

Original: ( x 1 , . . .
Revised: ( x 1 , . . .

Original: , x d , 0 , . . . ) , and it holds that (cid:107) f − f (cid:48) (cid:107) 

Revised: , x d , 0 , . . . ) , and it holds that (cid:107) f − f (cid:48) (cid:107)

Original: = (cid:107) f
Revised: = (cid:107) f

Original: d − f (cid:48)
Revised: d − f (cid:48)

Original: d (cid:107) 2 .
Revised: d (cid:107) 2 .

Original: Hence, by the same argument as the proof of Theorem 7, we have the same bound for F γp,θ,d ([0 , 1] ∞ ) .
Revised: Hence, by the same argument as the proof of Theorem 7, we have the same bound for F γp,θ,d ([0 , 1] ∞ ) .

Original: Thus, the same statement as Theorem 7 holds for the γ -smooth functions on [0 , 1] d .
Revised: Thus, the same statement as Theorem 7 holds for the γ -smooth functions on [0 , 1] d .

Original: This shows that the difﬁculty of the approximation depends only on the smoothness γ ( s ) and independent of the dimensionality d .
Revised: This shows that the difﬁculty of the approximation depends only on the smoothness γ ( s ) and independent of the dimensionality d .

Original: In particular, we can establish the bound even for d (cid:29) n .
Revised: In particular, we can establish the bound even for d (cid:29) n .

Original: B A NALYSIS OF ESTIMATION ERROR BY CONVOLUTIONAL NEURAL NETWORK
Revised: B A NALYSIS OF ESTIMATION ERROR BY CONVOLUTIONAL NEURAL NETWORK

Original: In this section, we discuss approximation and estimation errors by CNNs and dilated CNNs.
Revised: In this section, we discuss approximation and estimation errors by CNNs and dilated CNNs.

Original: In
Revised: In

Original: Theorem 7, we considered approximating a function in F γp,θ by FNNs.
Revised: Theorem 7, we considered approximating a function in F γp,θ by FNNs.

Original: According to the analysis, the index required to achieve the derived approximation error bound is determined by I ( T, γ ) , and only the coordinates corresponding to the index set I ( T, γ ) in X should be taken as input.
Revised: According to the analysis, the index required to achieve the derived approximation error bound is determined by I ( T, γ ) , and only the coordinates corresponding to the index set I ( T, γ ) in X should be taken as input.

Original: However, in practice, it is not given which index is required as input, and it should be estimated from the data.
Revised: However, in practice, it is not given which index is required as input, and it should be estimated from the data.

Original: In this section, we show that, under certain conditions, it is possible to ﬁnd the required indices from the data by using CNN and dilated CNN type architectures.
Revised: In this section, we show that, under certain conditions, it is possible to ﬁnd the required indices from the data by using CNN and dilated CNN type architectures.

Original: We also show that these architectures can achieve a favorable approximation and estimation errors for functions in mixed and anisotropic smooth spaces that depends on the smoothness of the function classes.
Revised: We also show that these architectures can achieve a favorable approximation and estimation errors for functions in mixed and anisotropic smooth spaces that depends on the smoothness of the function classes.

Original: C R ELATIONSHIP TO EXISTING FUNCTION SPACES
Revised: C R ELATIONSHIP TO EXISTING FUNCTION SPACES

Original: In this section, we discuss the relationship between the space F γp,θ , its ﬁnite dimensional counter-part and some related function classes.
Revised: In this section, we discuss the relationship between the space F γp,θ , its ﬁnite dimensional counter-part and some related function classes.

Original: C. M IXED SMOOTHNESS
Revised: C. M IXED SMOOTHNESS

Original: First, we introduce a ﬁnite dimensional counter part of or function space with mixed smoothness (Schmeisser, 1987; Sickel & Ullrich, 2009), where the domain of the input is [0 , 1] d .
Revised: First, we introduce a ﬁnite dimensional counter part of or function space with mixed smoothness (Schmeisser, 1987; Sickel & Ullrich, 2009), where the domain of the input is [0 , 1] d .

Original: Deﬁnition 15 (Mixed smooth modulus of smoothness) .
Revised: Deﬁnition 15 (Mixed smooth modulus of smoothness) .

Original: For r ∈ N and h ∈ R > 0 , let
Revised: For r ∈ N and h ∈ R > 0 , let

Original: [Equation]
Revised: [Equation]

Original: ( otherwise ) , be the r -th order discrete differentiation for a function f : [0 , 1] → R .
Revised: ( otherwise ) , be the r -th order discrete differentiation for a function f : [0 , 1] → R .

Original: By applying this discrete differential operator to each coordinate of a subset e ⊂ { 1 , . . .
Revised: By applying this discrete differential operator to each coordinate of a subset e ⊂ { 1 , . . .

Original: , d } , the mixed discrete differential operator for a step length h ∈ R d> 0 and the order parameter r ∈ N d is deﬁned as
Revised: , d } , the mixed discrete differential operator for a step length h ∈ R d> 0 and the order parameter r ∈ N d is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: for x ∈ [0 , 1] d .
Revised: for x ∈ [0 , 1] d .

Original: Then, the modulus of mixed smoothness is deﬁned by
Revised: Then, the modulus of mixed smoothness is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: for t ∈ R d> 0 and 1 ≤ p , where (cid:107) · (cid:107) p is the L p -norm with respect to the Lebesgue measure on [0 , 1] d .
Revised: for t ∈ R d> 0 and 1 ≤ p , where (cid:107) · (cid:107) p is the L p -norm with respect to the Lebesgue measure on [0 , 1] d .

Original: Then, based on this modulus of mixed smoothness, we can deﬁne the mixed smooth Besov space as follows.
Revised: Then, based on this modulus of mixed smoothness, we can deﬁne the mixed smooth Besov space as follows.

Original: Deﬁnition 16 (Mixed smooth Besov space) .
Revised: Deﬁnition 16 (Mixed smooth Besov space) .

Original: Let 1 ≤ p, q ≤ ∞ .
Revised: Let 1 ≤ p, q ≤ ∞ .

Original: For a given smoothness parameter a ∈ R d> 0 , let r i
Revised: For a given smoothness parameter a ∈ R d> 0 , let r i

Original: = (cid:98) a
Revised: = (cid:98) a

Original: i (cid:99) + 1 ( i ∈ [ d ]) and deﬁne the seminorm | · | MB a,ep,q as
Revised: i (cid:99) + 1 ( i ∈ [ d ]) and deﬁne the seminorm | · | MB a,ep,q as

Original: [Equation]
Revised: [Equation]

Original: 0 , 1] d (cid:2)(cid:0)(cid:81) i ∈
Revised: , 1] d (cid:2)(cid:0)(cid:81) i ∈

Original: e t − a
Revised: e t − a

Original: i
Revised: i

Original: i (cid:1)w e r
Revised: i (cid:1)

Original: [Equation] p ( f [Equation] t ) (cid:3) 
Revised: w e r,p ( f, t ) (cid:3) q dt (cid:81)

Original: [Equation]
Revised: i ∈

Original: dt (cid:81) i ∈ e t i (cid:111) 1 /q (1 ≤ q < ∞ ) , sup t ∈ [0 , 1] d (cid:0)(cid:81)
Revised: e t i (cid:111) 1 /q (1 ≤ q < ∞ ) , sup t ∈ [0 , 1] d (cid:0)(cid:81)

Original: i ∈
Revised: i ∈

Original: e t − a
Revised: e t − a

Original: i
Revised: i

Original: i (cid:1) w er,p ( f, t ) ( q = ∞ ) .
Revised: i (cid:1) w er,p ( f, t ) ( q = ∞ ) .

Original: Then we deﬁne the norm of the mixed smooth Besov space as
Revised: Then we deﬁne the norm of the mixed smooth Besov space as

Original: [Equation]
Revised: [Equation]

Original: so that the mixed smooth Besov space is given by MB αp,q ([0 , 1] d ) :
Revised: so that the mixed smooth Besov space is given by MB αp,q ([0 , 1] d ) :

Original: = { f ∈ L p ([0 , 1] d ) : (cid:107) f (cid:107) MB ap,q < ∞} .
Revised: = { f ∈ L p ([0 , 1] d ) : (cid:107) f (cid:107) MB ap,q < ∞} .

Original: The mixed smooth Besov space deﬁned above can be seen as the ﬁnite dimensional counter part of our space with mixed smoothness γ .
Revised: The mixed smooth Besov space deﬁned above can be seen as the ﬁnite dimensional counter part of our space with mixed smoothness γ .

Original: Here, for s ∈ N d , let
Revised: Here, for s ∈ N d , let

Original: [Equation]
Revised: [Equation]

Original: s
Revised: f

Original: [Equation]
Revised: [Equation]

Original: where ψ l : [0 , 1] d → R is deﬁned in the same way as in the inﬁnite dimensional setting and f ∈
Revised: where ψ l : [0 , 1] d → R is deﬁned in the same way as in the inﬁnite dimensional setting and f ∈

Original: L 2 ([0 , 1] d ) .
Revised: L 2 ([0 , 1] d ) .

Original: Then, it is known that
Revised: Then, it is known that

Original: [Equation]
Revised: [Equation]

Original: holds for 1 < p < ∞ (see Section 3.3 of Dung et al.
Revised: holds for 1 < p < ∞ (see Section 3.3 of Dung et al.

Original: (2018), Yanchenko (2020);
Revised: (2018), Yanchenko (2020);

Original: Lizorkin &
Revised: Lizorkin &

Original: Nikol’skii (1990) and references therein).
Revised: Nikol’skii (1990) and references therein).

Original: Therefore, the mixed frequency space can be viewed as an extension of the ﬁnite-dimensional mixed smooth Besov space to the inﬁnite dimensional one.
Revised: Therefore, the mixed frequency space can be viewed as an extension of the ﬁnite-dimensional mixed smooth Besov space to the inﬁnite dimensional one.

Original: Approximation and estimation abilities of deep learning with ReLU activation for the true function in the ﬁnite-dimensional mixed smooth Besov space was investigated by Suzuki (2019) when a 1 = · · · = a d .
Revised: Approximation and estimation abilities of deep learning with ReLU activation for the true function in the ﬁnite-dimensional mixed smooth Besov space was investigated by Suzuki (2019) when a 1 = · · · = a d .

Original: They showed that deep learning can achieve the (near) minimax optimal rate in this setting and the mixed smoothness alleviates the curse of dimensionality.
Revised: They showed that deep learning can achieve the (near) minimax optimal rate in this setting and the mixed smoothness alleviates the curse of dimensionality.

Original: C. A NISOTROPIC SMOOTHNESS
Revised: C. A NISOTROPIC SMOOTHNESS

Original: Ingster & Stepanova (2006) considered a general function class representated by
Revised: Ingster & Stepanova (2006) considered a general function class representated by

Original: [Equation]
Revised: [Equation]

Original:   for a given sequence c = ( c l ) l ∈ Z ∞ 0 with c l ∈ R .
Revised:   for a given sequence c = ( c l ) l ∈ Z ∞ 0 with c l ∈ R .

Original: This class includes our γ -smooth space with p = θ = 2 .
Revised: This class includes our γ -smooth space with p = θ = 2 .

Original: In particular, Ingster & Stepanova (2011) analyzed an estimation problem when ( c l ) l ∈ Z ∞ 0 is given by
Revised: In particular, Ingster & Stepanova (2011) analyzed an estimation problem when ( c l ) l ∈ Z ∞ 0 is given by

Original: [Equation]
Revised: [Equation]

Original: for monotonically increasing sequence a = ( a i ) ∞
Revised: for monotonically increasing sequence a = ( a i ) ∞

Original: i =1 ∈ R ∞ > 0 .
Revised: i =1 ∈ R ∞ > 0 .

Original: It can be shown that, in this setting, F c is speciﬁcally given by the following anisotropic Sobolev classe:
Revised: It can be shown that, in this setting, F c is speciﬁcally given by the following anisotropic Sobolev classe:

Original: [Equation]
Revised: [Equation]

Original: This characterization highlights the intuition of the anisotropic smoothness, i.e., we assume the functions f ∈ F c have different smoothness ( a i ) with respect to each coordinate x i .
Revised: This characterization highlights the intuition of the anisotropic smoothness, i.e., we assume the functions f ∈ F c have different smoothness ( a i ) with respect to each coordinate x i .

Original: Here, by noticing that (cid:107) δ s ( f ) (cid:107) 

Revised: Here, by noticing that (cid:107) δ s ( f ) (cid:107)

Original: = (cid:80)
Revised: = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: | l i | < 2 si (cid:104) f, φ l (cid:105) 2 , we have that (cid:107)
Revised: (cid:99)≤| l i | < 2 si (cid:104) f, φ l (cid:105) 2 , we have that (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: for γ ( s ) =
Revised: for γ ( s ) =

Original: max i { a i s i } .
Revised: max i { a i s i } .

Original: Therefore, we see that the unit ball of the anisotropic Sobolev space W ais included in the unit ball of F γ 2 , 2 .
Revised: Therefore, we see that the unit ball of the anisotropic Sobolev space W ais included in the unit ball of F γ 2 , 2 .

Original: Hence, our bounds in the following also give bounds for the anisotropic Sobolev space.
Revised: Hence, our bounds in the following also give bounds for the anisotropic Sobolev space.

Original: Suzuki & Nitanda (2021) also analyzed approximation and estimation error bounds of deep learning with the ReLU activation for the anisotropic Besov spaces deﬁned on a ﬁnite dimensional space [0 , 1] d (Nikol’skii, 1975; Vybiral, 2006; Triebel, 2011).
Revised: Suzuki & Nitanda (2021) also analyzed approximation and estimation error bounds of deep learning with the ReLU activation for the anisotropic Besov spaces deﬁned on a ﬁnite dimensional space [0 , 1] d (Nikol’skii, 1975; Vybiral, 2006; Triebel, 2011).

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: We can see that δ s can be decomposed as
Revised: We can see that δ s can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: for c k ∈ C and the imaginary number i .
Revised: for c k ∈ C and the imaginary number i .

Original: Thus, when 1 ≤ p ≤ 2 , from Theorem 1 of Nessel &
Revised: Thus, when 1 ≤ p ≤ 2 , from Theorem 1 of Nessel &

Original: Wilmes (1978), we have that
Revised: Wilmes (1978), we have that

Original: [Equation]
Revised: [Equation]

Original: where v =
Revised: where v =

Original: ( 1 p − 12 ) + and 2 vs = 2 v (cid:80) ∞ i =
Revised: ( 1 p − 12 ) + and 2 vs = 2 v (cid:80) ∞ i =

Original: 1 s i .
Revised: 1 s i .

Original: Furthermore, for 2 < p , from the Cauchy-Schwarz inequality, we obtain
Revised: Furthermore, for 2 < p , from the Cauchy-Schwarz inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore, Eq.
Revised: Therefore, Eq.

Original: (4) holds all over the range of 1 ≤ p < ∞ .
Revised: (4) holds all over the range of 1 ≤ p < ∞ .

Original: To show the approximation error in the assertion of the theorem, we consider approximating f by
Revised: To show the approximation error in the assertion of the theorem, we consider approximating f by

Original: R T ( f ) deﬁned as
Revised: R T ( f ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Then, we further approximate R T ( f ) by a fully connected neural network.
Revised: Then, we further approximate R T ( f ) by a fully connected neural network.

Original: For that purpose, we ﬁrst analyze the approximation error (cid:107)
Revised: For that purpose, we ﬁrst analyze the approximation error (cid:107)

Original: f − R T ( f ) (cid:107) 2 by R T ( f ) .
Revised: f − R T ( f ) (cid:107) 2 by R T ( f ) .

Original: This can be evaluated in the following lemma.
Revised: This can be evaluated in the following lemma.

Original: Lemma 17.
Revised: Lemma 17.

Original: Under the same setting as Theorem [Equation] we have that
Revised: Under the same setting as Theorem 7, we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We show the inequality for the settings 1 ≤ θ ≤ 2 and 2 < θ separately.
Revised: We show the inequality for the settings 1 ≤ θ ≤ 2 and 2 < θ separately.

Original: Approximation error (cid:107) f 
Revised: Approximation error (cid:107) f − R T

Original: [Equation]
Revised: [Equation]

Original:  2 by R T ( f ) for 1 ≤ θ [Equation]
Revised: f ) (cid:107) 2 by R T ( f ) for 1 ≤ θ ≤ 2 :

Original: Using the orthogonality of δ s between different s
Revised: Using the orthogonality of δ s between different s

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where Eq.
Revised: where Eq.

Original: (4) is used to show Eq.
Revised: (4) is used to show Eq.

Original: Then, for s ∈ N ∞ 0 with T ≤ γ ( s ) , using the assumption v [Equation]
Revised: Then, for s ∈ N ∞ 0 with T ≤ γ ( s ) , using the assumption v (cid:80) ∞ i =

Original:  ≤ vα ( γ ) < 1 , we have that
Revised: 1 s i γ ( s ) ≤ vα ( γ ) < 1 , we have that

Original: [Equation]
Revised: [Equation]

Original: Then, applying this inequality to Eq.
Revised: Then, applying this inequality to Eq.

Revised: (6), we obtain

Original: [Equation]
Revised: [Equation]

Revised: 2. Approximation error (cid:107)

Original: we obtain
Revised: f − R

Original: [Equation]
Revised: [Equation]

Original: 2. Approximation error (cid:107) f− R T ( f ) (cid:107) 2 by R T ( f ) for 2 < θ :
Revised: f

Original: Using again the orthogonality of δ s ( f ) between different s

Original: [Equation]
Revised: [Equation]

Original:  we obtain
Revised: Using again the orthogonality of δ s ( f ) between different s , we obtain

Original: [Equation]
Revised: [Equation]

Original: and (cid:88)
Revised: and (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Then, using the assumption vα (cid:48) < 1 for α (cid:48)
Revised: Then, using the assumption vα (cid:48) < 1 for α (cid:48)

Original: =
Revised: =

Original: α ( γ (cid:48) ) , we obtain 2 vs − γ (cid:48) ( s ) ≤ 2 ( vα (cid:48) − 1) γ (cid:48) ( s ) ≤ 2 ( vα (cid:48) − 1) T .
Revised: α ( γ (cid:48) ) , we obtain 2 vs − γ (cid:48) ( s ) ≤ 2 ( vα (cid:48) − 1) γ (cid:48) ( s ) ≤ 2 ( vα (cid:48) − 1) T .

Original: Applying this inequality to Eq.
Revised: Applying this inequality to Eq.

Original: (7), we obtain
Revised: (7), we obtain

Original: [Equation]
Revised: [Equation]

Original: Then, by using Cauchy-Schwarz inequality, we obtain
Revised: Then, by using Cauchy-Schwarz inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original:   
Revised:   (1 −

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: ≤ 2 − [Equation] 1 − vα [Equation]
Revised: 

Original: T 

Original: [Equation]
Revised: [Equation]

Original: θ [Equation] θ . Next, consider approximating R T by a neural network.
Revised: Next, consider approximating R T by a neural network.

Original: We show it only for 1 ≤ θ ≤ 2 , but the same argument can be applied for 2 < θ .
Revised: We show it only for 1 ≤ θ ≤ 2 , but the same argument can be applied for 2 < θ .

Original: Theorem 4.1 in Perekrestenko et al.
Revised: Theorem 4.1 in Perekrestenko et al.

Original: (2018) asserts that there exist constants C 1 , C 2 > 0 such that, for[Equation] ˜ [Equation]
Revised: (2018) asserts that there exist constants C 1 , C 2 > 0 such that, for

Original: C

Original: [Equation]
Revised: [Equation]

Original: ) (cid:35) , there exists a neural network ˜ ψ l
Revised: , there exists a neural network ˜ ψ l

Original: i ∈
Revised: i ∈

Original: Φ( L ˜ ψ , 21 , C 2 , 21 2 L ˜ ψ ) that can approximate ψ l i as
Revised: Φ( L ˜ ψ , 21 , C 2 , 21 2 L ˜ ψ ) that can approximate ψ l i as

Original: [Equation]
Revised: [Equation]

Original: Moreover, since ψ l i ∈ [ −√ 2 , √ 2] ,
Revised: Moreover, since ψ l i ∈ [ −√ 2 , √ 2] ,

Original: [Equation]
Revised: [Equation]

Original: Since, we can write
Revised: Since, we can write

Original: [Equation]
Revised: [Equation]

Original: for x ∈ R , it holds that max {−√ 2 min {√ 2 , x }} ∈ Φ(2 , 4 , √ 2 , 16) and then we can see that
Revised: for x ∈ R , it holds that max {−√ 2 min {√ 2 , x }} ∈ Φ(2 , 4 , √ 2 , 16) and then we can see that

Original: [Equation]
Revised: [Equation]

Original: By using these facts
Revised: By using these facts

Original: [Equation]
Revised: [Equation]

Original: we can construct the neural network that takes value in [ −√ 2 , √ 2] .
Revised: we can construct the neural network that takes value in [ −√ 2 , √ 2] .

Original: In the following, let
Revised: In the following, let

Original: [Equation]
Revised: [Equation]

Original: Since we need to approximate a trigonometric polynomial
Revised: Since we need to approximate a trigonometric polynomial

Original: [Equation]
Revised: [Equation]

Original: we need to (approximately) realize multiplications by neural networks.
Revised: we need to (approximately) realize multiplications by neural networks.

Original: By Proposition 3 of Yarotsky (2017), using
Revised: By Proposition 3 of Yarotsky (2017), using

Original: [Equation]
Revised: [Equation]

Original: and a constant B × > 0 , there exists a neural network φ × ∈
Revised: and a constant B × > 0 , there exists a neural network φ × ∈

Original: Φ( L × , W × , B × , S × ) that satisﬁes
Revised: Φ( L × , W × , B × , S × ) that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: L
Revised: φ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: − d max (cid:89) i = x i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: L ∞ ([ − 1 , 1]

Original: [Equation]
Revised: [Equation]

Original:  ≤ (cid:15).
Revised: max ) ≤ (cid:15).

Original: Since ˆ ψ l i ≤ √ 2 is satisﬁed, we know that 
Revised: Since ˆ ψ l i ≤ √ 2 is satisﬁed, we know that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Here, note that Eq.
Revised: [Equation] ⇒ [Equation] [Equation]

Original: (8) yields that
Revised: − [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original:  

Original: [Equation][Equation] [Equation] ) =
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 
Revised: (cid:15). Here, note that Eq. (8) yields that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) d max

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: =

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation]
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Then, by using triangle inequality and Eq. (9), we know that (cid:13)(cid:13)(cid:13)
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( √
Revised: ψ

Original: [Equation]
Revised: [Equation]

Original: − [Equation]
Revised: ∞ [Equation]

Original: (cid:13)
Revised: = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) d

Original: [Equation]
Revised: [Equation]

Revised: j =

Original: [Equation]
Revised: [Equation]

Revised: Then, by using triangle inequality and Eq.

Original: [Equation]
Revised: [Equation]

Original: ) ≤(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Table]
Revised: we know that [Equation] ≤

Revised: [Equation]

Revised: [Table]

Original: Therefore, if we set
Revised: Therefore, if we set

Original: [Equation]
Revised: [Equation]

Original: where J ( s ) :
Revised: where J ( s ) :

Original: = { l ∈ Z ∞ 0 : (cid:98) 

Revised: = { l ∈ Z ∞ 0 : (cid:98)

Original: s i − 1 (cid:99) ≤ | l i | <
Revised: s i − 1 (cid:99) ≤ | l i | <

Original: 2 s i } , we obtain that
Revised: 2 s i } , we obtain that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact (cid:104) f, ψ l (cid:105) ≤ (cid:107) f (cid:107) 2 ≤ B f in the last inequality.
Revised: where we used the fact (cid:104) f, ψ l (cid:105) ≤ (cid:107) f (cid:107) 2 ≤ B f in the last inequality.

Original: Hence, if we put
Revised: Hence, if we put

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: then we have that
Revised: then we have that

Original: [Equation]
Revised: [Equation]

Original: where Q ( T ) =
Revised: where Q ( T ) =

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: ( θ > 2) .
Revised: ( θ > 2) .

Original: Noting that 2 − T (cid:46) Q ( T ) , we have that (cid:107) f − ˆ R T (cid:107) 2 (cid:46) Q ( T ) (cid:107) f (cid:107) F
Revised: Noting that 2 − T (cid:46) Q ( T ) , we have that (cid:107) f − ˆ R T (cid:107) 2 (cid:46) Q ( T ) (cid:107) f (cid:107) F

Original: γp,θ .
Revised: γp ,θ

Revised: [Equation]

Original: Finally, we evaluate the network size to achieve this approximation error.
Revised: Finally, we evaluate the network size to achieve this approximation error.

Original: Since ˆ R T is the linear combination of neural network φ ×
Revised: Since ˆ R T is the linear combination of neural network φ ×

Original: [Equation]
Revised: [Equation]

Original: , by putting
Revised: , by putting

Original: [Equation]
Revised: [Equation]

Original: we have that ˆ R T ∈
Revised: we have that ˆ R T ∈

Original: Φ( L, W, S, B ) .
Revised: Φ( L, W, S, B ) .

Original: Substituting Eq.
Revised: Substituting Eq.

Original: (10) to L × and L ˆ φ , we can evaluate as
Revised: (10) to L × and L ˆ φ , we can evaluate as

Original: L ˆ ψ = C(cid:38)(cid:18)
Revised: L ˆ ψ = C(cid:38)(cid:18)

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:108) C 1 (6 max { log B f , log 2 } ) 2 (cid:109) (cid:108)(cid:16) max { d 2max , T 2 , (log S ( γ, T )) 2 , log f max , 2 } (cid:17)(cid:109) and
Revised: ≤ (cid:108) C 1 (6 max { log B f , log 2 } ) 2 (cid:109) (cid:108)(cid:16) max { d 2max , T 2 , (log S ( γ, T )) 2 , log f max , 2 } (cid:17)(cid:109) and

Original: L × = (cid:24) log (cid:18) 3 d max (cid:15) +(cid:19)(cid:25) (cid:100) log d max (cid:101) ≤ (cid:24) max (cid:26) log (cid:18) 3 d max (cid:15) (cid:19) , log(cid:27) + log(cid:25) (cid:100) log d max (cid:101) ≤ (cid:100) 6 max { log B f , log 5 }(cid:101) (cid:100) max { d max , T, log G ( T, γ ) }(cid:101) (cid:100) log d max (cid:101) .
Revised: L × = (cid:24) log (cid:18) 3 d max (cid:15) +(cid:19)(cid:25) (cid:100) log d max (cid:101) ≤ (cid:24) max (cid:26) log (cid:18) 3 d max (cid:15) (cid:19) , log(cid:27) + log(cid:25) (cid:100) log d max (cid:101) ≤ (cid:100) 6 max { log B f , log 5 }(cid:101) (cid:100) max { d max , T, log G ( T, γ ) }(cid:101) (cid:100) log d max (cid:101) .

Original: Therefore, if we set K = 2 max (cid:110) (cid:100) 6 max { log B f , log 5 }(cid:101) , (cid:108) C 1 (6 max { log B f , log 2 } ) 2 (cid:109)(cid:111) , then it holds that
Revised: Therefore, if we set K = 2 max (cid:110) (cid:100) 6 max { log B f , log 5 }(cid:101) , (cid:108) C 1 (6 max { log B f , log 2 } ) 2 (cid:109)(cid:111) , then it holds that

Original: L ≤ 2 K max (cid:8) d 2 max , T 2 , (log G ( T, γ )) 2 , log f max (cid:9) , and thus we also have
Revised: L ≤ 2 K max (cid:8) d 2 max , T 2 , (log G ( T, γ )) 2 , log f max (cid:9) , and thus we also have

Original: [Equation]
Revised: [Equation]

Original: For the setting of 2 < θ , we can prove the bound by the same procedure.
Revised: For the setting of 2 < θ , we can prove the bound by the same procedure.

Original: Thus, we obtain the assertion.
Revised: Thus, we obtain the assertion.

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Before we prove Theorem
Revised: Before we prove Theorem

Original: [Equation]
Revised: [Equation]

Original: we show the following lemma.
Revised: we show the following lemma.

Original: The following lemma is inspired by
Revised: The following lemma is inspired by

Original: Lemma 1.2 of Temlyakov (1983).
Revised: Lemma 1.2 of Temlyakov (1983).

Original: Lemma 18.
Revised: Lemma 18.

Original: Suppose that a = ( a i ) ∞
Revised: Suppose that a = ( a i ) ∞

Original: i =1 and a (cid:48)
Revised: i =1 and a (cid:48)

Original: = ( a (cid:48) i ) ∞
Revised: = ( a (cid:48) i ) ∞

Original: i =1 are positive monotonically non-decreasing sequences such that 1 ≤ a 1 = a (cid:48) 1 and
Revised: i =1 are positive monotonically non-decreasing sequences such that 1 ≤ a 1 = a (cid:48) 1 and

Original: [Equation]
Revised: [Equation]

Original: for a positive constant β >
Revised: for a positive constant β >

Original: [Equation]
Revised: [Equation]

Original: Then we have the following inequalities:
Revised: Then we have the following inequalities:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, note that
Revised: First, note that

Original: [Equation]
Revised: [Equation]

Original: If T satisﬁes (cid:80) ∞ i =2 a (cid:48)
Revised: If T satisﬁes (cid:80) ∞ i =2 a (cid:48)

Original: i s i < T
Revised: i s i < T

Original: [Equation]
Revised: [Equation]

Original: then we have that
Revised: then we have that

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: the second term of the right hand side of Eq.
Revised: the second term of the right hand side of Eq.

Original: (23) can be evaluated as
Revised: (23) can be evaluated as

Original: [Equation]
Revised: [Equation]

Original: Next, we evaluate the ﬁrst term of the right hand side of Eq.
Revised: Next, we evaluate the ﬁrst term of the right hand side of Eq.

Original: We see that
Revised: We see that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus the ﬁrst term of the right hand side of Eq.
Revised: Thus the ﬁrst term of the right hand side of Eq.

Original: (23) can be bounded by
Revised: (23) can be bounded by

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: by noticing that
Revised: by noticing that

Original: [Equation]
Revised: [Equation]

Original: and combining the evaluations above
Revised: and combining the evaluations above

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (23) yields that
Revised: (23) yields that

Original: [Equation]
Revised: [Equation]

Original: which yields the ﬁrst inequality.
Revised: which yields the ﬁrst inequality.

Original: Next, we show the second inequality.
Revised: Next, we show the second inequality.

Original: We decompose (cid:80) s ∈ N ∞
Revised: We decompose (cid:80) s ∈ N ∞

Original: [Equation]
Revised: [Equation]

Revised: s

Revised: [Equation]

Original: <
Revised: <

Original: [Equation]
Revised: [Equation]

Original: 2 s as (cid:80) Tt =

Revised: 2 s as (cid:80) Tt =

Original: I t where I t is deﬁned as
Revised: I t where I t is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Here, for 1 ≤ t ≤ T , we can evaluate I t as
Revised: Here, for 1 ≤ t ≤ T , we can evaluate I t as

Original: I [Equation] t
Revised: I t ≤ 2 2 t

Original: [Equation]
Revised: [Equation]

Original: Since a i ≥ 1 , we have a i ≤ 2 a i − 1 ( i ≥ 1) and thus by Eq.
Revised: Since a i ≥ 1 , we have a i ≤ 2 a i − 1 ( i ≥ 1) and thus by Eq.

Original: (21) with a (cid:48)
Revised: (21) with a (cid:48)

Original: i = a 1 ( ∀ i ∈ N ) and β = 1 , we have (cid:88)
Revised: i = a 1 ( ∀ i ∈ N ) and β = 1 , we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where we used the ﬁrst assertion (21) in the last inequality.
Revised: where we used the ﬁrst assertion (21) in the last inequality.

Original: Therefore [Equation] we can obtain
Revised: Therefore, we can obtain

Original: [Equation]
Revised: [Equation]

Original: Now using this bound [Equation] we can verify that
Revised: Now using this bound, we can verify that

Original: [Equation]
Revised: [Equation]

Original: which yields the second assertion (22)
Revised: which yields the second assertion (22).

Original: [Equation]

Original: Now, we are ready to show Theorem 9.
Revised: Now, we are ready to show Theorem 9.

Original: First, we give the proof in the setting of γ ( s ) =
Revised: First, we give the proof in the setting of γ ( s ) =

Original: (cid:104) a, s (cid:105) .
Revised: (cid:104) a, s (cid:105) .

Original: Proof for mixed smoothness ( γ ( s ) =
Revised: Proof for mixed smoothness ( γ ( s ) =

Original: (cid:104) a, s (cid:105) ).
Revised: (cid:104) a, s (cid:105) ).

Original: First, we consider the setting 1 ≤ θ ≤ 2 .
Revised: First, we consider the setting 1 ≤ θ ≤ 2 .

Original: Lemma 18 yields
Revised: Lemma 18 yields

Original: [Equation]
Revised: [Equation]

Original: Moreover, we can easily see that
Revised: Moreover, we can easily see that

Original: [Equation]
Revised: [Equation]

Original: because a is a positive monotonically non-decreasing sequence.
Revised: because a is a positive monotonically non-decreasing sequence.

Original: By the assumption a i = Ω( i q ) , 
Revised: By the assumption a i = Ω( i q ) , d

Original: [Equation]
Revised: [Equation]

Original: are satisﬁed.
Revised: are satisﬁed.

Original: Now, by using the ﬁlter w ∈ R C × 1 × W (cid:48) with the width W (cid:48)
Revised: Now, by using the ﬁlter w ∈ R C × 1 × W (cid:48) with the width W (cid:48)

Original: =
Revised: =

Original: d max , the number of output channels
Revised: d max , the number of output channels

Original: C =
Revised: C =

Original: d max and the number of input channels C (cid:48)
Revised: d max and the number of input channels C (cid:48)

Original: = 1 given by w i, 1 ,j =
Revised: = 1 given by [Equation] i, 1 ,

Original: [Equation]
Revised: [Equation]

Original:  1 ( i= j )
Revised: = [Equation] 1 ( i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: j )

Original: [Equation]
Revised: [Equation]

Original: = j ) , for i, j ∈ [ d max ] , we can see that
Revised: j ) , for i, j ∈ [ d max ] , we can see that

Original: [Equation]
Revised: [Equation]

Original: By Theorem 7, if we set
Revised: By Theorem 7, if we set

Original: [Equation]
Revised: [Equation]

Original: where K, K (cid:48) > 0 are constants, for any function
Revised: where K, K (cid:48) > 0 are constants, for any function

Original: f ∈ U ( F (cid:104) a,s (cid:105) p,θ ) , there exists a neural network
Revised: f ∈ U ( F (cid:104) a,s (cid:105) p,θ ) , there exists a neural network

Original: ˆ R T ∈
Revised: ˆ R T ∈

Original: Φ( L, W, S, B ) such that
Revised: Φ( L, W, S, B ) such that

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Since f (cid:48)
Revised: Since f (cid:48)

Original: ( X ) =
Revised: ( X ) =

Original: (cid:16)
Revised: (cid:16)

Original: ˆ R T ◦ Conv 1 ,w ( X ) (cid:17) 1 , we can see that f (cid:48) is a dilated CNN: f (cid:48) ∈
Revised: ˆ R T ◦ Conv 1 ,w ( X ) (cid:17) 1 , we can see that f (cid:48) is a dilated CNN: f (cid:48) ∈

Original: P ( L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, B, S )
Revised: P ( L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, B, S )

Original: where L (cid:48)
Revised: where L (cid:48)

Original: = 1 , B (cid:48)
Revised: = 1 , B (cid:48)

Original: = 1 and W (cid:48)
Revised: = 1 and W (cid:48)

Original: =
Revised: =

Original: C = d max .
Revised: C = d max .

Original: Next, we consider the setting 2 < θ .
Revised: Next, we consider the setting 2 < θ .

Original: Let a (cid:48) 

Revised: Let a (cid:48)

Original: = a 1 2 , and for δ = a 2 − a 1 (which is positive by the assumption a 2 > a 1 ) and a constant u that satisﬁes 2 < u < 2+ 2 δ a 1 , we set a (cid:48)
Revised: = a 1 2 , and for δ = a 2 − a 1 (which is positive by the assumption a 2 > a 1 ) and a constant u that satisﬁes 2 < u < 2+ 2 δ a 1 , we set a (cid:48)

Original: i = a i u for i ≥ 1 .
Revised: i = a i u for i ≥ 1 .

Original: Then, a (cid:48) 1 < a (cid:48) 2 ≤ . . .
Revised: Then, a (cid:48) 1 < a (cid:48) 2 ≤ . . .

Original: is satisﬁed, that is, a (cid:48)
Revised: is satisﬁed, that is, a (cid:48)

Original: = ( a (cid:48) i ) ∞
Revised: = ( a (cid:48) i ) ∞

Original: i =1 is a positive monotonically increasing sequence.
Revised: i =1 is a positive monotonically increasing sequence.

Original: Moreover, by the assumption a i = Ω( i q ) , it holds that, for all c > 0 ,
Revised: Moreover, by the assumption a i = Ω( i q ) , it holds that, for all c > 0 ,

Original: [Equation]
Revised: [Equation]

Original: By noticing that
Revised: By noticing that

Original: [Equation]
Revised: [Equation]

Original: a 
Revised: a (cid:48)

Original: [Equation]
Revised: [Equation]

Original: 1 where we used u > 2 , we also have that
Revised: 1 where we used u > 2 , we also have that

Original: [Equation]
Revised: [Equation]

Original: Moreover, since we have veriﬁed Eq.
Revised: Moreover, since we have veriﬁed Eq.

Original: (25), we can apply Lemma 18 to the right hand side of this inequality and it can be further bounded as
Revised: (25), we can apply Lemma 18 to the right hand side of this inequality and it can be further bounded as

Original: [Equation]
Revised: [Equation]

Original: By setting γ (cid:48) ( s ) = (cid:104) a (cid:48) , s (cid:105) and using Theorem 7, we can see that if we deﬁne the set of CNN P by the same argument as the case of 1 ≤ θ ≤ 2 , for all f ∈ U ( F γp,θ ) , there exists a dilated CNN f (cid:48) ∈ P such that
Revised: By setting γ (cid:48) ( s ) = (cid:104) a (cid:48) , s (cid:105) and using Theorem 7, we can see that if we deﬁne the set of CNN P by the same argument as the case of 1 ≤ θ ≤ 2 , for all f ∈ U ( F γp,θ ) , there exists a dilated CNN f (cid:48) ∈ P such that

Original: [Equation]
Revised: [Equation]

Original: is satisﬁed, where we used α ( γ (cid:48) )
Revised: is satisﬁed, where we used α ( γ (cid:48) )

Original: = 2 a 1 by the deﬁnition of a (cid:48) .
Revised: = 2 a 1 by the deﬁnition of a (cid:48) .

Original: Here, as in Eq.
Revised: Here, as in Eq.

Original: (24), we have that
Revised: (24), we have that

Original: [Equation]
Revised: [Equation]

Original: T, γ
Revised: T, γ

Original: [Equation]
Revised: [Equation]

Original: Therefore, by resetting T ← 2 T , we obtain the same result as in 1 ≤ θ ≤ 2 .
Revised: Therefore, by resetting T ← 2 T , we obtain the same result as in 1 ≤ θ ≤ 2 .

Original: Proof for anisotropic smoothness ( γ ( s ) =
Revised: Proof for anisotropic smoothness ( γ ( s ) =

Original: max i { a i s i } i ).
Revised: max i { a i s i } i ).

Original: Here, we consider the setting of mixed smoothness γ ( s ) =
Revised: Here, we consider the setting of mixed smoothness γ ( s ) =

Original: max i { a i s i } i .
Revised: max i { a i s i } i .

Original: We note that
Revised: We note that

Original: [Equation]
Revised: [Equation]

Original: The condition sup i { a i s i } ≤ T is equivalent to the condition that s i ≤
Revised: The condition sup i { a i s i } ≤ T is equivalent to the condition that s i ≤

Original: Ta i for all i ∈ N , the right hand side can be further bounded as
Revised: Ta i for all i ∈ N , the right hand side can be further bounded as

Original: [Equation]
Revised: [Equation]

Original: Therefore, in the setting of 1 ≤ θ ≤ 2 , Theorem 7 with α = (cid:80) ∞
Revised: Therefore, in the setting of 1 ≤ θ ≤ 2 , Theorem 7 with α = (cid:80) ∞

Original: i
Revised: i

Original: =1 1 a i yields that
Revised: =1 1 a i yields that

Original: [Equation]
Revised: [Equation]

Original: Since it holds that
Revised: Since it holds that

Original: [Equation]
Revised: [Equation]

Original: (cid:100)

Original: Tai [Equation] [Equation]  ≤
Revised: 

Original: [Equation]
Revised: [Equation]

Original: G ( T, γ ) (cid:46) 2 (cid:80) ∞
Revised: G ( T, γ ) (cid:46) 2 (cid:80) ∞

Original: i =1 (cid:100)
Revised: i =1 (cid:100)

Original: Tai (cid:101) is satisﬁed.
Revised: Tai (cid:101) is satisﬁed.

Original: By noticing that a is a positive monotonically increasing sequence with polynomial order growth, the same argument as in the setting of γ ( s ) =
Revised: By noticing that a is a positive monotonically increasing sequence with polynomial order growth, the same argument as in the setting of γ ( s ) =

Original: (cid:104) a, s (cid:105) can be applied.
Revised: (cid:104) a, s (cid:105) can be applied.

Original: Then, we obtain the assertion.
Revised: Then, we obtain the assertion.

Original: F P ROOF OF T HEOREM
Revised: F P ROOF OF T HEOREM

Original: The estimation error can be derived by evaluating the bias and variance trade-off.
Revised: The estimation error can be derived by evaluating the bias and variance trade-off.

Original: The bias can be evaluated by the approximation error which has been analyzed in the previous sections, and the variance can be characterized by the complexity of the model.
Revised: The bias can be evaluated by the approximation error which has been analyzed in the previous sections, and the variance can be characterized by the complexity of the model.

Original: As a measure of complexity of the model, we utilize the covering number of the model (van der Vaart & Wellner, 1996).
Revised: As a measure of complexity of the model, we utilize the covering number of the model (van der Vaart & Wellner, 1996).

Original: Deﬁnition 19 (Covering number) .
Revised: Deﬁnition 19 (Covering number) .

Original: For a norm space F equipped with a norm (cid:107) · (cid:107) , the (cid:15) -covering number N ( F , (cid:15), (cid:107) · (cid:107) ) the minimum number of balls with radius (cid:15) (measured by the norm (cid:107) · (cid:107) ) to cover the norm space F :
Revised: For a norm space F equipped with a norm (cid:107) · (cid:107) , the (cid:15) -covering number N ( F , (cid:15), (cid:107) · (cid:107) ) the minimum number of balls with radius (cid:15) (measured by the norm (cid:107) · (cid:107) ) to cover the norm space F :

Original: [Equation]
Revised: [Equation]

Original: The covering number of the model of the dilated CNNs can be evaluated as in the following lemma.
Revised: The covering number of the model of the dilated CNNs can be evaluated as in the following lemma.

Original: Lemma 20.
Revised: Lemma 20.

Original: The log-covering number of the set of the dilated CNNs P ( L
Revised: The log-covering number of the set of the dilated CNNs P ( L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , W(cid:48)
Revised: , W

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]

Original: B

Original: [Equation]
Revised: [Equation]

Original:  ) can be bounded as
Revised: S ) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The assertion can be shown by evaluating how strongly a small perturbation of parameters can deviate the function.
Revised: The assertion can be shown by evaluating how strongly a small perturbation of parameters can deviate the function.

Original: For w ∈ R C (cid:48)
Revised: For w ∈ R C (cid:48)

Original: × W (cid:48) and h ∈ N, we have that
Revised: × W (cid:48) and h ∈ N

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Therefore, for F
Revised: Therefore [Equation] for F

Original: [Equation]
Revised: [Equation]

Original: F (cid:48) ∈ R C (cid:48)
Revised: F (cid:48) ∈ R C (cid:48)

Original: × C (cid:48)
Revised: × C (cid:48)

Original: × W (cid:48)
Revised: × W (cid:48)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Conv

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]

Original: [Equation]

Original: Here, for CNNs f
Revised: Here, for CNNs f

Original: [Equation]
Revised: [Equation]

Original: g such that
Revised: g such that

Original: [Equation]
Revised: [Equation]

Original: we deﬁne
Revised: we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where A 1 ( f )( X ) =
Revised: where A 1 ( f )( X ) =

Original: B L (cid:48)
Revised: B L (cid:48)

Original: ( f )( X ) =
Revised: ( f )( X ) =

Original: X .
Revised: X .

Original: By using these notation and the triangle inequality
Revised: By using these notation and the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: we can see that it holds that
Revised: we can see that it holds that

Original: [Equation]
Revised: [Equation]

Original: Hence, by applying the inequalities (27) and (26)
Revised: Hence, by applying the inequalities (27) and (26)

Original: [Equation]
Revised: [Equation]

Original: we can see that
Revised: we can see that

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised:  

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation]
Revised: [Table] [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table] [Equation] [Table] [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original:  

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now, for f FNN ∈ Φ( L, W, B, S ) , we have
Revised: Now, for f FNN ∈ Φ( L, W, B, S ) , we have

Original: [Equation]
Revised: [Equation]

Original: and thus we also have
Revised: and thus we also have

Original: [Equation]
Revised: [Equation]

Original: On the other hands
Revised: On the other hands

Original: [Equation]
Revised: [Equation]

Original: f FNN
Revised: f FNN

Original: [Equation]
Revised: [Equation]

Original: g FNN ∈
Revised: g FNN ∈

Original: Φ( L, W, B, S ) can be represented as
Revised: Φ( L, W, B, S ) can be represented as

Original: [Equation]
Revised: [Equation]

Original: Then, by noticing that (cid:107) g
Revised: Then, by noticing that (cid:107) g

Original: ( X ) (cid:107) ∞
Revised: ( X ) (cid:107) ∞

Original: [Equation]

Original: ( W

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: L(cid:48)
Revised: L

Original: [Equation]
Revised: [Equation]

Original: the same argument as Lemma 3 of Suzuki (2019) yields that, if
Revised: the same argument as Lemma 3 of Suzuki (2019) yields that, if

Original: [Equation]
Revised: [Equation]

Original: 28) then the L ∞ distance between the composite functions f FNN ◦ g and g FNN (cid:48) ◦ g can be bounded as
Revised: 28) then the L ∞ distance between the composite functions f FNN ◦ g and g FNN (cid:48) ◦ g can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Since the triangle inequality yields
Revised: Since the triangle inequality yields

Original: [Equation]
Revised: [Equation]

Original: by combining these inequalities, under the condition (28) and max l =1 ,...
Revised: by combining these inequalities, under the condition (28) and max l =1 ,...

Original: L (cid:48) (cid:107) F l − F (cid:48) l (cid:107) ∞
Revised: L (cid:48) (cid:107) F l − F (cid:48) l (cid:107) ∞

Original: ≤
Revised: ≤

Original: δ , we obtain
Revised: δ , we obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: by noticing that the number of combinations of non-zero parameter conﬁgurations in Φ( L, W, B, S ) is bounded by ( W + 1) LS
Revised: by noticing that the number of combinations of non-zero parameter conﬁgurations in Φ( L, W, B, S ) is bounded by ( W + 1) LS

Original: [Equation]
Revised: [Equation]

Original: we can again utilize the same argument as Lemma 3 of
Revised: we can again utilize the same argument as Lemma 3 of

Original: Suzuki (2019) to we obtain
Revised: Suzuki (2019) to we obtain

Original: [Equation]
Revised: [Equation]

Original: Using a covering number bound of a model
Revised: Using a covering number bound of a model

Original: [Equation]
Revised: [Equation]

Original: a standard analysis of the ERM estimator gives the following bound on its estimation error.
Revised: a standard analysis of the ERM estimator gives the following bound on its estimation error.

Original: Lemma 21 (Theorem 2.6 of Hayakawa & Suzuki (2020); Schmidt-Hieber (2020)) .
Revised: Lemma 21 (Theorem 2.6 of Hayakawa & Suzuki (2020); Schmidt-Hieber (2020)) .

Original: Let (cid:98) f be any
Revised: Let (cid:98) f be any

Original: ERM estimator that takes its value in a model F ⊂ L ∞ ([0 , 1] ∞ ) .
Revised: ERM estimator that takes its value in a model F ⊂ L ∞ ([0 , 1] ∞ ) .

Original: Suppose that there exists a constant F > 0 such that (cid:107) f ◦ (cid:107) ∞ ≤ F and (cid:107) f (cid:107) ≤ F for any f ∈ F .
Revised: Suppose that there exists a constant F > 0 such that (cid:107) f ◦ (cid:107) ∞ ≤ F and (cid:107) f (cid:107) ≤ F for any f ∈ F .

Original: Then for any 0 < δ <satisfying N ( F , δ, (cid:107) · (cid:107) ∞ ) ≥ 3 , it holds that
Revised: Then for any 0 < δ <satisfying N ( F , δ, (cid:107) · (cid:107) ∞ ) ≥ 3 , it holds that

Original: [Equation]
Revised: [Equation]

Original: where C is a universal constant.
Revised: where C is a universal constant.

Original: Since we have assumed P X is absolutely continuous to the Lebesgue measure λ ∞ and (cid:107) d P X d λ ∞ (cid:107) ∞ < ∞ , (cid:107)·(cid:107) P X in the right hand side can be replaced by (cid:107)·(cid:107) 2 with a constant factor multiplication.
Revised: Since we have assumed P X is absolutely continuous to the Lebesgue measure λ ∞ and (cid:107) d P X d λ ∞ (cid:107) ∞ < ∞ , (cid:107)·(cid:107) P X in the right hand side can be replaced by (cid:107)·(cid:107) 2 with a constant factor multiplication.

Original: Now, we are ready to show Theorem 10.
Revised: Now, we are ready to show Theorem 10.

Original: Proof of Theorem 10 First, we consider the setting of γ ( s ) =
Revised: Proof of Theorem 10 First, we consider the setting of γ ( s ) =

Original: (cid:104) a, s (cid:105) .
Revised: (cid:104) a, s (cid:105) .

Original: By Theorem 9, by setting
Revised: By Theorem 9, by setting

Original: [Equation]
Revised: [Equation]

Original: we have that, for any f ◦ ∈ U ( F γ p,θ ) , there exists a dilated 
Revised: we have that, for any f ◦ ∈ U ( F γ p,θ ) , there exists a dilated CNN f ∈ P (

Original: [Equation]
Revised: [Equation]

Original: f ∈ P
Revised: L

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: S )
Revised: S )

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Moreover, by Lemma 20, the covering number of P can be bounded as
Revised: Moreover, by Lemma 20, the covering number of P can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Here, under the assumption (cid:107) f ◦ (cid:107) ∞
Revised: Here, under the assumption (cid:107) f ◦ (cid:107) ∞

Original: ≤ B f , we can also obtain that there exists f ∈ ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, B, S ) such that (cid:107) f − f ◦ (cid:107) 22 ≤ 2 − 2(1 − v/a 1 ) T , and a covering number evaluation log (cid:0) N ( ¯ P , δ, (cid:107) · (cid:107) ∞ ) (cid:1) ≤ log ( N ( P , δ, (cid:107) · (cid:107) ∞ )) .
Revised: ≤ B f , we can also obtain that there exists f ∈ ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, B, S ) such that (cid:107) f − f ◦ (cid:107) 22 ≤ 2 − 2(1 − v/a 1 ) T , and a covering number evaluation log (cid:0) N ( ¯ P , δ, (cid:107) · (cid:107) ∞ ) (cid:1) ≤ log ( N ( P , δ, (cid:107) · (cid:107) ∞ )) .

Original: Therefore, by Lemma 21, the ERM estimator (cid:98) f taking its value in ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, B, S ) satisﬁes
Revised: Therefore, by Lemma 21, the ERM estimator (cid:98) f taking its value in ¯ P ( B f , L (cid:48) , B (cid:48) , W (cid:48) , C, L, W, B, S ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: n + δ 
Revised: n + δ ( B f + σ

Original: [Equation]
Revised: [Equation]

Original: Thus, by taking T to satisfy 2 Ta 1 = n 1 2( a 1 − v )+1 and letting δ = 1 n , we obtain that
Revised: Thus, by taking T to satisfy 2 Ta 1 = n 1 2( a 1 − v )+1 and letting δ = 1 n , we obtain that

Original: [Equation]
Revised: [Equation]

Original: which yields the ﬁrst assertion.
Revised: which yields the ﬁrst assertion.

Original: Next, we consider the setting of γ = max i { a i s i } i .
Revised: Next, we consider the setting of γ = max i { a i s i } i .

Original: We again apply Theorem 9 so that, by setting
Revised: We again apply Theorem 9 so that, by setting

Revised: L

Revised: [Equation]

Revised: = 1 ,

Original: [Equation]
Revised: [Equation]

Original: we have that, for any f ◦ ∈ U ( F γp,θ ) , there exists a dilated CNN f ∈P ( L
Revised: we have that, for any f ◦ ∈ U ( F γp,θ ) , there exists a dilated CNN [Equation] ∈

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Therefore, by the same argument as the setting of mix smoothness, we can verify
Revised: Therefore, by the same argument as the setting of mix smoothness, we can verify

Original: [Equation]
Revised: [Equation]

Original: which yields the second assertion.
Revised: which yields the second assertion.

Original: G P ROOFS OF T HEOREMS 13 AND
Revised: G P ROOFS OF T HEOREMS 13 AND

Original: First, we prove the following lemma.
Revised: First, we prove the following lemma.

Original: Lemma 22.
Revised: Lemma 22.

Original: Let X =
Revised: Let X =

Original: ( x i ) ∞
Revised: ( x i ) ∞

Original: i =1 ∈ R ∞ .
Revised: i =1 ∈ R ∞ .

Original: For any sequence ( x i 1 , . . .
Revised: For any sequence ( x i 1 , . . .

Original: , x i N ) satisfying i j ≤ H L (cid:48) with arbitrary integer H > 

Revised: , x i N ) satisfying i j ≤ H L (cid:48) with arbitrary integer H >

Original: and i 1 < i 2 < · · · < i N , there exists a dilated CNN where the number of layers is L (cid:48) , the width of ﬁlters is H , the number of channels of each layer is N and the ﬁlters
Revised: and i 1 < i 2 < · · · < i N , there exists a dilated CNN where the number of layers is L (cid:48) , the width of ﬁlters is H , the number of channels of each layer is N and the ﬁlters

Original: W l ∈ R N × N × H (2 ≤ l ≤ T ) and W 1 ∈ R N × H in each layer satisfy (cid:107) W l (cid:107) ∞ ≤ 1 ( l = 1 , . . .
Revised: W l ∈ R N × N × H (2 ≤ l ≤ T ) and W 1 ∈ R N × H in each layer satisfy (cid:107) W l (cid:107) ∞ ≤ 1 ( l = 1 , . . .

Original: , L (cid:48) )
Revised: , L (cid:48) )

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Revised: ◦

Original:  First, we let ( W 1 ) k,j = 1 if ( i k − 1 mod T ) =
Revised: Conv [Equation] W 1 ◦ X. First, we let ( W 1 ) k,j = 1 if ( i k − 1 mod T ) =

Original: j − 1 and ( W 1 ) k,j = 0 otherwise, for k ∈ [ N ] and j ∈ [ T ] .
Revised: j − 1 and ( W 1 ) k,j = 0 otherwise, for k ∈ [ N ] and j ∈ [ T ] .

Original: Then, for each k ∈ [ N ] , there exists i (1) k ∈
Revised: Then, for each k ∈ [ N ] , there exists i (1) k ∈

Original: [ H L (cid:48) − 1 ] such that A 1 ( X ) k, ( i (1) k − 1) H
Revised: [ H L (cid:48) − 1 ] such that A 1 ( X ) k, ( i (1) k − 1) H

Original: +

Revised: +

Original: = x i k , indeed, i (1) k can be obtained as i (1) k
Revised: = x i k , indeed, i (1) k can be obtained as i (1) k

Original: = (cid:98)
Revised: = (cid:98)

Original: ( i k − 1) /H
Revised: ( i k − 1) /H

Original: (cid:99)
Revised: (cid:99)

Original: + 1 .
Revised: + 1 .

Original: In the following, we recursively deﬁne W l and i ( l ) k for l = 2 , . . .
Revised: In the following, we recursively deﬁne W l and i ( l ) k for l = 2 , . . .

Original: , L (cid:48)
Revised: , L (cid:48)

Original: so that
Revised: so that

Original: [Equation]
Revised: [Equation]

Original: is satisﬁed.
Revised: is satisﬁed.

Original: We remark that this is satisﬁed for l = 2 .
Revised: We remark that this is satisﬁed for l = 2 .

Original: Suppose that, for l , we have W l and i ( l ) k that satisfy the condition (29).
Revised: Suppose that, for l , we have W l and i ( l ) k that satisfy the condition (29).

Original: If we deﬁne
Revised: If we deﬁne

Original: [Equation]
Revised: [Equation]

Original: for i ∈ N , then we have that
Revised: for i ∈ N , then we have that

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: Therefore, by letting i ( l +1) k as i ( l +1) k
Revised: Therefore, by letting i ( l +1) k as i ( l +1) k

Original: = (cid:98) ( i ( l ) i − 1) /H
Revised: = (cid:98) ( i ( l ) i − 1) /H

Original: (cid:99)
Revised: (cid:99)

Original: + 

Revised: +

Original: (i.e., the integer such that
Revised: (i.e., the integer such that

Original: ( i ( l +1)
Revised: ( i ( l +1)

Original: k − 1) H + 1 ≤ i ( l ) k < i ( l +1) k
Revised: k − 1) H + 1 ≤ i ( l ) k < i ( l +1) k

Original: H ) and setting W l +1 so that ( W l +1 ) k,j = 1 if ( i ( l ) k − 1 mod T ) =
Revised: H ) and setting W l +1 so that ( W l +1 ) k,j = 1 if ( i ( l ) k − 1 mod T ) =

Original: j − 1 and ( W l +1 ) k,j = 0 otherwise ( k ∈ [ N ] and j ∈ [ T ] ), we can see that the condition (29) ( A l +1 ( X ) k, ( i ( l +1) k − 1) T l
Revised: j − 1 and ( W l +1 ) k,j = 0 otherwise ( k ∈ [ N ] and j ∈ [ T ] ), we can see that the condition (29) ( A l +1 ( X ) k, ( i ( l +1) k − 1) T l

Original: +1 +

Revised: +1 +

Original: = x i k for all k ∈ [ N ] ) holds for l + 1 .
Revised: = x i k for all k ∈ [ N ] ) holds for l + 1 .

Original: Therefore, by the inductive argument, we have that
Revised: Therefore, by the inductive argument, we have that

Original: A

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: because i ( L (cid:48) )
Revised: because i ( L (cid:48) )

Original: k satisﬁes 0 ≤ i ( L (cid:48) )
Revised: k satisﬁes 0 ≤ i ( L (cid:48) )

Original: k − 1 ≤
Revised: k − 1 ≤

Original: ( i k − 1) /H
Revised: ( i k − 1) /H

Original: L (cid:48) which yields i ( L (cid:48) )
Revised: L (cid:48) which yields i ( L (cid:48) )

Original: k
Revised: k

Original: = 1 for all k ∈ [ N ] by the assumption i k ≤ H L (cid:48) .
Revised: = 1 for all k ∈ [ N ] by the assumption i k ≤ H L (cid:48) .

Original: A pictorial illustration of the proof of Lemma 22 with H = W (cid:48) , N = C (cid:48) and L (cid:48)
Revised: A pictorial illustration of the proof of Lemma 22 with H = W (cid:48) , N = C (cid:48) and L (cid:48)

Original: = 2 is given in
Revised: = 2 is given in

Original: Figure 1.
Revised: Figure 1.

Original: [Figure]
Revised: [Figure]

Original: Proofs of Theorems 13 and 14.
Revised: Proofs of Theorems 13 and 14.

Original: Let ( i j ) ∞
Revised: Let ( i j ) ∞

Original: j =1 be the sequence of indices that yields a i 1 < a i 2 < · · · for the given positive sequence
Revised: j =1 be the sequence of indices that yields a i 1 < a i 2 < · · · for the given positive sequence

Original: a =
Revised: a =

Original: ( a i ) ∞ i =1 .
Revised: ( a i ) ∞ i =1 .

Original: We only give the proof for the mixed smoothness setting γ ( s ) =
Revised: We only give the proof for the mixed smoothness setting γ ( s ) =

Original: (cid:104) a, s (cid:105) .
Revised: (cid:104) a, s (cid:105) .

Original: As for the anisotropic smoothness setting, the same reasoning can be applied.
Revised: As for the anisotropic smoothness setting, the same reasoning can be applied.

Original: Since (cid:107) a (cid:107) wl q ≤ 1 is satisﬁed by the assumption, we have a i j ≥ j q for all j ∈ N .
Revised: Since (cid:107) a (cid:107) wl q ≤ 1 is satisﬁed by the assumption, we have a i j ≥ j q for all j ∈ N .

Original: Therefore, we can verify that for all T > 0 , it holds that
Revised: Therefore, we can verify that for all T > 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Hence, we can see that d max = T 1 q , f max = T 1 q by their deﬁnitions.
Revised: Hence, we can see that d max = T 1 q , f max = T 1 q by their deﬁnitions.

Original: Since a i = Ω(log i ) , there exists a constant Q > 0 such that
Revised: Since a i = Ω(log i ) , there exists a constant Q > 0 such that

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ [ N ] .
Revised: for all i ∈ [ N ] .

Original: Then, if a i ≤ T , i ≤ exp (cid:16) TQ (cid:17) is satisﬁed.
Revised: Then, if a i ≤ T , i ≤ exp (cid:16) TQ (cid:17) is satisﬁed.

Original: By using Lemma 22 with N = d max , we can construct the dilated convolutional structure
Revised: By using Lemma 22 with N = d max , we can construct the dilated convolutional structure

Original: [Equation]
Revised: [Equation]

Original: that extracts all of the elements in I ( T, γ ) .
Revised: that extracts all of the elements in I ( T, γ ) .

Original: In particular, for a fully connected neural network ˆ R T with d max dimensional input, we have that
Revised: In particular, for a fully connected neural network ˆ R T with d max dimensional input, we have that

Original: [Equation]
Revised: [Equation]

Original: that is, the convolutional neural network in the right hand side depends only on the elements in
Revised: that is, the convolutional neural network in the right hand side depends only on the elements in

Original: I ( T, γ ) .
Revised: I ( T, γ ) .

Original: On the other hand, by Lemma 18, we have that
Revised: On the other hand, by Lemma 18, we have that

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Theorem 7, if we set
Revised: Therefore, by Theorem 7, if we set

Original: [Equation]
Revised: [Equation]

Original: there exists a neural network
Revised: there exists a neural network

Original: ˆ R T ∈
Revised: ˆ R T ∈

Original: Φ( L, W, B, S ) such that f (cid:48) 
Revised: Φ( L, W, B, S ) such that f (cid:48) ( X

Original: [Equation]
Revised: [Equation]

Original: X
Revised: = [Equation] R T (

Original: [Equation]
Revised: [Equation]

Original: =
Revised: I [Equation] T

Original: [Equation]
Revised: [Equation]

Original: ) satisﬁes
Revised: γ )) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Therefore, by combining Eq.
Revised: Therefore, by combining Eq.

Original: (32) and Eq.
Revised: (32) and Eq.

Original: (35), we can set
Revised: (35), we can set

Revised: L (cid:48)

Revised: = (cid:24)

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original:  T, W
Revised: ∼ T, W

Original: [Equation]
Revised: [Equation]

Original: = 3 , 
Revised: = 3 , C

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = T 1 q

Original: [Equation]
Revised: [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: = 1 , and ( L, W, S, B ) as above so that there exists f (cid:48)
Revised: = 1 , and ( L, W, S, B ) as above so that there exists f (cid:48)

Original: ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, B, S )
Revised: ∈ P ( L (cid:48) , B (cid:48) , W (cid:48) , C (cid:48) , L, W, B, S )

Original: such that (cid:107)
Revised: such that (cid:107)

Original: [Equation]
Revised: [Equation]

Original: This yields the proof of Theorem 13.
Revised: This yields the proof of Theorem 13.

Original: The remaining proof for the estimation error (Theorem 14) can be done in the same manner as that of Theorem 10, which yields Theorem 14.
Revised: The remaining proof for the estimation error (Theorem 14) can be done in the same manner as that of Theorem 10, which yields Theorem 14.

Original: H N UMERICAL EXPERIMENTS H. T HE EXPERIMENT OF D IMENSION I NDEPENDENCE
Revised: H N UMERICAL EXPERIMENTS H. T HE EXPERIMENT OF D IMENSION I NDEPENDENCE

Original: In this section, we verify the dimension independence of CNNs by numerical experiments using functions with anisotropic smoothness.
Revised: In this section, we verify the dimension independence of CNNs by numerical experiments using functions with anisotropic smoothness.

Original: The experiments were conducted in the following settings:
Revised: The experiments were conducted in the following settings:

Original: Smoothness with Polynomial order increase:
Revised: Smoothness with Polynomial order increase:

Original: We consider the following function as the true function:
Revised: We consider the following function as the true function:

Original: [Equation]
Revised: [Equation]

Original: where p, d ∈ N are parameters.
Revised: where p, d ∈ N are parameters.

Original: Data generation model: y i =
Revised: Data generation model: y i =

Original: f ◦ ( x i )
Revised: f ◦ ( x i )

Original: ( i = 1 , . . . , n ) 
Revised: ( i = 1 , . . . , n ) .

Original: Parameter settings: n =128 , p =1 . 2 , 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 , d= 100 , 200 , 500 , 1000 .
Revised: Parameter settings: n = p = , 4 . 0 , 5 . 0 , d

Revised: =

Original: The trained model: CNN with the kernel width W (cid:48)
Revised: The trained model: CNN with the kernel width W (cid:48)

Original: = 10 , the number of channels C (cid:48)
Revised: = 10 , the number of channels C (cid:48)

Original: = 10 , and the depth L (cid:48)
Revised: = 10 , and the depth L (cid:48)

Original: = 1 . 2.
Revised: = 1 . 2.

Original: Smoothness with sparse structure:
Revised: Smoothness with sparse structure:

Original: We consider the following function as the true function:
Revised: We consider the following function as the true function:

Original: [Equation]
Revised: [Equation]

Original: where p, d ∈ N are parameters.
Revised: where p, d ∈ N are parameters.

Original: Data generation model: y i =
Revised: Data generation model: y i =

Original: f ◦x i (  i 
Revised: f ◦ ( x i )

Original:  =  . , n 
Revised: ( i = 1 , . . . , n ) .

Original: Parameter setting: n = 128 , p = 1 . 2 , 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 , d= 100 , 200 , 500 , 1000 .
Revised: Parameter setting: n = 128 , p = 1 . 2 , 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 , d

Original: The
Revised: =

Original:  trained model: Dilated CNN with the depth L (cid:48)
Revised: The trained model: Dilated CNN with the depth L (cid:48)

Original: = 3 layers, the kernel width W (cid:48)
Revised: = 3 layers, the kernel width W (cid:48)

Original: = 3 , and the number of channels C (cid:48)
Revised: = 3 , and the number of channels C (cid:48)

Original: = 10 .
Revised: = 10 .

Original: In each experiment, we estimated f ◦ from n observations
Revised: In each experiment, we estimated f ◦ from n observations

Original: x i , y i )
Revised: x i , y i )

Original: ni
Revised: ni

Original: =1 using a CNN or dilated CNN.
Revised: =1 using a CNN or dilated CNN.

Original: The results are shown in Figure 2 and Figure 3.
Revised: The results are shown in Figure 2 and Figure 3.

Original: These results show that for the same p , we don’t see any deterioration of the estimation error by increasing the dimensionality d of the input.
Revised: These results show that for the same p , we don’t see any deterioration of the estimation error by increasing the dimensionality d of the input.

Original: These results are consistent with the fact that the theoretical upper bound is dimension independent and depends only on smoothness.
Revised: These results are consistent with the fact that the theoretical upper bound is dimension independent and depends only on smoothness.

Original: Moreover, in the setting of sparse smoothness, we again observe that the estimation error is dimension-independent by using dilated CNN, which is also consistent with our theoretical ﬁndings.
Revised: Moreover, in the setting of sparse smoothness, we again observe that the estimation error is dimension-independent by using dilated CNN, which is also consistent with our theoretical ﬁndings.

Original: I E XTENSION OF 2D CONVOLUTION
Revised: I E XTENSION OF 2D CONVOLUTION

Original: Here, we present that a similar argument can be applied to 2D convolution for image inputs.
Revised: Here, we present that a similar argument can be applied to 2D convolution for image inputs.

Original: For that purpose, we consider a quite simple setting where the true function f o has a mixed/anisotropic smoothness with respect to the wavelet coefﬁcients of the input image.
Revised: For that purpose, we consider a quite simple setting where the true function f o has a mixed/anisotropic smoothness with respect to the wavelet coefﬁcients of the input image.

Original: More precisely, suppose that the input image X has size N × N where N = 2 K for an integer
Revised: More precisely, suppose that the input image X has size N × N where N = 2 K for an integer

Original: K ( X ∈ [0 , 1] N × N ) and let ( α k,j ) 3( N/ 2 k ) 2 j = for k ∈ K be the wavelet coefﬁcient at the level k .
Revised: K ( X ∈ [0 , 1] N × N ) and let ( α k,j ) 3( N/ 2 k ) 2 j = for k ∈ K be the wavelet coefﬁcient at the level k .

Original: Then, we assume the true function is γ -smooth with respect to the wavelet coefﬁcients ( α 1 , 1 , α 1 , 2 , . .
Revised: Then, we assume the true function is γ -smooth with respect to the wavelet coefﬁcients ( α 1 , 1 , α 1 , 2 , . .

Original: , α 1 , 3( N/ 2) 2 , α 2 , 1 , . . .
Revised: , α 1 , 3( N/ 2) 2 , α 2 , 1 , . . .

Original: , α 2 , 3( N/ 2 2 ) , . .
Revised: , α 2 , 3( N/ 2 2 ) , . .

Original: . , α K, 1 ) ∈ R N 2 .
Revised: . , α K, 1 ) ∈ R N 2 .

Original: Usually, N 2 is large and a standard analysis suffers from the curse of dimensionality.
Revised: Usually, N 2 is large and a standard analysis suffers from the curse of dimensionality.

Original: The construction of wavelet coefﬁcients is as follows (see Daubechies (1992) for more details).
Revised: The construction of wavelet coefﬁcients is as follows (see Daubechies (1992) for more details).

Original: The wavelet transform is obtained based on a pair of a wavelet ﬁlter ϕ ∈ R N and scaling ﬁlter ψ ∈ R N .
Revised: The wavelet transform is obtained based on a pair of a wavelet ﬁlter ϕ ∈ R N and scaling ﬁlter ψ ∈ R N .

Original: We assume this pair of ﬁlters are given.
Revised: We assume this pair of ﬁlters are given.

Original: For u ∈ R M 1 and v ∈ R M 2 , let u ⊗ v :=   u 1 v. . .
Revised: For u ∈ R M 1 and v ∈ R M 2 , let u ⊗ v :=   u 1 v. . .

Original: u 1 v M 2 .
Revised: u 1 v M 2 .

Original: u M 1 v. . .
Revised: u M 1 v. . .

Original: u M 1 v M  ∈ R M 1 × M 2 , and for x, y ∈ R M 1 × M 2 , let (cid:104) x, y (cid:105) :
Revised: u M 1 v M  ∈ R M 1 × M 2 , and for x, y ∈ R M 1 × M 2 , let (cid:104) x, y (cid:105) :

Original: =
Revised: =

Original: M 1 (cid:88) i =
Revised: M 1 (cid:88) i =

Original: M 2 (cid:88)
Revised: M 2 (cid:88)

Original: j = x i,j y i,j .
Revised: j = x i,j y i,j .

Original: The level k wavelet and scaling bases are obtained by
Revised: The level k wavelet and scaling bases are obtained by

Original: [Equation]
Revised: [Equation]

Original: We apply a shift operation on these basis functions as
Revised: We apply a shift operation on these basis functions as

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]

Original: and we deﬁne ψ ( k,j ) in the same way.
Revised: and we deﬁne ψ ( k,j ) in the same way.

Original: Using these ﬁlter, we can calculate the wavelet coefﬁcients in an inductive manner.
Revised: Using these ﬁlter, we can calculate the wavelet coefﬁcients in an inductive manner.

Original: At the ﬁrst level, we obtain
Revised: At the ﬁrst level, we obtain

Original: [Equation]
Revised: [Equation]

Original: Let A (1 ,q ) =
Revised: Let A (1 ,q ) =

Original: ( α (1 ,q ) i,j ) i,j for q = 1 , 2 , 3 , 4 .
Revised: ( α (1 ,q ) i,j ) i,j for q = 1 , 2 , 3 , 4 .

Original: At the k -th level, we assume that we have already have
Revised: At the k -th level, we assume that we have already have

Original: A ( k − 1 ,q ) ∈ R N/ 2 k − 1 ( q ∈ [4]) .
Revised: A ( k − 1 ,q ) ∈ R N/ 2 k − 1 ( q ∈ [4]) .

Original: Then, the k -th level coefﬁcients can be obtained by
Revised: Then, the k -th level coefﬁcients can be obtained by

Original: [Equation]
Revised: [Equation]

Original: ϕ( k
Revised: ϕ

Original: [Equation]
Revised: [Equation]

Original: i ) ⊗ ψ ( k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: j ) , A (k −
Revised: i ) ⊗ ψ (

Original: [Equation]
Revised: [Equation]

Original: 1) (cid:105) ( i

Original: [Equation]
Revised: [Equation]

Original: ∈ [ N/ 2 k ]) , α
Revised: A

Original: [Equation]
Revised: [Equation]

Original: =
Revised: ) (cid:105) (

Original: [Equation]
Revised: [Equation]

Original: We again let A ( k,q ) =
Revised: We again let A ( k,q ) =

Original: ( α ( k,q ) i,j ) i,j for q = 1 , 2 , 3 , 4 .
Revised: ( α ( k,q ) i,j ) i,j for q = 1 , 2 , 3 , 4 .

Original: Then, we obtain the wavelet coefﬁcient as
Revised: Then, we obtain the wavelet coefﬁcient as

Original: A ( X ) =
Revised: A ( X ) =

Original: ( A ( k, 2) , A ( k, 3) , A ( k, 4) )
Revised: ( A ( k, 2) , A ( k, 3) , A ( k, 4) )

Original: K − 1 k
Revised: K − 1 k

Original: =1 in a recursive manner that represents strength of each frequency component k of the input image X at each location j .
Revised: =1 in a recursive manner that represents strength of each frequency component k of the input image X at each location j .

Original: We assume that the true function f o is γ -smooth with respect to this coefﬁcient (more precisely its vectorization vec( A ( X ))
Revised: We assume that the true function f o is γ -smooth with respect to this coefﬁcient (more precisely its vectorization vec( A ( X ))

Original: ∈ R N 2 ).
Revised: ∈ R N 2 ).

Original: Assumption 23.
Revised: Assumption 23.

Original: The true function f
Revised: The true function f

Original: o : R N × N → N is γ -smooth with respect to vec( A ( X )) .
Revised: o : R N × N → N is γ -smooth with respect to vec( A ( X )) .

Original: The smoothness satisﬁes the sparsity (Assumption 12).
Revised: The smoothness satisﬁes the sparsity (Assumption 12).

Original: Moreover, we also assume Assumptions 3 is satisﬁed.
Revised: Moreover, we also assume Assumptions 3 is satisﬁed.

Original: From now on, we construct a 2D-CNN structure to extract variables in I ( T, γ ) from vec( A ( X )) .
Revised: From now on, we construct a 2D-CNN structure to extract variables in I ( T, γ ) from vec( A ( X )) .

Original: Suppose that the support of ϕ and ψ are included in the ﬁrst H components, then the inner product appeared above are executable just in a local H × H region.
Revised: Suppose that the support of ϕ and ψ are included in the ﬁrst H components, then the inner product appeared above are executable just in a local H × H region.

Original: For example, the Haar wavelet has
Revised: For example, the Haar wavelet has

Original: H = 2 .
Revised: H = 2 .

Original: Therefore, the inner product to extract the coefﬁcients can be realized by the standard convolution operation in CNNs.
Revised: Therefore, the inner product to extract the coefﬁcients can be realized by the standard convolution operation in CNNs.

Original: From now on, we employ the same notations used in the proof of Theorems 13 and 14.
Revised: From now on, we employ the same notations used in the proof of Theorems 13 and 14.

Original: We T >be any positive real.
Revised: We T >be any positive real.

Original: We deﬁne ( d max , f max , G ) as in Theorem 7.
Revised: We deﬁne ( d max , f max , G ) as in Theorem 7.

Original: If we can show that a CNN can extract variables included in I ( T, γ ) , then we can apply the same argument to Theorems 13 and 14.
Revised: If we can show that a CNN can extract variables included in I ( T, γ ) , then we can apply the same argument to Theorems 13 and 14.

Original: Let the number of channels of of 2D convolution be C k = d max + 4 for k = 1 , . .
Revised: Let the number of channels of of 2D convolution be C k = d max + 4 for k = 1 , . .

Original: , K and C 

Revised: , K and C

Original: = 1 .
Revised: = 1 .

Original: For each layer k ∈ [ K − 1] , we will deﬁne
Revised: For each layer k ∈ [ K − 1] , we will deﬁne

Original: a ﬁlter W ( k ) ∈ R C k × C k − 1 ×
Revised: a ﬁlter W ( k ) ∈ R C k × C k − 1 ×

Original: H k × H k where H k = min { H, N/ 2 k } .
Revised: H k × H k where H k = min { H, N/ 2 k } .

Original: Let
Revised: Let

Original: A k ( X ) ∈
Revised: A k ( X ) ∈

Original: R C k × N/ 2 k × N/ 2 k be the output from the k -th layer of the CNN.
Revised: R C k × N/ 2 k × N/ 2 k be the output from the k -th layer of the CNN.

Original: Then, A k ( X ) is updated by the following convolution with the interval h = 2 : A 0 ( X ) =
Revised: Then, A k ( X ) is updated by the following convolution with the interval h = 2 : A 0 ( X ) =

Original: X
Revised: X

Original: and
Revised: and

Original: A k ( X ) q,i,j =
Revised: A k ( X ) q,i,j =

Original: [Equation]
Revised: [Equation]

Original:  where, by convention, we let A k − 1 ( X ) i,
Revised: , where, by convention, we let A k − 1 ( X ) i,

Original: j =
Revised: j =

Original: A k − 1 ( X ) ( i − 1 mod N/ 2 k − 1 )+1 , ( j − 1 mod N/ 2 k − 1 )+1 for i and j which are out of the range { 1 , . . .
Revised: A k − 1 ( X ) ( i − 1 mod N/ 2 k − 1 )+1 , ( j − 1 mod N/ 2 k − 1 )+1 for i and j which are out of the range { 1 , . . .

Original: , N/ 2 k − 1 } .
Revised: , N/ 2 k − 1 } .

Original: Next, we show that an appropriately deﬁned ﬁlter can extract I ( T, γ ) .
Revised: Next, we show that an appropriately deﬁned ﬁlter can extract I ( T, γ ) .

Original: First, we let a part of W ( k ) as follows to extract ( A ( k, 1) , A ( k, 2) , A ( k, 3) , A ( k, 4) ) :
Revised: First, we let a part of W ( k ) as follows to extract ( A ( k, 1) , A ( k, 2) , A ( k, 3) , A ( k, 4) ) :

Original: [Equation]
Revised: [Equation]

Original: where x i : i (cid:48) =
Revised: where x i : i (cid:48) =

Original: ( x i , x i +1 , . . .
Revised: ( x i , x i +1 , . . .

Original: , x i (cid:48) )
Revised: , x i (cid:48) )

Original: (cid:62) for a vector x .
Revised: (cid:62) for a vector x .

Original: Second, for k ≥ 2 , we extract the “important coefﬁcients” included in I ( T, γ ) from ( A ( k − 1 , 2) , A ( k − 1 , 3) , A ( k − 1 , 4) ) that are already extracted in the last layer.
Revised: Second, for k ≥ 2 , we extract the “important coefﬁcients” included in I ( T, γ ) from ( A ( k − 1 , 2) , A ( k − 1 , 3) , A ( k − 1 , 4) ) that are already extracted in the last layer.

Original: For that purpose, let
Revised: For that purpose, let

Original: W ( k ) j,q,i 1 ,j 1 = 1 for 2 ≤ q ≤ 4 , j ≥ 5 and i 1 , j 1 ∈ [2] if A ( k − 1 ,q ) contains the ( j − 4) -th element of I ( T, γ ) at its ( i 1 , j 1 ) -th component.
Revised: W ( k ) j,q,i 1 ,j 1 = 1 for 2 ≤ q ≤ 4 , j ≥ 5 and i 1 , j 1 ∈ [2] if A ( k − 1 ,q ) contains the ( j − 4) -th element of I ( T, γ ) at its ( i 1 , j 1 ) -th component.

Original: Otherwise, let W ( k ) j,q,i 1 ,j 1 = 0 .
Revised: Otherwise, let W ( k ) j,q,i 1 ,j 1 = 0 .

Original: Finally, we extract the important coefﬁcients included in A k − 1 ( X ) q, : , : for q ≥ 5 .
Revised: Finally, we extract the important coefﬁcients included in A k − 1 ( X ) q, : , : for q ≥ 5 .

Original: For 5 ≤ q ≤ N , let W j,q,i 1 ,i 2 = 1 for j ≥ 5 and i 1 , j 1 ∈ [2] if A k − 1 ( X ) q, : , : contains the ( j − 4) -th element of I ( T, γ ) at its ( i 1 , j 1 ) -th component.
Revised: For 5 ≤ q ≤ N , let W j,q,i 1 ,i 2 = 1 for j ≥ 5 and i 1 , j 1 ∈ [2] if A k − 1 ( X ) q, : , : contains the ( j − 4) -th element of I ( T, γ ) at its ( i 1 , j 1 ) -th component.

Original: Otherwise, we let W j,q,i 1 ,i 2 = 0 .
Revised: Otherwise, we let W j,q,i 1 ,i 2 = 0 .

Original: Since the interval of convolution is h = 2 , the size of A K ( X ) is C K × 1 × 1 which can be seen as a “vector.”
Revised: Since the interval of convolution is h = 2 , the size of A K ( X ) is C K × 1 × 1 which can be seen as a “vector.”

Original: By the construction above, it contains all coefﬁcients included in I ( T, γ ) .
Revised: By the construction above, it contains all coefﬁcients included in I ( T, γ ) .

Original: Therefore, by feeding ˜ X = vec( A K ( X )) to an FNN ˆ R T as considered in the proof of Theorems 13 and 14, we obtain the same approximation and estimation error bounds as those theorems under Assumption 23.
Revised: Therefore, by feeding ˜ X = vec( A K ( X )) to an FNN ˆ R T as considered in the proof of Theorems 13 and 14, we obtain the same approximation and estimation error bounds as those theorems under Assumption 23.

Original: Here, note that the number of parameters in the CNN constructed above has the same order as the 1D-situation thanks.
Revised: Here, note that the number of parameters in the CNN constructed above has the same order as the 1D-situation thanks.

Original: Thus, we obtain the same convergence rate as Theorems 13 and 14.
Revised: Thus, we obtain the same convergence rate as Theorems 13 and 14.