Original: M ACHINE L EARNING F OR E LLIPTIC PDE S : F AST R ATE G ENERALIZATION B OUND , N EURAL S CALING L AW AND M INIMAX O PTIMALITY
Revised: M ACHINE L EARNING F OR E LLIPTIC PDE S : F AST R ATE G ENERALIZATION B OUND , N EURAL S CALING L AW AND M INIMAX O PTIMALITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep
Revised: In this paper, we study the statistical limits of deep learning techniques for solving elliptic partial differential equations (PDEs) from random samples using the Deep

Original: Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs).
Revised: Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs).

Original: To simplify the problem, we focus on a prototype elliptic PDE: the Schrödinger equation on a hypercube with zero Dirichlet boundary condition, which is applied in quantummechanical systems.
Revised: To simplify the problem, we focus on a prototype elliptic PDE: the Schrödinger equation on a hypercube with zero Dirichlet boundary condition, which is applied in quantummechanical systems.

Original: We establish upper and lower bounds for both methods, which improve upon concurrently developed upper bounds for this problem via a fast rate generalization bound.
Revised: We establish upper and lower bounds for both methods, which improve upon concurrently developed upper bounds for this problem via a fast rate generalization bound.

Original: We discover that the current Deep Ritz Method is sub-optimal and propose a modiﬁed version of it.
Revised: We discover that the current Deep Ritz Method is sub-optimal and propose a modiﬁed version of it.

Original: We also prove that PINN and the modiﬁed version of DRM can achieve minimax optimal bounds over
Revised: We also prove that PINN and the modiﬁed version of DRM can achieve minimax optimal bounds over

Original: Sobolev spaces.
Revised: Sobolev spaces.

Original: Empirically, following recent work which has shown that the deep model accuracy will improve with growing training sets according to a power law, we supply computational experiments to show similar-behavior of dimension dependent power law for deep PDE solvers.
Revised: Empirically, following recent work which has shown that the deep model accuracy will improve with growing training sets according to a power law, we supply computational experiments to show similar-behavior of dimension dependent power law for deep PDE solvers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Partial differential equations (PDEs) play a prominent role in many disciplines of science and engineering.
Revised: Partial differential equations (PDEs) play a prominent role in many disciplines of science and engineering.

Original: The recent deep learning breakthrough and the rapid development of sensors, computational power, and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods (Long et al., 2018; 2019; Raissi et al., 2019; Han et al., 2018; Sirignano &
Revised: The recent deep learning breakthrough and the rapid development of sensors, computational power, and data storage in the past decade has drawn attention to numerically solving PDEs via machine learning methods (Long et al., 2018; 2019; Raissi et al., 2019; Han et al., 2018; Sirignano &

Original: Spiliopoulos, 2018; Khoo et al., 2017), especially in high dimensions where conventional methods become impractical.
Revised: Spiliopoulos, 2018; Khoo et al., 2017), especially in high dimensions where conventional methods become impractical.

Original: The set of applications that motivate this interest is wide-ranging, including computational physics (Han et al., 2018;
Revised: The set of applications that motivate this interest is wide-ranging, including computational physics (Han et al., 2018;

Original: Long et al., 2018; Raissi et al., 2019), inverse problem (Zhang et al., 2018; Gilton et al., 2019; Fan & Ying, 2020) and quantitative ﬁnance (Heaton et al., 2017; Germain et al., 2021).
Revised: Long et al., 2018; Raissi et al., 2019), inverse problem (Zhang et al., 2018; Gilton et al., 2019; Fan & Ying, 2020) and quantitative ﬁnance (Heaton et al., 2017; Germain et al., 2021).

Original: The numerical methods generated by the use of deep learning techniques are mesh-less methods, see the discussion in (Xu, 2020).
Revised: The numerical methods generated by the use of deep learning techniques are mesh-less methods, see the discussion in (Xu, 2020).

Original: A natural deep learning technique in the problems that are based on a standard feed-forward type of architecture takes advantage (when available) of a variational formulation, whose solution coincides with the solution of the PDE of interest.
Revised: A natural deep learning technique in the problems that are based on a standard feed-forward type of architecture takes advantage (when available) of a variational formulation, whose solution coincides with the solution of the PDE of interest.

Original: Despite the success and popularity of adopting neural networks for solving high-dimensional
Revised: Despite the success and popularity of adopting neural networks for solving high-dimensional

Original: PDEs, the following question still remains poorly answered
Revised: PDEs, the following question still remains poorly answered

Original: For a given PDE and a data-driven approximation architecture, how large the sample size and how complex the model are needed to reach a prescribed performance level?
Revised: For a given PDE and a data-driven approximation architecture, how large the sample size and how complex the model are needed to reach a prescribed performance level?

Original: In this paper, we aim to establish the numerical analysis of such deep learning based PDE solvers.
Revised: In this paper, we aim to establish the numerical analysis of such deep learning based PDE solvers.

Original: Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number, known as the neural scaling law (Kaplan et al., 2020;
Revised: Inspired by recent works which showed that the empirical performance of a model is remarkably predictable via a power law of the data number, known as the neural scaling law (Kaplan et al., 2020;

Original: Hestness et al., 2017; Sharma & Kaplan, 2020), we aim to explore the neural scaling law for deep
Revised: Hestness et al., 2017; Sharma & Kaplan, 2020), we aim to explore the neural scaling law for deep

Original: PDE solvers and compare its performance to Fourier approximation.
Revised: PDE solvers and compare its performance to Fourier approximation.

Original: Among the various approaches of using deep learning methods for solving PDEs, in this work, we focus on the Deep Ritz method (DRM) (E & Yu, 2018; Khoo et al., 2017) and the Physics-Informed
Revised: Among the various approaches of using deep learning methods for solving PDEs, in this work, we focus on the Deep Ritz method (DRM) (E & Yu, 2018; Khoo et al., 2017) and the Physics-Informed

Original: Neural Networks (PINN) approach (Sirignano & Spiliopoulos, 2018; Raissi et al., 2019), both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs.
Revised: Neural Networks (PINN) approach (Sirignano & Spiliopoulos, 2018; Raissi et al., 2019), both of which are based on minimizing neural network parameters according to some loss functional related to the PDEs.

Original: To provide theoretical guarantees for DRM and PINN, following (Lu et al., 2021b;
Revised: To provide theoretical guarantees for DRM and PINN, following (Lu et al., 2021b;

Original: Duan et al., 2021; Bai et al., 2021), we decompose the error into approximation error (Yarotsky, 2017;
Revised: Duan et al., 2021; Bai et al., 2021), we decompose the error into approximation error (Yarotsky, 2017;

Original: Suzuki, 2018; Shen et al., 2021) and generalization error (Bartlett et al., 2005; Xu & Zeevi, 2020;
Revised: Suzuki, 2018; Shen et al., 2021) and generalization error (Bartlett et al., 2005; Xu & Zeevi, 2020;

Original: Farrell et al., 2021; Schmidt-Hieber et al., 2020; Suzuki, 2018).
Revised: Farrell et al., 2021; Schmidt-Hieber et al., 2020; Suzuki, 2018).

Original: However, instead of the O (1 / √ n ) ( n is the number of data sampled) slow rate generalization bounds established in prior work (Lu et al., 2021b;
Revised: However, instead of the O (1 / √ n ) ( n is the number of data sampled) slow rate generalization bounds established in prior work (Lu et al., 2021b;

Original: Shen et al., 2021; Xu, 2020; Shin et al., 2020), we utilize the strongly convex structure of the DRM and PINN objectives and provide an O (1 /n ) fast rate generalization bound (Bartlett et al., 2005; Xu & Zeevi, 2020) that leads us to a non-parametric estimation bound.
Revised: Shen et al., 2021; Xu, 2020; Shin et al., 2020), we utilize the strongly convex structure of the DRM and PINN objectives and provide an O (1 /n ) fast rate generalization bound (Bartlett et al., 2005; Xu & Zeevi, 2020) that leads us to a non-parametric estimation bound.

Original: Our theory also suggests an optimal selection of network size with respect to the number of sampled data.
Revised: Our theory also suggests an optimal selection of network size with respect to the number of sampled data.

Original: Moreover, to illustrate the optimality of our upper bound, we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modiﬁed version of DRM.
Revised: Moreover, to illustrate the optimality of our upper bound, we also establish an information-theoretic lower bound which matches our upper bound for PINN and a modiﬁed version of DRM.

Original: We also test our theory by numerical experiments.
Revised: We also test our theory by numerical experiments.

Original: Recent works (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Mikami et al., 2021) studying a variety of deep learning algorithms all ﬁnd the same polynomial scaling relation between the testing error and the number of data.
Revised: Recent works (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Mikami et al., 2021) studying a variety of deep learning algorithms all ﬁnd the same polynomial scaling relation between the testing error and the number of data.

Original: As the number of training data n increases, the population loss L of well-trained and well-tuned models scales with n as a power-law L ∝ 1 n α for some α .
Revised: As the number of training data n increases, the population loss L of well-trained and well-tuned models scales with n as a power-law L ∝ 1 n α for some α .

Original: (Sharma & Kaplan, 2020) also scans over a large range of α and problem dimension d and ﬁnds an approximately α ∝ 1 d scaling law.
Revised: (Sharma & Kaplan, 2020) also scans over a large range of α and problem dimension d and ﬁnds an approximately α ∝ 1 d scaling law.

Original: In Section 4, we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory.
Revised: In Section 4, we conduct numerical experiments to show that this phenomenon still appears for deep PDE solvers and this neural scaling law tests more idiosyncratic features of the theory.

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: Neural Scaling Law
Revised: Neural Scaling Law

Original: The starting point of our work is the recent observation across speech, vision and text (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Rosenfeld, 2021) that the empirical performance of a model satisﬁes a power law scales as a power-law with model size and dataset size.
Revised: The starting point of our work is the recent observation across speech, vision and text (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2019; Rosenfeld, 2021) that the empirical performance of a model satisﬁes a power law scales as a power-law with model size and dataset size.

Original: (Sharma & Kaplan, 2020) further ﬁnds out that the power of the scaling law depends on the intrinsic dimension of the dataset.
Revised: (Sharma & Kaplan, 2020) further ﬁnds out that the power of the scaling law depends on the intrinsic dimension of the dataset.

Original: Theoretical works (Schmidt-Hieber et al., 2020; Suzuki, 2018;
Revised: Theoretical works (Schmidt-Hieber et al., 2020; Suzuki, 2018;

Original: Suzuki & Nitanda, 2019; Chen et al., 2019b; Imaizumi & Fukumizu, 2020; Farrell et al., 2021; Jiao et al., 2021c) explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network.
Revised: Suzuki & Nitanda, 2019; Chen et al., 2019b; Imaizumi & Fukumizu, 2020; Farrell et al., 2021; Jiao et al., 2021c) explore the optimal power law under the non-parametric curve estimation setting via a plug-in neural network.

Original: Our work extends this line of research to solving PDEs.
Revised: Our work extends this line of research to solving PDEs.

Original: Deep Network Based PDE Solver.
Revised: Deep Network Based PDE Solver.

Original: Solving high dimensional partial differential equations (PDEs) has been a long-standing challenge due to the curse of dimensionality.
Revised: Solving high dimensional partial differential equations (PDEs) has been a long-standing challenge due to the curse of dimensionality.

Original: At the same time, deep learning has shown superior ﬂexibility and adaptivity in approximating high dimensional functions, which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing.
Revised: At the same time, deep learning has shown superior ﬂexibility and adaptivity in approximating high dimensional functions, which leads to state-of-the-art performances in a wide range of tasks ranging from computer vision to natural language processing.

Original: Recent years, pioneer works (Han et al., 2018; Raissi et al., 2019;
Revised: Recent years, pioneer works (Han et al., 2018; Raissi et al., 2019;

Original: Long et al., 2018; Sirignano & Spiliopoulos, 2018; Khoo et al., 2017) try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks (Lu et al., 2021a; Li et al., 2020).
Revised: Long et al., 2018; Sirignano & Spiliopoulos, 2018; Khoo et al., 2017) try to utilize the deep neural networks to solve different types of PDEs and achieve impressive results in many tasks (Lu et al., 2021a; Li et al., 2020).

Original: Based on the natural idea of representing solutions of PDEs by (deep) neural networks, different loss functions for solving PDEs are proposed.
Revised: Based on the natural idea of representing solutions of PDEs by (deep) neural networks, different loss functions for solving PDEs are proposed.

Original: (Han et al., 2018; 2020) utilize the
Revised: (Han et al., 2018; 2020) utilize the

Original: Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network (Zang et al., 2020) solves the weak formulations of PDEs via an adversarial network.
Revised: Feynman-Kac formulation which turns solving PDE to a stochastic control problem and the weak adversarial network (Zang et al., 2020) solves the weak formulations of PDEs via an adversarial network.

Original: In this paper, we focus on the convergence rate of the Deep Ritz Method (DRM) (E & Yu, 2018; Khoo et al., 2017) and Physics-Informed neural network (PINN) (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018).
Revised: In this paper, we focus on the convergence rate of the Deep Ritz Method (DRM) (E & Yu, 2018; Khoo et al., 2017) and Physics-Informed neural network (PINN) (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018).

Original: DRM (E & Yu, 2018; Khoo et al., 2017) utilizes the variational structure of the PDE, which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs, and trains a neural network to minimize the variational objective.
Revised: DRM (E & Yu, 2018; Khoo et al., 2017) utilizes the variational structure of the PDE, which is similar to the Ritz-Galerkin method in classical numerical analysis of PDEs, and trains a neural network to minimize the variational objective.

Original: PINN (Raissi et al., 2019; Sirignano &
Revised: PINN (Raissi et al., 2019; Sirignano &

Original: Spiliopoulos, 2018) trains a neural network directly to minimize the residual of the PDE, i.e., using the strong form of the PDE.
Revised: Spiliopoulos, 2018) trains a neural network directly to minimize the residual of the PDE, i.e., using the strong form of the PDE.

Original: Theoretical Guarantees For Machine Learning Based PDE Solvers.
Revised: Theoretical Guarantees For Machine Learning Based PDE Solvers.

Original: Theoretical convergence results for deep learning based PDE solvers raises wide interest recently.
Revised: Theoretical convergence results for deep learning based PDE solvers raises wide interest recently.

Original: Speciﬁcally, (Lu et al., 2021b;
Revised: Speciﬁcally, (Lu et al., 2021b;

Original: Grohs & Herrmann, 2020; Marwah et al., 2021; Wojtowytsch et al., 2020; Xu, 2020; Shin et al., 2020; Bai et al., 2021) investigate the regularity of PDEs approximated by neural network and (Lu et al., 2021b;
Revised: Grohs & Herrmann, 2020; Marwah et al., 2021; Wojtowytsch et al., 2020; Xu, 2020; Shin et al., 2020; Bai et al., 2021) investigate the regularity of PDEs approximated by neural network and (Lu et al., 2021b;

Original: Luo & Yang, 2020) further provide a generalization analysis.
Revised: Luo & Yang, 2020) further provide a generalization analysis.

Original: (Nickl et al., 2020) introduces a prior over the solution of the PDE and considers an equivalent white noise model (Brown & Low, 1996).
Revised: (Nickl et al., 2020) introduces a prior over the solution of the PDE and considers an equivalent white noise model (Brown & Low, 1996).

Original: (Nickl et al., 2020) provides the rate of convergence of the posterior.
Revised: (Nickl et al., 2020) provides the rate of convergence of the posterior.

Original: Our paper does
Revised: Our paper does

Original: [Table]
Revised: [Table]

Original: not need to introduce the prior on the target function and provides a non-asymptotic guarantee for ﬁnite number of data.
Revised: not need to introduce the prior on the target function and provides a non-asymptotic guarantee for ﬁnite number of data.

Original: At the same time, (Nickl et al., 2020) can only be applied to linear PDEs while our proof technique can be extended to nonlinear ones.
Revised: At the same time, (Nickl et al., 2020) can only be applied to linear PDEs while our proof technique can be extended to nonlinear ones.

Original: All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate.
Revised: All these papers also fail to answer the question that how to determine the network size corresponding to the sampled data number to achieve a desired statistical convergence rate.

Original: (Hütter & Rigollet, 2019; Manole et al., 2021) consider the similar problem for the optimal transport problem, i.e. Monge-ampere equation.
Revised: (Hütter & Rigollet, 2019; Manole et al., 2021) consider the similar problem for the optimal transport problem, i.e. Monge-ampere equation.

Original: Nevertheless, the variational problem we considered is different from (Hütter & Rigollet, 2019; Manole et al., 2021) and leads to technical difference.
Revised: Nevertheless, the variational problem we considered is different from (Hütter & Rigollet, 2019; Manole et al., 2021) and leads to technical difference.

Original: The most related works to ours are two concurrent papers (Duan et al., 2021; Jiao et al., 2021a;b).
Revised: The most related works to ours are two concurrent papers (Duan et al., 2021; Jiao et al., 2021a;b).

Original: However, our upper bound is faster than (Duan et al., 2021;
Revised: However, our upper bound is faster than (Duan et al., 2021;

Original: Jiao et al., 2021a;b).
Revised: Jiao et al., 2021a;b).

Original: In this paper, we also show that generalization analysis in (Lu et al., 2021b; Duan et al., 2021; Luo & Yang, 2020) are loose due to the lack of a localization technique (De Boor &
Revised: In this paper, we also show that generalization analysis in (Lu et al., 2021b; Duan et al., 2021; Luo & Yang, 2020) are loose due to the lack of a localization technique (De Boor &

Original: De Boor, 1978; Bartlett et al., 2005; Koltchinskii, 2011; Xu, 2020).
Revised: De Boor, 1978; Bartlett et al., 2005; Koltchinskii, 2011; Xu, 2020).

Original: With observation of the strong convexity of the loss function, we follow the fast rate results for ERM (Schmidt-Hieber et al., 2020;
Revised: With observation of the strong convexity of the loss function, we follow the fast rate results for ERM (Schmidt-Hieber et al., 2020;

Original: Xu & Zeevi, 2020; Farrell et al., 2021) and provide a near optimal bound for both DRM and PINN.
Revised: Xu & Zeevi, 2020; Farrell et al., 2021) and provide a near optimal bound for both DRM and PINN.

Original: 1. C ONTRIBUTION
Revised: 1. C ONTRIBUTION

Original: In short, we summarize our contribution as follows
Revised: In short, we summarize our contribution as follows

Original: • In this paper, we ﬁrst considered the statistical limit of learning a PDE solution from sampled observations.
Revised: • In this paper, we ﬁrst considered the statistical limit of learning a PDE solution from sampled observations.

Original: The lower bound shows a non-standard exponent different from non-parametric estimation of a function.
Revised: The lower bound shows a non-standard exponent different from non-parametric estimation of a function.

Original: • Instead of the O (1 / √ n ) slow rate generalization bounds in (Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;c), we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods (Van De Geer, 1987; Bartlett et al., 2005; Koltchinskii, 2011; Srebro et al., 2010; Xu & Zeevi, 2020).
Revised: • Instead of the O (1 / √ n ) slow rate generalization bounds in (Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;c), we utilized the strongly convex nature of the variational form and provided a fast rate generalization bound via the localization methods (Van De Geer, 1987; Bartlett et al., 2005; Koltchinskii, 2011; Srebro et al., 2010; Xu & Zeevi, 2020).

Original: We discovered that the current Deep Ritz Methods is sub-optimal and propose a modiﬁed version of it.
Revised: We discovered that the current Deep Ritz Methods is sub-optimal and propose a modiﬁed version of it.

Original: We showed that PINN and the modiﬁed version of DRM can achieve nearly min-max optimal convergence rate.
Revised: We showed that PINN and the modiﬁed version of DRM can achieve nearly min-max optimal convergence rate.

Original: Our result is listed in Table 1.
Revised: Our result is listed in Table 1.

Original: • We tested the recently discovered neural scaling law (Hestness et al., 2017;
Revised: • We tested the recently discovered neural scaling law (Hestness et al., 2017;

Original: Kaplan et al., 2020; Rosenfeld et al., 2019; Hashimoto, 2021) for deep PDE solvers numerically.
Revised: Kaplan et al., 2020; Rosenfeld et al., 2019; Hashimoto, 2021) for deep PDE solvers numerically.

Original: The empirical results veriﬁed our theory.
Revised: The empirical results veriﬁed our theory.

Original:  S ET - UP
Revised:  S ET - UP

Original: We consider the static Schrödinger equation with zero Dirichlet boundary condition on the domain Ω , which we assume to be the unit hypercube in R d .
Revised: We consider the static Schrödinger equation with zero Dirichlet boundary condition on the domain Ω , which we assume to be the unit hypercube in R d .

Original: In order to precisely introduce the problem, we recall some standard notions.
Revised: In order to precisely introduce the problem, we recall some standard notions.

Original: We consider our domain as Ω =
Revised: We consider our domain as Ω =

Original: [0 , 1] d and use L 2 (Ω) to denote the space of square integrable functions on Ω with respect to the Lebesgue measure.
Revised: [0 , 1] d and use L 2 (Ω) to denote the space of square integrable functions on Ω with respect to the Lebesgue measure.

Original: We let L ∞ (Ω) be the space of essentially bounded (with respect to the Lebesgue measure) functions on Ω and C ( ∂ Ω) denotes the space of continuous functions on ∂ Ω .
Revised: We let L ∞ (Ω) be the space of essentially bounded (with respect to the Lebesgue measure) functions on Ω and C ( ∂ Ω) denotes the space of continuous functions on ∂ Ω .

Original: Let f ∈ L 2 (Ω) , V ∈ L ∞ (Ω) , and , g ∈ L ∞ (Ω) .
Revised: Let f ∈ L 2 (Ω) , V ∈ L ∞ (Ω) , and , g ∈ L ∞ (Ω) .

Original: Our focus is on the analysis of Deep-Learning-based numerical methods to solve the elliptic PDE:
Revised: Our focus is on the analysis of Deep-Learning-based numerical methods to solve the elliptic PDE:

Original: [Equation]
Revised: [Equation]

Original: 2. L OSS F UNCTIONS FOR S OLVING PDE S AND I NDUCED E VALUATION M ETRIC
Revised: 2. L OSS F UNCTIONS FOR S OLVING PDE S AND I NDUCED E VALUATION M ETRIC

Original: In this paper, we mainly focus on analyzing Deep Ritz Methods (DRM) and Physics-Informed Neural
Revised: In this paper, we mainly focus on analyzing Deep Ritz Methods (DRM) and Physics-Informed Neural

Original: Network (PINN).
Revised: Network (PINN).

Original: In this subsection, we ﬁrst introduce the objective function and algorithm of the two methods.
Revised: In this subsection, we ﬁrst introduce the objective function and algorithm of the two methods.

Original: Deep Ritz Methods (E & Yu, 2018; Sirignano & Spiliopoulos, 2018) Recall that the equation 2. is equivalent to following variational form
Revised: Deep Ritz Methods (E & Yu, 2018; Sirignano & Spiliopoulos, 2018) Recall that the equation 2. is equivalent to following variational form

Original: [Equation]
Revised: [Equation]

Original: where u is minimized over H 10 (Ω) with boundary condition given by g on ∂ Ω .
Revised: where u is minimized over H 10 (Ω) with boundary condition given by g on ∂ Ω .

Original: This variational form provides the basis for the DRM type method for solving the static Schrödinger equation based on neural network ansatz.
Revised: This variational form provides the basis for the DRM type method for solving the static Schrödinger equation based on neural network ansatz.

Original: More speciﬁcally, the energy functional given in equation 2.2 is viewed as the population risk function to train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H 10 (Ω) .
Revised: More speciﬁcally, the energy functional given in equation 2.2 is viewed as the population risk function to train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H 10 (Ω) .

Original: In this paper, we also rely on the strong convexity of the DRM objective with respect to the H 1 norm.
Revised: In this paper, we also rely on the strong convexity of the DRM objective with respect to the H 1 norm.

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: For DRM, we further assume 0 < V min ≤ V ( x ) ≤ V max , then we have
Revised: For DRM, we further assume 0 < V min ≤ V ( x ) ≤ V max , then we have

Original: [Equation]
Revised: [Equation]

Original: holds for all u ∈ H 10 (Ω)
Revised: holds for all u ∈ H 10 (Ω)

Original: Physics-Informed Neural Network (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018).
Revised: Physics-Informed Neural Network (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018).

Original: PINN solves 2.1 via minimizing the following objective function
Revised: PINN solves 2.1 via minimizing the following objective function

Original: [Equation]
Revised: [Equation]

Original: The objective function E PINN can also be viewed as the population risk function and we can train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H 10 (Ω) .
Revised: The objective function E PINN can also be viewed as the population risk function and we can train an optimal estimator approximation of the solution to the PDE within a parameterized hypothesis function class F ⊂ H 10 (Ω) .

Original: In this paper, we also rely on the strong convexity of the PINN objective with respect to the H 2 norm, for which we need some additional assumptions on the potential.
Revised: In this paper, we also rely on the strong convexity of the PINN objective with respect to the H 2 norm, for which we need some additional assumptions on the potential.

Original: Proposition 2.2.
Revised: Proposition 2.2.

Original: For PINN, we further assume V ∈ L ∞ (Ω) with 0 < C min < V 2 − ∆ V, 0 <
Revised: For PINN, we further assume V ∈ L ∞ (Ω) with 0 < C min < V 2 − ∆ V, 0 <

Original: C min < V ( x ) ≤ V max and − ∆ V ( x ) ≤ V max , then we have for all u ∈ H 10 (Ω)
Revised: C min < V ( x ) ≤ V max and − ∆ V ( x ) ≤ V max , then we have for all u ∈ H 10 (Ω)

Original: [Equation]
Revised: [Equation]

Original: 2. E STIMATOR S ETTING
Revised: 2. E STIMATOR S ETTING

Original: Empirical Loss Minimization
Revised: Empirical Loss Minimization

Original: In order to access the d -dimensional integrals, DRM (E & Yu, 2018; Khoo et al., 2017) and PINN (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018) employ a Monte-Carlo method for computing the high dimensional integrals, which leads to the so-called empirical risk minimization training for neural networks.
Revised: In order to access the d -dimensional integrals, DRM (E & Yu, 2018; Khoo et al., 2017) and PINN (Raissi et al., 2019; Sirignano & Spiliopoulos, 2018) employ a Monte-Carlo method for computing the high dimensional integrals, which leads to the so-called empirical risk minimization training for neural networks.

Original: To deﬁne the empirical loss, let { X j } nj =1 be an i.i.d.
Revised: To deﬁne the empirical loss, let { X j } nj =1 be an i.i.d.

Original: sequence of random variables distributed according to the uniform distribution P Ω in domain Ω .
Revised: sequence of random variables distributed according to the uniform distribution P Ω in domain Ω .

Original: We also have access to noisy observations f
Revised: We also have access to noisy observations f

Original: j = f ( X j ) + ξ j (1 ≤ j ≤ n ) of the right hand side of the with and
Revised: j = f ( X j ) + ξ j (1 ≤ j ≤ n ) of the right hand side of the with and

Original: Deﬁne the empirical losses E n by setting
Revised: Deﬁne the empirical losses E n by setting

Original: [Equation]
Revised: [Equation]

Original: where | Ω | represents the Lebesgue measure of the domain.
Revised: where | Ω | represents the Lebesgue measure of the domain.

Original: Once given an empirical loss E n , we apply the empirical loss minimization to seek the estimation u n , i.e. u n = arg min u ∈ F E n ( u ) , where F is the parametrized hypothesis function space we consider.
Revised: Once given an empirical loss E n , we apply the empirical loss minimization to seek the estimation u n , i.e. u n = arg min u ∈ F E n ( u ) , where F is the parametrized hypothesis function space we consider.

Original: Some examples can be reproducing kernel Hilbert space (Chen et al., 2021b) and tensor training format (Richter et al., 2021;
Revised: Some examples can be reproducing kernel Hilbert space (Chen et al., 2021b) and tensor training format (Richter et al., 2021;

Original: Chen et al., 2021a).
Revised: Chen et al., 2021a).

Original: In this paper, we consider sparse neural network and truncated Fourier basis, which can achieve min-max optimal estimation rate for the non-parametric function estimation (Tsybakov, 2008;
Revised: In this paper, we consider sparse neural network and truncated Fourier basis, which can achieve min-max optimal estimation rate for the non-parametric function estimation (Tsybakov, 2008;

Original: Schmidt-Hieber et al., 2020; Farrell et al., 2021; Suzuki, 2018;
Revised: Schmidt-Hieber et al., 2020; Farrell et al., 2021; Suzuki, 2018;

Original: Chen et al., 2019b; Jiao et al., 2021c; Nitanda & Suzuki, 2020).
Revised: Chen et al., 2019b; Jiao et al., 2021c; Nitanda & Suzuki, 2020).

Original: Sparse Neural Network Function Space
Revised: Sparse Neural Network Function Space

Original: In this paper, the hypothesis function space F is expressed by the neural network function space following (Schmidt-Hieber et al., 2020; Suzuki, 2018;
Revised: In this paper, the hypothesis function space F is expressed by the neural network function space following (Schmidt-Hieber et al., 2020; Suzuki, 2018;

Original: Farrell et al., 2021).
Revised: Farrell et al., 2021).

Original: Let us denote the ReLU 3 activation by η 3 ( x ) = max { x 3 , 0 } ( x ∈ R ) , which is used in (E & Yu, 2018).
Revised: Let us denote the ReLU 3 activation by η 3 ( x ) = max { x 3 , 0 } ( x ∈ R ) , which is used in (E & Yu, 2018).

Original: For a vector x , η ( x ) is operated in an element-wise manner.
Revised: For a vector x , η ( x ) is operated in an element-wise manner.

Original: Deﬁne the space of all neural networks with height L , width W , sparsity constraint S and norm constraint B as Φ( L, W, S, B ) :=
Revised: Deﬁne the space of all neural networks with height L , width W , sparsity constraint S and norm constraint B as Φ( L, W, S, B ) :=

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: where ◦ denotes the function composition, (cid:107) · (cid:107) 

Revised: where ◦ denotes the function composition, (cid:107) · (cid:107)

Original: is the (cid:96) 0 -norm of the matrix (the number of non-zero elements of the matrix) and (cid:107) · (cid:107) ∞ , ∞ is the (cid:96) ∞ -norm of the matrix (maximum of the absolute values of the elements).
Revised: is the (cid:96) 0 -norm of the matrix (the number of non-zero elements of the matrix) and (cid:107) · (cid:107) ∞ , ∞ is the (cid:96) ∞ -norm of the matrix (maximum of the absolute values of the elements).

Original: Truncated Fourier Basis Estimator We also consider the Truncated Fourier basis as our estimator.
Revised: Truncated Fourier Basis Estimator We also consider the Truncated Fourier basis as our estimator.

Original: Denote the domain we are interested in by Ω ⊆ [0 , 1] d .
Revised: Denote the domain we are interested in by Ω ⊆ [0 , 1] d .

Original: For any z ∈ N d , we consider the corresponding Fourier basis function φ z ( x ) :=
Revised: For any z ∈ N d , we consider the corresponding Fourier basis function φ z ( x ) :=

Original: e 2 πi (cid:104)
Revised: e 2 πi (cid:104)

Original: z,x (cid:105) ( x ∈ Ω) .
Revised: z,x (cid:105) ( x ∈ Ω) .

Original: Any function f ∈ L 2 (Ω) can be represented as a weighted sum of the Fourier basis f ( x ) :
Revised: Any function f ∈ L 2 (Ω) can be represented as a weighted sum of the Fourier basis f ( x ) :

Original: = (cid:80) z
Revised: = (cid:80) z

Original: ∈ N d f z φ z ( x ) , where f z :=
Revised: ∈ N d f z φ z ( x ) , where f z :=

Original: (cid:82) Ω f ( x ) φ z ( x ) dx ( ∀ z ∈ N d ) is the Fourier coefﬁcient.
Revised: (cid:82) Ω f ( x ) φ z ( x ) dx ( ∀ z ∈ N d ) is the Fourier coefﬁcient.

Original: This inspires us to use the Fourier basis whose index lies in a truncated set Z ξ = { z ∈ Z|(cid:107)
Revised: This inspires us to use the Fourier basis whose index lies in a truncated set Z ξ = { z ∈ Z|(cid:107)

Original: z (cid:107) ∞ ≤ ξ } to represent the function class F as
Revised: z (cid:107) ∞ ≤ ξ } to represent the function class F as

Original: F ξ = { (cid:80) (cid:107) z (cid:107) ∞
Revised: F ξ = { (cid:80) (cid:107) z (cid:107) ∞

Original: ≤ ξ a z φ z | a z ∈ R , (cid:107) z (cid:107) ∞ ≤ ξ } .
Revised: ≤ ξ a z φ z | a z ∈ R , (cid:107) z (cid:107) ∞ ≤ ξ } .

Original:  L OWER B OUND
Revised:  L OWER B OUND

Original: In this section, we aim to consider the statistical limit of learning the solution of a PDE.
Revised: In this section, we aim to consider the statistical limit of learning the solution of a PDE.

Original: As discussed in Propositions 2.1 and 2.2, we directly consider the H 1 norm for DRM and H 2 norm for PINN as the evaluation metric.
Revised: As discussed in Propositions 2.1 and 2.2, we directly consider the H 1 norm for DRM and H 2 norm for PINN as the evaluation metric.

Original: The lower bounds are shown as follows.
Revised: The lower bounds are shown as follows.

Original: Theorem 3.1 (Lower bound) .
Revised: Theorem 3.1 (Lower bound) .

Original: We denote u ∗ ( f ) to be the solution of the PDE 2.1 and we can access randomly sampled data { X i , f i } i =1 , ··· ,n as described in Section 2.2.
Revised: We denote u ∗ ( f ) to be the solution of the PDE 2.1 and we can access randomly sampled data { X i , f i } i =1 , ··· ,n as described in Section 2.2.

Original: We further assume u ∗ ( f ) ∈ H s for a given s ∈ Z + .
Revised: We further assume u ∗ ( f ) ∈ H s for a given s ∈ Z + .

Original: Then we have the following lower bounds.
Revised: Then we have the following lower bounds.

Original: DRM Lower Bound.
Revised: DRM Lower Bound.

Original: For all estimators ψ : (cid:0) R d (cid:1) ⊗ n × R ⊗ n → H s (Ω) , we have
Revised: For all estimators ψ : (cid:0) R d (cid:1) ⊗ n × R ⊗ n → H s (Ω) , we have

Original: [Equation]
Revised: [Equation]

Original: PINN Lower Bound.
Revised: PINN Lower Bound.

Original: For all estimators ψ : (cid:0) R d (cid:1) ⊗ n × R ⊗ n → H s (Ω) , we have
Revised: For all estimators ψ : (cid:0) R d (cid:1) ⊗ n × R ⊗ n → H s (Ω) , we have

Original: [Equation]
Revised: [Equation]

Original: Given that n − 2( β − k ) d +2 β is the minimax rate of estimating the k -th derivative of a β -smooth density in L(Liu & Wang, 2012; Prakasa Rao, 1996; Müller & Gasser, 1979), the lower bound obtained here is the rate of estimating the right hand side function f in terms of the H − 1 norm.
Revised: Given that n − 2( β − k ) d +2 β is the minimax rate of estimating the k -th derivative of a β -smooth density in L(Liu & Wang, 2012; Prakasa Rao, 1996; Müller & Gasser, 1979), the lower bound obtained here is the rate of estimating the right hand side function f in terms of the H − 1 norm.

Original: Given the H − 1 norm error estimate on f , we can achieve an estimate of u , which provides an alternative way to understand our upper bound.
Revised: Given the H − 1 norm error estimate on f , we can achieve an estimate of u , which provides an alternative way to understand our upper bound.

Original: (See discussion in Appendix E.)
Revised: (See discussion in Appendix E.)

Original: The lower bound is non-standard, for the 2 s − 2 in the numerator is different from the 2 s − 4 in the denominator.
Revised: The lower bound is non-standard, for the 2 s − 2 in the numerator is different from the 2 s − 4 in the denominator.

Original:  U PPER B OUND
Revised:  U PPER B OUND

Original: To theoretically understand the empirical success of Physics-Informed Neural Networks and the
Revised: To theoretically understand the empirical success of Physics-Informed Neural Networks and the

Original: Deep Ritz solver, in this section, we aim to prove that the excess risk ∆ E n := E ( u n ) − E ( u ∗ ) of a well-trained neural networks on the PINN/DRM loss function will follow precise power-law scaling relations with the size of the training dataset.
Revised: Deep Ritz solver, in this section, we aim to prove that the excess risk ∆ E n := E ( u n ) − E ( u ∗ ) of a well-trained neural networks on the PINN/DRM loss function will follow precise power-law scaling relations with the size of the training dataset.

Original: Similar to (Xu, 2020; Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;b), we decompose the excess risk into approximation error and generalization error.
Revised: Similar to (Xu, 2020; Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;b), we decompose the excess risk into approximation error and generalization error.

Original: Different from the concurrent bound (Duan et al., 2021; Jiao et al., 2021a), we provide a fast rate O (1 /n ) by utilizing the strong convexity of the objective function established in Section 2.1 and achieve a faster and near optimal upper bound.
Revised: Different from the concurrent bound (Duan et al., 2021; Jiao et al., 2021a), we provide a fast rate O (1 /n ) by utilizing the strong convexity of the objective function established in Section 2.1 and achieve a faster and near optimal upper bound.

Original: We show that the generalization error can be bounded by the ﬁxed point ( i.e. the solution of φ ( r )
Revised: We show that the generalization error can be bounded by the ﬁxed point ( i.e. the solution of φ ( r )

Original: = r ) of the local Rademacher complexity
Revised: = r ) of the local Rademacher complexity

Original: [Equation]
Revised: [Equation]

Original: where R n is the Rademacher complexity, I ( u )
Revised: where R n is the Rademacher complexity, I ( u )

Original: =
Revised: =

Original: ∆ u + V u, (cid:107) · (cid:107) A = (cid:107) · (cid:107) H 2 for PINN and
Revised: ∆ u + V u, (cid:107) · (cid:107) A = (cid:107) · (cid:107) H 2 for PINN and

Original: I ( u ) =
Revised: I ( u ) =

Original: (cid:107)∇ u (cid:107) 2 + V u, (cid:107)·(cid:107)
Revised: (cid:107)∇ u (cid:107) 2 + V u, (cid:107)·(cid:107)

Original: A = (cid:107)·(cid:107) H 1 for DRM.
Revised: A = (cid:107)·(cid:107) H 1 for DRM.

Original: We put detailed deﬁnition and analysis in Appendix
Revised: We put detailed deﬁnition and analysis in Appendix

Original: B.4.
Revised: B.4.

Original: We ﬁrst provide a meta theorem to decompose the error into approximation and a fast rate generalization error.
Revised: We ﬁrst provide a meta theorem to decompose the error into approximation and a fast rate generalization error.

Original: Then we plug in the approximation and generalization error calculated in
Revised: Then we plug in the approximation and generalization error calculated in

Original: Appendix B.3 and Appendix B.2 and ﬁnally achieve the following upper bounds.
Revised: Appendix B.3 and Appendix B.2 and ﬁnally achieve the following upper bounds.

Original: We also put a more detailed proof sketch in Appendix A.2.
Revised: We also put a more detailed proof sketch in Appendix A.2.

Original: Physics Informed Neural Network.
Revised: Physics Informed Neural Network.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: (Informal Upper Bound of PINN with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ( L, W, S, B ) with parameters L
Revised: (Informal Upper Bound of PINN with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ( L, W, S, B ) with parameters L

Original: = O (1) , W = O ( n d d
Revised: = O (1) , W = O ( n d d

Original: +2 s − 4 ) , S = O ( n d d +2 s − 4 ) , B = O ( n d d +2 s − 4 ) , then the Physics
Revised: +2 s − 4 ) , S = O ( n d d +2 s − 4 ) , B = O ( n d d +2 s − 4 ) , then the Physics

Original: Informed estimator ˆ u DNNPINN = min u ∈ Φ( L,W,S,B ) E PINN n ( u ) satisﬁes the following upper bound with high probability
Revised: Informed estimator ˆ u DNNPINN = min u ∈ Φ( L,W,S,B ) E PINN n ( u ) satisﬁes the following upper bound with high probability

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: (Informal Upper Bound of PINN with Truncated Fourier Series Estimator) With proper assumptions, consider the Physics Informed Neural Network objective with a plug-in Fourier Series estimator ˆ u FourierPINN = min u ∈ F ξ (Ω) E PINN n ( u ) with ξ = Θ( n 1 d +2 s − 4 ) , then with high probability we have
Revised: (Informal Upper Bound of PINN with Truncated Fourier Series Estimator) With proper assumptions, consider the Physics Informed Neural Network objective with a plug-in Fourier Series estimator ˆ u FourierPINN = min u ∈ F ξ (Ω) E PINN n ( u ) with ξ = Θ( n 1 d +2 s − 4 ) , then with high probability we have

Original: [Equation]
Revised: [Equation]

Original: Deep Ritz Methods.
Revised: Deep Ritz Methods.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: (Informal Upper Bound of DRM with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ( L, W, S, B ) with parameters
Revised: (Informal Upper Bound of DRM with Deep Neural Network Estimator) With proper assumptions, consider the sparse Deep Neural Network function space Φ( L, W, S, B ) with parameters

Original: L = O (1) , W = O ( n d d +2 s − 2 ) , S = O ( n d d +2 s − 2 ) , B = O ( n d d +2 s − 2 ) , then the Deep ritz estimator ˆ u DNNDRM = min u ∈ Φ( L,W,S,B ) E DRM n ( u ) satisﬁes the following upper bound with high probability
Revised: L = O (1) , W = O ( n d d +2 s − 2 ) , S = O ( n d d +2 s − 2 ) , B = O ( n d d +2 s − 2 ) , then the Deep ritz estimator ˆ u DNNDRM = min u ∈ Φ( L,W,S,B ) E DRM n ( u ) satisﬁes the following upper bound with high probability

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: (Informal Upper Bound of DRM with Truncated Fourier Series Estimator) With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator ˆ u FourierDRM =
Revised: (Informal Upper Bound of DRM with Truncated Fourier Series Estimator) With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator ˆ u FourierDRM =

Original: min u ∈ F ξ (Ω) E DRM n ( u ) with ξ = Θ( n 1 d +2 s − 2 ) , then with high probability we have
Revised: min u ∈ F ξ (Ω) E DRM n ( u ) with ξ = Θ( n 1 d +2 s − 2 ) , then with high probability we have

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: • There is a common belief that Machine learning based PDE solvers can break the curse of dimensionality (E & Yu, 2018;
Revised: • There is a common belief that Machine learning based PDE solvers can break the curse of dimensionality (E & Yu, 2018;

Original: Grohs et al., 2018; Lanthaler et al., 2021).
Revised: Grohs et al., 2018; Lanthaler et al., 2021).

Original: However, we obtained an n − 2 s − 2 2 s − 4+ d convergence rate, which can become super slow in high dimension.
Revised: However, we obtained an n − 2 s − 2 2 s − 4+ d convergence rate, which can become super slow in high dimension.

Original: Our analysis showed that it is essential to constrain the function space to break the curse of dimensionality.
Revised: Our analysis showed that it is essential to constrain the function space to break the curse of dimensionality.

Original: (Lu et al., 2021b) considered the DRM in Barron spaces.
Revised: (Lu et al., 2021b) considered the DRM in Barron spaces.

Original: (Ongie et al., 2019) showed that functions in the Barron space enjoy a smoothness s at the same magnitude as d , which will also lead to convergence rate independent of the dimension using our upper bound.
Revised: (Ongie et al., 2019) showed that functions in the Barron space enjoy a smoothness s at the same magnitude as d , which will also lead to convergence rate independent of the dimension using our upper bound.

Original: Neural network can also approximate mixed sparse grid spaces (Montanelli & Du, 2019; Suzuki, 2018) and functions on manifold (Nitanda & Suzuki, 2020; Chen et al., 2019b) without curse of dimensionality.
Revised: Neural network can also approximate mixed sparse grid spaces (Montanelli & Du, 2019; Suzuki, 2018) and functions on manifold (Nitanda & Suzuki, 2020; Chen et al., 2019b) without curse of dimensionality.

Original: Combined with these approximation bounds, we can also achieve a bound that breaks the curse of dimensionality using Theorem B.12 and B.9.
Revised: Combined with these approximation bounds, we can also achieve a bound that breaks the curse of dimensionality using Theorem B.12 and B.9.

Original: In this paper, we aim to consider the statistical power of the loss function in common function spaces and leave the curse of dimensionality as a separate topic.
Revised: In this paper, we aim to consider the statistical power of the loss function in common function spaces and leave the curse of dimensionality as a separate topic.

Original: • Our bound is faster than the concurrent bound (Duan et al., 2021;
Revised: • Our bound is faster than the concurrent bound (Duan et al., 2021;

Original: Jiao et al., 2021a) for we provided a fast rate O (1 /n ) by utilizing the strong convexity of the objective function and improved the convergence rate from n − 2 s − 2 d +4 s − 4 to n − 2 s − 2 d
Revised: Jiao et al., 2021a) for we provided a fast rate O (1 /n ) by utilizing the strong convexity of the objective function and improved the convergence rate from n − 2 s − 2 d +4 s − 4 to n − 2 s − 2 d

Original: +2 s − 2 for Deep Ritz and from n − 2 s − 4 d +4 s − 8 to n − 2 s − 4 d +2 s − 4 for PINN.
Revised: +2 s − 2 for Deep Ritz and from n − 2 s − 4 d +4 s − 8 to n − 2 s − 4 d +2 s − 4 for PINN.

Original: Compared to the lower bound provided in Section 3, our bounds for PINN is near-optimal while the upper bound for DRM is sub-optimal.
Revised: Compared to the lower bound provided in Section 3, our bounds for PINN is near-optimal while the upper bound for DRM is sub-optimal.

Original: We believe our bound is tight and put the discussion in Appendix E. We’ll propose a modiﬁed version of DRM to match the upper and lower bound in the next section.
Revised: We believe our bound is tight and put the discussion in Appendix E. We’ll propose a modiﬁed version of DRM to match the upper and lower bound in the next section.

Original: • For upper bound of DRM, due to a technical issue, we assumed that the observation we access is
Revised: • For upper bound of DRM, due to a technical issue, we assumed that the observation we access is

Original: clean, i.e f i = f
Revised: clean, i.e f i = f

Original: ( X i ) .
Revised: ( X i ) .

Original: We conjecture that add noising on observation will not effect the rate and leave this to future work.
Revised: We conjecture that add noising on observation will not effect the rate and leave this to future work.

Original:  M ODIFIED D EEP R ITZ M ETHODS
Revised:  M ODIFIED D EEP R ITZ M ETHODS

Original: Comparing the lower bound in Section 3 and the upper bound in Section 4, we ﬁnd out that
Revised: Comparing the lower bound in Section 3 and the upper bound in Section 4, we ﬁnd out that

Original: the
Revised: the

Original: Physics-Informed Neural Network achieves min-max optimality while the Deep Ritz Method does not.
Revised: Physics-Informed Neural Network achieves min-max optimality while the Deep Ritz Method does not.

Original: In this section, we propose a modiﬁed version of Deep Ritz which can be statistically optimal.
Revised: In this section, we propose a modiﬁed version of Deep Ritz which can be statistically optimal.

Original: As discussed in Appendix E, the reason behind the suboptimality of DRM comes from the high complexity introduced via the uniform concentration bound of the gradient term in the variational form.
Revised: As discussed in Appendix E, the reason behind the suboptimality of DRM comes from the high complexity introduced via the uniform concentration bound of the gradient term in the variational form.

Original: At the same time, we further observed that the (cid:82) (cid:107)∇ u (cid:107) 2 dx does not require any query from the right hand side function f , which means that we can easily make another splitted sample to approximate the (cid:82) (cid:107)∇ u (cid:107) 2 dx term more precisely.
Revised: At the same time, we further observed that the (cid:82) (cid:107)∇ u (cid:107) 2 dx does not require any query from the right hand side function f , which means that we can easily make another splitted sample to approximate the (cid:82) (cid:107)∇ u (cid:107) 2 dx term more precisely.

Original: [Equation]
Revised: [Equation]

Original: Once we sampled more data for approximating (cid:82) |∇ u | 2 dx , we can achieve an near optimal bound for the Truncated Fourier Estimator when Nn (cid:38) n
Revised: Once we sampled more data for approximating (cid:82) |∇ u | 2 dx , we can achieve an near optimal bound for the Truncated Fourier Estimator when Nn (cid:38) n

Original: [Equation]
Revised: [Equation]

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: (Informal Upper Bound of DRM with Truncated Fourier Series Estimator)With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator ˆ u FourierMDRM = min u ∈ F ξ (Ω) E MDRM N,n ( u ) with ξ = Θ( n
Revised: (Informal Upper Bound of DRM with Truncated Fourier Series Estimator)With proper assumptions, consider the Deep Ritz objective with a plug in Fourier Series estimator ˆ u FourierMDRM = min u ∈ F ξ (Ω) E MDRM N,n ( u ) with ξ = Θ( n

Original: [Equation]
Revised: [Equation]

Original: and Nn (cid:38) n
Revised: and Nn (cid:38) n

Original: [Equation]
Revised: [Equation]

Original: 2 s − 4 , then we have
Revised: 2 s − 4 , then we have

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: We still cannot achieve optimal rate for neural network even with modiﬁed DRM methods.
Revised: We still cannot achieve optimal rate for neural network even with modiﬁed DRM methods.

Original: The reason is because the number of neurons is not a good complexity measure for the gradient of the function.
Revised: The reason is because the number of neurons is not a good complexity measure for the gradient of the function.

Original: Thus, the bound for ψ ( r ) =
Revised: Thus, the bound for ψ ( r ) =

Original: R n ( {I ( u ) | (cid:107) u − u ∗ (cid:107) 2 H 1 ≤ r } ) is not enough for achieving optimal convergence rate.
Revised: R n ( {I ( u ) | (cid:107) u − u ∗ (cid:107) 2 H 1 ≤ r } ) is not enough for achieving optimal convergence rate.

Original: However, following (Schmidt-Hieber et al., 2020; Suzuki, 2018; Imaizumi & Fukumizu, 2020; Chen et al., 2019b; Farrell et al., 2021) to use deep networks for estimating functions, we optimize the best neural network with constrained sparsity in our paper.
Revised: However, following (Schmidt-Hieber et al., 2020; Suzuki, 2018; Imaizumi & Fukumizu, 2020; Chen et al., 2019b; Farrell et al., 2021) to use deep networks for estimating functions, we optimize the best neural network with constrained sparsity in our paper.

Original: Here we conjecture that there exists a computable complexity measure that can make DRM statistically optimal and leave ﬁnding the right complexity of the neural network’s gradient to be future work.
Revised: Here we conjecture that there exists a computable complexity measure that can make DRM statistically optimal and leave ﬁnding the right complexity of the neural network’s gradient to be future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct several numerical experiments to verify our theory.
Revised: In this section, we conduct several numerical experiments to verify our theory.

Original: We follow the neural network and hyper-parameter setting in (Chen et al., 2020).
Revised: We follow the neural network and hyper-parameter setting in (Chen et al., 2020).

Original: Due to the page limit, we only put the experiments for Deep Ritz Methods here.
Revised: Due to the page limit, we only put the experiments for Deep Ritz Methods here.

Original: 6. T HE M ODIFIED D EEP R ITZ M ETHODS
Revised: 6. T HE M ODIFIED D EEP R ITZ M ETHODS

Original: In this section, we conduct experiments which substantiate our theoretical results for modiﬁed
Revised: In this section, we conduct experiments which substantiate our theoretical results for modiﬁed

Original: Deep
Revised: Deep

Original: Ritz methods.
Revised: Ritz methods.

Original: For simplicity, we take V ( x ) = 1 in our experiment.
Revised: For simplicity, we take V ( x ) = 1 in our experiment.

Original: We conduct experiment in 2-dimension and select the solution of the PDE as u ∗ =
Revised: We conduct experiment in 2-dimension and select the solution of the PDE as u ∗ =

Original: (cid:80) z (cid:107) z (cid:107) − s φ z ( x ) ∈ H s .
Revised: (cid:80) z (cid:107) z (cid:107) − s φ z ( x ) ∈ H s .

Original: We show the log-log plot of the H 1 loss against the number of sampled data for s = 4 in Figure 1.
Revised: We show the log-log plot of the H 1 loss against the number of sampled data for s = 4 in Figure 1.

Original: We use an OLS estimator to ﬁt the log-log plot and put the estimated slope and corresponding R 2 score in Figure 1.
Revised: We use an OLS estimator to ﬁt the log-log plot and put the estimated slope and corresponding R 2 score in Figure 1.

Original: As our theory predicts, the modiﬁed Deep Ritz Method converges faster than the original one.
Revised: As our theory predicts, the modiﬁed Deep Ritz Method converges faster than the original one.

Original: All the derivation of the two estimators is listed in Appendix E.
Revised: All the derivation of the two estimators is listed in Appendix E.

Original: 6. D IMENSION D EPENDENT S CALING L AW .
Revised: 6. D IMENSION D EPENDENT S CALING L AW .

Original: [Figure]
Revised: [Figure]

Original: We conduct experiments to illustrate that the population loss of well-trained and well-tuned
Revised: We conduct experiments to illustrate that the population loss of well-trained and well-tuned

Original: Deep Ritz method will scale with the ddimensional training data number N as a powerlaw L ∝ 1 N α .
Revised: Deep Ritz method will scale with the ddimensional training data number N as a powerlaw L ∝ 1 N α .

Original: We also scan over a range of d and α and verify an approximately α ∝ 1 d scaling law as our theory suggests.
Revised: We also scan over a range of d and α and verify an approximately α ∝ 1 d scaling law as our theory suggests.

Original: We use the same test function in Section 6.1 as the solution of our
Revised: We use the same test function in Section 6.1 as the solution of our

Original: PDE.
Revised: PDE.

Original: For simplicity, we take V ( x ) = 1 in our experiment.
Revised: For simplicity, we take V ( x ) = 1 in our experiment.

Original: We train the deep Ritz method on 20, 80, 320, 1280, 10240 sampled data points for 5,6,7,8,9,10 dimensional problems and we plot our results on the log-log scale.
Revised: We train the deep Ritz method on 20, 80, 320, 1280, 10240 sampled data points for 5,6,7,8,9,10 dimensional problems and we plot our results on the log-log scale.

Original: Results are shown in Figure 2.
Revised: Results are shown in Figure 2.

Original: We discover the L ∝ n 1 d +2 scaling law in practical situations.
Revised: We discover the L ∝ n 1 d +2 scaling law in practical situations.

Original: 11 'LPHQVLRQ        0 X OWL S OL FD WL Y H
Revised: 11 'LPHQVLRQ        0 X OWL S OL FD WL Y H

Original:  , QY H U V H  R I   FD OL QJ  / D Z  & R H II L F L H Q W  1 / 0 2 

Revised:  , QY H U V H  R I   FD OL QJ  / D Z  & R H II L F L H Q W  1 / 0 2

Original: 6 8 Log of Training Data Number.
Revised: 6 8 Log of Training Data Number.

Original: 5-Dimension 1.
Revised: 5-Dimension 1.

Original: L o g
Revised: L o g

Original: o f R e l a t i v e T e s t i n g L o ss .
Revised: o f R e l a t i v e T e s t i n g L o ss .

Original: 0 2 4 6 8 Log of Training Data Number.
Revised: 0 2 4 6 8 Log of Training Data Number.

Original: 6-Dimension 0.
Revised: 6-Dimension 0.

Original: L o g
Revised: L o g

Original: o f R e l a t i v e T e s t i n g L o ss .
Revised: o f R e l a t i v e T e s t i n g L o ss .

Original: 0 2 4 6 8 Log of Training Data Number.
Revised: 0 2 4 6 8 Log of Training Data Number.

Original: 7-Dimension 0.6 0.
Revised: 7-Dimension 0.6 0.

Original: L o g
Revised: L o g

Original: o f R e l a t i v e T e s t i n g L o ss .
Revised: o f R e l a t i v e T e s t i n g L o ss .

Original: 0 2 4 6 8 Log of Training Data Number.
Revised: 0 2 4 6 8 Log of Training Data Number.

Original: 8-Dimension 0.6 0.
Revised: 8-Dimension 0.6 0.

Original: L o g
Revised: L o g

Original: o f R e l a t i v e T e s t i n g L o ss .
Revised: o f R e l a t i v e T e s t i n g L o ss .

Original: 0 2 4 6 8 Log of Training Data Number.
Revised: 0 2 4 6 8 Log of Training Data Number.

Original: 9-Dimension 0.
Revised: 9-Dimension 0.

Original: L o g
Revised: L o g

Original: o f R e l a t i v e T e s t i n g L o ss .
Revised: o f R e l a t i v e T e s t i n g L o ss .

Original: 0 2 4 6 8 Log of Training Data Number.
Revised: 0 2 4 6 8 Log of Training Data Number.

Original: 10-Dimension 0.
Revised: 10-Dimension 0.

Original: L o g
Revised: L o g

Original: o f R e l a t i v e T e s t i n g L o ss .
Revised: o f R e l a t i v e T e s t i n g L o ss .

Original: 6. A DAPTATION T O T HE S IMPLER F UNCTIONS .
Revised: 6. A DAPTATION T O T HE S IMPLER F UNCTIONS .

Original: (Sharma & Kaplan, 2020) showed that the neural scaling law will adapt to the structure that the target function enjoys.
Revised: (Sharma & Kaplan, 2020) showed that the neural scaling law will adapt to the structure that the target function enjoys.

Original: This adaptivity enables the neural network to break the cure of the dimensionality for simple functions in high dimension.
Revised: This adaptivity enables the neural network to break the cure of the dimensionality for simple functions in high dimension.

Original: (Suzuki & Nitanda, 2019; Chen et al., 2019a) also observed this theoretically.
Revised: (Suzuki & Nitanda, 2019; Chen et al., 2019a) also observed this theoretically.

Original: For solving PDEs, we also observed this adaptivity in practice.
Revised: For solving PDEs, we also observed this adaptivity in practice.

Original: Here we tested the following two hypothesis
Revised: Here we tested the following two hypothesis

Original: • Random Neural Network Teacher.
Revised: • Random Neural Network Teacher.

Original: Following (Sharma & Kaplan, 2020), we also tested random neural network using He initialization (He et al., 2015) as the ground turth solution u ∗ .
Revised: Following (Sharma & Kaplan, 2020), we also tested random neural network using He initialization (He et al., 2015) as the ground turth solution u ∗ .

Original: (De Palma et al., 2018) showed that random deep neural networks are biased towards simple functions and in practice we observed a scaling law at the parametric rate.
Revised: (De Palma et al., 2018) showed that random deep neural networks are biased towards simple functions and in practice we observed a scaling law at the parametric rate.

Original: Speciﬁcally, we obtained a linear estimate with slope α = − 0 .
Revised: Speciﬁcally, we obtained a linear estimate with slope α = − 0 .

Original: 50679429 and a R 2 score = 0 .
Revised: 50679429 and a R 2 score = 0 .

Original: 96 in the log-log plot.
Revised: 96 in the log-log plot.

Original: See Figure 3(a).
Revised: See Figure 3(a).

Original: • Simple Polynomials.
Revised: • Simple Polynomials.

Original: Neural network can approximate simple polynomials exponentially fast (Wang et al., 2018).
Revised: Neural network can approximate simple polynomials exponentially fast (Wang et al., 2018).

Original: Thus, we select the ground truth solution to be the following simple polynomial in 10 dimensional spaces u ∗ ( x ) = x 1 x 2 + · · · + x 9 x 10 .
Revised: Thus, we select the ground truth solution to be the following simple polynomial in 10 dimensional spaces u ∗ ( x ) = x 1 x 2 + · · · + x 9 x 10 .

Original: In this example, we obtained a linear estimate with slope α = − 0 .
Revised: In this example, we obtained a linear estimate with slope α = − 0 .

Original: 49755418 and a R 2 score = 0 .
Revised: 49755418 and a R 2 score = 0 .

Original: 99 in the log-log plot.
Revised: 99 in the log-log plot.

Original: See Figure 3(b).
Revised: See Figure 3(b).

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: L og o f R e l a t i v e T e s t i n g
Revised: L og o f R e l a t i v e T e s t i n g

Original: L o ss
Revised: L o ss

Original: Log of Training Data Number.
Revised: Log of Training Data Number.

Original: (a) Random Neural Network Teacher (b) y = x 1 x 2 + x 3 x 4 +⋯+ x 9 x
Revised: (a) Random Neural Network Teacher (b) y = x 1 x 2 + x 3 x 4 +⋯+ x 9 x

Original: Log of Training Data Number.
Revised: Log of Training Data Number.

Original: L og o f R e l a t i v e T e s t i n g
Revised: L og o f R e l a t i v e T e s t i n g

Original: L o ss
Revised: L o ss

Original: Conclusion In this paper, We considered the statistical min-max optimality of solving a PDE from random samples.
Revised: Conclusion In this paper, We considered the statistical min-max optimality of solving a PDE from random samples.

Original: We improved the previous bounds (Xu, 2020;
Revised: We improved the previous bounds (Xu, 2020;

Original: Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a) by
Revised: Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a) by

Original: providing the ﬁrst fast rate generalization bound for learning PDE solutions via the strongly convex nature of the two objective functions.
Revised: providing the ﬁrst fast rate generalization bound for learning PDE solutions via the strongly convex nature of the two objective functions.

Original: We achieved the optimal rate via the PINN and a modiﬁed Deep Ritz method.
Revised: We achieved the optimal rate via the PINN and a modiﬁed Deep Ritz method.

Original: We veriﬁed our theory via numerical experiments and explored the dimension dependent scaling laws of Deep PDE solvers.
Revised: We veriﬁed our theory via numerical experiments and explored the dimension dependent scaling laws of Deep PDE solvers.

Original: Discussion and Future Work Here we discuss several drawbacks of our theory
Revised: Discussion and Future Work Here we discuss several drawbacks of our theory

Original: • We restricted our target function and estimators in W 1 , ∞ instead of H 1 due to boundedness assumption made in the local Rademacher complexity arguments.
Revised: • We restricted our target function and estimators in W 1 , ∞ instead of H 1 due to boundedness assumption made in the local Rademacher complexity arguments.

Original: However, typical functional used in physics is always unbounded, such as the Newtonian potential
Revised: However, typical functional used in physics is always unbounded, such as the Newtonian potential

Original: [Table]
Revised: [Table]

Original: which limits the application of our theory.
Revised: which limits the application of our theory.

Original: • This paper did not discuss any optimization aspect of the deep PDE solvers and always assumed that global optimum can be achieved.
Revised: • This paper did not discuss any optimization aspect of the deep PDE solvers and always assumed that global optimum can be achieved.

Original: However, it is important to investigate whether the optimization error (Suzuki & Akiyama, 2020; Chizat, 2021) will ﬁnally dominate.
Revised: However, it is important to investigate whether the optimization error (Suzuki & Akiyama, 2020; Chizat, 2021) will ﬁnally dominate.

Original: • Instead of solving a single PDE, recent works (Long et al., 2018; 2019;
Revised: • Instead of solving a single PDE, recent works (Long et al., 2018; 2019;

Original: Li et al., 2020; Lanthaler et al., 2021; Bhattacharya et al., 2020; Fan & Ying, 2020; Feliu-Faba et al., 2020) considered the so-called "operator learning", which aims to learn a family of PDE/inverse problems using a single network.
Revised: Li et al., 2020; Lanthaler et al., 2021; Bhattacharya et al., 2020; Fan & Ying, 2020; Feliu-Faba et al., 2020) considered the so-called "operator learning", which aims to learn a family of PDE/inverse problems using a single network.

Original: It is interesting to investigate the generalization bound and neural scaling law there.
Revised: It is interesting to investigate the generalization bound and neural scaling law there.

Original: • We ﬁnd out that the sparsity of the neural network is not a good complexity measure of neural network’s gradient.
Revised: • We ﬁnd out that the sparsity of the neural network is not a good complexity measure of neural network’s gradient.

Original: We conjecture that there exists an oracle complexity measure, whose approximation and generalization bounds can lead Modiﬁed DRM to achieve the optimal convergence rate.
Revised: We conjecture that there exists an oracle complexity measure, whose approximation and generalization bounds can lead Modiﬁed DRM to achieve the optimal convergence rate.

Original: Appendix A A PPENDIX O RGANIZATION AND P ROOF S KETCH A. N OTATIONS
Revised: Appendix A A PPENDIX O RGANIZATION AND P ROOF S KETCH A. N OTATIONS

Original: In this section, we provide all the notations we need in the proof.
Revised: In this section, we provide all the notations we need in the proof.

Original: Let Ω ⊂ R d be some open set.
Revised: Let Ω ⊂ R d be some open set.

Original: We denote C (Ω) the space of continuous functions on Ω and C k (Ω) the space of all functions that are k times continuously differentiable on Ω ( ∀ k ∈ Z + ).
Revised: We denote C (Ω) the space of continuous functions on Ω and C k (Ω) the space of all functions that are k times continuously differentiable on Ω ( ∀ k ∈ Z + ).

Original: For any n ∈ N 0 ( N 0 :
Revised: For any n ∈ N 0 ( N 0 :

Original: = Z + ∪ { 0 } is the set of all non-negative integers) and 1 ≤ p ≤ ∞ , we deﬁne the Sobolev space W n,p (Ω) by
Revised: = Z + ∪ { 0 } is the set of all non-negative integers) and 1 ≤ p ≤ ∞ , we deﬁne the Sobolev space W n,p (Ω) by

Original: [Equation]
Revised: [Equation]

Original: In particular, when p = 2 , we deﬁne H n (Ω) :
Revised: In particular, when p = 2 , we deﬁne H n (Ω) :

Original: = W n, 2 (Ω) for any n ∈ N 0 .
Revised: = W n, 2 (Ω) for any n ∈ N 0 .

Original: Moreover, for any f ∈ W n,p (Ω) with 1 ≤ p < ∞ , we deﬁne the Sobolev norm by:
Revised: Moreover, for any f ∈ W n,p (Ω) with 1 ≤ p < ∞ , we deﬁne the Sobolev norm by:

Original: [Equation]
Revised: [Equation]

Original: In particular, when p = ∞ , we have:
Revised: In particular, when p = ∞ , we have:

Original: [Equation]
Revised: [Equation]

Original: Consider the Fourier expansion f := (cid:80) z ∈ N d f z φ z ( x ) of the function f ∈ W n,p (Ω) .
Revised: Consider the Fourier expansion f := (cid:80) z ∈ N d f z φ z ( x ) of the function f ∈ W n,p (Ω) .

Original: We can equivalently express the Sobolev norm as:
Revised: We can equivalently express the Sobolev norm as:

Original: [Equation]
Revised: [Equation]

Original: where f z = (cid:82) Ω f ( x ) φ z ( x ) dx =
Revised: where f z = (cid:82) Ω f ( x ) φ z ( x ) dx =

Original: (cid:82) Ω f ( x ) e − 2 πi (cid:104)
Revised: (cid:82) Ω f ( x ) e − 2 πi (cid:104)

Original: z,x (cid:105) dx ( x ∈ Ω) is the z −
Revised: z,x (cid:105) dx ( x ∈ Ω) is the z −

Original: th Fourier coefﬁcient of f .
Revised: th Fourier coefﬁcient of f .

Original: Moreover, we use W 1 ,p 0 (Ω) to denote the closure of C 1 c (Ω) in W 1 ,p (Ω) .
Revised: Moreover, we use W 1 ,p 0 (Ω) to denote the closure of C 1 c (Ω) in W 1 ,p (Ω) .

Original: In particular, when p = 2 , we deﬁne H 10 (Ω) :
Revised: In particular, when p = 2 , we deﬁne H 10 (Ω) :

Original: = W 1 , 2 0 (Ω) .
Revised: = W 1 , 2 0 (Ω) .

Original: Furthermore, we use (cid:107) · (cid:107) to present the vector 2 norm and, given a data sample { X i } ni =1 ⊂ Ω , (cid:107) · (cid:107) n,p = ( E n · p ) 1 /p denote the empirical p norm, where E n : L 2 (Ω) → R is the corresponding empirical average operator deﬁned as E n f := 1 n (cid:80) ni =1 f ( X i ) , ∀ f ∈ L 2 (Ω) .
Revised: Furthermore, we use (cid:107) · (cid:107) to present the vector 2 norm and, given a data sample { X i } ni =1 ⊂ Ω , (cid:107) · (cid:107) n,p = ( E n · p ) 1 /p denote the empirical p norm, where E n : L 2 (Ω) → R is the corresponding empirical average operator deﬁned as E n f := 1 n (cid:80) ni =1 f ( X i ) , ∀ f ∈ L 2 (Ω) .

Original: Given two quantities
Revised: Given two quantities

Original: X and Y , we write X (cid:46) Y when the inequality X ≤ CY holds, where C is some constant.
Revised: X and Y , we write X (cid:46) Y when the inequality X ≤ CY holds, where C is some constant.

Original: For two functions f and g mapping from R + to R , we write f = O ( g ) when there exist two constants C (cid:48) and x 0 independent of f and g , such that the inequality f ( x ) ≤ C (cid:48) g
Revised: For two functions f and g mapping from R + to R , we write f = O ( g ) when there exist two constants C (cid:48) and x 0 independent of f and g , such that the inequality f ( x ) ≤ C (cid:48) g

Original: ( x ) holds for any x ≥ x 0 .
Revised: ( x ) holds for any x ≥ x 0 .

Original: We use
Revised: We use

Original: X (cid:39) Y to denote X (cid:46) Y and Y (cid:46) X .
Revised: X (cid:39) Y to denote X (cid:46) Y and Y (cid:46) X .

Original: A. A PPENDIX O RGANIZATION AND P ROOF S KETCH
Revised: A. A PPENDIX O RGANIZATION AND P ROOF S KETCH

Original: In this section, we list the organization of the appendix and put a more detailed proof sketch of our main results.
Revised: In this section, we list the organization of the appendix and put a more detailed proof sketch of our main results.

Original: We put all the proof of upper bounds in Appendix B and all the proof of lower bounds in Appendix D. All the proof of the results about modiﬁed Deep Ritz method is given in Appendix C.
Revised: We put all the proof of upper bounds in Appendix B and all the proof of lower bounds in Appendix D. All the proof of the results about modiﬁed Deep Ritz method is given in Appendix C.

Original: The proof of lower bounds is based on standard Fano method.
Revised: The proof of lower bounds is based on standard Fano method.

Original: In this section, we focus on the proof sketch of the fast rate upper bound.
Revised: In this section, we focus on the proof sketch of the fast rate upper bound.

Original: Error Decomposition.
Revised: Error Decomposition.

Original: We ﬁrst decompose the excess risk ∆ E n := E ( u n ) − E ( u ∗ ) of a welltrained neural network on the PINN/DRM loss function into approximation error and generalization error, similar to (Xu, 2020;
Revised: We ﬁrst decompose the excess risk ∆ E n := E ( u n ) − E ( u ∗ ) of a welltrained neural network on the PINN/DRM loss function into approximation error and generalization error, similar to (Xu, 2020;

Original: Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;b).
Revised: Lu et al., 2021b; Duan et al., 2021; Jiao et al., 2021a;b).

Original: The regularity results used in the decomposition are proved in Appendix B.1.
Revised: The regularity results used in the decomposition are proved in Appendix B.1.

Original: Explicitly, for any u F ∈ F (Ω) , we can decompose the excess risk as
Revised: Explicitly, for any u F ∈ F (Ω) , we can decompose the excess risk as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is on uniformly sampled data, F (Ω) is the space of parametrized estimators we used like truncated Fourier series or sparse neural networks, ˆ u is the minimizer of the empirical loss E n in F (Ω) and u ∗ is the minimizer of the population loss E (i.e, ground truth solution).
Revised: where the expectation is on uniformly sampled data, F (Ω) is the space of parametrized estimators we used like truncated Fourier series or sparse neural networks, ˆ u is the minimizer of the empirical loss E n in F (Ω) and u ∗ is the minimizer of the population loss E (i.e, ground truth solution).

Original: The inequality in the third line follows from the fact that ˆ u is the minimizer of the empirical loss E n in the space F (Ω) , which implies E n (ˆ u ) ≤ E n ( u F ) .
Revised: The inequality in the third line follows from the fact that ˆ u is the minimizer of the empirical loss E n in the space F (Ω) , which implies E n (ˆ u ) ≤ E n ( u F ) .

Original: We call the ﬁrst term generalization error as it’s measuring the difference between E n and E .
Revised: We call the ﬁrst term generalization error as it’s measuring the difference between E n and E .

Original: We call the second term approximation error as it seeks for a parametrized estimator u F that approximates the ground truth solution u ∗ well in F (Ω) .
Revised: We call the second term approximation error as it seeks for a parametrized estimator u F that approximates the ground truth solution u ∗ well in F (Ω) .

Original: The upper bounds on generalization and approximation error that we achieved in this paper are listed in
Revised: The upper bounds on generalization and approximation error that we achieved in this paper are listed in

Original: Table 2.
Revised: Table 2.

Original: Let n denote the number of sampled datapoints.
Revised: Let n denote the number of sampled datapoints.

Original: For the generalization error, different from the concurrent upper bound O ( 1 √ n ) (Duan et al., 2021; Jiao et al., 2021a), we provide a faster and near optimal upper bound O ( 1 n ) by utilizing the strong convexity of the objective function established in
Revised: For the generalization error, different from the concurrent upper bound O ( 1 √ n ) (Duan et al., 2021; Jiao et al., 2021a), we provide a faster and near optimal upper bound O ( 1 n ) by utilizing the strong convexity of the objective function established in

Original: Appendix B.1.
Revised: Appendix B.1.

Original: Via using the Peeling Lemma (Lemma B.4, for completeness, we also provide a proof), we show that the generalization error can be bounded by the ﬁxed point of the local Rademacher complexity
Revised: Via using the Peeling Lemma (Lemma B.4, for completeness, we also provide a proof), we show that the generalization error can be bounded by the ﬁxed point of the local Rademacher complexity

Original: [Equation]
Revised: [Equation]

Original: where R n is the Rademacher complexity, I ( u )
Revised: where R n is the Rademacher complexity, I ( u )

Original: =
Revised: =

Original: ∆ u + V u, (cid:107) · (cid:107) A = (cid:107) · (cid:107) H 2 for PINN and
Revised: ∆ u + V u, (cid:107) · (cid:107) A = (cid:107) · (cid:107) H 2 for PINN and

Original: I ( u ) =
Revised: I ( u ) =

Original: (cid:107)∇ u (cid:107) 2 + V u, (cid:107) · (cid:107)
Revised: (cid:107)∇ u (cid:107) 2 + V u, (cid:107) · (cid:107)

Original: A = (cid:107) · (cid:107) H 1 for DRM.
Revised: A = (cid:107) · (cid:107) H 1 for DRM.

Original: Once we show that φ ( r ) is of magnitude O
Revised: Once we show that φ ( r ) is of magnitude O

Original: ( (cid:112) rn )
Revised: ( (cid:112) rn )

Original: , we can achieve the O ( 1 n ) convergence rate via solving the ﬁx point equation φ ( r ) =
Revised: , we can achieve the O ( 1 n ) convergence rate via solving the ﬁx point equation φ ( r ) =

Original: O ( (cid:112) rn ) =
Revised: O ( (cid:112) rn ) =

Original: r ⇒ r
Revised: r ⇒ r

Original: = O ( 1 n ) .
Revised: = O ( 1 n ) .

Original: Using the solution of the ﬁxed point equation of the local Rademacher complexity to bound the generalization error is a standard result in empirical process Bartlett et al.
Revised: Using the solution of the ﬁxed point equation of the local Rademacher complexity to bound the generalization error is a standard result in empirical process Bartlett et al.

Original: ; Srebro et al.
Revised: ; Srebro et al.

Original: ; Koltchinskii (2011); Xu & Zeevi (2020); Farrell et al.
Revised: ; Koltchinskii (2011); Xu & Zeevi (2020); Farrell et al.

Original: The difference is that we used the H 1 / H 2 norm to deﬁne the localized set, while the previous papers used the (cid:96) 2 distance.
Revised: The difference is that we used the H 1 / H 2 norm to deﬁne the localized set, while the previous papers used the (cid:96) 2 distance.

Original: The way to obtain the fast rate generalization bound is using the Peeling Lemma.
Revised: The way to obtain the fast rate generalization bound is using the Peeling Lemma.

Original: We present the error decomposition results as a meta theorem, which is shown in Theorem B.12 for
Revised: We present the error decomposition results as a meta theorem, which is shown in Theorem B.12 for

Original: PINN, Theorem B.9 for DRM and Theorem C.1 for MDRM, respectively.
Revised: PINN, Theorem B.9 for DRM and Theorem C.1 for MDRM, respectively.

Original: To make the ﬁnal rate depend on the data number only, we need bounds of the approximation error in Appendix B.3 and bounds of the local Rademacher complexity in Appendix B.2.
Revised: To make the ﬁnal rate depend on the data number only, we need bounds of the approximation error in Appendix B.3 and bounds of the local Rademacher complexity in Appendix B.2.

Original: Approximation Error.
Revised: Approximation Error.

Original: The proof of the approximation results of truncated Fourier series is easy and intuitive.
Revised: The proof of the approximation results of truncated Fourier series is easy and intuitive.

Original: For completeness, we provide it in Appendix B.3.1.
Revised: For completeness, we provide it in Appendix B.3.1.

Original: The proof of the approximation results of neural networks follows from the fact that a B-spline approximation can be formulated as a
Revised: The proof of the approximation results of neural networks follows from the fact that a B-spline approximation can be formulated as a

Original: ReLU3 neural network efﬁciently.
Revised: ReLU3 neural network efﬁciently.

Original: Our proof basically follows Duan et al.
Revised: Our proof basically follows Duan et al.

Original: ; Jiao et al.
Revised: ; Jiao et al.

Original: (2021a), while the only difference is the activation function.
Revised: (2021a), while the only difference is the activation function.

Original: Our proof is also very similar to (Yarotsky, 2017;
Revised: Our proof is also very similar to (Yarotsky, 2017;

Original: Suzuki, 2018), but the depth of our network is of constant magnitude instead of O ( 1 log (cid:15) ) magnitude, where (cid:15) denotes the desired approximation error.
Revised: Suzuki, 2018), but the depth of our network is of constant magnitude instead of O ( 1 log (cid:15) ) magnitude, where (cid:15) denotes the desired approximation error.

Original: Such improvement of depth results from the fact that ReLU3 activations can approximate B-splines more easily than the ReLU activations, which is useful in our generalization analysis.
Revised: Such improvement of depth results from the fact that ReLU3 activations can approximate B-splines more easily than the ReLU activations, which is useful in our generalization analysis.

Original: Although the proof of the approximation results of neural networks in the Sobolev space is standard, we still list it in Appendix B.3.2.
Revised: Although the proof of the approximation results of neural networks in the Sobolev space is standard, we still list it in Appendix B.3.2.

Original: Generalization Error.
Revised: Generalization Error.

Original: As we discussed above, the generalization error can be bounded by the ﬁx point of the local Rademacher complexity, i.e. the solution of φ ( r )
Revised: As we discussed above, the generalization error can be bounded by the ﬁx point of the local Rademacher complexity, i.e. the solution of φ ( r )

Original: = r .
Revised: = r .

Original: Once we have a O ( (cid:112) rn ) bound of φ ( r ) , we can achieve the O ( 1 n ) fast rate generalization bound we want.
Revised: Once we have a O ( (cid:112) rn ) bound of φ ( r ) , we can achieve the O ( 1 n ) fast rate generalization bound we want.

Original: It remains to upper bound the the local Rademacher complexity φ ( r ) .
Revised: It remains to upper bound the the local Rademacher complexity φ ( r ) .

Original: For the upper bound on the local Rademacher complexity of truncated Fourier series estimators, our proof technique is similar to that of the kernel estimators, whose Rademacher complexity can be bounded by the trace of the Gram matrix ( i.e. the effective number of basis).
Revised: For the upper bound on the local Rademacher complexity of truncated Fourier series estimators, our proof technique is similar to that of the kernel estimators, whose Rademacher complexity can be bounded by the trace of the Gram matrix ( i.e. the effective number of basis).

Original: One interesting thing we showed is that the ﬁnal upper bound of the Rademacher complexity localized by H 1 norm is (cid:113) ξ d − 2 r n .
Revised: One interesting thing we showed is that the ﬁnal upper bound of the Rademacher complexity localized by H 1 norm is (cid:113) ξ d − 2 r n .

Original: The term ξ d − 2 in the numerator is smaller than ξ d , which is the exact number of Fourier basis.
Revised: The term ξ d − 2 in the numerator is smaller than ξ d , which is the exact number of Fourier basis.

Original: This improvement results from the H 1 norm localization.
Revised: This improvement results from the H 1 norm localization.

Original: The detailed proof is given in Lemma B.6,
Revised: The detailed proof is given in Lemma B.6,

Original: Lemma B.7 and Lemma B.8.
Revised: Lemma B.7 and Lemma B.8.

Original: For the upper bound on the local Rademacher complexity bound for neural network, we follow
Revised: For the upper bound on the local Rademacher complexity bound for neural network, we follow

Original: Schmidt-Hieber et al.
Revised: Schmidt-Hieber et al.

Original: Suzuki (2018); Farrell et al.
Revised: Suzuki (2018); Farrell et al.

Original: (2021) to use a Dudley integral theorem and a covering number argument.
Revised: (2021) to use a Dudley integral theorem and a covering number argument.

Original: The covering number arguments are shown in Theorem B.4, Theorem
Revised: The covering number arguments are shown in Theorem B.4, Theorem

Original: B.5 and Theorem B.6.
Revised: B.5 and Theorem B.6.

Original: The ﬁnal local Rademacher complexity bounds are given in Lemma B.17 and
Revised: The ﬁnal local Rademacher complexity bounds are given in Lemma B.17 and

Original: Lemma B.18.
Revised: Lemma B.18.

Original: The difference is that the complexity of gradient of ReLU3 activation function makes the covering number depend exponentially on the neural network’s depth.
Revised: The difference is that the complexity of gradient of ReLU3 activation function makes the covering number depend exponentially on the neural network’s depth.

Original: However, the improvement of neural network’s depth to constant magnitude mentioned above in the approximation results saves this problem.
Revised: However, the improvement of neural network’s depth to constant magnitude mentioned above in the approximation results saves this problem.

Original: One drawback of our proof is that the H 1 norm localization wouldn’t improve the bound for Rademacher complexity and leads to sub-optimal upper bounds.
Revised: One drawback of our proof is that the H 1 norm localization wouldn’t improve the bound for Rademacher complexity and leads to sub-optimal upper bounds.

Original: We hypothesize that our bound is tight for sparse neural network and put seeking a right complexity measure of neural network for solving PDEs as a future work.
Revised: We hypothesize that our bound is tight for sparse neural network and put seeking a right complexity measure of neural network for solving PDEs as a future work.

Original: [Table]
Revised: [Table]

Original: B P ROOF OF THE U PPER B OUNDS B. R EGULARITY R ESULT F OR THE PDE MODEL . Regularity Results of the DRM Objective Function
Revised: B P ROOF OF THE U PPER B OUNDS B. R EGULARITY R ESULT F OR THE PDE MODEL . Regularity Results of the DRM Objective Function

Original: Theorem B.1.
Revised: Theorem B.1.

Original: We consider the static Schrödinger equation on the unit hypercube on R d with the zero Direchlet boundary condition:
Revised: We consider the static Schrödinger equation on the unit hypercube on R d with the zero Direchlet boundary condition:

Original: [Equation]
Revised: [Equation]

Original: (B.1) where f ∈ L 2 (Ω) and V ∈ L ∞ (Ω) with 0 < V min ≤ V ( x ) ≤ V max > 0 .
Revised: (B.1) where f ∈ L 2 (Ω) and V ∈ L ∞ (Ω) with 0 < V min ≤ V ( x ) ≤ V max > 0 .

Original: There exists a unique weak solution u ∗ S to the equivalent variational problem (Evans, 1998):
Revised: There exists a unique weak solution u ∗ S to the equivalent variational problem (Evans, 1998):

Original: [Equation]
Revised: [Equation]

Original: arg
Revised: arg

Original: [Equation]
Revised: [Equation]

Original: Then for any u ∈ H 1 (Ω)
Revised: Then for any u ∈ H 1 (Ω)

Original: [Equation]
Revised: [Equation]

Original: we have: min(1 , V min )
Revised: we have: min(1 , V min )

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To show that u ∗ S satisﬁes estimate B
Revised: To show that u ∗ S satisﬁes estimate B

Original: [Equation]
Revised: [Equation]

Original: 3, we ﬁrst claim that for any u ∈ H 1 (Ω) ,
Revised: 3, we ﬁrst claim that for any u ∈ H 1 (Ω) ,

Original: [Equation]
Revised: [Equation]

Original: In fact
Revised: In fact

Original: [Equation]
Revised: [Equation]

Original: by plugging in the ﬁrst equation of B
Revised: by plugging in the ﬁrst equation of B

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: one has that
Revised: one has that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, applying Green’s formula to the true solution u ∗ S yields:
Revised: Furthermore, applying Green’s formula to the true solution u ∗ S yields:

Original: [Equation]
Revised: [Equation]

Original: where the last identity above follows from the second equality in B.1.
Revised: where the last identity above follows from the second equality in B.1.

Original: Now for any u ∈ H 1 (Ω) , applying Green’s formula to u and the true solution u ∗ S implies:
Revised: Now for any u ∈ H 1 (Ω) , applying Green’s formula to u and the true solution u ∗ S implies:

Original: [Equation]
Revised: [Equation]

Original: where the last identity above again follows from the second equality in B
Revised: where the last identity above again follows from the second equality in B

Original: [Equation]
Revised: [Equation]

Original: This completes our proof of identity B.4.
Revised: This completes our proof of identity B.4.

Original: Using the assumptions on the potential function V then implies:
Revised: Using the assumptions on the potential function V then implies:

Original: [Equation]
Revised: [Equation]

Original: This completes our proof of B.1.
Revised: This completes our proof of B.1.

Original: Regularity Results of the PINN Objective Function
Revised: Regularity Results of the PINN Objective Function

Original: Theorem B.2.
Revised: Theorem B.2.

Original: We consider the static Schrödinger equation on the unit hypercube on R d with the
Revised: We consider the static Schrödinger equation on the unit hypercube on R d with the

Original: Neumann boundary condition:
Revised: Neumann boundary condition:

Original: [Equation]
Revised: [Equation]

Original: where f ∈ L 2 (Ω) and V ∈ L ∞ (Ω) with V 2 − ∆ V > C min , 0 < C min < V ( x ) ≤ V max and − ∆ V ( x ) ≤ V max .
Revised: where f ∈ L 2 (Ω) and V ∈ L ∞ (Ω) with V 2 − ∆ V > C min , 0 < C min < V ( x ) ≤ V max and − ∆ V ( x ) ≤ V max .

Original: Then there exists a unique solution u ∗ S ∈ H 10 (Ω) to the following minimization problem (Brezis, 2010):
Revised: Then there exists a unique solution u ∗ S ∈ H 10 (Ω) to the following minimization problem (Brezis, 2010):

Original: [Equation]
Revised: [Equation]

Original: Then for any u ∈ H 10 (Ω) , we have:
Revised: Then for any u ∈ H 10 (Ω) , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any u ∈ H 10 (Ω) , we let ˜ u
Revised: For any u ∈ H 10 (Ω) , we let ˜ u

Original: = u − u ∗ , then we have ˜ u ∈ H 10 (Ω) .
Revised: = u − u ∗ , then we have ˜ u ∈ H 10 (Ω) .

Original: [Equation]
Revised: [Equation]

Original: Using Green’s formula
Revised: Using Green’s formula

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: where the last equality above follows from the fact that ˜ u ∈ H 10 (Ω) .
Revised: where the last equality above follows from the fact that ˜ u ∈ H 10 (Ω) .

Original: This further implies:
Revised: This further implies:

Original: [Equation]
Revised: [Equation]

Original: Using Green’s formula again, we have:
Revised: Using Green’s formula again, we have:

Original: [Equation]
Revised: [Equation]

Original: Then we can further deduce that:
Revised: Then we can further deduce that:

Original: [Equation]
Revised: [Equation]

Original: For we have assumed V ∈ L ∞ (Ω) with 0 < C min < V 2 − ∆ V, 0 < C min < V ( x ) ≤ V max and − ∆ V ( x ) ≤ V max , thus we have
Revised: For we have assumed V ∈ L ∞ (Ω) with 0 < C min < V 2 − ∆ V, 0 < C min < V ( x ) ≤ V max and − ∆ V ( x ) ≤ V max , thus we have

Original: [Equation]
Revised: [Equation]

Original: B. A UXILIARY DEFINITIONS AND LEMMATA O N G ENERALIZATION E RROR
Revised: B. A UXILIARY DEFINITIONS AND LEMMATA O N G ENERALIZATION E RROR

Original: To bound the generalization error
Revised: To bound the generalization error

Original: [Equation]
Revised: [Equation]

Original: we use the localized Rademacher complexity (Bartlett et al., 2005)
Revised: we use the localized Rademacher complexity (Bartlett et al., 2005)

Original: [Equation]
Revised: [Equation]

Original: Recall that the Rademacher complexity of a function class G is deﬁned by
Revised: Recall that the Rademacher complexity of a function class G is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where Z i are i.i.d samples according to the data distributions and σ j are i.i.d
Revised: where Z i are i.i.d samples according to the data distributions and σ j are i.i.d

Original: Rademacher random variables which take the value 1 with probability 12 and value − 1 with probability 12 .
Revised: Rademacher random variables which take the value 1 with probability 12 and value − 1 with probability 12 .

Original: The following important symmetrization lemma makes the connection between the uniform law of large numbers and the Rademacher complexity.
Revised: The following important symmetrization lemma makes the connection between the uniform law of large numbers and the Rademacher complexity.

Original: Lemma B.1 (Symmetrization Lemma) .
Revised: Lemma B.1 (Symmetrization Lemma) .

Original: Let F be a set of functions.
Revised: Let F be a set of functions.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Lemma B.2 (Ledoux-Talagrand contraction (Ledoux & Talagrand, 2013, Theorem 4.12)) .
Revised: Lemma B.2 (Ledoux-Talagrand contraction (Ledoux & Talagrand, 2013, Theorem 4.12)) .

Original: Assume that φ : R → R is L -Lipschitz with φ (0)
Revised: Assume that φ : R → R is L -Lipschitz with φ (0)

Original: = 0 .
Revised: = 0 .

Original: Let { σ i } ni
Revised: Let { σ i } ni

Original: =1 be independent Rademacher random variables.
Revised: =1 be independent Rademacher random variables.

Original: Then for any T ⊂ R n
Revised: Then for any T ⊂ R n

Original: [Equation]
Revised: [Equation]

Original: Let ( E, ρ ) be a metric space with metric ρ .
Revised: Let ( E, ρ ) be a metric space with metric ρ .

Original: A δ -cover of a set A ⊂ E with respect to ρ is a collection of points { x 1 , · · · , x n } ⊂
Revised: A δ -cover of a set A ⊂ E with respect to ρ is a collection of points { x 1 , · · · , x n } ⊂

Original: A such that for every x ∈ A , there exists i ∈ { 1 , · · · , n } such that ρ ( x, x i ) ≤ δ .
Revised: A such that for every x ∈ A , there exists i ∈ { 1 , · · · , n } such that ρ ( x, x i ) ≤ δ .

Original: The δ -covering number N ( δ, A, ρ ) is the cardinality of the smallest δ -cover of the set
Revised: The δ -covering number N ( δ, A, ρ ) is the cardinality of the smallest δ -cover of the set

Original: A with respect to the metric ρ .
Revised: A with respect to the metric ρ .

Original: Equivalently, the δ -covering number N ( δ, A, ρ ) is the minimal number of balls B ρ ( x, δ ) of radius δ needed to cover the set A .
Revised: Equivalently, the δ -covering number N ( δ, A, ρ ) is the minimal number of balls B ρ ( x, δ ) of radius δ needed to cover the set A .

Original: Theorem B.3 (Dudley’s Integral theorem) .
Revised: Theorem B.3 (Dudley’s Integral theorem) .

Original: Let F be a function class such that sup f ∈ F (cid:107) f
Revised: Let F be a function class such that sup f ∈ F (cid:107) f

Original: (cid:107) n, 2 ≤ M .
Revised: (cid:107) n, 2 ≤ M .

Original: Then the Rademacher complexity R n ( F ) satisﬁes that
Revised: Then the Rademacher complexity R n ( F ) satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: Lemma B.3 (Talagrand Concentration Inequality) . Consider a function class F deﬁned on a probability measure µ such that for all f ∈ F , we have (cid:107) f (cid:107) ∞ ≤ β, E µ [ f ] = 0 , E µ [ f 2 ] ≤ σ 2 .
Revised: Lemma B.3 (Talagrand Concentration Inequality) . Consider a function class F deﬁned on a probability measure µ such that for all f ∈ F , we have (cid:107) f (cid:107) ∞ ≤ β, E µ [ f ] = 0 , E µ [ f 2 ] ≤ σ 2 .

Original: Then for any t > 0 , we can have the following concentration results.
Revised: Then for any t > 0 , we can have the following concentration results.

Original: [Equation]
Revised: [Equation]

Original: Lemma B.4 (Peeling lemma (Bartlett et al., 2005)) .
Revised: Lemma B.4 (Peeling lemma (Bartlett et al., 2005)) .

Original: Consider some measurable function class F .
Revised: Consider some measurable function class F .

Original: Assume that there exists a sub-root function φ ( r ) satisfying
Revised: Assume that there exists a sub-root function φ ( r ) satisfying

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Denote F ( r ) = { f ∈ F | E [ f ] ≤ r } to be the localized set with radius r .
Revised: Denote F ( r ) = { f ∈ F | E [ f ] ≤ r } to be the localized set with radius r .

Original: Then for a ﬁxed set of datapoints { z i } ni =1 and a ﬁxed set of Rademacher random variables { σ i } ni =1 , we have:
Revised: Then for a ﬁxed set of datapoints { z i } ni =1 and a ﬁxed set of Rademacher random variables { σ i } ni =1 , we have:

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: φ
Revised: φ

Original: [Equation]
Revised: [Equation]

Original: r .
Revised: r .

Original: We also modify the peeling lemma above, as we aim to apply it to derive the upper bound for the
Revised: We also modify the peeling lemma above, as we aim to apply it to derive the upper bound for the

Original: Modiﬁed Deep Ritz Method (MDRM).
Revised: Modiﬁed Deep Ritz Method (MDRM).

Original: Lemma B.5 (Peeling Lemma For MDRM) .
Revised: Lemma B.5 (Peeling Lemma For MDRM) .

Original: Given some measurable function class F and two continuous mappings g, h : F → R , we deﬁne a class F of vector-valued functions by:
Revised: Given some measurable function class F and two continuous mappings g, h : F → R , we deﬁne a class F of vector-valued functions by:

Original: F := { ( g ◦ f, h ◦ f ) | f ∈ F } .
Revised: F := { ( g ◦ f, h ◦ f ) | f ∈ F } .

Original: For any f ∈ F , we use g f and h f to denote the two compositions g ◦ f
Revised: For any f ∈ F , we use g f and h f to denote the two compositions g ◦ f

Original: and h ◦ f , respectively.
Revised: and h ◦ f , respectively.

Original: For any r > 0 , the localized set F r is deﬁned by:
Revised: For any r > 0 , the localized set F r is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: Moreover, the modiﬁed Rademacher Complexity of F r is deﬁned by:
Revised: Moreover, the modiﬁed Rademacher Complexity of F r is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: Assume that there exists some function φ : [0 , ∞ ) → [0 , ∞ ) and some r (cid:63) > 0 , such that for any r > r (cid:63) , we have:
Revised: Assume that there exists some function φ : [0 , ∞ ) → [0 , ∞ ) and some r (cid:63) > 0 , such that for any r > r (cid:63) , we have:

Original: [Equation]
Revised: [Equation]

Original: Then for any r > r (cid:63) , we have:
Revised: Then for any r > r (cid:63) , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is the same as the original peeling lemma, thus we omit the detailed proof here.
Revised: The proof is the same as the original peeling lemma, thus we omit the detailed proof here.

Original: B.2. L OCAL R ADEMACHER C OMPLEXITY OF T RUNCATED F OURIER B ASIS
Revised: B.2. L OCAL R ADEMACHER C OMPLEXITY OF T RUNCATED F OURIER B ASIS

Original: Deﬁnition B.1.
Revised: Deﬁnition B.1.

Original: (Fourier Series)
Revised: (Fourier Series)

Original: Given a domain Ω ⊆ [0 , 1] d .
Revised: Given a domain Ω ⊆ [0 , 1] d .

Original: For any z ∈ N d , we consider the corresponding Fourier basis function φ z ( x ) :=
Revised: For any z ∈ N d , we consider the corresponding Fourier basis function φ z ( x ) :=

Original: e 2 πi (cid:104)
Revised: e 2 πi (cid:104)

Original: z,x (cid:105) ( x ∈ Ω) .
Revised: z,x (cid:105) ( x ∈ Ω) .

Original: With respect to the Fourier basis, any function f ∈ L 2 (Ω) can be decomposed as the following sum:
Revised: With respect to the Fourier basis, any function f ∈ L 2 (Ω) can be decomposed as the following sum:

Original: [Equation]
Revised: [Equation]

Original: where for any z ∈ N d , the Fourier coefﬁcient f z = (cid:82) Ω f ( x ) φ z ( x ) dx .
Revised: where for any z ∈ N d , the Fourier coefﬁcient f z = (cid:82) Ω f ( x ) φ z ( x ) dx .

Original: Deﬁnition B.2.
Revised: Deﬁnition B.2.

Original: (Truncated Fourier Series)
Revised: (Truncated Fourier Series)

Original: For a ﬁxed positive integer ξ ∈ Z + , we deﬁne the space
Revised: For a ﬁxed positive integer ξ ∈ Z + , we deﬁne the space

Original: F ξ (Ω) of truncated Fourier series as follows:
Revised: F ξ (Ω) of truncated Fourier series as follows:

Original: [Equation]
Revised: [Equation]

Original: Equivalently, we can decompose any f ∈ F ξ (Ω) as f := (cid:80)
Revised: Equivalently, we can decompose any f ∈ F ξ (Ω) as f := (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Lemma B.6.
Revised: Lemma B.6.

Original: (Local Rademacher Complexity of Localized Truncated Fourier Series)
Revised: (Local Rademacher Complexity of Localized Truncated Fourier Series)

Original: For a ﬁxed ξ ∈ Z + , we consider a localized class of functions F ρ,ξ (Ω) =
Revised: For a ﬁxed ξ ∈ Z + , we consider a localized class of functions F ρ,ξ (Ω) =

Original: (cid:110)
Revised: (cid:110)

Original: f ∈ F ξ (Ω) (cid:12)(cid:12)(cid:12) (cid:107) f (cid:107) 2 H 1 (Ω) ≤ ρ (cid:111) , where ρ > 0 is ﬁxed.
Revised: f ∈ F ξ (Ω) (cid:12)(cid:12)(cid:12) (cid:107) f (cid:107) 2 H 1 (Ω) ≤ ρ (cid:111) , where ρ > 0 is ﬁxed.

Original: Then we have the following upper bound on the local Rademacher complexity:
Revised: Then we have the following upper bound on the local Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Take an arbitrary function f ∈ F ρ,ξ (Ω) .
Revised: Take an arbitrary function f ∈ F ρ,ξ (Ω) .

Original: Let f = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z
Revised: Let f = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z

Original: φ
Revised: φ

Original: z be the Fourier basis expansion of f .
Revised: z be the Fourier basis expansion of f .

Original: ρ ≥ (cid:107) f (cid:107) 2 H 1 (Ω) implies constraint (cid:80) (cid:107)
Revised: ρ ≥ (cid:107) f (cid:107) 2 H 1 (Ω) implies constraint (cid:80) (cid:107)

Original: z (cid:107) ∞
Revised: z (cid:107) ∞

Original: ≤ ξ | f z | 2 (cid:107)
Revised: ≤ ξ | f z | 2 (cid:107)

Original: z (cid:107) 2 (cid:46) ρ on the Fourier coefﬁcients(Adams & Fournier, 2003).
Revised: z (cid:107) 2 (cid:46) ρ on the Fourier coefﬁcients(Adams & Fournier, 2003).

Original: On the other hand, substituting the Fourier expansion into the average sum 1 n (cid:80) ni =1 σ i f ( X i ) and using Cauchy-Schwarz inequality imply:
Revised: On the other hand, substituting the Fourier expansion into the average sum 1 n (cid:80) ni =1 σ i f ( X i ) and using Cauchy-Schwarz inequality imply:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we have used the constraint (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ | f z | 2 (cid:107) z (cid:107) 2 (cid:46) ρ in the last step above.
Revised: where we have used the constraint (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ | f z | 2 (cid:107) z (cid:107) 2 (cid:46) ρ in the last step above.

Original: Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i (1 ≤ i ≤ n ) and the uniformly sampled data points { X i } ni =1 on both sides and applying Jensen’s inequality, we can deduce that:
Revised: Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i (1 ≤ i ≤ n ) and the uniformly sampled data points { X i } ni =1 on both sides and applying Jensen’s inequality, we can deduce that:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Using independence between the random variables σ i (1 ≤ i ≤ n )
Revised: Using independence between the random variables σ i (1 ≤ i ≤ n )

Original: [Equation]
Revised: [Equation]

Original: we can further simplify the expectation inside the square root above as below:
Revised: we can further simplify the expectation inside the square root above as below:

Original: [Equation]
Revised: [Equation]

Original: Combining the two bounds above yields the desired upper bound:
Revised: Combining the two bounds above yields the desired upper bound:

Original: [Equation]
Revised: [Equation]

Original: Lemma B.7.
Revised: Lemma B.7.

Original: (Local Rademacher Complexity of Localized Truncated Fourier Series’ Gradient)
Revised: (Local Rademacher Complexity of Localized Truncated Fourier Series’ Gradient)

Original: For a ﬁxed ξ ∈ Z + , we consider a localized class of functions G ρ,ξ (Ω) =
Revised: For a ﬁxed ξ ∈ Z + , we consider a localized class of functions G ρ,ξ (Ω) =

Original: {(cid:107)∇ f (cid:107) | f ∈ F ρ,ξ (Ω) } , where ρ > 0 is ﬁxed.
Revised: {(cid:107)∇ f (cid:107) | f ∈ F ρ,ξ (Ω) } , where ρ > 0 is ﬁxed.

Original: Then for any sample { X i } ni =1 ⊂ Ω , we have the following upper bound on the local
Revised: Then for any sample { X i } ni =1 ⊂ Ω , we have the following upper bound on the local

Original: Rademacher complexity:
Revised: Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Take an arbitrary function f ∈ F ρ,ξ (Ω) .
Revised: Take an arbitrary function f ∈ F ρ,ξ (Ω) .

Original: Let f = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z
Revised: Let f = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z

Original: φ
Revised: φ

Original: z be the Fourier basis expansion of f .
Revised: z be the Fourier basis expansion of f .

Original: Similarly, the norm restriction condition (cid:107) f (cid:107) 2 H 1 (Ω) ≤ ρ can be reduced to the following condition about Fourier coefﬁcients:
Revised: Similarly, the norm restriction condition (cid:107) f (cid:107) 2 H 1 (Ω) ≤ ρ can be reduced to the following condition about Fourier coefﬁcients:

Original: [Equation]
Revised: [Equation]

Original: Moreover, substituting the Fourier expansion into the average sum 1 n (cid:80) ni =1 σ i
Revised: Moreover, substituting the Fourier expansion into the average sum 1 n (cid:80) ni =1 σ i

Original: [Equation]
Revised: [Equation]

Original: f ( X i ) (cid:107) and using
Revised: f ( X i ) (cid:107) and using

Original: Cauchy-Schwarz inequality imply:
Revised: Cauchy-Schwarz inequality imply:

Original: [Equation]
Revised: [Equation]

Original: where we have used the constraint (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ | f z | 2 (cid:107) z (cid:107) 2 (cid:46) ρ in the last step above.
Revised: where we have used the constraint (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ | f z | 2 (cid:107) z (cid:107) 2 (cid:46) ρ in the last step above.

Original: Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i (1 ≤ i ≤ n ) and the uniformly sampled data points { X i } ni =1 on both sides and applying Jensen’s inequality, we can deduce that:
Revised: Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i (1 ≤ i ≤ n ) and the uniformly sampled data points { X i } ni =1 on both sides and applying Jensen’s inequality, we can deduce that:

Original: [Equation]
Revised: [Equation]

Original: Using independence between the random variables σ i (1 ≤ i ≤ n )
Revised: Using independence between the random variables σ i (1 ≤ i ≤ n )

Original: [Equation]
Revised: [Equation]

Original: we can further simplify the expectation inside the square root above as below:
Revised: we can further simplify the expectation inside the square root above as below:

Original: [Equation]
Revised: [Equation]

Original: Combining the two bounds above yields the desired upper bound:
Revised: Combining the two bounds above yields the desired upper bound:

Original: [Equation]
Revised: [Equation]

Original: Lemma B.8.
Revised: Lemma B.8.

Original: (Local Rademacher Complexity of Localized Truncated Fourier Series’ Laplacian)
Revised: (Local Rademacher Complexity of Localized Truncated Fourier Series’ Laplacian)

Original: For a ﬁxed ξ ∈ Z + , we consider a localized class of functions J
Revised: For a ﬁxed ξ ∈ Z + , we consider a localized class of functions J

Original: [Equation]
Revised: [Equation]

Original: where ρ > 0 is ﬁxed.
Revised: where ρ > 0 is ﬁxed.

Original: Correspondingly, we deﬁne a localized class of Laplacians K ρ,ξ (Ω) := { ∆ f | f ∈ J ρ,ξ (Ω) } .
Revised: Correspondingly, we deﬁne a localized class of Laplacians K ρ,ξ (Ω) := { ∆ f | f ∈ J ρ,ξ (Ω) } .

Original: Then for any sample { X i } ni =1 ⊂ Ω , we have the following upper bound on the local Rademacher complexity:
Revised: Then for any sample { X i } ni =1 ⊂ Ω , we have the following upper bound on the local Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Take an arbitrary function f ∈ J ρ,ξ (Ω) .
Revised: Take an arbitrary function f ∈ J ρ,ξ (Ω) .

Original: Let f = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z
Revised: Let f = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z

Original: φ
Revised: φ

Original: z be the Fourier basis expansion of f .
Revised: z be the Fourier basis expansion of f .

Original: Similarly, the norm restriction condition (cid:107) f (cid:107) 2 H 2 (Ω) ≤ ρ can be reduced to the following condition about Fourier coefﬁcients:
Revised: Similarly, the norm restriction condition (cid:107) f (cid:107) 2 H 2 (Ω) ≤ ρ can be reduced to the following condition about Fourier coefﬁcients:

Original: [Equation]
Revised: [Equation]

Original: Moreover, substituting the Fourier expansion into the average sum 1 n (cid:80) ni
Revised: Moreover, substituting the Fourier expansion into the average sum 1 n (cid:80) ni

Original: =

Revised: =

Original: σ i ∆ f
Revised: σ i ∆ f

Original: ( X i ) and using
Revised: ( X i ) and using

Original: Cauchy-Schwarz inequality imply:
Revised: Cauchy-Schwarz inequality imply:

Original: [Equation]
Revised: [Equation]

Original: where we have used the constraint (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ | f z | 2 (cid:107) z (cid:107) 4 (cid:46) ρ in the last step above.
Revised: where we have used the constraint (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ | f z | 2 (cid:107) z (cid:107) 4 (cid:46) ρ in the last step above.

Original: Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i (1 ≤ i ≤ n ) and the uniformly sampled data points { X i } ni =1 on both sides and applying Jensen’s inequality, we can deduce that:
Revised: Moreover, by taking expectation with respect to the i.i.d Rademacher random variables σ i (1 ≤ i ≤ n ) and the uniformly sampled data points { X i } ni =1 on both sides and applying Jensen’s inequality, we can deduce that:

Original: [Equation]
Revised: [Equation]

Original: Using independence between the random variables σ i (1 ≤ i ≤ n )
Revised: Using independence between the random variables σ i (1 ≤ i ≤ n )

Original: [Equation]
Revised: [Equation]

Original: we can further simplify the expectation inside the square root above as below:
Revised: we can further simplify the expectation inside the square root above as below:

Original: [Equation]
Revised: [Equation]

Original: Combining the two bounds above yields the desired upper bound:
Revised: Combining the two bounds above yields the desired upper bound:

Original: [Equation]
Revised: [Equation]

Original: B.2. L OCAL R ADEMACHER C OMPLEXITY OF THE D EEP N EURAL N ETWORK M ODEL
Revised: B.2. L OCAL R ADEMACHER C OMPLEXITY OF THE D EEP N EURAL N ETWORK M ODEL

Original: In this section we aim to bound the local Rademacher Complexity of a Deep Neural Network.
Revised: In this section we aim to bound the local Rademacher Complexity of a Deep Neural Network.

Original: We ﬁrst bound the covering number of the function space composed by the gradient of all possible neural networks and then apply a Duley Integral to achieve the ﬁnal bound.
Revised: We ﬁrst bound the covering number of the function space composed by the gradient of all possible neural networks and then apply a Duley Integral to achieve the ﬁnal bound.

Original: Deﬁnition B.3.
Revised: Deﬁnition B.3.

Original: Let η l denote the l -ReLU activiation function.
Revised: Let η l denote the l -ReLU activiation function.

Original: Here we use η 3 :=
Revised: Here we use η 3 :=

Original: max { 0 , x } 3 (E & Yu, 2018) as the activation function to ensure smoothness.
Revised: max { 0 , x } 3 (E & Yu, 2018) as the activation function to ensure smoothness.

Original: We can deﬁne the space consisting of all neural network models with depth L , width W , sparsity constraint S and norm constraint B as follows:
Revised: We can deﬁne the space consisting of all neural network models with depth L , width W , sparsity constraint S and norm constraint B as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) · (cid:107) 

Revised: where (cid:107) · (cid:107)

Original: measures the number of nonzero entries in a matrix and (cid:107) · (cid:107) ∞ , ∞ measures the maximum of the absolute values of the entries in a matrix.
Revised: measures the number of nonzero entries in a matrix and (cid:107) · (cid:107) ∞ , ∞ measures the maximum of the absolute values of the entries in a matrix.

Original: For any d ∈ Z + , we refer to an arbitrary element in Φ( L, W, S, B ) as a ReLU3 Deep Neural Network.
Revised: For any d ∈ Z + , we refer to an arbitrary element in Φ( L, W, S, B ) as a ReLU3 Deep Neural Network.

Original: Then for any index 1 ≤ k ≤ L , we use F k to denote the k − ReLU3 Deep Neural Network composed by the ﬁrst k layers, i.e:
Revised: Then for any index 1 ≤ k ≤ L , we use F k to denote the k − ReLU3 Deep Neural Network composed by the ﬁrst k layers, i.e:

Original: [Equation]
Revised: [Equation]

Original: Also, we use Φ k ( L, W, S, B ) to denote the space consisting of all F k .
Revised: Also, we use Φ k ( L, W, S, B ) to denote the space consisting of all F k .

Original: In particular, when k = L , we have:
Revised: In particular, when k = L , we have:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, given that the domain Ω ⊂ [0 , 1] d is bounded, we have sup x ∈ Ω (cid:107) x (cid:107) ∞
Revised: Furthermore, given that the domain Ω ⊂ [0 , 1] d is bounded, we have sup x ∈ Ω (cid:107) x (cid:107) ∞

Original: = 1 .
Revised: = 1 .

Original: Lemma B.9.
Revised: Lemma B.9.

Original: (Upper bound on ∞ -norm of functions in DNN space)
Revised: (Upper bound on ∞ -norm of functions in DNN space)

Original: For any 1 ≤ k ≤ L , the following inequality holds:
Revised: For any 1 ≤ k ≤ L , the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use induction to prove this claim.
Revised: We use induction to prove this claim.

Original: Base cases: When k = 1 , we have that for any x ∈ Ω and any F 1 ∈ Φ 1 ( L, W, S, B ) , the following holds:
Revised: Base cases: When k = 1 , we have that for any x ∈ Ω and any F 1 ∈ Φ 1 ( L, W, S, B ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: B.19)
Revised: B.19)

Original: When k = 2 , we have that for any x ∈ Ω and any F 2 ∈ Φ 2 ( L, W, S, B ) , the following holds:
Revised: When k = 2 , we have that for any x ∈ Ω and any F 2 ∈ Φ 2 ( L, W, S, B ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: By applying the bound proved in the case when k =
Revised: By applying the bound proved in the case when k =

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the assumption that W ≥ 2 .
Revised: where the last inequality follows from the assumption that W ≥ 2 .

Original: Inductive Step: Now we assume that the claim has been proved for k − 1 , where 3 ≤ k ≤ L .
Revised: Inductive Step: Now we assume that the claim has been proved for k − 1 , where 3 ≤ k ≤ L .

Original: Similarly, for any x ∈ Ω and any F k ∈ Φ k ( L, W, S, B )
Revised: Similarly, for any x ∈ Ω and any F k ∈ Φ k ( L, W, S, B )

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Using inductive hypothesis, we can further deduce that:
Revised: Using inductive hypothesis, we can further deduce that:

Original: [Equation]
Revised: [Equation]

Original: Taking supremum with respect to x ∈ Ω and F k ∈ Φ k ( L, W, S, B ) on the LHS implies that the given upper bound also holds for k .
Revised: Taking supremum with respect to x ∈ Ω and F k ∈ Φ k ( L, W, S, B ) on the LHS implies that the given upper bound also holds for k .

Original: By induction, the claim is proved.
Revised: By induction, the claim is proved.

Original: We also need to show that the ReLU3 activation function is a Lipschitzness functions over a bounded domain.
Revised: We also need to show that the ReLU3 activation function is a Lipschitzness functions over a bounded domain.

Original: Lemma B.10.
Revised: Lemma B.10.

Original: For any k ∈ Z + , consider the k − ReLU activation function η k deﬁned on some bounded domain D ⊂ R d (i.e, sup x ∈D (cid:107) x (cid:107) ∞ ≤ C for some C > 0 ).
Revised: For any k ∈ Z + , consider the k − ReLU activation function η k deﬁned on some bounded domain D ⊂ R d (i.e, sup x ∈D (cid:107) x (cid:107) ∞ ≤ C for some C > 0 ).

Original: Then we have that for any x, y ∈ D , the following inequalities hold:
Revised: Then we have that for any x, y ∈ D , the following inequalities hold:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This is because |∇ η 1 ( x ) | =
Revised: This is because |∇ η 1 ( x ) | =

Original: | max { 1 , 0 }| = 1 , |∇ η 2 ( x ) | = | 2 max { x, 0 }| ≤ 2 C and |∇ η 3 ( x ) | = | 3 max { x, 0 } 2 | ≤ 3 C 2 .
Revised: | max { 1 , 0 }| = 1 , |∇ η 2 ( x ) | = | 2 max { x, 0 }| ≤ 2 C and |∇ η 3 ( x ) | = | 3 max { x, 0 } 2 | ≤ 3 C 2 .

Original: Lemma B.11.
Revised: Lemma B.11.

Original: (Relation between the covering number of DNN space and parameter space)
Revised: (Relation between the covering number of DNN space and parameter space)

Original: For any 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L, W, S, B ) are given by:
Revised: For any 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L, W, S, B ) are given by:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, assume that the (cid:107) (cid:107) ∞ norm of the distance between the parameter spaces of F k and
Revised: Furthermore, assume that the (cid:107) (cid:107) ∞ norm of the distance between the parameter spaces of F k and

Original: G k is uniformly upper bounded by δ , i.e
Revised: G k is uniformly upper bounded by δ , i.e

Original: [Equation]
Revised: [Equation]

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let’s prove the claim by using induction on k .
Revised: Let’s prove the claim by using induction on k .

Original: Base Case: When k = 1 , we have that for any x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L, W, S, B ) satisfying constraint B.20, the following holds:
Revised: Base Case: When k = 1 , we have that for any x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L, W, S, B ) satisfying constraint B.20, the following holds:

Original: [Equation]
Revised: [Equation]

Original: When k = 2 , we have that for any x ∈ Ω and any F 2 , G 2 ∈
Revised: When k = 2 , we have that for any x ∈ Ω and any F 2 , G 2 ∈

Original: Φ 2 ( L, W, S, B ) satisfying constraint
Revised: Φ 2 ( L, W, S, B ) satisfying constraint

Original: B.20, the following inequality holds:
Revised: B.20, the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: By applying the upper bound proved in equation B.19, we can upper bound the ﬁrst part
Revised: By applying the upper bound proved in equation B.19, we can upper bound the ﬁrst part

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: G
Revised: G

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: by:
Revised: by:

Original: [Equation]
Revised: [Equation]

Original: By applying the Lipschitz condition proved in Lemma B.10 and the bound proved in equation B.
Revised: By applying the Lipschitz condition proved in Lemma B.10 and the bound proved in equation B.

Original: [Equation]
Revised: [Equation]

Original: we can further upper bound the second part (cid:107)W (2) G η 3 ( F 1 ( x )) − W
Revised: we can further upper bound the second part (cid:107)W (2) G η 3 ( F 1 ( x )) − W

Original: [Equation]
Revised: [Equation]

Original: 2) G η 3 ( G 1 ( x ))
Revised: 2) G η 3 ( G 1 ( x ))

Original: (cid:107) ∞ by:
Revised: (cid:107) ∞ by:

Original: [Equation]
Revised: [Equation]

Original: Summing the two upper bounds above yields:
Revised: Summing the two upper bounds above yields:

Original: [Equation]
Revised: [Equation]

Original: where we again use the assumption d ≥ 2 in the last step.
Revised: where we again use the assumption d ≥ 2 in the last step.

Original: Inductive Step: Now we assume that the claim has been proved for k − 1 , where k ≥ 3 .
Revised: Inductive Step: Now we assume that the claim has been proved for k − 1 , where k ≥ 3 .

Original: For any x ∈ Ω and F k ∈ Φ k ( L, W, S, B ) , we have that:
Revised: For any x ∈ Ω and F k ∈ Φ k ( L, W, S, B ) , we have that:

Original: [Equation]
Revised: [Equation]

Original: Applying triangle inequality helps us upper bound the ﬁrst term above as follows:
Revised: Applying triangle inequality helps us upper bound the ﬁrst term above as follows:

Original: [Equation]
Revised: [Equation]

Original: From Lemma B.9, we can upper bound the ﬁrst term δW (cid:107) F k − 1 ( x ) (cid:107) 3 ∞ by:
Revised: From Lemma B.9, we can upper bound the ﬁrst term δW (cid:107) F k − 1 ( x ) (cid:107) 3 ∞ by:

Original: [Equation]
Revised: [Equation]

Original: Moreover, applying Lemma B.10 and the inductive hypothesis let us upper bound the second term
Revised: Moreover, applying Lemma B.10 and the inductive hypothesis let us upper bound the second term

Original: [Equation]
Revised: [Equation]

Original: follows:
Revised: follows:

Original: [Equation]
Revised: [Equation]

Original: Combining the two upper bounds derived above yields:
Revised: Combining the two upper bounds derived above yields:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality above follows from k ≥ 3 .
Revised: where the last inequality above follows from k ≥ 3 .

Original: Taking supremum with respect to x ∈ Ω on the
Revised: Taking supremum with respect to x ∈ Ω on the

Original: LHS implies the given upper bound also holds for k .
Revised: LHS implies the given upper bound also holds for k .

Original: By induction, the claim is proved.
Revised: By induction, the claim is proved.

Original: Theorem B.4.
Revised: Theorem B.4.

Original: (Bounding the DNN space covering number)
Revised: (Bounding the DNN space covering number)

Original: Fix some sufﬁciently large N ∈ Z + .
Revised: Fix some sufﬁciently large N ∈ Z + .

Original: Consider a Deep Neural Network space Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =
Revised: Consider a Deep Neural Network space Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) .
Revised: = O ( N ) .

Original: Then the log value of the covering number of this DNN space with respect to the inf-norm (cid:107) F ( x ) (cid:107) ∞ := sup x ∈ Ω | F ( x ) | , which is denoted by N ( δ, Φ( L, W, S, B ) , (cid:107) · (cid:107) ∞ ) , can be upper bounded by:
Revised: Then the log value of the covering number of this DNN space with respect to the inf-norm (cid:107) F ( x ) (cid:107) ∞ := sup x ∈ Ω | F ( x ) | , which is denoted by N ( δ, Φ( L, W, S, B ) , (cid:107) · (cid:107) ∞ ) , can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrstly ﬁx a sparsity pattern (i.e, the locations of the non-zero entries are ﬁxed).
Revised: We ﬁrstly ﬁx a sparsity pattern (i.e, the locations of the non-zero entries are ﬁxed).

Original: By picking k = L in Lemma B.11, we get the following upper bound on the covering number with respect to (cid:107) ·
Revised: By picking k = L in Lemma B.11, we get the following upper bound on the covering number with respect to (cid:107) ·

Original: [Equation]
Revised: [Equation]

Original: ∞ :
Revised: ∞ :

Original: [Equation]
Revised: [Equation]

Original: Furthermore, note that the number of feasible conﬁgurations is upper bounded by (cid:0) ( W +1)
Revised: Furthermore, note that the number of feasible conﬁgurations is upper bounded by (cid:0) ( W +1)

Original: L S (cid:1) ≤ ( W + 1) LS .(Schmidt-Hieber et al., 2020; Farrell et al., 2021)
Revised: L S (cid:1) ≤ ( W + 1) LS .(Schmidt-Hieber et al., 2020; Farrell et al., 2021)

Original: Plug in the previous inequality and yields:
Revised: Plug in the previous inequality and yields:

Original: [Equation]
Revised: [Equation]

Original: ( δ, Φ( L
Revised: ( δ, Φ( L

Original: [Equation]
Revised: [Equation]

Original: W, S, B ) , (cid:107) · (cid:107) ∞ ) ≤ log
Revised: W, S, B ) , (cid:107) · (cid:107) ∞ ) ≤ log

Original: [Equation]
Revised: [Equation]

Original: δ 3 L
Revised: δ 3 L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3 L −
Revised: 3 L −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: 5 × 3 L − 1 −
Revised: 5 × 3 L − 1 −

Original: [Equation]
Revised: [Equation]

Original: 2 3 L − 1 2 − L
Revised: 2 3 L − 1 2 − L

Original: [Equation]
Revised: [Equation]

Original: (cid:17) −
Revised: (cid:17) −

Original: [Equation]
Revised: [Equation]

Original: Note that here the dimension d is some constant.
Revised: Note that here the dimension d is some constant.

Original: Thus, by plugging in thee given magnitudes
Revised: Thus, by plugging in thee given magnitudes

Original: L = O (1) , W = O ( N ) , S =
Revised: L = O (1) , W = O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) , we can further deduce that:
Revised: = O ( N ) , we can further deduce that:

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes our proof.
Revised: This ﬁnishes our proof.

Original: Now let’s consider upper bounding the covering number of the l 2 norm of the sparse Deep Neural
Revised: Now let’s consider upper bounding the covering number of the l 2 norm of the sparse Deep Neural

Original: Networks’ gradients.
Revised: Networks’ gradients.

Original: Note that for any 1 ≤ k ≤ L − 1 , any k − ReLU3 Deep Neural Network
Revised: Note that for any 1 ≤ k ≤ L − 1 , any k − ReLU3 Deep Neural Network

Original: F k ∈ Φ k ( L, W, S, B ) is a map from R d to R W .
Revised: F k ∈ Φ k ( L, W, S, B ) is a map from R d to R W .

Original: For any 1 ≤ l ≤ W , we use F k,l ( x ) to denote the l -th component of the map F k .
Revised: For any 1 ≤ l ≤ W , we use F k,l ( x ) to denote the l -th component of the map F k .

Original: This helps us write the map F k ( x ) and its Jacobian matrix J [ F k ]( x ) explicitly as:
Revised: This helps us write the map F k ( x ) and its Jacobian matrix J [ F k ]( x ) explicitly as:

Original: [Equation]
Revised: [Equation]

Original: In particular, when k = L , we have that any F L ∈ Φ L ( L, W, S, B ) =
Revised: In particular, when k = L , we have that any F L ∈ Φ L ( L, W, S, B ) =

Original: Φ( L, W, S, B ) is a map from R d to R .
Revised: Φ( L, W, S, B ) is a map from R d to R .

Original: Thus, its Jacobian can be explicitly written as the following row vector:
Revised: Thus, its Jacobian can be explicitly written as the following row vector:

Original: [Equation]
Revised: [Equation]

Original: Lemma B.12.
Revised: Lemma B.12.

Original: (Upper bound on ∞ -norm of Jacobian/Gradient of elements in the DNN space)
Revised: (Upper bound on ∞ -norm of Jacobian/Gradient of elements in the DNN space)

Original: For any 1 ≤ k ≤ L , the following inequality holds:
Revised: For any 1 ≤ k ≤ L , the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use induction on k to prove the claim.
Revised: We use induction on k to prove the claim.

Original: Base case: k = 1 .
Revised: Base case: k = 1 .

Original: By the deﬁnition of Jacobian matrix, we have that for any x ∈ Ω and any
Revised: By the deﬁnition of Jacobian matrix, we have that for any x ∈ Ω and any

Original: F 1 ∈ Φ 1 ( L, W, S, B ) , the following holds:
Revised: F 1 ∈ Φ 1 ( L, W, S, B ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: Inductive Step: Assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .
Revised: Inductive Step: Assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .

Original: For any x ∈ Ω and any F k ∈ Φ k ( L, W, S, B ) , by applying the Chain Rule, we can write the Jacobian matrix
Revised: For any x ∈ Ω and any F k ∈ Φ k ( L, W, S, B ) , by applying the Chain Rule, we can write the Jacobian matrix

Original: J [ F k ]( x ) as J [ F k ]( x ) =
Revised: J [ F k ]( x ) as J [ F k ]( x ) =

Original: W ( k ) F J [ η 3 ◦ F k − 1 ]( x ) , where the ReLU3 activation function η 3 is applied to each component F k − 1 ,l (1 ≤ l ≤ W ) of the map F k .
Revised: W ( k ) F J [ η 3 ◦ F k − 1 ]( x ) , where the ReLU3 activation function η 3 is applied to each component F k − 1 ,l (1 ≤ l ≤ W ) of the map F k .

Original: Then we have the following upper bound:
Revised: Then we have the following upper bound:

Original: [Equation]
Revised: [Equation]

Original: Note that the composition η 3 ◦ F k − 1 is a map from R d to R W .
Revised: Note that the composition η 3 ◦ F k − 1 is a map from R d to R W .

Original: Hence, the Jacobian matrix J [ η 3 ◦
Revised: Hence, the Jacobian matrix J [ η 3 ◦

Original: F k − 1 ]( x ) is of shape R W × d .
Revised: F k − 1 ]( x ) is of shape R W × d .

Original: Applying the Chain Rule again implies:
Revised: Applying the Chain Rule again implies:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for any 1 ≤ l ≤ W , the summation on the RHS above can be upper bounded by:
Revised: Furthermore, for any 1 ≤ l ≤ W , the summation on the RHS above can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Now let’s take supremum with respect to l and apply the inductive hypothesis and Lemma B.9.
Revised: Now let’s take supremum with respect to l and apply the inductive hypothesis and Lemma B.9.

Original: This yields:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: By substituting equation B.25 into equation B.
Revised: By substituting equation B.25 into equation B.

Original: [Equation]
Revised: [Equation]

Original: we can derive the ﬁnal bound:
Revised: we can derive the ﬁnal bound:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality above follows from k ≥ 2 .
Revised: where the last inequality above follows from k ≥ 2 .

Original: Taking supremum with respect to x ∈ Ω and
Revised: Taking supremum with respect to x ∈ Ω and

Original: F k ∈ Φ k ( L, W, S, B ) on the LHS implies that the given upper bound also holds for k .
Revised: F k ∈ Φ k ( L, W, S, B ) on the LHS implies that the given upper bound also holds for k .

Original: By induction, the claim is proved.
Revised: By induction, the claim is proved.

Original: For the convenience of the following proof, we ﬁrst prove this lemma for vector 2 norm and ∞ norm.
Revised: For the convenience of the following proof, we ﬁrst prove this lemma for vector 2 norm and ∞ norm.

Original: Lemma B.13.
Revised: Lemma B.13.

Original: Given any two row vectors u , v ∈ R 1 × d , we have:
Revised: Given any two row vectors u , v ∈ R 1 × d , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Assume that the two vectors u , v ∈ R
Revised: Assume that the two vectors u , v ∈ R

Original: d can be explicitly written as u =
Revised: d can be explicitly written as u =

Original: [ u 1 , u 2 , · · · , u d ] and v = [ v 1 , v 2 , · · · , v d ] , respectively.
Revised: [ u 1 , u 2 , · · · , u d ] and v = [ v 1 , v 2 , · · · , v d ] , respectively.

Original: By applying Cauchy-Schwarz inequality, we have:
Revised: By applying Cauchy-Schwarz inequality, we have:

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Taking the square root on both sides yields the desired inequality.
Revised: Taking the square root on both sides yields the desired inequality.

Original: Then we upper bound the Lipschitz constant of the gradient of the neural network.
Revised: Then we upper bound the Lipschitz constant of the gradient of the neural network.

Original: Given a DNN space Φ( L, W, S, B ) , we deﬁne a corresponding DNN Gradient space ∇ Φ( L, W, S, B ) as:
Revised: Given a DNN space Φ( L, W, S, B ) , we deﬁne a corresponding DNN Gradient space ∇ Φ( L, W, S, B ) as:

Original: [Equation]
Revised: [Equation]

Original: Lemma B.14.
Revised: Lemma B.14.

Original: (Relation between the covering number of the DNN Gradient space and parameter space)
Revised: (Relation between the covering number of the DNN Gradient space and parameter space)

Original: For any 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L, W, S, B ) are given by:
Revised: For any 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L, W, S, B ) are given by:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, assume that the (cid:107) (cid:107) ∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ , i.e (cid:107) W (
Revised: Furthermore, assume that the (cid:107) (cid:107) ∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ , i.e (cid:107) W (

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: ( l ) G (cid:107) ∞ , ∞ ≤ δ, (cid:107) b ( l ) F − b ( l
Revised: ( l ) G (cid:107) ∞ , ∞ ≤ δ, (cid:107) b ( l ) F − b ( l

Original: [Equation]
Revised: [Equation]

Original: G (cid:107) ∞ ≤ δ, ( ∀ 1 ≤ l ≤ k ) .
Revised: G (cid:107) ∞ ≤ δ, ( ∀ 1 ≤ l ≤ k ) .

Original: (B.27)
Revised: (B.27)

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: In particular, when k = L , we have:
Revised: In particular, when k = L , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use induction on k to prove the claim.
Revised: We use induction on k to prove the claim.

Original: Base case: When k = 1 , we have that for any x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L, W, S, B ) , the following holds:
Revised: Base case: When k = 1 , we have that for any x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L, W, S, B ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: Inductive Step: assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .
Revised: Inductive Step: assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .

Original: Then for any x ∈ Ω and F k , G k ∈
Revised: Then for any x ∈ Ω and F k , G k ∈

Original: Φ k ( L, W, S, B ) satisfying constraint B.27, applying the Chain Rule and triangle inequality help us upper
Revised: Φ k ( L, W, S, B ) satisfying constraint B.27, applying the Chain Rule and triangle inequality help us upper

Original: [Equation]
Revised: [Equation]

Original: the
Revised: the

Original: [Equation]
Revised: [Equation]

Original: Using equation B
Revised: Using equation B

Original: [Equation]
Revised: [Equation]

Original: 25 helps us upper bound the ﬁrst term by:
Revised: 25 helps us upper bound the ﬁrst term by:

Original: [Equation]
Revised: [Equation]

Original: Note that the two compositions η 3 ◦ F k − 1 and η 3 ◦ G k − 1 both map from R d to R W .
Revised: Note that the two compositions η 3 ◦ F k − 1 and η 3 ◦ G k − 1 both map from R d to R W .

Original: Hence, the two
Revised: Hence, the two

Original: Jacobian matrices J [ η 3 ◦ F k − 1 ]( x ) and J [ η 3 ◦ G k − 1 ]( x ) are of shape R W × d .
Revised: Jacobian matrices J [ η 3 ◦ F k − 1 ]( x ) and J [ η 3 ◦ G k − 1 ]( x ) are of shape R W × d .

Original: Applying the Chain
Revised: Applying the Chain

Original: Rule again implies:
Revised: Rule again implies:

Original: [Equation]
Revised: [Equation]

Original: For any 1 ≤ l ≤ W
Revised: For any 1 ≤ l ≤ W

Original: [Equation]
Revised: [Equation]

Original: the summation on the RHS above can be upper bounded by:
Revised: the summation on the RHS above can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: We denote the two summations above by T 1 and T 2 , respectively:
Revised: We denote the two summations above by T 1 and T 2 , respectively:

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst sum T
Revised: For the ﬁrst sum T

Original: [Equation]
Revised: [Equation]

Original: applying Lemma B.9, Lemma B.10, Lemma B.11 and Lemma B.12 yields the following upper bound:
Revised: applying Lemma B.9, Lemma B.10, Lemma B.11 and Lemma B.12 yields the following upper bound:

Original: [Equation]
Revised: [Equation]

Original: For the second sum T
Revised: For the second sum T

Original: [Equation]
Revised: [Equation]

Original: applying Lemma B.9 and inductive hypothesis yields:
Revised: applying Lemma B.9 and inductive hypothesis yields:

Original: [Equation]
Revised: [Equation]

Original: Combining the two upper bounds on T 1 and T 2 yields:
Revised: Combining the two upper bounds on T 1 and T 2 yields:

Original: [Equation]
Revised: [Equation]

Original: By taking supremum with respect to 1 ≤ l ≤ W on the LHS yields:
Revised: By taking supremum with respect to 1 ≤ l ≤ W on the LHS yields:

Original: [Equation]
Revised: [Equation]

Original: By adding the two upper bounds in B.31 and B.32, we can deduce that:
Revised: By adding the two upper bounds in B.31 and B.32, we can deduce that:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality above follows from k ≥ 2 .
Revised: where the last inequality above follows from k ≥ 2 .

Original: Taking supremum with respect to x ∈ Ω on the
Revised: Taking supremum with respect to x ∈ Ω on the

Original: LHS implies the given upper bound also holds for k .
Revised: LHS implies the given upper bound also holds for k .

Original: By induction, the claim is proved.
Revised: By induction, the claim is proved.

Original: In particular, when k = L , we have ∇ F L ( x ) =
Revised: In particular, when k = L , we have ∇ F L ( x ) =

Original: J [ F L ]( x ) T for any x ∈ Ω .
Revised: J [ F L ]( x ) T for any x ∈ Ω .

Original: Applying Lemma B. then yields:
Revised: Applying Lemma B. then yields:

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes our proof of the Lemma.
Revised: This ﬁnishes our proof of the Lemma.

Original: Theorem B.5.
Revised: Theorem B.5.

Original: (Bounding the DNN Gradient space covering number)
Revised: (Bounding the DNN Gradient space covering number)

Original: Fix some sufﬁciently large N ∈ Z + .
Revised: Fix some sufﬁciently large N ∈ Z + .

Original: Consider a Deep Neural Network space Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =
Revised: Consider a Deep Neural Network space Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) .
Revised: = O ( N ) .

Original: Then the log value of the covering number of the DNN Gradient space with respect to the (cid:107) · (cid:107) ∞ norm (cid:107) F ( x ) (cid:107) ∞ := sup x ∈ Ω | F ( x ) | , which is denoted by N ( δ, ∇ Φ( L, W, S, B ) , (cid:107) · (cid:107) ∞ ) , can be upper bounded by:
Revised: Then the log value of the covering number of the DNN Gradient space with respect to the (cid:107) · (cid:107) ∞ norm (cid:107) F ( x ) (cid:107) ∞ := sup x ∈ Ω | F ( x ) | , which is denoted by N ( δ, ∇ Φ( L, W, S, B ) , (cid:107) · (cid:107) ∞ ) , can be upper bounded by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrstly ﬁx a sparsity pattern (i.e, the locations of the non-zero entries are ﬁxed).
Revised: We ﬁrstly ﬁx a sparsity pattern (i.e, the locations of the non-zero entries are ﬁxed).

Original: Using equation B.29 in Lemma B.14, yields the following upper bound on the covering number with respect to (cid:107) · (cid:107) ∞ :
Revised: Using equation B.29 in Lemma B.14, yields the following upper bound on the covering number with respect to (cid:107) · (cid:107) ∞ :

Original: [Equation]
Revised: [Equation]

Original: Furthermore, note that the number of feasible conﬁgurations is upper bounded by: (cid:0) ( W +1)
Revised: Furthermore, note that the number of feasible conﬁgurations is upper bounded by: (cid:0) ( W +1)

Original: L S (cid:1) ≤ ( W + 1) LS .
Revised: L S (cid:1) ≤ ( W + 1) LS .

Original: (Schmidt-Hieber et al., 2020; Farrell et al., 2021).
Revised: (Schmidt-Hieber et al., 2020; Farrell et al., 2021).

Original: Plug this inequality into the previous estimation then yields:
Revised: Plug this inequality into the previous estimation then yields:

Original: [Equation]
Revised: [Equation]

Original: Note that here the dimension d is some constant.
Revised: Note that here the dimension d is some constant.

Original: Thus, by plugging in thee given magnitudes
Revised: Thus, by plugging in thee given magnitudes

Original: L = O (1) , W = O ( N ) , S =
Revised: L = O (1) , W = O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) , we can further deduce that:
Revised: = O ( N ) , we can further deduce that:

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes our proof.
Revised: This ﬁnishes our proof.

Original: Now let’s consider upper bounding the covering number of the Laplacian of the sparse Deep
Revised: Now let’s consider upper bounding the covering number of the Laplacian of the sparse Deep

Original: Neural Networks.
Revised: Neural Networks.

Original: Note that for any 1 ≤ k ≤ L − 1 , any k − ReLU3 Deep Neural Network
Revised: Note that for any 1 ≤ k ≤ L − 1 , any k − ReLU3 Deep Neural Network

Original: F k ∈ Φ k ( L, W, S, B ) is a vector-valued function mapping from R d to R W .
Revised: F k ∈ Φ k ( L, W, S, B ) is a vector-valued function mapping from R d to R W .

Original: Moreover, we deﬁne the
Revised: Moreover, we deﬁne the

Original: Laplacian of F k ( x ) , which is denoted by ∆[ F k ]( x ) , as follows:
Revised: Laplacian of F k ( x ) , which is denoted by ∆[ F k ]( x ) , as follows:

Original: [Equation]
Revised: [Equation]

Original: where for any 1 ≤ l ≤ W , we have:
Revised: where for any 1 ≤ l ≤ W , we have:

Original: [Equation]
Revised: [Equation]

Original: In particular, when k = L , we have that any F L ∈ Φ L ( L, W, S, B ) =
Revised: In particular, when k = L , we have that any F L ∈ Φ L ( L, W, S, B ) =

Original: Φ( L, W, S, B ) is a scalarvalued function mapping from R d to R .
Revised: Φ( L, W, S, B ) is a scalarvalued function mapping from R d to R .

Original: Thus, its Laplacian can be explicitly written as:
Revised: Thus, its Laplacian can be explicitly written as:

Original: [Equation]
Revised: [Equation]

Original: For both Lemma B.15 and Lemma B.16 below, we consider a ﬁxed Deep Neural Network space Φ( L, W, S, B ) with L
Revised: For both Lemma B.15 and Lemma B.16 below, we consider a ﬁxed Deep Neural Network space Φ( L, W, S, B ) with L

Original: = O (1) , W = O ( N ) , S =
Revised: = O (1) , W = O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) , where N ∈ Z + is ﬁxed and sufﬁciently large.
Revised: = O ( N ) , where N ∈ Z + is ﬁxed and sufﬁciently large.

Original: Lemma B.15.
Revised: Lemma B.15.

Original: (Upper bound on ∞ -norm of Laplacian of elements in the DNN space)
Revised: (Upper bound on ∞ -norm of Laplacian of elements in the DNN space)

Original: For any 1 ≤ k ≤ L , we have the following upper bound:
Revised: For any 1 ≤ k ≤ L , we have the following upper bound:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use induction on k to prove the claim.
Revised: We use induction on k to prove the claim.

Original: Base case: k = 1 .
Revised: Base case: k = 1 .

Original: Note that any F 1 ∈ Φ 1 ( L, W, S, B ) is a linear transform, so the Laplacian ∆[ F 1 ]( x ) must be the zero vector for any x ∈ Ω .
Revised: Note that any F 1 ∈ Φ 1 ( L, W, S, B ) is a linear transform, so the Laplacian ∆[ F 1 ]( x ) must be the zero vector for any x ∈ Ω .

Original: This implies:
Revised: This implies:

Original: [Equation]
Revised: [Equation]

Original: Inductive Step: Assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .
Revised: Inductive Step: Assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .

Original: For any x ∈ Ω and any F k ∈ Φ k ( L, W, S, B ) , using linearity of the Laplacian operator implies:
Revised: For any x ∈ Ω and any F k ∈ Φ k ( L, W, S, B ) , using linearity of the Laplacian operator implies:

Original: [Equation]
Revised: [Equation]

Original: Taking the inf-norm on both sides of the identity above implies:
Revised: Taking the inf-norm on both sides of the identity above implies:

Original: [Equation]
Revised: [Equation]

Original: It now remains to upper bound the term (cid:107) ∆[ η 3 ◦ F k − 1 ]( x ) (cid:107) ∞ .
Revised: It now remains to upper bound the term (cid:107) ∆[ η 3 ◦ F k − 1 ]( x ) (cid:107) ∞ .

Original: For any 1 ≤ l ≤ W , we will use the
Revised: For any 1 ≤ l ≤ W , we will use the

Original: Chain Rule to write the l -th component (cid:16) ∆[ η 3 ◦ F k − 1 ]( x ) (cid:17) l in an explicit form.
Revised: Chain Rule to write the l -th component (cid:16) ∆[ η 3 ◦ F k − 1 ]( x ) (cid:17) l in an explicit form.

Original: For any 1 ≤ j ≤ d , we have:
Revised: For any 1 ≤ j ≤ d , we have:

Original: [Equation]
Revised: [Equation]

Original: Differentiating with respect to x j on both sides above yields:
Revised: Differentiating with respect to x j on both sides above yields:

Original: [Equation]
Revised: [Equation]

Original: Summing the expression above from j = 1 to j = d implies:
Revised: Summing the expression above from j = 1 to j = d implies:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: We denote the two summations above by U 1 and U 2 , respectively:
Revised: We denote the two summations above by U 1 and U 2 , respectively:

Original: [Equation]
Revised: [Equation]

Original: On the one hand, by applying Lemma B.9 and Lemma B.12, we can upper bound U 1 by:
Revised: On the one hand, by applying Lemma B.9 and Lemma B.12, we can upper bound U 1 by:

Original: [Equation]
Revised: [Equation]

Original: where the last step above follows from k ≤ L and L
Revised: where the last step above follows from k ≤ L and L

Original: = O (1) .
Revised: = O (1) .

Original: On the other hand, by applying Lemma B.9 and the inductive hypothesis
Revised: On the other hand, by applying Lemma B.9 and the inductive hypothesis

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: where the last step above follows from k ≤ L and L
Revised: where the last step above follows from k ≤ L and L

Original: = O (1) .
Revised: = O (1) .

Original: Summing the two bounds on U 1 and U 2 implies that for any 1 ≤ l ≤ W
Revised: Summing the two bounds on U 1 and U 2 implies that for any 1 ≤ l ≤ W

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Taking supremum with respect to 1 ≤ l ≤ W then yields:
Revised: Taking supremum with respect to 1 ≤ l ≤ W then yields:

Original: [Equation]
Revised: [Equation]

Original: Taking supremum with respect to x ∈ Ω and F k ∈ Φ k ( L, W, S, B ) on the LHS implies that the given upper bound also holds for k .
Revised: Taking supremum with respect to x ∈ Ω and F k ∈ Φ k ( L, W, S, B ) on the LHS implies that the given upper bound also holds for k .

Original: By induction, the claim is proved.
Revised: By induction, the claim is proved.

Original: Lemma B.16.
Revised: Lemma B.16.

Original: (Relation between the covering number of the DNN Laplacian space and parameter space)
Revised: (Relation between the covering number of the DNN Laplacian space and parameter space)

Original: For any 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L, W, S, B ) are given by:
Revised: For any 1 ≤ k ≤ L , suppose that a pair of different two networks F k , G k ∈ Φ k ( L, W, S, B ) are given by:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, assume that the (cid:107) (cid:107) ∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ , i.e
Revised: Furthermore, assume that the (cid:107) (cid:107) ∞ norm of the distance between the parameter spaces is uniformly upper bounded by δ , i.e

Original: [Equation]
Revised: [Equation]

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use induction on k to prove the claim.
Revised: We use induction on k to prove the claim.

Original: Base case: k = 1 .
Revised: Base case: k = 1 .

Original: Note that any F 1 ∈ Φ 1 ( L, W, S, B ) is a linear transform, so the Laplacian ∆[ F 1 ]( x ) must be the zero vector for any x ∈ Ω .
Revised: Note that any F 1 ∈ Φ 1 ( L, W, S, B ) is a linear transform, so the Laplacian ∆[ F 1 ]( x ) must be the zero vector for any x ∈ Ω .

Original: Hence, for any x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L, W, S, B ) , we have:
Revised: Hence, for any x ∈ Ω and any F 1 , G 1 ∈ Φ 1 ( L, W, S, B ) , we have:

Original: [Equation]
Revised: [Equation]

Original: Inductive Step: assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .
Revised: Inductive Step: assume that the claim has been proved for k − 1 , where 2 ≤ k ≤ L .

Original: Then for any x ∈ Ω and F k , G k ∈
Revised: Then for any x ∈ Ω and F k , G k ∈

Original: Φ k ( L, W, S, B ) satisfying constraint B.36, applying linearity of the Laplacian operator indicates:
Revised: Φ k ( L, W, S, B ) satisfying constraint B.36, applying linearity of the Laplacian operator indicates:

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term (cid:107)W ( k ) F − W ( k ) G (cid:107) ∞
Revised: For the ﬁrst term (cid:107)W ( k ) F − W ( k ) G (cid:107) ∞

Original: [Equation]
Revised: [Equation]

Original: ∆[ η
Revised: ∆[ η

Original: [Equation]
Revised: [Equation]

Original: ◦ F k − 1 ]( x ) (cid:107) ∞ , applying the bound in equation B.35 and equation B.36 yields:
Revised: ◦ F k − 1 ]( x ) (cid:107) ∞ , applying the bound in equation B.35 and equation B.36 yields:

Original: [Equation]
Revised: [Equation]

Original: For the second term (cid:107)W ( k ) G
Revised: For the second term (cid:107)W ( k ) G

Original: [Equation]
Revised: [Equation]

Original: ∞
Revised: ∞

Original: [Equation]
Revised: [Equation]

Original: ∆[ η 3 ◦ F k − 1 ]( x ) − ∆[ η 3 ◦ G k − 1 ]( x ) (cid:107) ∞ , we need to upper bound the norm (cid:107) ∆[ η 3 ◦ F k − 1 ]( x ) − ∆[ η 3 ◦ G k − 1 ]( x ) (cid:107) ∞ at ﬁrst.
Revised: ∆[ η 3 ◦ F k − 1 ]( x ) − ∆[ η 3 ◦ G k − 1 ]( x ) (cid:107) ∞ , we need to upper bound the norm (cid:107) ∆[ η 3 ◦ F k − 1 ]( x ) − ∆[ η 3 ◦ G k − 1 ]( x ) (cid:107) ∞ at ﬁrst.

Original: Note that for any 1 ≤ l ≤ W , we can use equation B.34 to write the l -th component of ∆[ η 3 ◦ F k − 1 ]( x ) − ∆[ η 3 ◦ G k − 1 ]( x ) as: (cid:16)
Revised: Note that for any 1 ≤ l ≤ W , we can use equation B.34 to write the l -th component of ∆[ η 3 ◦ F k − 1 ]( x ) − ∆[ η 3 ◦ G k − 1 ]( x ) as: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: We denote the four summations above by V 1 , V 2 , V 3 and V
Revised: We denote the four summations above by V 1 , V 2 , V 3 and V

Original: [Equation]
Revised: [Equation]

Original: respectively:
Revised: respectively:

Original: [Equation]
Revised: [Equation]

Original: j = (cid:16) ∂ ∂x j F k − 1 ,l ( x ) (cid:17) 2 ,
Revised: j = (cid:16) ∂ ∂x j F k − 1 ,l ( x ) (cid:17) 2 ,

Original: V 2 := 6 η 1 [ G k − 1 ,l ( x )]
Revised: V 2 := 6 η 1 [ G k − 1 ,l ( x )]

Original: d (cid:88)
Revised: d (cid:88)

Original: j = (cid:16) ∂
Revised: j = (cid:16) ∂

Original: ∂x j F k − 1 ,l ( x ) (cid:17) 2 − 6 η 

Revised: ∂x j F k − 1 ,l ( x ) (cid:17) 2 − 6 η

Original: [ G k − 1 ,l ( x )] d (cid:88)
Revised: [ G k − 1 ,l ( x )] d (cid:88)

Original: j = (cid:16) ∂
Revised: j = (cid:16) ∂

Original: ∂x j G k − 1 ,l ( x ) (cid:17) 2 ,
Revised: ∂x j G k − 1 ,l ( x ) (cid:17) 2 ,

Original: V 3 :
Revised: V 3 :

Original: =
Revised: =

Original: 3 η 2 [ F k − 1 ,l ( x )]
Revised: 3 η 2 [ F k − 1 ,l ( x )]

Original: d (cid:88)
Revised: d (cid:88)

Original: j = ∂∂x 

Revised: j = ∂∂x

Original: j F k − 1 ,l ( x ) − 3 η 

Revised: j F k − 1 ,l ( x ) − 3 η

Original: [ G k − 1 ,l ( x )] d (cid:88)
Revised: [ G k − 1 ,l ( x )] d (cid:88)

Original: j = ∂∂x 

Revised: j = ∂∂x

Original: j F k − 1 ,l ( x ) ,
Revised: j F k − 1 ,l ( x ) ,

Original: V 4 :
Revised: V 4 :

Original: =
Revised: =

Original: 3 η 2 [ G k − 1 ,l ( x )] d (cid:88)
Revised: 3 η 2 [ G k − 1 ,l ( x )] d (cid:88)

Original: j = ∂∂x 

Revised: j = ∂∂x

Original: j F k − 1 ,l ( x ) − 3 η 

Revised: j F k − 1 ,l ( x ) − 3 η

Original: [ G k − 1 ,l ( x )] d (cid:88)
Revised: [ G k − 1 ,l ( x )] d (cid:88)

Original: j = ∂∂x 

Revised: j = ∂∂x

Original: j G k − 1 ,l ( x ) .
Revised: j G k − 1 ,l ( x ) .

Original: By applying Lemma B.9, Lemma B.10, Lemma B.11 and Lemma B.12, we can upper bound V 1 by:
Revised: By applying Lemma B.9, Lemma B.10, Lemma B.11 and Lemma B.12, we can upper bound V 1 by:

Original: V
Revised: V

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last step above follows from k ≤ L and L
Revised: where the last step above follows from k ≤ L and L

Original: = O (1) .
Revised: = O (1) .

Original: Furthermore, note that for any 1 ≤ j ≤ d , we can upper bound the difference (cid:16) ∂∂x j F k − 1 ,l ( x ) (cid:17) 2 −
Revised: Furthermore, note that for any 1 ≤ j ≤ d , we can upper bound the difference (cid:16) ∂∂x j F k − 1 ,l ( x ) (cid:17) 2 −

Original: [Equation]
Revised: [Equation]

Original: ∂∂x j G k − 1 ,l ( x ) (cid:17) 2 as follows:
Revised: ∂∂x j G k − 1 ,l ( x ) (cid:17) 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that η 1 ( G k − 1 ,l ( x )) ≥ 0 .
Revised: Note that η 1 ( G k − 1 ,l ( x )) ≥ 0 .

Original: Combining the non-negativity with equation B.39, Lemma B.9,
Revised: Combining the non-negativity with equation B.39, Lemma B.9,

Original: Lemma B.12 and Lemma B.14 helps us upper bound V 2 by:
Revised: Lemma B.12 and Lemma B.14 helps us upper bound V 2 by:

Original: [Equation]
Revised: [Equation]

Original: 6 η 1 [ G k − 1 ,l ( x )]
Revised: 6 η 1 [ G k − 1 ,l ( x )]

Original: d (cid:88) j =
Revised: d (cid:88) j =

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last step above follows from k ≤ L and L
Revised: where the last step above follows from k ≤ L and L

Original: = O (1)
Revised: = O (1)

Original: [Equation]
Revised: [Equation]

Original: Moreover, using Lemma B.9, Lemma B.10 and Lemma B.15 helps us upper bound V 3 by:
Revised: Moreover, using Lemma B.9, Lemma B.10 and Lemma B.15 helps us upper bound V 3 by:

Original: [Equation]
Revised: [Equation]

Original: where the last step above follows from k ≤ L and L
Revised: where the last step above follows from k ≤ L and L

Original: = O (1) .
Revised: = O (1) .

Original: Finally, applying Lemma B.9 and inductive hypothesis helps us upper bound V 4 by:
Revised: Finally, applying Lemma B.9 and inductive hypothesis helps us upper bound V 4 by:

Original: [Equation]
Revised: [Equation]

Original: where the last step above follows from k ≤ L and L
Revised: where the last step above follows from k ≤ L and L

Original: = O (1) .
Revised: = O (1) .

Original: Combining the four bounds on V 1 , V 2 , V 3 and V 4 implies:
Revised: Combining the four bounds on V 1 , V 2 , V 3 and V 4 implies:

Original: [Equation]
Revised: [Equation]

Original: Taking supremum with respect to 1 ≤ l ≤ W gives us an upper bound on the second term
Revised: Taking supremum with respect to 1 ≤ l ≤ W gives us an upper bound on the second term

Original: [Equation]
Revised: [Equation]

Original: Combining the two bounds derived in equation B.38 and equation B.40 then implies:
Revised: Combining the two bounds derived in equation B.38 and equation B.40 then implies:

Original: [Equation]
Revised: [Equation]

Original: Taking supremum with respect to x ∈ Ω on the LHS implies that the given upper bound also holds for k .
Revised: Taking supremum with respect to x ∈ Ω on the LHS implies that the given upper bound also holds for k .

Original: By induction, the claim is proved.
Revised: By induction, the claim is proved.

Original: Given a Neural Network function space Φ( L, W, S, B ) , we deﬁne a corresponding Neural Network
Revised: Given a Neural Network function space Φ( L, W, S, B ) , we deﬁne a corresponding Neural Network

Original: Laplacian space ∆Φ( L, W, S, B ) as:
Revised: Laplacian space ∆Φ( L, W, S, B ) as:

Original: [Equation]
Revised: [Equation]

Original: Theorem B.6.
Revised: Theorem B.6.

Original: (Bounding the Neural Network Laplacian space covering number)
Revised: (Bounding the Neural Network Laplacian space covering number)

Original: Fix some sufﬁciently large N ∈ Z + .
Revised: Fix some sufﬁciently large N ∈ Z + .

Original: Consider a Deep Neural Network space Φ( L, W, S, B ) with L = O (1) , W =
Revised: Consider a Deep Neural Network space Φ( L, W, S, B ) with L = O (1) , W =

Original: O ( N ) , S =
Revised: O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) .
Revised: = O ( N ) .

Original: Then the log value of the covering number of the DNN
Revised: Then the log value of the covering number of the DNN

Original: Laplacian space with respect to the (cid:107) · (cid:107) ∞ norm (cid:107) F ( x ) (cid:107) ∞ :
Revised: Laplacian space with respect to the (cid:107) · (cid:107) ∞ norm (cid:107) F ( x ) (cid:107) ∞ :

Original: = sup x ∈ Ω | F ( x ) | , which is denoted by
Revised: = sup x ∈ Ω | F ( x ) | , which is denoted by

Original: N ( δ, ∆Φ
Revised: N ( δ, ∆Φ

Original: ( L, W, S, B ) , (cid:107) · (cid:107) ∞ ) , can be upper bounded by:
Revised: ( L, W, S, B ) , (cid:107) · (cid:107) ∞ ) , can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrstly ﬁx a sparsity pattern (i.e, the locations of the non-zero entries are ﬁxed).
Revised: We ﬁrstly ﬁx a sparsity pattern (i.e, the locations of the non-zero entries are ﬁxed).

Original: Applying
Revised: Applying

Original: Lemma B.16 yields that there exists some constant C
Revised: Lemma B.16 yields that there exists some constant C

Original: = O (1) , such that the covering number with respect to (cid:107) · (cid:107) ∞ can be upper bounded by:
Revised: = O (1) , such that the covering number with respect to (cid:107) · (cid:107) ∞ can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, note that the number of feasible conﬁgurations is upper bounded by (cid:0) ( W +1)
Revised: Furthermore, note that the number of feasible conﬁgurations is upper bounded by (cid:0) ( W +1)

Original: L S
Revised: L S

Original: [Equation]
Revised: [Equation]

Original: ≤ ( W + 1) LS (Schmidt-Hieber et al., 2020; Farrell et al., 2021).
Revised: ≤ ( W + 1) LS (Schmidt-Hieber et al., 2020; Farrell et al., 2021).

Original: Then we plug this into the pervious estimation and yields:
Revised: Then we plug this into the pervious estimation and yields:

Original: [Equation]
Revised: [Equation]

Original: Note that here the dimension d is some constant.
Revised: Note that here the dimension d is some constant.

Original: Thus, by plugging in thee given magnitudes
Revised: Thus, by plugging in thee given magnitudes

Original: L = O (1) , W = O ( N ) , S =
Revised: L = O (1) , W = O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) , we can further deduce that:
Revised: = O ( N ) , we can further deduce that:

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes our proof.
Revised: This ﬁnishes our proof.

Original: Lemma B.17 (Local Rademacher Complexity Bound of DNN Estimator for Deep Ritz Method) .
Revised: Lemma B.17 (Local Rademacher Complexity Bound of DNN Estimator for Deep Ritz Method) .

Original: Consider a Deep Neural Network space F (Ω) =
Revised: Consider a Deep Neural Network space F (Ω) =

Original: Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =
Revised: Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =

Original: O ( N ) and B =
Revised: O ( N ) and B =

Original: O ( N ) , where N ∈ Z
Revised: O ( N ) , where N ∈ Z

Original: + is ﬁxed to be sufﬁciently large.
Revised: + is ﬁxed to be sufﬁciently large.

Original: Moreover, assume that the gradients and function value of F (Ω) , u ∗ , V and f are uniformly bounded max
Revised: Moreover, assume that the gradients and function value of F (Ω) , u ∗ , V and f are uniformly bounded max

Original: [Equation]
Revised: [Equation]

Original: For any ρ > 0 , we consider a localized set
Revised: For any ρ > 0 , we consider a localized set

Original: L ρ deﬁned by:
Revised: L ρ deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: Then for any ρ (cid:38) n − 2 , the Rademacher complexity of a localized function space S ρ (Ω) := (cid:110)
Revised: Then for any ρ (cid:38) n − 2 , the Rademacher complexity of a localized function space S ρ (Ω) := (cid:110)

Original: [Equation]
Revised: [Equation]

Original: can be upper bounded by a sub-root function
Revised: can be upper bounded by a sub-root function

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. we have φ
Revised: e. we have φ

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: and R n
Revised: and R n

Original: [Equation]
Revised: [Equation]

Original: S ρ
Revised: S ρ

Original: [Equation]
Revised: [Equation]

Original: Ω
Revised: Ω

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: holds for all ρ (cid:38) n − 2 .
Revised: holds for all ρ (cid:38) n − 2 .

Original: Proof.
Revised: Proof.

Original: Firstly, we will check that for any u ∈ L ρ (Ω) , the corresponding function h in S ρ (Ω) is
Revised: Firstly, we will check that for any u ∈ L ρ (Ω) , the corresponding function h in S ρ (Ω) is

Original: Lipschitz with respect to u − u ∗ and (cid:107)∇ u (cid:107) − (cid:107)∇ u ∗ (cid:107) .
Revised: Lipschitz with respect to u − u ∗ and (cid:107)∇ u (cid:107) − (cid:107)∇ u ∗ (cid:107) .

Original: Note that for any u 1 , u 2 ∈ L ρ (Ω) with corresponding functions h 1 , h 2 ∈ S ρ (Ω) , applying boundedness condition B.43 yields:
Revised: Note that for any u 1 , u 2 ∈ L ρ (Ω) with corresponding functions h 1 , h 2 ∈ S ρ (Ω) , applying boundedness condition B.43 yields:

Original: [Equation]
Revised: [Equation]

Original: Let’s pick L = C 2 + C > C .
Revised: Let’s pick L = C 2 + C > C .

Original: Applying the Talagrand Contraction Lemma B.2 helps us upper bound the local Rademacher complexity R n ( S ρ (Ω)) by
Revised: Applying the Talagrand Contraction Lemma B.2 helps us upper bound the local Rademacher complexity R n ( S ρ (Ω)) by

Original: [Equation]
Revised: [Equation]

Original: From the localization constraint ρ ≥ (cid:107) u − u ∗ (cid:107) 2 H 1 (Ω) = (cid:107) u − u ∗ (cid:107) 2 L 2 (Ω) + (cid:107)∇ u − ∇ u ∗ (cid:107)
Revised: From the localization constraint ρ ≥ (cid:107) u − u ∗ (cid:107) 2 H 1 (Ω) = (cid:107) u − u ∗ (cid:107) 2 L 2 (Ω) + (cid:107)∇ u − ∇ u ∗ (cid:107)

Original: L 2 (Ω) , we can deduce that (cid:107)
Revised: L 2 (Ω) , we can deduce that (cid:107)

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: and (cid:107)∇ u − ∇ u ∗ (cid:107)
Revised: and (cid:107)∇ u − ∇ u ∗ (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ≤ √
Revised: ≤ √

Original: [Equation]
Revised: [Equation]

Original: (B.45)
Revised: (B.45)

Original: Moreover, note that Ω ⊂ [0 , 1] d .
Revised: Moreover, note that Ω ⊂ [0 , 1] d .

Original: Applying triangle inequality yields:
Revised: Applying triangle inequality yields:

Original: [Equation]
Revised: [Equation]

Original: Using inequality B.45 and inequality
Revised: Using inequality B.45 and inequality

Original: B.
Revised: B.

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: (Geer & van de Geer, 2000; Rakhlin et al., 2017) showed a “upper isometry” property
Revised: (Geer & van de Geer, 2000; Rakhlin et al., 2017) showed a “upper isometry” property

Original: [Equation]
Revised: [Equation]

Original: where the metric (cid:107) · (cid:107) L 2 is equivalent to (cid:107) · (cid:107) n, 2 with high probability.
Revised: where the metric (cid:107) · (cid:107) L 2 is equivalent to (cid:107) · (cid:107) n, 2 with high probability.

Original: Combining this fact with Theorem B
Revised: Combining this fact with Theorem B

Original: [Equation]
Revised: [Equation]

Original: 3, we can bound the local Rademacher complexities using Dudley integral:
Revised: 3, we can bound the local Rademacher complexities using Dudley integral:

Original: [Equation]
Revised: [Equation]

Original: For any ρ (cid:38) 1 n 2 , we pick α = 1 n (cid:46) √ ρ and plug in the upper bounds proved in Theorem B.4 and
Revised: For any ρ (cid:38) 1 n 2 , we pick α = 1 n (cid:46) √ ρ and plug in the upper bounds proved in Theorem B.4 and

Original: Theorem B.5, which implies:
Revised: Theorem B.5, which implies:

Original: [Equation]
Revised: [Equation]

Original: Lemma B.18 (Local Rademacher Complexity Bound of DNN Estimator for Physics Informed Neural
Revised: Lemma B.18 (Local Rademacher Complexity Bound of DNN Estimator for Physics Informed Neural

Original: Network) .
Revised: Network) .

Original: Consider a Deep Neural Network space F (Ω) =
Revised: Consider a Deep Neural Network space F (Ω) =

Original: Φ( L, W, S, B ) with L = O (1) , W =
Revised: Φ( L, W, S, B ) with L = O (1) , W =

Original: O ( N ) , S =
Revised: O ( N ) , S =

Original: O ( N ) and B
Revised: O ( N ) and B

Original: = O ( N ) , where N ∈ Z + is ﬁxed to be sufﬁciently large.
Revised: = O ( N ) , where N ∈ Z + is ﬁxed to be sufﬁciently large.

Original: Moreover, assume that the gradients and function value of F (Ω) , u ∗ , V and f are uniformly bounded
Revised: Moreover, assume that the gradients and function value of F (Ω) , u ∗ , V and f are uniformly bounded

Original: [Equation]
Revised: [Equation]

Original: (B.47)
Revised: (B.47)

Original: For any ρ > 0 , we consider a localized set
Revised: For any ρ > 0 , we consider a localized set

Original: M ρ deﬁned by:
Revised: M ρ deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: Then for any ρ (cid:38) n − 2 , the Rademacher complexity of a localized function space T ρ (Ω) :=
Revised: Then for any ρ (cid:38) n − 2 , the Rademacher complexity of a localized function space T ρ (Ω) :=

Original: [Equation]
Revised: [Equation]

Original: h := | Ω
Revised: h := | Ω

Original: [Equation]
Revised: [Equation]

Original: (Ω) (cid:111) can be upper bounded by a sub-root function
Revised: (Ω) (cid:111) can be upper bounded by a sub-root function

Original: [Equation]
Revised: [Equation]

Original: i.e. we have
Revised: i.e. we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: holds for all ρ (cid:38) n − 2 .
Revised: holds for all ρ (cid:38) n − 2 .

Original: Proof.
Revised: Proof.

Original: Firstly, we will check that for any u ∈ L ρ (Ω) , the corresponding function h in S ρ (Ω) is
Revised: Firstly, we will check that for any u ∈ L ρ (Ω) , the corresponding function h in S ρ (Ω) is

Original: Lipschitz with respect to u − u ∗ and ∆ u − ∆ u ∗ .
Revised: Lipschitz with respect to u − u ∗ and ∆ u − ∆ u ∗ .

Original: Note that for any u 1 , u 2 ∈ L ρ (Ω) with corresponding functions h 1 , h 2 ∈ S ρ (Ω) , applying boundedness condition B.47 yields:
Revised: Note that for any u 1 , u 2 ∈ L ρ (Ω) with corresponding functions h 1 , h 2 ∈ S ρ (Ω) , applying boundedness condition B.47 yields:

Original: [Equation]
Revised: [Equation]

Original: Let’s pick L = max { 2 C 2 +4 C, 2 C 3 +4 C 2 } .
Revised: Let’s pick L = max { 2 C 2 +4 C, 2 C 3 +4 C 2 } .

Original: Applying the Talagrand Contraction Lemma B.2 helps us upper bound the local Rademacher complexity R n ( T ρ (Ω)) by
Revised: Applying the Talagrand Contraction Lemma B.2 helps us upper bound the local Rademacher complexity R n ( T ρ (Ω)) by

Original: [Equation]
Revised: [Equation]

Original: ∆ u − ∆ u
Revised: ∆ u − ∆ u

Original: [Equation]
Revised: [Equation]

Original: (cid:107) L 2 (Ω) ≤ √ ρ (cid:111)(cid:17) ≤ R n (cid:16)(cid:110)
Revised: (cid:107) L 2 (Ω) ≤ √ ρ (cid:111)(cid:17) ≤ R n (cid:16)(cid:110)

Original: u − u ∗ : u ∈ Φ( L, W, S, B ) , (cid:107) u − u ∗ (cid:107) n, 2 ≤ 2 √ ρ (cid:111)(cid:17) + R n (cid:16)(cid:110)
Revised: u − u ∗ : u ∈ Φ( L, W, S, B ) , (cid:107) u − u ∗ (cid:107) n, 2 ≤ 2 √ ρ (cid:111)(cid:17) + R n (cid:16)(cid:110)

Original: ∆ u − ∆ u ∗ : u ∈ Φ( L, W, S, B ) , (cid:107) ∆ u − ∆ u ∗ (cid:107) n, 2 ≤ 2 √ ρ (cid:111)(cid:17) (cid:46) inf 0 <
Revised: ∆ u − ∆ u ∗ : u ∈ Φ( L, W, S, B ) , (cid:107) ∆ u − ∆ u ∗ (cid:107) n, 2 ≤ 2 √ ρ (cid:111)(cid:17) (cid:46) inf 0 <

Original: [Equation]
Revised: [Equation]

Original: (cid:8)
Revised: (cid:8)

Original: [Equation]
Revised: [Equation]

Original: (cid:113)
Revised: (cid:113)

Original: [Equation]
Revised: [Equation]

Original: N ( δ,
Revised: N ( δ,

Original: [Equation]
Revised: [Equation]

Original: ( L, W, S, B ) , (cid:107) · (cid:107) n, 2 ) dδ
Revised: ( L, W, S, B ) , (cid:107) · (cid:107) n, 2 ) dδ

Original: [Equation]
Revised: [Equation]

Original: + inf
Revised: + inf

Original: [Equation]
Revised: [Equation]

Original: inf
Revised: inf

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: + inf
Revised: + inf

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: N ( δ
Revised: N ( δ

Original: [Equation]
Revised: [Equation]

Original: ( L
Revised: ( L

Original: [Equation]
Revised: [Equation]

Original: W, S, B ) , (cid:107) · (cid:107) ∞ ) dδ (cid:9) .
Revised: W, S, B ) , (cid:107) · (cid:107) ∞ ) dδ (cid:9) .

Original: For any ρ (cid:38) 1 n 2 , we pick α = 1 n (cid:46) √ ρ and plug in the upper bounds proved in Theorem B.4 and
Revised: For any ρ (cid:38) 1 n 2 , we pick α = 1 n (cid:46) √ ρ and plug in the upper bounds proved in Theorem B.4 and

Original: Theorem B.5, which implies:
Revised: Theorem B.5, which implies:

Original: [Equation]
Revised: [Equation]

Original: B. A UXILIARY DEFINITIONS AND LEMMATA O N A PPROXIMATION E RROR B.3. A PPROXIMATION USING T RUNCATED F OURIER B ASIS
Revised: B. A UXILIARY DEFINITIONS AND LEMMATA O N A PPROXIMATION E RROR B.3. A PPROXIMATION USING T RUNCATED F OURIER B ASIS

Original: Lemma B.19.
Revised: Lemma B.19.

Original: Given α > 0 and a ﬁxed integer ξ ∈ Z + .
Revised: Given α > 0 and a ﬁxed integer ξ ∈ Z + .

Original: For any function f ∈ H α (Ω)
Revised: For any function f ∈ H α (Ω)

Original: [Equation]
Revised: [Equation]

Original: we let f ξ = (cid:80)
Revised: we let f ξ = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: z (cid:107) ∞
Revised: z (cid:107) ∞

Original: ≤ ξ f z φ z be the best approximation of f in the space F ξ (Ω) .
Revised: ≤ ξ f z φ z be the best approximation of f in the space F ξ (Ω) .

Original: Then for any 0 < β ≤ α , we have the following inequality:
Revised: Then for any 0 < β ≤ α , we have the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For f ∈ H
Revised: For f ∈ H

Original: α (Ω)
Revised: α (Ω)

Original: [Equation]
Revised: [Equation]

Original: we know the Fourier coefﬁcient satisﬁes
Revised: we know the Fourier coefﬁcient satisﬁes

Original: [Equation]
Revised: [Equation]

Original: We directly construct f ξ = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z φ z to be the truncated Fourier series of the function f , then we have
Revised: We directly construct f ξ = (cid:80) (cid:107) z (cid:107) ∞ ≤ ξ f z φ z to be the truncated Fourier series of the function f , then we have

Original: [Equation]
Revised: [Equation]

Original: B.3. A PPROXIMATION USING N EURAL N ETWORK
Revised: B.3. A PPROXIMATION USING N EURAL N ETWORK

Original: In this section, we aim to provide approximation bound for deep neural network.
Revised: In this section, we aim to provide approximation bound for deep neural network.

Original: Our proof of the approximation upper bound is based on the observation that the B-spline approximation(De Boor &
Revised: Our proof of the approximation upper bound is based on the observation that the B-spline approximation(De Boor &

Original: De Boor, 1978; Schumaker, 2007) can be formulated as a ReLU3 neural network efﬁciently(Suzuki, 2018; Gühring et al., 2020;
Revised: De Boor, 1978; Schumaker, 2007) can be formulated as a ReLU3 neural network efﬁciently(Suzuki, 2018; Gühring et al., 2020;

Original: Duan et al., 2021; Jiao et al., 2021a).
Revised: Duan et al., 2021; Jiao et al., 2021a).

Original: Although the proof of the approximation of the neural network to the Sobolev spaces is a standard approach, we still demonstrate the proof sketch here.
Revised: Although the proof of the approximation of the neural network to the Sobolev spaces is a standard approach, we still demonstrate the proof sketch here.

Original: Deﬁnition B.4.
Revised: Deﬁnition B.4.

Original: (Univariate and Multivariate B-splines)
Revised: (Univariate and Multivariate B-splines)

Original: Fix an arbitrary integer l ∈ Z + .
Revised: Fix an arbitrary integer l ∈ Z + .

Original: Consider a corresponding uniform partition π l of [0 , 1] :
Revised: Consider a corresponding uniform partition π l of [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: where t ( l ) i = il ( ∀ 0 ≤ i ≤ l ) .
Revised: where t ( l ) i = il ( ∀ 0 ≤ i ≤ l ) .

Original: Now for any k ∈ Z + , we can deﬁne an extended partition π l,k as:
Revised: Now for any k ∈ Z + , we can deﬁne an extended partition π l,k as:

Original: [Equation]
Revised: [Equation]

Original: Based on the extended partition π l,k , the univariate B-splines of order k with respect to partition π l are deﬁned by:
Revised: Based on the extended partition π l,k , the univariate B-splines of order k with respect to partition π l are deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: ) where I l,k = {− k + 1 , − k + 2 , · · · , l − 1 } and (cid:104) t ( l ) i , · · · , t ( l ) i + k ] denotes the divided difference operator.
Revised: ) where I l,k = {− k + 1 , − k + 2 , · · · , l − 1 } and (cid:104) t ( l ) i , · · · , t ( l ) i + k ] denotes the divided difference operator.

Original: Equivalently, for any x ∈ [0 , 1]
Revised: Equivalently, for any x ∈ [0 , 1]

Original: [Equation]
Revised: [Equation]

Original: we can rewrite the univariate B-splines N ( k ) l,i ( x ) in an explicit form:
Revised: we can rewrite the univariate B-splines N ( k ) l,i ( x ) in an explicit form:

Original: [Equation]
Revised: [Equation]

Original: where { a ij | − k + 1 ≤ i ≤ 0 , 0 ≤ j ≤ k − 1 } , { b in | − k + 1 ≤ i ≤ 0 , 1 ≤ n ≤ k − 2 } and { c ij | l − k + 1 ≤ i ≤ l − 1 , l − k + 1 ≤ j ≤ l − 1 } are some ﬁxed constants.
Revised: where { a ij | − k + 1 ≤ i ≤ 0 , 0 ≤ j ≤ k − 1 } , { b in | − k + 1 ≤ i ≤ 0 , 1 ≤ n ≤ k − 2 } and { c ij | l − k + 1 ≤ i ≤ l − 1 , l − k + 1 ≤ j ≤ l − 1 } are some ﬁxed constants.

Original: For any index vector i
Revised: For any index vector i

Original: =
Revised: =

Original: ( i 1 , i 2 , · · · , i d ) ∈ I dl,k , we can deﬁne a corresponding multivariate B-spline as a product of univariate B-splines:
Revised: ( i 1 , i 2 , · · · , i d ) ∈ I dl,k , we can deﬁne a corresponding multivariate B-spline as a product of univariate B-splines:

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition B.5.
Revised: Deﬁnition B.5.

Original: (Interpolation Operator(Schumaker, 2007))
Revised: (Interpolation Operator(Schumaker, 2007))

Original: Take some domain Ω ⊂ [0 , 1] d and two arbitrary integers k, l ∈ Z + .
Revised: Take some domain Ω ⊂ [0 , 1] d and two arbitrary integers k, l ∈ Z + .

Original: Consider the extended partition π l,k and the corresponding set of multivariate B-splines { N ( k ) l, i ( x ) } i ∈ I dl,k deﬁned in Deﬁnition B.4.
Revised: Consider the extended partition π l,k and the corresponding set of multivariate B-splines { N ( k ) l, i ( x ) } i ∈ I dl,k deﬁned in Deﬁnition B.4.

Original: For any i ∈ I dl,k , we deﬁne the domain
Revised: For any i ∈ I dl,k , we deﬁne the domain

Original: Ω i := { x ∈ Ω : x j ∈ [ t i j , t i j + k ] , 1 ≤ j ≤ d } .
Revised: Ω i := { x ∈ Ω : x j ∈ [ t i j , t i j + k ] , 1 ≤ j ≤ d } .

Original: There exists a set of linear functionals { λ i } i ∈
Revised: There exists a set of linear functionals { λ i } i ∈

Original: I dk,l , where λ i : L 1 (Ω) → R ( ∀ i ∈
Revised: I dk,l , where λ i : L 1 (Ω) → R ( ∀ i ∈

Original: I dk,l ) , such that for any i ∈ I dk,l and p ∈ [1 , ∞ ] , we have:
Revised: I dk,l ) , such that for any i ∈ I dk,l and p ∈ [1 , ∞ ] , we have:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The corresponding interpolation operator Q k,l is deﬁned as:
Revised: The corresponding interpolation operator Q k,l is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Theorem B.7.
Revised: Theorem B.7.

Original: [(Schumaker, 2007)]
Revised: [(Schumaker, 2007)]

Original: Fix f ∈ W s (Ω) with Ω ⊆ [0 , 1] d , s ∈ Z + and p ∈ [1 , ∞ ) .
Revised: Fix f ∈ W s (Ω) with Ω ⊆ [0 , 1] d , s ∈ Z + and p ∈ [1 , ∞ ) .

Original: Then for any k, l, r ∈ Z + with k ≥ s and 0 ≤ r ≤ s , we have that there exists some constant
Revised: Then for any k, l, r ∈ Z + with k ≥ s and 0 ≤ r ≤ s , we have that there exists some constant

Original: C = C ( k, s, r, p, d ) , such that:
Revised: C = C ( k, s, r, p, d ) , such that:

Original: [Equation]
Revised: [Equation]

Original: Theorem B.8.
Revised: Theorem B.8.

Original: (Approximation result of Deep Neural Network)
Revised: (Approximation result of Deep Neural Network)

Original: Fix some dimension d ∈ Z + , some domain Ω ⊆ [0 , 1] d .
Revised: Fix some dimension d ∈ Z + , some domain Ω ⊆ [0 , 1] d .

Original: We pick some l = N 1 d ≥ 2 , for any s, r ∈ Z + with 0 ≤ r ≤ s and any function u ∗ ∈ H s (Ω) , there exists some sparse Deep Neural Network u DNN ∈
Revised: We pick some l = N 1 d ≥ 2 , for any s, r ∈ Z + with 0 ≤ r ≤ s and any function u ∗ ∈ H s (Ω) , there exists some sparse Deep Neural Network u DNN ∈

Original: Φ( L, W, S, B ) with
Revised: Φ( L, W, S, B ) with

Original: L = O (1) , W = O ( N ) , S =
Revised: L = O (1) , W = O ( N ) , S =

Original: O ( N ) , B = O ( N ) , such that:
Revised: O ( N ) , B = O ( N ) , such that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrstly show that the given function u ∗ can be approximated well by some linear combination of multivariate splines, which is denoted by u sp .
Revised: We ﬁrstly show that the given function u ∗ can be approximated well by some linear combination of multivariate splines, which is denoted by u sp .

Original: Note that N is assumed to be sufﬁciently large.
Revised: Note that N is assumed to be sufﬁciently large.

Original: Hence, we may pick l = (cid:100) N 1 d (cid:101) =
Revised: Hence, we may pick l = (cid:100) N 1 d (cid:101) =

Original: Θ( N 1 d ) ∈
Revised: Θ( N 1 d ) ∈

Original: Z + to be the partition size of the B-splines.
Revised: Z + to be the partition size of the B-splines.

Original: Moreover, by picking k = 4 and p = 2 in Theorem B.7, we have that the linear combination u sp :=
Revised: Moreover, by picking k = 4 and p = 2 in Theorem B.7, we have that the linear combination u sp :=

Original: Q
Revised: Q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: We will then show that the linear combination u sp = (cid:80) i ∈ I d 4 ,l λ i ( f ) N (4) l, i can be implemented by some Deep Neural Network u DNN ∈
Revised: We will then show that the linear combination u sp = (cid:80) i ∈ I d 4 ,l λ i ( f ) N (4) l, i can be implemented by some Deep Neural Network u DNN ∈

Original: Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =
Revised: Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =

Original: O ( N ) and
Revised: O ( N ) and

Original: B =
Revised: B =

Original: O (log N ) .
Revised: O (log N ) .

Original: Firstly, note that for x ≥ 0 , both x and x 2 can be expressed in terms of the ReLU activation function η 3 with no error:
Revised: Firstly, note that for x ≥ 0 , both x and x 2 can be expressed in terms of the ReLU activation function η 3 with no error:

Original: [Equation]
Revised: [Equation]

Original: Applying the explicit formula listed in equation B.50 implies that for any − 3 ≤ i ≤ l − 1 , the univariate B-spline function N (4) l,i ( x ) ( x ∈ [0 , 1]) can be implemented by some ReLU3 Deep Neural
Revised: Applying the explicit formula listed in equation B.50 implies that for any − 3 ≤ i ≤ l − 1 , the univariate B-spline function N (4) l,i ( x ) ( x ∈ [0 , 1]) can be implemented by some ReLU3 Deep Neural

Original: Network v DNN with both scalar input and scalar output.
Revised: Network v DNN with both scalar input and scalar output.

Original: We have that for v DNN , the depth L v is 2 and the maximum width W v is upper bounded by 11 .
Revised: We have that for v DNN , the depth L v is 2 and the maximum width W v is upper bounded by 11 .

Original: Secondly, for any x, y ≥ 0 , we have that the product operation x · y can be expressed in terms of the
Revised: Secondly, for any x, y ≥ 0 , we have that the product operation x · y can be expressed in terms of the

Original: ReLU3 activation function η 3 with no error:
Revised: ReLU3 activation function η 3 with no error:

Original: [Equation]
Revised: [Equation]

Original: In (Schumaker, 2007), it has been proved that the B-splines are always non-negative, i.e N (4) l,i ( x ) ≥ 0 , ∀ x ∈ [0 , 1] .
Revised: In (Schumaker, 2007), it has been proved that the B-splines are always non-negative, i.e N (4) l,i ( x ) ≥ 0 , ∀ x ∈ [0 , 1] .

Original: Therefore, by multiplying the non-negative univariate B-splines, we can implement any multivariate B-spline N (4) l, i =
Revised: Therefore, by multiplying the non-negative univariate B-splines, we can implement any multivariate B-spline N (4) l, i =

Original: Π dj =1 N (4) l,i j ( x j ) with some ReLU3 Deep Neural Network p DNN .
Revised: Π dj =1 N (4) l,i j ( x j ) with some ReLU3 Deep Neural Network p DNN .

Original: We have that for p DNN , the depth L p = (cid:100) log 2 d (cid:101) + 2 and the maximum width W p = max { 11 d, 92 d } .
Revised: We have that for p DNN , the depth L p = (cid:100) log 2 d (cid:101) + 2 and the maximum width W p = max { 11 d, 92 d } .

Original: Hence, we can further claim that u ∗ =
Revised: Hence, we can further claim that u ∗ =

Original: (cid:80) i ∈ I d 4 ,
Revised: (cid:80) i ∈ I d 4 ,

Original: l λ i ( u ∗ ) N (4) l, i , which is a linear combination of the multivariate B-splines N (4) l, i , can be implemented by some ReLU3 Deep Neural Network u DNN .
Revised: l λ i ( u ∗ ) N (4) l, i , which is a linear combination of the multivariate B-splines N (4) l, i , can be implemented by some ReLU3 Deep Neural Network u DNN .

Original: It remains to check that u
Revised: It remains to check that u

Original: DNN ∈
Revised: DNN ∈

Original: Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =
Revised: Φ( L, W, S, B ) with L = O (1) , W = O ( N ) , S =

Original: O ( N ) and
Revised: O ( N ) and

Original: B = O ( N ) .
Revised: B = O ( N ) .

Original: Note that we can ensure that the hidden layers of u DNN are of the same dimension W by adding inactive neurons.
Revised: Note that we can ensure that the hidden layers of u DNN are of the same dimension W by adding inactive neurons.

Original: For the depth L of u DNN , we have that L is equal to L p + 1 , where L p denotes the depth of the
Revised: For the depth L of u DNN , we have that L is equal to L p + 1 , where L p denotes the depth of the

Original: ReLU3 Deep Neural Network p DNN .
Revised: ReLU3 Deep Neural Network p DNN .

Original: Thus, we have L =
Revised: Thus, we have L =

Original: L p + 1 = (cid:100) log 2 d (cid:101) + 3 , which implies that
Revised: L p + 1 = (cid:100) log 2 d (cid:101) + 3 , which implies that

Original: L = O (1) .
Revised: L = O (1) .

Original: For the width W of u DNN , we have that W ≤ | I dk,l | W p , where W p denotes the width of the ReLU
Revised: For the width W of u DNN , we have that W ≤ | I dk,l | W p , where W p denotes the width of the ReLU

Original: Deep Neural Network p DNN .
Revised: Deep Neural Network p DNN .

Original: This implies:
Revised: This implies:

Original: [Equation]
Revised: [Equation]

Original: For the sparsity constraint S of u DNN , starting from the third layer, the number of activated neurons is half of the number of activated neurons at previous layer.
Revised: For the sparsity constraint S of u DNN , starting from the third layer, the number of activated neurons is half of the number of activated neurons at previous layer.

Original: This yields the following upper bound on
Revised: This yields the following upper bound on

Original: S :
Revised: S :

Original: [Equation]
Revised: [Equation]

Original: For the norm constraint B of u DNN , we have the following upper bound on B from equation B. and equation B.52:
Revised: For the norm constraint B of u DNN , we have the following upper bound on B from equation B. and equation B.52:

Original: [Equation]
Revised: [Equation]

Original: Now we have shown that parameters L, W, S, B of the Deep Neural Network u DNN are of the desired magnitude, which completes our proof.
Revised: Now we have shown that parameters L, W, S, B of the Deep Neural Network u DNN are of the desired magnitude, which completes our proof.

Original: B. F INAL U PPER B OUND
Revised: B. F INAL U PPER B OUND

Original: In this subsection, we provide the proof of upper bounds for PINN and DRM.
Revised: In this subsection, we provide the proof of upper bounds for PINN and DRM.

Original: For both estimator, we ﬁrst provide a meta-theorem to illustrate the approximation and generalization decomposition with a O (1 /n ) fast rate generalization bound(Bartlett et al., 2005; Xu, 2020).
Revised: For both estimator, we ﬁrst provide a meta-theorem to illustrate the approximation and generalization decomposition with a O (1 /n ) fast rate generalization bound(Bartlett et al., 2005; Xu, 2020).

Original: Then we use truncated
Revised: Then we use truncated

Original: Fourier basis estimator and neural network estimator as example to obtain the ﬁnal rate.
Revised: Fourier basis estimator and neural network estimator as example to obtain the ﬁnal rate.

Original: B.4. D EEP R ITZ M ETHODS
Revised: B.4. D EEP R ITZ M ETHODS

Original: Theorem B.9 (Meta-theorem for Upper Bounds of Deep Ritz Methods) .
Revised: Theorem B.9 (Meta-theorem for Upper Bounds of Deep Ritz Methods) .

Original: Let u ∗ ∈ H s (Ω) denote the true solution to the PDE model with Dirichlet boundary condition:
Revised: Let u ∗ ∈ H s (Ω) denote the true solution to the PDE model with Dirichlet boundary condition:

Original: [Equation]
Revised: [Equation]

Original: where f ∈ L 2 (Ω) and V ∈ L ∞ (Ω) with 0 < V min ≤ V ( x ) ≤ V max > 0 .
Revised: where f ∈ L 2 (Ω) and V ∈ L ∞ (Ω) with 0 < V min ≤ V ( x ) ≤ V max > 0 .

Original: In Theorem B.1, it has been proved that u ∗ can be obtained by minimizing the loss E ( u ) :
Revised: In Theorem B.1, it has been proved that u ∗ can be obtained by minimizing the loss E ( u ) :

Original: [Equation]
Revised: [Equation]

Original: For a ﬁxed function space F (Ω) , consider the empirical loss induced by the Deep Ritz Method:
Revised: For a ﬁxed function space F (Ω) , consider the empirical loss induced by the Deep Ritz Method:

Original: [Equation]
Revised: [Equation]

Original: where { X j } nj =1 are datapoints uniformly sampled from the domain Ω .
Revised: where { X j } nj =1 are datapoints uniformly sampled from the domain Ω .

Original: Then the Deep Ritz estimator associated with function space F (Ω) is deﬁned as the minimizer of E n ( u ) over the function space
Revised: Then the Deep Ritz estimator associated with function space F (Ω) is deﬁned as the minimizer of E n ( u ) over the function space

Original: F (Ω) :
Revised: F (Ω) :

Original: [Equation]
Revised: [Equation]

Original: Moreover, we assume that there exists some constant C > 0 such that all function u in the function space F (Ω) , the real solution u ∗ and f, V satisfy the following two conditions.
Revised: Moreover, we assume that there exists some constant C > 0 such that all function u in the function space F (Ω) , the real solution u ∗ and f, V satisfy the following two conditions.

Original: • The gradients and function value are uniformly bounded
Revised: • The gradients and function value are uniformly bounded

Original: [Equation]
Revised: [Equation]

Original: • All the functions in the function space F (Ω) satisﬁes the boundary condition u
Revised: • All the functions in the function space F (Ω) satisﬁes the boundary condition u

Original: [Equation]
Revised: [Equation]

Original: 0 on ∂ Ω
Revised: 0 on ∂ Ω

Original: [Equation]
Revised: [Equation]

Original: At the the same time, for any ρ >
Revised: At the the same time, for any ρ >

Original: [Equation]
Revised: [Equation]

Original: we assume the Rademacher complexity of a localized function space S ρ
Revised: we assume the Rademacher complexity of a localized function space S ρ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: u ∗ (cid:107) 2 H 1 ≤ ρ (cid:111) can be upper bounded by a sub-root function φ = φ ( ρ ) :
Revised: u ∗ (cid:107) 2 H 1 ≤ ρ (cid:111) can be upper bounded by a sub-root function φ = φ ( ρ ) :

Original: [0 , ∞ )
Revised: [0 , ∞ )

Original: → [0 , ∞ ) , i.e. φ (4 ρ ) ≤ 2 φ ( ρ ) and R n ( S ρ (Ω)) ≤ φ ( ρ ) ( ∀ ρ > 0) .
Revised: → [0 , ∞ ) , i.e. φ (4 ρ ) ≤ 2 φ ( ρ ) and R n ( S ρ (Ω)) ≤ φ ( ρ ) ( ∀ ρ > 0) .

Original: [Equation]
Revised: [Equation]

Original: For all constant t > 0 .
Revised: For all constant t > 0 .

Original: We denote r ∗ to be the solution of the ﬁx point equation of local Rademacher complexity r = φ ( r ) .
Revised: We denote r ∗ to be the solution of the ﬁx point equation of local Rademacher complexity r = φ ( r ) .

Original: There exist two constants C p , C q such that with probability 1 − C p exp( − C q t ) , we have the following upper bound for the Deep Ritz Estimator
Revised: There exist two constants C p , C q such that with probability 1 − C p exp( − C q t ) , we have the following upper bound for the Deep Ritz Estimator

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To upper bound the excess risk ∆ E ( n ) :
Revised: To upper bound the excess risk ∆ E ( n ) :

Original: =
Revised: =

Original: E (ˆ u DRM ) − E ( u ∗ ) , following(Xu, 2020; Lu et al., 2021b; Duan et al., 2021), we decompose the excess risk into approximation error and generalization error with probability 1 − e − C q t , where C q > 0 is some constant:
Revised: E (ˆ u DRM ) − E ( u ∗ ) , following(Xu, 2020; Lu et al., 2021b; Duan et al., 2021), we decompose the excess risk into approximation error and generalization error with probability 1 − e − C q t , where C q > 0 is some constant:

Original: [Equation]
Revised: [Equation]

Original: ) where the expectation is on all sampled data.
Revised: ) where the expectation is on all sampled data.

Original: The inequality of the third line is because ˆ u DRM is the minimizer of the empirical loss E n in the solution
Revised: The inequality of the third line is because ˆ u DRM is the minimizer of the empirical loss E n in the solution

Original: set F (Ω) , so we have E n (ˆ u DRM ) ≤
Revised: set F (Ω) , so we have E n (ˆ u DRM ) ≤

Original: E n ( u F ) .
Revised: E n ( u F ) .

Original: The last inequality is based on the Bernstein inequality.
Revised: The last inequality is based on the Bernstein inequality.

Original: The variance of h = | Ω | · (cid:104) 

Revised: The variance of h = | Ω | · (cid:104)

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: u ∗ (cid:107) 2 (cid:17) + 1 2 V ( | u | 2 − | u ∗ | 2 ) − f ( u − u ∗ ) (cid:105) can be upper bounded by α (cid:48)
Revised: u ∗ (cid:107) 2 (cid:17) + 1 2 V ( | u | 2 − | u ∗ | 2 ) − f ( u − u ∗ ) (cid:105) can be upper bounded by α (cid:48)

Original: α (cid:2) E ( u F ) −
Revised: α (cid:2) E ( u F ) −

Original: E ( u (cid:63) ) (cid:3) due to the strong convexity of the variation objective (B.60).
Revised: E ( u (cid:63) ) (cid:3) due to the strong convexity of the variation objective (B.60).

Original: According to the Bernstein inequality, there exists some constant C q > 0 , such that with probability 1 − e − C q t we have:
Revised: According to the Bernstein inequality, there exists some constant C q > 0 , such that with probability 1 − e − C q t we have:

Original: [Equation]
Revised: [Equation]

Original: Note that B.58 holds for all function lies in the function space F .
Revised: Note that B.58 holds for all function lies in the function space F .

Original: Thus, we can take u F := arg min u 0 ∈ F (Ω) (cid:16) E ( u 0 ) − E ( u (cid:63) ) (cid:17) and ﬁnally get ∆ E ( n ) ≤ E (ˆ u DRM
Revised: Thus, we can take u F := arg min u 0 ∈ F (Ω) (cid:16) E ( u 0 ) − E ( u (cid:63) ) (cid:17) and ﬁnally get ∆ E ( n ) ≤ E (ˆ u DRM

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: This inequality decompose the excess risk to the generalization error ∆ E gen := E (ˆ u DRM ) − E ( u ∗ )
Revised: This inequality decompose the excess risk to the generalization error ∆ E gen := E (ˆ u DRM ) − E ( u ∗ )

Original: +
Revised: +

Original: E n ( u ∗ ) − E n (ˆ u DRM ) and the approximation error ∆ E app = inf u F ∈ F (Ω) (cid:16) E ( u F ) − E ( u (cid:63) ) (cid:17) .
Revised: E n ( u ∗ ) − E n (ˆ u DRM ) and the approximation error ∆ E app = inf u F ∈ F (Ω) (cid:16) E ( u F ) − E ( u (cid:63) ) (cid:17) .

Original: From the lemmata proved in Section B.3, we already have an estimation of the approximation error’s convergence rate.
Revised: From the lemmata proved in Section B.3, we already have an estimation of the approximation error’s convergence rate.

Original: So now we’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization technique(Bartlett et al., 2005; Xu, 2020).
Revised: So now we’ll focus on providing fast rate upper bounds of the generalization error for the two estimators using the localization technique(Bartlett et al., 2005; Xu, 2020).

Original: To achieve the fast generalization bound, we focus on the following normalized empirical process
Revised: To achieve the fast generalization bound, we focus on the following normalized empirical process

Original: [Equation]
Revised: [Equation]

Original: First, we try to bound the expectation of the normalized empirical process.
Revised: First, we try to bound the expectation of the normalized empirical process.