Original: L EARNING R E LU NETWORKS TO HIGH UNIFORM ACCURACY IS INTRACTABLE
Revised: L EARNING R E LU NETWORKS TO HIGH UNIFORM ACCURACY IS INTRACTABLE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Statistical learning theory provides bounds on the necessary number of training samples needed to reach a prescribed accuracy in a learning problem formulated over a given target class.
Revised: Statistical learning theory provides bounds on the necessary number of training samples needed to reach a prescribed accuracy in a learning problem formulated over a given target class.

Original: This accuracy is typically measured in terms of a generalization error, that is, an expected value of a given loss function.
Revised: This accuracy is typically measured in terms of a generalization error, that is, an expected value of a given loss function.

Original: However, for several applications — for example in a security-critical context or for problems in the computational sciences — accuracy in this sense is not sufﬁcient.
Revised: However, for several applications — for example in a security-critical context or for problems in the computational sciences — accuracy in this sense is not sufﬁcient.

Original: In such cases, one would like to have guarantees for high accuracy on every input value, that is, with respect to the uniform norm.
Revised: In such cases, one would like to have guarantees for high accuracy on every input value, that is, with respect to the uniform norm.

Original: In this paper we precisely quantify the number of training samples needed for any conceivable training algorithm to guarantee a given uniform accuracy on any learning problem formulated over target classes containing (or consisting of) ReLU neural networks of a prescribed architecture.
Revised: In this paper we precisely quantify the number of training samples needed for any conceivable training algorithm to guarantee a given uniform accuracy on any learning problem formulated over target classes containing (or consisting of) ReLU neural networks of a prescribed architecture.

Original: We prove that, under very general assumptions, the minimal number of training samples for this task scales exponentially both in the depth and the input dimension of the network architecture.
Revised: We prove that, under very general assumptions, the minimal number of training samples for this task scales exponentially both in the depth and the input dimension of the network architecture.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The basic goal of supervised learning is to determine a function 1 u : r 0 , 1 s d Ñ R from (possibly noisy) samples p u p x 1 q , . .
Revised: The basic goal of supervised learning is to determine a function 1 u : r 0 , 1 s d Ñ R from (possibly noisy) samples p u p x 1 q , . .

Original: , u p x m qq .
Revised: , u p x m qq .

Original: As the function u can take arbitrary values between these samples, this problem is, of course, not solvable without any further information on u .
Revised: As the function u can take arbitrary values between these samples, this problem is, of course, not solvable without any further information on u .

Original: In practice, one typically leverages domain knowledge toa priori estimate the structure and regularity of u , for instance in terms of symmetries, its smoothness or compositionality.
Revised: In practice, one typically leverages domain knowledge to estimate the structure and regularity of u a priori, for instance, in terms of symmetries, smoothness, or compositionality.

Original: Such additional information can be encoded via a suitable target class U Ă C pr 0 , 1 s d q that u is known to be a member of.
Revised: Such additional information can be encoded via a suitable target class U Ă C pr 0 , 1 s d q that u is known to be a member of.

Original: We are interested in the optimal accuracy for reconstructing u that can be achieved by any algorithm which utilizes m point samples.
Revised: We are interested in the optimal accuracy for reconstructing u that can be achieved by any algorithm which utilizes m point samples.

Original: To make this mathematically precise, we assume that this accuracy is measured by a norm } ¨ } Y of a suitable Banach space Y Ą U .
Revised: To make this mathematically precise, we assume that this accuracy is measured by a norm } ¨ } Y of a suitable Banach space Y Ą U .

Original: Formally, an algorithm can thus be described by a map
Revised: Formally, an algorithm can thus be described by a map

Original: A : U Ñ Y that can query the function u at m points x i and that outputs a function A p u q with A p u q « u (see Section 2.1 for a precise deﬁnition that incorporates adaptivity and stochasticity).
Revised: A : U Ñ Y that can query the function u at m points x i and that outputs a function A p u q with A p u q « u (see Section 2.1 for a precise deﬁnition that incorporates adaptivity and stochasticity).

Original: We will be interested in upper and lower bounds on the accuracy that can be reached by any such algorithm — equivalently, we are interested in the minimal number m of point samples needed for any algorithm to achieve a given accuracy ε for every u P U .
Revised: We will be interested in upper and lower bounds on the accuracy that can be reached by any such algorithm — equivalently, we are interested in the minimal number m of point samples needed for any algorithm to achieve a given accuracy ε for every u P U .

Original: This m would then establish a fundamental benchmark on the sample complexity (and the algorithmic complexity) of learning functions in U to a given accuracy.
Revised: This m would then establish a fundamental benchmark on the sample complexity (and the algorithmic complexity) of learning functions in U to a given accuracy.

Original: The choice of the Banach space Y — in other words how we measure accuracy — is very crucial here.
Revised: The choice of the Banach space Y — in other words how we measure accuracy — is very crucial here.

Original: For example, statistical learning theory provides upper bounds on the optimal accuracy in terms of an expected loss, or with respect to Y “ L 2 pr 0 , 1 s d , d P q , where P is a (generally unknown)  data generating distribution (Devroye et al., 2013; Shalev-Shwartz & Ben-David,2014; Mohri et al., 2018;
Revised: For example, statistical learning theory provides upper bounds on the optimal accuracy in terms of an expected loss, i.e., with respect to Y “ L 2 pr 0 , 1 s d , d P q , where P is a (generally unknown) [Figure] data generating distribution (Devroye et al., 2013; Shalev-Shwartz & Ben-David, et al

Original:  Kim et al., 2021).
Revised: , 2018; Kim et al., 2021).

Revised: This offers a powerful approach to ensure a small average reconstruction error.

Original: This offers a powerful approach for achieving a small reconstruction error in an averaged sense, but there are many important scenarios where such bounds on the accuracy are not sufﬁcient: For several applications in the sciences, for example in the context of the numerical solution of partial differential equations (Raissi et al., 2019; Han et al., 2018), any security-critical application (for example, facial ID authentication schemes (Guo & Zhang, 2019)), as well as any application with[Figure] a distribution-shift (i.e., the data generating distribution is different from the distribution in which the accuracy is measured (Quiñonero-Candela et al., 2008)) one would like to obtain an approximation of u that is close to u not only on average, but that can be guaranteed to be close for every x P r 0 , 1 s d .
Revised: However, there are many important scenarios where such bounds on the accuracy are not sufﬁcient and one would like to obtain an approximation of u that is close to u not only on average, but that can be guaranteed to be close for every x P r 0 , 1 s d . This includes several applications in the sciences, for example in the context of the numerical solution of partial differential equations (Raissi et al., 2019; Han et al., 2018; Richter & Berner, 2022), any security-critical application, for example, facial ID authentication schemes (Guo & Zhang, 2019), as well as any application with a distribution-shift , i.e., where the data generating distribution is different from the distribution in which the accuracy is measured (Quiñonero-Candela et al., 2008).

Original: Such applications can only be efﬁciently solved if there exists an efﬁcient algorithm A that achieves uniform accuracy , i.e., a small error sup u P U } u ´ A p u q} L 8 pr 0 , 1 s d q with respect to the uniform norm given by Y “ L 8 pr 0 , 1 s d q , i.e., } f } L 8 pr 0 , 1 s d q  “ esssup x Pr 0 , 1 s d | f p x q| .
Revised: Such applications can only be efﬁciently solved if there exists an efﬁcient algorithm A that achieves uniform accuracy , i.e., a small error sup u P U } u ´ A p u q} L 8 pr 0 , 1 s d q with respect to the uniform norm given by Y “ L 8 pr 0 , 1 s d q , i.e., } f } L 8 pr 0 , 1 s d q : “ esssup x Pr 0 , 1 s d | f p x q| .

Original: Inspired by recent successes of deep learning across a plethora of tasks in machine learning (LeCun et al., 2015) and also increasingly the sciences (Jumper et al., 2021; Pfau et al., 2020), we will be particularly interested in the case where the target class U consists of — or contains — realizations of (feed-forward) neural networks of a speciﬁc architecture 2 .
Revised: Inspired by recent successes of deep learning across a plethora of tasks in machine learning (LeCun et al., 2015) and also increasingly the sciences (Jumper et al., 2021; Pfau et al., 2020), we will be particularly interested in the case where the target class U consists of — or contains — realizations of (feed-forward) neural networks of a speciﬁc architecture 2 .

Original: Neural networks have been proven and observed to be extremely powerful in terms of their expressivity, that is, their ability to accurately approximate large classes of complicated functions with only relatively few parameters (Elbrächter et al., 2021).
Revised: Neural networks have been proven and observed to be extremely powerful in terms of their expressivity, that is, their ability to accurately approximate large classes of complicated functions with only relatively few parameters (Elbrächter et al., 2021; Berner et al., 2022).

Original: However, it has also been repeatedly observed that the training of neural networks (e.g., ﬁtting a neural network to data samples) to high uniform accuracy presents a big challenge: conventional training algorithms (such as SGD and its variants) often ﬁnd neural networks that perform well on average (meaning that they achieve a small generalization error), but there are typically some regions in the input space where the error is large (Fiedler et al., 2023); see Figurefor an illustrative example.
Revised: However, it has also been repeatedly observed that the training of neural networks (e.g., ﬁtting a neural network to data samples) to high uniform accuracy presents a big challenge: conventional training algorithms (such as SGD and its variants) often ﬁnd neural networks that perform well on average (meaning that they achieve a small generalization error), but there are typically some regions in the input space where the error is large (Fiedler et al., 2023); see Figurefor an illustrative example.

Original: This phenomenon has been systematically studied on an empirical level by Adcock & Dexter (2021).
Revised: This phenomenon has been systematically studied on an empirical level by Adcock & Dexter (2021).

Original: It is also at the heart of several observed instabilities in the training of deep neural networks, including adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015) or so-called hallucinations emerging in generative modeling, e.g., tomographic reconstructions (Bhadra et al., 2021) or machine translation (Müller et al., 2020).
Revised: It is also at the heart of several observed instabilities in the training of deep neural networks, including adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015) or so-called hallucinations emerging in generative modeling, e.g., tomographic reconstructions (Bhadra et al., 2021) or machine translation (Müller et al., 2020).

Original: Note that additional knowledge on the target functions could potentially help circumvent these issues, see Remark 1.2.
Revised: Note that additional knowledge on the target functions could potentially help circumvent these issues, see Remark 1.3.

Original: However, for many applications, it is not possible to precisely describe the regularity of the target functions.
Revised: However, for many applications, it is not possible to precisely describe the regularity of the target functions.

Original: We thus analyze the case where no additional information is given besides the fact that one aims to recover a (unknown) neural network of a speciﬁed architecture and regularization from given samples – i.e., we assume that U contains a class of neural networks of a given architecture, subject to various regularization methods.
Revised: We thus analyze the case where no additional information is given besides the fact that one aims to recover a (unknown) neural network of a speciﬁed architecture and regularization from given samples – i.e., we assume that U contains a class of neural networks of a given architecture, subject to various regularization methods.

Original: This is satisﬁed in several applications of interest, e.g., model extraction attacks (Tramèr et al., 2016;
Revised: This is satisﬁed in several applications of interest, e.g., model extraction attacks (Tramèr et al., 2016;

Original: He et al., 2022) and teacher-student settings (Mirzadeh et al., 2020; Xie et al., 2020).
Revised: He et al., 2022) and teacher-student settings (Mirzadeh et al., 2020; Xie et al., 2020).

Original: It is also in line with standard settings in the statistical query literature , in neural network identiﬁcation , and in statistical learning theory, see Section 1.1.
Revised: It is also in line with standard settings in the statistical query literature , in neural network identiﬁcation , and in statistical learning theory (Anthony & Bartlett, 1999; Mohri et al., 2018), see Section 1.1.

Original: For such settings we can rigorously show that learning a class of neural networks is prone to instabilities.
Revised: For such settings we can rigorously show that learning a class of neural networks is prone to instabilities.

Original: Speciﬁcally, any conceivable learning algorithm (in particular, any version of SGD), which recovers the neural network to high uniform accuracy, needs intractably many samples.
Revised: Speciﬁcally, any conceivable learning algorithm (in particular, any version of SGD), which recovers the neural network to high uniform accuracy, needs intractably many samples.

Original: Theorem 1.1.
Revised: Theorem 1.1.

Original: Suppose that U contains all neural networks with d -dimensional input, ReLU activation function, L layers of width up to 3 d , and coefﬁcients bounded by c in the (cid:96) q -norm.
Revised: Suppose that U contains all neural networks with d -dimensional input, ReLU activation function, L layers of width up to 3 d , and coefﬁcients bounded by c in the (cid:96) q norm.

Original: Assume that there exists an algorithm that reconstructs all functions in U to uniform(i.e., L 8 pr 0 , 1 s d q ) accuracy ε from m point samples.
Revised: Assume that there exists an algorithm that reconstructs all functions in U to uniform accuracy ε from m point samples.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where Ω “ [Equation] Theorem 1.1 is a special case of Theorem 2.2 (covering Y “ L p pr 0 , 1 s d q for all p P r 1 , 8s , as well asmore general network architectures) which will be stated and proven in Section 2.3.
Revised: Theorem 1.1 is a special case of Theorem 2.2 (covering Y “ L p pr 0 , 1 s d q for all p P r 1 , 8s , as well as network architectures with arbitrary width) which will be stated and proven in Section 2.3.

Original: To give a concrete example, we consider the problem of learning neural networks with ReLU activation function, L layers of width at most 3 d , and coefﬁcients bounded by c to uniform accuracy ε “ 1 { 1024 .
Revised: To give a concrete example, we consider the problem of learning neural networks with ReLU activation function, L layers of width at most 3 d , and coefﬁcients bounded by c to uniform accuracy ε “ 1 { 1024 .

Original: According to our results we would need at least
Revised: According to our results we would need at least

Original: [Equation]
Revised: [Equation]

Original: many samples — the sample complexity thus depends exponentially on the input dimension d , the network width, and the network depth , becoming intractable even for moderate values of d, c, L (for d “ 15 , c “ 2 , and L “ 7 , the sample size m would already have to exceed the estimated number of atoms in our universe).
Revised: many samples — the sample complexity thus depends exponentially on the input dimension d , the network width, and the network depth , becoming intractable even for moderate values of d, c, L (for d “ 15 , c “ 2 , and L “ 7 , the sample size m would already have to exceed the estimated number of atoms in our universe).

Original: If, on the other hand, reconstruction only with respect to the L 2 -norm were required, standard results in statistical learning theory (see, for example, Berner et al., 2020) show that m only needs to depend polynomially on d .
Revised: If, on the other hand, reconstruction only with respect to the L 2 norm were required, standard results in statistical learning theory (see, for example, Berner et al., 2020) show that m only needs to depend polynomially on d .

Original: We conclude that uniform reconstruction is vastly harder than reconstruction with respect to the L 2 -norm and, in particular, intractable.
Revised: We conclude that uniform reconstruction is vastly harder than reconstruction with respect to the L 2 norm and, in particular, intractable.

Original: Our results are further corroborated by numerical experiments presented in Section 3 below.
Revised: Our results are further corroborated by numerical experiments presented in Section 3 below.

Original: Remark 1.2.
Revised: Remark 1.2.

Original: Our results do not rule out the possibility that there exist training algorithms for neural networks that achieve high accuracy on some restricted class of target functions, if the knowledge about the target class can be incorporated into the algorithm design.

Original: For example, if it were known that the target function can be efﬁciently approximated by polynomials one could ﬁrst compute an approximating polynomial (using polynomial regression which is tractable) and then represent the approximating polynomial by a neural network.

Original: The resulting numerical problem would however be very different from the way deep learning is used in practice, since most neural network coefﬁcients (namely those corresponding to the approximating polynomial) would be ﬁxed a priori.

Original: Our results apply to the situation where such additional information on the target class U is not available and no problem speciﬁc knowledge is incorporated into the algorithm design besides the network architecture and regularization procedure.

Original: Remark 1.3.

Original: For other target classes U , uniform reconstruction is tractable (i.e., the number of required samples for recovery does not massively exceed the number of parameters deﬁning the class).
Revised: For other target classes U , uniform reconstruction is tractable (i.e., the number of required samples for recovery does not massively exceed the number of parameters deﬁning the class).

Original: A simple example are univariate polynomials of degree less than m which can be exactly determined from m samples.
Revised: A simple example are univariate polynomials of degree less than m which can be exactly determined from m samples.

Original: One can show similar results for sparse multivariate polynomials using techniques from the ﬁeld of compressed sensing (Rauhut, 2007).
Revised: One can show similar results for sparse multivariate polynomials using techniques from the ﬁeld of compressed sensing (Rauhut, 2007).

Original: Further, one can show that approximation rates in suitable reproducing kernel Hilbert spaces with bounded kernel can be realized using point samples with respect to the uniform norm (Pozharska & Ullrich, 2022).
Revised: Further, one can show that approximation rates in suitable reproducing kernel Hilbert spaces with bounded kernel can be realized using point samples with respect to the uniform norm (Pozharska & Ullrich, 2022).

Original: Our results uncover an opposing behavior of neural network classes: There exist functions that can be arbitrarily well approximated (in fact, exactly represented) by small neural networks, but these representations cannot be inferred from samples.
Revised: Our results uncover an opposing behavior of neural network classes: There exist functions that can be arbitrarily well approximated (in fact, exactly represented) by small neural networks, but these representations cannot be inferred from samples.

Original: Our results are thus highly speciﬁc to classes of neural networks.
Revised: Our results are thus highly speciﬁc to classes of neural networks.

Revised: Remark 1.3.

Revised: Our results do not rule out the possibility that there exist training algorithms for neural networks that achieve high accuracy on some restricted class of target functions, if the knowledge about the target class can be incorporated into the algorithm design.

Revised: For example, if it were known that the target function can be efﬁciently approximated by polynomials one could ﬁrst compute an approximating polynomial (using polynomial regression which is tractable) and then represent the approximating polynomial by a neural network.

Revised: The resulting numerical problem would however be very different from the way deep learning is used in practice, since most neural network coefﬁcients (namely those corresponding to the approximating polynomial) would be ﬁxed a priori.

Revised: Our results apply to the situation where such additional information on the target class U is not available and no problem speciﬁc knowledge is incorporated into the algorithm design besides the network architecture and regularization procedure.

Original: We also complement the lower bounds of Theorem 1.1 with corresponding upper bounds.
Revised: We also complement the lower bounds of Theorem 1.1 with corresponding upper bounds.

Original: Theorem 1.4.
Revised: Theorem 1.4.

Original: Suppose that U consists of all neural networks with d -dimensional input, ReLU activation function, L layers of width at most p N 1 , . . , N L ´ 1 , 1 q , and coefﬁcients bounded by c in the (cid:96) q -norm.
Revised: Suppose that U consists of all neural networks with d -dimensional input, ReLU activation function, L layers of width at most B , and coefﬁcients bounded by c in the (cid:96) q norm.

Original: Then, there exists an algorithm that reconstructs all functions in U to uniform (i.e.,
Revised: Then, there exists an algorithm that reconstructs all functions in U to uniform accuracy ε from m point samples with m ď C d [Equation] d , where C : “ # ? d [Equation]

Original: L 8 pr 0 , 1 s d q ) accuracy ε from m point samples with
Revised: c [Equation] if q ďd [Equation]

Revised: c

Revised: [Equation]

Revised: q

Revised: [Equation]

Revised: if

Revised: [Equation]

Revised: ě

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.4 follows from Theorem 2.4 that will be stated in Section 2.4.
Revised: Theorem 1.4 follows from Theorem 2.4 that will be stated in Section 2.4.

Original: We refer to Remark B.4 for a discussion of the gap between the upper and lower bounds.
Revised: We refer to Remark B.4 for a discussion of the gap between the upper and lower bounds.

Original: Remark 1.5.
Revised: Remark 1.5.

Original: Our setting allows for an algorithm to choose the sample points p x 1 , . . .
Revised: Our setting allows for an algorithm to choose the sample points p x 1 , . . .

Original: , x m q in an adaptive way for each u P U ; see Section 2.1 for a precise deﬁnition of the class of adaptive (possibly randomized) algorithms.
Revised: , x m q in an adaptive way for each u P U ; see Section 2.1 for a precise deﬁnition of the class of adaptive (possibly randomized) algorithms.

Original: This implies that even a very clever sampling strategy (as would be employed in active learning) cannot break the bounds established in this paper.
Revised: This implies that even a very clever sampling strategy (as would be employed in active learning) cannot break the bounds established in this paper.

Original: Remark 1.6.
Revised: Remark 1.6.

Original: Our results also shed light on the impact of different regularization methods.
Revised: Our results also shed light on the impact of different regularization methods.

Original: While picking a stronger regularizer (e.g., a small value of q ) yields quantitative improvements (in the sense of a smaller Ω ),for small ε the sampling size  required for approximation in L 8 always scales exponentially in the input dimension.
Revised: While picking a stronger regularizer (e.g., a small value of q ) yields quantitative improvements (in the sense of a smaller Ω ), the sample size m required for approximation in L 8 can still increase exponentially with the input dimension d .

Original: However, this scaling is only visible for very small ε .
Revised: However, this scaling is only visible for very small ε .

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Several other works have established “hardness” results for neural network training.
Revised: Several other works have established “hardness” results for neural network training.

Original: For example, the foundational works by Blum & Rivest (1992); Vu (1998) show that for certainspeciﬁc architectures the training process can be NP-complete.
Revised: For example, the seminal works by Blum & Rivest (1992); Vu (1998) show that for certain architectures the training process can be NP -complete.

Original: By contrast, our results do not directly consider algorithm runtime at all; our results are stronger in the sense of showing that even if it were possible to efﬁciently learn a neural network from samples, the necessary number of data points would be too large to be tractable.
Revised: By contrast, our results do not directly consider algorithm runtime at all; our results are stronger in the sense of showing that even if it were possible to efﬁciently learn a neural network from samples, the necessary number of data points would be too large to be tractable.

Original: We also want to mention a series of hardness results in the setting of statistical query  algorithms, see, for instance, Chen et al.
Revised: We also want to mention a series of hardness results in the setting of statistical query (SQ) algorithms, see, e.g., Chen et al.

Original: ; Diakonikolas et al.
Revised: ; Diakonikolas et al.

Original: ; Goel et al.
Revised: ; Goel et al.

Original: (2020b); Reyzin (2020); Song et al.
Revised: (2020b); Reyzin (2020); Song et al.

Original: For instance, Chen et al.
Revised: For instance, Chen et al.

Original: (2022) shows that any SQ algorithm able to learn ReLU networks with two hidden layers and width poly p
Revised: (2022) shows that any SQ algorithm capable of learning ReLU networks with two hidden layers and width poly p

Original: d q up to L 2 error 1 { poly p d q must use a number of samples that scales superpolynomially in d , or must use SQ queries withquery tolerance smaller than the reciprocal of any polynomial in d .
Revised: d q up to L 2 error 1 { poly p d q must use a number of samples that scales superpolynomially in d , or must use SQ queries with tolerance smaller than the reciprocal of any polynomial in d .

Original: As this example illustrates, these hardness results apply to so-called SQ algorithms,meaning that the learner has access to an oracle that produces approximations (potentially corrupted by adversarial noise ) of certain expectations E r h p X, u p X qqs , where u is the unknown function to be learned, X is a random variable representing the data, and h is a function chosen by the learner (potentially subject to some restrictions, e.g. Lipschitz continuity).
Revised: In such SQ algorithms, the learner has access to an oracle that produces approximations (potentially corrupted by adversarial noise ) of certain expectations E r h p X, u p X qqs , where u is the unknown function to be learned, X is a random variable representing the data, and h is a function chosen by the learner (potentially subject to some restrictions, e.g. Lipschitz continuity).

Original: This possibility of the oracle to inject adversarial  noise into the learning procedure—which does not entirely reﬂect the usual mathematical formulation of learning problems—is crucial for several of these results.
Revised: The possibility of the oracle to inject adversarial (instead of just stochastic) noise into the learning procedure — which does not entirely reﬂect the typical mathematical formulation of learning problems — is crucial for several of these results.

Original: We also mention that due to this possibility of adversarial(not stochastic) noise, not every gradient-based optimization method (for instance, SGD) is strictly speaking an SQ algorithm; see also the works by Goel et al.
Revised: We also mention that due to this possibility of adversarial noise, not every gradient-based optimization method (for instance, SGD) is strictly speaking an SQ algorithm; see also the works by Goel et al.

Original: (2020a, Page 3) and Abbe et al.
Revised: (2020a, Page 3) and Abbe et al.

Original: for a more detailed discussion.
Revised: for a more detailed discussion.

Original: There also exist hardness results for learning algorithms based on label queries (i.e., noise-free point samples), which constitutes a setting similar to ours (Chen et al., 2022).
Revised: There also exist hardness results for learning algorithms based on label queries (i.e., noise-free point samples), which constitutes a setting similar to ours. More precisely, Chen et al.

Original: More precisely, this work shows that ReLU neural networks with constant depth and polynomial size constraints are not efﬁciently learnable up to a small squared loss with respect to a Gaussian distribution.
Revised: (2022) show that ReLU neural networks with constant depth and polynomial size constraints are not efﬁciently learnable up to a small squared loss with respect to a Gaussian distribution.

Original: However, the existing hardness results are in terms of runtime of the algorithm and are contingent on several (difﬁcult and unproven) conjectures from the area of cryptography (the decisional Difﬁe-Hellmann assumption or the "Learning with Errors" assumption); the correctness of these conjectures in particular would imply that P ‰ NP .
Revised: However, the existing hardness results are in terms of runtime of the algorithm and are contingent on several (difﬁcult and unproven) conjectures from the area of cryptography (the decisional Difﬁe-Hellmann assumption or the “Learning with Errors” assumption); the correctness of these conjectures in particular would imply that P ‰ NP .

Original: By contrast, our results are completely free of such assumptions and show that the considered problem is information-theoretically hard, not just computationally.
Revised: By contrast, our results are completely free of such assumptions and show that the considered problem is information-theoretically hard, not just computationally.

Original: As already hinted at in the introduction, our results further extend the broad literature on statistical learning theory (Anthony & Bartlett, 1999; Vapnik, 1999; Cucker & Smale, 2002b;
Revised: As already hinted at in the introduction, our results further extend the broad literature on statistical learning theory (Anthony & Bartlett, 1999; Vapnik, 1999; Cucker & Smale, 2002b;

Original: Bousquet et al., 2003; Vapnik, 2013; Mohri et al., 2018).
Revised: Bousquet et al., 2003; Vapnik, 2013; Mohri et al., 2018).

Original: Speciﬁcally, we provide fully explicit upper and lower bounds on the sample complexity of (regularized) neural network hypothesis classes.
Revised: Speciﬁcally, we provide fully explicit upper and lower bounds on the sample complexity of (regularized) neural network hypothesis classes .

Original: In the context of PAC learning, we analyze the realizable case, where the target function is contained in the hypothesis class Mohri et al. (2018, Theorem 3.20).
Revised: In the context of PAC learning, we analyze the realizable case, where the target function is contained in the hypothesis class (Mohri et al., 2018, Theorem 3.20).

Original: Contrary to standard results, we do not pose any assumptions, such as IID, on the data distribution, and even allow for adaptive sampling.
Revised: Contrary to standard results, we do not pose any assumptions, such as IID, on the data distribution, and even allow for adaptive sampling.

Original: Moreover, we analyze the complexity for all L p -norms with p P r 1 , 8s , whereas classical results mostly deal with the squared loss.
Revised: Moreover, we analyze the complexity for all L p norms with p P r 1 , 8s , whereas classical results mostly deal with the squared loss.

Original: As an example of such  results, we mention that (bounded)  classes with ﬁnite pseudodimension D can be learned to squared L 2 loss (cid:15) with O p D(cid:15) ´
Revised: As an example of such classical results, we mention that (bounded) hypothesis classes with ﬁnite pseudodimension D can be learned to squared L 2 loss (cid:15) with O p D(cid:15) ´

Original: 2 q point samples; see e.g. (Mohri et al., 2018,
Revised: 2 q point samples; see e.g., Mohri et al.

Original:  Theorem 11.8).
Revised: (2018, Theorem 11.8).

Original: Bounds for the pseudodimension of neural networks are readily available in the literature; see e.g. Bartlett et al.
Revised: Bounds for the pseudodimension of neural networks are readily available in the literature; see e.g., Bartlett et al.

Original: These bounds imply that learning ReLU networks in L 2 is tractable, in contrast to the L 8 setting.
Revised: These bounds imply that learning ReLU networks in L 2 is tractable, in contrast to the L 8 setting.

Original: Another related area is the identiﬁcation of (equivalence classes of) neural network parameters from its input-output map.
Revised: Another related area is the identiﬁcation of (equivalence classes of) neural network parameters from their input-output maps.

Original: While most works focus on scenarios where one has access to an inﬁnite number of queries (Fefferman & Markel, 1993; Vlaˇci´c & Bölcskei, 2022), there arealso recent results employing only ﬁnitely many samples (Rolnick & Kording, 2020; Fiedler et al., 2023).
Revised: While most works focus on scenarios where one has access to an inﬁnite number of queries (Fefferman & Markel, 1993; Vlaˇci´c & Bölcskei, 2022), there are recent results employing only ﬁnitely many samples (Rolnick & Kording, 2020; Fiedler et al., 2023).

Original: While robust identiﬁcation of the  network parameters is sufﬁcient to guarantee uniform accuracy,  it is not a necessary condition.
Revised: Robust identiﬁcation of the neural network parameters is sufﬁcient to guarantee uniform accuracy, but it is not a necessary condition.

Revised: Speciﬁcally, proximity of input-output maps does not necessarily imply proximity of corresponding neural network parameters (Berner et al., 2019).

Original: More generally, our results show that efﬁcient identiﬁcation from samples cannot be possible unless (as done in the previously mentioned works) further prior information is incorporated.
Revised: More generally, our results show that efﬁcient identiﬁcation from samples cannot be possible unless (as done in the previously mentioned works) further prior information is incorporated.

Original: In the same spirit, this restricts the applicability of model extraction attacks, such as model inversion or evasion attacks (Tramèr et al., 2016;
Revised: In the same spirit, this restricts the applicability of model extraction attacks, such as model inversion or evasion attacks (Tramèr et al., 2016;

Original: He et al., 2022).
Revised: He et al., 2022).

Original: Our results are most closely related to recent results by Grohs & Voigtlaender (2021) where target classes consisting of neural network approximation spaces are considered.
Revised: Our results are most closely related to recent results by Grohs & Voigtlaender (2021) where target classes consisting of neural network approximation spaces are considered.

Original: The results of Grohs &
Revised: The results of Grohs &

Original: Voigtlaender (2021), however, are purely asymptotic.
Revised: Voigtlaender (2021), however, are purely asymptotic.

Original: Since the asymptotic behavior incurred by the rate is often only visible for very ﬁne accuracies, the results of Grohs & Voigtlaender (2021)therefore cannot be applied to obtain concrete lower bounds on the required sample size.
Revised: Since the asymptotic behavior incurred by the rate is often only visible for very ﬁne accuracies, the results of Grohs & Voigtlaender (2021) cannot be applied to obtain concrete lower bounds on the required sample size.

Original: Our results are completely explicit in all parameters and readily yield practically relevant bounds.
Revised: Our results are completely explicit in all parameters and readily yield practically relevant bounds.

Original: They also elucidate the role of adaptive sampling and differentcoefﬁcient regularization methods.
Revised: They also elucidate the role of adaptive sampling and different regularization methods.

Original: 1. N OTATION
Revised: 1. N OTATION

Original: For d P N , we denote by C pr 0 , 1 s d q the space of continuous functions f : r 0 , 1 s d Ñ R .
Revised: For d P N , we denote by C pr 0 , 1 s d q the space of continuous functions f : r 0 , 1 s d Ñ R .

Original: For a ﬁnite set
Revised: For a ﬁnite set

Original: I and p a i q i P I P
Revised: I and p a i q i P I P

Original: R
Revised: R

Original: I , we write ř i P
Revised: I , we write ř i P

Original: I a i : “ 1 | I | ř i P I a i .
Revised: I a i : “ 1 | I | ř i P I a i .

Original: For m P N , we write r m s : “ t 1 , . . .
Revised: For m P N , we write r m s : “ t 1 , . . .

Original: , m u .
Revised: , m u .

Original: For
Revised: For

Original: A Ă R d , we denote by A o the set of interior points of A .
Revised: A Ă R d , we denote by A o the set of interior points of A .

Original: For any subset A of a vector space V , any c P R , and any y P V , we further deﬁne y ` c
Revised: For any subset A of a vector space V , any c P R , and any y P V , we further deﬁne y ` c

Original: ¨ A : “ t y ` ca : a P A u .
Revised: ¨ A : “ t y ` ca : a P A u .

Original: For a matrix W P R n ˆ k and q P r 1 , 8q , we write } W } (cid:96) q : “ ` ř i,j | W i,j | q ˘ 1 { q , and for q “ 8 we write } W } (cid:96) 8 : “ max i,j | W i,j | .
Revised: For a matrix W P R n ˆ k and q P r 1 , 8q , we write } W } (cid:96) q : “ ` ř i,j | W i,j | q ˘ 1 { q , and for q “ 8 we write } W } (cid:96) 8 : “ max i,j | W i,j | .

Original: For vectors b P R n , we use the analogously deﬁned notation } b } (cid:96) q .
Revised: For vectors b P R n , we use the analogously deﬁned notation } b } (cid:96) q .

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: This section contains our main theoretical results.
Revised: This section contains our main theoretical results.

Original: We introduce the considered classes of algorithms in Section 2.1 and target classes in Section 2.2.
Revised: We introduce the considered classes of algorithms in Section 2.1 and target classes in Section 2.2.

Original: Our main lower and upper bounds are formulated and proven in Section 2.3 and Section 2.4, respectively.
Revised: Our main lower and upper bounds are formulated and proven in Section 2.3 and Section 2.4, respectively.

Original: 2. A DAPTIVE ( RANDOMIZED ) ALGORITHMS BASED ON POINT SAMPLES
Revised: 2. A DAPTIVE ( RANDOMIZED ) ALGORITHMS BASED ON POINT SAMPLES

Original: As described in the introduction, our goal is to analyze how well one can recover an unknown function u from a target class U in a Banach space Y based on point samples.
Revised: As described in the introduction, our goal is to analyze how well one can recover an unknown function u from a target class U in a Banach space Y based on point samples.

Original: This is one of the main problems in information-based complexity (Traub, 2003), and in this section we brieﬂy recall the most important related notions.
Revised: This is one of the main problems in information-based complexity (Traub, 2003), and in this section we brieﬂy recall the most important related notions.

Original: Given U Ă C pr 0 , 1 s d q X Y for a Banach space Y , we say that a map A : U Ñ Y is an adaptive deterministic method using m P N point samples if there are f 1 P r 0 , 1 s d and mappings[Equation] i
Revised: Given U Ă C pr 0 , 1 s d q X Y for a Banach space Y , we say that a map A : U Ñ Y is an adaptive deterministic method using m P N point samples if there are f 1 P r 0 , 1 s d and mappings

Original: [Equation]
Revised: [Equation]

Original: such that for every u P U , using the point sequence x p u q “ p x 1 , . .
Revised: such that for every u P U , using the point sequence x p u q “ p x 1 , . .

Original: , x m q Ă r 0 , 1 s d deﬁned as
Revised: , x m q Ă r 0 , 1 s d deﬁned as

Original: [Equation]
Revised: [Equation]

Original: the map A is of the form
Revised: the map A is of the form

Original: A p u q “ Q p x 1 , . . .
Revised: A p u q “ Q p x 1 , . . .

Original: , x m , u p x 1 q , . . .
Revised: , x m , u p x 1 q , . . .

Original: , u p x m qq P Y .
Revised: , u p x m qq P Y .

Original: The set of all deterministic methods using m point samples will be denoted  Alg m p U, Y q .
Revised: The set of all deterministic methods using m point samples is denoted by Alg m p U, Y q .

Original: In addition to such deterministic methods, we also study randomized methods deﬁned as follows:
Revised: In addition to such deterministic methods, we also study randomized methods deﬁned as follows:

Original: A tuple p A , m q is called an adaptive random method using m P N point samples on average if A “ p A ω q ω P Ω where p Ω , F , P q is a probability space, and where m : Ω Ñ N is such that the following conditions hold:
Revised: A tuple p A , m q is called an adaptive random method using m P N point samples on average if A “ p A ω q ω P Ω where p Ω , F , P q is a probability space, and where m : Ω Ñ N is such that the following conditions hold:

Original: m is measurable, and E r m s ď m ; 2. @
Revised: m is measurable, and E r m s ď m ; 2. @

Original: u P U : ω ÞÑ A ω
Revised: u P U : ω ÞÑ A ω

Original: p u q is measurable with respect to the Borel σ -algebra on Y ;3. @
Revised: p u q is measurable with respect to the Borel σ -algebra on Y ;

Original: ω P Ω : A ω P Alg m p ω q p U, Y q .
Revised: [Equation]

Original: The set of all random methods using m point samples on average will be denoted  Alg MCm p U, Y q , since such methods are sometimes called Monte-Carlo algorithms .
Revised: The set of all random methods using m point samples on average will be denoted by Alg MCm p U, Y q , since such methods are sometimes called Monte-Carlo (MC) algorithms.

Original: For a target class U , we denote the optimal (randomized) error as
Revised: For a target class U , we deﬁne the optimal (randomized) error as

Original: [Equation]
Revised: [Equation]

Original: We note that Alg m p U, Y q Ă Alg MCm p U, Y q , since each deterministic method can be interpreted as a randomized method over a trivial probability space.
Revised: We note that Alg m p U, Y q Ă Alg MCm p U, Y q , since each deterministic method can be interpreted as a randomized method over a trivial probability space.

Original: 2. N EURAL NETWORK CLASSES
Revised: 2. N EURAL NETWORK CLASSES

Original: We will be concerned with target classes related to ReLU neural networks.
Revised: We will be concerned with target classes related to ReLU neural networks.

Original: These will be deﬁned in the present subsection.
Revised: These will be deﬁned in the present subsection.

Original: Let (cid:37) : R Ñ R , (cid:37) p x q “ max t 0 , x u  be the ReLU activation function .
Revised: Let (cid:37) : R Ñ R , (cid:37) p x q “ max t 0 , x u , be the ReLU activation function .

Original: Given a depth L P N , an architecture p N 0 , N 1 , . . .
Revised: Given a depth L P N , an architecture p N 0 , N 1 , . . .

Original: , N L q P N L [Equation] 1 , and neural network coefﬁcients
Revised: , N L q P N L ` 1 , and neural network coefﬁcients

Original: [Equation]
Revised: [Equation]

Original: we deﬁne their realization R p
Revised: we deﬁne their realization R p

Original: Φ q P C p R N 0 , R N L q asR p Φ qp x q “ x L , where
Revised: Φ q P C p R N 0 , R N L q as

Original: [Equation]
Revised: [Equation]

Original: with (cid:37)  applied componentwise.
Revised: where (cid:37) is applied componentwise and φ i : R N i ´ 1 Ñ R N i , x ÞÑ W i x ` b i , for i P r L s .

Original: Given c ą 0 and q P r 1 , 8s , deﬁne the class
Revised: Given c ąand q P r 1 , 8s , deﬁne the class

Original: [Equation]
Revised: [Equation]

Original: [Equation] where } Φ } 
Revised: where } Φ } (cid:96) q : “ max 1 ď i ď L max t} W i } (cid:96) q

Original: [Equation]
Revised: [Equation]

Original:  u [Equation]
Revised: } b i } (cid:96) q u .

Original: For studying target classes related to neural networks, the following deﬁnition will be useful.
Revised: To study target classes related to neural networks, the following deﬁnition will be useful.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Let U, H Ă C pr 0 , 1 s d q .
Revised: Let U, H Ă C pr 0 , 1 s d q .

Original: We say that U contains a copy of H (attached to u 0 P U with constant c 0 ą 0 ) if u 0 ` c 0 ¨ H Ă U.
Revised: We say that U contains a copy of H , attached to u 0 P U with constant c 0 P p 0 , 8q , if u 0 ` c 0 ¨ H Ă U.

Original: 2. L OWER BOUND
Revised: 2. L OWER BOUND

Original: The following result constitutes the main result of the present paper.
Revised: The following result constitutes the main result of the present paper.

Original: Theorem 1.1 readily follows from it as a special case.
Revised: Theorem 1.1 readily follows from it as a special case.

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Let L P N ě 3 , d, B P N , p, q P r 1 , 8s , and
Revised: Let L P N ě 3 , d, B P N , p, q P r 1 , 8s , and

Original: c P p 0 , 8q .
Revised: c P p 0 , 8q .

Original: Suppose that the target class U Ă C pr 0 , 1 s d q contains a copy of H q p d,B...,B, 1 q ,c with constant c 0  , where the B in p d, B, . . .
Revised: Suppose that the target class U Ă C pr 0 , 1 s d q contains a copy of H q p d,B...,B, 1 q ,c with constant c 0 P p 0 , 8q , where the B in p d, B, . . .

Original: , B, 1 q appears L ´ 1 times.
Revised: , B, 1 q appears L ´ 1 times.

Original: Then, for any s P N with s ď min [Equation] we have 
Revised: Then, for any s P N with s ď min (cid:32) B 3 , d ( we have err

Original: [Equation]
Revised: [Equation]

Original: whereΩ
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This follows by combining Theorem A.5 with Lemmas A.2 and A.3 in the appendix.
Revised: This follows by combining Theorem A.5 with Lemmas A.2 and A.3 in the appendix.

Original: Remark 2.3.
Revised: Remark 2.3.

Original: For p !
Revised: For p !

Original: 8 , the bound from above does not necessarily imply that an intractable number of training samples is needed.
Revised: 8 , the bound from above does not necessarily imply that an intractable number of training samples is needed.

Original: This is a reﬂection of the fact that  learning isefﬁciently possible (at least if one only considers the number of training samples and not the runtime of the algorithm) in this regime.
Revised: This is a reﬂection of the fact that efﬁcient learning is possible (at least if one only considers the number of training samples and not the runtime of the algorithm) in this regime.

Original: Indeed, it is well-known in statistical learning theory(see for instance (Cucker & Smale, 2002a, Proposition 7)) thatwhen the error is measured in L 2 , one can get good learning bounds based on the  L 8 -entropy numbers of the class of target functions.
Revised: Indeed, it is well-known in statistical learning theory that one obtains learning bounds based on the entropy numbers (w.r.t. the L 8 norm) of the class of target functions, when the error is measured in L 2 , see, for instance, Cucker & Smale (2002a, Proposition 7).

Original: The ε -entropy numbers of a class of neural networks with L layers and W (bounded) weights scale linearly in W, L and logarithmically in 1 { ε , so that one gets good L 2 learning bounds.
Revised: The ε -entropy numbers of a class of neural networks with L layers and w (bounded) weights scale linearly in w, L and logarithmically in 1 { ε , so that one gets tractable L 2 learning bounds.

Original: By interpolation for L p -norms (noting that in our case the target functions are bounded, so that the L 8 -reconstruction error is bounded, even though the decay with m is very bad), this also implies L p -learning bounds, but these get worse and worse as p Ñ 8 .
Revised: By interpolation for L p norms (noting that in our case the target functions are bounded, so that the L 8 reconstruction error is bounded, even though the decay with m is very bad), this also implies L p learning bounds, but these get worse and worse as p Ñ 8 .

Original: We remark that these learning bounds are based on empirical risk minimization, which might be computationally infeasible (Vu, 1998);but since our lower bounds should hold for any feasible algorithm (irrespective of its computational complexity), this means that one cannot expect to get an intractable lower bound for p ! 8 in our setting.
Revised: We remark that these learning bounds are based on empirical risk minimization, which might be computationally infeasible (Vu, 1998); since our lower bounds should hold for any feasible algorithm (irrespective of its computational complexity), this means that one cannot expect to get an intractable lower bound for p ! 8 in our setting.

Original: The idea of the proof of Theorem 2.2 (here only presented for u 0 “ 0 and s “ d , which implies that
Revised: The idea of the proof of Theorem 2.2 (here only presented for u 0 “ 0 and s “ d , which implies that

Original: B ě 3 d ) is as follows:
Revised: B ě 3 d ) is as follows:

Original: We ﬁrst show (see Lemmas A.2 and A.3) that the neural network set H q p d,B,...,B, 1 q ,c contains a large class of “bump functions” of the form λ ¨ ϑ M,y .
Revised: We ﬁrst show (see Lemmas A.2 and A.3) that the neural network set H q p d,B,...,B, 1 q ,c contains a large class of “bump functions” of the form λ ¨ ϑ M,y .

Original: Here, ϑ M,y is supported on the set y ` r´ 1 M , 1 M s d and satisﬁes } ϑ M,y } L p pr 0 , 1 s d q — M ´ d { p , and M P N , y P r 0 , 1 s d can be chosen arbitrarily; see Lemma A.1.
Revised: Here, ϑ M,y is supported on the set y ` r´ 1 M , 1 M s d and satisﬁes } ϑ M,y } L p pr 0 , 1 s d q — M ´ d { p , where M P N and y P r 0 , 1 s d can be chosen arbitrarily; see Lemma A.1.

Original: The size of the scaling factor λ “ λ p M, c, q, d, L q depends crucially on the regularization parameters c, q .
Revised: The size of the scaling factor λ “ λ p M, c, q, d, L q depends crucially on the regularization parameters c and q .

Original: This is the main technical part of the proof, requiring to construct suitable neural networks adhering to the imposed (cid:96) q -restrictions on the weightsbut for which λ is as big as possible.
Revised: This is the main technical part of the proof, requiring to construct suitable neural networks adhering to the imposed (cid:96) q restrictions on the weights for which λ is as big as possible.

Original: If one learns using m points samples x 1 , . .
Revised: If one learns using m points samples x 1 , . .

Original: , x m and if M “ O p m´ 1 { d q , then a volume packing argument shows that there exists y P r 0 , 1 s d such that ϑ M,
Revised: , x m and if M “ O p m 1 { d q , then a volume packing argument shows that there exists y P r 0 , 1 s d such that ϑ M,

Original: y p x i q “ 0 for all i  .
Revised: y p x i q “ 0 for all i P r m s .

Original: This means that the learner cannot distinguish the function λ ¨ ϑ M,y P H q p d,B,...,B, 1 q ,c from the zero function and will thus make an error of roughly } λ ¨ ϑ M,y } L p — λ ¨ M ´ d { p .
Revised: This means that the learner cannot distinguish the function λ ¨ ϑ M,y P H q p d,B,...,B, 1 q ,c from the zero function and will thus make an error of roughly } λ ¨ ϑ M,y } L p — λ ¨ M ´ d { p .

Original: This already implies the lower bound in Theorem 2.2 for the case of deterministic algorithms.
Revised: This already implies the lower bound in Theorem 2.2 for the case of deterministic algorithms.

Original: To get the lower bound for randomized algorithms, we employ a technique from informationbased complexity (see, e.g., Heinrich, 1994): For randomized algorithms using m point samples, we again choose M “ O p m´ 1 { d q .
Revised: To get the lower bound for randomized algorithms using m point samples on average, we employ a technique from information-based complexity (see, e.g., Heinrich, 1994): We again set M “ O p m 1 { d q and deﬁne p

Original: Then, using a volume packing argument one can show that if one chooses y “ y (cid:96) “ (cid:96)M for (cid:96) P t 0 , . . . , M u d  ,  then  for any choice of m sampling points x 1 , . . .
Revised: y (cid:96) q (cid:96) Pr M { 2 s d as the nodes of a uniform grid on r 0 , 1 s d with width 2 { M . Using a volume packing argument, we then show that for any choice of m sampling points x 1 , . . .

Original: , x m , “at least half of the functions
Revised: , x m , “at least half of the functions

Original: ϑ M,y (cid:96) avoid all the sampling points”, i e., for at least half of the indices (cid:96) , it holds that ϑ M,y (cid:96) p x i q “ 0 for all i  .
Revised: ϑ M,y (cid:96) avoid all the sampling points”, i.e., for at least half of the indices (cid:96) , it holds that ϑ M,y (cid:96) p x i q “ 0 for all i P r m s .

Original: A learner using the samples x 1 , . . .
Revised: A learner using the samples x 1 , . . .

Original: , x m can thus not distinguish between the zero function and λ ¨ ϑ M,y (cid:96) P H q p d,B,...,B, 1 q ,c for at least half of the parameters (cid:96) .
Revised: , x m can thus not distinguish between the zero function and λ ¨ ϑ M,y (cid:96) P H q p d,B,...,B, 1 q ,c for at least half of the indices (cid:96) .

Original: Therefore, any deterministic algorithm will make an error of Ω p λ ¨ M ´ d { p q on average with respect to (cid:96) .
Revised: Therefore, any deterministic algorithm will make an error of Ω p λ ¨ M ´ d { p q on average with respect to (cid:96) .

Original: Since each randomized algorithm A “ p A ω q ω P Ω is a collection of deterministic algorithms and since taking an average commutes with taking the expectation, this implies that any randomized algorithmusing m point samples will have  expected error of Ω p λ ¨ M ´ d { p q on average with respect to (cid:96) .
Revised: Since each randomized algorithm A “ p A ω q ω P Ω is a collection of deterministic algorithms and since taking an average commutes with taking the expectation, this implies that any randomized algorithm will have an expected error of Ω p λ ¨ M ´ d { p q on average with respect to (cid:96) .

Original: This easily implies the stated bound.
Revised: This easily implies the stated bound.

Original: As mentioned in the introduction, we want to emphasize that well-trained neural networks can indeed exhibit such bump functions, see Figure 1 and Adcock & Dexter (2021); Fiedler et al.
Revised: As mentioned in the introduction we want to emphasize that well-trained neural networks can indeed exhibit such bump functions, see Figure 1 and Adcock & Dexter (2021); Fiedler et al.

Original: 2. U PPER BOUND
Revised: 2. U PPER BOUND

Original: In this section we present our main upper bound, which directly implies the statement of Theorem 1.4.
Revised: In this section we present our main upper bound, which directly implies the statement of Theorem 1.4.

Original: Theorem 2.4.
Revised: Theorem 2.4.

Original: Let L, d P N , q P r 1 , 8s , c P p 0 , 8q , and N 1 , . .
Revised: Let L, d P N , q P r 1 , 8s , c P p 0 , 8q , and N 1 , . .

Original: . , N L ´ 1 P N .
Revised: . , N L ´ 1 P N .

Original: Then, we have 
Revised: Then, we have err MCm

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This follows by combining Lemmas B.2 and B.3 in the appendix.
Revised: This follows by combining Lemmas B.2 and B.3 in the appendix.

Original: Let us outline the main idea of the proof.
Revised: Let us outline the main idea of the proof.

Original: We ﬁrst show that each neural network
Revised: We ﬁrst show that each neural network

Original: R p Φ q P H q p N 0 ,...,N L q ,c is Lipschitz-continuous, where the Lipschitz constant can be conveniently bounded in terms of the parameters N 0 , . . .
Revised: R p Φ q P H q p N 0 ,...,N L q ,c is Lipschitz-continuous, where the Lipschitz constant can be conveniently bounded in terms of the parameters N 0 , . . .

Original: , N L , c, q , see Lemma B.2 in the appendix.
Revised: , N L , c, q , see Lemma B.2 in the appendix.

Original: In Lemma B.3, we then show that any function with moderate Lipschitz constant can be reconstructed from samples by piecewise constant interpolation.
Revised: In Lemma B.3, we then show that any function with moderate Lipschitz constant can be reconstructed from samples by piecewise constant interpolation.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: Having established fundamental bounds on the performance of any learning algorithm, we want to numerically evaluate the performance of commonly used deep learning methods.
Revised: Having established fundamental bounds on the performance of any learning algorithm, we want to numerically evaluate the performance of commonly used deep learning methods.

Original: To illustrate our main result in Theorem 2.2, we estimate the error in (2) by a tractable approximation in a student-teacher setting.
Revised: To illustrate our main result in Theorem 2.2, we estimate the error in (2) by a tractable approximation in a student-teacher setting.

Original: Speciﬁcally, we estimate the minimal error over neural network target functions (“teachers”)
Revised: Speciﬁcally, we estimate the minimal error over neural network target functions (“teachers”)

Original: p U Ă H q p d,N 1 ,...,N L ´ 1 , 1 q ,c for deep learning algorithms p
Revised: p U Ă H q p d,N 1 ,...,N L ´ 1 , 1 q ,c for deep learning algorithms p

Original: A Ă Alg MCm p U, L p q via Monte-Carlo sampling, i.e.,
Revised: A Ă Alg MCm p U, L p q via Monte-Carlo sampling, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where p X j q Jj “ 1 are independent evaluation samples uniformly distributed on 3 r´ 0 .
Revised: where p X j q Jj “ 1 are independent evaluation samples uniformly distributed on 3 r´ 0 .

Original: 5 s d and p Ω represents the seeds for the algorithms.
Revised: 5 s d and p Ω represents the seeds for the algorithms.

Original: We obtain teacher networks u P H 8p d,N 1 ,...,N L ´ 1 , 1 q ,c by sampling their coefﬁcients Φ componentwise according to a uniform distribution on r´ c, c s .
Revised: We obtain teacher networks u P H 8p d,N 1 ,...,N L ´ 1 , 1 q ,c by sampling their coefﬁcients Φ componentwise according to a uniform distribution on r´ c, c s .

Original: For every algorithm p A , m q P p A and seed ω P p Ω we consider point
Revised: For every algorithm p A , m q P p A and seed ω P p Ω we consider point

Original: sequences x p u q uniformly distributed in r´ 0 .
Revised: sequences x p u q uniformly distributed in r´ 0 .

Original: 5 s d with m p ω q “ m .
Revised: 5 s d with m p ω q “ m .

Original: The corresponding point samples are used to train the coefﬁcients of a neural network (“student”) using the Adam optimizer (Kingma & Ba, 2015) with exponentially decaying learning rate.
Revised: The corresponding point samples are used to train the coefﬁcients of a neural network (“student”) using the Adam optimizer (Kingma & Ba, 2015) with exponentially decaying learning rate.

Original: We consider input dimensions d “ 1 and d “ 3 , for each of which we compute the error in (6) for 4 different sample sizes m over 40 teacher networks u .
Revised: We consider input dimensions d “ 1 and d “ 3 , for each of which we compute the error in (3) for 4 different sample sizes m over 40 teacher networks u .

Original: For each combination, we train student networks with 3 different seeds, 3 different widths, and 3 different batch-sizes.
Revised: For each combination, we train student networks with 3 different seeds, 3 different widths, and 3 different batch-sizes.

Original: In summary, this yields
Revised: In summary, this yields

Original: [Equation]
Revised: [Equation]

Original: “ 8640 experiments each executed on a single GPU.
Revised: 8640 experiments each executed on a single GPU.

Original: The precise hyperparameters can be found in Tables 1 and 3 of Appendix C.
Revised: The precise hyperparameters can be found in Tables 1 and 3 in Appendix C.

Original: Figure 2 shows that there is a clear gap between the errors x err m p p U, L p ; p
Revised: Figure 2 shows that there is a clear gap between the errors x err m p p U, L p ; p

Original: A q for p P t 1 , 2 u and p “ 8 .
Revised: A q for p P t 1 , 2 u and p “ 8 .

Original: Especially in the one-dimensional case, the rate  w.r.t.
Revised: Especially in the one-dimensional case, the rate x err m p p U, L 8 ; p A q w.r.t.

Original: the number of samples  also seems to stagnate at a precision that might be insufﬁcient for certain applications.
Revised: the number of samples m also seems to stagnate at a precision that might be insufﬁcient for certain applications.

Original: Figure 3 illustrates that the errors are caused by spikes of the teacher network which are not covered by any sample.
Revised: Figure 3 illustrates that the errors are caused by spikes of the teacher network which are not covered by any sample.

Original: Note that this is very similar to the construction in the proof of our main result, see Section 2.3.
Revised: Note that this is very similar to the construction in the proof of our main result, see Section 2.3.

Original: In general, the rates worsen when considering more teacher networks p U and improve when considering further deep learning algorithms p A , including other architectures or more elaborate training and sampling schemes.
Revised: In general, the rates worsen when considering more teacher networks p U and improve when considering further deep learning algorithms p A , including other architectures or more elaborate training and sampling schemes.

Original: Note, however, that each setting needs to be evaluated for a number of teacher networks, sample sizes, and seeds.
Revised: Note, however, that each setting needs to be evaluated for a number of teacher networks, sample sizes, and seeds.

Original: We provide an extensible implementation 4 in PyTorch (Paszke et al., 2019) featuring multi-node experiment execution and hyperparameter tuning using Ray Tune (Liaw et al., 2018), experiment tracking using Weights & Biases and TensorBoard, and ﬂexible experiment conﬁguration.
Revised: We provide an extensible implementation 4 in PyTorch (Paszke et al., 2019) featuring multi-node experiment execution and hyperparameter tuning using Ray Tune (Liaw et al., 2018), experiment tracking using Weights & Biases and TensorBoard, and ﬂexible experiment conﬁguration.

Original: Building upon our work, research teams with sufﬁcient computational resources can provide further numerical evidence on an even larger scale.
Revised: Building upon our work, research teams with sufﬁcient computational resources can provide further numerical evidence on an even larger scale.

Original:  D ISCUSSION AND L IMITATIONS Discussion
Revised:  D ISCUSSION AND L IMITATIONS Discussion.

Original: We derived fundamental upper and lower bounds for the number oftraining samples needed for any algorithm to reconstruct an arbitrary function from a target class consisting of (or containing) realizations of neural networks with ReLU activation function of a given architecture and subject to (cid:96) q -regularization constraints on the network coefﬁcients.
Revised: We derived fundamental upper and lower bounds for the number of samples needed for any algorithm to reconstruct an arbitrary function from a target class containing realizations of neural networks with ReLU activation function of a given architecture and subject to (cid:96) q regularization constraints on the network coefﬁcients, see Theorems 2.2 and 2.4.

Original: These bounds are completely explicit in the network architecture, the type of regularization, and the norm in which the reconstruction error is measured.
Revised: These bounds are completely explicit in the network architecture, the type of regularization, and the norm in which the reconstruction error is measured.

Original: We observe that our lower bounds are severely more restrictive if the error is measured in the uniform L 8 -norm rather than the (more commonly studied) L 2 -norm.
Revised: We observe that our lower bounds are severely more restrictive if the error is measured in the uniform L 8 norm rather than the (more commonly studied) L 2 norm.

Original: Particularly, learning a class of neural networks with ReLU activation function with moderately high accuracy in the L 8 -norm is intractable for moderate input dimensions, as well as network width and depth.
Revised: Particularly, learning a class of neural networks with ReLU activation function with moderately high accuracy in the L 8 norm is intractable for moderate input dimensions, as well as network widths and depths.

Original: We anticipate that further investigations into the sample complexity of neural network classes can eventually contribute to a better understanding of possible circumstances under which it is possible to design reliable deep learning algorithms and help explain well-known instability phenomena such
Revised: We anticipate that further investigations into the sample complexity of neural network classes can eventually contribute to a better understanding of possible circumstances under which it is possible to design reliable deep learning algorithms and help explain well-known instability phenomena such

Original: [Figure]
Revised: [Figure]

Revised: d

Original: [Figure]
Revised: [Figure]

Revised: L ∞ L

Revised: [Figure]

Revised: [Figure]

Original: as adversarial examples.
Revised: as adversarial examples.

Original: Such an understanding can be beneﬁcial in assessing the potential and limitations of machine learning methods applied to security- and safety-critical scenarios.
Revised: Such an understanding can be beneﬁcial in assessing the potential and limitations of machine learning methods applied to security- and safety-critical scenarios.

Original: Limitations and Outlook.
Revised: Limitations and Outlook.

Original: We ﬁnally discuss some possible implications and also limitations of our work.
Revised: We ﬁnally discuss some possible implications and also limitations of our work.

Original: First of all, our results are highly speciﬁc to neural networks with the ReLU activation function.
Revised: First of all, our results are highly speciﬁc to neural networks with the ReLU activation function.

Original: We expect that obtaining similar results for other activation functions will require substantially new methods.
Revised: We expect that obtaining similar results for other activation functions will require substantially new methods.

Original: We plan to investigate this in future work.
Revised: We plan to investigate this in future work.

Original: The explicit nature of our results reveal a discrepancy between the lower and upper bound, especially for high dimensions.
Revised: The explicit nature of our results reveal a discrepancy between the lower and upper bound, especially for high dimensions.

Original: We conjecture that both the current upper and lower bounds are not quite optimal.
Revised: We conjecture that both the current upper and lower bounds are not quite optimal.

Original: Determining to which extent one can tighten the bounds is an interesting open problem.
Revised: Determining to which extent one can tighten the bounds is an interesting open problem.

Original: Our analysis is a worst-case analysis in the sense that we show that for any given algorithm A , there exists at least one u in our target class U on which A performs poorly.
Revised: Our analysis is a worst-case analysis in the sense that we show that for any given algorithm A , there exists at least one u in our target class U on which A performs poorly.

Original: The question of whether this poor behavior is actually generic will be studied in future work.
Revised: The question of whether this poor behavior is actually generic will be studied in future work.

Original: One way to establish such generic results could be to prove that our considered target classes contain copies of neural network realizations attached to many different u ’s.
Revised: One way to establish such generic results could be to prove that our considered target classes contain copies of neural network realizations attached to many different u ’s.

Original: Finally, we consider target classes U that contain  realizations ofall neural networks  subject to different regularizations.
Revised: Finally, we consider target classes U that contain all realizations of neural networks with a given architecture subject to different regularizations.

Original: This is justiﬁed because whenever a deep learning method is employed to reconstruct a function  by representing it approximately by a neural network of a given architecture (and without further knowledge about u ), a natural minimal requirement is that the method should perform well if the sought function is in fact equal to a neural network.
Revised: This can be justiﬁed as follows: Whenever a deep learning method is employed to reconstruct a function u by representing it approximately by a neural network (without further knowledge about u ), a natural minimal requirement is that the method should perform well if the sought function is in fact equal to a neural network.

Original: However, if additional problem information about u can be incorporated into the learning problem it may be possible to overcome the barriers shown in this work.
Revised: However, if additional problem information about u can be incorporated into the learning problem it may be possible to overcome the barriers shown in this work.

Original: The degree to which this is possible, as well as the extension of our results to other architectures, such as convolutional neural networks, transformer networks, and graph neural networks will be the subject of future work. 
Revised: The degree to which this is possible, as well as the extension of our results to other architectures, such as convolutional neural networks, transformers, and graph neural networks will be the subject of future work. A CKNOWLEDGMENTS

Revised: The research of Julius Berner was supported by the Austrian Science Fund (FWF) under grant

Revised: I3403-N32 and by the Vienna Science and Technology Fund (WWTF) under grant ICT19-041.

Revised: The computational results presented have been achieved in part using the Vienna Scientiﬁc Cluster (VSC).

Revised: Felix Voigtlaender acknowledges support by the DFG in the context of the Emmy Noether junior research group VO 2594/1-

Original: A P ROOF OF THE LOWER BOUND INS ECTION 2. A. C ONSTRUCTION OF HAT FUNCTIONS IMPLEMENTED BY R E LU NETWORKS
Revised: A P ROOF OF THE LOWER BOUND IN

Revised:  A. C ONSTRUCTION OF HAT FUNCTIONS IMPLEMENTED BY R E LU NETWORKS

Original: For d P N , M ą 0 , σ P R , s P r d s  and y P R d , deﬁne Λ M,σ :
Revised: For d P N , M ą 0 , σ P R , s P r d s , and y P R d , deﬁne Λ M,σ :

Original: R Ñ p´8 , 1 s ,t
Revised: R Ñ p´8 , 1 s ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −−−  0 .0 .1 .1 .2 .0 .0 .1 .1 .2 .Λ 2 , 1 Λ 4 ,Λ 4 , and furthermore
Revised: and furthermore

Original: [Equation]
Revised: [Equation]

Original: where, as before, (cid:37) : R Ñ R , x ÞÑ max t 0 , x u  denotes the ReLU activation function.
Revised: where, as before, (cid:37) : R Ñ R , x ÞÑ max t 0 , x u , denotes the ReLU activation function.

Original: A plot of Λ M,σ is shown in Figure 4.
Revised: A plot of Λ M,σ is shown in Figure 4.

Original: With these deﬁnitions, the function ϑ p s q M,y satisﬁes the following properties:
Revised: With these deﬁnitions, the function ϑ p s q M,y satisﬁes the following properties:

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For d P N , s P r d s , M ě 1 , y P r 0 , 1 s d , and p P p 0 , 8s , we have
Revised: For d P N , s P r d s , M ě 1 , y P r 0 , 1 s d , and p P p 0 , 8s , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us ﬁrst give a quick overview of the proof.
Revised: Let us ﬁrst give a quick overview of the proof.

Original: The statement on the support of ϑ p s q M,y follows by observing that ∆ p s q M,y p x q ą 0 can only happen if Λ M,y i
Revised: The statement on the support of ϑ p s q M,y follows by observing that ∆ p s q M,y p x q ą 0 can only happen if Λ M,y i

Original: p x i q ą 0 for all
Revised: p x i q ą 0 for all

Original: i P r s s .
Revised: i P r s s .

Original: As 0 ď ϑ p s q M,y ď 1 , the upper bound on the L p pr 0 , 1 s d q -norm can then be estimated by the Lebesgue measure of the intersection of the support of ϑ p s q M,y and the hypercube r 0 , 1 s d .
Revised: As 0 ď ϑ p s q M,y ď 1 , the upper bound on the L p pr 0 , 1 s d q norm can then be estimated by the Lebesgue measure of the intersection of the support of ϑ p s q M,y and the hypercube r 0 , 1 s d .

Original: For the lower bound we compute the measure of the intersection with a subset of the support on which it holds that ϑ p s q M,y ě 12 .
Revised: For the lower bound we compute the measure of the intersection with a subset of the support on which it holds that ϑ p s q M,y ě 12 .

Original: We start by proving the statement on the support of ϑ p s q M,y .
Revised: We start by proving the statement on the support of ϑ p s q M,y .

Original: If 0 ‰ ϑ p s q M,y p x q , then ∆ p s q M,y p x q ą 0 , meaning ř si “ 1 Λ M,y i p x i q ą s ´ 1 .
Revised: If 0 ‰ ϑ p s q M,y p x q , then ∆ p s q M,y p x q ą 0 , meaning ř si “ 1 Λ M,y i p x i q ą s ´ 1 .

Original: Because of Λ M,y i p x i q P p´8 , 1 s for all
Revised: Because of Λ M,y i p x i q P p´8 , 1 s for all

Original: i P r s s ,
Revised: i P r s s ,

Original: this is only possible if Λ M,y i
Revised: this is only possible if Λ M,y i

Original: p x i q ą 0 for all
Revised: p x i q ą 0 for all

Original: i P r s s .
Revised: i P r s s .

Original: Directly from the deﬁnition of Λ M,y i (see also Figure 4), this implies | x
Revised: Directly from the deﬁnition of Λ M,y i (see also Figure 4), this implies | x

Original: i ´ y i | ď 1 M for all
Revised: i ´ y i | ď 1 M for all

Original: i P r s s , meaning x P y ` p M ´ 1 r´ 1 , 1 s s ˆ R d ´ s q .
Revised: i P r s s , meaning x P y ` p M ´ 1 r´ 1 , 1 s s ˆ R d ´ s q .

Original: This proves the ﬁrst claim.
Revised: This proves the ﬁrst claim.

Original: Regarding the second claim, deﬁne y ˚ : “ p y 1 , . .
Revised: Regarding the second claim, deﬁne y ˚ : “ p y 1 , . .

Original: , y s q P R s , and, for k P N , denote by λ k
Revised: , y s q P R s , and, for k P N , denote by λ k

Original: the
Revised: the

Original: Lebesgue measure on R k .
Revised: Lebesgue measure on R k .

Original: Then, since r 0 , 1 s d X supp ϑ p s q M,y Ă p y ˚ ` M ´ 1 r´ 1 , 1 s s q ˆ r 0 , 1 s d ´ s and 0 ď ϑ p s q M,y ď 1 , we see that 
Revised: Then, since r 0 , 1 s d X supp ϑ p s q M,y Ă p y ˚ ` M ´ 1 r´ 1 , 1 s s q ˆ r 0 , 1 s d ´ s and 0 ď ϑ p s q M,y ď 1 , we see that } ϑ p s q M,y } L p pr 0 ,

Original: [Equation]
Revised: [Equation]

Revised: ď

Original:  For the converse estimate, let us also write x ˚ “ p x 1 , . . .
Revised: [Equation] ` [Equation] For the converse estimate, let us also write x ˚ “ p x 1 , . . .

Original: , x s q for x P R d .
Revised: , x s q for x P R d .

Original: Then, if x P R d satisﬁes x ˚ P
Revised: Then, if x P R d satisﬁes x ˚ P

Original: y ˚ ` 1 2 Ms r´ 1 , 
Revised: y ˚ ` 1 2 Ms r´ 1 , 1 s s , we see [Equation] for i P [Equation]

Original:  By deﬁnition of Λ M,y i , this implies Λ M,y i
Revised: s [Equation] By deﬁnition of Λ M,y i , this implies Λ M,y i

Original: p x i q “ 1 ´ M ¨ | x i
Revised: p x i q “ 1 ´ M ¨ | x i

Original: ´ y i | ě 1 ´ 12 s and hence
Revised: ´ y i | ě 1 ´ 12 s and hence

Original: [Equation]
Revised: [Equation]

Original: so that ϑ p s q M,y p x q ě 12 .
Revised: so that ϑ p s q M,y p x q ě 12 .

Original: Finally, it is not difﬁcult to show (see (Grohs & Voigtlaender, 2021, Lemma A.2) for the details) that
Revised: Finally, it is not difﬁcult to show, that

Original: [Equation]
Revised: [Equation]

Revised: see Grohs & Voigtlaender (2021, Equation (A.1)) for the details.

Original: Overall, we thus see
Revised: Overall, we thus see

Original: [Equation]
Revised: [Equation]

Original: Note that a compactly supported (non-trivial) function such as ϑ p s q M,y can only be represented by
Revised: Note that a compactly supported (non-trivial) function such as ϑ p s q M,y can only be represented by

Original: ReLU networks with more than two layers, see Blum & Li (1991, Section 3).
Revised: ReLU networks with more than two layers, see Blum & Li (1991, Section 3).

Original: For this reason, we focus on the case L P N ě 3 in this paper.
Revised: For this reason, we focus on the case L P N ě 3 in this paper.

Original: Next, we show that scaled versions of the hat functions ϑ p s q M,y can be represented using neural networks of a suitable architecture and with a suitable bound on the magnitude of the coefﬁcients.
Revised: Next, we show that scaled versions of the hat functions ϑ p s q M,y can be represented using neural networks of a suitable architecture and with a suitable bound on the magnitude of the coefﬁcients.

Original: We begin with the (more interesting) case where the exponent q that determines the regularization of the weights satisﬁes q ě 2 .
Revised: We begin with the (more interesting) case where the exponent q that determines the regularization of the weights satisﬁes q ě 2 .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let d P N , L P N ě 3 , B P N ě 3 , c ą 0 , q P r 2 , 8s , and s P N with s ď min t B 3 , d u .
Revised: Let d P N , L P N ě 3 , B P N ě 3 , c ą 0 , q P r 2 , 8s , and s P N with s ď min t B 3 , d u .

Original: Then, there exists a constant
Revised: Then, there exists a constant

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: where the B in p d, B, . . .
Revised: where the B in p d, B, . . .

Original: , B, 1 q appears L ´ 1 times.
Revised: , B, 1 q appears L ´ 1 times.

Original: Proof.
Revised: Proof.

Original: Let M P N , y P r 0 , 1 s d , and ν P t˘ 1 u be ﬁxed.
Revised: Let M P N , y P r 0 , 1 s d , and ν P t˘ 1 u be ﬁxed.

Original: We will now construct the coefﬁcients pp W 1 , b 1 q , . . .
Revised: We will now construct the coefﬁcients pp W 1 , b 1 q , . . .

Original: , p W L , b L qq of a neural network with the following properties:
Revised: , p W L , b L qq of a neural network with the following properties:

Original: The ﬁrst two layers pp W 1 , b 1 q , p W 2 , b 2 qq output at any of their B output dimensions the function C 1 ¨ Λ p s q M,y for a suitable scaling factor C 1 “ C 1 p c, M, s, B, q q ą 0 .
Revised: The ﬁrst two layers pp W 1 , b 1 q , p W 2 , b 2 qq output at any of their B output dimensions the function C 1 ¨ Λ p s q M,y for a suitable scaling factor C 1 “ C 1 p c, M, s, B, q q ą 0 .

Original: The following activation function yields C 1 ¨ ϑ p s q M,y “ (cid:37) ` C 1 ¨ Λ p s q M,y ˘ for all output dimensions.
Revised: The following activation function yields C 1 ¨ ϑ p s q M,y “ (cid:37) ` C 1 ¨ Λ p s q M,y ˘ for all output dimensions.

Original: Each of the layers pp W 3 , b 3 q , . .
Revised: Each of the layers pp W 3 , b 3 q , . .

Original: , p W L ´ 1 , b L ´ 1 qq scales the previous output by another factor C 2 “ C 2 p c, B, q q ą 0 , leading to the output C 1 C L ´
Revised: , p W L ´ 1 , b L ´ 1 qq scales the previous output by another factor C 2 “ C 2 p c, B, q q ą 0 , leading to the output C 1 C L ´

Original: 3¨ ϑ p s q M,y in any of the B output dimensions.
Revised: 3¨ ϑ p s q M,y in any of the B output dimensions.

Original: This construction uses the fact that all intermediate outputs are positive by construction such that 
Revised: This construction uses the fact that all intermediate outputs are positive by construction such that the

Original: the intermediate ReLU activationfunctions
Revised: intermediate ReLU activation

Original:  (cid:37) just act as identities
Revised: functions (cid:37) just act as identities.

Original: The last layer p W L , b L q now computes the sum of the previous outputs scaled by another factor C 3 “ C 3 p c, B, q q ą 0 and multiplied by ν , such that the ﬁnal one-dimensional output equals νBC 1 C L ´ 3C{+
 C+} 3 ¨ ϑ p s q M,y .
Revised: The last layer p W L , b L q now computes the sum of the previous outputs scaled by another factor C 3 “ C 3 p c, B, q q ą 0 and multiplied by ν , such that the ﬁnal one-dimensional output equals νBC 1 C L ´ {+
 C+} 3 ¨ ϑ p s q M,y .

Original: The result follows by setting λ “ BC 1 C L ´ 3C{+
 C+} 3 Ms and choosing the scaling factors C 1 , C 2 , and C 3 as large as possible, constrained by the width B and the regularization given by c and q .
Revised: The result follows by setting λ “ BC 1 C L ´ {+
 C+} 3 Ms and choosing the scaling factors C 1 , C 2 , and C 3 as large as possible, constrained by the width B and the regularization given by c and q .

Original: Deﬁne r : “ t B {p 3 s q u , noting that r ě 1 , since s ď B { 3 .
Revised: Deﬁne r : “ t B {p 3 s q u , noting that r ě 1 , since s ď B { 3 .

Original: We ﬁrst introduce a few notations: We write 0 k ˆ n for the k ˆ n matrix with all entries being zero; similarly, we write 1 k ˆ n for the k ˆ n matrix with all entries being one.
Revised: We ﬁrst introduce a few notations: We write 0 k ˆ n for the k ˆ n matrix with all entries being zero; similarly, we write 1 k ˆ n for the k ˆ n matrix with all entries being one.

Original: Furthermore, we denote by p e 1 , . . .
Revised: Furthermore, we denote by p e 1 , . . .

Original: , e d q the standard basis of R d , and deﬁne
Revised: , e d q the standard basis of R d , and deﬁne

Original: [Equation]
Revised: [Equation]

Original: We note that all entries of these matrices and vectors are elements of r´ 1 , 1 s .
Revised: We note that all entries of these matrices and vectors are elements of r´ 1 , 1 s .

Original: Using these matrices and vectors, we now deﬁne
Revised: Using these matrices and vectors, we now deﬁne

Original: [Equation]
Revised: [Equation]

Original: and furthermore
Revised: and furthermore

Original: [Equation]
Revised: [Equation]

Revised: s |´ 1 B ˆ s |´ 1 B ˆ s q | 0 B ˆp B ´ 3 rs q ¯

Revised: P R B ˆ B ,

Revised: [Equation]

Original: where we note that B ´ 3 rs ě 0 since r “ t B {p 3 s q u .
Revised: where we note that B ´ 3 rs ě 0 since r “ t B {p 3 s q u .

Original: It is straightforward to verify that } W 1 } (cid:96) q , } W 2 } (cid:96) q , } b 1 } (cid:96) q , } b 2 } (cid:96) q ď c .
Revised: It is straightforward to verify that } W 1 } (cid:96) q , } W 2 } (cid:96) q , } b 1 } (cid:96) q , } b 2 } (cid:96) q ď c .

Original: Furthermore, we deﬁne
Revised: Furthermore, we deﬁne

Original: W i : “ c
Revised: W i : “ c

Original: B 2 { q 1 B ˆ B and b i : “ p 0 | ¨ ¨ ¨ | 0 q T P R B for 3 ď i ď L ´ 1 , and ﬁnally W L : “ ν ¨ c B 1 { q p 1 | ¨ ¨ ¨ | 1 q P R 1 ˆ B and b L : “ p 0 q P R 1 .
Revised: B 2 { q 1 B ˆ B and b i : “ p 0 | ¨ ¨ ¨ | 0 q T P R B for 3 ď i ď L ´ 1 , and ﬁnally W L : “ ν ¨ c B 1 { q p 1 | ¨ ¨ ¨ | 1 q P R 1 ˆ B and b L : “ p 0 q P R 1 .

Original: Again, it is straightforward to verify that } W i } (cid:96) q , } b i } (cid:96) q ď c for 3 ď i ď L ´ 1 and also that } W L } (cid:96) q , } b L } (cid:96) q ď c .
Revised: Again, it is straightforward to verify that } W i } (cid:96) q , } b i } (cid:96) q ď c for 3 ď i ď L ´ 1 and also that } W L } (cid:96) q , } b L } (cid:96) q ď c .

Original: Therefore, setting Φ : “ pp W 1 , b 1 q , . .
Revised: Therefore, setting Φ : “ pp W 1 , b 1 q , . .

Original: , p W L , b L qq , we have R p Φ q P H q p d,B,...,B, 1 q ,c ; it thus remains to verify that R p Φ q “ ν ¨ λMs ¨ ϑ p s q M,y for a constant λ as in the statement of the lemma.
Revised: , p W L , b L qq , we have R p Φ q P H q p d,B,...,B, 1 q ,c ; it thus remains to verify that R p Φ q “ ν ¨ λMs ¨ ϑ p s q M,y for a constant λ as in the statement of the lemma.

Original: To see this, weﬁrst note for any  j P r d s that
Revised: To see this, we note for any x P R d and j P r d s that

Original: [Equation]
Revised: [Equation]

Revised: For notational convenience we further deﬁne φ i p x q : “ (cid:37) p W i x ` b i q for i P r L s .

Original: Therefore, we see for arbitrary i P r B sand x P R d that
Revised: Then, we observe for x P R B and i P r B s that

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Table] [Equation]
Revised: Therefore, we see for arbitrary x P R d

Original: and hence (cid:37)
Revised: and i P r B s that

Original: [Equation]
Revised: [Equation]

Original: Next [Equation] for 3 ď i ď L ´ 1 and φ i : R B Ñ R B , x ÞÑ (cid:37) p W i x ` b i q
Revised: Hence, it holds that

Original: [Equation]
Revised: [Equation]

Original:  we see for arbitrary κ ě 0 and [Equation] P r B s that “
Revised: Next, for 3 ď i ď L ´ 1 , we see for arbitrary κ ě 0 and j P r B s that “`

Original: [Equation]
Revised: [Equation]

Original: meaning
Revised: meaning

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] we conclude
Revised: Therefore, we conclude

Original: [Equation]
Revised: [Equation]

Original: All in all [Equation] this easily implies
Revised: All in all, this easily implies

Original: [Equation]
Revised: [Equation]

Original: It therefore remains to recall that r “ t B {p 3 s q u ě 1 , so that 2 r ě 1 ` r ą B 3 s and hence 3 rs ě B 2 .
Revised: It therefore remains to recall that r “ t B {p 3 s q u ě 1 , so that 2 r ě 1 ` r ą B 3 s and hence 3 rs ě B 2 .

Original: Since also 1 ´ 2 q ě 0 , this implies p 3 rs q 1 ´ 2 q ě p B { 2 q 1 ´
Revised: Since also 1 ´ 2 q ě 0 , this implies p 3 rs q 1 ´ 2 q ě p B { 2 q 1 ´

Original: [Equation]
Revised: [Equation]

Original: q ě B 1 ´ q 2 { 2 , which ﬁnally shows
Revised: q ě B 1 ´ q 2 { 2 , which ﬁnally shows

Original: [Equation]
Revised: [Equation]

Original: Now, we also consider the case q ď 2 .
Revised: Now, we also consider the case q ď 2 .

Original: We remark that in the case q “ 2 , the next lemma only agrees with Lemma A.2 up to a constant factor.
Revised: We remark that in the case q “ 2 , the next lemma only agrees with Lemma A.2 up to a constant factor.

Original: This is a proof artifact and is inconsequential for the questions we are interested in.
Revised: This is a proof artifact and is inconsequential for the questions we are interested in.

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Let d P N , L P N ě 3 , B P N ě 3 , c ą 0 ,
Revised: Let d P N , L P N ě 3 , B P N ě 3 , c ą 0 ,

Original: q P r 1 , 2 s , and s P N with s ď min t d, B 3 u .
Revised: q P r 1 , 2 s , and s P N with s ď min t d, B 3 u .

Original: Then, we have
Revised: Then, we have

Original: [Equation]

Original: u ,

Original: [Equation]

Original: y P r 0 , 1 s

Original: [Equation]
Revised: [Equation]

Original: where the B in p d, B, . . .
Revised: where the B in p d, B, . . .

Original: , B, 1 q appears L ´ 1 times.
Revised: , B, 1 q appears L ´ 1 times.

Original: Proof of Lemma A.3.
Revised: Proof.

Original: The proof idea is similar to the one of Lemma A.2.
Revised: The proof idea is similar to the one of Lemma A.2.

Original: However, we only realize a scaled version of the function ϑ p s q M,y in the ﬁrst coordinate of the outputs after the ﬁrst two layers.
Revised: However, we only realize a scaled version of the function ϑ p s q M,y in the ﬁrst coordinate of the outputs after the ﬁrst two layers.

Original: As in the proof of Lemma A.2, we denote by p e 1 , . . .
Revised: As in the proof of Lemma A.2, we denote by p e 1 , . . .

Original: , e d q the standard basis of R d , and we write 0 k ˆ n and 1 k ˆ n for the k ˆ n matrices which have all entries equal to zero or one, respectively.
Revised: , e d q the standard basis of R d , and we write 0 k ˆ n and 1 k ˆ n for the k ˆ n matrices which have all entries equal to zero or one, respectively.

Original: Moreover, we use the matrices and vectors I s , α, β, γ deﬁned in Equation (8).
Revised: Moreover, we use the matrices and vectors I s , α, β, γ deﬁned in Equation (5).

Original: With this setup, deﬁne
Revised: With this setup, deﬁne

Original: [Equation]
Revised: [Equation]

Original: Note that these deﬁnitions make sense since 2 s ď 3 s ď B .
Revised: Note that these deﬁnitions make sense since 2 s ď 3 s ď B .

Original: Further, deﬁne b 2 : “ p 0 | ¨ ¨ ¨ | 0 q T P R B and
Revised: Further, deﬁne b 2 : “ p 0 | ¨ ¨ ¨ | 0 q T P R B and

Original: W 2 : “ c 
Revised: W 2 : “ c p

Original: [Equation]
Revised: [Equation]

Revised: q

Original: [Equation]
Revised: [Equation]

Revised: { q ˆ

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: ˆp

Original: [Equation]
Revised: [Equation]

Original: qp B
Revised: p

Original: [Equation]
Revised: [Equation]

Revised: 0 p

Original: [Equation]
Revised: [Equation]

Original: 0 p
Revised: 0 p

Original: [Equation]
Revised: [Equation]

Original: qˆpB
Revised: qˆp

Original: [Equation]
Revised: [Equation]

Original:  ˙
Revised: q ˙

Original: P R B ˆ B .
Revised: P R B ˆ B .

Original: Next, for 3 ď i ď L ´ 1 , deﬁne b i : “ p 0 | ¨ ¨ ¨ | 0 q T P R B and
Revised: Next, for 3 ď i ď L ´ 1 , deﬁne b i : “ p 0 | ¨ ¨ ¨ | 0 q T P R B and

Original: W i : “ c ¨ ¨ ˚˚˝ 1 0 ¨ ¨ ¨ 0 0 ¨ ¨ ¨ 0 ...
Revised: W i : “ c ¨ ¨ ˚˚˝ 1 0 ¨ ¨ ¨ 0 0 ¨ ¨ ¨ 0 ...

Original: 0 0 ¨ ¨ ¨ ˛ ‹‹‚ P R B ˆ B , and ﬁnally let W L : “ ν ¨ c ¨ p 1 | 0 | ¨ ¨ ¨ | 0{+
+} q P R 1 ˆ B and b L : “ p 0 q P R 1 .
Revised: 0 0 ¨ ¨ ¨ ˛ [Equation] and ﬁnally let W L : “ ν ¨ c ¨ p 1 | 0 | ¨ ¨ ¨ | {+
+} q P R 1 ˆ B and b L : “ p 0 q P R 1 .

Original: It is straightforward to verify that } W j } (cid:96) q ď c and } b j } (cid:96) q ď c for all 1 ď j ď L .
Revised: It is straightforward to verify that } W j } (cid:96) q ď c and } b j } (cid:96) q ď c for all 1 ď j ď L .

Original: Therefore, R p Φ q P H q p d,B,...,B, 1 q ,c for Φ : “ pp W 1 , b 1 q , . . .
Revised: Therefore, R p Φ q P H q p d,B,...,B, 1 q ,c for Φ : “ pp W 1 , b 1 q , . . .

Original: , p W L , b L qq .
Revised: , p W L , b L qq .

Original: It therefore remains to show that R p Φ q “ ν
Revised: It therefore remains to show that R p Φ q “ ν

Original: [Equation]
Revised: [Equation]

Original: c L s
Revised: c L s

Original: [Equation]
Revised: [Equation]

Original: 2 q {
Revised: 2 q {

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: Ms ϑ

Original: [Equation]
Revised: [Equation]

Original: ϑ
Revised: q M

Original: [Equation]
Revised: [Equation]

Original: To see this, ﬁrst note for 3 ď i ď L ´ 1 and φ i : R B Ñ R B , φ i p x q  “ (cid:37) p W i x ` b i q that φ i p
Revised: For notational convenience we deﬁne φ i p x q : “ (cid:37) p W i x ` b i q for i P r L s . Then we note for 3 ď i ď L ´ 1 that

Original: [Equation]
Revised: [Equation]

Original: q “ ` c ¨ (cid:37) p x 1 q | 0 | ¨ ¨ ¨ |˘ T .

Original: This easily implies
Revised: This easily implies

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: Finally, an application of Equation (10) shows that 
Revised: Finally, an application of Equation (6) shows that [Equation]

Revised: ˘˙˙

Original: [Equation]
Revised: [Equation]

Original: Overall, we thus see as claimed that
Revised: Overall, we thus see as claimed that

Original: [Equation]
Revised: [Equation]

Original: Remark A.4.
Revised: Remark A.4.

Original: A straightforward adaptation of the proof shows that the same statement holds for
Revised: A straightforward adaptation of the proof shows that the same statement holds for

Original: H q p d,B,B 2 ,...,B L ´ 1 , 1 q ,c instead of H q p d,B,...,B, 1 q ,c , for arbitrary B 2 , . ..
Revised: H q p d,B,N 2 ,...,N L ´ 1 , 1 q ,c instead of H q p d,B,...,B, 1 q ,c , for arbitrary N 2 , . .

Original:  , B L ´ 1 P N .
Revised: . , N L ´ 1 P N .

Original: A. A GENERAL LOWER BOUND
Revised: A. A GENERAL LOWER BOUND

Original: We now show that any target class containing a large number of (shifted) hat functions has a large optimal error.
Revised: We now show that any target class containing a large number of (shifted) hat functions has a large optimal error.

Original: Theorem A.5.
Revised: Theorem A.5.

Original: Let d, m P N , s P r d s , and M : “ 8 r m 1 { s s .
Revised: Let d, m P N , s P r d s , and M : “ 8 r m 1 { s s .

Original: Assume that U Ă C pr 0 , 1 s d q satisﬁes
Revised: Assume that U Ă C pr 0 , 1 s d q satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  for certain λ ą 0 and u 0 P C pr 0 , 1 s d q .
Revised: ν P t˘ 1 u and y P r 0 , 1 s d for certain λ ą 0 and u 0 P C pr 0 , 1 s d q .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: The general idea of the proof is sketched in Section 2.3.
Revised: The general idea of the proof is sketched in Section 2.3.

Original: In what follows we provide the technical details.
Revised: In what follows we provide the technical details.

Original: Proof of Theorem A.5.
Revised: Proof.

Original: The proof is divided into ﬁve steps.
Revised: The proof is divided into ﬁve steps.

Original: Step 1: Deﬁne k : “ r m 1 { s s and let y (cid:96) : “ p 1 ,..., 1 q 8 k ` (cid:96) [Equation] [Equation]
Revised: Step 1: Deﬁne k : “ r m 1 { s s and let y (cid:96) : “ p 1 ,..., 1 q 8 k ` (cid:96) ´p 1 ,..., 1 q 4 k

Original: P 
Revised: P r 0 , 1 s d for (cid:96) P r 4 k s d . Furthermore, let Γ : “ r 4 k s s ˆ tp 1 , . . , 1 qu Ă r 4 k s d . With f (cid:96),

Original: [Equation]
Revised: [Equation]

Original: 0 , 1 s d for (cid:96) P r 4 k s d .

Original: Furthermore, let Γ m : “pr 4 k s s ˆ t 0 uq ˆ t˘ 1 u Ă R d ˆ t˘ 1 u . With
Revised: : “ u [Equation] ` ν ¨ λ Msϑ p s q M,

Original: [Equation]
Revised: [Equation]

Revised: (cid:96) for p (cid:96), ν q P

Original: [Equation]
Revised: [Equation]

Original:  it holds by assumption that
Revised: 7) it holds by assumption that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, since M “ 8 k , Lemma A.1 and a moment’s thought reveal that
Revised: Furthermore, since M “ 8 k , Lemma A.1 and a moment’s thought reveal that

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where we notefor later use that supp p f (cid:96),ν ´ u 0 q “ supp ϑ p s q M,y (cid:96) .
Revised: where we note that supp p f (cid:96),ν ´ u 0 q “ supp ϑ p s q M,y (cid:96) .

Original: Step 2: Let 5 A P Alg 2 m p U, L p q be arbitrary and x “ x p u 0 q “ p x 1 , . . .
Revised: Step 2: Let 5 A P Alg 2 m p U, L p q be arbitrary and x “ x p u 0 q “ p x 1 , . . .

Original: , x
Revised: , x

Original: 2 m q P

Original:  ` r [Equation] [Equation] 2 m as described before Equation (1).
Revised: 2 m q P ` r 0 , 1 s d ˘ 2 m as described before Equation (1).

Original: Put
Revised: Put

Revised: I x : “

Original: [Equation]
Revised: [Equation]

Original: We now show that
Revised: We now show that

Original: [Equation]
Revised: [Equation]

Original: To see this we will estimate the cardinality of the complement set
Revised: To see this we will estimate the cardinality of the complement set

Original: [Equation] uq [Equation] x from above.
Revised: I c x : “ Γ z I x from above.

Original: For (cid:96) P I c x there must exist i (cid:96) P r 2 m s with ϑ p s q M,y (cid:96) p x i (cid:96) q ‰ 0 and hence x i (cid:96) Pr supp  ϑ p s q M,y (cid:96) s o .
Revised: For (cid:96) P I c x there must exist i (cid:96) P r 2 m s with ϑ p s q M,y (cid:96) p x i (cid:96) q ‰ 0 and hence x i (cid:96) P supp ` ϑ p s q M,y (cid:96) ˘ o .

Original: The map I c x Ñ r 2 m s , (cid:96) ÞÑ i (cid:96)  is thus injective due to (14).
Revised: The map I c x Ñ r 2 m s , (cid:96) ÞÑ i (cid:96) , is thus injective due to (9).

Original: Therefore | I c x | ď 2 m and thus | I x | ě p 4 k q s ´ 2 m , which is (16).
Revised: Therefore | I c x | ď 2 m and thus | I x | ě | Γ | ´ 2 m , which is (10).

Original: Furthermore, the deﬁnition of I x , combined with the deﬁnition(12) of f (cid:96),ν , and the condition that A can only depend on the samples x and the values of the input function at these samples, directly imply that
Revised: Furthermore, the deﬁnition of I x , combined with the deﬁnition of f (cid:96),ν in (7) and the condition that A can only depend on the samples x and the values of the input function at these samples, directly imply that

Original: [Equation]
Revised: [Equation]

Original: Step 3:
Revised: Step 3:

Original: Recalling our notation for the average in Section 1.2, it holds thatÿ
Revised: Recalling our notation for the average in Section 1.2, it holds that

Original: [Equation]
Revised: [Equation]

Original: Here, (19) follows from the deﬁnition of Γ m ; (20) follows since I x Ă r 4 k s s ˆ t 0 u ; (22) follows from k “ r m 1 s s and (16); (23) follows from (17); (24) follows from the triangle inequality and (12); (25) follows from Lemma A.1; and (26) follows from the deﬁnition of M , which implies that 
Revised: Here, (12) follows since I x Ă Γ ; (13) follows from k “ r m 1 s s and (10); (14) follows from (11); (15) follows from the triangle inequality and (7); (16) follows from Lemma A.1; and (17) follows from the deﬁnition of M , which implies that M ď

Original: [Equation]
Revised: 8 m 1 { s ` 8 ď 16 m 1 { s .

Original: Step 4: Let p A , m q P Alg MCm p U, L p q be arbitrary with A “ p A ω q ω P Ω for a probability space p Ω , F , P q .
Revised: Step 4: Let p A , m q P Alg MCm p U, L p q be arbitrary with A “ p A ω q ω P Ω for a probability space p Ω , F , P q .

Original: Put Ω 0 : “ t ω P Ω : m p ω q ď 2 m u .
Revised: Put Ω 0 : “ t ω P Ω : m p ω q ď 2 m u .

Original: Since the Markov inequality implies that
Revised: Since the Markov inequality implies that

Original: [Equation]
Revised: [Equation]

Original: it follows that
Revised: it follows that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Step 5: We ﬁnally estimate for p A , m q as in Step 4 that
Revised: Step 5: We ﬁnally estimate for p A , m q as in Step 4 that

Original: [Equation]
Revised: [Equation]

Original: Here, (29) follows from (13); (32) follows from Step 3 (note that A ω P Alg 2 m p U, L p q for ω P Ω 0 ); and (33) follows from (27).
Revised: Here, (19) follows from (8); (20) follows from Step 3 (note that A ω P Alg 2 m p U, L p q for ω P Ω 0 ); and (21) follows from (18).

Original: Since p A , m q P Alg MCm p U, L p q was arbitrary, this implies the desired statement.A.
Revised: Since p A , m q P Alg MC m p U, L p q was arbitrary, this implies the desired statement.

Original: C OMPLETING THE PROOF OF T HEOREM 2.

Original: Theorem 2.2 follows by combining Theorem A.5 with Lemmas A.2 and A.3.

Original: Remark A.6.
Revised: Remark A.6.

Original: Close inspection of the proof of Theorem 2.2 shows that one can replace the point samples u p x i q by Tu p x
Revised: Close inspection of the proof of Theorem 2.2 shows that one can replace the point samples u p x i q by Tu p x

Original: i q , where T : U Ñ C pr 0 , 1 s d q is any local operator 6 .
Revised: i q , where T : U Ñ C pr 0 , 1 s d q is any local operator 6 .

Original: Since any differential operator is a local operator, our lower bounds also hold if we measure point samples of a differential operator applied to u , as it is commonly done in the context of so-called physics-informed neural networks (Raissi et al., 2019).
Revised: Since any differential operator is a local operator, our lower bounds also hold if we measure point samples of a differential operator applied to u , as it is commonly done in the context of so-called physics-informed neural networks (Raissi et al., 2019).

Original: B P ROOF OF THE UPPER BOUND IN S ECTION 2. 
Revised: B P ROOF OF THE UPPER BOUND IN S ECTION 2.

Original: We ﬁrst provide an auxiliary result which bounds the spectral norm } W } (cid:96) 2 Ñ (cid:96) 2 of a matrix W by its entry-wise (cid:96) q -norm.
Revised: We ﬁrst provide an auxiliary result which bounds the spectral norm } W } (cid:96) 2 Ñ (cid:96) 2 of a matrix W by its entry-wise (cid:96) q norm.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Let W P R N ˆ M and q P r 1 , 8s .
Revised: Let W P R N ˆ M and q P r 1 , 8s .

Original: Then it holds that
Revised: Then it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst note that } W } (cid:96) 2 “ } W } F , the Frobenius norm of the matrix W .
Revised: We ﬁrst note that } W } (cid:96) 2 “ } W } F , the Frobenius norm of the matrix W .

Original: It is well-known that the Frobenius norm satisﬁes } W } (cid:96) 2 Ñ (cid:96) 2 ď } W } F .
Revised: It is well-known that the Frobenius norm satisﬁes } W } (cid:96) 2 Ñ (cid:96) 2 ď } W } F .

Original: Since we could not locate a convenient reference, we reproduce the elementary proof: The Cauchy-Schwarz inequality implies that
Revised: Since we could not locate a convenient reference, we reproduce the elementary proof: The Cauchy-Schwarz inequality implies that

Original: [Equation]
Revised: [Equation]

Original: which implies the claim.
Revised: which implies the claim.

Original: Thus, we see for q ď 2 that } W } (cid:96) 2 Ñ (cid:96) 2 ď } W } (cid:96) 2 ď } W } (cid:96) q .
Revised: Thus, we see for q ď 2 that } W } (cid:96) 2 Ñ (cid:96) 2 ď } W } (cid:96) 2 ď } W } (cid:96) q .

Original: Clearly, the same estimate holds for complex-valued matrices and vectors as well.
Revised: Clearly, the same estimate holds for complex-valued matrices and vectors as well.

Original: Now, to handle the case q ě 2 , we ﬁrst note for q “ 8 and W P C N ˆ M and x P C M that
Revised: Now, to handle the case q ě 2 , we ﬁrst note for q “ 8 and W P C N ˆ M and x P C M that

Original: [Equation]
Revised: [Equation]

Original: This proves the claim in case of q “ 8 .
Revised: This proves the claim in case of q “ 8 .

Original: Finally, for q P p 2 , 8q , we choose θ “ 2 q , so that 1 q “ θ 2 ` 1 ´ θ 8 .
Revised: Finally, for q P p 2 , 8q , we choose θ “ 2 q , so that 1 q “ θ 2 ` 1 ´ θ 8 .

Original: Thus, applying the Riesz-Thorin interpolation theorem (see, e.g., (Folland, 1999, Theorem 6.27)) to the linear map p C N ˆ M , } ¨ } (cid:96) q q Ñ p C N , } ¨ } (cid:96) 2 q , W ÞÑ
Revised: Thus, applying the Riesz-Thorin interpolation theorem (see, e.g., Folland, 1999, Theorem 6.27) to the linear map p C N ˆ M , } ¨ } (cid:96) q q Ñ p C N , } ¨ } [Equation]

Original:  Wx  shows for each x P C M that
Revised: q , W ÞÑ Wx , shows for each x P C M that

Original: [Equation]
Revised: [Equation]

Original: which completes the proof 7 .
Revised: which completes the proof 7 .

Original: Next, let us deﬁne the Lipschitz constant Lip (cid:96) q p f q of a function f : R d Ñ R k with respect to the (cid:96) 2 -norm by
Revised: Next, let us deﬁne the Lipschitz constant Lip (cid:96) q p φ q of a function φ : R d Ñ R k with respect to the (cid:96)norm by

Original: [Equation]
Revised: [Equation]

Original: Note that the Lipschitz constant of an afﬁne-linear mapping x ÞÑ
Revised: Note that the Lipschitz constant of an afﬁne-linear mapping x ÞÑ

Original: Wx ` b equals the spectral norm } W } (cid:96) 2 Ñ (cid:96) 2 .
Revised: Wx ` b equals the spectral norm } W } (cid:96) 2 Ñ (cid:96) 2 .

Original: Thus, we can use the previous lemma to bound the Lipschitz constant of neural network realizations R p
Revised: Thus, we can use the previous lemma to bound the Lipschitz constant of neural network realizations R p

Original: Φ q P H q p N 0 ,...,N L q ,c in terms of their architecture p N 0 , . . .
Revised: Φ q P H q p N 0 ,...,N L q ,c in terms of their architecture p N 0 , . . .

Original: , N L q and  regularization on their weights (given by max 1 ď i ď L max t} W i } (cid:96) q , } b i } (cid:96) q u ď c ).
Revised: , N L q and the regularization on their weights (given by max 1 ď i ď L max t} W i } (cid:96) q , } b i } (cid:96) q u ď c ).

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let L P N , q P r 1 , 8s , c ą 0 , and N 0 , . .
Revised: Let L P N , q P r 1 , 8s , c ą 0 , and N 0 , . .

Original: , N L P N .
Revised: , N L P N .

Original: Then, each R p Φ q P H q p N 0 ,...,N L q ,c satisﬁes
Revised: Then, each R p Φ q P H q p N 0 ,...,N L q ,c satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma B.2.
Revised: Proof.

Original: Let R p Φ q P H q p N 0 ,...,N L q ,c be arbitrary.
Revised: Let R p Φ q P H q p N 0 ,...,N L q ,c be arbitrary.

Original: By deﬁnition, this means
Revised: By deﬁnition, this means

Original: [Equation]
Revised: [Equation]

Original: where (cid:37): R N Ñ R N acts componentwise, and where the afﬁne-linear maps T i : R N i
Revised: where (cid:37) acts componentwise, and where the afﬁne-linear maps φ i : R N i

Original: ´ 1 Ñ R N i are of the formT
Revised: ´ 1 Ñ R N i are of the form

Original:  i  x  “ W i x ` b i , with W i P
Revised: φ i p x q “ W i x ` b i , with W i P

Original: R N i ˆ N i ´ 1 and } W i } (cid:96) q ď c .
Revised: R N i ˆ N i ´ 1 and } W i } (cid:96) q ď c .

Revised: The ReLU activation function (cid:37) : R Ñ R , x ÞÑ max t 0 , x u , is easily seen to satisfy | (cid:37) p x q ´ (cid:37) p

Revised: y q| ď | x ´ y | for x,

Original: Lemma B.1 implies that
Revised: y P R . This implies that

Original: [Equation]
Revised: [Equation]

Original: Next, we note as a direct consequence of the deﬁnition that
Revised: Lemma B.1 establishes for i P r L s that

Original: [Equation]Moreover, the ReLU activation function (cid:37) : R Ñ R , x ÞÑ max t 0 , x u is easily seen to satisfy | (cid:37) p x q ´ (cid:37) p y q| ď | x ´ y | for x, y P R , and this implies } (cid:37) p x q ´ (cid:37) p y q} (cid:96) 2 ď } x ´
Revised: [Equation]

Original: y } (cid:96) 2 for x, y P R N . Combining all these observations easily implies the claim.
Revised: which, together with (22), proves the claim.

Original: Note that we can estimate the error of reconstructing Lipschitz continuous functions from samples by piecewise constant interpolation.
Revised: Note that we can estimate the error of reconstructing Lipschitz continuous functions from samples by piecewise constant interpolation.

Original: Together with Lemma B.2, this allows us to construct a (nonadaptive, deterministic) algorithm for reconstructing neural networks from samples.
Revised: Together with Lemma B.2, this allows us to construct a (nonadaptive, deterministic) algorithm for reconstructing neural networks from samples.

Original: Lemma B.3.
Revised: Lemma B.3.

Original: Let d P N .
Revised: Let d P N .

Original: Then, for every m P N , there exist points x 1 , . . .
Revised: Then, for every m P N , there exist points x 1 , . . .

Original: , x m P r 0 , 1 s d and a map Θ m : R m Ñ L 8 pr 0 , 1 s d q satisfying 
Revised: , x m P r 0 , 1 s d and a map Θ m : R m Ñ L 8 pr 0 , 1 s d q satisfying ›

Original: [Equation]
Revised: [Equation]

Original: for every function u : r 0 , 1 s d Ñ R with Lip (cid:96) 2 p u q ă 8 .
Revised: for every function u : r 0 , 1 s d Ñ R with Lip (cid:96) 2 p u q ă 8 .

Original: Proof.
Revised: Proof.

Original: Let m P
Revised: Let m P

Original: N be arbitrary and choose K : “ t m 1 { d u ě 1 .
Revised: N be arbitrary and choose K : “ t m 1 { d u ě 1 .

Original: Write t x 1 , . .
Revised: Write t x 1 , . .

Original: , x K d u “ (cid:32) K , 1  K , . . .
Revised: , x K d u “ p 1 ,..., 1 q 2 K `

Revised: [Equation]

Original: , K ´ 1 K ( d noting that r 0 , 1 s d “
Revised: , K ´ 1 K ( d noting that r 0 , 1 s d “

Original: K d ď i “x i ` r 0 , 1 K s d .
Revised: K d ď i “x i ` r´ 12 K , 12 K s d .

Original: Hence, choosing Q i : “ p x i ` r 0 , 1  K s d qz Ť i ´ 1 j “ 1 p x j ` r 0 , 1  K s d q , we get r 0 , 1 s d “ Ţ K d i “ 1 Q i , where the union is disjoint.
Revised: Hence, choosing Q i : “ p x i ` r´ 1 2 K , 1 2 K s d qz Ť i ´ 1 j “ 1 p x j ` r´ 1 2 K , 1 2 K s d q , we get r 0 , 1 s d “ Ţ K d i “ 1 Q i , where the union is disjoint.

Original: Note that K d ď m and choose arbitrary points x K d [Equation]
Revised: Note that K d ď m and choose arbitrary points x K d ` 1 , . . .

Original: [Equation]
Revised: , x m P r 0 , 1 s d .

Original: Furthermore, deﬁne 
Revised: Furthermore, deﬁne Θ m :

Original: [Equation]
Revised: R m Ñ L 8 pr 0 , 1 s d q , p a 1 , . . . , a m q ÞÑ K d ÿ i “a i ¨ 1 Q i .

Original: To prove Equation (35), let u : r 0 , 1 s d Ñ R be arbitrary with Lip (cid:96) 2 p u q ă 8 .
Revised: To prove Equation (23), let u : r 0 , 1 s d Ñ R be arbitrary with Lip (cid:96) 2 p u q ă 8 .

Original: For arbitrary x P r 0 , 1 s d , there then exists a unique i P r K d s satisfying x P Q
Revised: For arbitrary x P r 0 , 1 s d , there then exists a unique i P r K d s satisfying x P Q

Original: i Ă x i ` r 0 , 1 K s d , and in particular } x ´ x i } (cid:96) 2 ď ?
Revised: i Ă x i ` r´ 12 K , 12 K s d , and in particular } x ´ x i } (cid:96) 2 ď ?

Original: d { K  .
Revised: d {p 2 K q .

Original: Therefore,ˇ
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Since x P r 0 , 1 s d was arbitrary, this implies
Revised: Since x P r 0 , 1 s d was arbitrary, this implies

Original: [Equation]
Revised: [Equation]

Original: d K . It remains to note because of K “ t m 1 { d u ě 1 that 2 K ě 1 ` K ą m 1 { d , whence 1 K ď 2 ¨ m ´ 1 { d .
Revised: Finally, we note that K “ t m 1 { d u ě 1 implies 2 K ě 1 ` K ą m 1 { d , which proves the claim.

Original: Note that the proof above requires to convert a Lipschitz constant with respect to the (cid:96) 2 -norm to an (cid:96) 8 estimate which costs a factor ?
Revised: Note that the proof above requires to convert a Lipschitz constant with respect to the (cid:96) 2 norm to an (cid:96) 8 estimate which costs a factor ?

Original: d and contributes to the gap between our lower and upper bound.
Revised: d and contributes to the gap between our lower and upper bound.

Original: Remark B.4.
Revised: Remark B.4.

Original: Note that our upper and lower bounds in Theorems 1.1 and 1.4 are asymptotically sharp with respect to the number of samples m , the regularization parameter c , and the network depth L but not fully sharp with respect to the multiplicative factor depending on d, q only.
Revised: Note that our upper and lower bounds in Theorems 1.1 and 1.4 are asymptotically sharp with respect to the number of samples m , the regularization parameter c , and the network depth L but not fully sharp with respect to the multiplicative factor depending on d and q only.

Original: Given m many samples, a combination of Theorems 1.1 and 1.4 shows that the optimal achievable L 8 -reconstruction error ε for reconstructing neural networks with L layers up to width 3 d and coefﬁcients bounded by c in the (cid:96) q -norm satisﬁes #
Revised: Given m many samples, a combination of Theorems 1.1 and 1.4 shows that the optimal achievable L 8 reconstruction error ε for reconstructing neural networks with L layers up to width 3 d and coefﬁcients bounded by c in the (cid:96) q norm satisﬁes #

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: if q ď

Revised: [Table]

Original:  For moderate input dimensions d the upper and lower bounds are quite tight, but for larger d there remains a gap.
Revised: if q ą 2 . + For moderate input dimensions d the upper and lower bounds are quite tight, but for larger d there remains a gap.

Original: However, in that case the lower bound for m is already intractable (at least if ε !
Revised: However, in that case the lower bound for m is already intractable (at least if ε !

Original: 1 { d or if c " 1 and L is large) so that the upper bound is merely of academic interest.
Revised: 1 { d or if c " 1 and L is large) so that the upper bound is merely of academic interest.

Original: C H YPERPARAMETERS USED IN THE NUMERICAL EXPERIMENTS
Revised: C H YPERPARAMETERS USED IN THE NUMERICAL EXPERIMENTS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]