Original: T IGHTER
Revised: T IGHTER

Original: S PARSE
Revised: S PARSE

Original: A PPROXIMATION
Revised: A PPROXIMATION

Original: B OUNDS
Revised: B OUNDS

Original: FOR
Revised: FOR

Original: R E LU N EURAL N ETWORKS
Revised: R E LU N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski & Barron, 2018) provides bounds on the width n of a ReLU two-layer neural network needed to approximate a function f over the ball B R ( R d ) up to error (cid:15) , when the Fourier based quantity C f = (cid:82) R d (cid:107) ξ (cid:107) 2 | ˆ f ( ξ ) | dξ is ﬁnite.
Revised: A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski & Barron, 2018) provides bounds on the width n of a ReLU two-layer neural network needed to approximate a function f over the ball B R ( R d ) up to error (cid:15) , when the Fourier based quantity C f = (cid:82) R d (cid:107) ξ (cid:107) 2 | ˆ f ( ξ ) | dξ is ﬁnite.

Original: More recently Ongie et al.
Revised: More recently Ongie et al.

Original: (2019) used the Radon transform as a tool for analysis of inﬁnite-width ReLU two-layer networks.
Revised: (2019) used the Radon transform as a tool for analysis of inﬁnite-width ReLU two-layer networks.

Original: In particular, they introduce the concept of Radon-based R -norms and show that a function deﬁned on R d can be represented as an inﬁnite-width twolayer neural network if and only if its R -norm is ﬁnite.
Revised: In particular, they introduce the concept of Radon-based R -norms and show that a function deﬁned on R d can be represented as an inﬁnite-width twolayer neural network if and only if its R -norm is ﬁnite.

Original: In this work, we extend the framework of (Ongie et al., 2019) and deﬁne similar Radon-based semi-norms ( R , U -norms)
Revised: In this work, we extend the framework of (Ongie et al., 2019) and deﬁne similar Radon-based semi-norms ( R , U -norms)

Original: such that a function admits an inﬁnite-width neural network representation on a bounded open set U
Revised: such that a function admits an inﬁnite-width neural network representation on a bounded open set U

Original: ⊆ R d when its R , U -norm is ﬁnite.
Revised: ⊆ R d when its R , U -norm is ﬁnite.

Original: Building on this, we derive sparse (ﬁnite-width) neural network approximation bounds that reﬁne those of Breiman (1993); Klusowski & Barron (2018).
Revised: Building on this, we derive sparse (ﬁnite-width) neural network approximation bounds that reﬁne those of Breiman (1993); Klusowski & Barron (2018).

Original: Finally, we show that inﬁnite-width neural network representations on bounded open sets are not unique and study their structure, providing a functional view of mode connectivity.
Revised: Finally, we show that inﬁnite-width neural network representations on bounded open sets are not unique and study their structure, providing a functional view of mode connectivity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Extensive work has shown that for a neural network to be able to generalize, the size or magnitude of the parameters is more important than the size of the network, when the latter is large enough (Bartlett, 1997; Neyshabur et al., 2015; Zhang et al., 2016).
Revised: Extensive work has shown that for a neural network to be able to generalize, the size or magnitude of the parameters is more important than the size of the network, when the latter is large enough (Bartlett, 1997; Neyshabur et al., 2015; Zhang et al., 2016).

Original: Under certain regimes, the size of the neural networks used in practice is so large that the training data is ﬁt perfectly and an inﬁnite-width approximation is appropriate.
Revised: Under certain regimes, the size of the neural networks used in practice is so large that the training data is ﬁt perfectly and an inﬁnite-width approximation is appropriate.

Original: In this setting, what matters to obtain good generalization is to ﬁt the data using the right inductive bias, which is speciﬁed by how network parameters are controlled (Wei et al., 2020) together with the training algorithm used (Lyu & Li, 2020).
Revised: In this setting, what matters to obtain good generalization is to ﬁt the data using the right inductive bias, which is speciﬁed by how network parameters are controlled (Wei et al., 2020) together with the training algorithm used (Lyu & Li, 2020).

Original: The inﬁnite-width two-layer neural network model has been studied from several perspectives due to its simplicity.
Revised: The inﬁnite-width two-layer neural network model has been studied from several perspectives due to its simplicity.

Original: One can replace the ﬁnite-width ReLU network 1 n (cid:80) ni
Revised: One can replace the ﬁnite-width ReLU network 1 n (cid:80) ni

Original: =

Revised: =

Original: a i ( (cid:104) ω
Revised: a i ( (cid:104) ω

Original: i , x (cid:105) − b i ) + by an integral over the parameter space with respect to a signed Radon measure: (cid:82) ( (cid:104) ω, x (cid:105) − b )
Revised: i , x (cid:105) − b i ) + by an integral over the parameter space with respect to a signed Radon measure: (cid:82) ( (cid:104) ω, x (cid:105) − b )

Original: + dα ( ω, b ) .
Revised: + dα ( ω, b ) .

Original: Thus, controlling the magnitude of the neural network parameters is akin to controlling the measure α
Revised: Thus, controlling the magnitude of the neural network parameters is akin to controlling the measure α

Original: according to a certain norm.
Revised: according to a certain norm.

Original: Bach (2017) introduced the F 1 -space, which is the inﬁnitewidth neural network space with norm inf { (cid:82)
Revised: Bach (2017) introduced the F 1 -space, which is the inﬁnitewidth neural network space with norm inf { (cid:82)

Original: | b | d | α | ( ω, b ) } , derived from the ﬁnite-width regularizer 1 n (cid:80) ni
Revised: | b | d | α | ( ω, b ) } , derived from the ﬁnite-width regularizer 1 n (cid:80) ni

Original: =1 | a i |(cid:107) ( ω i , b i ) (cid:107) 

Revised: =1 | a i |(cid:107) ( ω i , b i ) (cid:107)

Original: (the inﬁmum is over all the measures α which represent the function at hand).
Revised: (the inﬁmum is over all the measures α which represent the function at hand).

Original: A different line of work (Savarese et al., 2019; Ongie et al., 2019) consider the inﬁnite-width spaces with norm inf {(cid:107) α (cid:107) TV =
Revised: A different line of work (Savarese et al., 2019; Ongie et al., 2019) consider the inﬁnite-width spaces with norm inf {(cid:107) α (cid:107) TV =

Original: (cid:82) d | α | ( ω, b ) } , which is derived from the ﬁnite-width regularizern (cid:80) ni
Revised: (cid:82) d | α | ( ω, b ) } , which is derived from the ﬁnite-width regularizern (cid:80) ni

Original: =1 | a i |(cid:107) ω
Revised: =1 | a i |(cid:107) ω

Original: i (cid:107) 2 (i.e. omitting the bias term).
Revised: i (cid:107) 2 (i.e. omitting the bias term).

Original: Both of these works seek to ﬁnd expressions for this norm, leading to characterizations of the functions that are representable by inﬁnite-width networks.
Revised: Both of these works seek to ﬁnd expressions for this norm, leading to characterizations of the functions that are representable by inﬁnite-width networks.

Original: Savarese et al.
Revised: Savarese et al.

Original: (2019) solves the problem in the one-dimensional case: they show that for a function f on R , this norm takes value max { (cid:82) R | f (cid:48)(cid:48)
Revised: (2019) solves the problem in the one-dimensional case: they show that for a function f on R , this norm takes value max { (cid:82) R | f (cid:48)(cid:48)

Original: ( x ) | dx, | f (cid:48) ( −∞ ) + f (cid:48) ( ∞ ) |} .
Revised: ( x ) | dx, | f (cid:48) ( −∞ ) + f (cid:48) ( ∞ ) |} .

Original: Ongie et al.
Revised: Ongie et al.

Original: (2019) give an expression for this norm (the R -norm ) for functions on R d , making use of Radon transforms (see
Revised: (2019) give an expression for this norm (the R -norm ) for functions on R d , making use of Radon transforms (see

Original: Subsec. 2.3).
Revised: Subsec. 2.3).

Original: Although we mentioned in the ﬁrst paragraph that in many occasions the network size is large enough that the speciﬁc number of neurons is irrelevant, when the target function is hard to approx- imate it is interesting to have an idea of how many neurons one needs to approximate it.
Revised: Although we mentioned in the ﬁrst paragraph that in many occasions the network size is large enough that the speciﬁc number of neurons is irrelevant, when the target function is hard to approx- imate it is interesting to have an idea of how many neurons one needs to approximate it.

Original: The ﬁrst contribution in this direction was by Cybenko (1989);
Revised: The ﬁrst contribution in this direction was by Cybenko (1989);

Original: Hornik et al.
Revised: Hornik et al.

Original: (1989), which show that twolayer neural networks with enough neurons can approximate any reasonable function on bounded sets in the uniform convergence topology.
Revised: (1989), which show that twolayer neural networks with enough neurons can approximate any reasonable function on bounded sets in the uniform convergence topology.

Original: Later on, Barron (1993); Breiman (1993) provided sparse approximation bounds stating that if a function f is such that a certain quantity C f constructed from the Fourier transform ˆ f is ﬁnite, then there exists a neural network of width n such that the L 2 approximation error with respect to a distribution of bounded support is lower than O ( C f /n ) .
Revised: Later on, Barron (1993); Breiman (1993) provided sparse approximation bounds stating that if a function f is such that a certain quantity C f constructed from the Fourier transform ˆ f is ﬁnite, then there exists a neural network of width n such that the L 2 approximation error with respect to a distribution of bounded support is lower than O ( C f /n ) .

Original: More recently, Klusowski & Barron (2018) provided alternative sparse approximation bounds of Breiman (1993) by restricting to networks with bounded weights and a slightly better dependency on n at the expense of a constant factor increasing with d (see Subsec. 2.2).
Revised: More recently, Klusowski & Barron (2018) provided alternative sparse approximation bounds of Breiman (1993) by restricting to networks with bounded weights and a slightly better dependency on n at the expense of a constant factor increasing with d (see Subsec. 2.2).

Original: Contributions .
Revised: Contributions .

Original: In our work, we seek to characterize the functions that coincide with an inﬁnitewidth two-layer neural network on a ﬁxed bounded open set.
Revised: In our work, we seek to characterize the functions that coincide with an inﬁnitewidth two-layer neural network on a ﬁxed bounded open set.

Original: This endeavor is interesting in itself because in practice, we want to learn target functions for which we know samples on a bounded set, and we are typically unconcerned with the values that the learned functions take at inﬁnity.
Revised: This endeavor is interesting in itself because in practice, we want to learn target functions for which we know samples on a bounded set, and we are typically unconcerned with the values that the learned functions take at inﬁnity.

Original: Moreover, the tools that we develop allow us to derive state-of-the-art sparse approximation bounds.
Revised: Moreover, the tools that we develop allow us to derive state-of-the-art sparse approximation bounds.

Original: Our main contributions are the following:
Revised: Our main contributions are the following:

Original: • In the spirit of the R -norm introduced by Ongie et al.
Revised: • In the spirit of the R -norm introduced by Ongie et al.

Original: (2019), for any bounded open set U
Revised: (2019), for any bounded open set U

Original: ⊆
Revised: ⊆

Original: R d we deﬁne the R , U -norm of a function on R d , and show that when the R , U -norm of f is ﬁnite, f ( x ) can admits a representation of the form (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b )
Revised: R d we deﬁne the R , U -norm of a function on R d , and show that when the R , U -norm of f is ﬁnite, f ( x ) can admits a representation of the form (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b )

Original: + dα ( ω, b ) + (cid:104) v, x (cid:105)
Revised: + dα ( ω, b ) + (cid:104) v, x (cid:105)

Original: + c for x ∈ U , where v ∈ R d , c ∈ R and α is an even signed Radon measure.
Revised: + c for x ∈ U , where v ∈ R d , c ∈ R and α is an even signed Radon measure.

Original: • Using the R , U -norm, we derive function approximation bounds for neural networks with a ﬁxed ﬁnite width.
Revised: • Using the R , U -norm, we derive function approximation bounds for neural networks with a ﬁxed ﬁnite width.

Original: We compute the R , U -norm of a function in terms of its Fourier representation, and show that it admits an upper bound by the quantity C f .
Revised: We compute the R , U -norm of a function in terms of its Fourier representation, and show that it admits an upper bound by the quantity C f .

Original: This shows that our approximation bound is tighter than the previous bound by Breiman (1993), and meaningful in more instances (e
Revised: This shows that our approximation bound is tighter than the previous bound by Breiman (1993), and meaningful in more instances (e

Original: g. for ﬁnite-width neural networks).
Revised: g. for ﬁnite-width neural networks).

Original: We also show R , U -norm-based bounds analogous to the ones of Klusowski & Barron (2018).
Revised: We also show R , U -norm-based bounds analogous to the ones of Klusowski & Barron (2018).

Original: • Setting U as the open unit ball of radius R , we show that neural network representations of f on U hold for multiple even Radon measures, which contrasts with the uniqueness result provided by Ongie et al.
Revised: • Setting U as the open unit ball of radius R , we show that neural network representations of f on U hold for multiple even Radon measures, which contrasts with the uniqueness result provided by Ongie et al.

Original: (2019) for the case of R d .
Revised: (2019) for the case of R d .

Original: We study the structure of the sets of Radon measures which give rise to the same function on U .
Revised: We study the structure of the sets of Radon measures which give rise to the same function on U .

Original: The non-uniqueness of the measure representing a measure could be linked to the phenomenon of mode connectivity.
Revised: The non-uniqueness of the measure representing a measure could be linked to the phenomenon of mode connectivity.

Original: Additional related work .
Revised: Additional related work .

Original: There have been other recent works which have used the Radon transform to study neural networks in settings different from ours (Parhi & Nowak, 2021a; Bartolucci et al., 2021).
Revised: There have been other recent works which have used the Radon transform to study neural networks in settings different from ours (Parhi & Nowak, 2021a; Bartolucci et al., 2021).

Original: These two works consider the R -norm as a regularizer for an inverse problem, and proceed to prove representer theorems: there exists a solution of the regularized problem which is a twolayer neural network equal to the number of datapoints.
Revised: These two works consider the R -norm as a regularizer for an inverse problem, and proceed to prove representer theorems: there exists a solution of the regularized problem which is a twolayer neural network equal to the number of datapoints.

Original: Regarding inﬁnite-width network spaces, E & Wojtowytsch (2020) present several equivalent deﬁnitions and provides a review.
Revised: Regarding inﬁnite-width network spaces, E & Wojtowytsch (2020) present several equivalent deﬁnitions and provides a review.

Original: A well-known line of work (Mei et al., 2018; Chizat & Bach, 2018; Rotskoff & Vanden-Eijnden, 2018) studies the convergence of gradient descent for inﬁnite-width two-layer neural networks.
Revised: A well-known line of work (Mei et al., 2018; Chizat & Bach, 2018; Rotskoff & Vanden-Eijnden, 2018) studies the convergence of gradient descent for inﬁnite-width two-layer neural networks.

Original:  F RAMEWORK 2. N OTATION
Revised:  F RAMEWORK 2. N OTATION

Original: S d − 1 denotes the ( d − 1)
Revised: S d − 1 denotes the ( d − 1)

Original: -dimensional hypersphere (as a submanifold of R d ) and B R ( R d ) is the
Revised: -dimensional hypersphere (as a submanifold of R d ) and B R ( R d ) is the

Original: Euclidean open ball of radius R .
Revised: Euclidean open ball of radius R .

Original: For U ⊆ R d measurable, the space C 

Revised: For U ⊆ R d measurable, the space C

Original: ( U ) of functions vanishing at inﬁnity contains the continuous functions f such that for any (cid:15) > 0 , there exists compact K ⊆
Revised: ( U ) of functions vanishing at inﬁnity contains the continuous functions f such that for any (cid:15) > 0 , there exists compact K ⊆

Original: U depending on f such that | f ( x ) | < (cid:15) for x ∈ U \ K .
Revised: U depending on f such that | f ( x ) | < (cid:15) for x ∈ U \ K .

Original: P ( U ) is the set of Borel probability measures, M ( U ) is the space of ﬁnite signed Radon measures (which may be seen as the dual of
Revised: P ( U ) is the set of Borel probability measures, M ( U ) is the space of ﬁnite signed Radon measures (which may be seen as the dual of

Original: C 0 ( U ) ).
Revised: C 0 ( U ) ).

Original: Throughout the paper, the term Radon measure refers to a ﬁnite signed Radon measure for shortness.
Revised: Throughout the paper, the term Radon measure refers to a ﬁnite signed Radon measure for shortness.

Original: If γ ∈ M ( U ) , then (cid:107) γ (cid:107) TV is the total variation (TV) norm of γ .
Revised: If γ ∈ M ( U ) , then (cid:107) γ (cid:107) TV is the total variation (TV) norm of γ .

Original: M C ( U ) denotes the space of complex-valued ﬁnite signed Radon measures, deﬁned as the dual space of C 0 ( U, C ) (the space of complex-valued functions vanishing at inﬁnity).
Revised: M C ( U ) denotes the space of complex-valued ﬁnite signed Radon measures, deﬁned as the dual space of C 0 ( U, C ) (the space of complex-valued functions vanishing at inﬁnity).

Original: We denote by S ( R d ) the space of
Revised: We denote by S ( R d ) the space of

Original: Schwartz functions, which contains the functions in C ∞ ( R d ) whose derivatives of any order decay faster than polynomials of all orders, i.e. for all k, p ∈ ( N 0 ) d , sup x ∈ R d | x k ∂ ( p ) ϕ
Revised: Schwartz functions, which contains the functions in C ∞ ( R d ) whose derivatives of any order decay faster than polynomials of all orders, i.e. for all k, p ∈ ( N 0 ) d , sup x ∈ R d | x k ∂ ( p ) ϕ

Original: ( x ) | < + ∞ .
Revised: ( x ) | < + ∞ .

Original: For f ∈ L 1 ( R d ) , we use ˆ f to denote the unitary Fourier transforms with angular frequency, deﬁned as ˆ f ( ξ ) =
Revised: For f ∈ L 1 ( R d ) , we use ˆ f to denote the unitary Fourier transforms with angular frequency, deﬁned as ˆ f ( ξ ) =

Original: [Equation]
Revised: [Equation]

Original: If ˆ f ∈ L 1 ( R d ) as well, we have the inversion formula f
Revised: If ˆ f ∈ L 1 ( R d ) as well, we have the inversion formula f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: The Fourier transform is a continuous automorphism on S ( R d ) .
Revised: The Fourier transform is a continuous automorphism on S ( R d ) .

Original: 2. E XISTING SPARSE APPROXIMATION BOUNDS
Revised: 2. E XISTING SPARSE APPROXIMATION BOUNDS

Original: One of the classical results of the theory of two-layer neural networks (Breiman (1993), building on (Barron, 1993)) states that given a probability measure p ∈ P ( B R ( R d )) and a function f :
Revised: One of the classical results of the theory of two-layer neural networks (Breiman (1993), building on (Barron, 1993)) states that given a probability measure p ∈ P ( B R ( R d )) and a function f :

Original: B R ( R d ) →
Revised: B R ( R d ) →

Original: R admitting a Fourier representation of the form f ( x ) =[Equation] R
Revised: R admitting a Fourier representation of the form f ( x ) =

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: d ˆ f ( ξ ) , where ˆ f ∈ M C ( R d ) is a complex-valued Radon measure such that C f =
Revised: d ˆ f ( ξ ) , where ˆ f ∈ M C ( R d ) is a complex-valued Radon measure such that C f =

Original: [Equation]
Revised: [Equation]

Original: there exists a two-layer neural network ˜ f
Revised: there exists a two-layer neural network ˜ f

Original: ( x ) = 1 m (cid:80) 
Revised: ( x ) = 1 n (cid:80) ni

Original: [Equation]
Revised: [Equation]

Original: i ( [Equation] x
Revised: i ( (cid:104) x

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: ω

Original: [Equation]
Revised: [Equation]

Original:  − b i ) + such that
Revised: (cid:105) − b i ) + such that

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: These classical results do not provide bounds on the magnitude of the neural network weights.
Revised: These classical results do not provide bounds on the magnitude of the neural network weights.

Original: More recently, Klusowski & Barron (2018) showed similar approximation bounds for two-layer
Revised: More recently, Klusowski & Barron (2018) showed similar approximation bounds for two-layer

Original: ReLU networks under additional l 1 and l 0 bounds on the weights a i , ω i .
Revised: ReLU networks under additional l 1 and l 0 bounds on the weights a i , ω i .

Original: Namely, if ˜ C f = 1 (
Revised: Namely, if ˜ C f = [Equation] π )

Original: [Equation]
Revised: [Equation]

Original: π ) d/ 2 (cid:82) R d (cid:107) ξ (cid:107) 21 d | ˆ f | ( ξ ) < + ∞ there exists a two-layer neural network ˜ f
Revised: / 2 (cid:82) R d (cid:107) [Equation] (cid:107) 21 d | ˆ f | ( ξ ) < + ∞ there exists a two-layer neural network ˜ f

Original: ( x ) = a 0 + (cid:104) ω 0 , x (cid:105)
Revised: ( x ) = a 0 + (cid:104) ω 0 , x (cid:105)

Original: + κm (cid:80) mi
Revised: + κn (cid:80) ni

Original: ={+=
+}
Revised: {+=
+}

Original: a i  (cid:104) w i , x (cid:105) − b i ) + with | a i | ≤ 1 , (cid:107) ω
Revised: a i ( (cid:104) w i , x (cid:105) − b i ) + with | a i | ≤ 1 , (cid:107) ω

Original: i (cid:107) ≤ 1 , b i ∈ [0 , 1] , and κ ≤ 2 ˜ C f , and sup
Revised: i (cid:107) ≤ 1 , b i ∈ [0 , 1] , and κ ≤ 2 ˜ C f , and sup

Original: [Equation]
Revised: [Equation]

Original: where c is a universal constant.
Revised: where c is a universal constant.

Original: 2. R EPRESENTATION RESULTS ON R d BASED ON THE R ADON TRANSFORM
Revised: 2. R EPRESENTATION RESULTS ON R d BASED ON THE R ADON TRANSFORM

Original: One deﬁnes P d denotes the space of hyperplanes on R d , whose elements may be represented by points in S d − 1 × R by identifying { x |(cid:104) ω, x (cid:105) =
Revised: One deﬁnes P d denotes the space of hyperplanes on R d , whose elements may be represented by points in S d − 1 × R by identifying { x |(cid:104) ω, x (cid:105) =

Original: b } with both ( ω, b ) and ( − ω, − b ) .
Revised: b } with both ( ω, b ) and ( − ω, − b ) .

Original: Thus, functions on P d are even functions on S d − 1 × R and we will use both notions interchangeably 1 .
Revised: Thus, functions on P d are even functions on S d − 1 × R and we will use both notions interchangeably 1 .

Original: The Radon transform and the dual Radon transform.
Revised: The Radon transform and the dual Radon transform.

Original: If f : R d → R is a function which is integrable over all the hyperplanes of R d , we may deﬁne the Radon transform R f : P d → R as
Revised: If f : R d → R is a function which is integrable over all the hyperplanes of R d , we may deﬁne the Radon transform R f : P d → R as

Original: [Equation]
Revised: [Equation]

Original: That is, one integrates the function f over the hyperplane ( ω, b ) .
Revised: That is, one integrates the function f over the hyperplane ( ω, b ) .

Original: If Φ : P d → R is a continuous function, the dual Radon transform R ∗ Φ : R d → R is deﬁned as
Revised: If Φ : P d → R is a continuous function, the dual Radon transform R ∗ Φ : R d → R is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where the integral is with respect to the Hausdorff measure over S d − 1 .
Revised: where the integral is with respect to the Hausdorff measure over S d − 1 .

Original: R and R ∗ are adjoint operators in the appropriate domains (see Lemma 13).
Revised: R and R ∗ are adjoint operators in the appropriate domains (see Lemma 13).

Original: The Radon inversion formula.
Revised: The Radon inversion formula.

Original: When f ∈ C ∞ ( R d ) , one has that (Theorem 3.1, Helgason (2011))
Revised: When f ∈ C ∞ ( R d ) , one has that (Theorem 3.1, Helgason (2011))

Original: [Equation]
Revised: [Equation]

Original: where c d = 1 2(2 π ) d − 1 and ( − ∆) s/ 2 denotes the (negative) fractional Laplacian, deﬁned via its
Revised: where c d = 1 2(2 π ) d − 1 and ( − ∆) s/ 2 denotes the (negative) fractional Laplacian, deﬁned via its

Original: Fourier transform as
Revised: Fourier transform as

Original: [Equation]
Revised: [Equation]

Original: The R -norm.
Revised: The R -norm.

Original: Given a function f : R d → R , Ongie et al.
Revised: Given a function f : R d → R , Ongie et al.

Original: (2019) introduce the quantity (cid:107) f
Revised: (2019) introduce the quantity (cid:107) f

Original: [Equation]
Revised: [Equation]

Original: if f Lipschitz
Revised: if f Lipschitz

Original: [Equation]
Revised: [Equation]

Original: otherwise .
Revised: otherwise .

Original: [Equation]
Revised: [Equation]

Original: They call it the R -norm of f , although it is formally a semi-norm.
Revised: They call it the R -norm of f , although it is formally a semi-norm.

Original: Here, the space S ( S d − 1 × R ) of
Revised: Here, the space S ( S d − 1 × R ) of

Original: Schwartz functions on S d − 1 × R is deﬁned, in analogy with S ( R d ) , as the space of C ∞ functions ψ on S d − 1 × R which for any integers k, l ≥ 0 and any differential operator D on S d − 1 satisfy sup ( ω,b ) ∈ S d − 1 × R | (1 + | b | k )
Revised: Schwartz functions on S d − 1 × R is deﬁned, in analogy with S ( R d ) , as the space of C ∞ functions ψ on S d − 1 × R which for any integers k, l ≥ 0 and any differential operator D on S d − 1 satisfy sup ( ω,b ) ∈ S d − 1 × R | (1 + | b | k )

Original: ∂ kb ( Dψ )( ω, b ) | < + ∞ (Helgason (2011), p. 5).
Revised: ∂ kb ( Dψ )( ω, b ) | < + ∞ (Helgason (2011), p. 5).

Original: Moreover, S ( P d ) = { ψ ∈ S ( S d − 1 × R ) | ψ even } , which means the conditions on ψ in (4) can be written as ψ ∈
Revised: Moreover, S ( P d ) = { ψ ∈ S ( S d − 1 × R ) | ψ even } , which means the conditions on ψ in (4) can be written as ψ ∈

Original: S ( P d ) , (cid:107) ψ (cid:107) ∞ ≤ 1 .
Revised: S ( P d ) , (cid:107) ψ (cid:107) ∞ ≤ 1 .

Original: The ﬁniteness of the R -norm indicates whether a function on R d admits an exact representation as an inﬁnitely wide neural network.
Revised: The ﬁniteness of the R -norm indicates whether a function on R d admits an exact representation as an inﬁnitely wide neural network.

Original: Namely, Ongie et al.
Revised: Namely, Ongie et al.

Original: (2019) in their Lemma 10 show that (cid:107) f (cid:107) R is ﬁnite if and only if there exists a (unique) even measure α ∈ M ( S d − 1 × R ) and (unique) v ∈ R d , c ∈ R such that for any x ∈ R d ,
Revised: (2019) in their Lemma 10 show that (cid:107) f (cid:107) R is ﬁnite if and only if there exists a (unique) even measure α ∈ M ( S d − 1 × R ) and (unique) v ∈ R d , c ∈ R such that for any x ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: in which case, (cid:107) f (cid:107) R
Revised: in which case, (cid:107) f (cid:107) R

Original: =
Revised: =

Original: (cid:107) α (cid:107) TV .
Revised: (cid:107) α (cid:107) TV .

Original: Remark the following differences between this result and the bounds by (Breiman, 1993; Klusowski & Barron, 2018) shown in equations (1) and (2):
Revised: Remark the following differences between this result and the bounds by (Breiman, 1993; Klusowski & Barron, 2018) shown in equations (1) and (2):

Original: (i) in (5) we have an exact representation with inﬁnite-width neural networks instead of an approximation result with ﬁnite-width, (ii) in (5) the representation holds on R d instead of a bounded domain.
Revised: (i) in (5) we have an exact representation with inﬁnite-width neural networks instead of an approximation result with ﬁnite-width, (ii) in (5) the representation holds on R d instead of a bounded domain.

Original: In our work, we derive representation results similar to the ones of Ongie et al.
Revised: In our work, we derive representation results similar to the ones of Ongie et al.

Original: for functions deﬁned on bounded open sets, which naturally give rise to sparse approximation results that reﬁne those of (Breiman, 

Revised: for functions deﬁned on bounded open sets, which naturally give rise to sparse approximation results that reﬁne those of (Breiman,

Original: ; Klusowski & Barron, 2018).
Revised: ; Klusowski & Barron, 2018).

Original: One property that makes the Radon transform and its dual useful to analyze neural networks can be understood at a very high level via the following argument: if f ( x ) = (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b )
Revised: One property that makes the Radon transform and its dual useful to analyze neural networks can be understood at a very high level via the following argument: if f ( x ) = (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b )

Original: + ρ ( ω, b ) d ( ω, b ) + (cid:104) v, x (cid:105) + c for some smooth rapidly decreasing function ρ , then ∆ f ( x ) =
Revised: + ρ ( ω, b ) d ( ω, b ) + (cid:104) v, x (cid:105) + c for some smooth rapidly decreasing function ρ , then ∆ f ( x ) =

Original: (cid:82)
Revised: (cid:82)

Original: S d − 1 × R δ (cid:104) ω,x (cid:105) =
Revised: S d − 1 × R δ (cid:104) ω,x (cid:105) =

Original: b ρ ( ω, b ) d ( ω, b ) = (cid:82) S
Revised: b ρ ( ω, b ) d ( ω, b ) = (cid:82) S

Original: d − 1 ρ ( ω, (cid:104) ω, x (cid:105) )
Revised: d − 1 ρ ( ω, (cid:104) ω, x (cid:105) )

Original: dω =
Revised: dω =

Original: ( R ∗ ρ )( x ) .
Revised: ( R ∗ ρ )( x ) .

Original: For a general function f of the form (5), one has similarly that (cid:104) ∆ f, ϕ (cid:105) =
Revised: For a general function f of the form (5), one has similarly that (cid:104) ∆ f, ϕ (cid:105) =

Original: (cid:104) α, R ϕ (cid:105) for any ϕ ∈ S ( R d ) .
Revised: (cid:104) α, R ϕ (cid:105) for any ϕ ∈ S ( R d ) .

Original: This property relates the evaluations of the measure α to the function ∆ f via the Radon transform, and is the main ingredient in the proof of Lemma 10 of Ongie et al.
Revised: This property relates the evaluations of the measure α to the function ∆ f via the Radon transform, and is the main ingredient in the proof of Lemma 10 of Ongie et al.

Original: While we also rely on it, we need many additional tools to deal with the case of bounded open sets.
Revised: While we also rely on it, we need many additional tools to deal with the case of bounded open sets.

Original:  R EPRESENTATION RESULTS ON BOUNDED OPEN SETS
Revised:  R EPRESENTATION RESULTS ON BOUNDED OPEN SETS

Original: Schwartz functions on open sets.
Revised: Schwartz functions on open sets.

Original: Let U ⊆
Revised: Let U ⊆

Original: R d be an open subset.
Revised: R d be an open subset.

Original: The space of Schwartz functions on U may be deﬁned as S ( U ) =
Revised: The space of Schwartz functions on U may be deﬁned as S ( U ) =

Original: (cid:84) z ∈ R d \U (cid:84) k ∈ ( N 0 )
Revised: (cid:84) z ∈ R d \U (cid:84) k ∈ ( N 0 )

Original: d { f ∈ S
Revised: d { f ∈ S

Original: ( R d ) | ∂ ( k ) f
Revised: ( R d ) | ∂ ( k ) f

Original: ( z ) = 0 } , i.e. they are those Schwartz functions on R d such that the derivatives of all orders vanish outside of U (c.f.
Revised: ( z ) = 0 } , i.e. they are those Schwartz functions on R d such that the derivatives of all orders vanish outside of U (c.f.

Original: Def. 3.2, Shaviv (2020)).
Revised: Def. 3.2, Shaviv (2020)).

Original: The structure of S ( U ) is similar to S ( R d ) in that its topology is given by a family of semi-norms indexed by (( N 0 ) d ) 2 : (cid:107) f (cid:107) k,k (cid:48)
Revised: The structure of S ( U ) is similar to S ( R d ) in that its topology is given by a family of semi-norms indexed by (( N 0 ) d ) 2 : (cid:107) f (cid:107) k,k (cid:48)

Original: =
Revised: =

Original: sup x ∈U | x k · f ( k (cid:48) )
Revised: sup x ∈U | x k · f ( k (cid:48) )

Original: ( x ) | .
Revised: ( x ) | .

Original: Similarly, if V ⊆ P d is open, we deﬁne S ( V ) =
Revised: Similarly, if V ⊆ P d is open, we deﬁne S ( V ) =

Original: (cid:84) ( ω,b ) ∈ ( S d − 1 × R ) \V (cid:84) k ∈ ( N 0 ) 2 { f ∈ S ( P d ) | ∂ k 1 b ˆ∆ k 2 f ( ω, b ) = 0 } , where ˆ∆ is the spherical Laplacian.
Revised: (cid:84) ( ω,b ) ∈ ( S d − 1 × R ) \V (cid:84) k ∈ ( N 0 ) 2 { f ∈ S ( P d ) | ∂ k 1 b ˆ∆ k 2 f ( ω, b ) = 0 } , where ˆ∆ is the spherical Laplacian.

Original: The R , U -norm.
Revised: The R , U -norm.

Original: Let U ⊆
Revised: Let U ⊆

Original: R d be a bounded open set, and let ˜ U :
Revised: R d be a bounded open set, and let ˜ U :

Original: = { ( ω, (cid:104) ω, x (cid:105) )
Revised: = { ( ω, (cid:104) ω, x (cid:105) )

Original: ∈ S d − 1 × R | x ∈
Revised: ∈ S d − 1 × R | x ∈

Original: U} .
Revised: U} .

Original: For any function f : R d → R , we deﬁne the R , U -norm of f as
Revised: For any function f : R d → R , we deﬁne the R , U -norm of f as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note the similarity between this quantity and the R -norm deﬁned in (4); the main differences are that the supremum here is taken over the even Schwartz functions on ˜ U instead of S d − 1 × R , and that the non-Lipschitz case does not need a separate treatment.
Revised: Note the similarity between this quantity and the R -norm deﬁned in (4); the main differences are that the supremum here is taken over the even Schwartz functions on ˜ U instead of S d − 1 × R , and that the non-Lipschitz case does not need a separate treatment.

Original: Remark that (cid:107) f (cid:107) R , U ≤ (cid:107) f (cid:107) R .
Revised: Remark that (cid:107) f (cid:107) R , U ≤ (cid:107) f (cid:107) R .

Original: If f has enough regularity, we may write (cid:107) f (cid:107) R , U = (cid:82)
Revised: If f has enough regularity, we may write (cid:107) f (cid:107) R , U = (cid:82)

Original: ˜
Revised: ˜

Original: U |R ( − ∆) ( d +1) /
Revised: U |R ( − ∆) ( d +1) /

Original: 2 f | ( ω, b ) d ( ω, b ) , using that the fractional Laplacian is self-adjoint and R ∗ is the adjoint of R .
Revised: 2 f | ( ω, b ) d ( ω, b ) , using that the fractional Laplacian is self-adjoint and R ∗ is the adjoint of R .

Original: Deﬁne P d U to be the bounded open set of hyperplanes of R d that intersect U , which in analogy with Subsec. 2.3, is equal to ˜ U up to the identiﬁcation of ( ω, b ) with ( − ω, − b ) .
Revised: Deﬁne P d U to be the bounded open set of hyperplanes of R d that intersect U , which in analogy with Subsec. 2.3, is equal to ˜ U up to the identiﬁcation of ( ω, b ) with ( − ω, − b ) .

Original: Similarly, note that S ( P d U ) =
Revised: Similarly, note that S ( P d U ) =

Original: { ψ ∈ S ( ˜ U ) , ψ even } , which allows to rewrite the conditions in (6) as ψ ∈ S ( P d U ) , (cid:107) ψ (cid:107) ∞ ≤ 1 .
Revised: { ψ ∈ S ( ˜ U ) , ψ even } , which allows to rewrite the conditions in (6) as ψ ∈ S ( P d U ) , (cid:107) ψ (cid:107) ∞ ≤ 1 .

Original: The following proposition, which is based on the Riesz-Markov-Kakutani representation theorem, shows that when the R , U -norm is ﬁnite, it can be associated to a unique Radon measure over P d U .
Revised: The following proposition, which is based on the Riesz-Markov-Kakutani representation theorem, shows that when the R , U -norm is ﬁnite, it can be associated to a unique Radon measure over P d U .

Original: Proposition 1.
Revised: Proposition 1.

Original: If (cid:107) f (cid:107) R , U < + ∞ , there exists a unique Radon measure α ∈ M
Revised: If (cid:107) f (cid:107) R , U < + ∞ , there exists a unique Radon measure α ∈ M

Original: ( P d U ) such that − c d (cid:104) f, ( − ∆)
Revised: ( P d U ) such that − c d (cid:104) f, ( − ∆)

Original: ( d +1) / 2 R ∗ ψ (cid:105) =
Revised: ( d +1) / 2 R ∗ ψ (cid:105) =

Original: (cid:82) P d U ψ ( ω, b ) dα ( ω, b ) for any ψ ∈ S ( P d U ) .
Revised: (cid:82) P d U ψ ( ω, b ) dα ( ω, b ) for any ψ ∈ S ( P d U ) .

Original: Moreover, (cid:107) f (cid:107) R , U =
Revised: Moreover, (cid:107) f (cid:107) R , U =

Original: (cid:107) α (cid:107) TV .
Revised: (cid:107) α (cid:107) TV .

Original: Building on this, we see that a neural network representation for bounded U holds when the R , Unorm is ﬁnite:
Revised: Building on this, we see that a neural network representation for bounded U holds when the R , Unorm is ﬁnite:

Original: Theorem 1.
Revised: Theorem 1.

Original: Let U be a open, bounded subset of R d .
Revised: Let U be a open, bounded subset of R d .

Original: Let f : R d → R such that (cid:107) f (cid:107) R , U < + ∞ .
Revised: Let f : R d → R such that (cid:107) f (cid:107) R , U < + ∞ .

Original: Let α ∈ M
Revised: Let α ∈ M

Original: ( P d U ) be given by Proposition 1.
Revised: ( P d U ) be given by Proposition 1.

Original: For any ϕ ∈ S ( U ) , there exist unique v ∈ R d and c ∈ R such that
Revised: For any ϕ ∈ S ( U ) , there exist unique v ∈ R d and c ∈ R such that

Original: [Equation]
Revised: [Equation]

Original: That is, f ( x ) =
Revised: That is, f ( x ) =

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: + c for x a.e. (almost everywhere) in U .
Revised: + c for x a.e. (almost everywhere) in U .

Original: If f is continuous, then the equality holds for all x ∈ U .
Revised: If f is continuous, then the equality holds for all x ∈ U .

Original: Remark that this theorem does not claim that the representation given by α, v, c is unique, unlike
Revised: Remark that this theorem does not claim that the representation given by α, v, c is unique, unlike

Original: Lemma 10 by Ongie et al.
Revised: Lemma 10 by Ongie et al.

Original: concerning analogous representations on R d .
Revised: concerning analogous representations on R d .

Original: In Sec.
Revised: In Sec.

Original: 5 we see that such representations are in fact not unique, for particular choices of the set U .
Revised: 5 we see that such representations are in fact not unique, for particular choices of the set U .

Original: We want to underline that the proof of Theorem 1 uses completely different tools from the ones of Lemma 10 by Ongie et al.
Revised: We want to underline that the proof of Theorem 1 uses completely different tools from the ones of Lemma 10 by Ongie et al.

Original: (2019): their result relies critically on the fact that the only harmonic Lipschitz functions on R d are afﬁne functions, which is not true for functions on bounded subsets in our setting.
Revised: (2019): their result relies critically on the fact that the only harmonic Lipschitz functions on R d are afﬁne functions, which is not true for functions on bounded subsets in our setting.

Original:  S PARSE APPROXIMATION FOR FUNCTIONS WITH BOUNDED R , U - NORM
Revised:  S PARSE APPROXIMATION FOR FUNCTIONS WITH BOUNDED R , U - NORM

Original: In this section, we show how to obtain approximation bounds of a function f on a bounded open set U using a ﬁxed-width neural network with bounded coefﬁcients, in terms of the R , U -norm introduced in the previous section.
Revised: In this section, we show how to obtain approximation bounds of a function f on a bounded open set U using a ﬁxed-width neural network with bounded coefﬁcients, in terms of the R , U -norm introduced in the previous section.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let U ⊆ B R ( R d ) be a bounded open set.
Revised: Let U ⊆ B R ( R d ) be a bounded open set.

Original: Suppose that f : R d → R is such that (cid:107) f (cid:107) R , U is ﬁnite, where (cid:107) ·
Revised: Suppose that f : R d → R is such that (cid:107) f (cid:107) R , U is ﬁnite, where (cid:107) ·

Original: (cid:107) R , U is deﬁned in (6) .
Revised: (cid:107) R , U is deﬁned in (6) .

Original: Let v ∈ R d , c ∈ R as in Theorem 1.
Revised: Let v ∈ R d , c ∈ R as in Theorem 1.

Original: Then, there exists { ( ω i , b i ) } ni
Revised: Then, there exists { ( ω i , b i ) } ni

Original: =1 ⊆
Revised: =1 ⊆

Original: ˜ U and { a i } ni =1 ⊆ {± 1 } such that the function ˜ f : R d → R deﬁned as
Revised: ˜ U and { a i } ni =1 ⊆ {± 1 } such that the function ˜ f : R d → R deﬁned as

Original: [Equation]
Revised: [Equation]

Original: fulﬁlls, for x a.e. in U ,
Revised: fulﬁlls, for x a.e. in U ,

Original: [Equation]
Revised: [Equation]

Original: The equality holds for all x ∈ U if f is continuous.
Revised: The equality holds for all x ∈ U if f is continuous.

Original: The proof of Theorem 2 (in App. B) uses the neural network representation (7) and a probabilistic argument.
Revised: The proof of Theorem 2 (in App. B) uses the neural network representation (7) and a probabilistic argument.

Original: If one samples { ( ω i , b i ) } n i =1 from a probability distribution proportional to | α | , a
Revised: If one samples { ( ω i , b i ) } n i =1 from a probability distribution proportional to | α | , a

Original: Rademacher complexity bound upper-bounds the expectation of the supremum norm between ˜ f and f , which yields the result.
Revised: Rademacher complexity bound upper-bounds the expectation of the supremum norm between ˜ f and f , which yields the result.

Original: Note the resemblance of (8) with the bound (1); the R , U norm of f replaces the quantity C f .
Revised: Note the resemblance of (8) with the bound (1); the R , U norm of f replaces the quantity C f .

Original: We can also use the R , U -norm to obtain a bound analogous to (2), that is, with a slightly better dependency in the exponent of n at the expense of a constant factor growing with the dimension.
Revised: We can also use the R , U -norm to obtain a bound analogous to (2), that is, with a slightly better dependency in the exponent of n at the expense of a constant factor growing with the dimension.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let f : R d → R and U ⊆
Revised: Let f : R d → R and U ⊆

Original: B 1 ( R d ) open such that (cid:107)
Revised: B 1 ( R d ) open such that (cid:107)

Original: f
Revised: f

Original: (cid:107) R , U < + ∞ .
Revised: (cid:107) R , U < + ∞ .

Original: Then, then there exist { a i } ni =1 ⊆
Revised: Then, then there exist { a i } ni =1 ⊆

Original: [ − 1 , 1] , { ω i } ni =1 ⊆ { ω ∈ R d |(cid:107) ω (cid:107) 1 = 1 } and { b i } ni =1 ⊆
Revised: [ − 1 , 1] , { ω i } ni =1 ⊆ { ω ∈ R d |(cid:107) ω (cid:107) 1 = 1 } and { b i } ni =1 ⊆

Original: [0 , 1] and κ < √ d (cid:107) f (cid:107) R , U such that the function
Revised: [0 , 1] and κ < √ d (cid:107) f (cid:107) R , U such that the function

Original: [Equation]
Revised: [Equation]

Original: fulﬁlls, for x a.e. in U and some universal constant c > 0 ,
Revised: fulﬁlls, for x a.e. in U and some universal constant c > 0 ,

Original: [Equation]
Revised: [Equation]

Original: The proof of this result (in App. B) follows readily from the representation (7) and Theorem 1 of
Revised: The proof of this result (in App. B) follows readily from the representation (7) and Theorem 1 of

Original: Klusowski & Barron (2018).
Revised: Klusowski & Barron (2018).

Original: 4. L INKS WITH THE F OURIER SPARSE APPROXIMATION BOUNDS
Revised: 4. L INKS WITH THE F OURIER SPARSE APPROXIMATION BOUNDS

Original: The following result shows that setting U =
Revised: The following result shows that setting U =

Original: B R ( R d ) , the R , U -norm can be bounded by the Fourierbased quantities C f , ˜ C
Revised: B R ( R d ) , the R , U -norm can be bounded by the Fourierbased quantities C f , ˜ C

Original: f introduced in Subsec.
Revised: f introduced in Subsec.

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume that the function f : R d → R admits a Fourier representation of the form f ( x ) =
Revised: Assume that the function f : R d → R admits a Fourier representation of the form f ( x ) =

Original: [Equation]
Revised: [Equation]

Original: f ∈ M C ( R d ) a complex-valued Radon measure.
Revised: f ∈ M C ( R d ) a complex-valued Radon measure.

Original: Let C f be the quantity used in the sparse approximation bound by Breiman (1993) (see Subsec. 2.2).
Revised: Let C f be the quantity used in the sparse approximation bound by Breiman (1993) (see Subsec. 2.2).

Original: Then, one has that
Revised: Then, one has that

Original: [Equation]
Revised: [Equation]

Original: As a direct consequence of Theorem 3, when U =
Revised: As a direct consequence of Theorem 3, when U =

Original: B R ( R d ) the right-hand side of (8) can be upper-bounded by R 2 C f / √ n .
Revised: B R ( R d ) the right-hand side of (8) can be upper-bounded by R 2 C f / √ n .

Original: This allows to reﬁne the bound (1) from Breiman (1993) to a bound in the supremum norm over B R ( R d ) , and where the approximating neural network ˜ f ( x ) =m
Revised: This allows to reﬁne the bound (1) from Breiman (1993) to a bound in the supremum norm over B R ( R d ) , and where the approximating neural network ˜ f ( x ) =n [Equation] ∈

Original: [Equation]
Revised: [Equation]

Original: and b i ∈ ( − R, R ) .
Revised: R [Equation] R ) .

Original: While we are able to prove the bound (9), the Fourier representation of f does not allow for a manageable expression for the measure α described in Proposition 1.
Revised: While we are able to prove the bound (9), the Fourier representation of f does not allow for a manageable expression for the measure α described in Proposition 1.

Original: For that, the following theorem starts from a slightly modiﬁed Fourier representation of f , under which one can describe the measure α and provide a formula for the R , U -norm.
Revised: For that, the following theorem starts from a slightly modiﬁed Fourier representation of f , under which one can describe the measure α and provide a formula for the R , U -norm.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let f : R d → R admitting the representation
Revised: Let f : R d → R admitting the representation

Original: [Equation]
Revised: [Equation]

Original: for some complex-valued Radon measures µ ∈ M C ( S d − 1 × R ) such that dµ ( ω, b ) = dµ ( − ω, − b )
Revised: for some complex-valued Radon measures µ ∈ M C ( S d − 1 × R ) such that dµ ( ω, b ) = dµ ( − ω, − b )

Original: =
Revised: =

Original: d ¯ µ ( − ω, b ) =
Revised: d ¯ µ ( − ω, b ) =

Original: d ¯ µ ( ω, − b ) , and (cid:82) S d − 1 × R b 2 d | µ | ( ω, b ) < + ∞ .
Revised: d ¯ µ ( ω, − b ) , and (cid:82) S d − 1 × R b 2 d | µ | ( ω, b ) < + ∞ .

Original: Choosing U
Revised: Choosing U

Original: = B R ( R d ) , the unique measure α ∈ M ( P dR ) speciﬁed by Proposition 1 takes the following form:
Revised: = B R ( R d ) , the unique measure α ∈ M ( P dR ) speciﬁed by Proposition 1 takes the following form:

Original: [Equation]
Revised: [Equation]

Original: where K = 2 π d/ 2 / Γ( d 2 ) .
Revised: where K = 2 π d/ 2 / Γ( d 2 ) .

Original: Note that α is real-valued because (cid:82) R t 2 e − itb dµ ( ω, t )
Revised: Note that α is real-valued because (cid:82) R t 2 e − itb dµ ( ω, t )

Original: ∈
Revised: ∈

Original: R as t 2 dµ ( ω, t ) = ( − t ) 2 dµ ( ω, − t ) .
Revised: R as t 2 dµ ( ω, t ) = ( − t ) 2 dµ ( ω, − t ) .

Original: Consequently, the R , B R ( R d ) -norm of f is
Revised: Consequently, the R , B R ( R d ) -norm of f is

Original: [Equation]
Revised: [Equation]

Original: Remark that µ is the pushforward of the measure ˆ f by the mappings ξ (cid:55)→ ( ± ξ/ (cid:107) ξ (cid:107) , ± ξ ) .
Revised: Remark that µ is the pushforward of the measure ˆ f by the mappings ξ (cid:55)→ ( ± ξ/ (cid:107) ξ (cid:107) , ± ξ ) .

Original: When the
Revised: When the

Original: Fourier transform ˆ f admits a density, one may obtain the density of µ via a change from Euclidean to spherical coordinates: dµ ( ω, b ) = 12 vol ( S d − 1 )
Revised: Fourier transform ˆ f admits a density, one may obtain the density of µ via a change from Euclidean to spherical coordinates: dµ ( ω, b ) = 12 vol ( S d − 1 )

Original: ˆ f ( bω )
Revised: ˆ f ( bω )

Original: | b | d − 1 d ( ω, b ) .
Revised: | b | d − 1 d ( ω, b ) .

Original: Hence, Theorem 4 provides an operative way to compute the R , U -norm of f if one has access to the Fourier transform of ˆ f .
Revised: Hence, Theorem 4 provides an operative way to compute the R , U -norm of f if one has access to the Fourier transform of ˆ f .

Revised: Note that equation (11) implies that the R , B R ( R d ) -norm of f increases with R , and in particular is smaller than the R -norm of f , which corresponds to setting R = ∞ .

Original: Theorems 3 and 4 are proven jointly in App.
Revised: Theorems 3 and 4 are proven jointly in App.

Original: B. Note that from the expression (11) one can easily see that (cid:107) f (cid:107) R , B R ( R d ) is upper-bounded by RC f :
Revised: B. Note that from the expression (11) one can easily see that (cid:107) f (cid:107) R , B R ( R d ) is upper-bounded by RC f :

Original: [Equation]
Revised: [Equation]

Original: R
Revised: R

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: where the equality holds since µ is the pushforward of ˆ f .
Revised: where the equality holds since µ is the pushforward of ˆ f .

Original: Equation (12) makes apparent the norm (cid:107) f (cid:107) R , B R ( R d ) is sometimes much smaller than the quantities C f , ˜ C
Revised: Equation (12) makes apparent the norm (cid:107) f (cid:107) R , B R ( R d ) is sometimes much smaller than the quantities C f , ˜ C

Original: f , as is showcased by the following one-dimensional example (see the proof in App. B).
Revised: f , as is showcased by the following one-dimensional example (see the proof in App. B).

Original: In these situations, the sparse approximation bounds that we provide in Theorem 2 and Proposition 2 are much better than the ones in (1)-(2).
Revised: In these situations, the sparse approximation bounds that we provide in Theorem 2 and Proposition 2 are much better than the ones in (1)-(2).

Original: Example 1.
Revised: Example 1.

Original: Take the function f : R → R deﬁned as f ( x ) = cos( x ) − cos((1 + (cid:15) ) x ) , with (cid:15) > 0 .
Revised: Take the function f : R → R deﬁned as f ( x ) = cos( x ) − cos((1 + (cid:15) ) x ) , with (cid:15) > 0 .

Original: f admits the Fourier representation f
Revised: f admits the Fourier representation f

Original: ( x ) = 1 (2 π ) 1 / 2 (cid:82) R (cid:112) π 2 ( δ 1 ( ξ ) + δ − 1 ( ξ ) − δ 1+ (cid:15) ( ξ ) − δ − 1 − (cid:15) ( ξ ))
Revised: ( x ) = [Equation] δ − 1 − (cid:15) ( ξ ))

Original: e iξx dξ .
Revised: e iξx dξ .

Original: We have that C f = 2 + 2 (cid:15)
Revised: We have that C f = 2 + 2 (cid:15)

Original: + (cid:15) 2 , and (cid:107) f
Revised: + (cid:15) 2 , and (cid:107) f

Original: (cid:107) R , B R ( R d ) ≤ R (cid:0) R(cid:15)
Revised: (cid:107) R , B R ( R d ) ≤ R (cid:0) R(cid:15)

Original: + 2 (cid:15)
Revised: + 2 (cid:15)

Original: + (cid:15) 2 (cid:1) .
Revised: + (cid:15) 2 (cid:1) .

Original: (cid:107) f (cid:107) R , B R ( R d ) goes to zero as (cid:15) → 0 , while C f converges to 2.
Revised: (cid:107) f (cid:107) R , B R ( R d ) goes to zero as (cid:15) → 0 , while C f converges to 2.

Original: An interesting class of functions for which (cid:107) f (cid:107) R , B R ( R d ) is ﬁnite but C f , ˜ C f are inﬁnite are functions that can be written as a ﬁnite-width neural network on B R ( R d ) , as shown in the following proposition.
Revised: An interesting class of functions for which (cid:107) f (cid:107) R , B R ( R d ) is ﬁnite but C f , ˜ C f are inﬁnite are functions that can be written as a ﬁnite-width neural network on B R ( R d ) , as shown in the following proposition.

Original: Proposition 3.
Revised: Proposition 3.

Original: Let f : R d → R deﬁned as f ( x ) = 1 n (cid:80) ni
Revised: Let f : R d → R deﬁned as f ( x ) = 1 n (cid:80) ni

Original: =

Revised: =

Original: a i ( (cid:104) ω
Revised: a i ( (cid:104) ω

Original: i , x (cid:105) − b i ) + for all x ∈ R d , with { ω i } ni =1 ⊆ S
Revised: i , x (cid:105) − b i ) + for all x ∈ R d , with { ω i } ni =1 ⊆ S

Original: d − 1 , { a i } ni =1 , { b i } ni =1 ⊆ R .
Revised: d − 1 , { a i } ni =1 , { b i } ni =1 ⊆ R .

Original: Then, for any bounded open set U , we have (cid:107) f (cid:107) R , U ≤n (cid:80) ni
Revised: Then, for any bounded open set U , we have (cid:107) f (cid:107) R , U ≤n (cid:80) ni

Original: =1 | a i | , while C f , ˜ C f
Revised: =1 | a i | , while C f , ˜ C f

Original: = + ∞ if f is not an afﬁne function.
Revised: = + ∞ if f is not an afﬁne function.

Original: Proposition 3 makes use of the fact that the R , U -norm is always upper-bounded by the R -norm, which also means that all the bounds developed in Ongie et al.
Revised: Proposition 3 makes use of the fact that the R , U -norm is always upper-bounded by the R -norm, which also means that all the bounds developed in Ongie et al.

Original: (2019) apply for the R , U -norm.
Revised: (2019) apply for the R , U -norm.

Original: The fact that ﬁnite-width neural networks have inﬁnite C f was stated by E & Wojtowytsch (2020), that used them to show the gap between the functions with ﬁnite C f and functions representable by inﬁnite-width neural networks (belonging to the Barron space, in their terminology).
Revised: The fact that ﬁnite-width neural networks have inﬁnite C f was stated by E & Wojtowytsch (2020), that used them to show the gap between the functions with ﬁnite C f and functions representable by inﬁnite-width neural networks (belonging to the Barron space, in their terminology).

Original: It remains to be seen whether the gap is closed when considering functions with ﬁnite R , U -norm, i.e., whether any function admitting an inﬁnite-width representation (7) on U has a ﬁnite R , U -norm.
Revised: It remains to be seen whether the gap is closed when considering functions with ﬁnite R , U -norm, i.e., whether any function admitting an inﬁnite-width representation (7) on U has a ﬁnite R , U -norm.

Original: Moving to the non-linear Radon transform.
Revised: Moving to the non-linear Radon transform.

Original: In many applications the function of interest f may be better represented as (cid:82) ( (cid:104) ω, ϕ ( x ) (cid:105) − t )
Revised: In many applications the function of interest f may be better represented as (cid:82) ( (cid:104) ω, ϕ ( x ) (cid:105) − t )

Original: + dα ( ω, t ) + (cid:104) v, x (cid:105) + c , where ϕ is a ﬁxed ﬁnite dimensional, non-linear and bounded feature map.
Revised: + dα ( ω, t ) + (cid:104) v, x (cid:105) + c , where ϕ is a ﬁxed ﬁnite dimensional, non-linear and bounded feature map.

Original: Our results trivially extend to this case where in the
Revised: Our results trivially extend to this case where in the

Original: Radon transform hyperplanes are replaced by hyperplanes in the feature space.
Revised: Radon transform hyperplanes are replaced by hyperplanes in the feature space.

Original: This can be seen as the “kernel trick” applied to the Radon transform.
Revised: This can be seen as the “kernel trick” applied to the Radon transform.

Original: The corresponding (cid:107) f (cid:107) R ,ϕ ( U ) corresponds to the sparsity of the decomposition in the feature space, and we have better approximation when (cid:107) f (cid:107) R ,ϕ ( U ) < (cid:107) f (cid:107) R , U .
Revised: The corresponding (cid:107) f (cid:107) R ,ϕ ( U ) corresponds to the sparsity of the decomposition in the feature space, and we have better approximation when (cid:107) f (cid:107) R ,ϕ ( U ) < (cid:107) f (cid:107) R , U .

Original: This gives a simple condition for when transfer learning is successful, and explains the success of using random fourier features as a preprocessing in implicit neural representations of images and surfaces (Tancik et al., 2020).
Revised: This gives a simple condition for when transfer learning is successful, and explains the success of using random fourier features as a preprocessing in implicit neural representations of images and surfaces (Tancik et al., 2020).

Original: In order to go beyond the ﬁxed feature maps and tackle deeper ReLU networks, we think that the non-linear Radon transform (Ehrenpreis, 2003) is an interesting tool to explore.
Revised: In order to go beyond the ﬁxed feature maps and tackle deeper ReLU networks, we think that the non-linear Radon transform (Ehrenpreis, 2003) is an interesting tool to explore.

Original: We note that Parhi & Nowak (2021b) introduced recently a representer theorem for deep ReLU networks using Radon transforms as a regularizer.
Revised: We note that Parhi & Nowak (2021b) introduced recently a representer theorem for deep ReLU networks using Radon transforms as a regularizer.

Original:  I NFINITE - WIDTH REPRESENTATIONS ARE NOT UNIQUE ON BOUNDED SETS
Revised:  I NFINITE - WIDTH REPRESENTATIONS ARE NOT UNIQUE ON BOUNDED SETS

Original: Ongie et al.
Revised: Ongie et al.

Original: (2019) show that when the R -norm of f is ﬁnite, there is a unique measure α ∈ M ( R d ) such that the representation (5) holds for x ∈ R d .
Revised: (2019) show that when the R -norm of f is ﬁnite, there is a unique measure α ∈ M ( R d ) such that the representation (5) holds for x ∈ R d .

Original: In this section we show that when we only ask the representation to hold for x in a bounded open set, there exist several measures that do the job; in fact, they span an inﬁnite-dimensional space.
Revised: In this section we show that when we only ask the representation to hold for x in a bounded open set, there exist several measures that do the job; in fact, they span an inﬁnite-dimensional space.

Original: Let U =
Revised: Let U =

Original: B R ( R d ) be the open ball of radius R > 0 in R d , which means that ˜ U
Revised: B R ( R d ) be the open ball of radius R > 0 in R d , which means that ˜ U

Original: = S
Revised: = S

Original: d − 1 × ( − R, R ) and P d U is the set of hyperplanes { x |(cid:104) ω, x (cid:105) =
Revised: d − 1 × ( − R, R ) and P d U is the set of hyperplanes { x |(cid:104) ω, x (cid:105) =

Original: b } such that (cid:107) ω (cid:107) =
Revised: b } such that (cid:107) ω (cid:107) =

Original: 1 and b ∈ ( − R, R ) , which we denote by P dR for simplicity.
Revised: 1 and b ∈ ( − R, R ) , which we denote by P dR for simplicity.

Original: In the following we will construct a space of Radon measures α ∈ M ( P dR ) whose neural network representation (5) coincide for all x ∈ B R ( R d ) .
Revised: In the following we will construct a space of Radon measures α ∈ M ( P dR ) whose neural network representation (5) coincide for all x ∈ B R ( R d ) .

Original: Note that since any bounded subset of R d is included in some open ball, our results imply that such representations are non-unique on any bounded set.
Revised: Note that since any bounded subset of R d is included in some open ball, our results imply that such representations are non-unique on any bounded set.

Original: Remark 1.
Revised: Remark 1.

Original: When one considers representations on B R ( R d ) of the sort (5) with the measure α lying in the larger space M ( S d − 1 × R ) , the non-uniqueness is apparent because there are two ‘trivial’ kinds of symmetry at play:
Revised: When one considers representations on B R ( R d ) of the sort (5) with the measure α lying in the larger space M ( S d − 1 × R ) , the non-uniqueness is apparent because there are two ‘trivial’ kinds of symmetry at play:

Original: (i) Related to parity: when the measure α is odd, we have (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b ) + dα ( ω, b ) = 12 (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b ) + − ( −(cid:104) ω, x (cid:105) + b ) + dα ( ω, b ) =
Revised: (i) Related to parity: when the measure α is odd, we have (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b ) + dα ( ω, b ) = 12 (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b ) + − ( −(cid:104) ω, x (cid:105) + b ) + dα ( ω, b ) =

Original: (cid:104)
Revised: (cid:104)

Original: 12 (cid:82) S d − 1 × R ω dα ( ω, b ) , x (cid:105) − 12 (cid:82) S
Revised: 12 (cid:82) S d − 1 × R ω dα ( ω, b ) , x (cid:105) − 12 (cid:82) S

Original: d − 1 × R b dα ( ω, b ) , which is an afﬁne function of x . (ii) Related to boundedness: if ( ω, b ) ∈ S d − 1 × ( R \ ( − R, R )) , x (cid:55)→ ( (cid:104) ω, x (cid:105)− b ) + restricted to B R ( R d ) is an afﬁne function of x .
Revised: d − 1 × R b dα ( ω, b ) , which is an afﬁne function of x . (ii) Related to boundedness: if ( ω, b ) ∈ S d − 1 × ( R \ ( − R, R )) , x (cid:55)→ ( (cid:104) ω, x (cid:105)− b ) + restricted to B R ( R d ) is an afﬁne function of x .

Original: Hence, if α is supported on S d − 1 × S d − 1 × ( R \ ( − R, R )) , x [Equation] ) is an afﬁne function when restricted to B R ( R d ) .
Revised: Hence, if α is supported on S d − 1 × S d − 1 × ( R \ ( − R, R )) , x (cid:55)→ (cid:82) S d − 1 × R ( (cid:104) ω, x (cid:105) − b ) + dα ( ω, b ) is an afﬁne function when restricted to B R ( R d ) .

Original: Since in Sec.
Revised: Since in Sec.

Original: 3 we restrict our scope to measures α lying in M ( P d U ) , these two kinds of symmetries are already quotiented out in our analysis.
Revised: 3 we restrict our scope to measures α lying in M ( P d U ) , these two kinds of symmetries are already quotiented out in our analysis.

Original: The third kind of non-uniqueness that we discuss in this section is conceptually deeper, taking place within M ( P d U ) .
Revised: The third kind of non-uniqueness that we discuss in this section is conceptually deeper, taking place within M ( P d U ) .

Original: Let { Y k,j | k ∈ Z
Revised: Let { Y k,j | k ∈ Z

Original: + , 1 ≤ j ≤ N k,d } be the orthonormal basis of spherical harmonics of the space
Revised: + , 1 ≤ j ≤ N k,d } be the orthonormal basis of spherical harmonics of the space

Original: L 2 ( S d − 1 ) (Atkinson & Han, 2012).
Revised: L 2 ( S d − 1 ) (Atkinson & Han, 2012).

Original: It is well known that for any k , the functions { Y k,j | 1 ≤ j ≤
Revised: It is well known that for any k , the functions { Y k,j | 1 ≤ j ≤

Original: N k,d } are the restrictions to S d − 1 of homogeneous polynomials of degree k , and in fact N k,d is the dimension of the space of homogeneous harmonic polynomials of degree k .
Revised: N k,d } are the restrictions to S d − 1 of homogeneous polynomials of degree k , and in fact N k,d is the dimension of the space of homogeneous harmonic polynomials of degree k .

Original: Consider the following subset of even functions in C ∞ ( S d − 1 × ( − R, R )) :
Revised: Consider the following subset of even functions in C ∞ ( S d − 1 × ( − R, R )) :

Original: [Equation]
Revised: [Equation]

Original: where X k (cid:48) denotes the monomial of degree k (cid:48) on ( − R, R ) .
Revised: where X k (cid:48) denotes the monomial of degree k (cid:48) on ( − R, R ) .

Original: We have the following result regarding the non-uniqueness of neural network representations:
Revised: We have the following result regarding the non-uniqueness of neural network representations:

Original: Theorem 5.
Revised: Theorem 5.

Original: If α ∈ M ( P dR ) is such that α ∈ cl w ( span ( A )) , then we have that 

Revised: If α ∈ M ( P dR ) is such that α ∈ cl w ( span ( A )) , then we have that

Original: = (cid:82)
Revised: = (cid:82)

Original: S d − 1 × ( − R,R ) ( (cid:104) ω, x (cid:105) − b )
Revised: [Equation]

Original: + dα ( ω, b ) for any x ∈ B R ( R d ) .
Revised: + dα ( ω, b ) for any x ∈ B R ( R d ) .

Original: That is, α yields a neural network representation of the zero-function on B R ( R d ) .
Revised: That is, α yields a neural network representation of the zero-function on B R ( R d ) .

Original: Here, we consider span ( A ) as a subset of M ( P dR ) by the
Revised: Here, we consider span ( A ) as a subset of M ( P dR ) by the

Original: Riesz-Markov-Kakutani representation theorem via the action (cid:104) g, ϕ (cid:105) =
Revised: Riesz-Markov-Kakutani representation theorem via the action (cid:104) g, ϕ (cid:105) =

Original: (cid:82) P dR ϕ ( ω, b ) g ( ω, b ) d ( ω, b ) for any g ∈ span ( A ) , ϕ ∈ C 

Revised: (cid:82) P dR ϕ ( ω, b ) g ( ω, b ) d ( ω, b ) for any g ∈ span ( A ) , ϕ ∈ C

Original: ( P dR ) , and cl w denotes the closure in the topology of weak convergence of M ( S d − 1 × R ) .
Revised: ( P dR ) , and cl w denotes the closure in the topology of weak convergence of M ( S d − 1 × R ) .

Original: In particular, any measure whose density is in the span of A will yield a function which is equal to zero when restricted to B R ( R d ) .
Revised: In particular, any measure whose density is in the span of A will yield a function which is equal to zero when restricted to B R ( R d ) .

Original: As an example of this result, we show a simple measure in M ( P d 1 ) which represents the zero function on B 1 ( R 2 ) .
Revised: As an example of this result, we show a simple measure in M ( P d 1 ) which represents the zero function on B 1 ( R 2 ) .

Original: Example 2 (Non-zero measure representing the zero function on B 1 ( R 2 ) ) .
Revised: Example 2 (Non-zero measure representing the zero function on B 1 ( R 2 ) ) .

Original: We deﬁne the even Radon measure α ∈ M ( S 1 × ( − 1 , 1)) with density dα ( ω, b ) =
Revised: We deﬁne the even Radon measure α ∈ M ( S 1 × ( − 1 , 1)) with density dα ( ω, b ) =

Original: (8 ω 40 − 8 ω 20 +1) d ( ω, b ) where ω = ( ω 0 , ω 1 ) .
Revised: (8 ω 40 − 8 ω 20 +1) d ( ω, b ) where ω = ( ω 0 , ω 1 ) .

Original: Then, for any x ∈ B 1 ( R 2 ) , 0 = (cid:82) S 1 × ( − 1 , 1) ( (cid:104) ω, x (cid:105) − b )
Revised: Then, for any x ∈ B 1 ( R 2 ) , 0 = (cid:82) S 1 × ( − 1 , 1) ( (cid:104) ω, x (cid:105) − b )

Original: + dα ( ω, x ) .
Revised: + dα ( ω, x ) .

Original: On the one hand, Proposition 1 states that there exists a unique measure α ∈ M ( P d U ) such that − c d (cid:104) f, ( − ∆) ( d +1) / 2 R ∗ ψ (cid:105) =
Revised: On the one hand, Proposition 1 states that there exists a unique measure α ∈ M ( P d U ) such that − c d (cid:104) f, ( − ∆) ( d +1) / 2 R ∗ ψ (cid:105) =

Original: (cid:82) P d U ψ ( ω, b ) dα ( ω, b ) for any ψ ∈ S ( P d U ) if (cid:107) f (cid:107) R , U is ﬁnite.
Revised: (cid:82) P d U ψ ( ω, b ) dα ( ω, b ) for any ψ ∈ S ( P d U ) if (cid:107) f (cid:107) R , U is ﬁnite.

Original: On the other hand, Theorem 5 claims that functions admit distinct representations by measures in M ( P d U ) .
Revised: On the other hand, Theorem 5 claims that functions admit distinct representations by measures in M ( P d U ) .

Original: The following theorem clariﬁes these two seemingly contradictory statements.
Revised: The following theorem clariﬁes these two seemingly contradictory statements.

Original: Consider the following subset of even functions in C ∞ ( S d − 1 × ( − R, R )) , which contains A :B [Equation] ≡ k
Revised: Consider the following subset of even functions in C ∞ ( S d − 1 × ( − R, R )) , which contains A :

Original: [Equation]

Original: mod

Original: [Equation]
Revised: [Equation]

Original: ≤ N d

Original: [Equation]

Original: [Equation]

Original: Proposition 4.
Revised: Proposition 4.

Original: Let 0 < R < R (cid:48) .
Revised: Let 0 < R < R (cid:48) .

Original: Let f : R d → R such that (cid:107) f (cid:107) R , B R (cid:48)
Revised: Let f : R d → R such that (cid:107) f (cid:107) R , B R (cid:48)

Original: ( R d ) <
Revised: ( R d ) <

Original: + ∞ and let α ∈ M ( P d R ) be the unique measure speciﬁed by Proposition 1.
Revised: + ∞ and let α ∈ M ( P dR ) be the unique measure speciﬁed by Proposition 1.

Original: Then, α is the unique measure in
Revised: Then, α is the unique measure in

Original: M ( P dR ) such that
Revised: M ( P dR ) such that

Original: [Equation]
Revised: [Equation]

Original: The condition (13) holds for any measure α (cid:48)
Revised: The condition (13) holds for any measure α (cid:48)

Original: ∈ M ( P dR ) for which f admits a representation of the form (7) on B R ( R d ) .
Revised: ∈ M ( P dR ) for which f admits a representation of the form (7) on B R ( R d ) .

Original: Thus, α can be characterized as the unique measure in M ( P dR ) such that f admits a representation of the form (7) on B R ( R d ) and the condition (14) holds.
Revised: Thus, α can be characterized as the unique measure in M ( P dR ) such that f admits a representation of the form (7) on B R ( R d ) and the condition (14) holds.

Original: In (14), the quantity (cid:104) f, ( − ∆) ( d +1) / 2 R ∗ ( Y k,j ⊗ 1 | X | <R X k (cid:48) )
Revised: In (14), the quantity (cid:104) f, ( − ∆) ( d +1) / 2 R ∗ ( Y k,j ⊗ 1 | X | <R X k (cid:48) )

Original: (cid:105) is well deﬁned despite 1 | X | <R X k (cid:48) not being continuous on R ; we deﬁne it as (cid:104) f, ( − ∆) 
Revised: (cid:105) is well deﬁned despite 1 | X | <R X k (cid:48) not being continuous on R ; we deﬁne it as (cid:104) f, ( − ∆) [Equation] d [Equation]

Revised: [Equation]

Original: ( d +1) / 2 R ∗ (( Y k,j ⊗ 1 | X | <R X k (cid:48) )
Revised: R [Equation] Y [Equation]

Revised: |

Original: + ˜ g ) (cid:105) , where ˜ g is any function in S ( P dR (cid:48) )
Revised: [Equation] | < [Equation] , where ˜ g is any function in S ( P dR (cid:48) )

Original: such that ( Y k,j ⊗ 1 | X | <R X k (cid:48) )
Revised: such that ( Y k,j ⊗ 1 | X | <R X k (cid:48) )

Original: +
Revised: +

Original: ˜ g ∈ S
Revised: ˜ g ∈ S

Original: ( P dR (cid:48) )
Revised: ( P dR (cid:48) )

Original: (which do exist, see App. C).
Revised: (which do exist, see App. C).

Original: In short, Proposition 4 characterizes the measure α from Proposition 1 in terms of its evaluations on the spaces R ( S ( B R
Revised: In short, Proposition 4 characterizes the measure α from Proposition 1 in terms of its evaluations on the spaces R ( S ( B R

Original: ( R d )))
Revised: ( R d )))

Original: and span ( B ) , and by Corollary 1 the direct sum of these two spaces dense in C 0 ( P d R ) , which by the Riesz-Markov-Kakutani representation theorem is the predual of
Revised: and span ( B ) , and by Corollary 1 the direct sum of these two spaces dense in C 0 ( P dR ) , which by the Riesz-Markov-Kakutani representation theorem is the predual of

Original: M ( P dR ) .
Revised: M ( P dR ) .

Original: Interestingly, the condition (13) holds for any measure α ∈ M ( P dR ) which represents the function f on B R ( R d ) , but it is easy to see that the condition (14) does not: by Theorem 5 we have that if ψ ∈ span ( A ) ⊆ span ( B ) , the measure α (cid:48) deﬁned as dα (cid:48) ( ω, b )
Revised: Interestingly, the condition (13) holds for any measure α ∈ M ( P dR ) which represents the function f on B R ( R d ) , but it is easy to see that the condition (14) does not: by Theorem 5 we have that if ψ ∈ span ( A ) ⊆ span ( B ) , the measure α (cid:48) deﬁned as dα (cid:48) ( ω, b )

Original: = dα ( ω, b ) + ψ ( ω, b ) db represents the function f on B R ( R d ) , and
Revised: = dα ( ω, b ) + ψ ( ω, b ) db represents the function f on B R ( R d ) , and

Original: (cid:104) α (cid:48) , ψ (cid:105)
Revised: (cid:104) α (cid:48) , ψ (cid:105)

Original: = (cid:104) α, ψ (cid:105)
Revised: = (cid:104) α, ψ (cid:105)

Original: + (cid:107) ψ (cid:107) 22 .
Revised: + (cid:107) ψ (cid:107) 22 .

Original: It remains an open question to see whether Theorem 5 captures all the measures which represent the zero function on B R ( R d ) , which we hypothesize.
Revised: It remains an open question to see whether Theorem 5 captures all the measures which represent the zero function on B R ( R d ) , which we hypothesize.

Original: If that was the case, we would obtain a complete characterization of the Radon measures which represent a given function on B R ( R d ) .
Revised: If that was the case, we would obtain a complete characterization of the Radon measures which represent a given function on B R ( R d ) .

Original: Mode connectivity .
Revised: Mode connectivity .

Original: Mode connectivity is the phenomenon that optima of neural network losses (at least the ones found by gradient descent) turn out to be connected by paths where the loss value is almost constant, and was observed empirically by Garipov et al.
Revised: Mode connectivity is the phenomenon that optima of neural network losses (at least the ones found by gradient descent) turn out to be connected by paths where the loss value is almost constant, and was observed empirically by Garipov et al.

Original: ; Draxler et al.
Revised: ; Draxler et al.

Original: Kuditipudi et al.
Revised: Kuditipudi et al.

Original: (2019) provided an algorithmic explanation based on dropout, and an explanation based on the noise stability property.
Revised: (2019) provided an algorithmic explanation based on dropout, and an explanation based on the noise stability property.

Original: Theorem 5 suggests an explanation for mode connectivity from a functional perspective: one can construct ﬁnitely-supported measures which approximate a measure α ∈ cl w ( span ( A )) , yielding ﬁnite-width neural networks with non-zero weights which approximate the zero function on B R ( R d ) .
Revised: Theorem 5 suggests an explanation for mode connectivity from a functional perspective: one can construct ﬁnitely-supported measures which approximate a measure α ∈ cl w ( span ( A )) , yielding ﬁnite-width neural networks with non-zero weights which approximate the zero function on B R ( R d ) .

Original: Assuming that the data distribution is supported in
Revised: Assuming that the data distribution is supported in

Original: B R ( R d ) , adding a multiple of one such network to an optimal network will produce little change in the loss value because the function being represented is essentially unchanged.
Revised: B R ( R d ) , adding a multiple of one such network to an optimal network will produce little change in the loss value because the function being represented is essentially unchanged.

Original: More work is required to conﬁrm or discard this intuition.
Revised: More work is required to conﬁrm or discard this intuition.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We provided in this paper tighter sparse approximation bounds for two-layer ReLU neural networks.
Revised: We provided in this paper tighter sparse approximation bounds for two-layer ReLU neural networks.

Original: Our results build on the introduction of Radon-based R , U -norms for functions deﬁned on a bounded open set U .
Revised: Our results build on the introduction of Radon-based R , U -norms for functions deﬁned on a bounded open set U .

Original: Our bounds reﬁne Fourier-based approximation bounds of Breiman (1993); Klusowski & Barron (2018).
Revised: Our bounds reﬁne Fourier-based approximation bounds of Breiman (1993); Klusowski & Barron (2018).

Original: We also showed that the representation of inﬁnite width neural networks on bounded open sets are not unique, which can be seen as a functional view of mode connectivity observed in training deep neural networks.
Revised: We also showed that the representation of inﬁnite width neural networks on bounded open sets are not unique, which can be seen as a functional view of mode connectivity observed in training deep neural networks.

Original: We leave two open questions: whether any function admitting an inﬁnite-width representation on U has a ﬁnite R , U -norm, and whether Theoremcaptures all the measures which represent the zero function on B R ( R d ) .
Revised: We leave two open questions: whether any function admitting an inﬁnite-width representation on U has a ﬁnite R , U -norm, and whether Theoremcaptures all the measures which represent the zero function on B R ( R d ) .

Original: Finally, in order to extend our theory to deeper ReLU networks we believe that non-linear Radon transforms (Ehrenpreis, 2003) are interesting tools to explore.
Revised: Finally, in order to extend our theory to deeper ReLU networks we believe that non-linear Radon transforms (Ehrenpreis, 2003) are interesting tools to explore.