Original: T HE I MPLICIT B IAS OF M INIMA S TABILITY IN M ULTIVARIATE S HALLOW R E LU N ETWORKS
Revised: T HE I MPLICIT B IAS OF M INIMA S TABILITY IN M ULTIVARIATE S HALLOW R E LU N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the type of solutions to which stochastic gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss.
Revised: We study the type of solutions to which stochastic gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss.

Original: Our results are based on a dynamical stability analysis.
Revised: Our results are based on a dynamical stability analysis.

Original: In the univariate case, it was shown that linearly stable minima correspond to network functions (predictors) f , whose second derivative has a bounded weighted L 1 norm.
Revised: In the univariate case, it was shown that linearly stable minima correspond to network functions (predictors), whose second derivative has a bounded weighted L 1 norm.

Original: The bound on the norm gets smaller as the step size increases, implying that training with a large step size leads to ‘smoother’ predictors.
Revised: Notably, the bound gets smaller as the step size increases, implying that training with a large step size leads to ‘smoother’ predictors.

Original: Here we generalize this result to the multivariate case, showing that a similar result applies to the Laplacian of the predictor, ∆ f .
Revised: Here we generalize this result to the multivariate case, showing that a similar result applies to the Laplacian of the predictor.

Original: We demonstrate the tightness of our bound on the MNIST dataset, and show that it accurately captures the behavior of the solutions as a function of the step size.
Revised: We demonstrate the tightness of our bound on the MNIST dataset, and show that it accurately captures the behavior of the solutions as a function of the step size.

Original: Additionally, we prove a depth separation result on the approximation power of ReLU networks corresponding to stable minima of the loss.
Revised: Additionally, we prove a depth separation result on the approximation power of ReLU networks corresponding to stable minima of the loss.

Original: Specifically, although shallow ReLU networks are universal approximators, we prove that stable shallow networks are not.
Revised: Specifically, although shallow ReLU networks are universal approximators, we prove that stable shallow networks are not.

Original: Namely, there is a function that cannot be well-approximated by stable single hidden-layer ReLU networks trained with a non-vanishing step size.
Revised: Namely, there is a function that cannot be wellapproximated by stable single hidden-layer ReLU networks trained with a nonvanishing step size.

Original: In contrast, we show that the same function can be realized as a stable two hidden-layer ReLU network.
Revised: This is while the same function can be realized as a stable two hidden-layer ReLU network.

Original: Finally, we prove that if a function is sufficiently smooth (Sobolev) then it can be approximated arbitrarily well using single hidden-layer ReLU networks that correspond to stable solutions of gradient descent.
Revised: Finally, we prove that if a function is sufficiently smooth (in a Sobolev sense) then it can be approximated arbitrarily well using shallow ReLU networks that correspond to stable solutions of gradient descent.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks (NNs) have been demonstrating phenomenal performance in a wide array of fields, from computer vision and speech processing to medical sciences.
Revised: Neural networks (NNs) have been demonstrating phenomenal performance in a wide array of fields, from computer vision and speech processing to medical sciences.

Original: Modern networks are typically taken to be highly overparameterized.
Revised: Modern networks are typically taken to be highly overparameterized.

Original: In such setting, the training loss usually has multiple global minima, which correspond to models that perfectly fit the training data.
Revised: In such setting, the training loss usually has multiple global minima, which correspond to models that perfectly fit the training data.

Original: Some of those models are clearly sub-optimal in terms of generalization.
Revised: Some of those models are clearly sub-optimal in terms of generalization.

Original: Yet, the training process seems to consistently avoid those bad global minima, and somehow steer the model towards global minima that generalize well.
Revised: Yet, the training process seems to consistently avoid those bad global minima, and somehow steer the model towards global minima that generalize well.

Original: A long line of works attributed this behavior to “implicit biases” of the training algorithms (e.g., Zhang et al. ; Gunasekar et al. (2017); Soudry et al.
Revised: A long line of works attributed this behavior to “implicit biases” of the training algorithms, e.g. , (Zhang et al., 2017; Gunasekar et al., 2017; Soudry et al., 2018; Arora et al., 2019).

Original: Recently, it has been recognized that a dominant factor affecting the implicit bias of gradient descent (GD) and stochastic GD (SGD), is associated with dynamical stability.
Revised: Recently, it has been recognized that a dominant factor affecting the implicit bias of gradient descent (GD) and stochastic gradient descent (SGD), is associated with dynamical stability.

Original: Roughly speaking, the dynamical stability of a minimum point refers to the ability of the optimizer to stably converge to that point.
Revised: Roughly speaking, the dynamical stability of a minimum point refers to the ability of the optimizer to stably converge to that point.

Original: Particular research efforts have been devoted to understanding linear stability , namely the dynamical stability of the optimizer’s linearized dynamics around the minimum (Wu et al., 2018;
Revised: Particular research efforts have been devoted to understanding linear stability , namely the dynamical stability of the optimizer’s linearized dynamics around the minimum (Wu et al., 2018;

Original: Nar & Sastry, 2018; Mulayoff et al., 2021; Ma & Ying, 2021).
Revised: Nar & Sastry, 2018; Mulayoff et al., 2021; Ma & Ying, 2021).

Original: For GD and SGD, it is well known that a minimum is linearly stable if the loss terrain is sufficiently flat w.r.t.
Revised: For GD and SGD, it is well known that a minimum is linearly stable if the loss terrain is sufficiently flat w.r.t.

Original: the step size η .
Revised: the step size η .

Original: Concretely, a necessary condition for a minimum to be linearly stable for GD and SGD is that the top eigenvalue of the Hessian at that minimum point be smaller than 2 /η (see Sec. 2).
Revised: Concretely, a necessary condition for a minimum to be linearly stable for GD and SGD is that the top eigenvalue of the Hessian at that minimum point be smaller than 2 /η (see Sec. 2).

Original: Although this  condition only characterizes the linearized dynamics, it has been empirically shown to hold in realworld neural-network training (Cohen et al., 2020; Gilmer et al., 2022).
Revised: Although this [Figure] condition only characterizes the linearized dynamics, it has been empirically shown to hold in realworld neural-network training (Cohen et al., 2020; Gilmer et al., 2022).

Original: The linear stability condition turns out to have a strong effect on the nature of the network that is obtained upon convergence, both in terms of the end-to-end predictor function (Mulayoff et al., 2021), and in terms of the way this function is implemented by the network (Mulayoff & Michaeli, 2020).[Figure]
Revised: The linear stability condition turns out to have a strong effect on the nature of the network that is obtained upon convergence, both in terms of the end-to-end predictor function (Mulayoff et al., 2021), and in terms of the way this function is implemented by the network (Mulayoff & Michaeli, 2020).

Original: Mulayoff et al.
Revised: Mulayoff et al.

Original: (2021) studied how linear stability affects a single hidden-layer univariate ReLU network, when trained with the quadratic loss.
Revised: (2021) studied how linear stability affects a single hidden-layer univariate ReLU network, when trained with the quadratic loss.

Original: They showed that in this setting, stable solutions of
Revised: They showed that in this setting, stable solutions of

Original: SGD with step size η correspond to functions f satisfying
Revised: SGD with step size η correspond to functions f satisfying

Original: [Equation]
Revised: [Equation]

Original: ) where f denotes the network input-output function, and g is a weight function that depends only on the training data.
Revised: [Equation] where f denotes the network input-output function, and g is a weight function that depends only on the training data.

Original: This result implies that for univariate shallow ReLU networks, SGD is biased towards ‘smooth’ solutions 1 .
Revised: This result implies that for univariate shallow ReLU networks, SGD is biased towards ‘smooth’ solutions 1 .

Original: Moreover, the larger the step size η , the smoother the solution becomes.
Revised: Moreover, the larger the step size η , the smoother the solution becomes.

Original: In this paper we study the stable solutions of single hidden-layer ReLU networks with multidimensional inputs, trained using SGD and the quadratic loss.
Revised: In this paper, we study the stable solutions of single hidden-layer ReLU networks with multidimensional inputs, trained using SGD and the quadratic loss.

Original: Particularly, in Sec.
Revised: Particularly, in Sec.

Original: 3 we generalize the result of Mulayoff et al.
Revised: 3 we generalize the result of Mulayoff et al.

Original: (2021) to the multivariate setting.
Revised: (2021) to the multivariate setting.

Original: As it turns out, the natural extension of (1) involves the Radon transform of the Laplacian of the predictor function, ∆ f (see Theorem 1).
Revised: As it turns out, the natural extension of (1) involves the Radon transform of the Laplacian of the predictor function, ∆ f (see Thm. 1).

Original: However, we show this result can also be interpreted in primal space as
Revised: However, we show this result can also be interpreted in primal space as

Original: [Equation]
Revised: [Equation]

Original: where ρ is some weighting function.
Revised: where ρ is some weighting function.

Original: Thus, stable solutions of SGD in the multivariate case also correspond to smooth predictors ( i.e., functions whose Laplacian has a small weighted L 1 norm).
Revised: Thus, stable solutions of SGD in the multivariate case also correspond to smooth predictors ( i.e. , functions whose Laplacian has a small weighted L 1 norm).

Original: The larger the step size, the smoother the function becomes.
Revised: The larger the step size, the smoother the function becomes.

Original: Figure 1 illustrates this phenomenon.
Revised: Figure 1 illustrates this phenomenon.

Original: Additionally, we study the approximation power of single hidden-layer ReLU networks that correspond to stable minima of the loss.
Revised: Additionally, we study the approximation power of single hidden-layer ReLU networks corresponding to stable minima.

Original: It is well known that shallow ReLU networks can approximatearbitrarily well any continuous function over a compact set (Pinkus, 1999).
Revised: It is well known that shallow ReLU networks can approximate any continuous function over a compact set (Pinkus, 1999).

Revised: However this does not imply that SGD can stably converge to such approximations.

Original: However, some approximationscorrespond to unstable minima and are thus of limited practical interest.
Revised: If there exist functions whose approximations are all unstable, then this property may be of limited practical interest.

Original: In Sec.
Revised: In Sec.

Original: 4 we prove that every convergent sequence of stable networks has a limit function that also satisfies the stability condition (Theorem 1).
Revised: 4 we prove that every convergent sequence of stable networks has a limit function that also satisfies the stability condition (Thm. 1).

Original: Building on this, we prove a depth separation result.
Revised: Building on this, we prove a depth separation result.

Original: Specifically, we show that there exists a function that does not satisfy the stability condition for any positive step size.
Revised: Specifically, we show that there exists a function that does not satisfy the stability condition for any positive step size.

Original: Namely, it cannot be stably approximated by a single hidden-layer ReLU network trained with a non-vanishing step size.
Revised: Namely, it cannot be stably approximated by a single hidden-layer ReLU network trained with a non-vanishing step size.

Original: Yet, the same function can be realized as a two hidden-layer ReLU network corresponding to a stable minimum.
Revised: Yet, the same function can be realized as a two hidden-layer ReLU network corresponding to a stable minimum.

Original: Moreover, in Sec. 5 we show that if a function is sufficiently smooth (Sobolev) then it can be approximated arbitrarily well using single hidden-layer ReLU networks that correspond to stable solutions of GD.
Revised: Moreover, in Sec. 5 we show that if a function is sufficiently smooth (Sobolev) then it can be approximated arbitrarily well using single hidden-layer ReLU networks that correspond to stable solutions of GD.

Original: Finally, in Sec. 3.4 and 6 we demonstrate our results.
Revised: Finally, in Sec. 3.3 and 6 we demonstrate our results.

Original: Particularly, we illustrate how our stable minima characterization (Theorem 1) can be used to predict certain properties of the solution.
Revised: Particularly, we illustrate how our stable minima characterization (Thm. 1) can be used to predict certain properties of the solution.

Original: For example, for certain isotropic data ( e.g.  Gaussian), we show that a large step size tends to increase the biases of all neurons.
Revised: For example, for certain isotropic data ( e.g. , Gaussian), we show that a large step size tends to increase the biases of all neurons.

Original: We also demonstrate on the MNIST dataset the tightness of our stability bound, and that it predicts well the dependence of the stability and generalization performance on the step size.
Revised: We also demonstrate on the MNIST dataset the tightness of our stability bound, and that it predicts well the dependence of the stability and generalization performance on the step size.

Original:  B ACKGROUND : M INIMA STABILITY OF SGD
Revised:  B ACKGROUND : M INIMA STABILITY OF SGD

Original: In this section we give a brief survey on minima stability.
Revised: In this section we give a brief survey on minima stability.

Original: Let us consider the problem of minimizing an empirical loss using SGD.
Revised: Let us consider the problem of minimizing an empirical loss using SGD.

Original: We are interested in the typical regime of overparameterized models.
Revised: We are interested in the typical regime of overparameterized models.

Original: In this setting, there exist multiple global minimizers of the loss.
Revised: In this setting, there exist multiple global minimizers of the loss.

Original: Yet, SGD cannot stably converge to any minimum.
Revised: Yet, SGD cannot stably converge to any minimum.

Original: The stability of a minimum is associated with the dynamics of SGD in its vicinity.
Revised: The stability of a minimum is associated with the dynamics of SGD in its vicinity.

Original: Specifically, a minimum is said to be stable if once SGD arrives near the minimum, it stays in its vicinity.
Revised: Specifically, a minimum is said to be stable if once SGD arrives near the minimum, it stays in its vicinity.

Original: If SGD repels from the minimum, then we say that it is unstable.
Revised: If SGD repels from the minimum, then we say that it is unstable.

Original: Formally, let ℓ j : R d → R be differentiable almost everywhere for all j ∈ [ n ] .
Revised: Formally, let ℓ j : R d → R be differentiable almost everywhere for all j ∈ [ n ] .

Original: Here we consider a loss function L and its stochastic analogue,
Revised: Here we consider a loss function L and its stochastic analogue,

Original: [Equation]
Revised: [Equation]

Original: where B t is a batch of size B sampled at iteration t .
Revised: where B t is a batch of size B sampled at iteration t .

Original: We assume that the batches { B t } are drawn uniformly from the dataset, independently across iterations.
Revised: We assume that the batches { B t } are drawn uniformly from the dataset, independently across iterations.

Original: SGD’s update rule is given by
Revised: SGD’s update rule is given by

Original: [Equation]
Revised: [Equation]

Original: where η is the step size.
Revised: where η is the step size.

Original: Analyzing the full dynamics of this system is intractable in most cases.
Revised: Analyzing the full dynamics of this system is intractable in most cases.

Original: Therefore, several works studied the behavior of this system near minima using linearized dynamics (Wu et al., 2018; Ma & Ying, 2021; Nar & Sastry, 2018; Mulayoff et al., 2021), which is a common practice for characterizing the stability of nonlinear systems.
Revised: Therefore, several works studied the behavior of this system near minima using linearized dynamics (Wu et al., 2018; Ma & Ying, 2021; Nar & Sastry, 2018; Mulayoff et al., 2021), which is a common practice for characterizing the stability of nonlinear systems.

Original: Definition 1 (Linear stability) .
Revised: Definition 1 (Linear stability) .

Original: Let θ ∗ be a twice differentiable minimum of L .
Revised: Let θ ∗ be a twice differentiable minimum of L .

Original: Consider the linearized stochastic dynamical system
Revised: Consider the linearized stochastic dynamical system

Original: [Equation]
Revised: [Equation]

Original: Then θ ∗ is ε linearly stable if for any θ 0 in the ε -ball B ε ( θ ∗ ) , we have lim sup t →∞ E
Revised: Then θ ∗ is ε linearly stable if for any θ 0 in the ε -ball B ε ( θ ∗ ) , we have lim sup t →∞ E

Original: [ ∥ θ t − θ ∗ ∥ ] ≤ ε .
Revised: [ ∥ θ t − θ ∗ ∥ ] ≤ ε .

Original: Namely, a minimum is ε linearly stable if once θ t enters an ε -ball around the minimum, it ends up at a distance no greater than ε from it in expectation.
Revised: Namely, a minimum is ε linearly stable if once θ t enters an ε -ball around the minimum, it ends up at a distance no greater than ε from it in expectation.

Original: Under mild conditions, any stable minimum of the nonlinear system is also linearly stable (Vidyasagar, 2002, p. 268).
Revised: Under mild conditions, any stable minimum of the nonlinear system is also linearly stable (Vidyasagar, 2002, p. 268).

Original: We have the following condition.
Revised: We have the following condition.

Original: Lemma 1 (Necessary condition for linear stability (Mulayoff et al., 2021, Lemma 1)) .
Revised: Lemma 1 (Necessary condition for linear stability (Mulayoff et al., 2021, Lemma 1)) .

Original: Consider
Revised: Consider

Original: SGD with step size η , where batches are drawn uniformly from the training set, independently across iterations.
Revised: SGD with step size η , where batches are drawn uniformly from the training set, independently across iterations.

Original: If θ ∗ is an ε linearly stable minimum of L , then 
Revised: If θ ∗ is an ε linearly stable minimum of L , then [Equation]

Original: [Equation]
Revised: [Equation]

Original: This condition states that stable minima of SGD are flat w.r.t.
Revised: This condition states that stable minima of SGD are flat w.r.t.

Original: the step size.
Revised: the step size.

Original: Although this result was proved for the linearized dynamics, it was observed to hold also in practice, where the full nonlinear dynamics apply.
Revised: Although this result was proved for the linearized dynamics, it was observed to hold also in practice, where the full nonlinear dynamics apply.

Original: Particularly, much empirical evidence on real-world neural-network training (Cohen et al., 2020; Gilmer et al., 2022) points out that GD and SGD converge only to linearly stable minima, i.e., minima satisfying (6).
Revised: Particularly, much empirical evidence on real-world neural-network training (Cohen et al., 2020; Gilmer et al., 2022) points out that GD and SGD converge only to linearly stable minima, i.e. , minima satisfying (6).

Original: More on dynamical stability and its interaction with common practices ( e.g. , learning rate decay, absence of ε and B in Lemma 1 result, etc. )
Revised: More on dynamical stability and its interaction with common practices ( e.g. , learning rate decay, absence of ε and B in Lemma 1 result, etc. )

Original: in App. A.
Revised: in App. A.

Original:  L ARGE STEP SIZE BIASES TO SMOOTH FUNCTIONS
Revised:  L ARGE STEP SIZE BIASES TO SMOOTH FUNCTIONS

Original: Consider the set of multivariate functions over R d that can be implemented by a single hidden-layer
Revised: Consider the set of multivariate functions over R d that can be implemented by a single hidden-layer

Original: ReLU network with k neurons,
Revised: ReLU network with k neurons,

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) denotes the ReLU activation function.
Revised: where σ ( · ) denotes the ReLU activation function.

Original: Each f ∈ F k is a piecewise linear function with at most k knots 2 .
Revised: Each f ∈ F k is a piecewise linear function with at most k knots 2 .

Original: Given some training set { x j , y j } n j =1 , we are interested in functions that globally minimize the quadratic loss
Revised: Given some training set { x j , y j } nj =1 , we are interested in functions that globally minimize the quadratic loss

Original: [Equation]
Revised: [Equation]

Original: Definition 2 (Solution) .
Revised: Definition 2 (Solution) .

Original: A function f ∈ F k is a ‘ solution ’ if L ( f ) = 0 , i.e., f ( x j ) =
Revised: A function f ∈ F k is a ‘ solution ’ if L ( f ) = 0 , i.e., f ( x j ) =

Original: y j ∀ j ∈ [ n ] .
Revised: y j ∀ j ∈ [ n ] .

Original: We focus on the overparameterized regime ( kd > n ) where there are multiple solutions.
Revised: We focus on the overparameterized regime ( kd > n ) in which there exist multiple solutions.

Original: We want to study the properties of solutions which correspond to stable minima of SGD.
Revised: We want to study the properties of solutions which correspond to stable minima of SGD.

Original: However, a key challenge is that any solution f ∈ F k typically has infinitely many different parameterizations.
Revised: However, a key challenge is that any solution f ∈ F k typically has infinitely many different parameterizations.

Original: In other words, there are various parameter vectors
Revised: In other words, there are various parameter vectors

Original: [Equation]
Revised: [Equation]

Original: that can implement the same function f .
Revised: that can implement the same function f .

Original: Different parameterizations correspond to different minima, which may have different Hessian eigenvalues.
Revised: Different parameterizations correspond to different minima, which may have different Hessian eigenvalues.

Original: Therefore, for a given step size η , some parameterizations of f may be stable while others may not.
Revised: Therefore, for a given step size η , some parameterizations of f may be stable while others may not.

Original: Thus, to determine whether SGD can stably converge to a solution f , we need to consider whether there exists some stable minimum θ , which corresponds to a parametrization of f .
Revised: Thus, to determine whether SGD can stably converge to a solution f , we need to check whether there exists some stable minimum θ , which corresponds to a parametrization of f .

Original: Hence, we use the following definition.
Revised: We therefore use the following definition.

Original: Definition 3 (Stable solution) .
Revised: Definition 3 (Stable solution) .

Original: A solution f ∈ F k is said to be stable for step size η if there exists a minimum θ ∗ of the loss that corresponds to f , where θ ∗ is linearly stable for SGD with step size η .
Revised: A solution f ∈ F k is said to be stable for step size η if there exists a minimum θ ∗ of the loss that corresponds to f , where θ ∗ is linearly stable for SGD with step size η .

Original: In light of these definitions, we characterize stable solutions through the inverse of the dual Radon transform,denoted by ( R ∗ ) − 1 (see App. C), and the Laplacian ∆ f, which should be interpreted in the weak sense ( i.e., as a sum of weighted Dirac delta functions, see App. D).
Revised: The next theorem characterizes stable solutions using the Radon transform R (see App. C) and the Laplace operator ∆ . Particularly, we use the inverse of the dual Radon transform, ( R ∗ ) − 1 , and interpret ∆ f in the weak sense, i.e. , as a sum of weighted Dirac delta functions (see App. D).

Original: Theorem 1 (Properties of stable solutions) .
Revised: Theorem 1 (Properties of stable solutions) .

Original: Let f be a linearly stable solution for SGD with step size η .
Revised: Let f be a linearly stable solution for SGD with step size η .

Original: Assume that the knots of f do not coincide with any training point.
Revised: Assume that the knots of f do not coincide with any training point.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ R ,g is the stability norm, defined as
Revised: where ∥·∥ R ,g is the stability norm, defined as

Original: [Equation]
Revised: [Equation]

Original: (cid:90) S d − 1 × R (cid:12)(cid:12)(cid:2) ( R ∗ ) − 1 ∆ f (cid:3) ( v , b ) (cid:12)(cid:12) g ( v , b )d s ( v )d b, (11) and g ( v , b ) ≜ min (cid:0) ˜g ( v , b ) , ˜ g
Revised: and g ( v , b ) ≜ min (cid:0) ˜

Original:  ( − v , − b ) (cid:1) is a non-negative weighting function, with ˜ g given by[Equation] b [Equation]
Revised: g ( v , b ) , ˜ g ( − v , − b ) (cid:1) is a non-negative weighting function, with ˜ g given by

Original: (cid:114)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:104)

Original: [Equation]

Original: (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)(cid:13)(cid:13)(cid:13)

Original: [Equation] Here X is a random vector drawn from the dataset’s distribution (i.e., sampled uniformly from { x j } ).
Revised: Here X is a random vector drawn from the dataset’s distribution (i.e., sampled uniformly from { x j } ).

Original: This theorem shows that the step size constraints the obtained solution through the stability norm.
Revised: This theorem, whose proof is provided in App. E, shows that the step size constrains the stability norm of the solution.

Original: The constraint becomes more strict as the step size increases.
Revised: Notably, the constraint becomes stricter as the step size increases.

Original: While this result depends only on the step size,considering other hyper-parameters ( e.g.  batch size, initialization) might improve this bound.
Revised: Before interpreting this result, let us note that although it depends only on the step size, other hyper-parameters ( e.g. , batch size, initialization) may potentially improve the bound.

Original: However, as we discuss in App.
Revised: Yet, as we discuss in App.

Original: A,for practical settings the effect of other hyper-parameters seems secondary.
Revised: A, the effect of other hyper-parameters seems secondary in practical settings.

Original: The implications of Theorem 1 can be understood in primal space and in Radon space.
Revised: The implications of Thm. 1 can be understood in primal space and in Radon space.

Original: In the following sections we discuss both interpretations and give examples.
Revised: In the following, we discuss both interpretations and give examples.

Original: 3. P RIMAL SPACE INTERPRETATIONOF T HEOREM
Revised: 3. P RIMAL SPACE INTERPRETATION

Original: Theorem 1 is stated in Radon space, which may be difficult to interpret.
Revised: Theorem 1 is stated in Radon space, which may be difficult to interpret.

Original: However, in some cases it can also be interpreted in primal space, by deriving an alternative form for the stability norm ∥ · ∥ R ,g .
Revised: However, in some cases it can also be interpreted in primal space, by deriving an alternative form for the stability norm ∥ · ∥ R ,g .

Original: Specifically, in App.
Revised: Specifically, in App.

Original: G we show that if g is piecewise continuous and L 1 -integrable 4 , then for all f ∈ F k  we have
Revised: G we show that if g is piecewise continuous and L 1 -integrable 4 , then for all f ∈ F k and ρ = R − 1 g we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Figure]

Original: where ρ = R − 1 g .

Original: In this presentation of the stability norm, ρ is not necessarily non-negative.
Revised: In this presentation of the stability norm, ρ is not necessarily non-negative.

Original: Nevertheless, all its hyper-plane integrals are non-negative sincefrom its definition, R ρ = g ≥ 0 .
Revised: Nevertheless, all its hyper-plane integrals are non-negative, since R ρ = g ≥ 0 .

Original: Thus,by (13), the stability norm can be interpreted as a non-negative linear combination of hyperplane integrals of ρ along the knots of f .
Revised: Thus, the stability norm can be interpreted as a non-negative linear combination of hyper-plane integrals of ρ along the knots of f .

Original: This is visualized in Fig.
Revised: This is visualized in Fig.

Original: Hence, Theorem 1 combined with (13) implies that, with large η , stable solutions correspond to smooth functions, i.e., functions with small weighted | ∆ f | . [Figure]
Revised: Hence, Thm. 1 combined with (13) implies that the larger the step size η , the smoother the solution becomes.

Original: 3. R ADON SPACE INTERPRETATIONOF T HEOREM
Revised: 3. R ADON SPACE INTERPRETATION

Original: Another interesting interpretation of Theorem 1 can be derived in Radon space.
Revised: Another interesting interpretation of Thm. 1 can be derived in Radon space.

Original: First, let us examine how the weight function g ( v , b ) behaves as a function of b .
Revised: First, let us examine how the weight function g ( v , b ) behaves as a function of b .

Original: For every fixed v , the function g ( v , · ) has a finite support, [min j { x ⊤ j v } , max j { x ⊤ j v } ] .
Revised: For every fixed v , the function g ( v , · ) has a finite support, [min j { x ⊤ j v } , max j { x ⊤ j v } ] .

Original: Moreover, g ( v , · ) typically has most of its mass concentrated around the center of the distribution of the projected data points { x ⊤ j v } , and it decays towards the endpoints (see e.g.  Fig. 3).
Revised: Moreover, g ( v , · ) typically has most of its mass concentrated around the center of the distribution of the projected data points { x ⊤ j v } , and it decays towards the endpoints (see e.g. , Fig. 3).

Original: Next, let us interpret how the term ( R ∗ ) − 1 ∆ f behaves.
Revised: Next, let us interpret how the term ( R ∗ ) − 1 ∆ f behaves.

Original: For a single hidden-layer ReLU network, ( R ∗ ) − 1 ∆ f is a sum of Dirac deltas.
Revised: For a single hidden-layer ReLU network, ( R ∗ ) − 1 ∆ f is a sum of Dirac deltas.

Original: Specifically, as shown in (Ongie et al., 2020), if f is a function of the form f ( x ) = (cid:80) ki
Revised: Specifically, as shown in (Ongie et al., 2020), if f is a function of the form f ( x ) = (cid:80) ki

Original: =1 a i σ ( v ⊤ i x − b i )
Revised: =1 a i σ ( v ⊤ i x − b i )

Original: + c with ∥ v i ∥ 2 = 1 for all i ∈ [ k ] , then (see App.
Revised: + c with ∥ v i ∥ 2 = 1 for all i ∈ [ k ] , then (see App.

Original: F.3)
Revised: F.3)

Original: [Equation] (cid:88)
Revised: ( R ∗

Original: [Equation]
Revised: [Equation]

Original: (14) where ∆ f is the (distributional) Laplacian of f , and δ ( v ,b ) denotes a Dirac delta centered at ( v , b ) ∈
Revised: where ∆ f is the (distributional) Laplacian of f , and δ ( v ,b ) denotes a Dirac delta centered at ( v , b ) ∈

Original: S d − 1 × R .
Revised: S d − 1 × R .

Original: Thus, as we show in Appendix F.3, we can  define a parameter space representation for the stability norm as 
Revised: We can thus define a parameter space representation for the stability norm as (see App. F.3)

Original: [Equation]
Revised: [Equation]

Original: [Equation] For a representation of  f in which the ReLUunits have knots that do not align, i.e., there is one Dirac function for each ReLU unit, both of the stability norm representations are equivalent,
Revised: Generally, this parametric representation of the stability norm satisfies ∥ f ∥ R ,g ≤ S θ , where equality happens whenever the ReLU knots of the representation do not coincide ( i.e. , there is one Dirac function for each ReLU unit).

Original: i.e., ∥ f ∥ R ,g = S θ . For a representation in which the ReLU units knots do align, we have that ∥ f ∥ R ,g ≤ S θ , but we still have, as in (10), that (see App.
Revised: Yet, this parametric view of the stability norm also obeys (see App.

Original: F.3)
Revised: F.3)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence, larger step sizes η push S θ to be smaller, and from (15) we see that | a i | will tend to be small.
Revised: Hence, larger step sizes η push S θ to be smaller, and from (15) we see that | a i | will tend to be small.

Original: Also, since g ( v , · ) typically decays towards the boundary of its support, this pushes the biases of the neurons, b i , away from the center of the distribution.
Revised: Also, since g ( v , · ) typically decays towards the boundary of its support, this pushes the neurons’ biases, b i , away from the center of the distribution.

Original: The resulting effect is that the predictor function f becomes flatter, especially near the center of the distribution.
Revised: The resulting effect is that the predictor function f becomes flatter, especially near the center of the distribution.

Original: This is illustrated in Figs.
Revised: This is illustrated in Figs.

Original: 1 and 4(b).
Revised: 1 and 4(b).

Original: 3. P ROOF IDEA
Revised: 3. E XAMPLES

Original: Theorem 1 is a result of two properties of twice-differentiable minima. First, we show in Lemmain the appendix that these minima satisfy λ max ( ∇ 2 θ L ) ≥ 1 + 2 ∥ f ∥ R ,g  .Second, we know from
Revised: Earlier we introduced two interpretations for the stability norm ∥·∥ R ,g : one in primal space, which uses the weight function ρ , and one in Radon space, which uses the weight function g .

Original: Lemma 1 that stable minima satisfy λ max ( ∇ 2 θ L ) ≤ 2 /η . Together, these properties imply that 2 /η ≥ 1 + 2 ∥ f ∥ R ,g , from which we get the result of the theorem, ∥ f ∥ R ,g ≤ 1 /η − 1 / 2 .
Revised: In this section, we compute g and ρ for two toy examples, for which (13) holds.

Original: Note that twice-differentiable minima correspond to functions whose knots do not coincide with any training point.

Original: Although we prove our result only for such functions, we observed that in practice the condition λ max ( ∇ 2 θ L ) ≤ 2 /η is always met around minima to which SGD converges (see Sec.

Original: 6 and App.

Original: A).

Original: For full derivation of the theorem see App.

Original: E.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: E XAMPLES

Original: Earlier we introduced two interpretations for the stability norm ∥·∥ R ,g : one in primal space, which uses the weight function ρ , and one in Radon space, which uses the weight function g .

Original: In this section, we compute g and ρ for two toy examples, for which (13) holds.

Original: Example 1: Two data points in R 2 .
Revised: Example 1: Two data points in R 2 .

Original: Assume the dataset contains two points, x 1 = (1 , 0) and x 2 = ( − 1 , 0) .
Revised: Assume the dataset contains two points: x 1 = (1 , 0) and x 2 = ( − 1 , 0) .

Original: In this case we can analytically calculate g and ρ (see App.
Revised: In this case we can analytically calculate g and ρ (see App.

Original: H.1).
Revised: H.1).

Original: Figure 3 depictsthe values of these functions.
Revised: Figure 3 depicts these functions.

Original: Here, ρ has singularities at x 1 , x 2 and at the origin.
Revised: Here, ρ has singularities at x 1 , x 2 and at the origin.

Original: Yet, despite these singularities, all line integrals of ρ are finite and thus the expression (cid:82) R d | ∆ f | ( x ) ρ ( x )d x is well-defined.
Revised: Yet, despite these singularities, all line integrals of ρ are finite and thus the expression (cid:82) R d | ∆ f | ( x ) ρ ( x )d x is well-defined.

Original: Moreover, while ρ can take negative values, all its line integrals take positive values.
Revised: Moreover, while ρ takes negative values, all its line integrals take positive values.

Original: Example 2: Isotropic distribution.
Revised: Example 2: Isotropic distribution.

Original: Suppose the data is isotropically distributed, i.e., P ( X ⊤ v > b ) does not depend on the direction of v .
Revised: Suppose the data is isotropically distributed, i.e. , P ( X ⊤ v > b ) does not depend on the direction of v .

Original: Thus g is independent of v , which implies that ρ = R − 1 g is a radial function.
Revised: Thus g is independent of v , which implies that ρ = R − 1 g is a radial function.

Original: In App.
Revised: In App.

Original: H.2 we give an analytic expression to g for any isotropic distribution.
Revised: H.2 we give an analytic expression for g for any isotropic distribution.

Original: In the special case where X ∼ N ( 0 , I ) , g ( v , · ) decays monotonically with b , and thus, as discussed in Sec. 3.2, large step sizes will tend to increase the biases of all neurons.
Revised: In the special case of X ∼ N ( 0 , I ) , g ( v , · ) decays monotonically with b , and thus, as discussed in Sec. 3.2, large step sizes will tend to increase the biases of all neurons.

Original: Additionally, we show in
Revised: Additionally, we show in

Original: App.
Revised: App.

Original: H.2 that for 2D data, ρ is positive and strictly decreasing in ∥ x ∥ , and it satisfies the asymptotics ρ ( x )
Revised: H.2 that for 2D data, ρ is positive and strictly decreasing in ∥ x ∥ , and it satisfies the asymptotics ρ ( x )

Original: =
Revised: =

Original: O (log( ∥ x ∥ )) as ∥ x ∥ → 0 , and ρ ( x ) =
Revised: O (log( ∥ x ∥ )) as ∥ x ∥ → 0 , and ρ ( x ) =

Original: O ( ∥ x
Revised: O ( ∥ x

Original: ∥ − 1 ) as ∥ x ∥ → ∞ .
Revised: ∥ − 1 ) as ∥ x ∥ → ∞ .

Original: Figure 3 visualizes g and ρ for 2D Gaussian data.
Revised: Figure 3 visualizes g and ρ for two dimensional Gaussian data.

Original:  S TABILITY LEADS TO DEPTH SEPARATION
Revised:  S TABILITY LEADS TO DEPTH SEPARATION

Original: Single hidden-layer neural networks are universal approximators, i.e., they can approximate arbitrarily well any continuous function over compact sets (Pinkus, 1999).
Revised: Single hidden-layer neural networks are universal approximators, i.e. , they can approximate arbitrarily well any continuous function over compact sets (Pinkus, 1999).

Original: However, some of these approximations may correspond to unstable minima that are virtually unreachable by training via SGD.
Revised: However, some of these approximations may correspond to unstable minima that are virtually unreachable by training via SGD.

Original: To understand what is the effective approximation power of these nets, we need to account for functions which have stable approximations.
Revised: To understand what is the effective approximation power of neural networks, we need to identify the class of functions that have stable approximations.

Original: Here we have the following (see proof in App. I.1).
Revised: We have the following (see proof in App. I.1).

Original: Proposition 1.
Revised: Proposition 1.

Original: Let X⊂ R d be the interior of the convex hull of theset of training points, andlet f : X → R be any function.
Revised: Let X be the interior of the convex hull of the training points, and f : X → R be any function.

Original: Suppose there exists a sequence{ f k } ∞ k =1 of single hidden-layer ReLU networks  with bounded stability norm that converges to f in L 1-norm over X .
Revised: Suppose there exists a sequence of single hidden-layer ReLU networks { f k } with bounded stability norm that converges to f in L 1 over X .

Original: Then ∥ f ∥ R ,g is finite, and lim k →∞ ∥ f k ∥
Revised: Then ∥ f ∥ R ,g is finite, and lim k →∞ ∥ f k ∥

Original: R ,g = ∥
Revised: R ,g = ∥

Original: f ∥ R ,g .
Revised: f ∥ R ,g .

Original: Let { f k }∞ k =1 be a convergent sequence of stable solutions, i.e., ∀ f k ∈ F k : ∥ f k ∥
Revised: Let { f k } be a convergent sequence of stable solutions with a growing number of knots, i.e. , ∀ f k ∈ F k : ∥ f k ∥

Original: R ,g ≤ 1 /η − 1 /(see Th. 1).
Revised: R ,g ≤ 1 /η − 1 / 2 (see Thm. 1).

Original: Then, by this proposition  we have that the limit function f also satisfies this inequality.
Revised: Then, by the proposition above we have that the limit function f also satisfies this inequality.

Original: Therefore, the effective set of functions that can be approximated arbitrarily well by single hiddenlayer ReLU networks includes only continuous functions f that satisfy  ∥
Revised: Therefore, the effective class of functions that can be approximated arbitrarily well by single hidden-layer ReLU networks includes only continuous functions f that satisfy the stability condition ∥

Original: f ∥ R ,g ≤ 1 /η − 1 / 2 .
Revised: f ∥ R ,g ≤ 1 /η − 1 / 2 .

Original: As the step size decreases, more functions satisfy this condition, suggesting that more functions can be stably approximated by the network.
Revised: As the step size decreases, more functions satisfy this condition, suggesting that more functions can be stably approximated by single hidden-layer ReLU networks.

Original: Surprisingly, there exists at least one continuous function p which does not satisfy the stability condition for any positive step size (see proof in App. I.2).
Revised: Surprisingly, there exists at least one continuous function p that has ∥ p ∥ R ,g = ∞ and therefore does not satisfy the stability condition for any positive step size (see proof in App. I.2).

Revised: Therefore from Prop. 1 and Thm. 1, this function cannot be approximated arbitrarily well by single hidden-layer ReLU networks trained with a non-vanishing step size.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let p ( x ) =
Revised: Assume the input dimension d ≥ 2 , and let p ( x ) =

Original: σ (1 − ∥ x ∥ 1 ) .
Revised: σ (1 −∥ x ∥ 1 ) .

Original: Assume the support of p is contained in the interior of the convex hull of the training points.
Revised: Suppose the support of p is contained in the interior of the convex hull of the training points.

Original: Then ∥ p ∥ R ,g =+ ∞ .
Revised: Then ∥ p ∥ R ,g = ∞ .

Original: In words, the function p has unbounded stability norm, and thus cannot be approximated arbitrarily well by single hidden-layer ReLU networks trained with a non-vanishing step size.

Original: Intriguingly, this function has an implementation by a finite two hidden-layer network, which can correspond to a stable solution.
Revised: Intriguingly, this function does have an implementation as a finite-width two hidden-layer network, p ( x ) =

Original: Specifically, the two hidden-layer network implementation of p is given by σ (1 − (cid:80) di =1 (cid:0) σ ( x i ) + σ ( − x i ) (cid:1) ) .
Revised: σ (1 − (cid:80) di =1 ( σ ( x i ) + σ ( − x i ))) , which is a stable solution for a fixed step size.

Original: In App.
Revised: Indeed, in App.

Original: I.3 we show that for an appropriate choice of step size, GD is able to converge to this implementation.
Revised: I.3 we demonstrate that for an appropriate choice of η , GD is able to converge to this implementation.

Original: Thus, we have a depth separation result: the function p cannot be approximated by stable minima of one hidden-layer networks trained with a non-vanishing step size, yet with two hidden-layers, GD can converge to this function with a fixed step size.
Revised: Thus, we have a depth separation result: the function p cannot be approximated by stable minima of one hidden-layer networks trained with a non-vanishing step size, yet with two hidden-layers, GD can converge to this function with a fixed step size.

Original:  S HALLOW NETWORK APPROXIMATIONS OF SMOOTH FUNCTIONS
Revised:  S HALLOW NETWORK APPROXIMATIONS OF SMOOTH FUNCTIONS

Original: In Sec.
Revised: In Sec.

Original: 4 we showed that stable single-hidden layer ReLU networks are not universal approximators.
Revised: 4 we showed that stable single-hidden layer ReLU networks are not universal approximators.

Original: In this section we give an approximation guarantee under smoothness assumptions.
Revised: In this section we give an approximation guarantee under smoothness assumptions.

Original: That is, we show that if a function is sufficiently smooth, then it can be approximated arbitrarily well using single hidden-layer networks that correspond to stable solutions of GD.
Revised: That is, we show that if a function is sufficiently smooth, then it can be approximated arbitrarily well using single hidden-layer networks that correspond to stable solutions of GD.

Original: Let W d +1 , 1 w ( R d ) denote the weighted Sobolev space of all functions whose weak partial derivatives up to order d +1 are bounded in a weighted L 1 -norm ∥·∥ 1 ,w with weight function w ( x ) :
Revised: Let W d +1 , 1 w ( R d ) denote the weighted Sobolev space of all functions whose weak partial derivatives up to order d +1 are bounded in a weighted L 1 -norm ∥·∥ 1 ,w with weight function w ( x ) :

Original: = R ∗ [1+ | b | ]( x ) .
Revised: = R ∗ [1+ | b | ]( x ) .

Original: Let ∥ · ∥ W d
Revised: Let ∥ · ∥ W d

Original: +1 , 1 w ( R d ) denote the corresponding Sobolev norm 
Revised: +1 , 1 w ( R [Equation] ) denote the corresponding Sobolev norm [Equation]

Original: [Equation]
Revised: [Equation]

Original: where β is a multi-index.
Revised: where β is a multi-index.

Original: For technical convenience, we restrict ourselves to odd input dimensions d only.
Revised: For technical convenience, we restrict ourselves to odd input dimensions d only.

Original: Our results use the “ R -norm” ∥ · ∥ R introduced by Ongie et al.
Revised: Our results use the “ R -norm” ∥ · ∥ R introduced by Ongie et al.

Original: (2020) (see Sec.
Revised: (2020) (see Sec.

Original: 7 for details), and the stability norm ∥ · ∥ R , ˆ g with a different weight function ˆ g defined below (see proof in App.
Revised: 7 for details), and the stability norm ∥ · ∥ R , ˆ g with a different weight function ˆ g defined below (see proof in App.

Original: L).
Revised: L).

Original: Proposition 3.
Revised: Proposition 3.

Original: Assume d is odd and let f ∈ W d
Revised: Assume d is odd and let f ∈ W d

Original: +1 , 1 w ( R d ) .
Revised: +1 , 1 w ( R d ) .

Original: Then, there exists a sequence of single hidden-layer ReLU network functions { f k }∞ k =1 such that f k ∈ F k converges to f in L 1-norm over any compact subset K ⊂ R d , i.e., lim k →∞ (cid:82) K | f k ( x ) − f ( x ) | d x = 0 , and satisfies the bounds ∥ f k ∥
Revised: Then, there exists a sequence of single hidden-layer ReLU network functions { f k } such that f k ∈ F k converges to f in L 1 over any compact subset K ⊂ R d , i.e., lim k →∞ (cid:82) K | f k ( x ) − f ( x ) | d x = 0 , and satisfies the bounds ∥ f k ∥

Original: R + ∥ f k ∥ R , ˆ g ≤ c d, ˆ g ∥ f∥ W
Revised: R + ∥ f k ∥ R , ˆ g ≤ c d, ˆ g ∥ f

Original: [Equation]

Original:  +1 , 1 w ( R d ) for all k , where
Revised: ∥ W d +1 , 1 w ( R d ) for all k , where

Original: [Equation]
Revised: [Equation]

Original: and c d, ˆ g is a constant depending on d and ˆ g but independent of f .
Revised: and c d, ˆ g is a constant depending on d and ˆ g but independent of f .

Original: Here X is drawn uniformly at random from the dataset.
Revised: Here X is drawn uniformly at random from the dataset.

Original: This proposition asserts that for any f ∈ W d +1 , 1 w ( R d ) there exists a sequence of single hidden-layer
Revised: This proposition shows that for any f ∈ W d +1 , 1 w ( R d ) there exists a sequence of single hidden-layer

Original: ReLU network approximations { f k } for which {∥ f k ∥ R } and {∥ f k ∥ R , ˆ g } are bounded.
Revised: ReLU network approximations { f k } for which {∥ f k ∥ R } and {∥ f k ∥ R , ˆ g } are bounded.

Original: To prove that these functions can have stable implementations for GD, we need to show that if both the stability norm and R -norm are bounded (in function space), this implies bounded sharpness 6 in parameter space.
Revised: To prove that these functions can have stable parameterizations for GD, we need to show that if both the stability norm and R -norm are bounded (a function space property), then there exists a corresponding minimum with bounded sharpness 6 in parameter space.

Original: To this end, we derived a novel upper bound on the minimal sharpness that agiven solution can have among its different implementations (see proof in App. K).
Revised: To this end, we derive an upper bound on the minimal sharpness of a solution f among its different parameterizations in terms of the stability norm and the R -norm (see proof in App. K).

Original: Lemma 2.
Revised: Lemma 2.

Original: Let f ∈ F k be a solution for which the knots do not coincide with any training point.
Revised: Let f ∈ F k be a solution for which the knots do not coincide with any training point.

Original: Then there exists an implementation θ ∗ corresponding to f such that
Revised: Then there exists an implementation θ ∗ corresponding to f such that

Original: [Equation]
Revised: [Equation]

Original: Here X is drawn uniformly at random from the dataset, and Σ X is the covariance matrix of X .
Revised: Here X is drawn uniformly at random from the dataset, and Σ X is the covariance matrix of X .

Original: Combining Prop. 3 and Lemma 2 we get that any f ∈ W d +1 , 1 w ( R d ) can be approximated arbitrarily well by a sequence of stable solutions for GD with a fixed step size η .
Revised: Combining Prop. 3 and Lemma 2 we get that any f ∈ W d +1 , 1 w ( R d ) can be approximated arbitrarily well by a sequence of stable solutions for GD with a fixed step size η .

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose the input dimension d is odd, and let f ∈ W d
Revised: Suppose the input dimension d is odd, and let f ∈ W d

Original: +1 , 1 w ( R d ) .
Revised: +1 , 1 w ( R d ) .

Original: Then, there exist η >and a sequence of single hidden-layer ReLU networks { f k }∞ k =1 such that f k ∈ F k converges to f in 
Revised: Then, there exist η >and a sequence of single hidden-layer ReLU network functions { f k } such that f k ∈ F k converges to f in L 1 over any compact subset K ⊂ R d , and every f k is stable for GD with step size η . This theorem state that any sufficiently smooth function can be stably approximated in the limit of infinitely many neurons. We can also use Lemma 2 to guarantee the stability of solutions in the finite case. Since λ max ≤ 2 /η is a sufficient condition for stability in GD, we have the following.

Revised: Theorem 3.

Revised: Let f ∈ F k be a solution for which the knots do not coincide with any training point.

Original: L 1 -norm over any compact subset K ⊂ R d and every fk is  stable  for GD with step size η .
Revised: If [Equation] [Equation] then f is a stable solution for GD with step size η .

Revised: Theorem 3 complements Thm. 1, as it gives a sufficient condition for stability in function space.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now demonstrate our theoretical results.
Revised: We now demonstrate our theoretical results.

Original: We start with a regression task on synthetic data.
Revised: We start with a regression task on synthetic data.

Original: Here, we drew n = 100 pairs ( x j , y j ) in R 20 × R from the standard normal distribution to serve as our training set.
Revised: Here, we drew n = 100 pairs ( x j , y j ) in R 20 × R from the standard normal distribution to serve as our training set.

Original: We fit a single hidden-layer ReLU network with k = 40 neurons to the data using GD with various step sizes (runs were stopped when the loss dropped bellow 10 − 8 ).
Revised: We fit a single hidden-layer ReLU network with k = 40 neurons to the data using GD with various step sizes (runs were stopped when the loss dropped bellow 10 − 8 ).

Original: For each minimum θ ∗ to which GD converged, we computed the loss’ sharpness, λ max ( ∇ 2 L ( θ ∗ )) , and our lower bound on the sharpness, 1 + 2 ∥
Revised: For each minimum θ ∗ to which GD converged, we computed the loss’ sharpness, λ max ( ∇ 2 L ( θ ∗ )) , and our lower bound on the sharpness, 1 + 2 ∥

Original: f
Revised: f

Original: ∥
Revised: ∥

Original: R ,g (Lemma 3 in the appendix).
Revised: R ,g (Lemma 3 in the appendix).

Original: Additionally, we numerically determined the sharpness of the flattest implementation for every solution.
Revised: Additionally, we numerically determined the sharpness of the flattest implementation for every solution.

Original: Figure 4(a) depicts the results for this experiment.
Revised: Figure 4(a) depicts the results for this experiment.

Original: The red line marks the border of the stable region, which is 2 /η .
Revised: The red line marks the border of the stable region, which is 2 /η .

Original: Namely, (S)GD cannot stably converge to a minimum whose sharpness is above this line.
Revised: Namely, (S)GD cannot stably converge to a minimum whose sharpness is above this line.

Original: The dashed yellow line shows the sharpness of the minima to which GD converged in practice.
Revised: The dashed yellow line shows the sharpness of the minima to which GD converged in practice.

Original: As can be seen, here GD converged at the edge of stability (the two lines coincide), a phenomenon discussed in (Cohen et al., 2020).
Revised: As can be seen, here GD converged at the edge of stability (the two lines coincide), a phenomenon discussed in (Cohen et al., 2020).

Original: The blue curve is the sharpness of the flattest implementation of each solution, while the the purple curve is our lower bound.
Revised: The blue curve is the sharpness of the flattest implementation of each solution (see App. A), while the the purple curve is our lower bound.

Original: Here we see that our bound is quite tight (blue vs. purple).
Revised: We see that our bound is quite tight (blue vs. purple).

Original: Furthermore, asexpected, as the step size increases the minima get flatter in parameter space (yellow curve), which translates to smoother predictors in function space (purple curve).
Revised: Furthermore, as the step size increases the minima get flatter in parameter space (yellow curve), which translates to smoother predictors in function space (purple curve).

Original: Additionally, we see from Fig. 4(b) that the norm of the bias vector bin (14) increases with the step size, as our theory predicts (Sec. 3.2).
Revised: Additionally, we see from Fig. 4(b) that the norm of the bias vector b increases with the step size, as our theory predicts (Sec. 3.2).

Original: Next, we present an experiment with binary classification on MNIST (LeCun, 1998) using SGD.
Revised: Next, we present an experiment with binary classification on MNIST (LeCun, 1998) using SGD.

Original: In this experiment we used two classes from MNIST, ‘ 0 ’ and ‘ 1 ’, with 256 samples each ( n = 512 ).
Revised: In this experiment we used n = 512 samples from two MNIST classes, ‘ 0 ’ and ‘ 1 ’.

Original: The classes were labeled as y = 1 and y = − 1 , respectively.
Revised: The classes were labeled as y = 1 and y = − 1 , respectively.

Original: For the validation set we usedimages from the remaining samples in each class.
Revised: For the validation set we used 4000 images from the remaining samples in each class.

Original: We trained a single hidden-layer ReLU network with k = 200 neurons using SGD with batch size B = 16 .
Revised: We trained a single hidden-layer ReLU network with k =neurons using SGD with batch size B = 16 , and the quadratic loss.

Original: At train time we used the quadratic loss.

Original: To perform classification at inference time, we thresholded the net’s output at 0 .
Revised: To perform classification at inference time, we thresholded the net’s output at 0 .

Original: We ran SGD until the loss dropped below 10 − 8 for 2000 consecutive epochs.
Revised: We ran SGD until the loss dropped below 10 −for 2000 consecutive epochs.

Original: Figure 5(a) shows the same indices as in the previous experiment.
Revised: Figure 5(a) shows the same indices as in the previous experiment.

Original: Here we see again that as the step size increases, the minima get flatter in parameter space (yellow curve), which translates to smoother predictors in function space (purple curve).
Revised: Here we see again that as the step size increases, the minima get flatter in parameter space (yellow curve), which translates to smoother predictors in function space (purple curve).

Original: Figure 5(b) shows the classification accuracy on the validation set, where we see that the network generalizes better as the step size increases, as past work showed, e.g.  (Keskar et al., 2016).
Revised: Figure 5(b) shows the classification accuracy on the validation set, where we see that the network generalizes better as the step size increases, as past work showed, e.g. , (Keskar et al., 2017). More experiments in App. N.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Dynamical stability analysis was applied to neural network training in several works.
Revised: Dynamical stability analysis was applied to neural network training in several works.

Original: In particular,
Revised: In particular,

Original: Nar & Sastry (2018) analyzed Lyapunov stability of one hidden-layer ReLU networks without bias .
Revised: Nar & Sastry (2018) analyzed Lyapunov stability of one hidden-layer ReLU networks without bias .

Original: They proved a bound on the network’s output which depends on the step size and training data and implies that the network’s output should be smaller for training samples with larger magnitude.
Revised: They proved a bound on the network’s output which depends on the step size and training data and implies that the network’s output should be smaller for training samples with larger magnitude.

Original: Mulayoff & Michaeli (2020) characterized the flattest minima for linear networks trained with the square loss. They showed that these minima have unique properties, e.g. , balancedness and layer
Revised: Mulayoff & Michaeli (2020) characterized the flattest minima for linear nets and showed that these minima have unique properties. Yet, in their setting all minima implement the same input-output

Original: [Figure]
Revised: [Figure]

Original: coupling.
Revised: function.

Original: However, in their setting all minima implement the same input-output function. Thus, their results only show that SGD is biased toward certain implementations of the same input-output function, whereas our result shows that SGD is biased toward certain input-output functions .
Revised: functions .

Original: Wu et al.
Revised: Wu et al.

Original: (2018) proved a sufficient condition for dynamical stability of SGD in expectation using second moment.
Revised: (2018) proved a sufficient condition for dynamical stability of SGD in expectation using second moment.

Original: Ma & Ying (2021) extended their result by showing  necessary and sufficient condition for dynamical stability in higher moments.
Revised: Ma & Ying (2021) extended their result by showing a necessary and sufficient condition for dynamical stability in higher moments.

Original: In addition, the authors combined this condition with the multiplicative structure of neural networks to prove an upper bound on the Sobolev seminorm of the model’s input-output function at stable interpolating solutions.
Revised: In addition, the authors combined this condition with the multiplicative structure of neural nets to prove an upper bound on the Sobolev seminorm of the model’s input-output function at stable interpolating solutions.

Original: However, this upper bound  depends on the norm of the first layer of the network, which in general can bearbitrarily large.
Revised: Their upper bound extends to deep nets, yet it depends on the norm of the first layer of the network, which in general can be large.

Original: Mulayoff et al.
Revised: Mulayoff et al.

Original: characterized the stable solutions of SGD for univariate single hidden-layer
Revised: characterized the stable solutions of SGD for univariate single hidden-layer

Original: ReLU networks with the square loss.
Revised: ReLU networks with the square loss.

Original: Our Theorem 1 is the natural extension of (Mulayoff et al., 2021,
Revised: Our Thm. 1 is the natural extension of (Mulayoff et al., 2021,

Original: Th. 1) to the multivariate case.
Revised: Thm. 1) to the multivariate case.

Original: To prove it, we combine the proof technique of lower bounding the top eigenvalue of the Hessian, used by Mulayoff et al.
Revised: To prove it, we combine the proof technique of lower bounding the top eigenvalue of the Hessian, used by Mulayoff et al.

Original: (2021), with the Radon transforms analysis used by Ongie et al.
Revised: (2021), with the Radon transforms analysis used by Ongie et al.

Original: Combining these techniques is not a priori trivial, since Radon transform was not used before for Hessian analysis.
Revised: Combining these techniques is not a priori trivial, since Radon transform was not used before for Hessian analysis.

Original: Also, it required several subtle steps that are not encountered in the univariate setting nor in (Ongie et al., 2020)
Revised: Also, it required several subtle steps that are not encountered in the univariate setting nor in (Ongie et al., 2020)

Original: ( e.g.  working with the inverse of the dual Radon transform to obtain the primal space representation).
Revised: ( e.g. , working with the inverse of the dual Radon transform to obtain the primal space representation).

Original: Ongie et al.
Revised: Ongie et al.

Original: (2020) studied the space of functions realizable as infinite-width single hidden-layer
Revised: (2020) studied the space of functions realizable as infinite-width single hidden-layer

Original: ReLU nets with bounded weights norm.
Revised: ReLU nets with bounded weights norm.

Original: Their settings assumes explicit regularization, i.e., min-norm solution, whereas here we derived our results for SGD without regularization, via implicit bias.
Revised: Their settings assumes explicit regularization, i.e. , min-norm solution, whereas here we derived our results for SGD without regularization, via implicit bias.

Original: On the technical level, they introduced the “ R -norm” ∥ · ∥ R that is closely related to the stability norm.
Revised: On the technical level, they introduced the “ R -norm” ∥ · ∥ R that is closely related to the stability norm.

Original: Particularly, ∥ · ∥ R = ∥ · ∥ R ,g , for g = 1 the constant 1 function.
Revised: Particularly, ∥ · ∥ R = ∥ · ∥ R ,g , for g = 1 the constant 1 function.

Original: They proved similar results of depth separation and approximation guarantees, shown here in Secs.
Revised: They proved similar results of depth separation and approximation guarantees, shown here in Secs.

Original: More related work in App.
Revised: More related work in App.

Original: B.
Revised: B.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Large step sizes are often used to improve generalization (Li et al., 2019).
Revised: Large step sizes are often used to improve generalization (Li et al., 2019).

Original: This work suggests an explanation to this practice.
Revised: This work suggests an explanation to this practice.

Original: Specifically, we showed that large step sizes lead to smaller stability norm and thus can bias toward smooth predictors in shallow multivariate ReLU networks.
Revised: Specifically, we showed that large step sizes lead to smaller stability norm and thus can bias towards smooth predictors in shallow multivariate ReLU networks.

Original: We find the smoothness measure depends on the data via specific functions g or ρ , and exemplify their properties.
Revised: We find the smoothness measure depends on the data via specific functions g or ρ , and exemplify their properties.

Original: Moreover, we studied the approximation power of ReLU networks that correspond to stable solutions.
Revised: Moreover, we studied the approximation power of ReLU networks that correspond to stable solutions.

Original: Although shallow networks are universal approximators, we proved that stable solutions of these networks are not.
Revised: Although shallow networks are universal approximators, we proved that stable solutions of these networks are not.

Original: Namely, there is a function that cannot be well-approximated by stable single hidden-layer ReLU networks trained with a non-vanishing step size.
Revised: Namely, there is a function that cannot be well-approximated by stable single hidden-layer ReLU networks trained with a non-vanishing step size.

Original: Yet we showed that the same function can be realized as a stable two hidden-layer network, leading to a depth separation result.
Revised: Yet we showed that the same function can be realized as a stable two hidden-layer network, leading to a depth separation result.

Original: This result can explain the success of deep models over shallow ones.
Revised: This result can explain the success of deep models over shallow ones.

Original: Finally, we gave approximation guarantees for stable shallow ReLU networks.
Revised: Finally, we gave approximation guarantees for stable shallow ReLU networks.

Original: In particular, we proved that any Sobolev function can be approximated arbitrarily well using GD with single hidden-layer ReLU networks. 
Revised: In particular, we proved that any Sobolev function can be approximated arbitrarily well using GD with single hidden-layer ReLU networks. A CKNOWLEDGMENTS

Revised: The research of RM was supported by the Planning and Budgeting Committee of the Israeli Council for Higher Education, and by the Andrew and Erna Finci Viterbi Graduate Fellowship.

Revised: GO was supported by NSF CRII award CCF-2153371.

Original: A A DDITIONAL DISCUSSION
Revised: The research of DS was funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research

Revised: Council Executive Agency (ERCEA).

Revised: Neither the European Union nor the granting authority can be held responsible for them.

Revised: DS also acknowledges the support of Schmidt Career Advancement Chair in AI.

Revised: TM was supported by grant 2318/22 from the Israel Science Foundation and by the Ollendorff

Revised: Center of the Viterbi Faculty of Electrical and Computer Engineering at the Technion.

Revised: A A DDITIONAL DISCUSSION

Original: The independence of Lemma 1 and Theorem 1 on the batch size B .
Revised: The independence of Lemma 1 and Theorem 1 on the batch size B .

Original: Theorem 1 relies on
Revised: Theorem 1 relies on

Original: Lemma 1 (see App. E), which was proved in (Mulayoff et al., 2021).
Revised: Lemma 1 (see App. E), which was proved in (Mulayoff et al., 2021).

Original: This lemma states that if a minimum θ ∗ is linearly stable for SGD with batch-size B , then the Hessian H of the loss at θ ∗ must satisfy λ max ( H ) ≤ 2 /η , where η is the step-size.
Revised: This lemma states that if a minimum θ ∗ is linearly stable for SGD with batch-size B , then the Hessian H of the loss at θ ∗ must satisfy λ max ( H ) ≤ 2 /η , where η is the step-size.

Original: Importantly, this necessary condition holds true for any batch size B , and thus Theorem 1 is independent of B .
Revised: Importantly, this necessary condition holds true for any batch size B , and thus Theorem 1 is independent of B .

Original: We note, however, that the precise stability threshold of SGD might depend on B , yet the important points to notice are: (1)
Revised: We note, however, that the precise stability threshold of SGD might depend on B , yet the important points to notice are: (1)

Original: Here all we need is a necessary condition, and Lemma 1 provides a simple bound that holds for any B .
Revised: Here all we need is a necessary condition, and Lemma 1 provides a simple bound that holds for any B .

Original: (2) Empirical evidence shows that there is not much room for improvement upon this batch-size-independent bound in real-world settings.
Revised: (2) Empirical evidence shows that there is not much room for improvement upon this batch-size-independent bound in real-world settings.

Original: Specifically, for practical batch sizes, the gap between 2 /η and the stability threshold of SGD is often very small (see Fig.
Revised: Specifically, for practical batch sizes, the gap between 2 /η and the stability threshold of SGD is often very small (see Fig.

Original: 5 in our paper and Figures 2-3 in (Gilmer et al., 2022)).
Revised: 5 in our paper and Figures 2-3 in (Gilmer et al., 2022)).

Original: The proof of Lemma 1 is actually quite short and easy to follow (see (Mulayoff et al., 2021, App. II)).
Revised: The proof of Lemma 1 is actually quite short and easy to follow (see (Mulayoff et al., 2021, App. II)).

Original: The idea is that if θ ∗ is an ε linearly stable minimum, then by definition we have
Revised: The idea is that if θ ∗ is an ε linearly stable minimum, then by definition we have

Original: [Equation]
Revised: [Equation]

Original: (20) where { θ t } ∞ t =0 are governed by the linearized stochastic dynamics given in Eq.
Revised: (21) where { θ t } ∞ t =0 are governed by the linearized stochastic dynamics given in Eq.

Original: Using Jensen’s inequality, for all t > 0 we get ∥ E [ θ t ] − θ ∗ ∥ ≤ E
Revised: Using Jensen’s inequality, for all t > 0 we get ∥ E [ θ t ] − θ ∗ ∥ ≤ E

Original: [ ∥ θ t − θ ∗ ∥ ] .
Revised: [ ∥ θ t − θ ∗ ∥ ] .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Note that under the linearized dynamics, { E [ θ t ] } ∞ t =0 are precisely GD steps.
Revised: Note that under the linearized dynamics, { E [ θ t ] } ∞ t =0 are precisely GD steps.

Original: Therefore, we have that if θ ∗ is linearly stable for SGD, then it must be linearly stable also for GD.
Revised: Therefore, we have that if θ ∗ is linearly stable for SGD, then it must be linearly stable also for GD.

Original: Now, a well-known fact is that θ ∗ is linearly stable for GD if and only if λ max ( H ) ≤ 2 /η .
Revised: Now, a well-known fact is that θ ∗ is linearly stable for GD if and only if λ max ( H ) ≤ 2 /η .

Original: This is how we get the necessary condition in Lemma 1, which does not depend on the batch size.
Revised: This is how we get the necessary condition in Lemma 1, which does not depend on the batch size.

Original: The independence of Lemma 1 and Theorem 1 on ε .
Revised: The independence of Lemma 1 and Theorem 1 on ε .

Original: Lemma 1 states that a necessary condition for a twice differentiable minimum to be ε linearly stable is
Revised: Lemma 1 states that a necessary condition for a twice differentiable minimum to be ε linearly stable is

Original: λ max (cid:0) ∇ 2 L ( θ ∗ ) (cid:1) ≤ 2 /η.
Revised: λ max (cid:0) ∇ 2 L ( θ ∗ ) (cid:1) ≤ 2 /η.

Original: That is, the condition does not depend on ε which might seem not intuitive.
Revised: That is, the condition does not depend on ε which might seem not intuitive.

Original: However, the reason Lemma 1 does not depend on ε is because it refers to linear stability, opposed to non-linear dynamical stability.
Revised: However, the reason Lemma 1 does not depend on ε is because it refers to linear stability, opposed to non-linear dynamical stability.

Original: In linear stability for twice-differentiable minima, all we care about is the second-order Taylor approximation of the loss at the minimum.
Revised: In linear stability for twice-differentiable minima, all we care about is the second-order Taylor approximation of the loss at the minimum.

Original: In see previous paragraph we explained that Lemmagives a necessary condition through reduction to GD.
Revised: In see previous paragraph we explained that Lemmagives a necessary condition through reduction to GD.

Original: Now, when applying GD on a quadratic loss (with a PSD matrix), for any ε > 0 only one of two things can happen:
Revised: Now, when applying GD on a quadratic loss (with a PSD matrix), for any ε > 0 only one of two things can happen:

Original: Either ∃ θ 0 ∈ B ε ( θ ∗ ) : lim sup t →∞ ∥ θ t − θ ∗ ∥ = + ∞ (unstable for any ε > 0 ), 2. or ∀ θ 0 ∈
Revised: Either ∃ θ 0 ∈ B ε ( θ ∗ ) : lim sup t →∞ ∥ θ t − θ ∗ ∥ = + ∞ (unstable for any ε > 0 ), 2. or ∀ θ 0 ∈

Original: B ε ( θ ∗ ) : lim sup t →∞ ∥ θ t − θ ∗ ∥ ≤ ∥ θ 0 − θ ∗ ∥ ≤ ε (stable for any ε > 0 ).
Revised: B ε ( θ ∗ ) : lim sup t →∞ ∥ θ t − θ ∗ ∥ ≤ ∥ θ 0 − θ ∗ ∥ ≤ ε (stable for any ε > 0 ).

Original: In any outcome, the result does not depend on ε , and therefore ε does not appear in the result of
Revised: In any outcome, the result does not depend on ε , and therefore ε does not appear in the result of

Original: Lemma 1.
Revised: Lemma 1.

Original: Note that for non-differentiable minima, ε does affect linear stability in GD, however
Revised: Note that for non-differentiable minima, ε does affect linear stability in GD, however

Original: Lemma 1 only refers to twice-differentiable minima.
Revised: Lemma 1 only refers to twice-differentiable minima.

Original: Theorem 1 is based on Lemma 1, and therefore does not depend on ε .
Revised: Theorem 1 is based on Lemma 1, and therefore does not depend on ε .

Original: Yet, beyond this technical reasoning, it is important to note that here we consider interpolating solutions .
Revised: Yet, beyond this technical reasoning, it is important to note that here we consider interpolating solutions .

Original: For those solutions, the global minimum of the loss is also a global minimum w.r.t.
Revised: For those solutions, the global minimum of the loss is also a global minimum w.r.t.

Original: each data sample ( x j , y j ) separately.
Revised: each data sample ( x j , y j ) separately.

Original: Therefore, despite the stochasticity of SGD, every step points towards a global minimum.
Revised: Therefore, despite the stochasticity of SGD, every step points towards a global minimum.

Original: This implies that if the stability criterion is satisfied, then SGD converges to the minimum ( lim sup t →∞ E [ ∥ θ t − θ ∗ ∥ ] = 0 ) and if it is not satisfied, then SGD repels from the minimum ( lim sup t →∞ E [ ∥ θ t − θ ∗ ∥ ] = ∞ for the linearized dynamics).
Revised: This implies that if the stability criterion is satisfied, then SGD converges to the minimum ( lim sup t →∞ E [ ∥ θ t − θ ∗ ∥ ] = 0 ) and if it is not satisfied, then SGD repels from the minimum ( lim sup t →∞ E [ ∥ θ t − θ ∗ ∥ ] = ∞ for the linearized dynamics).

Original: This is also seen in simulations where models are overfit to training data using
Revised: This is also seen in simulations where models are overfit to training data using

Original: SGD, e.g.  (Ma et al., 2018b).
Revised: SGD, e.g. , (Ma et al., 2018b).

Original: Particularly, in our simulations the loss always converged to 0 when it converged (we arbitrarily decided to stop each run when the loss dropped below 10 − 8 ).
Revised: Particularly, in our simulations the loss always converged to 0 when it converged (we arbitrarily decided to stop each run when the loss dropped below 10 − 8 ).

Original: In the general case of non-interpolating solutions, the expected final distance to the minimum in mini-batch SGD ( lim sup t →∞ E [ ∥ θ t − θ ∗ ∥ ] ) can be a strictly positive finite number, and therefore in those cases ε does play a role.
Revised: In the general case of non-interpolating solutions, the expected final distance to the minimum in mini-batch SGD ( lim sup t →∞ E [ ∥ θ t − θ ∗ ∥ ] ) can be a strictly positive finite number, and therefore in those cases ε does play a role.

Original: Large step size training and warmup.
Revised: Large step size training and warmup.

Original: While Theorem 1 applies to any positive step size, it is most interesting when considering large step sizes.
Revised: While Theorem 1 applies to any positive step size, it is most interesting when considering large step sizes.

Original: High learning rates are standard practice, as they are associated with good generalization (Li et al., 2019).
Revised: High learning rates are standard practice, as they are associated with good generalization (Li et al., 2019).

Original: However, there are cases, e.g.  large initialization, in which high learning rate might cause the training to diverge.
Revised: However, there are cases, e.g. , large initialization, in which high learning rate might cause the training to diverge.

Original: In these cases, a learning rate warmup is applied, enabling training with large step sizes.
Revised: In these cases, a learning rate warmup is applied, enabling training with large step sizes.

Original: Learning rate decay.
Revised: Learning rate decay.

Original: Practitioners often work with learning rate schedule, which typically reduces the step size toward the end of training.
Revised: Practitioners often work with learning rate schedule, which typically reduces the step size toward the end of training.

Original: In this scenario, 2 /η can be quite high at the end, making
Revised: In this scenario, 2 /η can be quite high at the end, making

Original: Theorem 1 loose.
Revised: Theorem 1 loose.

Original: Here, empirical evidence shows that when reducing the step size at a late stage, the sharpness of the obtained minimum is often still controlled by the initial step size, e.g. , Figs.and 3 in (Gilmer et al., 2022).
Revised: Here, empirical evidence shows that when reducing the step size at a late stage, the sharpness of the obtained minimum is often still controlled by the initial step size, e.g. , Figs.and 3 in (Gilmer et al., 2022).

Original: Moreover, although learning rate decay is a popular technique, there are other popular training schemes in which the learning rate is not reduced, e.g.  (Smith et al., 2018).
Revised: Moreover, although learning rate decay is a popular technique, there are other popular training schemes in which the learning rate is not reduced, e.g. , (Smith et al., 2018).

Original: Lastly, as for the depth separation results (Sec. 4) and the approximation results (Sec. 5), they apply for any fixed positive step size.
Revised: Lastly, as for the depth separation results (Sec. 4) and the approximation results (Sec. 5), they apply for any fixed positive step size.

Original: In other words, the learning rate decay does not affect these results.
Revised: In other words, the learning rate decay does not affect these results.

Original: Initialization independent results.
Revised: Initialization independent results.

Original: Our results are independent of the initialization.
Revised: Our results are independent of the initialization.

Original: In the past it was shown that initialization can have large effect on which minimum GD converges under certain conditions.
Revised: In the past it was shown that initialization can have large effect on which minimum GD converges under certain conditions.

Original: However, this do not contradict our results, as explained below.
Revised: However, this do not contradict our results, as explained below.

Original: For very small step sizes, the GD trajectory follows that of gradient flow (GF).
Revised: For very small step sizes, the GD trajectory follows that of gradient flow (GF).

Original: Under certain conditions, e.g.  infinite width or vanishing initialization, it was shown that the network does not change much on the evolution trajectory of GF.
Revised: Under certain conditions, e.g. , infinite width or vanishing initialization, it was shown that the network does not change much on the evolution trajectory of GF.

Original: In this case, the initialization dominates the properties of the obtained solution.
Revised: In this case, the initialization dominates the properties of the obtained solution.

Original: This is known as kernel regime or Neural Tangent Kernel (NTK) regime (Jacot et al., 2018; Chizat et al., 2019).
Revised: This is known as kernel regime or Neural Tangent Kernel (NTK) regime (Jacot et al., 2018; Chizat et al., 2019).

Original: However, for practical step sizes and standard initialization, recent work (Cohen et al., 2020) showed that GD typically deviates from the GF trajectory, while entering the Edge of Stability regime.
Revised: However, for practical step sizes and standard initialization, recent work (Cohen et al., 2020) showed that GD typically deviates from the GF trajectory, while entering the Edge of Stability regime.

Original: This occurs when the stability threshold is achieved during training, i.e., λ max ( ∇ 2 L ( θ t )) ≥ 2 /η for some t > 0 .
Revised: This occurs when the stability threshold is achieved during training, i.e. , λ max ( ∇ 2 L ( θ t )) ≥ 2 /η for some t > 0 .

Original: In this case, GD converges to a different minimum than GF, i.e., GD escapes the NTK regime.
Revised: In this case, GD converges to a different minimum than GF, i.e. , GD escapes the NTK regime.

Original: Similar behavior was shown also for SGD (Gilmer et al., 2022).
Revised: Similar behavior was shown also for SGD (Gilmer et al., 2022).

Original: Theorem 1 assumption and minima that are not twice-differentiable.
Revised: Theorem 1 Proof idea. Theorem 1 is a result of two properties of twice-differentiable minima.

Revised: First, we show in Lemma 3 in the appendix that these minima satisfy λ max ( ∇ 2 θ L ) ≥ 1+2 ∥ f

Revised: ∥ R ,g .

Revised: Second, we know from Lemma 1 that stable minima satisfy λ max ( ∇ 2 θ L ) ≤ 2 /η .

Revised: Together, these properties imply that 2 /η ≥ 1 + 2 ∥ f

Revised: ∥ R ,g , from which we get the result of the theorem, ∥ f

Revised: ∥ R ,g ≤ 1 /η − 1 / 2 .

Revised: Note that twice-differentiable minima correspond to functions whose knots do not coincide with any training point.

Revised: Although we prove our result only for such functions, we observed that in practice the condition λ max ( ∇ 2 θ L ) ≤ 2 /η is always met around minima to which SGD converges (see Sec.

Revised: 6 and the next paragraph).

Revised: For full derivation of the theorem see App.

Revised: E.

Revised: Theorem 1 assumption and minima that are not twice-differentiable.

Original: Theorem 1 assumes that the knots of f do not coincide with any training point.
Revised: Theorem 1 assumes that the knots of f do not coincide with any training point.

Original: This assumption is done for technical simplicity, that is to ensure the minimum is twice-differentiable.
Revised: This assumption is done for technical simplicity, that is to ensure the minimum is twice-differentiable.

Original: In practice, we observed that usually only a small fraction of the knots coincide with training points.
Revised: In practice, we observed that usually only a small fraction of the knots coincide with training points.

Original: For example, in our MNIST experiment we had only 30 training points coinciding with knots of f (out of n = 512 training points and k =neurons).
Revised: For example, in our MNIST experiment we had only 30 training points coinciding with knots of f (out of n = 512 training points and k =neurons).

Original: Note that the twice-differentiability assumption in Theorem 1 is required for Lemma 1 to hold as Theorem 1 invokes Lemma 1.
Revised: Note that the twice-differentiability assumption in Theorem 1 is required for Lemma 1 to hold as Theorem 1 invokes Lemma 1.

Original: However, we observed that in practical settings, Lemma 1 still applies in settings where the assumption is violated.
Revised: However, we observed that in practical settings, Lemma 1 still applies in settings where the assumption is violated.

Original: This can be appreciated by the red curve upper bounding the orange curve in Figs. 4(a) and 5(a).
Revised: This can be appreciated by the red curve upper bounding the orange curve in Figs. 4(a) and 5(a).

Original: Namely, despite the fact that the minima are not always twice-differentiable in our experiments, the stability criterion for SGD is still observed to be upper bounded by 2 /η .
Revised: Namely, despite the fact that the minima are not always twice-differentiable in our experiments, the stability criterion for SGD is still observed to be upper bounded by 2 /η .

Original: It is possible to extend the analysis to minima that are not twice-differentiable by using the same method as in (Mulayoff et al., 2021).
Revised: It is possible to extend the analysis to minima that are not twice-differentiable by using the same method as in (Mulayoff et al., 2021).

Original: However, this makes the analysis much more complex.
Revised: However, this makes the analysis much more complex.

Revised: The meaning of min λ max in figures 4(a) and 5(a).

Revised: The curve min λ max shows the sharpness of the flattest implementation of each solution f .

Revised: In more detail, as discussed earlier, Theorem 1 is a result of two properties of twice-differentiable minima.

Revised: On the one hand, we know from Lemmathat stable minima satisfy λ max ( ∇ 2 θ L ) ≤ 2 /η .

Revised: On the other hand, Lemma 3 in App.

Original: B A DDITIONAL RELATED WORK
Revised: E asserts that these minima satisfy λ max ( ∇ 2 θ L ) ≥ 1 + 2 ∥ f ∥ R ,g . Note that each function f ∈ F k has multiple implementations, i.e. , different minima in parameter space which all correspond to f . These minima can have different sharpness.

Revised: Here, we can look at the best implementation, i.e. , a solution to min λ max , where the minimum is taken over all loss’ minima { θ } that implement f .

Revised: Overall, given a minimum θ with a corresponding function f , we have

Revised: [Equation]

Revised: These inequalities give us the result of Theorem 1, 1 + 2 ∥ f

Revised: ∥ R ,g ≤ 1 /η − 1 / 2 .

Revised: To understand the tightness of each part of our analysis, we added to the plots λ max and min λ max .

Revised: In both figures, λ max ( ∇ 2 θ L ) equals or just below 2 /η , a phenomenon known as edge of stability (Cohen et al., 2020).

Revised: Additionally, min λ max ( ∇ 2 θ L ) is close to 1 + 2 ∥ f

Revised: ∥ R ,g in these experiments.

Revised: Yet, Figure 4 shows that λ max ( ∇ 2 θ L ) can be quite larger than min λ max ( ∇ 2 θ L ) , meaning that there exists a far flatter minimum that implements the same function.

Revised: This fact was used by Dinh et al.

Revised: (2017) to show that sharp minima can generalize.

Revised: B A DDITIONAL RELATED WORK

Original: Implicit bias.
Revised: Implicit bias.

Original: A long line of works studied the implicit bias of the training procedure in an attempt to better understand generalization in overparameterized models.
Revised: A long line of works studied the implicit bias of the training procedure in an attempt to better understand generalization in overparameterized models.

Original: For the classification setting, in the case of linear prediction function, linearly separable data, and exponentially tailed loss functions ( e.g.  logistic and exponential), Soudry et al.
Revised: For the classification setting, in the case of linear prediction function, linearly separable data, and exponentially tailed loss functions ( e.g. , logistic and exponential), Soudry et al.

Original: (2018) showed that GD converges in the direction of the SVM solution.
Revised: (2018) showed that GD converges in the direction of the SVM solution.

Original: This result was later extended to linear fully connected and convolutional neural networks (Gunasekar et al., 2018b; Ji & Telgarsky, 2019a), more loss functions (Nacson et al., 2019b; Ji &
Revised: This result was later extended to linear fully connected and convolutional neural networks (Gunasekar et al., 2018b; Ji & Telgarsky, 2019a), more loss functions (Nacson et al., 2019b; Ji &

Original: Telgarsky, 2021), SGD optimization algorithm (Nacson et al., 2019c), other generic optimization methods (Gunasekar et al., 2018a), non-separable data (Ji & Telgarsky, 2019b), and homogeneous prediction functions (Nacson et al., 2019a; Lyu & Li, 2020; Ji et al., 2020).
Revised: Telgarsky, 2021), SGD optimization algorithm (Nacson et al., 2019c), other generic optimization methods (Gunasekar et al., 2018a), non-separable data (Ji & Telgarsky, 2019b), and homogeneous prediction functions (Nacson et al., 2019a; Lyu & Li, 2020; Ji et al., 2020).

Original: However, all those results do not depend on the step size, except for the requirement that it be sufficiently small.
Revised: However, all those results do not depend on the step size, except for the requirement that it be sufficiently small.

Original: Another line of works studied the implicit bias in the context of linear models with quadratic loss such as matrix factorization (Gunasekar et al., 2017; Li et al., 2018; Arora et al., 2018; 2019; Belabbas, 2020; Eftekhari & Zygalakis, 2020; Gidel et al., 2019; Ma et al., 2018a; Woodworth et al., 2020;
Revised: Another line of works studied the implicit bias in the context of linear models with quadratic loss such as matrix factorization (Gunasekar et al., 2017; Li et al., 2018; Arora et al., 2018; 2019; Belabbas, 2020; Eftekhari & Zygalakis, 2021; Gidel et al., 2019; Ma et al., 2018a; Woodworth et al., 2020;

Original: Azulay et al., 2021).
Revised: Azulay et al., 2021).

Original: However, all of these works relied on either small or infinitesimal step size ( i.e., gradient flow).
Revised: However, all of these works relied on either small or infinitesimal step size ( i.e. , gradient flow).

Original: Thus, they do not capture how the step size affects the implicit bias.
Revised: Thus, they do not capture how the step size affects the implicit bias.

Original: Moreover, they assumed a manifold property (Azulay et al., 2021).
Revised: Moreover, they assumed a manifold property (Azulay et al., 2021).

Original: As pointed out by Razin & Cohen (2020) and Vardi & Shamir (2021), these assumptions do not always apply.
Revised: As pointed out by Razin & Cohen (2020) and Vardi & Shamir (2021), these assumptions do not always apply.

Original: In contrast, our result is based on a stability condition of SGD, which depends on the step size and does not require the manifold assumption.
Revised: In contrast, our result is based on a stability condition of SGD, which depends on the step size and does not require the manifold assumption.

Original: How the step size affects the implicit bias.
Revised: How the step size affects the implicit bias.

Original: To investigate the implicit bias of the step size, Barrett & Dherin (2021) and Smith et al.
Revised: To investigate the implicit bias of the step size, Barrett & Dherin (2021) and Smith et al.

Original: (2021) suggested using a modified loss.
Revised: (2021) suggested using a modified loss.

Original: Under this modified loss, gradient flow approximates the trajectory of (S)GD on the original loss.
Revised: Under this modified loss, gradient flow approximates the trajectory of (S)GD on the original loss.

Original: However, the step size should be sufficiently small for the approximation to hold true.
Revised: However, the step size should be sufficiently small for the approximation to hold true.

Original: Moreover, the induced regularization term this method yields is expressed in terms of the model’s parameters.
Revised: Moreover, the induced regularization term this method yields is expressed in terms of the model’s parameters.

Original: Additionally, this term increases linearly with the step size and vanishes at any stationary point.
Revised: Additionally, this term increases linearly with the step size and vanishes at any stationary point.

Original: Radon transform analysis of shallow networks.
Revised: Radon transform analysis of shallow networks.

Original: Radon transform analysis has previously been used in studies of the approximation capabilities of single hidden-layer neural networks with bounded activation functions (Carroll & Dickinson, 1989; Ito, 1991), and more general activation functions in the ridgelet framework (Candès & Donoho, 1999; Candès, 1999).
Revised: Radon transform analysis has previously been used in studies of the approximation capabilities of single hidden-layer neural networks with bounded activation functions (Carroll & Dickinson, 1989; Ito, 1991), and more general activation functions in the ridgelet framework (Candès & Donoho, 1999; Candès, 1999).

Original: More recently, Sonoda & Murata (2017) used ridgelet transform analysis to study the approximation properties of two-layer neural networks with unbounded activation functions, including the ReLU.
Revised: More recently, Sonoda & Murata (2017) used ridgelet transform analysis to study the approximation properties of two-layer neural networks with unbounded activation functions, including the ReLU.

Original: Parts of this work extend results by Ongie et al.
Revised: Parts of this work extend results by Ongie et al.

Original: (2020), which defined a similar Radon-domain seminorm (the “ R -norm”) to determine the space of functions realizable as an infinite-width single hidden-layer ReLU networks with square-summable weights.
Revised: (2020), which defined a similar Radon-domain seminorm (the “ R -norm”) to determine the space of functions realizable as an infinite-width single hidden-layer ReLU networks with square-summable weights.

Original: Parhi & Nowak (2021) proved a representer theorem for single hidden-layer ReLU networks using the R -norm.
Revised: Parhi & Nowak (2021) proved a representer theorem for single hidden-layer ReLU networks using the R -norm.

Original: Finally, an L 2 version of the R -norm is used by Jin & Montúfar (2020) to describe the function space implicit bias of training a single hidden-layer ReLU network using gradient descent in the neural tangent kernel regime.
Revised: Finally, an L 2 version of the R -norm is used by Jin & Montúfar (2020) to describe the function space implicit bias of training a single hidden-layer ReLU network using gradient descent in the neural tangent kernel regime.

Original: C T HE R ADON TRANSFORM
Revised: C T HE R ADON TRANSFORM

Original: For a function f : R d → R , the d -dimensional Radon transform R f is the collection of all integrals of f over ( d − 1)
Revised: For a function f : R d → R , the d -dimensional Radon transform R f is the collection of all integrals of f over ( d − 1)

Original: -dimensional affine hyperplanes in R d .
Revised: -dimensional affine hyperplanes in R d .

Original: Every hyperplane can be parametrized by a pair ( v , b ) ∈ S
Revised: Every hyperplane can be parametrized by a pair ( v , b ) ∈ S

Original: d − 1 × R , where v is a unit normal to the hyperplane and b ∈ R is its distance from the origin.
Revised: d − 1 × R , where v is a unit normal to the hyperplane and b ∈ R is its distance from the origin.

Original: Therefore, the Radon transform R f is the function over ( v , b ) ∈ S d − 1 × R given by
Revised: Therefore, the Radon transform R f is the function over ( v , b ) ∈ S d − 1 × R given by

Original: [Equation]
Revised: [Equation]

Original: 22) where d s ( x ) represents integration with respect to the ( d − 1) -dimensional surface measure on the hyperplane v ⊤ x = b .
Revised: where d s ( x ) represents integration with respect to the ( d − 1) -dimensional surface measure on the hyperplane v ⊤ x = b .

Original: Note that the Radon transform is an even function, i.e., R f ( v , b ) =
Revised: Note that the Radon transform is an even function, i.e. , R f ( v , b ) =

Original: R f ( − v , − b ) , since ( v , b ) and ( − v , − b ) describe the same hyperplane.
Revised: R f ( − v , − b ) , since ( v , b ) and ( − v , − b ) describe the same hyperplane.

Original: The dual Radon transform R ∗ maps functions defined on S d − 1 × R to functions on R d by
Revised: The dual Radon transform R ∗ maps functions defined on S d − 1 × R to functions on R d by

Original: [Equation]
Revised: [Equation]

Original: (23) for all x ∈ R d , where d s ( v ) represents integration with respect to the ( d − 1) -dimensional surface measure on the unit sphere S d − 1 .
Revised: 25) for all x ∈ R d , where d s ( v ) represents integration with respect to the ( d − 1) -dimensional surface measure on the unit sphere S d − 1 .

Original: The Radon transform and its dual are invertible over spaces of smooth functions via the inversion formulas: 
Revised: The Radon transform and its dual are invertible over spaces of smooth functions via the inversion formulas: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where the fractional Laplacian operator ( − ∆)
Revised: (27) where the fractional Laplacian operator ( − ∆)

Original: d − 12 is defined by application of a ramp function in
Revised: d − 12 is defined by application of a ramp function in

Original: Fourier domain ( i.e., multiplication by ∥ ω
Revised: Fourier domain ( i.e. , multiplication by ∥ ω

Original: ∥ d − 1 in Fourier domain), and
Revised: ∥ d − 1 in Fourier domain), and

Original: [Equation]
Revised: [Equation]

Original:  is a dimension dependent constant.
Revised: (28) is a dimension dependent constant.

Original: These transforms may be extended to spaces of distributions ( e.g.  Dirac deltas) in a standard way (Ludwig, 1966; Helgason, 1999), which we summarize in App.
Revised: These transforms may be extended to spaces of distributions ( e.g. , Dirac deltas) in a standard way (Ludwig, 1966; Helgason, 1999), which we summarize in App.

Original: D. Important for this work is the distributional dual inverse Radon transform ( R ∗ ) − 1 , which maps a distribution defined over Euclidean space R d to a distribution in Radon domain S d − 1 × R .
Revised: D. Important for this work is the distributional dual inverse Radon transform ( R ∗ ) − 1 , which maps a distribution defined over Euclidean space R d to a distribution in Radon domain S d − 1 × R .

Original: D D ISTRIBUTIONAL FRAMEWORK
Revised: D D ISTRIBUTIONAL FRAMEWORK

Original: Let f : R d → R be any locally integrable function.
Revised: Let f : R d → R be any locally integrable function.

Original: Then its Laplacian ∆ f can be interpreted as a tempered distribution, meaning that ∆ f is defined via the duality pairing
Revised: Then its Laplacian ∆ f can be interpreted as a tempered distribution, meaning that ∆ f is defined via the duality pairing

Original: [Equation]
Revised: [Equation]

Original: (27) where φ is any Schwartz test function on R d , i.e., a smooth function such that the function and its partial derivatives of all orders have sufficiently fast decay at infinity; denote this space of functions by S ( R d ) .
Revised: (29) where φ is any Schwartz test function on R d , i.e. , a smooth function such that the function and its partial derivatives of all orders have sufficiently fast decay at infinity; denote this space of functions by S ( R d ) .

Original: For example, if f consists of a single ReLU unit, i.e., f ( x ) = σ ( v ⊤ x − b ) such that ∥ v ∥ = 1 , then it is easy to show ∆ f = δ ( v ⊤ x − b ) , meaning ⟨ ∆ f, φ ⟩ = (cid:82) { x : v ⊤ x = b } φ ( x )d x =
Revised: For example, if f consists of a single ReLU unit, i.e. , f ( x ) = σ ( v ⊤ x − b ) such that ∥ v ∥ = 1 , then it is easy to show ∆ f = δ ( v ⊤ x − b ) , meaning ⟨ ∆ f, φ ⟩ = (cid:82) { x : v ⊤ x = b } φ ( x )d x =

Original: R φ ( v , b ) .
Revised: R φ ( v , b ) .

Original: In other words, ∆ f is the distribution given by evaluation of the Radon transform of a test function at the point ( v , b ) ∈ S d − 1 × R .
Revised: In other words, ∆ f is the distribution given by evaluation of the Radon transform of a test function at the point ( v , b ) ∈ S d − 1 × R .

Original: Next, we describe how to understand the operator ( R ∗ ) − 1 in a distributional sense.
Revised: Next, we describe how to understand the operator ( R ∗ ) − 1 in a distributional sense.

Original: Let S H ( S d − 1 × R ) denote the image of Schwartz functions S ( R d ) under the classical Radon transform R .
Revised: Let S H ( S d − 1 × R ) denote the image of Schwartz functions S ( R d ) under the classical Radon transform R .

Original: The space
Revised: The space

Original: S H ( S d − 1 × R ) is characterized in (Ludwig, 1966; Helgason, 1999); it is the space of all even Schwartz functions defined on S d − 1 × R that additionally satisfy some moment conditions 7 .
Revised: S H ( S d − 1 × R ) is characterized in (Ludwig, 1966; Helgason, 1999); it is the space of all even Schwartz functions defined on S d − 1 × R that additionally satisfy some moment conditions 7 .

Original: It is also shown by Ludwig (1966) that the classical inverse Radon transform R − 1 is a linear homeomorphism of
Revised: It is also shown by Ludwig (1966) that the classical inverse Radon transform R − 1 is a linear homeomorphism of

Original: S H ( S d − 1 × R ) onto S ( R d ) .
Revised: S H ( S d − 1 × R ) onto S ( R d ) .

Original: Therefore, we may define its distributional transpose ( R − 1 ) ∗ =
Revised: Therefore, we may define its distributional transpose ( R − 1 ) ∗ =

Original: ( R ∗ ) −applied to any tempered distribution h by
Revised: ( R ∗ ) −applied to any tempered distribution h by

Original: [Equation]
Revised: [Equation]

Original: 28) for all ϕ ∈ S H ( S d − 1 × R ) .
Revised: ) for all ϕ ∈ S H ( S d − 1 × R ) .

Original: Note that ( R ∗ ) − 1 h is a distribution belonging to S
Revised: Note that ( R ∗ ) − 1 h is a distribution belonging to S

Original: ′ H ( S d − 1 × R ) , the topological dual of S H ( S d − 1 × R ) .
Revised: ′ H ( S d − 1 × R ) , the topological dual of S H ( S d − 1 × R ) .

Original: Returning to the example where f is a single ReLU unit, i.e., f ( x ) = σ ( v ⊤ x − b ) with ∥ v ∥ = 1 , then for any test function ϕ ∈ S H ( S d − 1 × R ) we have
Revised: Returning to the example where f is a single ReLU unit, i.e. , f ( x ) = σ ( v ⊤ x − b ) with ∥ v ∥ = 1 , then for any test function ϕ ∈ S H ( S d − 1 × R ) we have

Original: [Equation]
Revised: [Equation]

Original: This shows ( R ∗ ) − 1 ∆ f = δ ( v ,b ) , i.e., a Dirac delta centered at ( v , b ) .
Revised: This shows ( R ∗ ) − 1 ∆ f = δ ( v ,b ) , i.e. , a Dirac delta centered at ( v , b ) .

Original: If f ( x ) = (cid:80) ki =1 a i σ ( v ⊤ i x − b i )
Revised: If f ( x ) = (cid:80) ki =1 a i σ ( v ⊤ i x − b i )

Original: + c is any single hidden-layer ReLU network such that ∥ v i ∥ = 1 for all i = 1 , ..., k , then by linearity we have
Revised: + c is any single hidden-layer ReLU network such that ∥ v i ∥ = 1 for all i = 1 , ..., k , then by linearity we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we may define the total variation ∥ · ∥ TV for any distribution α ∈ S ′ H ( S d − 1 × R ) by
Revised: Finally, we may define the total variation ∥ · ∥ TV for any distribution α ∈ S ′ H ( S d − 1 × R ) by

Original: [Equation]
Revised: [Equation]

Original: If ∥ α ∥ TV is finite, then α is a distribution of order- 0 .
Revised: If ∥ α ∥ TV is finite, then α is a distribution of order- 0 .

Original: In this case, since S H ( S d − 1 × R ) is dense in the space of even and continuous functions on S d − 1 × R that vanish at infinity, α can be extended uniquely to an even signed measure on S d − 1 × R , and ∥ α ∥ TV is equal to the total variation norm of α .
Revised: In this case, since S H ( S d − 1 × R ) is dense in the space of even and continuous functions on S d − 1 × R that vanish at infinity, α can be extended uniquely to an even signed measure on S d − 1 × R , and ∥ α ∥ TV is equal to the total variation norm of α .

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: In the proof of the theorem we use the following lemma (for the proof of this lemma see Appendix F).
Revised: In the proof of the theorem we use the following lemma (for the proof of this lemma see Appendix F).

Original: Lemma 3 (Top eigenvalue lower bound) .
Revised: Lemma 3 (Top eigenvalue lower bound) .

Original: Let f ∈ F k be a twice-differentiable minimizer of the loss function, then
Revised: Let f ∈ F k be a twice-differentiable minimizer of the loss function, then

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ R ,g denotes the stability norm.
Revised: where ∥·∥ R ,g denotes the stability norm.

Original: Let f ∈ F k be a stable solution of the loss function.
Revised: Let f ∈ F k be a stable solution of the loss function.

Original: Then, according to Definition 3, there exists a linearly stable minimum point θ ∈ R ( d +2)
Revised: Then, according to Definition 3, there exists a linearly stable minimum point θ ∈ R ( d +2)

Original: k +1 such that the network at this minimum implements f .
Revised: k +1 such that the network at this minimum implements f .

Original: Due to the fact that the knots of f do not contain any training point, we have that θ is a twicedifferentiable minimum.
Revised: Due to the fact that the knots of f do not contain any training point, we have that θ is a twicedifferentiable minimum.

Original: From Lemma 1, since θ is a twice differentiable stable minimum then
Revised: From Lemma 1, since θ is a twice differentiable stable minimum then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: On the other hand, from Lemma 3 we have that
Revised: On the other hand, from Lemma 3 we have that

Original: [Equation]
Revised: [Equation]

Original: Using (33) and (34) we get
Revised: Using (35) and (36) we get

Original: [Equation]
Revised: [Equation]

Original: F P ROOF OF L EMMA
Revised: F P ROOF OF L EMMA

Original: The proof of the Lemma consists of the following steps:
Revised: The proof of the Lemma consists of the following steps:

Original: Calculating ∇ 2 θ L , and showing that at a global minimum it takes the form ∇ 2 θ L = 1 n ΦΦ ⊤ (Appendix F.1).
Revised: Calculating ∇ 2 θ L , and showing that at a global minimum it takes the form ∇ 2 θ L = 1 n ΦΦ ⊤ (Appendix F.1).

Original: Lower bounding λ max ( ∇ 2 θ L ) by usingλ max (cid:0) ∇ [Equation]
Revised: Lower bounding λ max ( ∇ 2 θ L ) by using

Original: [Equation]
Revised: [Equation]

Original: and lower bounding the RHS (Appendix F.2).
Revised: and lower bounding the right hand side (Appendix F.2).

Original: Simplifying the lower bound to obtain a more interpretable version which does not depend on the specific implementation of f (Appendix F.3)
Revised: Simplifying the lower bound to obtain a more interpretable version which does not depend on the specific implementation of f (Appendix F.3)

Original: F. H ESSIAN COMPUTATION
Revised: F. H ESSIAN COMPUTATION

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: 37) where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We denote
Revised: We denote

Original: [Equation]
Revised: [Equation]

Original: 39) and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: [Equation] Using these notations

Original: [Equation]
Revised: [Equation]

Original: Using these notations, assuming that θ ∗ is a twice differentiable global minimum of L , we have that the gradient is 
Revised: assuming that θ ∗ is a twice differentiable global minimum of L [Equation] we have that the gradient is [Equation]

Original: [Equation]
Revised: [Equation]

Original: The Hessian is given by
Revised: The Hessian is given by

Original: [Equation]
Revised: [Equation]

Original: where in the last transition we used ∀ j ∈ [ n ] : f ( x j )
Revised: where in the last transition we used ∀ j ∈ [ n ] : f ( x j )

Original: =
Revised: =

Original: y j (see Def 2).
Revised: y j (see Def. 2).

Original: From direct calculation we obtain 
Revised: From direct calculation we obtain ∇

Original: [Equation]
Revised: [Equation]

Original: where ⊙ denotes the Hadamard product
Revised: where ⊙ denotes the Hadamard product

Original: [Equation]
Revised: [Equation]

Original: ⊗ represents the Kronecker product and I : R d ×
Revised: ⊗ represents the Kronecker product and I : R d ×

Original: R ( d +2) k +

Revised: R ( d +2) k +

Original: → { 0 , 1 } k is the activation pattern of all neurons for input x , namely [ I ( x ; θ )] i = 

Revised: → { 0 , 1 } k is the activation pattern of all neurons for input x , namely [ I ( x ; θ )] i =

Original: if x ⊤ w (1) i + b (1) i > 0 and [ I ( x ; θ )] i = 

Revised: if x ⊤ w (1) i + b (1) i > 0 and [ I ( x ; θ )] i =

Original: otherwise.
Revised: otherwise.

Original: Let us denote the tangent features matrix by
Revised: Let us denote the tangent features matrix by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then the Hessian can be expressed as ∇ 2 θ L = ΦΦ ⊤ /n , and its maximal eigenvalue can be written as
Revised: Then the Hessian can be expressed as ∇ 2 θ L = ΦΦ ⊤ /n , and its maximal eigenvalue can be written as

Original: [Equation]
Revised: [Equation]

Original: F. L OWER BOUNDING THE TOP EIGENVALUE
Revised: [Equation]

Original: Continuing from the previous section’s calculation, if we take u = 1 √ n 1 , we obtain max
Revised: Continuing from the previous section’s calculation, if we take u = 1 √ n 1 , we obtain max

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: = 1 + 1 nk (cid:88) i =   d (cid:88) l =   n (cid:88) j = w (2) i x j,l I j,i  
Revised: [Table]

Original: +   n (cid:88)
Revised: [Equation]

Original: j = w (2) i I j,i  
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: +  
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation] [Equation]
Revised: [Table] [Equation]

Original: (cid:16)
Revised: [Table]

Original: x ⊤

Original: [Equation]
Revised: [Equation]

Original: w (1) i + b (1)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:17)   2   = 1 + 1 nk (cid:88) i
Revised: [Table]

Original: =  (cid:16) w (2) i
Revised: [Equation]

Original: (cid:17)  d (cid:88) l =   n (cid:88) j = x j,l I j,i   +   n (cid:88) j = I j,i 
Revised: [Table]

Original:  2   +   n (cid:88) j = σ (cid:16) x ⊤ j w (1) i + b (1) i (cid:17) 2 
Revised: [Equation]

Original:  ( ∗ ) ≥ 1 +nk (cid:88) i = (cid:12)(cid:12)(cid:12) w (2) i (cid:12)(cid:12)(cid:12) ( cid:118)(cid:117)(cid:117)(cid:117)(cid:116) d (cid:88) l =  
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: j = x [Equation] ,l I j,i   +
Revised: [Table]

Original:  

Original: [Equation] (cid:88) j =
Revised: +   n (cid:88) j =

Original: I j,i 
Revised: I j,i 

Original:  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)  n (cid:88)
Revised:  2 (cid:12) (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) n (cid:88)

Original: j
Revised: j

Original: = σ (cid:16) x
Revised: = σ (cid:16) x

Original: ⊤ j w (1) i + b (1) i (cid:17)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:12)  , (46) where in ( ∗ ) we used α 2 + β 2 ≥ 2 | αβ | .
Revised: ⊤ j w (1) i + b (1) i (cid:17) (cid:12) (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) , (48) where in ( ∗ ) we used α 2 + β 2 ≥ 2 | αβ | .

Original: Let C
Revised: Let C

Original: i ⊆ { x j } be the set of training points for which the i th neuron is active, and denote n i =
Revised: i ⊆ { x j } be the set of training points for which the i th neuron is active, and denote n i =

Original: | C
Revised: | C

Original: i | , that is n i = n (cid:88) j =
Revised: i | , that is n i = n (cid:88) j =

Original: I j,i .
Revised: I j,i .

Original: [Table]
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where X is a random sample from the dataset under uniform distribution
Revised: (cid:118)(cid:117)(cid:117)(cid:116)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: (cid:17)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: where X is a random sample from the dataset under uniform distribution.

Original:  we define
Revised: Next, we define

Original: [Equation]
Revised: [Equation]

Original: Using these notations we obtain λ 
Revised: Using these notations we obtain λ max (cid:0) ∇ 2 θ L (cid:1) ≥ 1 +

Original: [Equation]
Revised: [Equation]

Revised: w (2) i (cid:12)(cid:12)(cid:12) (cid:13)(cid:13)(cid:13) w (1) i (cid:13)(cid:13)(cid:13)

Revised: ( P ( X

Original: [Equation]
Revised: [Equation]

Original: where in ( ∗ ) and ( ∗∗ ) we defined
Revised: C i )) 2 (cid:113)

Original: [Equation]
Revised: [Equation]

Original:  respectively,
Revised: ( ∗ ) = 1 + 2 k (cid:88) i = (cid:12)(cid:12)(cid:12) w (2) i (cid:12)(cid:12)(cid:12) (cid:13)(cid:13)(cid:13) w (1) i (cid:13)(cid:13)(cid:13) ˜ g (cid:16) ¯ w (1) i , ¯ b (1) i (cid:17) ( ∗∗ ) ≥ 1 + 2 k (cid:88) i = (cid:12)(cid:12)(cid:12) w (2) i (cid:12)(cid:12)(cid:12) (cid:13)(cid:13)(cid:13) w (1) i (cid:13)(cid:13)(cid:13) g (cid:16) ¯ w (1) i , ¯ b (1) i (cid:17) , (52) where in ( ∗ ) and ( ∗∗ ) we defined, respectively,

Original: [Equation]
Revised: [Equation]

Original: From our derivation so far
Revised: From our derivation so far

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: We denote a i = w (2) i ∥ w (1) i ∥ and the representation dependent stability norm as
Revised: We denote a i = w (2) i ∥ w (1) i ∥ and the representation dependent stability norm as

Original: [Equation]
Revised: [Equation]

Original: F. I MPLEMENTATION FREE LOWER BOUND
Revised: F. I MPLEMENTATION FREE LOWER BOUND

Original: In this section, our goal is to give a simpler lower bound of the multivariate stability norm S θ , that does not depend on the specific representation of f .
Revised: In this section [Equation] our goal is to give a simpler lower bound of the multivariate stability norm S θ , that does not depend on the specific representation of f .

Original: Let α be the signed measure over S d − 1 ×
Revised: Let α be the signed measure over S d − 1 ×

Original: R given by α = (cid:80) k
Revised: R given by α = (cid:80) ki

Original: i =1 a i δ (cid:16) ¯ w (1) i , ¯ b (1) i (cid:17) , and whose total variation measure | α | is given by | α | = (cid:80) ki =1 | a i | δ (cid:16) ¯ w (1) i , ¯ b (1) i (cid:17) .
Revised: =1 a i δ (cid:16) ¯ w (1) i , ¯ b (1) i (cid:17) , and whose total variation measure | α | is given by | α | = (cid:80) ki =1 | a i | δ (cid:16) ¯ w (1) i , ¯ b (1) i (cid:17) .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: and thus
Revised: and thus

Original: [Equation]

Original: [Equation]

Original: Namely

Original: [Equation]
Revised: [Equation]

Original:  ∆ f is a weighted sum of Diracs supported on hyperplanes.
Revised: Namely, ∆ f is a weighted sum of Diracs supported on hyperplanes.

Original: From the last equation we obtain α
Revised: From the last equation we obtain α

Original: = ( R ∗ ) − 1 ∆ f .
Revised: = ( R ∗ ) − 1 ∆ f .

Original: Combining these results we get that
Revised: Combining these results we get that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the inequality in the third step is due to g being non-negative and the scenarios in which multiple deltas become active at the same location, namely ∃ i ̸= j : ¯ w (1) i [Equation]
Revised: where the inequality in the third step is due to g being non-negative and the scenarios in which multiple deltas become active at the same location, namely ∃ i ̸

Original:  ¯ w [Equation] [Equation] and ¯ b (1) i = ¯ b (1) j .
Revised: = j : ¯ w (1) i = ¯ w (1) j and ¯ b (1) i = ¯ b (1) j .

Original: Note that if the deltas do not align, then S θ = ∥
Revised: Note that if the deltas do not align, then S θ = ∥

Original: f ∥ R ,g .
Revised: f ∥ R ,g .

Original: Overall we have that
Revised: Overall we have that

Original: [Equation]
Revised: [Equation]

Original: G D ERIVATION OF THE STABILITY NORM IN PRIMAL SPACE
Revised: G D ERIVATION OF THE STABILITY NORM IN PRIMAL SPACE

Original: We defined the multivariate stability norm as ∥ f ∥ R ,g = ⟨| ( R ∗ ) − 1 ∆ f | , g ⟩ S d − 1 × R , where ⟨· , ·⟩ S d − 1 × R denotes the integral inner-product on S d − 1 × R .
Revised: We defined the multivariate stability norm as ∥ f ∥ R ,g = ⟨| ( R ∗ ) − 1 ∆ f | , g ⟩ S d − 1 × R , where ⟨· , ·⟩ S d − 1 × R denotes the integral inner-product on S d − 1 × R .

Original: Supposing that the inverse Radon transform of g exists, then by purely formal reasoning we ought to have
Revised: Supposing that the inverse Radon transform of g exists, then by purely formal reasoning we ought to have

Original: [Equation]
Revised: [Equation]

Original: Therefore, making the (formal) definitions | ∆ f | R ≜ R ∗ | ( R ∗ ) − 1 ∆ f | and ρ ≜ R − 1 g , we may also interpret the stability norm ∥ f ∥ R ,g as the quantity
Revised: Therefore, making the (formal) definitions | ∆ f | R ≜ R ∗ | ( R ∗ ) − 1 ∆ f | and ρ ≜ R − 1 g , we may also interpret the stability norm ∥ f ∥ R ,g as the quantity

Original: [Equation]
Revised: [Equation]

Original: In the event that f and g are smooth, and g is in the range of the classical Radon transform, then the above expression is equal to ∥ f
Revised: In the event that f and g are smooth, and g is in the range of the classical Radon transform, then the above expression is equal to ∥ f

Original: ∥ R ,g .
Revised: ∥ R ,g .

Original: However, this is not generally the case in our setting, and below we show how to give a more precise interpretation of this integral formula using distributional theory.
Revised: However, this is not generally the case in our setting, and below we show how to give a more precise interpretation of this integral formula using distributional theory.

Original: In particular, we show that when f is a finite-width ReLU network | ∆ f | R is equal to the total variation measure of ∆ f ( i.e., the measure-theoretic analog of the absolute value of a function).
Revised: In particular, we show that when f is a finite-width ReLU network | ∆ f | R is equal to the total variation measure of ∆ f ( i.e. , the measure-theoretic analog of the absolute value of a function).

Original: Additionally, in the event that g does not have a classically defined Radon inverse, we show how the integral in (59) can be interpreted using a smoothing approach.
Revised: Additionally, in the event that g does not have a classically defined Radon inverse, we show how the integral in (61) can be interpreted using a smoothing approach.

Original: Let f be a finite width single hidden-layer ReLU network, i.e., f ∈ F k for some finite k .
Revised: Let f be a finite width single hidden-layer ReLU network, i.e. , f ∈ F k for some finite k .

Original: Recall that ( R ∗ ) − 1 ∆ f is a finite weighted sum of Diracs in S d − 1 × R .
Revised: Recall that ( R ∗ ) − 1 ∆ f is a finite weighted sum of Diracs in S d − 1 × R .

Original: Let | ( R ∗ ) − 1 ∆ f | be the associated total variation measure, and define | ∆ f | R = R ∗ | ( R ∗ ) − 1 ∆ f | , where R ∗ is the distributional dual Radon transform.
Revised: Let | ( R ∗ ) − 1 ∆ f | be the associated total variation measure, and define | ∆ f | R = R ∗ | ( R ∗ ) − 1 ∆ f | , where R ∗ is the distributional dual Radon transform.

Original: Here | ∆ f | R is a tempered distribution given by a (positive) weighted sum of Diracs supported on hyperplanes.
Revised: Here | ∆ f | R is a tempered distribution given by a (positive) weighted sum of Diracs supported on hyperplanes.

Original: For example, if f is a single ReLU unit of the form f ( x ) = a σ ( x ⊤ v − b ) with ∥ v ∥ 2 = 1 , then | ∆ f | R is the distribution | a | δ ( x ⊤ v − b ) , that is, for any test function ϕ
Revised: For example, if f is a single ReLU unit of the form f ( x ) = a σ ( x ⊤ v − b ) with ∥ v ∥ 2 = 1 , then | ∆ f | R is the distribution | a | δ ( x ⊤ v − b ) , that is, for any test function ϕ

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] On the other hand, treating ∆ f as a measure defined over a compact subset of R d , its total variation measure | ∆ f | is also equal to | a | δ ( x ⊤ v − b ) .
Revised: On the other hand, treating ∆ f as a measure defined over a compact subset of R d , its total variation measure | ∆ f | is also equal to | a | δ ( x ⊤ v − b ) .

Original: The following result shows that, more generally, when f is any finite width ReLU net then | ∆ f | R and | ∆ f | are equal as measures.
Revised: The following result shows that, more generally, when f is any finite width ReLU net then | ∆ f | R and | ∆ f | are equal as measures.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let f ∈ F k , then | ∆ f | R = | ∆ f | as measures defined over any compact set of R d .
Revised: Let f ∈ F k , then | ∆ f | R = | ∆ f | as measures defined over any compact set of R d .

Original: Proof.
Revised: Proof.

Original: Since f ∈ F k , there exists a representation of f as f ( x ) = (cid:80)
Revised: Since f ∈ F k , there exists a representation of f as f ( x ) = (cid:80)

Original: k ′ i =
Revised: k ′ i =

Original: a i σ ( v ⊤ i x − b [Equation] + x ⊤ q + c where ∥ v i ∥ = 1 for all i , a i ̸ = 0 for all i , and ( v i , b i ) 
Revised: a i σ ( v ⊤ i x − b i ) + x ⊤ q + c where ∥ v i ∥ = 1 for all i , a i ̸ = 0 for all i , and ( v i , b i ) ̸

Original: ̸

Original: = ± ( v j , b j ) for all i ̸ =
Revised: = ± ( v j , b j ) for all i ̸ =

Original: j ( i.e., the knots of all
Revised: j ( i.e. , the knots of all

Original: ReLU units are distinct 8 ), and where k ′ ≤ k .
Revised: ReLU units are distinct 8 ), and where k ′ ≤ k .

Original: Therefore, each ReLU unit in this representation of f maps to a distinct Dirac δ ( v i ,b i ) in Radon space after applying the operator ( R ∗ ) − 1 ∆ , and so 
Revised: Therefore, each ReLU unit in this representation of f maps to a distinct Dirac δ ( v i ,b i ) in Radon space after applying the operator ( R ∗ ) − 1 ∆ , and so [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let X be any compact subset of R d , and let ϕ be any continuous test function defined over X .
Revised: Let X be any compact subset of R d , and let ϕ be any continuous test function defined over X .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Now, we show the same equality holds with | ∆ f | in place of | ∆ f | R .
Revised: Now, we show the same equality holds with | ∆ f | in place of | ∆ f | R .

Original: Let I + denote the set of indices i such that a i > 

Revised: Let I + denote the set of indices i such that a i >

Original: and I − denote the set of indices i such that a i < 0 .
Revised: and I − denote the set of indices i such that a i < 0 .

Original: Define the measures µ + = (cid:80) i ∈ I
Revised: Define the measures µ + = (cid:80) i ∈ I

Original: + a i δ ( v ⊤ i · − b i ) and µ − = − (cid:80) i ∈
Revised: + a i δ ( v ⊤ i · − b i ) and µ − = − (cid:80) i ∈

Original: I − a i δ ( v ⊤ i · − b i ) .
Revised: I − a i δ ( v ⊤ i · − b i ) .

Original: Observe that µ + and µ − are both positive measures whose supports only possibly intersect on a set of measure zero, and ∆ f = µ + − µ − .
Revised: Observe that µ + and µ − are both positive measures whose supports only possibly intersect on a set of measure zero, and ∆ f = µ + − µ − .

Original: This implies the total variation measure of ∆ f is given by | ∆ f | = µ + + µ − = (cid:80) k
Revised: This implies the total variation measure of ∆ f is given by | ∆ f | = µ + + µ − = (cid:80) k

Original: ′ i
Revised: ′ i

Original: =1 | a i | δ ( v ⊤ i ·− b i ) .
Revised: =1 | a i | δ ( v ⊤ i ·− b i ) .

Original: Hence,⟨| ∆ f | , ϕ ⟩ = k ′ (cid:88) i = | a
Revised: Hence,

Original: i |R ϕ ( v i , b i ) , (63) as claimed.
Revised: [Equation] [Equation] as claimed

Original: Note, however, that when f corresponds to finite-width ReLU network, | ∆ f | does not have finite total variation considered as a measure defined over all R d .
Revised: Note, however, that when f corresponds to finite-width ReLU network, | ∆ f | does not have finite total variation considered as a measure defined over all R d .

Original: Due to this technicality, when ρ is not compactly supported, we need to understand the integral in (59) as being with respect to the distribution | ∆ f | R in place of the measure | ∆ f | .
Revised: Due to this technicality, when ρ is not compactly supported, we need to understand the integral in (61) as being with respect to the distribution | ∆ f | R in place of the measure | ∆ f | .

Original: Now we show that when ρ = R − 1 g does not exist in a classical sense, the integral in (59) can still be interpreted using a smoothing approach.
Revised: Now we show that when ρ = R − 1 g does not exist in a classical sense, the integral in (61) can still be interpreted using a smoothing approach.

Original: Proposition 5.
Revised: Proposition 5.

Original: Let f ∈ F k and suppose g is an even, piecewise continuous L 1 function on S d − 1 × R and let ρ
Revised: Let f ∈ F k and suppose g is an even, piecewise continuous L 1 function on S d − 1 × R and let ρ

Original: =
Revised: =

Original: R − 1 g be its distibutional Radon inverse.
Revised: R − 1 g be its distibutional Radon inverse.

Original: Further, assume the support of | ( R ∗ ) − 1 ∆ f | does not intersect the set of points where g is discontinuous.
Revised: Further, assume the support of | ( R ∗ ) − 1 ∆ f | does not intersect the set of points where g is discontinuous.

Original: Then ∥ f ∥ R ,g is finite and
Revised: Then ∥ f ∥ R ,g is finite and

Original: [Equation]
Revised: [Equation]

Original: (64) where the integral above is understood as the finite limit
Revised: where the integral above is understood as the finite limit

Original: [Equation]
Revised: [Equation]

Original: (65) where ρ ϵ is a smooth approximation of ρ defined independently of f whose classical Radon transform g ϵ
Revised: where ρ ϵ is a smooth approximation of ρ defined independently of f whose classical Radon transform g ϵ

Original: =
Revised: =

Original: R ρ
Revised: R ρ

Original: ϵ exists for all ϵ > 0 , and g ϵ
Revised: ϵ exists for all ϵ > 0 , and g ϵ

Original: → g uniformly as ϵ → 0 on any closed subset of R d over which g is continuous.
Revised: → g uniformly as ϵ → 0 on any closed subset of R d over which g is continuous.

Original: Proof.
Revised: Proof.

Original: For any ϵ > 0 let ϕ ϵ ∈ S ( S d − 1 × R ) be a compactly supported even function acting as a smooth approximation of the identity, i.e., for any continuous, even function h vanishing at infinity we have ϕ ϵ ∗ h
Revised: For any ϵ > 0 let ϕ ϵ ∈ S ( S d − 1 × R ) be a compactly supported even function acting as a smooth approximation of the identity, i.e. , for any continuous, even function h vanishing at infinity we have ϕ ϵ ∗ h

Original: → h uniformly as ϵ → 0 , where ∗ denotes convolution of functions on S d − 1 × R .
Revised: → h uniformly as ϵ → 0 , where ∗ denotes convolution of functions on S d − 1 × R .

Original: Define g
Revised: Define g

Original: ϵ
Revised: ϵ

Original: =
Revised: =

Original: ( ϕ ϵ ∗ g ) · χ ϵ , where χ ϵ ( v , b ) is a smooth cutoff function that is equal to one if | b | ≤ 1 /ϵ and rapidly decays to zero for | b | ≥ 1 /ϵ .
Revised: ( ϕ ϵ ∗ g ) · χ ϵ , where χ ϵ ( v , b ) is a smooth cutoff function that is equal to one if | b | ≤ 1 /ϵ and rapidly decays to zero for | b | ≥ 1 /ϵ .

Original: Observe that g ϵ is an even Schwartz function by construction.
Revised: Observe that g ϵ is an even Schwartz function by construction.

Original: Furthermore, since g is piecewise continuous and L 1 (and in particular, it vanishes at infinity), this implies g ϵ
Revised: Furthermore, since g is piecewise continuous and L 1 (and in particular, it vanishes at infinity), this implies g ϵ

Original: → g uniformly over any closed set that does not intersect the set of points where g is discontinuous.
Revised: → g uniformly over any closed set that does not intersect the set of points where g is discontinuous.

Original: Therefore, if we let U ⊂ S d − 1 × R be any closed set containing the support of | ( R ∗ ) − 1 ∆ f | that does not intersect the set of points where g is discontinuous (which is guaranteed to exist since the support of | ( R ∗ ) − 1 ∆ f | is a finite set), then we have g ϵ → g uniformly over U .
Revised: Therefore, if we let U ⊂ S d − 1 × R be any closed set containing the support of | ( R ∗ ) − 1 ∆ f | that does not intersect the set of points where g is discontinuous (which is guaranteed to exist since the support of | ( R ∗ ) − 1 ∆ f | is a finite set), then we have g ϵ → g uniformly over U .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: where the limit is guaranteed to exist since the finite measure | ( R ∗ ) − 1 ∆ f | is a continuous linear functional over C 0 ( U ) , the space of continuous functions over U vanishing at infinity.
Revised: where the limit is guaranteed to exist since the finite measure | ( R ∗ ) − 1 ∆ f | is a continuous linear functional over C 0 ( U ) , the space of continuous functions over U vanishing at infinity.

Original: Finally, since g ϵ is Schwartz,Theorem 7.7 from (Solmon, 1987) guarantees ρ ϵ =
Revised: Finally, since g ϵ is Schwartz, (Solmon, 1987, Thm. 7.7) guarantees ρ ϵ =

Original: R − 1 g ϵ exists as a C ∞ -smooth function on R d that is also integrable along hyperplanes and for which the classical Radon inversion formula holds: R ρ ϵ =
Revised: R − 1 g ϵ exists as a C ∞ -smooth function on R d that is also integrable along hyperplanes and for which the classical Radon inversion formula holds: R ρ ϵ =

Original: g ϵ .
Revised: g ϵ .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original:  as claimed.
Revised: (69) as claimed.

Original: The assumption made above that the support of | ( R ∗ ) − 1 ∆ f | does not intersect the set of points where g is not overly restrictive.
Revised: The assumption made above that the support of | ( R ∗ ) − 1 ∆ f | does not intersect the set of points where g is not overly restrictive.

Original: For example, this assumption holds when f corresponds to a differentiable minimizer of the squared loss defined in terms of a finite set of training points and g is the data dependent weighting function defined in (12).
Revised: For example, this assumption holds when f corresponds to a differentiable minimizer of the squared loss defined in terms of a finite set of training points and g is the data dependent weighting function defined in (12).

Original: In this case, the discontinuity set of g ( v , b ) corresponds to the set of hyperplanes { x : R d : x ⊤ v = b } that intersect one or more of the training points.
Revised: In this case, the discontinuity set of g ( v , b ) corresponds to the set of hyperplanes { x : R d : x ⊤ v = b } that intersect one or more of the training points.

Original: And f corresponds to a differentiable minimizer if and only if the hyperplanes defined by the knots of the
Revised: And f corresponds to a differentiable minimizer if and only if the hyperplanes defined by the knots of the

Original: ReLU units making up f (i.e., the support of | ( R ∗ ) − 1 f | ) do not intersect any training points.
Revised: ReLU units making up f ( i.e. , the support of | ( R ∗ ) − 1 f | ) do not intersect any training points.

Original: H E XAMPLES OF g AND ρ H. T WO DATAPOINTS
Revised: H E XAMPLES OF g AND ρ H. T WO DATAPOINTS

Original: For this example, it is easy to calculate that
Revised: For this example, it is easy to calculate that

Original: [Equation]
Revised: [Equation]

Original: 68) where α is a positive constant.
Revised: where α is a positive constant.

Original: We compute ρ = R − 1 g by first determining its Laplacian, ∆ ρ , then inverting and Laplacian to recover ρ .
Revised: We compute ρ = R − 1 g by first determining its Laplacian, ∆ ρ , then inverting and Laplacian to recover ρ .

Original: First, the intertwining property of the Laplacian and the Radon transform gives
Revised: First, the intertwining property of the Laplacian and the Radon transform gives

Original: [Equation]
Revised: [Equation]

Original: [Equation] Therefore, by the Fourier slice theorem (Helgason, 1999), for all ( v , s ) ∈ S d − 1 × R we have[Equation]
Revised: Therefore, by the Fourier slice theorem (Helgason, 1999), for all ( v , s ) ∈ S d − 1 × R we have

Original: [Equation]
Revised: [Equation]

Original: where F {·} is the 2-D Fourier transform, and F b {·} is the Fourier transform in the b variable.
Revised: where F {·} is the 2-D Fourier transform, and F b {·} is the Fourier transform in the b variable.

Original: For fixed v , the function b → g ( v , b ) is continuous and piecewise linear with knots at 0 and ±| v 1 | , and it is easy to see that
Revised: For fixed v , the function b → g ( v , b ) is continuous and piecewise linear with knots at 0 and ±| v 1 | , and it is easy to see that

Original: [Equation]
Revised: [Equation]

Original: 71) which implies
Revised: ) which implies

Original: [Equation]
Revised: [Equation]

Original: If we restrict the unit-norm vector v = ( v 1 , v 2 ) to be such that v 1 ≥ 0 and define ξ = s v then we see that x ⊤ 1 ξ = s | v 1 | and x ⊤ 2 ξ = − s | v 1 | .
Revised: If we restrict the unit-norm vector v = ( v 1 , v 2 ) to be such that v 1 ≥ 0 and define ξ = s v then we see that x ⊤ 1 ξ = s | v 1 | and x ⊤ 2 ξ = − s | v 1 | .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original:  and inverting the Fourier transform gives
Revised: 75) and inverting the Fourier transform gives

Original: [Equation]
Revised: [Equation]

Original: Finally, since φ ( x ) = 12 π log( ∥ x ∥ ) is the fundamental solution of Poisson’s equation in 2D ( i.e., ∆ φ = δ , where δ is a Dirac centered at origin), we have
Revised: Finally, since φ ( x ) = 12 π log( ∥ x ∥ ) is the fundamental solution of Poisson’s equation in 2D ( i.e. , ∆ φ = δ , where δ is a Dirac centered at origin), we have

Original: [Equation]
Revised: [Equation]

Original: While each term in the sum above not absolutely integrable along lines, their sum is absolutely integrable along lines.
Revised: While each term in the sum above not absolutely integrable along lines, their sum is absolutely integrable along lines.

Original: This is because the function t → log( | t | ) is absolutely integrable over any neighborhood of the origin, and by a multipole expansion we may show that ρ ( x ) =
Revised: This is because the function t → log( | t | ) is absolutely integrable over any neighborhood of the origin, and by a multipole expansion we may show that ρ ( x ) =

Original: O ( ∥ x
Revised: O ( ∥ x

Original: ∥ − 2 ) as x → ∞ , which is also absolutely integrable along lines.
Revised: ∥ − 2 ) as x → ∞ , which is also absolutely integrable along lines.

Original: H. I SOTROPIC DATA DISTRIBUTION
Revised: H. I SOTROPIC DATA DISTRIBUTION

Original: For an isotropic data distribution, i.e., P (cid:0) x ⊤ v > b (cid:1) =
Revised: For an isotropic data distribution, i.e. , P (cid:0) x ⊤ v > b (cid:1) =

Original: M ( b ) for any v that satisfies ∥ v ∥ = 1 , we will have that
Revised: M ( b ) for any v that satisfies ∥ v ∥ = 1 , we will have that

Original: [Equation]
Revised: [Equation]

Original: Then, from symmetry and assuming that ˜ g is decreasing in b we obtain
Revised: Then, from symmetry and assuming that ˜ g is decreasing in b we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that g only depends on | b | and is decreasing in | b | .
Revised: Note that g only depends on | b | and is decreasing in | b | .

Original: Considering the parameter space representation for the stability norm
Revised: Considering the parameter space representation for the stability norm

Original: [Equation]
Revised: [Equation]

Original: we can see that solutions with larger | b i | , i.e., solutions which are more flat in function space, will have smaller stability norm.
Revised: we can see that solutions with larger | b i | , i.e. , solutions which are more flat in function space, will have smaller stability norm.

Original: We now characterize ρ = R − 1 g .
Revised: We now characterize ρ = R − 1 g .

Original: For simplicity, we focus on the two-dimensional setting ( d = 2 ).
Revised: For simplicity, we focus on the two-dimensional setting ( d = 2 ).

Original: Since g ( v , b ) does not depend on v , we drop this dependence and simply write g ( b ) .
Revised: Since g ( v , b ) does not depend on v , we drop this dependence and simply write g ( b ) .

Original: Note that this implies ρ is a radial function.
Revised: Note that this implies ρ is a radial function.

Original: Let ˜ ρ ( r ) denote the radial profile of ρ , i.e., ρ ( x ) =
Revised: Let ˜ ρ ( r ) denote the radial profile of ρ , i.e. , ρ ( x ) =

Original: ˜
Revised: ˜

Original: ρ ( ∥ x ∥ ) .
Revised: ρ ( ∥ x ∥ ) .

Original: We additionally make the following assumptions: g ( b ) is twice continuously differentiable away from the origin, and both g and its weak derivative g ′ are bounded and absolutely integrable.
Revised: We additionally make the following assumptions: g ( b ) is twice continuously differentiable away from the origin, and both g and its weak derivative g ′ are bounded and absolutely integrable.

Original: In this case, ˜ ρ has the integral formula
Revised: In this case, ˜ ρ has the integral formula

Original: [Equation]
Revised: [Equation]

Original: The assumptions on g above are sufficient to show the integrand in (79) is absolutely integrable over [ r, ∞ ) for r > 0 .
Revised: The assumptions on g above are sufficient to show the integrand in (81) is absolutely integrable over [ r, ∞ ) for r > 0 .

Original: Since g ( | b | ) is assumed to be decreasing in | b | , we have − g ′ ( b ) ≥ 0 for all b > 0 , which shows ˜ ρ ( r ) ≥ 0 for all r > 0 .
Revised: Since g ( | b | ) is assumed to be decreasing in | b | , we have − g ′ ( b ) ≥ 0 for all b > 0 , which shows ˜ ρ ( r ) ≥ 0 for all r > 0 .

Original: However, if g is not smooth at the origin, then g ′ ( b ) =
Revised: However, if g is not smooth at the origin, then g ′ ( b ) =

Original: O (1) as b → 0 + , and elementary analysis shows ˜ ρ ( r )
Revised: O (1) as b → 0 + , and elementary analysis shows ˜ ρ ( r )

Original: =
Revised: =

Original: O (log( r )) as r → 0 + and ˜ ρ ( r )
Revised: O (log( r )) as r → 0 + and ˜ ρ ( r )

Original: =
Revised: =

Original: O (1 /r ) as r → + ∞ .
Revised: O (1 /r ) as r → + ∞ .

Original: Finally, if we additionally assume g ′ ( b ) is non-increasing for b > 0 , then ρ ( r ) is strictly decreasing for r > 0 .
Revised: Finally, if we additionally assume g ′ ( b ) is non-increasing for b > 0 , then ρ ( r ) is strictly decreasing for r > 0 .

Original: To see this, fix any r ′ > r , and define δ = r ′ − r .
Revised: To see this, fix any r ′ > r , and define δ = r ′ − r .

Original: Using the change of variables
Revised: Using the change of variables

Original: b (cid:55)→ b − δ , we may show
Revised: b (cid:55)→ b − δ , we may show

Original: [Equation]
Revised: [Equation]

Original: Since g ′ is assumed to be non-increasing, we have g ′ ( b + δ ) ≤ g ′ ( b ) and it is elementary to show (( b + δ ) 2 − r 2 ) − 1 / 2 < ( b 2 − r 2 ) − 1 / 2 for all b > r , which shows the integrand in (80) is pointwise strictly bounded above by the integrand in (79) for all b > r , hence ˜
Revised: Since g ′ is assumed to be non-increasing, we have g ′ ( b + δ ) ≤ g ′ ( b ) and it is elementary to show (( b + δ ) 2 − r 2 ) − 1 / 2 < ( b 2 − r 2 ) − 1 / 2 for all b > r , which shows the integrand in (82) is pointwise strictly bounded above by the integrand in (81) for all b > r , hence ˜

Original: ρ
Revised: ρ

Original: ( r ′ ) <
Revised: ( r ′ ) <

Original: ˜ ρ ( r ) .
Revised: ˜ ρ ( r ) .

Original: In the case of 2D Gaussian distributed data X ∼ N ( 0 , I ) , then M ( b ) is the complementary of the
Revised: In the case of 2D Gaussian distributed data X ∼ N ( 0 , I ) , then M ( b ) is the complementary of the

Original: CDF of a normal random variable: M ( b ) = 1 √ 2 π (cid:82) ∞
Revised: CDF of a normal random variable: M ( b ) = 1 √ 2 π (cid:82) ∞

Original: b e − b 2 / 2 d b .
Revised: b e − b 2 / 2 d b .

Original: It is easy to verify that the resulting g function satisfies the above assumptions ( g ( b ) is decreasing, twice continuously differentiable away from the origin, both g and its weak derivative g ′ are bounded and absolutely integrable, and g ′ is non-increasing).
Revised: It is easy to verify that the resulting g function satisfies the above assumptions ( g ( b ) is decreasing, twice continuously differentiable away from the origin, both g and its weak derivative g ′ are bounded and absolutely integrable, and g ′ is non-increasing).

Original: Therefore, the resulting ρ has the all the properties outlined above.
Revised: Therefore, the resulting ρ has the all the properties outlined above.

Original: I D EPTH SEPARATION 
Revised: I D EPTH SEPARATION PROOFS

Original: Before giving the proofs in this section we introduce some additional notation.
Revised: Before giving the proofs in this section we introduce some additional notation.

Original: Let X denote the closed convex hull of the training points and X its open interior.
Revised: Let X denote the closed convex hull of the training points and X its open interior.

Original: Additionally, let Y =
Revised: Additionally, let Y =

Original: ( v , b ) ∈
Revised: ( v , b ) ∈

Original: S d − 1 × R : v ⊤ x > b for some x ∈ X } , and let Y denote its closure.
Revised: S d − 1 × R : v ⊤ x > b for some x ∈ X } , and let Y denote its closure.

Original: Note that for any smooth function ϕ with support contained in X , the Radon transform R ϕ has support contained in Y .
Revised: Note that for any smooth function ϕ with support contained in X , the Radon transform R ϕ has support contained in Y .

Original: Finally, for any distribution h and open set U , we let h | U denote its restriction to U .
Revised: Finally, for any distribution h and open set U , we let h | U denote its restriction to U .

Original: I. P ROOF OF P ROPOSITION
Revised: I. P ROOF OF P ROPOSITION

Original: First, we show that the convergence of a sequence of functions f k to f in L 1 -norm over X implies that the sequence of distributions ∆ f k | X converges weakly to the distribution ∆ f | X .
Revised: First, we show that the convergence of a sequence of functions f k to f in L 1 -norm over X implies that the sequence of distributions ∆ f k | X converges weakly to the distribution ∆ f | X .

Original: For all test functions ϕ ∈ S ( X ) we have
Revised: For all test functions ϕ ∈ S ( X ) we have

Original: [Equation]
Revised: [Equation]

Original: (81) where we used Holder’s inequality to achieve the final bound.
Revised: [Equation] where we used Holder’s inequality to achieve the final bound.

Original: Therefore, we have lim k →∞ ⟨ ∆ f k , ϕ ⟩ → ⟨ ∆ f, ϕ ⟩ , which proves the weak convergence.
Revised: Therefore, we have lim k →∞ ⟨ ∆ f k , ϕ ⟩ → ⟨ ∆ f, ϕ ⟩ , which proves the weak convergence.

Original: Next, we show ( R ∗ ) − 1 ∆ f k | Y converges weakly to ( R ∗ ) − 1 ∆ f | Y .
Revised: Next, we show ( R ∗ ) − 1 ∆ f k | Y converges weakly to ( R ∗ ) − 1 ∆ f | Y .

Original: For all test functions φ ∈ S H ( Y ) we have
Revised: For all test functions φ ∈ S H ( Y ) we have

Original: [Equation]
Revised: [Equation]

Original: Since ϕ vanishes outside Y , by the support theorem (Helgason, 1999, Corollary 2.8) we are ensured that R − 1 φ has support contained in X , hence R − 1 φ ∈ S ( X ) .
Revised: Since ϕ vanishes outside Y , by the support theorem (Helgason, 1999, Corollary 2.8) we are ensured that R − 1 φ has support contained in X , hence R − 1 φ ∈ S ( X ) .

Original: The desired result now follows immediately by weak convergence of ∆ f k | X to ∆ f | X .
Revised: The desired result now follows immediately by weak convergence of ∆ f k | X to ∆ f | X .

Original: Since g has support contained in Y , this further implies that the distribution g · ( R ∗ ) − 1 ∆ f k converges weakly to the distribution g · ( R ∗ ) − 1 ∆ f .
Revised: Since g has support contained in Y , this further implies that the distribution g · ( R ∗ ) − 1 ∆ f k converges weakly to the distribution g · ( R ∗ ) − 1 ∆ f .

Original: Finally, since ∥ f k ∥ R ,g = ∥ g · ( R ∗ ) − 1 ∆ f k ∥ TV is bounded by assumption, this implies each g · ( R ∗ ) − 1 ∆ f k is a measure having finite total variation, hence their weak limit g · ( R ∗ ) − 1 ∆ f is also a measure with finite total variation ( i.e., the weak limit of order-0 distributions that are bounded in TV-norm is also an order-0 distribution).
Revised: Finally, since ∥ f k ∥ R ,g = ∥ g · ( R ∗ ) − 1 ∆ f k ∥ TV is bounded by assumption, this implies each g · ( R ∗ ) − 1 ∆ f k is a measure having finite total variation, hence their weak limit g · ( R ∗ ) − 1 ∆ f is also a measure with finite total variation ( i.e. , the weak limit of order-0 distributions that are bounded in TV-norm is also an order-0 distribution).

Original: Therefore, ∥ f ∥ R ,g = ∥
Revised: Therefore, ∥ f ∥ R ,g = ∥

Original: g · ( R ∗ ) − 1 ∆ f ∥ TV is finite, as claimed.
Revised: g · ( R ∗ ) − 1 ∆ f ∥ TV is finite, as claimed.

Original: I. P ROOF OF P ROPOSITION
Revised: I. P ROOF OF P ROPOSITION

Original: To show the pyramid function p has infinite stability norm, we prove that g · ( R ∗ ) − 1 ∆ p is a distribution of order > 0 , which implies 
Revised: To show the pyramid function p has infinite stability norm, we prove that g · ( R ∗ ) − 1 ∆ p is a distribution of order > 0 , which implies [Equation]

Original: [Equation]
Revised: [Equation]

Original: First, observe that the Laplacian ∆ p is an order-0 distribution whose support is contained in the unit ℓ 1 -ball.
Revised: First, observe that the Laplacian ∆ p is an order-0 distribution whose support is contained in the unit ℓ 1 -ball.

Original: This implies the distribution ( R ∗ ) − 1 ∆ p , is supported on a compact set K in Radon domain.
Revised: This implies the distribution ( R ∗ ) − 1 ∆ p , is supported on a compact set K in Radon domain.

Original: By our assumption on the convex hull of the training points, g ( v , b ) > 0 for all ( v , b ) ∈ K .
Revised: By our assumption on the convex hull of the training points, g ( v , b ) > 0 for all ( v , b ) ∈ K .

Original: Since g is piecewise continuous and K is compact, this implies there exists constants
Revised: Since g is piecewise continuous and K is compact, this implies there exists constants

Original: c 1 , c 2 >such that c 1 ≤ g ( v , b ) ≤ c 2 for all ( v , b ) ∈ K .
Revised: c 1 , c 2 >such that c 1 ≤ g ( v , b ) ≤ c 2 for all ( v , b ) ∈ K .

Original: Therefore, we see that g · ( R ∗ ) − 1 ∆ p is an orderdistribution if and only if ( R ∗ ) − 1 ∆ p is an order-0 distribution.
Revised: Therefore, we see that g · ( R ∗ ) − 1 ∆ p is an orderdistribution if and only if ( R ∗ ) − 1 ∆ p is an order-0 distribution.

Original: However, by a result in (Ongie et al., 2020), we know ( R ∗ ) − 1 ∆ p has order > 0 ( i.e., in the terminology of (Ongie et al., 2020), p has infinite R -norm). 
Revised: However, by a result in (Ongie et al., 2020), we know ( R ∗ ) − 1 ∆ p has order > 0 ( i.e. , in the terminology of (Ongie et al., 2020), p has infinite R -norm). Additionally, we show this by direct calculation in App.

Original:  Therefore, g  ( R ∗ ) − 1 ∆ p must be a distribution of order > 0 and hence ∥ p ∥
Revised: M in the case of input dimension d = 2 . Therefore, g · ( R ∗ ) − 1 ∆ p must be a distribution of order > 0 and hence ∥ p ∥

Original: R ,g = + ∞ as claimed.
Revised: R ,g = + ∞ as claimed.

Original: I.  S TABILITY OF THE TWO HIDDEN - LAYER IMPLEMENTATION OF p ( x )
Revised: I.  S TABILITY OF THE TWO HIDDEN - LAYER IMPLEMENTATION OF p ( x )

Original: Let us focus on the under-parameterized setting, in which there exists a single optimal input-output predictor p ( x ) that globally minimizes the loss.
Revised: Let us focus on the under-parameterized setting, in which there exists a single optimal input-output predictor p ( x ) that globally minimizes the loss.

Original: In this case, the set of all global minima corresponds to different implementations of p ( x ) .
Revised: In this case, the set of all global minima corresponds to different implementations of p ( x ) .

Original: Under this setting, we will prove that there exists a set of nonzero Lebesgue measure such that for any initialization inside this set, GD necessarily converges to p ( x ) .
Revised: Under this setting, we will prove that there exists a set of nonzero Lebesgue measure such that for any initialization inside this set, GD necessarily converges to p ( x ) .

Original: To do so, we will first prove that for any minimum point θ ∗ ∈ R m corresponding to an implementation of p ( x ) , there exists a nonzero step size η with which θ ∗ is linearly stable.
Revised: To do so, we will first prove that for any minimum point θ ∗ ∈ R m corresponding to an implementation of p ( x ) , there exists a nonzero step size η with which θ ∗ is linearly stable.

Original: Furthermore, we will show that there exists a set T sloc ( θ ∗ ) embedded in a subspace of dimension m − m Null , in which any initialization converges to θ ∗ .
Revised: Furthermore, we will show that there exists a set T sloc ( θ ∗ ) embedded in a subspace of dimension m − m Null , in which any initialization converges to θ ∗ .

Original: Here m is the number of parameters in our two hidden-layer network, and m Null is the number of zero eigenvalues of ∇ 2 L at θ ∗ .
Revised: Here m is the number of parameters in our two hidden-layer network, and m Null is the number of zero eigenvalues of ∇ 2 L at θ ∗ .

Original: Next, we will show that there is a connected set of minima Θ ∗ around θ ∗ , such that the union (cid:83) θ ∈ Θ ∗ T sloc ( θ ) has a nonzero Lebesgue measure.
Revised: Next, we will show that there is a connected set of minima Θ ∗ around θ ∗ , such that the union (cid:83) θ ∈ Θ ∗ T sloc ( θ ) has a nonzero Lebesgue measure.

Original: Let us start with some minimum point θ ∗ , which corresponds to an implementation of p ( x ) .
Revised: Let us start with some minimum point θ ∗ , which corresponds to an implementation of p ( x ) .

Original: GD’s update rule is
Revised: GD’s update rule is

Original: [Equation]
Revised: [Equation]

Original: Define the mapping
Revised: Define the mapping

Original: [Equation]
Revised: [Equation]

Original: Then (84) can be written as
Revised: Then (86) can be written as

Original: [Equation]
Revised: [Equation]

Original: This equation describes the full dynamics of GD using the nonlinear mapping T .
Revised: This equation describes the full dynamics of GD using the nonlinear mapping T .

Original: Note that in this representation, θ ∗ is an equilibrium point of T , i.e., T ( θ ∗ )
Revised: Note that in this representation, θ ∗ is an equilibrium point of T , i.e. , T ( θ ∗ )

Original: = θ ∗ .
Revised: = θ ∗ .

Original: We would like to show that it is possible to converge to θ ∗ .
Revised: We would like to show that it is possible to converge to θ ∗ .

Original: Assume there is a finite number of training samples n , and none of them coincide with the knots of p .
Revised: Assume there is a finite number of training samples n , and none of them coincide with the knots of p .

Original: Then T is differentiable in a small neighborhood of θ ∗ .
Revised: Then T is differentiable in a small neighborhood of θ ∗ .

Original: The Jacobian matrix of T is
Revised: The Jacobian matrix of T is

Original: [Equation]
Revised: [Equation]

Original: and its eigenvalues are
Revised: and its eigenvalues are

Original: [Equation]
Revised: [Equation]

Original: In this setting, the loss’ Hessian at θ ∗ has non-negative and bounded eigenvalues.
Revised: In this setting, the loss’ Hessian at θ ∗ has non-negative and bounded eigenvalues.

Original: Particularly, there exists 10 a sufficiently small step size η that satisfies
Revised: Particularly, there exists 10 a sufficiently small step size η that satisfies

Original: [Equation]
Revised: [Equation]

Original: for all i .
Revised: for all i .

Original: Using the Center and Stable Manifold Theorem (Shub, 2013, Th. III.7), there exists a bounded set T sloc ( θ ∗ ) such that
Revised: Using the Center and Stable Manifold Theorem (Shub, 2013, Th. III.7), there exists a bounded set T sloc ( θ ∗ ) such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: for some 0 ≤ α < 1 .
Revised: for some 0 ≤ α < 1 .

Original: Here T sloc ( θ ∗ ) is tangent to the hyperplane that contains θ ∗ and is spanned by the nonzero eigenvectors of ∇ 2 L ( θ ∗ ) .
Revised: Here T sloc ( θ ∗ ) is tangent to the hyperplane that contains θ ∗ and is spanned by the nonzero eigenvectors of ∇ 2 L ( θ ∗ ) .

Original: Thus, assume that the initial point θ 0 ∈ T sloc ( θ ∗ ) , then
Revised: Thus, assume that the initial point θ 0 ∈ T sloc ( θ ∗ ) , then

Original: [Equation]
Revised: [Equation]

Original: 91) which shows that with any initialization in T sloc ( θ ∗ ) , GD’s iterations converge to θ ∗ .
Revised: which shows that with any initialization in T sloc ( θ ∗ ) , GD’s iterations converge to θ ∗ .

Original: Next, note that there is a neighborhood of θ ∗ within which the set of global minima of the loss form a smooth m Null -dimensional manifold 11 .
Revised: Next, note that there is a neighborhood of θ ∗ within which the set of global minima of the loss form a smooth m Null -dimensional manifold 11 .

Original: Let us denote this set of global minima around θ ∗ by Θ ∗ , and set η < 2 / max θ ∈ Θ ∗ { λ max ( ∇ 2 L ) } and limit the set Θ ∗ such that ∀ i : η ̸ = 1 /λ i .
Revised: Let us denote this set of global minima around θ ∗ by Θ ∗ , and set η < 2 / max θ ∈ Θ ∗ { λ max ( ∇ 2 L ) } and limit the set Θ ∗ such that ∀ i : η ̸ = 1 /λ i .

Original: Then, for each minimum θ ∈ Θ ∗ , according to the first part of the proof, there exists a ( m − m Null ) -dimensional set T sloc ( θ ) .
Revised: Then, for each minimum θ ∈ Θ ∗ , according to the first part of the proof, there exists a ( m − m Null ) -dimensional set T sloc ( θ ) .

Original: Now, since each T sloc ( θ ) is contained in a hyperplane that is orthogonal to the tangent of Θ ∗ at θ , and the dimension of the tangent of Θ ∗ at θ ∗ is m Null , we have that the dimension of the union of these sets, (cid:83) θ ∈ Θ ∗ T sloc ( θ ) , is m .
Revised: Now, since each T sloc ( θ ) is contained in a hyperplane that is orthogonal to the tangent of Θ ∗ at θ , and the dimension of the tangent of Θ ∗ at θ ∗ is m Null , we have that the dimension of the union of these sets, (cid:83) θ ∈ Θ ∗ T sloc ( θ ) , is m .

Original: Thus, the set (cid:83) θ ∈ Θ ∗ T sloc ( θ ) is of nonzero Lebesgue measure within R m and for any initialization in (cid:83) θ ∈ Θ ∗ T sloc ( θ ) , GD converges to p ( x ) .
Revised: Thus, the set (cid:83) θ ∈ Θ ∗ T sloc ( θ ) is of nonzero Lebesgue measure within R m and for any initialization in (cid:83) θ ∈ Θ ∗ T sloc ( θ ) , GD converges to p ( x ) .

Original: J G ENERAL LOSS FUNCTIONS
Revised: J G ENERAL LOSS FUNCTIONS

Original: In this section, we discuss how our results can be extended to general loss function with a unique finite root.
Revised: In this section, we discuss how our results can be extended to general loss function with a unique finite root.

Original: Assume some general loss function
Revised: Assume some general loss function

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( a, b ) is twice differentiable w.r.t.
Revised: where ℓ ( a, b ) is twice differentiable w.r.t.

Original: a and is minimized when a = b , i.e.,
Revised: a and is minimized when a = b , i.e. ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Then, we can calculate the loss’ gradient
Revised: Then, we can calculate the loss’ gradient

Original: [Equation]
Revised: [Equation]

Original: and Hessian matrix
Revised: and Hessian matrix

Original: [Equation]
Revised: [Equation]

Original:  where ℓ ′′ ( a, b ) ≜ ∂ 2 ∂a 2 ℓ ( a, b ) , and in the last transition we used f ( x j )
Revised: [Equation] where ℓ ′′ ( a, b ) ≜ ∂ 2 ∂a 2 ℓ ( a, b ) , and in the last transition we used f ( x j )

Original: =
Revised: =

Original: y j and ℓ ′ ( a, a ) = 0 for all a ∈ R .
Revised: y j and ℓ ′ ( a, a ) = 0 for all a ∈ R .

Original: If ℓ ′′ ( f ( x j ) , y j ) =
Revised: If ℓ ′′ ( f ( x j ) , y j ) =

Original: C > 0 for all training points, then we can generalize our results by simply multiplying the RHS of (45) by C .
Revised: C > 0 for all training points, then we can generalize our results by simply multiplying the RHS of (47) by C .

Original: If not, the analysis can still be used but we need to add a weigtning term to the stability norm which depends on the value of ℓ ′′ ( f ( x j ) , y j ) for each data point.
Revised: If not, the analysis can still be used but we need to add a weigtning term to the stability norm which depends on the value of ℓ ′′ ( f ( x j ) , y j ) for each data point.

Original: K P ROOF OF L EMMA
Revised: K P ROOF OF L EMMA

Original: In this section, our goal is to upper bound the top eigenvalue of the flattest implementation of a predictor function f .
Revised: In this section [Equation] our goal is to upper bound the top eigenvalue of the flattest implementation of a predictor function f .

Original: Here we use notation and some derivations form App. F.1.
Revised: Here we use notation and some derivations form App. F.1.

Original: Let q denote the top right singular vector of Φ , then
Revised: Let q denote the top right singular vector of Φ , then

Original: [Equation]
Revised: [Equation]

Revised: where in the inequality we used (cid:16)(cid:80) nj =1 u j (cid:17) 2 ≤ n for all u ∈ S n − 1 and substituted

Revised: [Equation]

Revised: Let Θ( f ) be the set of all implementations corresponding to f .

Revised: Since substituting

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: 1 i

Revised: [Equation]

Revised: [Equation]

Revised: i w (2)

Revised: [Equation]

Revised: (100) does not affect the network’s functionality f , we have

Revised: [Equation]

Revised: i = 

Revised: [Equation]

Revised:  

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: 
Revised:  

Revised: [Equation]

Revised: [Equation]

Revised: l

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: = [Equation] i

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ¯

Original: [Equation]

Original: [Equation]

Original: ¯

Original: [Equation]

Original: [Equation]

Original: (cid:17) 2 

Original:  
Revised:  

Original:  (96) where in the inequality we used (cid:16)(cid:80) nj =1 u j (cid:17) 2 ≤ n for all u ∈ S n − 1 and substituted

Original: [Equation]
Revised: [Equation]

Original: Let Θ( f ) be the set of all implementations corresponding to f .

Original: Since substituting

Original: [Equation]
Revised: [Equation]

Original: does not affect the network’s functionality f , we have
Revised:  

Original: [Equation]
Revised: [Equation]

Original: A necessary condition for optimality is that the derivative of the objective with respect to c
Revised: A necessary condition for optimality is that the derivative of the objective with respect to c

Original: i is equal to zero:
Revised: i is equal to zero:

Original: [Equation]
Revised: [Equation]

Original: It is easy to verify that these solutions for { c i } are indeed global minima.
Revised: It is easy to verify that these solutions for { c i } are indeed global minima.

Original: Plugging this in, we getmin θ ∈ Θ( f ) λ max (cid:0) ∇ 2 θ L (cid:1) ≤ min θ ∈ Θ( f ) n     n +k (cid:88) i =     (cid:13) (cid:13) (cid:13) w (1) i (cid:13) (cid:13) (cid:13) (cid:12) (cid:12) (cid:12) w (2) i (cid:12) (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12) n (cid:88) j = q j σ (cid:16) x ⊤ j ¯ w (1) i − ¯ b (1) i (cid:17)(cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12) (cid:118)(cid:117)(cid:117)(cid:117)
Revised: Plugging this in, we get

Original: (cid:116)
Revised: [Equation]

Original: d (cid:88) l =   n (cid:88)
Revised: n   n + [Table]

Original: j = q j x j,l I j,i   
Revised: l I j,i   [Equation]

Original: +   n (cid:88)
Revised: n [Equation]  [Equation]

Revised: I

Original: j =  q j I j,i 
Revised: [Equation] Since ∥ q ∥ = 1 , the right hand sides of these inequalities are independent of q . Thus, using these inequalities to further upper bound the top eigenvalue we have min [Equation]

Original: [- 
     -]
Revised: [- 
     -]l =

Original:    .
Revised: [Equation]

Original: Now, by Cauchy–Schwarz inequality three times we get (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88)

Original: j = q j σ (cid:16) x ⊤ j ¯ w (1) i − ¯ b (1) i (cid:17)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ ∥ q ∥ (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) j = σ 2 (cid:16) x ⊤ j ¯ w (1) i − ¯ b (1) i (cid:17) ,  n (cid:88) j = q j x j,l
Revised:  

Original: [Equation]
Revised: [Equation]

Original: j,i  
Revised: (cid:88) j =

Original: [Equation]
Revised: [Equation]

Original: ≤ ∥
Revised: j

Original: [Equation]
Revised: [Equation]

Original: ∥n (cid:88) j = x 2 j,l I[Equation] ,   n (cid:88) j=
Revised: I j

Original: [Equation]
Revised: [Equation]

Original: j I j,i  ≤ ∥
Revised: 

Original: [Equation] ∥n (cid:88)j
Revised:  +   n (cid:88)

Original:  =
Revised: j =

Original: I j
Revised: I j

Original: [Equation]
Revised: [Equation]

Original: i . Since ∥ q ∥ =
Revised: i     

Original: [Table] the right hand sides of these inequalities are independent of q . Thus, using these inequalities to further upper bound the top eigenvalue we have [Table]
Revised: 

Original: [Equation]
Revised: [Equation]

Original: To continue upper bounding the sharpness ( λ max ( ∇ 2 L ) ) of the flattest implementation, we can consider some implementation of f .
Revised: To continue upper bounding the sharpness ( λ max ( ∇ 2 L ) ) of the flattest implementation, we can consider some implementation of f .

Original: Specifically, since f ∈ F k , it can be represented as
Revised: Specifically, since f ∈ F k , it can be represented as

Original: [Equation]
Revised: [Equation]

Original: where ∥ v i ∥ = 1 for all i , ∥ h ∥ = 1 , a i ̸ = 0 for all i , and ( v i , b i ) ̸
Revised: where ∥ v i ∥ = 1 for all i , ∥ h ∥ = 1 , a i ̸ = 0 for all i , and ( v i , b i ) ̸

Original: = ± ( v j , b j ) for all i ̸ =
Revised: = ± ( v j , b j ) for all i ̸ =

Original: j ( i.e., the knots of all ReLU units are distinct 12 ), and where k ′ ≤ k .
Revised: j ( i.e. , the knots of all ReLU units are distinct 12 ), and where k ′ ≤ k .

Original: Thus, we use the following implementation of f
Revised: Thus, we use the following implementation of f

Original: [Equation]
Revised: [Equation]

Original: for i ∈ [ k ′ ] , where
Revised: for i ∈ [ k ′ ] , where

Original: [Equation]
Revised: [Equation]

Original: Additionally, if needed, we add two ReLU neurons to implement the linear component.
Revised: Additionally, if needed, we add two ReLU neurons to implement the linear component.

Original: [Equation]
Revised: [Equation]

Original: Thus, 
Revised: Thus, 1 +

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: min

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we used
Revised: where in the last inequality we used

Original: [Equation]
Revised: [Equation]

Original: and

Original:  ∥ x j ∥ 2 +1 > 0 and thus removing the indicator term only increases the RHS of (108). Recall that we denoted C i ⊆ { x j } be the set of training points for which the i th neuron is active, and n i = | 
Revised: and ∥ x j ∥ 2 +1 > 0 and thus removing the indicator term only increases the RHS of (110). Recall that we denoted C i ⊆ { x j } be the set of training points for which the i th neuron is active [Equation] and n i = | C

Original: C i | .
Revised: i | .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Additionally,
Revised: Additionally,

Original: [Equation]
Revised: [Equation]

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that,
Revised: Note that,

Original: [Equation]
Revised: [Equation]

Original: where Σ X is the covariance matrix of X .
Revised: where Σ x is the covariance matrix of x .

Original: Additionally,
Revised: Additionally,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any x ∈ R d

Original: [Equation]
Revised: [Equation]

Revised: Therefore, for any x ∈ R d | β | ≤ ∥∇ f ( x ) ∥ + ∥ f ∥ R .

Original: Taking the tightest bound we obtain 
Revised: Taking the tightest bound we obtain |

Original: [Equation]
Revised: [Equation]

Original:  Overall, combining (113), (114), and (117) we obtain 
Revised: f [Equation] [Equation] Overall, combining (115), (116), and (119) we obtain min

Original: [Equation]
Revised: [Equation]

Revised: ≤ 1 + 2 ∥ f ∥

Revised: [Equation]

Revised: [Equation]

Original: L P ROOF OF P ROPOSITION
Revised: L P ROOF OF P ROPOSITION

Original: Let f ∈ W d
Revised: Let f ∈ W d

Original: +1 , 1 w ( R d ) .
Revised: +1 , 1 w ( R d ) .

Original: First, we show that this implies both ∥ f ∥ R and ∥
Revised: First, we show that this implies both ∥ f ∥ R and ∥

Original: f ∥ R , ˆ g are finite.
Revised: f ∥ R , ˆ g are finite.

Original: Since we assume d is odd ( − ∆) ( d +1) /
Revised: Since we assume d is odd ( − ∆) ( d +1) /

Original: 2 f is an integral power of the negative Laplacian applied to f , hence can be expanded as a linear combination of d + 1 order partial derivatives, and so
Revised: 2 f is an integral power of the negative Laplacian applied to f , hence can be expanded as a linear combination of d + 1 order partial derivatives, and so

Original: [Equation]
Revised: [Equation]

Original: where a d is a constant depending on d but independent of f .
Revised: where a d is a constant depending on d but independent of f .

Original: Therefore, ∥ ( − ∆) ( d +1) / 2 f ∥ 1 ,w is finite.
Revised: Therefore, ∥ ( − ∆) ( d +1) / 2 f ∥ 1 ,w is finite.

Original: In particular, this shows ( − ∆)
Revised: In particular, this shows ( − ∆)

Original: ( d +1) / 2 ∈
Revised: ( d +1) / 2 ∈

Original: L 1 ( R d ) , and so R ( − ∆) ( d +1) / 2 f exists in a classical sense.
Revised: L 1 ( R d ) , and so R ( − ∆) ( d +1) / 2 f exists in a classical sense.

Original: This implies have the formulas ∥ f ∥ R = γ d ∥R ( − ∆) ( d +1) /
Revised: This implies have the formulas ∥ f ∥ R = γ d ∥R ( − ∆) ( d +1) /

Original: 2 f ∥ 1 and ∥
Revised: 2 f ∥ 1 and ∥

Original: f ∥ R , ˆ g = γ d ∥ ˆ g · [Equation]
Revised: f ∥ R , ˆ g = γ d ∥ ˆ g · R ( − ∆) ( d +1) /

Original: ( − ∆) ( d +1) / 2 f ∥ 1 where γ d = 1 2(2 π ) d − 1 (see (Ongie et al., 2020, Prop. 1)).
Revised: 2 f ∥ 1 where γ d = 1 2(2 π ) d − 1 (see (Ongie et al., 2020, Prop. 1)).

Original: Recall that w ( x ) =
Revised: Recall that w ( x ) =

Original: R ∗ [1 + | b | ]( x ) =
Revised: R ∗ [1 + | b | ]( x ) =

Original: c d + ζ d ∥ x ∥ , with c d = (cid:82) S
Revised: c d + ζ d ∥ x ∥ , with c d = (cid:82) S

Original: d − 1 d v and ζ d = (cid:82) S
Revised: d − 1 d v and ζ d = (cid:82) S

Original: d − 1 | v 1 | d v .
Revised: d − 1 | v 1 | d v .

Original: Therefore we have that
Revised: Therefore we have that

Original: [Equation]
Revised: [Equation]

Original: where C ˆ g = inf ( v ,b
Revised: where C ˆ g = inf ( v ,b

Original: ) ∈ S
Revised: ) ∈ S

Original: d − 1 × R 1+ | b | 1+ˆ g ( v ,b ) is finite and non-zero because for all v ∈ S d − 1 we have ˆ g ( v , b )
Revised: d − 1 × R 1+ | b | 1+ˆ g ( v ,b ) is finite and non-zero because for all v ∈ S d − 1 we have ˆ g ( v , b )

Original: =
Revised: =

Original: O ( | b | ) as | b | → ∞ where the implied constant is independent of v .
Revised: O ( | b | ) as | b | → ∞ where the implied constant is independent of v .

Original: Therefore, we have shown ∥ f ∥ R and ∥ f
Revised: Therefore, we have shown ∥ f ∥ R and ∥ f

Original: ∥ R , ˆ g are finite as claimed.
Revised: ∥ R , ˆ g are finite as claimed.

Original: Let α = ( R ∗ ) − 1 ∆ f = − γ d R ( − ∆)
Revised: Let α = ( R ∗ ) − 1 ∆ f = − γ d R ( − ∆)

Original: ( d +1) / 2 f .
Revised: ( d +1) / 2 f .

Original: Then ∥ f ∥ R = ∥
Revised: Then ∥ f ∥ R = ∥

Original: α ∥ 1 is finite, and so α is an Lfunction, which can be identified with a finite signed measure.
Revised: α ∥ 1 is finite, and so α is an Lfunction, which can be identified with a finite signed measure.

Original: Since ∥ f ∥ R ,g =
Revised: Since ∥ f ∥ R ,g =

Original: ∥
Revised: ∥

Original: ˆ g · α ∥ 1 is also finite, we see that ˜ α
Revised: ˆ g · α ∥ 1 is also finite, we see that ˜ α

Original: ( v , b ) :
Revised: ( v , b ) :

Original: = (1 + ˆ g ( v , b )))
Revised: = (1 + ˆ g ( v , b )))

Original: α ( v , b ) is also an L 1 function which can be identified with a finite signed measure.
Revised: α ( v , b ) is also an L 1 function which can be identified with a finite signed measure.

Original: By (Malliavin et al., 1995, Thm. 6.9), this implies there exists a sequence of finite atomic measures { ˜ α k }  converging narrowly 13 to ˜ α with ∥ ˜
Revised: By (Malliavin et al., 1995, Thm. 6.9), this implies there exists a sequence of finite atomic measures { ˜ α k } , such that each ˜ α k consists of a sum of at most k Diracs, converging narrowly 13 to ˜ α with ∥ ˜

Original: α k ∥ TV ≤ ∥ α ∥ 1 .
Revised: α k ∥ TV ≤ ∥ α ∥ 1 .

Original: Define α k
Revised: Define α k

Original: ( v , b ) =
Revised: ( v , b ) =

Original: ˜
Revised: ˜

Original: α k ( v , b ) / (1+ ˆ g ( v , b )) , which is also an atomic measure.
Revised: α k ( v , b ) / (1 + ˆ g ( v , b )) , which is also an atomic measure.

Original: Then it is easy to show α k
Revised: Then it is easy to show α k

Original: → α narrowly, as well.
Revised: → α narrowly, as well.

Original: By Lemma 5 of (Ongie et al., 2020), this implies there exists a sequence of single hidden-layer ReLU networks f k  converging to f pointwise.
Revised: By Lemma 5 of (Ongie et al., 2020), this implies there exists a sequence of single hidden-layer ReLU networks f k ∈ F k converging to f pointwise.

Original: Therefore, for all k we have
Revised: Therefore, for all k we have

Original: [Equation]
Revised: [Equation]

Original: Combining this inequality with the bound on ∥ f ∥ R + ∥ f
Revised: Combining this inequality with the bound on ∥ f ∥ R + ∥ f

Original: ∥ R , ˆ g given above, we see that
Revised: ∥ R , ˆ g given above, we see that

Original: [Equation]
Revised: [Equation]

Original: where c d, ˆ g = a d γ d C − 1 ˆ g is a constant defined independently of f .
Revised: where c d, ˆ g = a d γ d C − 1 ˆ g is a constant defined independently of f .

Original: Finally, if K is any compact subset, the pointwise convergence of f k to f on K can be upgraded to
Revised: Finally, if K is any compact subset, the pointwise convergence of f k to f on K can be upgraded to

Original: L 1 -convergence using Lebesgue’s dominated convergence theorem: by the bounds on the Lipschitz constant of a function given in terms of the R -norm in Proposition 8 of (Ongie et al., 2020), we have | f k ( x ) | ≤ ∥ x ∥ ( C + ∥ f k ∥ R ) ≤ B ∥ x ∥ for some constants C, B ≥ 0 , and since x
Revised: L 1 -convergence using Lebesgue’s dominated convergence theorem: by the bounds on the Lipschitz constant of a function given in terms of the R -norm in Proposition 8 of (Ongie et al., 2020), we have | f k ( x ) | ≤ ∥ x ∥ ( C + ∥ f k ∥ R ) ≤ B ∥ x ∥ for some constants C, B ≥ 0 , and since x

Original: (cid:55)→ B ∥ x ∥ is L 1 -integrable over any compact subset, the hypotheses of Lesbesgue’s dominated convergence theorem hold.
Revised: (cid:55)→ B ∥ x ∥ is L 1 -integrable over any compact subset, the hypotheses of Lesbesgue’s dominated convergence theorem hold.

Revised: M S TABILITY NORM OF “ PYRAMID ” FUNCTION

Revised: Here, to provide a better understanding of the depth separation result in Proposition 2, we show by direct calculation that the “pyramid” function in d = 2 dimensions, given by

Revised: [Equation]

Revised: [Equation]

Revised: fails to have finite stability norm.

Revised: In particular, we explicitly compute ( R ∗ ) − 1 ∆ p as a tempered distribution and show it is not a finite measure ( i.e. , must be a distribution of order > 0 ), which implies it cannot have finite stability norm under the assumptions in Proposition 2.

Revised: First, observe that ∆ p is linear combination of Diracs supported on finite line segments ℓ k defining the “edges” of the pyramid:

Revised: [Equation]

Revised: This means that if ϕ is any Schwartz class test function, then

Revised: [Equation]

Revised: [Equation]

Revised: Note that ∆ p ( x ) is a finite measure ( i.e. , a distribution of order zero), since

Revised: [Equation]

Revised: where | ℓ k | is the length of the line segment ℓ k .

Revised: See Fig.

Revised: 6 below for illustration.

Revised: (a) Pyramid function p - -1.

Revised: [Figure]

Revised: [Figure]

Revised: Now we compute R ∆ p , the Radon transform of ∆ p , which exists as a tempered distribution.

Revised: First, we show how to compute R δ ℓ where ℓ denotes a general line segment.

Revised: Let φ is any Schwartz class test function defined in Radon domain, then

Revised: [Equation]

Revised: where the exchange of integrals is justified by Fubini’s theorem since δ ℓ is a finite measure.

Revised: Suppose ℓ is a vertical line segment ℓ

Revised: = { (0 , t ) : t ∈ [ c, d ] } .

Revised: Assuming v is such that v 2 ̸ = 0 , then the inner integral above with ℓ in place of ℓ k above simplifies as

Revised: [Equation]

Original: {+In the event that v 2 = 
+}
Revised: {+In the event that v 2 = 
+}

Revised: we have

Revised: [Equation]

Revised: Therefore, we have shown

Revised: [Equation]

Revised: Now, consider one of line segments ℓ k coinciding with the edges of the pyramid.

Revised: This can be parameterized as ℓ k = { b k v k + t v ⊥ k : t ∈ [ c k , d k ] } where v k ∈ S 1 is a unit vector, b k ∈ R is a constant, and v ⊥ k is orthogonal to v k .

Revised: Let θ k is the angle such that v k

Revised: =

Revised: [cos( θ k ) , sin( θ k )] , then ℓ k is a rotation of the vertical line segment ℓ through the angle θ k , and translation by b k v k .

Revised: Therefore, by properties of Radon transforms,

Revised: [Equation]

Revised: where we set v ( θ )

Revised: =

Revised: [cos( θ ) , sin( θ )] for all θ ∈ [0 , π ) .

Revised: More concretely, we can express every slice

Revised: R δ ℓ k ( v , · ) as either a weighted indicator function when v ̸ = ± v k , which is non-zero when b is such that the line L v ,b :=

Revised: { x : v ⊤ x = b } intersects the line segment ℓ k , or as a weighted Dirac when v = ± v k , i.e. ,

Revised: [Equation]

Revised: For a fixed v ( θ ) ∈ S d − 1 the set of b ∈ R for which L v ( θ ) ,b ∩ ℓ 0 ̸ = ∅ is always a closed interval [ α θ , β θ ] .

Revised: Therefore, for θ ̸ = θ k we can write R δ ℓ k

Revised: ( θ, · ) = | sin( θ − θ k ) | − 1 1 [ α k ( θ ) ,β k ( θ )] for some α k ( θ ) and β k ( θ ) that vary continuously with θ .

Revised: Finally, by linearity, we obtain R ∆ p

Revised: = (cid:80) k

Revised: c k R δ ℓ k .

Revised: See Figure 7 for an approximate plot of R ∆ p .

Revised: Now we compute ( R ∗ ) − 1 ∆ p .

Revised: Recall that ( R ∗ ) − 1 = KR where K = H

Revised: ∂

Revised: b is a filtering step with

Revised: H being the Hilbert transform applied separably in the b -variable (Helgason, 1999).

Revised: For a smooth function g ,

Revised: [Equation]

Revised: (135) where p.v. indicates a principle value integral.

Revised: Therefore, for any θ ̸ = θ k , we have

Revised: [Equation]

Revised: and for θ = θ k we have

Revised: [Equation]

Revised: 3 angle (in radians) - -1.

Revised: o ff s e t b - - - - -       (a) Approximation of R ∆ p 0.

Revised: 3 angle (in radians) - -1.

Revised: o ff

Revised: [Figure]

Revised: e t

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Finally, by linearity of the operator KR , we have

Revised: [Equation]

Revised: Thus, [(

Revised: [Equation]

Revised: [Equation]

Revised: See Figure 7 for an approximate plot of ( R ∗ ) − 1 ∆ p = KR ∆ p .

Revised: As evidenced by the plot, this density has singularities along a 1-D manifold S in Radon domain.

Revised: This set corresponds to all lines in the primal domain passing through the corners of the pyramid.

Revised: Finally, we show that α :=

Revised: ( R ∗ ) − 1 ∆ p is not a finite measure ( i.e. , it is not an order zero distribution).

Revised: Intuitively, this is because the “density” α ( v , b ) is not absolutely integrable, since every 1-D angular slice has singularities like 1 / | b | .

Revised: Below we prove this more formally.

Revised: To prove α cannot be an order zero distribution, we construct a family of uniformly bounded test functions { φ ϵ } ϵ> 0 such that |⟨ α, φ ϵ ⟩| ≥ ρ ( ϵ ) ∥ φ ϵ ∥ ∞ , where ρ ( ϵ ) is a function such that ρ ( ϵ ) → + ∞ as ϵ → 0 + .

Revised: Let γ > 0 be a small fixed constant less than one.

Revised: For every 0 < ϵ < γ , consider the “rainbow-shaped” subset of Radon domain Ω ϵ defined by the inequalities − γ/ 2 < θ < γ/ 2 and cos( θ ) − ϵ < b < cos( θ ) where .

Revised: In primal domain, the set corresponds to a collection of lines that nearly intersect the corner point (1 , 0) .

Revised: Only three terms in the sum making up ( R ∗ ) − 1 ∆ p in (139) are dominant in the region Ω ϵ , corresponding to the three line segments in the support of ∆ p that arise from the right-most corner of the pyramid.

Revised: Elementary calculations show these three terms are specified by the parameters:

Revised: [Equation]

Revised: Therefore, α is well-approximated on Ω ϵ by

Revised: [Equation]

Revised: where we omit the terms p.v.

Revised: − 1 b − α k ( θ ) and p.v. − 1 ( b − b k ) 2 , since points in Ω ϵ are far from their singularity set.

Revised: In particular, we can show α − ˜

Revised: α is an order zero distribution when restricted to Ω ϵ ( i.e. , all other terms are locally smooth and bounded).

Revised: Let g ( θ ) be the function of θ in front of the principle value in ˜ α .

Revised: Note that g ( θ ) > B > 0 for all θ ∈ Ω ϵ where the constant B is independent of ϵ .

Revised: Let φ ϵ ( θ, b ) be a smooth function supported in Ω ϵ such that 0 ≤ φ ϵ ( θ, b ) ≤ 1 and φ ϵ ( θ, b )

Revised: = 1 on the region defined by the inequalities − γ/ 2 < θ < γ/ 2 and cos( θ ) − ϵ ≤ b ≤ cos( θ ) − ϵ 2 .

Revised: Then for any fixed θ ∈ ( − γ/ 2 , γ/ 2) , the integral p.v. (cid:82) φ ϵ ( θ, · ) cos( θ ) − b d b is bounded below by (cid:82) ϵ ϵ 2 1 b db = log( ϵ − 1 ) .

Revised: Therefore, we have

Revised: [Equation]

Revised: Since ∥ φ ϵ ∥ ∞ = 1 and γB log( ϵ − 1 ) − C

Revised: → + ∞ as ϵ → 0 + , this shows α cannot be a distribution of order zero, i.e. , it cannot be identified with a finite measure.

Revised: N A DDITIONAL EXPERIMENTS

Revised: The experiments in Sec.

Revised: 6 are designed to demonstrate Theorem 1 in a diverse range of step sizes ( [10 − 4 , 0 . 1] ).

Revised: Since flat minima of the loss landscape are concentrated near the origin in parameter space, and training with small step size near flat minima is inefficient, we used large initialization (about 10 times larger than standard methods).

Revised: Here we repeat the MNIST experiment using various initialization scales on a higher range of step sizes ( [10 − 3 , 0 . 2] ).

Revised: Figure 8 presents the sharpness curves for the different scales.

Revised: For large initialization, × 10 and × 15 , we get the same behavior as depicted in Sec. 6.

Revised: For small initialization, × 1 and × 5 , the sharpness of the obtained solutions is fixed for small learning rates up to a critical step size η ∗ .

Revised: At this threshold, the sharpness equals 2 /η ∗ , and any increment in the step size makes the minimum unstable.

Revised: This pushes SGD to flatter minima for larger step sizes, ones that satisfy the stability criterion.

Revised: Here it is important to note that for standard initialization, shown in Fig. 8(a), the threshold is well before the standard step size of η = 0 .

Revised: Namely, this phenomenon happens using standard initialization and standard learning rate.

Revised: [Figure]