Original: A LAW OF ROBUSTNESS FOR TWO - LAYERS NEURAL NETWORKS
Revised: A LAW OF ROBUSTNESS FOR TWO - LAYERS NEURAL NETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We initiate the study of the inherent tradeoffs between the size of a neural network and its robustness, as measured by its Lipschitz constant.
Revised: We initiate the study of the inherent tradeoffs between the size of a neural network and its robustness, as measured by its Lipschitz constant.

Original: We make a precise conjecture that, for any Lipschitz activation function and for most datasets, any two-layers neural network with k neurons that perfectly ﬁt the data must have its
Revised: We make a precise conjecture that, for any Lipschitz activation function and for most datasets, any two-layers neural network with k neurons that perfectly ﬁt the data must have its

Original: Lipschitz constant larger (up to a constant) than (cid:112) n/k where n is the number of datapoints.
Revised: Lipschitz constant larger (up to a constant) than (cid:112) n/k where n is the number of datapoints.

Original: In particular, this conjecture implies that overparametrization is necessary for robustness, since it means that one needs roughly one neuron per datapoint to ensure a O (1) -Lipschitz network, while mere data ﬁtting of d -dimensional data requires only one neuron per d datapoints.
Revised: In particular, this conjecture implies that overparametrization is necessary for robustness, since it means that one needs roughly one neuron per datapoint to ensure a O (1) -Lipschitz network, while mere data ﬁtting of d -dimensional data requires only one neuron per d datapoints.

Original: We prove a weaker version of this conjecture when the Lipschitz constant is replaced by an upper bound on it based on the spectral norm of the weight matrix.
Revised: We prove a weaker version of this conjecture when the Lipschitz constant is replaced by an upper bound on it based on the spectral norm of the weight matrix.

Original: We also prove the conjecturefor the ReLU activation function in the high-dimensional regime n ≈ d , and fora polynomial activation function of degree p when n ≈ d p .
Revised: We also prove the conjecture in the highdimensional regime n ≈ d (which we also refer to as the undercomplete case, since only k ≤ d is relevant here). Finally we prove the conjecture for polynomial activation functions of degree p when n ≈ d p .

Original: We complement these ﬁndings with experimental evidence supporting the conjecture.
Revised: We complement these ﬁndings with experimental evidence supporting the conjecture.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We study two-layered neural networks with inputs in R d , k neurons, and Lipschitz non-linearity ψ : R → R .
Revised: We study two-layers neural networks with inputs in R d , k neurons, and Lipschitz non-linearity ψ : R → R .

Original: These are functions of the form:
Revised: These are functions of the form:

Original: [Equation]
Revised: [Equation]

Original: ) with a (cid:96) , b (cid:96) ∈ R and w (cid:96) ∈ R d for any (cid:96) ∈ [ k ] .
Revised: (1) with a (cid:96) , b (cid:96) ∈ R and w (cid:96) ∈ R d for any (cid:96) ∈ [ k ] .

Original: We denote by F k ( ψ ) the set of functions of the form (1).
Revised: We denote by F k ( ψ ) the set of functions of the form (1).

Original: When k is large enough and ψ is non-polynomial, this set of functions can be used to ﬁt any given data set (Cybenko, 1989; Leshno et al., 1993).
Revised: When k is large enough and ψ is non-polynomial, this set of functions can be used to ﬁt any given data set (Cybenko, 1989; Leshno et al., 1993).

Original: That is, given a data set ( x i , y i ) i ∈
Revised: That is, given a data set ( x i , y i ) i ∈

Original: [ n ] ∈ ( R d × R ) n , one can ﬁnd f ∈ F k ( ψ ) such that
Revised: [ n ] ∈ ( R d × R ) n , one can ﬁnd f ∈ F k ( ψ ) such that

Original: [Equation]
Revised: [Equation]

Original: In a variety of scenarios one is furthermore interested in ﬁtting the data smoothly .
Revised: In a variety of scenarios one is furthermore interested in ﬁtting the data smoothly .

Original: For example, in machine learning, the data ﬁtting model f is used to make prediction at unseen points x (cid:54)∈ { x 1 , . . .
Revised: For example, in machine learning, the data ﬁtting model f is used to make predictions at unseen points x (cid:54)∈ { x 1 , . . .

Original: , x n } .
Revised: , x n } .

Original: It is reasonable to ask for these predictions to be stable, that is a small perturbation of x should result in a small perturbation of f ( x ) .
Revised: It is reasonable to ask for these predictions to be stable, that is a small perturbation of x should result in a small perturbation of f ( x ) .

Original: A natural question is: how “costly” is this stability restriction compared to mere data ﬁtting?
Revised: A natural question is: how “costly” is this stability restriction compared to mere data ﬁtting?

Original: In practice it seems much harder to ﬁnd robust models for large scale problems, as ﬁrst evidenced in the seminal paper (Szegedy et al., 2013).
Revised: In practice it seems much harder to ﬁnd robust models for large scale problems, as ﬁrst evidenced in the seminal paper (Szegedy et al., 2013).

Original: In theory the “cost” of ﬁnding robust models has been investigated from a computational complexity perspective in (Bubeck et al., 2019), from a statistical perspective in (Schmidt et al., 2018), and more generally from a model complexity perspective in (Degwekar et al., 2019; Raghunathan et al., 2019; Allen-Zhu and Li, 2020).
Revised: In theory the “cost” of ﬁnding robust models has been investigated from a computational complexity perspective in (Bubeck et al., 2019), from a statistical perspective in (Schmidt et al., 2018), and more generally from a model complexity perspective in (Degwekar et al., 2019; Raghunathan et al., 2019; Allen-Zhu and Li, 2020).

Original: We propose here a different angle of study within the broad model complexity perspective: does a model have to be larger for it to be robust?
Revised: We propose here a different angle of study within the broad model complexity perspective: does a model have to be larger for it to be robust?

Original: Empirical evidence (e.g., (Goodfellow et al., 2015; Madry et al., 2018)) suggests that bigger models (also known as “overparametrization”) does indeed help for robustness.
Revised: Empirical evidence (e.g., (Goodfellow et al., 2015; Madry et al., 2018)) suggests that bigger models (also known as “overparametrization”) do indeed help for robustness.

Original: Our main contribution is a conjecture (Conjecture 1 and Conjecture 2) on the precise tradeoffs between size of the model (i.e., the number of neurons k ) and robustness (i.e., the Lipschitz constant
Revised: Our main contribution is a conjecture (Conjecture 1 and Conjecture 2) on the precise tradeoffs between size of the model (i.e., the number of neurons k ) and robustness (i.e., the Lipschitz constant

Original: 60 m a x
Revised: m a x

Original: i m u m r ando m
Revised: i m u m r ando m

Original: g r ad i en t of the data ﬁtting model f ∈ F k ( ψ )) for generic data sets.
Revised: g r ad i en t of the data ﬁtting model f ∈ F k ( ψ )) for generic data sets.

Original: We say that a data set ( x i , y i ) i ∈ [ n ] is generic if it is i.i.d.
Revised: We say that a data set ( x i , y i ) i ∈ [ n ] is generic if it is i.i.d.

Original: with x i uniform (or approximately so, see below) on the sphere S d − 1 = { x ∈ R d : (cid:107) x (cid:107) = 1 } and y i uniform on {− 1 , +1 } .
Revised: with x i uniform (or approximately so, see below) on the sphere S d − 1 = { x ∈ R d : (cid:107) x (cid:107) = 1 } and y i uniform on {− 1 , +1 } .

Original: We give the precise conjecture in Section 2.
Revised: We give the precise conjecture in Section 2.

Original: We prove several weaker versions of Conjecture 1 and Conjecture 2 respectively in Section 4 and Section 3.
Revised: We prove several weaker versions of Conjecture 1 and Conjecture 2 respectively in Section 4 and Section 3.

Original: We also give empirical evidence for the conjecture in Section 5.
Revised: We also give empirical evidence for the conjecture in Section 5.

Original: A corollary of our conjecture.
Revised: A corollary of our conjecture.

Original: A key fact about generic data, established in Baum (1988);  Bubeck et al.
Revised: A key fact about generic data, established in Baum (1988); Yun et al. ; Bubeck et al.

Original: (2020), is that one can memorize arbitrary labels with k ≈ n/d , that is merely one neuron per d datapoints.
Revised: (2020), is that one can memorize arbitrary labels with k ≈ n/d , that is merely one neuron per d datapoints.

Original: Our conjecture implies that for such optimal-size neural networks it is impossible to be robust, in the sense that the Lipschitz constant must be of order √ d .
Revised: Our conjecture implies that for such optimal-size neural networks it is impossible to be robust, in the sense that the Lipschitz constant must be of order √ d .

Original: The conjecture also states that to be robust (i.e. attain Lipschitz constant O (1) ) one must necessarily have k ≈ n , that is roughly each datapoint must have its own neuron.
Revised: The conjecture also states that to be robust (i.e. attain Lipschitz constant O (1) ) one must necessarily have k ≈ n , that is roughly each datapoint must have its own neuron.

Original: Therefore, we obtain a trade off between size and robustness.
Revised: Therefore, we obtain a trade off between size and robustness, namely to make the network robust it needs to be d times larger than for mere data ﬁtting.

Original: We illustrate these two cases in Figure 1.
Revised: We illustrate these two cases in Figure 1.

Original: We train a neural network to ﬁt generic data, and plot the maximum gradient over several randomly drawn points (a proxy for the Lipschitz constant) for various values of √ d , when either k = n (blue dots) or k = 10 nd (red dots).
Revised: We train a neural network to ﬁt generic data, and plot the maximum gradient over several randomly drawn points (a proxy for the Lipschitz constant) for various values of √ d , when either k = n (blue dots) or k = 10 nd (red dots).

Original: As predicted, for the large neural network ( k = n ) the Lipschitz constant remains roughly constant, while for the optimally-sized one ( k = 10 nd ) the Lipschitz constant increases roughly linearly in √ d .
Revised: As predicted, for the large neural network ( k = n ) the Lipschitz constant remains roughly constant, while for the optimally-sized one ( k = 10 nd ) the Lipschitz constant increases roughly linearly in √ d .

Original: Notation.
Revised: Notation.

Original: For Ω ⊂ R d we deﬁne Lip Ω ( 
Revised: For Ω ⊂ R d we deﬁne Lip Ω ( f ) =

Original: [Equation] ) = sup 
Revised: sup x (cid:54) =

Original: [Equation]
Revised: x (cid:48) ∈ Ω | f ( x ) − f ( x (cid:48) ) | (cid:107) x − x (cid:48) (cid:107) (if Ω = R

Original: (if Ω = R d we omit the subscript and write Lip( f ) ), where (cid:107) · (cid:107) denotes the Euclidean.
Revised: d we omit the subscript and write Lip( f ) ), where (cid:107) · (cid:107) denotes the Euclidean norm.

Original: For matrices we use (cid:107) · (cid:107) op , (cid:107) · (cid:107) op , ∗ , (cid:107) · (cid:107) F and (cid:104)· , ·(cid:105) for respectively the operator norm, the nuclear norm (sum of singular values), the Frobenius norm, and the Frobenius inner product.
Revised: For matrices we use (cid:107) · (cid:107) op , (cid:107) · (cid:107) op , ∗ , (cid:107) · (cid:107) F and (cid:104)· , ·(cid:105) for respectively the operator norm, the nuclear norm (sum of singular values), the Frobenius norm, and the Frobenius inner product.

Original: We also use these notations for tensors of higher order, see Appendix A for more details on tensors.
Revised: We also use these notations for tensors of higher order, see Appendix A for more details on tensors.

Original: We denote c > 0 and C > 0 for universal numerical constants, respectively small enough and large enough, whose value can change in different occurences.
Revised: We denote c > 0 and C >for universal numerical constants, respectively small enough and large enough, whose values can change in different occurences.

Original: Similarly, by c p > 0 and C p > 0 we denote constants depending only on the parameter p .
Revised: Similarly, by c p > 0 and C p > 0 we denote constants depending only on the parameter p .

Original: We also write ReLU( t ) = max( t, 0) for the rectiﬁed linear unit.
Revised: We also write ReLU( t ) = max( t, 0) for the rectiﬁed linear unit.

Original: Generic data.
Revised: Generic data.

Original: We give some ﬂexibility in our deﬁnition of “generic data” in order to focus on the essence of the problem, rather than technical details.
Revised: We give some ﬂexibility in our deﬁnition of “generic data” in order to focus on the essence of the problem, rather than technical details.

Original: Namely, in addition to the spherical model mentioned above, where x i is i.i.d.
Revised: Namely, in addition to the spherical model mentioned above, where x i is i.i.d.

Original: uniform on the sphere S d − 1 = { x ∈ R
Revised: uniform on the sphere S d − 1 = { x ∈ R

Original: d : (cid:107) x (cid:107) = 1 } , we also consider the very closely related model where x i is iid from a centered Gaussian with covarianced I d (in particular E [ (cid:107) x i (cid:107) 2 ] = 1 , and in fact (cid:107) x
Revised: d : (cid:107) x (cid:107) = 1 } , we also consider the very closely related model where x i is i.i.d. from a centered Gaussian with covarianced I d (in particular E [ (cid:107) x i (cid:107) 2 ] = 1 , and in fact (cid:107) x

Original: i (cid:107) is tightly concentrated around 1 ).
Revised: i (cid:107) is tightly concentrated around 1 ).

Original: In both cases we consider y i to be i.i.d.
Revised: In both cases we consider y i to be i.i.d.

Original: random signs.
Revised: random signs.

Original: We say that a property holds with high probability for generic data , if it holds with high probability either for the spherical model or for the Gaussian model.
Revised: We say that a property holds with high probability for generic data , if it holds with high probability either for the spherical model or for the Gaussian model.

Original:  A CONJECTURED LAW OF ROBUSTNESS
Revised:  A CONJECTURED LAW OF ROBUSTNESS

Original: Our main contribution is the following conjecture, which asserts that, on generic data sets, increasing the size of a network is necessary to obtain robustness:
Revised: Our main contribution is the following conjecture, which asserts that, on generic data sets, increasing the size of a network is necessary to obtain robustness:

Original: Conjecture 1 For generic data sets, with high probability 1 , any f ∈ F k ( ψ ) ﬁtting the data  (i.e., satisfying (2) ) must also satisfy:
Revised: Conjecture 1 For generic data sets, with high probability 1 , any f ∈ F k ( ψ ) ﬁtting the data 2 (i.e., satisfying (2) ) must also satisfy:

Original: [Equation]
Revised: [Equation]

Original: Note that for generic data, with high probability (for n = poly( d ) ), there exists a smooth interpolation.
Revised: Note that for generic data, with high probability (for n = poly( d ) ), there exists a smooth interpolation.

Original: Namely there exists g : R d → R with g ( x i ) =
Revised: Namely there exists g : R d → R with g ( x i ) =

Original: y i , ∀ i ∈ [ n ] and Lip( g ) =
Revised: y i , ∀ i ∈ [ n ] and Lip( g ) =

Original: O (1) .
Revised: O (1) .

Original: This follows easily from the fact that with high probability (for large d ) one has (cid:107) x i − x j (cid:107) ≥ 1 , ∀ i (cid:54) =
Revised: This follows easily from the fact that with high probability (for large d ) one has (cid:107) x i − x j (cid:107) ≥ 1 , ∀ i (cid:54) =

Original: j .
Revised: j .

Original: Conjecture 1 puts restrictions on how smoothly one can interpolate data with small neural networks.
Revised: Conjecture 1 puts restrictions on how smoothly one can interpolate data with small neural networks.

Original: A striking consequence of the conjecture is that for a two-layers neural network f ∈ F k ( ψ ) to be as robust as this function g (i.e., Lip( f )
Revised: A striking consequence of the conjecture is that for a two-layers neural network f ∈ F k ( ψ ) to be as robust as this function g (i.e., Lip( f )

Original: =
Revised: =

Original: O (1) )
Revised: O (1) )

Original: and ﬁt the data, one must have k = Ω( n ) , i.e., roughly one neuron per data point.
Revised: and ﬁt the data, one must have k = Ω( n ) , i.e., roughly one neuron per data point.

Original: On the other hand with that many neurons it is quite trivial to smoothly interpolate the data, as we explain in Section 3.3.
Revised: On the other hand with that many neurons it is quite trivial to smoothly interpolate the data, as we explain in Section 3.3.

Original: Thus the conjecture makes a strong statement that  the trivial smooth interpolation isessentially the best thing one can do.
Revised: Thus the conjecture makes a strong statement that essentially the trivial smooth interpolation is the best thing one can do.

Original: In addition to making the prediction that one neuron per datapoint is necessary for optimal smoothness, the conjecture also gives a precise prediction on the possible tradeoff between size of the network and its robustness.
Revised: In addition to making the prediction that one neuron per datapoint is necessary for optimal smoothness, the conjecture also gives a precise prediction on the possible tradeoff between size of the network and its robustness.

Original: We also conjecture that this whole range of tradeoffs is actually achievable:
Revised: We also conjecture that this whole range of tradeoffs is actually achievable:

Original: Conjecture 2 Let n, d, k be such that C · nd ≤ k ≤ C · n and n ≤ d C where C is an arbitrarily large constant in the latter occurence.
Revised: Conjecture 2 Let n, d, k be such that C · nd ≤ k ≤ C · n and n ≤ d C where C is an arbitrarily large constant in the latter occurence.

Original: There exists ψ such that, for generic data sets, with high probability, there exists f ∈ F k ( ψ ) ﬁtting the data (i.e., satisfying (2) ) and such that
Revised: There exists ψ such that, for generic data sets, with high probability, there exists f ∈ F k ( ψ ) ﬁtting the data (i.e., satisfying (2) ) and such that

Original: [Equation]
Revised: [Equation]

Original: The condition k ≤ C · n in Conjecture 2 is necessary, for any interpolation of the data must have
Revised: The condition k ≤ C · n in Conjecture 2 is necessary, for any interpolation of the data must have

Original: Lipschitz constant at least a constant.
Revised: Lipschitz constant at least a constant.

Original: The other condition on k , namely k ≥ C · nd , is also necessary, for that many neurons is needed to merely guarantee the existence of a data-ﬁtting neural network with k neurons (see Baum (1988);
Revised: The other condition on k , namely k ≥ C · nd , is also necessary, for that many neurons is needed to merely guarantee the existence of a data-ﬁtting neural network with k neurons (see Baum (1988) ; Yun et al.

Original:  Bubeck et al.
Revised: ; Bubeck et al.

Revised: Finally the condition n ≤ d C is merely used to avoid explicitly stating a logarithmic term in our conjecture (indeed, equivalently one can replace this condition by adding a multiplicative polylogarithmic term in d in the claimed inequality).

Original: Our results around Conjecture 2 (Section 3).
Revised: Our results around Conjecture 2 (Section 3).

Original: We prove Conjecture 2 for both the optimal smoothness regime (which is quite straightforward, see Section 3.3) and for the optimal size regime (here more work is needed, and we use a certain tensor-based construction, see Section 3.4).
Revised: We prove Conjecture 2 for both the optimal smoothness regime (which is quite straightforward, see Section 3.3) and for the optimal size regime (here more work is needed, and we use a certain tensor-based construction, see Section 3.4).

Original: In the latter case we only prove approximate data ﬁtting (mostly to simplify the proofs), and more importantly we need to assume that n is of order d p for some even integer p .
Revised: In the latter case we only prove approximate data ﬁtting (mostly to simplify the proofs), and more importantly we need to assume that n is of order d p for some even integer p .

Original: It would be interesting to generalize the proof to any n .
Revised: It would be interesting to generalize the proof to any n .

Original: While the conjecture remains open between these two extreme regimes, we do give a construction in Section 3.3 which has the correct qualitative behavior (namely increasing k improves the Lipschitz constant), albeit the scaling we obtain is n/k instead of (cid:112) n/k , see Theorem 1.
Revised: While the conjecture remains open between these two extreme regimes, we do give a construction in Section 3.3 which has the correct qualitative behavior (namely increasing k improves the Lipschitz constant), albeit the scaling we obtain is n/k instead of (cid:112) n/k , see Theorem 1.

Original: Our results around Conjecture 1 (Section 4).
Revised: Our results around Conjecture 1 (Section 4).

Original: We prove a weaker version of Conjecture 1 where the Lipschitz constant on the sphere is replaced by a proxy involving the spectral norm of the weight matrix, see Theorem 3.
Revised: We prove a weaker version of Conjecture 1 where the Lipschitz constant on the sphere is replaced by a proxy involving the spectral norm of the weight matrix, see Theorem 3.

Original: We also prove the conjecture in the optimal size regime, speciﬁcally when n = d p for an integer p and one uses a polynomial activation function of degree p , see Theorem 6.
Revised: We also prove the conjecture in the optimal size regime, speciﬁcally when n = d p for an integer p and one uses a polynomial activation function of degree p , see Theorem 6.

Original: For p = 1 (i.e., n ≈ d ) we also proveit for the ReLU activation (with no bias terms, i.e., b (cid:96) = 0 , ∀ (cid:96) ∈ [ k ] in (1)) see Theorem4.
Revised: For p = 1 (i.e., n ≈ d ) we in fact prove the conjecture for abritrary non-linearities, see Theorem

Original: Further open problems.
Revised: Further open problems.

Original: Our proposed law of robustness is a ﬁrst mathematical formalization of the broader phenomenon that “overparametrization in neural networks is necessary for robustness”.
Revised: Our proposed law of robustness is a ﬁrst mathematical formalization of the broader phenomenon that “overparametrization in neural networks is necessary for robustness”.

Original: Ideally one would like a much more reﬁned understanding of the phenomenon than the one given in Conjecture 1.
Revised: Ideally one would like a much more reﬁned understanding of the phenomenon than the one given in Conjecture 1.

Original: For example, one could imagine that in greater generality, the law would read
Revised: For example, one could imagine that in greater generality, the law would read

Original: Lip Ω ( f ) ≥ F ( k, ( x i , y i ) i ∈ [ n ] , Ω) .
Revised: Lip Ω ( f ) ≥ F ( k, ( x i , y i ) i ∈ [ n ] , Ω) .

Original: That is, we would like to understand how the achievable level of smoothness depends on the particular data set at hand, but also on the set where we expect to be making predictions.
Revised: That is, we would like to understand how the achievable level of smoothness depends on the particular data set at hand, but also on the set where we expect to be making predictions.

Original: Another direction to generalize the law would be to extend it to multilayers neural networks.
Revised: Another direction to generalize the law would be to extend it to multilayers neural networks.

Original: In particular one could imagine the most general law would replace the parameter k (number of neurons) by the type of architecture being used and in turn predict the best architecture for a given data set and prediction set.
Revised: In particular one could imagine the most general law would replace the parameter k (number of neurons) by the type of architecture being used and in turn predict the best architecture for a given data set and prediction set.

Original: Finally note that our proposed law apply to all neural networks, but it would also be interesting to understand how the law interacts with algorithmic considerations (for example in Section 5 we use Adam( Kingma and Ba (2014)) to ﬁnd a set of weights that qualitatively match Conjecture 2).
Revised: Finally note that our proposed law apply to all neural networks, but it would also be interesting to understand how the law interacts with algorithmic considerations (for example in Section 5 we use Adam Kingma and Ba (2015) to ﬁnd a set of weights that qualitatively match Conjecture 2).

Original:  S MOOTH INTERPOLATION
Revised:  S MOOTH INTERPOLATION

Original: We start with a warm-up in Section 3.1 where we discuss the simplest case of interpolation with a linear model ( k = 1 , n ≤ d ) and in Section 3.2 for the optimal smoothness regime ( k = n ).
Revised: We start with a warm-up in Section 3.1 where we discuss the simplest case of interpolation with a linear model ( k = 1 , n ≤ d ) and in Section 3.2 for the optimal smoothness regime ( k = n ).

Original: We generalize the construction of Section 3.2 in Section 3.3 to obtain the whole range of tradeoffs between k and Lip( f ) , albeit with a suboptimal scaling, see Theorem 1.
Revised: We generalize the construction of Section 3.2 in Section 3.3 to obtain the whole range of tradeoffs between k and Lip( f ) , albeit with a suboptimal scaling, see Theorem 1.

Original: We also generalize the linear model calculations of Section 3.1 in Section 3.4 to obtain the optimal size regime for larger values of n via a certain tensor construction.
Revised: We also generalize the linear model calculations of Section 3.1 in Section 3.4 to obtain the optimal size regime for larger values of n via a certain tensor construction.

Original: 3. T HE SIMPLEST CASE : OPTIMAL SIZE REGIME WHEN n ≤ c · d
Revised: 3. T HE SIMPLEST CASE : OPTIMAL SIZE REGIME WHEN n ≤ c · d

Original: Let us consider k = 1 , n ≤ c · d and ψ ( t )
Revised: Let us consider k = 1 , n ≤ c · d and ψ ( t )

Original: =
Revised: =

Original: t .
Revised: t .

Original: Thus we are trying to ﬁnd w ∈ R d such that w · x i =
Revised: Thus we are trying to ﬁnd w ∈ R d such that w · x i =

Original: y i for all i ∈ [ n ] , or in other words
Revised: y i for all i ∈ [ n ] , or in other words

Original: Xw =
Revised: Xw =

Original: Y with X the n × d matrix whose i th row is x i , and Y =
Revised: Y with X the n × d matrix whose i th row is x i , and Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) .
Revised: , y n ) .

Original: The smoothest solution to this system (i.e., the one minimizing (cid:107) w (cid:107) ) is
Revised: The smoothest solution to this system (i.e., the one minimizing (cid:107) w (cid:107) ) is

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Using [Theorem 5.58, Vershynin (2012)] one has with probability at least 1 − exp( C − cd ) (and using that n ≤ c · d ) that
Revised: Using [Theorem 5.58, Vershynin (2012)] one has with probability at least 1 − exp( C − cd ) (and using that n ≤ c · d ) that

Original: [Equation]
Revised: [Equation]

Original:  and thus Lip( x (cid:55)→ w · x ) ≤ √ 2 · ·(cid:107) Y (cid:107) = √ 2 n .
Revised: , and thus Lip( x (cid:55)→ w · x ) ≤ √ 2 · (cid:107) Y (cid:107) = √ 2 n .

Original: This concludes the proof sketch of Conjecture 2 for the simplest case k = 1 and n ≤ d .
Revised: This concludes the proof sketch of Conjecture 2 for the simplest case k = 1 and n ≤ d .

Original:  A NOTHER SIMPLE CASE : OPTIMAL SMOOTHNESS REGIME
Revised: 3. A NOTHER SIMPLE CASE : OPTIMAL SMOOTHNESS REGIME

Original: Next we consider the optimal smoothness regime in Conjecture 2, namely k = n .
Revised: Next we consider the optimal smoothness regime in Conjecture 2, namely k = n .

Original: First note that, for generic data and n = poly( d ) , with high probability the caps C i :=
Revised: First note that, for generic data and n = poly( d ) , with high probability the caps C i :=

Original: (cid:8) x ∈ S d − 1 : x i · x ≥ 0 .
Revised: (cid:8) x ∈ S d − 1 : x i · x ≥ 0 .

Original: 9 (cid:9) are disjoint sets and moreover they each contain a single data point (namely x i ).
Revised: 9 (cid:9) are disjoint sets and moreover they each contain a single data point (namely x i ).

Original: With a single ReLU unit it is then easy to make a smooth function ( 10 -Lipschitz) which is 0 outside of C i and equal to +1 at x i (in other word the neuron activates for a single data point), namely x (cid:55)→ 10 · ReLU ( x i · x − 0 . 9) .
Revised: With a single ReLU unit it is then easy to make a smooth function ( 10 -Lipschitz) which is 0 outside of C i and equal to +1 at x i (in other words the neuron activates for a single data point), namely x (cid:55)→ 10 · ReLU ( x i · x − 0 . 9) .

Original: Thus one can ﬁt the entire data set with the following ReLU network which is 10 -Lipschitz on the sphere:
Revised: Thus one can ﬁt the entire data set with the following ReLU network which is 10 -Lipschitz on the sphere:

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof of Conjecture 2 for the optimal smoothness regime k = n .
Revised: This concludes the proof of Conjecture 2 for the optimal smoothness regime k = n .

Original: 3. I NTERMEDIATE REGIMES VIA R E LU NETWORKS
Revised: 3. I NTERMEDIATE REGIMES VIA R E LU NETWORKS

Original: We now combine the two constructions above (the linear model of Section 3.1 and the “isolation” strategy of Section 3.2) to give a construction that can trade off size for robustness (albeit not optimally according to Conjecture 2), see Appendix C for the proof.
Revised: We now combine the two constructions above (the linear model of Section 3.1 and the “isolation” strategy of Section 3.2) to give a construction that can trade off size for robustness (albeit not optimally according to Conjecture 2), see Appendix C for the proof.

Original: Theorem 1 Let n, d, k be such that C · n log( n ) d ≤ k ≤ C · n .
Revised: Theorem 1 Let n, d, k be such that C · n log( n ) d ≤ k ≤ C · n .

Original: For generic data sets, with probability at least 1 − 1 /n C , there exists f ∈ F k (ReLU) ﬁtting the data (i.e., satisfying (2) ) and such that
Revised: For generic data sets, with probability at least 1 − 1 /n C , there exists f ∈ F k (ReLU) ﬁtting the data (i.e., satisfying (2) ) and such that

Original: [Equation]
Revised: [Equation]

Original: f ) ≤ C · n log( d ) k .

Original: 3. O PTIMAL SIZE NETWORKS VIA TENSOR INTERPOLATION
Revised: 3. O PTIMAL SIZE NETWORKS VIA TENSOR INTERPOLATION

Original: In this section we essentially prove Conjecture 2 in the optimal size regime (namely k · d ≈ n ), with three caveats:
Revised: In this section we essentially prove Conjecture 2 in the optimal size regime (namely k · d ≈ n ), with three caveats:

Original: We allow a slack of a log n factor by considering k · d = Cn log( n ) instead of the optimal k · d = Cn as in Baum (1988);
Revised: We allow a slack of a log n factor by considering k · d = Cn log( n ) instead of the optimal k · d = Cn as in Baum (1988);

Original: Bubeck et al.
Revised: Bubeck et al.

Original: We only prove approximate ﬁt rather than exact ﬁt.
Revised: We only prove approximate ﬁt rather than exact ﬁt.

Original: It is likely that with more work one can use the core of our argument to obtain exact ﬁt.
Revised: It is likely that with more work one can use the core of our argument to obtain exact ﬁt.

Original: For that reason we did not make any attempt to optimize the dependency on ε in Theorem 2.
Revised: For that reason we did not make any attempt to optimize the dependency on ε in Theorem 2.

Original: For instance one could probably obtain log(1 /ε ) rather than 1 / poly( ε ) dependency by using an iterative scheme that ﬁts the residuals, as in (Bresler and Nagaraj, 2020; Bubeck et al., 2020).
Revised: For instance one could probably obtain log(1 /ε ) rather than 1 / poly( ε ) dependency by using an iterative scheme that ﬁts the residuals, as in (Bresler and Nagaraj, 2020; Bubeck et al., 2020).

Original: We assume that n is of order d p for some even integer p .
Revised: We assume that n is of order d p for some even integer p .

Original: While it might be that one can apply the same proof for odd integers, the whole construction crucially relies on p being an even integer as we essentially do a linear regression over the feature embedding x (cid:55)→ x ⊗ p .
Revised: While it might be that one can apply the same proof for odd integers, the whole construction crucially relies on p being an even integer as we essentially do a linear regression over the feature embedding x (cid:55)→ x ⊗ p .

Original: A possible approach to extend the proof to other values of n would be use the scheme of
Revised: A possible approach to extend the proof to other values of n would be use the scheme of

Original: Section 3.3 with the linear regression there replaced by the tensor regression used below.
Revised: Section 3.3 with the linear regression there replaced by the tensor regression used below.

Original: Theorem 2 Fix ε > 0 , p an even integer, and let ψ ( t ) =
Revised: Theorem 2 Fix ε > 0 , p an even integer, and let ψ ( t ) =

Original: t p .
Revised: t p .

Original: Let n, d, k be such that n log( n ) =
Revised: Let n, d, k be such that n log( n ) =

Original: ε 2 · d p and k
Revised: ε 2 · d p and k

Original: =
Revised: =

Original: C p · d p − 1 .
Revised: C p · d p − 1 .

Original: Then for generic data, with probability at least 1 − 1 /n C , there exists f ∈ F k ( ψ ) such that
Revised: Then for generic data, with probability at least 1 − 1 /n C , there exists f ∈ F k ( ψ ) such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We propose to approximately ﬁt with the following neural network:
Revised: We propose to approximately ﬁt with the following neural network:

Original: [Equation]
Revised: [Equation]

Original: Naively one might think that this neural network requires n neurons.
Revised: Naively one might think that this neural network requires n neurons.

Original: However, it turns out that one can always decompose a symmetric tensor of order p into k = 

Revised: However, it turns out that one can always decompose a symmetric tensor of order p into k =

Original: p d p − 1 rank- 1 symmetric tensors of order p , so that in fact f ∈ F k ( ψ ) .
Revised: p d p − 1 rank- 1 symmetric tensors of order p , so that in fact f ∈ F k ( ψ ) .

Original: For p = 2 this simply follows from eigendecomposition and for general p we give a simple proof in [Appendix A, Lemma 2].
Revised: For p = 2 this simply follows from eigendecomposition and for general p we give a simple proof in [Appendix A, Lemma 2].

Original: One also has by applying [Appendix B, Lemma 4] with τ = C p log( n ) and doing an union bound, that with probability at least 1 − 1 /n C , for any j ∈ [ n ] ,
Revised: One also has by applying [Appendix B, Lemma 4] with τ = C p log( n ) and doing an union bound, that with probability at least 1 − 1 /n C , for any j ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: In particular this proves (3).
Revised: In particular this proves (3).

Original: Thus it only remains to estimate the Lipschitz constant, which by [Appendix A, Lemma 1] is reduced to estimating the operator norm of the tensor (cid:80) ni
Revised: Thus it only remains to estimate the Lipschitz constant, which by [Appendix A, Lemma 1] is reduced to estimating the operator norm of the tensor (cid:80) ni

Original: =

Revised: =

Original: y i x ⊗ p i .
Revised: y i x ⊗ p i .

Original: We do so in [Appendix B, Lemma 5].
Revised: We do so in [Appendix B, Lemma 5].

Original: (cid:3)
Revised: (cid:3)

Original:  P ROVABLE WEAKER VERSIONS OF C ONJECTURE
Revised:  P ROVABLE WEAKER VERSIONS OF C ONJECTURE

Original: Conjecture 1 can be made weaker along several directions.
Revised: Conjecture 1 can be made weaker along several directions.

Original: For example the quantity of interest
Revised: For example the quantity of interest

Original: Lip S d − 1 ( f ) can be replaced by various upper bound proxies for the Lipschitz constant.
Revised: Lip S d − 1 ( f ) can be replaced by various upper bound proxies for the Lipschitz constant.

Original: A mild weakening would be to replace it by the Lipschitz constant on the whole space (we shall in fact only consider this notion here).
Revised: A mild weakening would be to replace it by the Lipschitz constant on the whole space (we shall in fact only consider this notion here).

Original: A much more severe weakening is to replace it by a quantity that depends on the spectral norm of the weight matrix (essentially ignoring the pattern of activation functions).
Revised: A much more severe weakening is to replace it by a quantity that depends on the spectral norm of the weight matrix (essentially ignoring the pattern of activation functions).

Original: For the latter proxy we actually give a complete proof, see Theorem 3, which in particular formally proves that “overparametrization is a law of robustness for generic data sets”.
Revised: For the latter proxy we actually give a complete proof, see Theorem 3, which in particular formally proves that “overparametrization is a law of robustness for generic data sets”.

Original: Other interesting directions to weaken the conjecture include specializing it to common activation functions, or simply having a smaller lower bound on the Lipschitz constant.
Revised: Other interesting directions to weaken the conjecture include specializing it to common activation functions, or simply having a smaller lower bound on the Lipschitz constant.

Original: In Section 4.2 weconsider the ReLU activation (with no bias terms) and prove Lip( f ) ≥ c (cid:113) dk , see Theorem 4.
Revised: In Section 4.2 we prove the conjecture when n is replaced by d in the lower bound.

Original: We say that this inequality is in the “very high-dimensional case”, in the sense that it matches the conjecture for n ≈ d .
Revised: We say that this inequality is in the “very high-dimensional case”, in the sense that it matches the conjecture for n ≈ d (alternatively we also refer to it as the “undercomplete case”, in the sense that only k ≤ d is relevant in this very high-dimensional scenario).

Original: In the moderately high-dimensional case ( n (cid:29) d ) the proof strategy we propose in Section 4.2 cannot work.
Revised: In the moderately high-dimensional case ( n (cid:29) d ) the proof strategy we propose in Section 4.2 cannot work.

Original: In Section 4.3 we give another argument for the latter case, speciﬁcally in the optimal size regime (i.e., k · d ≈ n ) and for a power activation function (again with no bias term), see Theorem 5.
Revised: In Section 4.3 we give another argument for the latter case, speciﬁcally in the optimal size regime (i.e., k · d ≈ n ) and for a power activation function, see Theorem 5.

Original: We generalize this to polynomial activation functions in Section D.1.
Revised: We generalize this to polynomial activation functions in Section D.1.

Original: In the speciﬁc case of a quadratic activation function we also show a lower bound that applies for any k and which is in fact larger than the one given in Conjecture 1, see Theorem 7 in Section D.2.
Revised: In the speciﬁc case of a quadratic activation function we also show a lower bound that applies for any k and which is in fact larger than the one given in Conjecture 1, see Theorem 7 in Section D.2.

Original: 4. S PECTRAL NORM PROXY FOR THE L IPSCHITZ CONSTANT
Revised: S PECTRAL NORM PROXY FOR THE L IPSCHITZ CONSTANT

Original: We can rewrite (1) as
Revised: We can rewrite (1) as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where a = ( a 1 , . . . ,
Revised: where a = ( a 1 , . . . ,

Original: a k ) ∈ R k , b = ( b 1 , . . .
Revised: a k ) ∈ R k , b = ( b 1 , . . .

Original: , b k ) ∈ R k ,
Revised: , b k ) ∈ R k ,

Original: W ∈ R k × d is the matrix whose (cid:96) th row is w (cid:96) , and ψ is extended from R → R to R k
Revised: W ∈ R k × d is the matrix whose (cid:96) th row is w (cid:96) , and ψ is extended from R → R to R k

Original: → R k by applying it coordinate-wise.
Revised: → R k by applying it coordinate-wise.

Original: We prove here the following:
Revised: We prove here the following:

Original: Theorem 3 Assume that ψ is L -Lipschitz.
Revised: Theorem 3 Assume that ψ is L -Lipschitz.

Original: For f ∈ F k ( ψ ) one has
Revised: For f ∈ F k ( ψ ) one has

Original: [Equation]
Revised: [Equation]

Original: For a generic data set, if f ( x i )
Revised: For a generic data set, if f ( x i )

Original: =
Revised: =

Original: y i , ∀ i ∈ [ n ] , then with positive probability one has:
Revised: y i , ∀ i ∈ [ n ] and f has no bias terms (i [Equation] e., b = 0 in (4) ), then with positive probability one has:

Original: [Equation]
Revised: [Equation]

Original: Note that we prove the inequality (5) only with positive probability (i.e., there exists a data set where the inequality is true), but in fact it is easy to derive the statement with high probability using classical concentration inequalities.
Revised: Note that we prove the inequality (6) only with positive probability (i.e., there exists a data set where the inequality is true), but in fact it is easy to derive the statement with high probability using classical concentration inequalities.

Original: Proof.
Revised: Proof.

Original: Since ψ : R → R is L -Lipschitz [Equation] we have:
Revised: Since ψ : R → R is L -Lipschitz, we have:

Original: [Equation]
Revised: [Equation]

Original: which directly proves (4)
Revised: which directly proves (5)

Original: [Equation]
Revised: [Equation]

Original: Next, following the proof of [Proposition
Revised: Next, following the proof of [Proposition

Original: [Equation]
Revised: [Equation]

Original: Bubeck et al.
Revised: Bubeck et al.

Original: (2020)] one obtains that for a generic data set, with positive probability, one has:
Revised: (2020)] one obtains that for a generic data set, with positive probability, one has (without bias terms):

Original: [Equation]
Revised: [Equation]

Original: It only remains to observe that:
Revised: It only remains to observe that:

Original: [Equation]
Revised: [Equation]

Original: op , which concludes the proof of (5).
Revised: which concludes the proof of (6).

Original: [Equation]
Revised: [Equation]

Original: 4. R E LU ACTIVATION
Revised: 4. U NDERCOMPLETE CASE

Original: Next we prove the conjecturefor ReLU with no biases in the high dimensional case n ≈ d (more precisely we replace n by d in the conjectured lower bound).
Revised: Next we prove the conjecture in the high dimensional case n ≈ d . More precisely we replace n by d in the conjectured lower bound.

Original: Theorem 4 Let f ∈ F k (ReLU) with no bias terms (i.e., b (cid:96) = 0 , ∀ (cid:96) ∈
Revised: Importantly note that the resulting lower bound then becomes non-trivial only in the regime k ≤ d (the “undercomplete case”). We consider in fact a slightly more general scenario than interpolation with a neural network, namely we simply assume that one interpolates the data with a function f ( x ) = g ( Px ) where P is a linear projection on a k -dimensional subspace (this clearly generalizes f ∈ F k ( ψ ) , in fact it even allows for the non-linearity ψ to depend on the data 3 , or to have a different non-linearity for each neuron). Theorem 4 Let n ≥ d .

Original: [ k ] in (1) ) and such that f ( x i ) = 
Revised: Let f : R d → R be a function such that f ( x i ) = y i , ∀ i ∈ [ n ] and moreover f ( x ) = g ( Px ) for some differentiable function g : R k

Original: y i , ∀ i ∈ [ n ] .
Revised: → R and matrix P ∈ R k × d .

Original: Then, for generic data, with probability at least 1 − exp( C − cd ) one must have
Revised: Then, for generic data, with probability at least 1 − exp( C − cd ) one must have

Original: [Equation]
Revised: [Equation]

Original: We note that the proof below does not rely on the randomness in the labels
Revised: Proof. Let us modify g so that P is simply an orthogonal projection operator (i.e., PP (cid:62)

Original: y
Revised: =

Original: i ’s, and thus it applies in particular to ﬁtting the value 1 everywhere.
Revised: I k ).

Original: This means thatassuming no bias terms is absolutely necessary for this proof (for otherwise one can ﬁt with just one neuron with bias 1 ). Moreover, even without bias, one can easily ﬁt smoothly the value 1 everywhere on the sphere with only k = d neurons, at least if one uses a quadratic activation function, namely with f ( x )
Revised: Let us also assume for sake of notational simplicity that we have a balanced data set of size 2 n , that is with: y 1 , . . . y n = +1 and y n +1 , . . .

Original: = (cid:80) di =1 ( x · e i ) 2 where e 1, . . 
Revised: y 2 n = − 1 . Let us denote x (cid:48)

Original: , e d is the canonical basis of R d . Thus in some sense replacing n by d in the conjectured lower bound is the best one can hope for without using the randomness of the labels. Proof.
Revised: i =

Original: For x ∈ R d , let a ( x ) ∈ R k be the “activation pattern” deﬁned by a ( x ) (cid:96) =
Revised: x i − x n + i for i ∈ [ n ] . The sequence x (cid:48) i is i.i.d.

Original: a (cid:96) 1 { w (cid:96) · x > 0 } , and let v ( x ) (cid:62) = a ( x ) (cid:62) W .{+(cid:48)(cid:62) i ] = 
+}
Revised: and satisﬁes E [ x (cid:48) i x {+(cid:48)(cid:62) i ] = 
+}

Original: Note that v ( x ) ∈ R d lives in a k -dimensional subspace L , namely the span of the rows of W , and we denote P for the projection on L (namely P = W (cid:62) ( WW (cid:62) ) − 1 W ).
Revised: d I d .

Original: With these notations we have that  the neural network computes the function f ( x ) = v ( x ) · x , and the gradient is ∇ f ( x ) = v ( x ) (cid:62) . In particular (cid:107) v ( x ) (cid:107) ≤ Lip( f ) and moreover by deﬁnition v ( x ) ∈ L which implies
Revised: Now observe that on the segment [ x i , x n + i ] (whose length is less than 2 ), the function f changes value from +1 to − 1 , and thus there exists z i ∈ [ x i , x n + i ] such that:

Original: [Equation]
Revised: [Equation]

Original: | Lip( f ) . In particular since | f ( xi )| = 1 we have (cid:107) 
Revised: Moreover one has (using that ∇ f ( x ) = P (cid:62) ∇ g ( x ) , and thus (cid:107)∇ g ( x ) (cid:107) =

Original: Px i (cid:107) ≥ 1 Lip( f ) .
Revised: (cid:107) P ∇ f ( x ) (cid:107) ≤ Lip( f ) )

Original: Hence:
Revised: [Equation] Combining the two above displays one has:

Original: [Equation]
Revised: [Equation]

Original: Using [Theorem 5.39, Vershynin (2012)]
Revised: Using [Theorem 5.39, Vershynin (2012)]

Original: (speciﬁcally (5.23))
Revised: (speciﬁcally (5.23))

Original: we know that with probability at least 1 − exp( C − cd ) we have (cid:13)(cid:13)(cid:80) ni
Revised: we know that with probability at least 1 − exp( C − cd ) we have (cid:13)(cid:13)(cid:80) ni

Original: =1 x  i x (cid:62) i (cid:13)(cid:13) op ≤ C nd .
Revised: =1 x (cid:48) i x (cid:48)(cid:62) i (cid:13)(cid:13) op ≤ C nd (here we use n ≥ d too).

Original: Moreoversince P is a projection on a kdimensional subspace we have (cid:107) P  (cid:107) op , ∗ ≤ k .
Revised: Moreover we have (cid:107) P (cid:62) P (cid:107) op , ∗ = Tr( P (cid:62) P ) = Tr( PP (cid:62) ) =

Original:  Thus we have (cid:104) (cid:80) ni
Revised: k . Thus we have (cid:104) (cid:80) ni

Original: =1 x  i x (cid:62) i , P  (cid:105) HS ≤ C n · kd so that with the above display one obtains:
Revised: =1 x (cid:48) i x (cid:48)(cid:62) i , P (cid:62) P (cid:105) HS ≤ C n · kd so that with the above display one obtains

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: concludes the proof

Original: [Equation]
Revised: [Equation]

Original: 4. P OWER ACTIVATION
Revised: 4. P OWER ACTIVATION

Original: We prove here the conjecture for the power activation function ψ ( t ) =
Revised: We prove here the conjecture for the power activation function ψ ( t ) =

Original: t p with p an integer and with no bias terms.
Revised: t p with p an integer and with no bias terms (we deal with general polynomials, including with bias, in Appendix D).

Original: Without bias such a network can be written as:
Revised: Without bias such a network can be written as:

Original: [Equation]
Revised: [Equation]

Original: where T = (cid:80) k(cid:96)
Revised: where T = (cid:80) k(cid:96)

Original: =
Revised: =

Original: 1 a (cid:96) w ⊗ p (cid:96) .
Revised: 1 a (cid:96) w ⊗ p (cid:96) .

Original: As we already saw in the proof of Theorem 2 (see speciﬁcally [Appendix
Revised: As we already saw in the proof of Theorem 2 (see speciﬁcally [Appendix

Original: A, Lemma 2]), without loss of generality we have k ≤ C p d p − 1 .
Revised: A, Lemma 2]), without loss of generality we have k ≤ C p d p − 1 .

Original: We now prove that tensor networks of the form (6) cannot obtain a Lipschitz constant 2 better than (cid:112) n/d p − 1 , in accordance with
Revised: We now prove that tensor networks of the form (7) cannot obtain a Lipschitz constant 4 better than (cid:112) n/d p − 1 , in accordance with

Original: Conjecture 1 for full rank tensors (where k ≈ d p − 1 ).
Revised: Conjecture 1 for full rank tensors (where k ≈ d p − 1 ).

Original: Theorem 5 Assume that we have a tensor T of order p such that

Original: [Equation]

Original: Then, for generic data, with probability at least 1 − C exp( − c p d ) , one must have

Original: [Equation]

Original: 0 1 2 3 4 5 6 7 8 9 10 11        m a x
Revised: 0 1 2 3 4 5 6 7 8 9 10 11        m a x

Original: i m u m r ando m g r ad i en t
Revised: i m u m r ando m g r ad i en t

Original: 60      m a x
Revised: 60      m a x

Original: i m u m r ando m g r ad i en t
Revised: i m u m r ando m g r ad i en t

Revised: Theorem 5 Assume that we have a tensor T of order p such that (cid:104) T, x ⊗ p

Revised: i (cid:105) =

Revised: y i , ∀ i ∈ [ n ] .

Revised: Then, for generic data, with probability at least 1 − C exp( − c p d ) , one must have

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Denoting Ω
Revised: Denoting Ω

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =

Revised: =

Original: y i x ⊗ p
Revised: y i x ⊗ p

Original: i , we have (using y 2 i = 1 for the ﬁrst equality and [Appendix A,
Revised: i , we have (using y 2 i = 1 for the ﬁrst equality and [Appendix A,

Original: Lemma 3] for the last inequality):
Revised: Lemma 3] for the last inequality):

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus we obtain (cid:107) T (cid:107) op ≥
Revised: Thus we obtain (cid:107) T (cid:107) op ≥

Original: [Equation]
Revised: [Equation]

Original: op , and it only remains to apply [Appendix B, Lemma 5] which states that with probability at least 1 − C exp( − c p d ) one has (cid:107) Ω (cid:107) op [Equation]
Revised: op , and it only remains to apply [Appendix B, Lemma 5] which states that with probability at least 1 − C exp( − c p d ) one has (cid:107) Ω (cid:107) op ≤ C p (cid:112) n d p − 1 .

Original: (cid:3)
Revised: (cid:3)

Original: In Appendix D we show how to extend this argument to any polynomial activation of degree p , and how to reﬁne the result for the special case of quadratic activations.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We consider a generic dataset from the Gaussian model (i.e., x 1 , . . .
Revised: We consider a generic dataset from the Gaussian model (i.e., x 1 , . . .

Original: , x n i.i.d.
Revised: , x n i.i.d.

Original: from N (0 , 1 d I d ) and labels y 1 , . . .
Revised: from N (0 , 1 d I d ) and labels y 1 , . . .

Original: , y n i.i.d from the uniform distribution over {− 1 , 1 } and independent of x 1 , . . .
Revised: , y n i.i.d from the uniform distribution over {− 1 , 1 } and independent of x 1 , . . .

Original: , x n ).
Revised: , x n ).

Original: For various values of ( n, d, k ) we train two-layers neural networks with k ReLU units and batch normalization (see Ioffe and Szegedy (2015)) between the linear layer and ReLU layer, using the Adam optimizer (Kingma and Ba, 2014) on the least squares loss.
Revised: For various values of ( n, d, k ) we train two-layers neural networks with k ReLU units and batch normalization (see Ioffe and Szegedy (2015)) between the linear layer and ReLU layer, using the Adam optimizer (Kingma and Ba, 2015) on the least squares loss.

Original: We keep the values of ( n, k, d ) where the network successfully memorizes the random labels (possibly after a rounding to {− 1 , +1 } , and such that prior to rounding the least squares loss is at most some small value ε to be speciﬁed later).
Revised: We keep the values of ( n, k, d ) where the network successfully memorizes the random labels (possibly after a rounding to {− 1 , +1 } , and such that prior to rounding the least squares loss is at most some small value ε to be speciﬁed later).

Original: Given a triple ( n, d, k ) , suppose the output of the trained network is f n,d,k : R d → R .
Revised: Given a triple ( n, d, k ) , suppose the output of the trained network is f n,d,k : R d → R .

Original: We then generate z 1 , . .
Revised: We then generate z 1 , . .

Original: , z T (where T = 1000 ) i.i.d from the distribution N (0 , 1 d I d ) , independently of everything else and deﬁne the “maximum random gradient” to be max i ∈
Revised: , z T (where T = 1000 ) i.i.d from the distribution N (0 , 1 d I d ) , independently of everything else and deﬁne the “maximum random gradient” to be max i ∈

Original: [ T ] (cid:107)∇ f n,k,d ( z i ) (cid:107) (it is our proxy for the true Lipschitz constant sup z ∈ S d − 1 (cid:107)∇ f n,d,k ( z ) (cid:107) ).
Revised: [ T ] (cid:107)∇ f n,k,d ( z i ) (cid:107) (it is our proxy for the true Lipschitz constant sup z ∈ S d − 1 (cid:107)∇ f n,d,k ( z ) (cid:107) ).

Original: Our experimental results are as follows:
Revised: Our experimental results are as follows:

Original: Experiment 1: We ran experiments with n between 100 and 2000 , d between ∼ 50 and ∼ n , and k between ∼ 10 and ∼ n (we also choose ε = 0 .
Revised: Experiment 1: We ran experiments with n between 100 and 2000 , d between ∼ 50 and ∼ n , and k between ∼ 10 and ∼ n (we also choose ε = 0 .

Original: 02 for the thresholding).
Revised: 02 for the thresholding).

Original: In Figure 2 we give a scatter plot of (cid:0)(cid:112) nk , max i ∈ [ T ] (cid:107)∇ f n,k,d ( z i ) (cid:107) (cid:1) , and as predicted we see a linear trend, thus providing empirical evidence for Conjecture 1.
Revised: In Figure 2 we give a scatter plot of (cid:0)(cid:112) nk , max i ∈ [ T ] (cid:107)∇ f n,k,d ( z i ) (cid:107) (cid:1) , and as predicted we see a linear trend, thus providing empirical evidence for Conjecture 1.

Original: Experiment 2: In this experiment, we investigate the two extreme cases k ∼ n and k ∼ n/d .
Revised: Experiment 2: In this experiment, we investigate the two extreme cases k ∼ n and k ∼ n/d .

Original: We ﬁx n = 10 4 and sweep the value of d between 10 to 5000 (we also choose ε = 0 .
Revised: We ﬁx n = 10 4 and sweep the value of d between 10 to 5000 (we also choose ε = 0 .

Original: 1 for the thresholding).
Revised: 1 for the thresholding).

Original: In the ﬁrst case, we let k = n and in the second case we let k = 10 n/d .
Revised: In the ﬁrst case, we let k = n and in the second case we let k = 10 n/d .

Original: In Figurewe plot √ d versus the maximum random gradient (as deﬁned above) for both cases.
Revised: In Figurewe plot √ d versus the maximum random gradient (as deﬁned above) for both cases.

Original: We observe a linear dependence between the maximum gradient value and √ d when we have k = 10 n/d , and roughly a constant maximum gradient value
Revised: We observe a linear dependence between the maximum gradient value and √ d when we have k = 10 n/d , and roughly a constant maximum gradient value

Original: when k = n , thus providing again evidence for
Revised: when k = n , thus providing again evidence for

Original: Conjecture
Revised: Conjecture

Original: A R ESULTS ON TENSORS
Revised: A R ESULTS ON TENSORS

Original: A tensor of order p is an array T =
Revised: A tensor of order p is an array T =

Original: ( T i 1 ,...,i p ) i 1 ,...,i p ∈ [ d ] .
Revised: ( T i 1 ,...,i p ) i 1 ,...,i p ∈ [ d ] .

Original: The Frobenius inner product for tensors is deﬁned by:
Revised: The Frobenius inner product for tensors is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: with the corresponding norm (cid:107) · (cid:107) F .
Revised: with the corresponding norm (cid:107) · (cid:107) F .

Original: A tensor is said to be of rank 1 if it can be written as:
Revised: A tensor is said to be of rank 1 if it can be written as:

Original: [Equation]
Revised: [Equation]

Original: for some u 1 , . . .
Revised: for some u 1 , . . .

Original: , u p ∈ R d .
Revised: , u p ∈ R d .

Original: The operator norm (cid:107) · (cid:107) op is deﬁned by:
Revised: The operator norm (cid:107) · (cid:107) op is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: For symmetric tensors (i.e., such that the entries of the array are invariant under permutation of the p indices),
Revised: For symmetric tensors (i.e., such that the entries of the array are invariant under permutation of the p indices),

Original: Banach’s Theorem (see e.g., [(2.32), Nemirovski (2004)]) states that in fact one has
Revised: Banach’s Theorem (see e.g., [(2.32), Nemirovski (2004)]) states that in fact one has

Original: [Equation]
Revised: [Equation]

Original: We refer to Friedland and Lim (2018) for more details and background on tensors.
Revised: We refer to Friedland and Lim (2018) for more details and background on tensors.

Original: We now list a couple of useful results, with short proofs.
Revised: We now list a couple of useful results, with short proofs.

Original: Lemma 

Revised: Lemma

Original: For a tensor T of order p , one has
Revised: For a tensor T of order p , one has

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: One has for any x, y ∈ S
Revised: One has for any x, y ∈ S

Original: d − 1 ,
Revised: d − 1 ,

Revised: [Equation]

Revised: [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: For any tensor T of order p , there exists w 1 , . . .
Revised: For any tensor T of order p , there exists w 1 , . . .

Original: , w 2 p d p − 1 ∈ R d and ξ 1 , . .
Revised: , w 2 p d p − 1 ∈ R d and ξ 1 , . .

Original: . , ξ 2 p d p − 1 ∈ {− 1 , +1 }
Revised: . , ξ 2 p d p − 1 ∈ {− 1 , +1 }

Original: such that for all x ∈ R d , (cid:104) T [Equation] x ⊗ p (cid:105) =
Revised: such that for all x ∈ R d , (cid:104) T, x ⊗ p (cid:105) =

Original: [Equation] (cid:96) =
Revised: 2 p d p − 1 (cid:88) (cid:96) =

Original: ξ (cid:96) · ( w (cid:96) · x ) p .
Revised: ξ (cid:96) · ( w (cid:96) · x ) p .

Original: Results like Lemma 2 go back at least to Reznick (1992).
Revised: Results like Lemma 2 go back at least to Reznick (1992).

Original: In fact much more precise results on minimal decomposition in rank- 1 tensors are known thanks to the work of Alexander and Hirschowitz (1995).
Revised: In fact much more precise results on minimal decomposition in rank- 1 tensors are known thanks to the work of Alexander and Hirschowitz (1995).

Original: We refer to (Comon et al., 2008) for more discussion on this topic.
Revised: We refer to (Comon et al., 2008) for more discussion on this topic.

Original: Proof.
Revised: Proof.

Original: First note that trivially T can be written as:
Revised: First note that trivially T can be written as:

Original: [Equation]
Revised: [Equation]

Original: Thus one only needs to prove that a function of the form x (cid:55)→ (cid:81) pq
Revised: Thus one only needs to prove that a function of the form x (cid:55)→ (cid:81) pq

Original: =1 ( w q · x ) can be written as the sum of 2 p functions of the form ( w (cid:48) · x ) p .
Revised: =1 ( w q · x ) can be written as the sum of 2 p functions of the form ( w (cid:48) · x ) p .

Original: To do so note that, with ε q i.i.d.
Revised: To do so note that, with ε q i.i.d.

Original: random signs,
Revised: random signs,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: For any tensor T of order p one has:
Revised: For any tensor T of order p one has:

Original: [Equation]
Revised: [Equation]

Original: The above result and its proof are directly taken from Li et al.
Revised: The above result and its proof are directly taken from Li et al.

Original: We only repeat the argument here for sake of completeness.
Revised: We only repeat the argument here for sake of completeness.

Original: Proof.
Revised: Proof.

Original: Note that the decomposition (9) is orthogonal, and thus for any tensor S of order p one has:
Revised: Note that the decomposition (10) is orthogonal, and thus for any tensor S of order p one has:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Thus one has (cid:107) T (cid:107) op , ∗ ≤ d p − 12 · (cid:107) T (cid:107) F .
Revised: Thus one has (cid:107) T (cid:107) op , ∗ ≤ d p − 12 · (cid:107) T (cid:107) F .

Original: By duality one also has (cid:107) T (cid:107) op ≥ d − p − 12 · (cid:107) T (cid:107) F , which concludes the proof.
Revised: By duality one also has (cid:107) T (cid:107) op ≥ d − p − 12 · (cid:107) T (cid:107) F , which concludes the proof.

Original: (cid:3)
Revised: (cid:3)

Original: B R ESULTS ON RANDOM TENSORS
Revised: B R ESULTS ON RANDOM TENSORS

Original: Lemma 

Revised: Lemma

Original: For any ﬁxed x ∈ S d − 1 and generic data, with probability at least 1 − C exp( − c p τ ) one has:
Revised: For any ﬁxed x ∈ S d − 1 and generic data, with probability at least 1 − C exp( − c p τ ) one has:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using [Theorem 1, Paouris et al.
Revised: Using [Theorem 1, Paouris et al.

Original: one has, for any ﬁxed x ∈ S
Revised: one has, for any ﬁxed x ∈ S

Original: d − 1 and τ ≤ n ,
Revised: d − 1 and τ ≤ n ,

Original: P (cid:32)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)
Revised: P (cid:32)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: where σ p denotes the p th moment of the standard Gaussian.
Revised: where σ p denotes the p th moment of the standard Gaussian.

Original: Let us denote n + = |{ i ∈ [ n ] : y i = +1 } and T + = (cid:80)
Revised: Let us denote n + = |{ i ∈ [ n ] : y i = +1 } and T + = (cid:80)

Original: i : y i =
Revised: i : y i =

Original: +1 x ⊗ p i , and similarly for n − , T − .
Revised: +1 x ⊗ p i , and similarly for n − , T − .

Original: Now with probability 1 − C exp( − cτ ) (with respect to the randomness of the y (cid:48)
Revised: Now with probability 1 − C exp( − cτ ) (with respect to the randomness of the y (cid:48)

Original: i s ) we have
Revised: i s ) we have

Original: [Equation]
Revised: [Equation]

Original: Thus combining the two above displays we obtain with probability at least 1 − C exp( − c p τ ) ,
Revised: Thus combining the two above displays we obtain with probability at least 1 − C exp( − c p τ ) ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 5 For generic data, with probability at least 1 − C exp( − c p d ) one has:
Revised: Lemma 5 For generic data, with probability at least 1 − C exp( − c p d ) one has:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let N be an 12 p -net of S d − 1 (in particular |N| ≤ C dp ).
Revised: Let N be an 12 p -net of S d − 1 (in particular |N| ≤ C dp ).

Original: By an union bound and Lemma 4 one has:
Revised: By an union bound and Lemma 4 one has:

Original: [Equation]
Revised: [Equation]

Original: Let T =
Revised: Let T =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =

Revised: =

Original: y i x ⊗ p i .
Revised: y i x ⊗ p i .

Original: Note that T is symmetric, and thus thanks to (8) and Lemma 1, one has:
Revised: Note that T is symmetric, and thus thanks to (9) and Lemma 1, one has:

Original: [Equation]
Revised: [Equation]

Original: and in particular (cid:107) T (cid:107) op ≤ 2 max x ∈N (cid:104) T, x ⊗ p (cid:105) , which together with (10) concludes the proof.
Revised: and in particular (cid:107) T (cid:107) op ≤ 2 max x ∈N (cid:104) T, x ⊗ p (cid:105) , which together with (11) concludes the proof.

Original: (cid:3)
Revised: (cid:3)

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: Let m =
Revised: Let m =

Original: nk (by assumption m ≤ c · d log( n ) ) and assume it is an integer.
Revised: nk (by assumption m ≤ c · d log( n ) ) and assume it is an integer.

Original: Let us choose m points with the same label, say it is the points x 1 , . . .
Revised: Let us choose m points with the same label, say it is the points x 1 , . . .

Original: , x m with label +1 .
Revised: , x m with label +1 .

Original: As in Section 3.1 let w ∈ R
Revised: As in Section 3.1 let w ∈ R

Original: d be the minimal norm vector that satisfy w · x i = 1 , and thus as we proved there with probability at least 1 − exp( C − cd ) one has (cid:107) w (cid:107) ≤ √ 2 m .
Revised: d be the minimal norm vector that satisfy w · x i = 1 , and thus as we proved there with probability at least 1 − exp( C − cd ) one has (cid:107) w (cid:107) ≤ √ 2 m .

Original: Crucially for the end of the proof, also note that the distribution of w is rotationally invariant.
Revised: Crucially for the end of the proof, also note that the distribution of w is rotationally invariant.

Original: Next observe that with probability at least 1 − 1 /n C (with respect to the sampling of x m +1 , . . .
Revised: Next observe that with probability at least 1 − 1 /n C (with respect to the sampling of x m +1 , . . .

Original: , x n ) one has max i ∈{ m +1 ,...,n } | w · x i | ≤ C ·(cid:107) w (cid:107) (cid:113) log( n ) d ≤ 12 .
Revised: , x n ) one has max i ∈{ m +1 ,...,n } | w · x i | ≤ C ·(cid:107) w (cid:107) (cid:113) log( n ) d ≤ 12 .

Original: In particular the cap C :
Revised: In particular the cap C :

Original: = { x ∈ S
Revised: = { x ∈ S

Original: d − 1 : w · x ≥ 12 } contains x 1 , . . .
Revised: d − 1 : w · x ≥ 12 } contains x 1 , . . .

Original: , x m but does not contain any x i , i > m .
Revised: , x m but does not contain any x i , i > m .

Original: Thus the neuron
Revised: Thus the neuron

Original: [Equation]
Revised: [Equation]

Original: ReLU
Revised: ReLU

Original: [Equation]
Revised: [Equation]

Original: computes the value 1 at points x 1 , . . .
Revised: computes the value 1 at points x 1 , . . .

Original: , x m and the value 

Revised: , x m and the value

Original: at the rest of the training set.
Revised: at the rest of the training set.

Original: One can now repeat this process, and build the neurons w 1 , . . .
Revised: One can now repeat this process, and build the neurons w 1 , . . .

Original: , w k (all with norm ≤ √ 2 m ), so that (with well-chosen signs ξ (cid:96) ∈ {− 1 , 1 } ) the data is perfectly ﬁtted by the function:
Revised: , w k (all with norm ≤ √ 2 m ), so that (with well-chosen signs ξ (cid:96) ∈ {− 1 , 1 } ) the data is perfectly ﬁtted by the function:

Original: [Equation]
Revised: [Equation]

Original: (cid:96) = 2 ·
Revised: (cid:96) = 2 ·

Original: [Equation]
Revised: [Equation]

Original: (cid:96) · ReLU (cid:18)
Revised: (cid:96) · ReLU (cid:18)

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: It only remains to estimate the Lipschitz constant.
Revised: It only remains to estimate the Lipschitz constant.

Original: Note that if a point x ∈ S d − 1 activates a certain subset A ⊂ { 1 , . . .
Revised: Note that if a point x ∈ S d − 1 activates a certain subset A ⊂ { 1 , . . .

Original: , k } of the neurons, then the gradient at this point is (cid:80) (cid:96) ∈
Revised: , k } of the neurons, then the gradient at this point is (cid:80) (cid:96) ∈

Original: A w (cid:48) (cid:96) with w (cid:48) (cid:96) = 2 ξ (cid:96) w (cid:96) .
Revised: A w (cid:48) (cid:96) with w (cid:48) (cid:96) = 2 ξ (cid:96) w (cid:96) .

Original: Using that the w (cid:96) (cid:48) are rotationally invariant, one also has with probability at least 1 − Cn exp( − cd ) that (cid:13)(cid:13)(cid:80) (cid:96)
Revised: Using that the w (cid:96) (cid:48) are rotationally invariant, one also has with probability at least 1 − Cn exp( − cd ) that (cid:13)(cid:13)(cid:80) (cid:96)

Original: ∈
Revised: ∈

Original: A w (cid:48) (cid:96) (cid:13)(cid:13) 2 ≤ C · | A | · m for all A ⊂ { 1 , . . . , k } .
Revised: A w (cid:48) (cid:96) (cid:13)(cid:13) 2 ≤ C · | A | · m for all A ⊂ { 1 , . . . , k } .

Original: Thus it only remains to control how large
Revised: Thus it only remains to control how large

Original: A can be.
Revised: A can be.

Original: We show below that |
Revised: We show below that |

Original: A | ≤ Cm log( d ) with probability at least 1 − C exp( − cd log( d )) which will conclude the proof.
Revised: A | ≤ Cm log( d ) with probability at least 1 − C exp( − cd log( d )) which will conclude the proof.

Original: If x activates neuron (cid:96) then w (cid:96) · x ≥ 12 ≥ (cid:107) w (cid:96) (cid:107) 4 √ m .
Revised: If x activates neuron (cid:96) then w (cid:96) · x ≥ 12 ≥ (cid:107) w (cid:96) (cid:107) 4 √ m .

Original: Now note that for any ﬁxed x ∈ S d − 1 and ﬁxed
Revised: Now note that for any ﬁxed x ∈ S d − 1 and ﬁxed

Original: A ⊂ [ k ] , P (cid:16) ∀ (cid:96) ∈
Revised: A ⊂ [ k ] , P (cid:16) ∀ (cid:96) ∈

Original: A, w (cid:96) · x ≥ (cid:107) w (cid:96) (cid:107) 4 √ m (cid:17) ≤ C exp (cid:0) − c | A | dm (cid:1) , so that
Revised: A, w (cid:96) · x ≥ (cid:107) w (cid:96) (cid:107) 4 √ m (cid:17) ≤ C exp (cid:0) − c | A | dm (cid:1) , so that

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: In particular we conclude that with a = Cm log( d ) the probability that a ﬁxed point on the sphere activates more than a neuron is exponentially small in d log( d ) (recall that m log( k ) ≤ cd by assumption).
Revised: In particular we conclude that with a = Cm log( d ) the probability that a ﬁxed point on the sphere activates more than a neuron is exponentially small in d log( d ) (recall that m log( k ) ≤ cd by assumption).

Original: Thus we can conclude via an union bound on an ε -net that the same holds for the entire sphere simultaneously.
Revised: Thus we can conclude via an union bound on an ε -net that the same holds for the entire sphere simultaneously.

Original: This concludes the proof.
Revised: This concludes the proof.

Original: D F URTHER RESULTS AROUND C ONJECTURED. P OLYNOMIAL ACTIVATION
Revised: D F URTHER RESULTS AROUND C ONJECTURED. P OLYNOMIAL ACTIVATION

Original: We now observe that one can generalize Theorem 5 to handle biases (the parameters b l in 1), and in fact even general polynomial activation function.
Revised: We now observe that one can generalize Theorem 5 to handle biases (the parameters b l in 1), and in fact even general polynomial activation function.

Original: Indeed, observe that any polynomial of (cid:104) w, x (cid:105) + b must also be a polynomial in (cid:104) w, x (cid:105) , albeit with different coefﬁcients.
Revised: Indeed, observe that any polynomial of (cid:104) w, x (cid:105) + b must also be a polynomial in (cid:104) w, x (cid:105) , albeit with different coefﬁcients.

Original: Theorem 6 Let ψ ( t ) =
Revised: Theorem 6 Let ψ ( t ) =

Original: (cid:80) pq =

Revised: (cid:80) pq =

Original: α q t q and assume that we have f ∈ F k ( ψ ) such that f ( x i ) =
Revised: α q t q and assume that we have f ∈ F k ( ψ ) such that f ( x i ) =

Original: y i , ∀ i ∈ [ n ] .
Revised: y i , ∀ i ∈ [ n ] .

Original: Then, for generic data, with probability at least 1 − C exp( − c p d ) one must have
Revised: Then, for generic data, with probability at least 1 − C exp( − c p d ) one must have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that for f ∈ F k ( ψ ) there exists tensors T 0 , . .
Revised: Note that for f ∈ F k ( ψ ) there exists tensors T 0 , . .

Original: , T p , such that T q is a tensor of order q , and f can be written as:
Revised: , T p , such that T q is a tensor of order q , and f can be written as:

Original: [Equation]
Revised: [Equation]

Original: Now let us deﬁne Ω q = (cid:80) ni =

Revised: Now let us deﬁne Ω q = (cid:80) ni =

Original: y i x ⊗ q i , and observe that n =
Revised: y i x ⊗ q i , and observe that n =

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: , and thus there exists q ∈ { 1 , . .
Revised: , and thus there exists q ∈ { 1 , . .

Original: , p } such that (cid:104) T q , Ω q (cid:105) ≥ c p n (we ignore the term q = 0 by considering the largest balanced subset of the data, i.e. we assume (cid:80) ni
Revised: , p } such that (cid:104) T q , Ω q (cid:105) ≥ c p n (we ignore the term q = 0 by considering the largest balanced subset of the data, i.e. we assume (cid:80) ni

Original: =

Revised: =

Original: y i = 0 ).
Revised: y i = 0 ).

Original: Now one can repeat the proof of Theorem 5 to obtain that with probability at least 1 − C exp( − c p d ) , one has (cid:107) T q (cid:107) op ≥ c p (cid:112) n
Revised: Now one can repeat the proof of Theorem 5 to obtain that with probability at least 1 − C exp( − c p d ) , one has (cid:107) T q (cid:107) op ≥ c p (cid:112) n

Original: d p − 1 .
Revised: d p − 1 .

Original: It only remains to observe that the Lipschitz constant of f on the unit ball is lower bounded by (cid:107) T q (cid:107) op .
Revised: It only remains to observe that the Lipschitz constant of f on the unit ball is lower bounded by (cid:107) T q (cid:107) op .

Original: As we mentioned in Section 4.3, without loss of generality we can assume T q is symmetric, and thus by (8) there exists x ∈ S d − 1 such that (cid:107) T q (cid:107) op
Revised: As we mentioned in Section 4.3, without loss of generality we can assume T q is symmetric, and thus by (9) there exists x ∈ S d − 1 such that (cid:107) T q (cid:107) op

Original: = (cid:104) T q , x ⊗ q (cid:105) .
Revised: = (cid:104) T q , x ⊗ q (cid:105) .

Original: Now consider the univariate polynomial P ( t ) =
Revised: Now consider the univariate polynomial P ( t ) =

Original: f ( tx ) .
Revised: f ( tx ) .

Original: By Markov brothers’ inequality one has max t ∈ [ − 1 , 1] P ( t ) ≥ | P ( q ) (0) |
Revised: By Markov brothers’ inequality one has max t ∈ [ − 1 , 1] P ( t ) ≥ | P ( q ) (0) |

Original: = q ! · |(cid:104) T q , x ⊗ q (cid:105)| = q ! · (cid:107) T q (cid:107) op , thus concluding the proof.
Revised: = q ! · |(cid:104) T q , x ⊗ q (cid:105)| = q ! · (cid:107) T q (cid:107) op , thus concluding the proof.

Original: (cid:3)
Revised: (cid:3)

Original: D. Q UADRATIC ACTIVATION
Revised: D. Q UADRATIC ACTIVATION

Original: In Section 4.3 we obtained a lower bound for tensor networks that match Conjecture 1 only when the rank of the corresponding tensor is maximal.
Revised: In Section 4.3 we obtained a lower bound for tensor networks that match Conjecture 1 only when the rank of the corresponding tensor is maximal.

Original: Here we show that for quadratic networks (i.e., p = 2 ) we can match Conjecture 1, and in fact even obtain a better bound, for any rank k :
Revised: Here we show that for quadratic networks (i.e., p = 2 ) we can match Conjecture 1, and in fact even obtain a better bound, for any rank k :

Original: Theorem 7 Assume that we have a matrix T ∈ R d × d with rank k such that:
Revised: Theorem 7 Assume that we have a matrix T ∈ R d × d with rank k such that:

Original: [Equation]
Revised: [Equation]

Original: Then, for generic data, with probability at least 1 − C exp( − cd ) , one must have
Revised: Then, for generic data, with probability at least 1 − C exp( − cd ) , one must have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is exactly the same as for Theorem 5, except that in (7), instead of using Lemma 3 we use the fact that for a matrix T of rank k one has:
Revised: The proof is exactly the same as for Theorem 5, except that in (8), instead of using Lemma 3 we use the fact that for a matrix T of rank k one has:

Original: [Equation]
Revised: [Equation]