Original: Learning Single-Index Models with Shallow Neural Networks
Revised: Learning Single-Index Models with Shallow Neural Networks

Original: Abstract
Revised: Abstract

Original: Single-index models are a class of functions given by an unknown univariate “link” function applied to an unknown one-dimensional projection of the input.
Revised: Single-index models are a class of functions given by an unknown univariate “link” function applied to an unknown one-dimensional projection of the input.

Original: These models are particularly relevant in high dimension, when the data might present low-dimensional structure that learning algorithms should adapt to.
Revised: These models are particularly relevant in high dimension, when the data might present low-dimensional structure that learning algorithms should adapt to.

Original: While several statistical aspects of this model, such as the sample complexity of recovering the relevant (one-dimensional) subspace, are well-understood, they rely on tailored algorithms that exploit the speciﬁc structure of the target function.
Revised: While several statistical aspects of this model, such as the sample complexity of recovering the relevant (one-dimensional) subspace, are well-understood, they rely on tailored algorithms that exploit the speciﬁc structure of the target function.

Original: In this work, we introduce a natural class of shallow neural networks and study its ability to learn single-index models via gradient ﬂow .
Revised: In this work, we introduce a natural class of shallow neural networks and study its ability to learn single-index models via gradient ﬂow .

Original: More precisely, we consider shallow networks in which biases of the neurons are frozen at random initialization.
Revised: More precisely, we consider shallow networks in which biases of the neurons are frozen at random initialization.

Original: We show that the corresponding optimization landscape is benign, which in turn leads to generalization guarantees that match the near-optimal sample complexity of dedicated semi-parametric methods.
Revised: We show that the corresponding optimization landscape is benign, which in turn leads to generalization guarantees that match the near-optimal sample complexity of dedicated semi-parametric methods.

Original:  Introduction
Revised:  Introduction

Original: High-dimensional learning with both computational and statistical guarantees, which is particularly relevant given the current scaling trends, remains an outstanding challenge.
Revised: High-dimensional learning with both computational and statistical guarantees, which is particularly relevant given the current scaling trends, remains an outstanding challenge.

Original: One important question which has received considerable attention is on understanding the advantages of using non-linear learning models, such as neural networks, over more mature (from a theoretical standpoint) counterparts, such as kernel methods [55, 56, 83, 22].
Revised: One important question which has received considerable attention is on understanding the advantages of using non-linear learning models, such as neural networks, over more mature (from a theoretical standpoint) counterparts, such as kernel methods [56, 57, 84, 22].

Original: Surprisingly, the question remains largely open even for shallow neural networks.
Revised: Perhaps surprisingly, the question remains largely open even for shallow neural networks.

Original: While approximation beneﬁts of shallow neural networks over non-adaptive kernels have been known for decades [9, 68], another important piece of the theoretical puzzle was provided by [5] whose analysis hinted at an inherent statistical advantage of neural networks for extracting information from high-dimensional data with a “hidden” low-dimensional structure.
Revised: While approximation beneﬁts of shallow neural networks over non-adaptive kernels have been known for decades [9, 69] another important piece of the theoretical puzzle was provided by [5] whose analysis hinted at an inherent statistical advantage of neural networks for extracting information from high-dimensional data with a “hidden” low-dimensional structure.

Original: Providing computational guarantees, the remaining piece of this puzzle, is still mostly unresolved.
Revised: Providing computational guarantees, the remaining piece of this puzzle, is still mostly unresolved.

Original: Several computational hardness results for learning functions that can be efﬁciently approximated by shallow neural networks have been established in the literature [30, 41, 28, 77, 20], ruling out positive results in the general setting.
Revised: Several computational hardness results for learning functions that can be efﬁciently approximated by shallow neural networks have been established in the literature [30, 41, 28, 78, 20], ruling out positive results in the general setting.

Original: On the other hand, progress has been made on the positive side [1, 2, 72] by focusing on function classes with strong structural properties, thereby showcasing the adaptive representation learning capabilities of neural networks.
Revised: On the other hand, progress has been made on the positive side [1, 2, 73] by focusing on function classes with strong structural properties, thereby showcasing the adaptive representation learning capabilities of neural networks.

Original: This work aligns with the latter effort, and focuses on the class of single-index models .
Revised: This work aligns with the latter effort, and focuses on the class of single-index models .

Original: Single-index models are high-dimensional functions F : R d !
Revised: Singleindex models are high-dimensional functions F : R d !

Original: R of the form F ⇤ ( x ) =
Revised: R of the form F ⇤ ( x ) =

Original: f
Revised: f

Original: ⇤ ( h ✓ ⇤ , x i ) , where both the univariate “link” function f
Revised: ⇤ ( h ✓ ⇤ , x i ) , where both the univariate “link” function f

Original: ⇤ : R !
Revised: ⇤ : R !

Original: R and relevant (one-dimensional) subspace ✓ ⇤ 

Revised: R and relevant (one-dimensional) subspace ✓ ⇤

Original: S d  1 are unknown.
Revised: S d  1 are unknown.

Original: These models have been extensively studied in the statistics literature [44, 47, 26, 45, 33], leading to dedicated algorithmic procedures, and can be provably approximated with shallow neural networks without incurring in a curse-of-dimensionality [5].
Revised: These models have been extensively studied in the statistics literature [45, 48, 26, 46, 33], leading to dedicated algorithmic procedures, and can be provably approximated with shallow neural networks without incurring in a curse-of-dimensionality [5].

Original: In contrast, the analysis of neural network learning using gradient-based methods hasbeen more heavily focused on the so-called 36th Conference on Neural Information Processing Systems (NeurIPS 2022). “Teacher-Student” setup [42, 39,84, 85,  10], where the link function f
Revised: In contrast, the analysis of neural network learning using gradient-based methods has focused on the so-called “TeacherStudent” setup [43, 39, 85, 86, 10], where the link function f

Original: ⇤ is assumed to be known and used as the activation function for the student.
Revised: ⇤ is assumed to be known and used as the activation function for the student.

Original: The mix of a high-dimensional parametric component (the hidden direction) with a non-parametric one in low dimension (the link function) in single-index models naturally suggests a shallow neural network architecture where the inner weights are shared and “active”, while the biases are “lazy” [23].
Revised: The mix of a high-dimensional parametric component (the hidden direction) with a non-parametric one in low dimension (the link function) in single-index models naturally suggests a shallow neural network architecture where the inner weights are shared and “active”, while the biases are “lazy” [23].

Original: We instantiate such an architecture by freezing the biases attheir random initialization, and analyze gradient descent on theremaining free parameters in the continuous-time limit.
Revised: We instantiate such an architecture by freezing the biases at random initialization, and analyze gradient descent on the free parameters in the continuous-time limit.

Original: Our main results establish that as soon as the width N of the network is larger than a quantity which depends solely on smoothness properties of the (univariate) link function f ⇤ , gradient ﬂow recovers the unknown direction ✓ ⇤ with near optimal sample complexity
Revised: Our main results establish that as soon as the width N of the network is larger than a quantity which depends solely on smoothness properties of the (univariate) link function f ⇤ , gradient ﬂow recovers the unknown direction ✓ ⇤ with near optimal sample complexity

Original: O ( d s ) , where s is the socalled information-exponent of the link function [10] (at least when s  3 , see Theorem 6.1 for the formal result), and approximates the univariate link function f ⇤ near-optimally (see Corollary 6.4).
Revised: O ( d s ) , where s is the socalled information-exponent of the link function [10] (at least when s  3 , see Theorem 6.1 for the formal result), and approximates the univariate link function f ⇤ near-optimally (see Corollary 6.4).

Original: The information exponent roughly captures the signal strength, which here refers to the alignment between the network direction ✓ and the hidden direction ✓ ⇤ , at typical initializations.
Revised: The information exponent roughly captures the signal strength, which here refers to the alignment between the network direction ✓ and the hidden direction ✓ ⇤ , at typical initializations.

Original: The success of gradient ﬂow relies on the benign optimization landscape of the empirical loss, though the presence of degenerate saddles necessitates a careful analysis leveraging uniform convergence of the empirical landscape [60, 35].
Revised: The success of gradient ﬂow relies on the benign optimization landscape of the empirical loss, though the presence of degenerate saddles necessitates a careful analysis leveraging uniform convergence of the empirical landscape [60, 35].

Original: We show that gradient ﬂow over our proposed neural network architecture solves two distinct problems—univariate non-parametric kernel ridge regression and non-convex optimization in high dimension— simultaneously and sample-efﬁciently , cementing its role as a versatile algorithm for high-dimensional learning.
Revised: We show that gradient ﬂow over our proposed neural network architecture solves two distinct problems—univariate non-parametric kernel ridge regression and non-convex optimization in high dimension— simultaneously and efﬁciently , cementing its role as a versatile algorithm for high-dimensional learning.

Original: We illustrate our theoretical results with experiments in Section B.
Revised: We illustrate our theoretical results with experiments in Section A.

Original:  Related Work
Revised:  Related Work

Original: Single–index and multi-index models.
Revised: Single-index and multi-index models.

Original: A useful modeling assumption in high-dimensional regression is that the regression function F ( x ) =
Revised: A useful modeling assumption in high-dimensional regression is that the regression function F ( x ) =

Original: E [ y | x ] only depends on one or a few directions.
Revised: E [ y | x ] only depends on one or a few directions.

Original: This leads to the single-index model F ( x ) =
Revised: This leads to the single-index model F ( x ) =

Original: f ⇤ ( h ✓ ⇤ , x i ) , and multi-index model F ( x ) =
Revised: f ⇤ ( h ✓ ⇤ , x i ) , and multi-index model F ( x ) =

Original: f ⇤ ( h ✓ ⇤ 1 , x i , . . .
Revised: f ⇤ ( h ✓ ⇤ 1 , x i , . . .

Original: , h ✓ ⇤ k , x i ) , with k typically much smaller than the dimension.
Revised: , h ✓ ⇤ k , x i ) , with k typically much smaller than the dimension.

Original: Such models have a long history in the statistics literature and different methods exist for various estimation problems, including projection pursuit [36, 46], slicing [51], gradient-based estimators [52], and moment-based estimators [26].
Revised: Such models have a long history in the statistics literature and different methods exist for various estimation problems, including projection pursuit [36, 47], slicing [52], gradient-based estimators [53], and moment-based estimators [26].

Original: When the function f
Revised: When the function f

Original: ⇤ is also to be estimated, we face a semi-parametric problem involving parameter recovery of ✓ ⇤ and non-parametric estimation of f ⇤ .
Revised: ⇤ is also to be estimated, we face a semi-parametric problem involving parameter recovery of ✓ ⇤ and non-parametric estimation of f ⇤ .

Original: Our work is closely related to [33], which also characterizes the population landscape of certain objectives by leveraging
Revised: Our work is closely related to [33], which also characterizes the population landscape of certain objectives by leveraging

Original: Gaussian data.
Revised: Gaussian data.

Original: Multi-index models are also studied in [5] in the context of shallow neural networks, where it is shown that certain models of inﬁnite-width shallow networks can adapt to such lowdimensional structure, though no tractable algorithms are introduced.
Revised: Multi-index models are also studied in [5] in the context of shallow neural networks, where it is shown that certain models of inﬁnite-width shallow networks can adapt to such lowdimensional structure, though no tractable algorithms are introduced.

Original: The works [8, 19, 65] show that certain neural networks trained close to initialization can learn certain sparse polynomials which take the form of multi-index models, but such networks do not directly aim to learn target directions.
Revised: The works [8, 19, 66] show that certain neural networks trained close to initialization can learn certain sparse polynomials which take the form of multi-index models, but such networks do not directly aim to learn target directions.

Original: Recently, [1] studied the learnability of functions on the hypercube by shallow neural networks with stochastic gradient descent and introduces the merged staircase property , which provides necessary and sufﬁcient conditions for learnability with linear sample complexity n = O ( d ) .
Revised: Recently, [1] studied the learnability of functions on the hypercube by shallow neural networks with stochastic gradient descent and introduces the merged staircase property , which provides necessary and sufﬁcient conditions for learnability with linear sample complexity n = O ( d ) .

Original: While they learn a broader class of functions (including multi-index model) for a more efﬁcient sample complexity regime ( O ( d ) vs O ( d s ) ), their setup is restricted to simple discrete data distributions, while our work captures the regime of semi-parametric estimation by considering Gaussian data without the sparsity requirements on F implied by their merged staircase property.
Revised: While they learn a broader class of functions (including multi-index model) for a more efﬁcient sample complexity regime ( O ( d ) vs O ( d s ) ), their setup is restricted to simple discrete data distributions, while our work captures the regime of semi-parametric estimation by considering Gaussian data without the sparsity requirements on F implied by their merged staircase property.

Original: Concurrently to our work, [4] and [27] studied the learnability of certain single and multi-index models on Gaussian data with shallow networks, by performing a single gradient step on the ﬁrst layer before ﬁtting the second layer.
Revised: Concurrently to our work, [4] and [27] studied the learnability of certain single and multi-index models on Gaussian data with shallow networks, by performing a single gradient step on the ﬁrst layer before ﬁtting the second layer.

Original: While the single step is sufﬁcient to provide a separation from kernel methods in these works, we show that optimizing both layers jointly until convergence (for a more simplistic architecture) can signiﬁcantly improve the rates, by fully decoupling the non-parametric learning part from the high-dimensional inference of the hidden direction.
Revised: While the single step is sufﬁcient to provide a separation from kernel methods in these works, we show that optimizing both layers jointly until convergence (for a more simplistic architecture) can signiﬁcantly improve the rates, by fully decoupling the non-parametric learning part from the high-dimensional inference of the hidden direction.

Original: Finally, recently [64] studied the ability of shallow neural networks to learn certain single and multi-index models, showing in particular that
Revised: Finally, recently [64] studied the ability of shallow neural networks to learn certain single and multi-index models, showing in particular that

Original: SGD-trained ReLU networks can learn single-index functions with monotonic index function (corresponding in our setting to s = 1 ) with linear (up to logarithmic factors) sample complexity.
Revised: SGD-trained ReLU networks can learn single-index functions with monotonic index function (corresponding in our setting to s = 1 ) with linear (up to logarithmic factors) sample complexity.

Original: Our results therefore extend such positive guarantees to a broader class of index functions with arbitrary information exponent.
Revised: Our results therefore extend such positive guarantees to a broader class of index functions with arbitrary information exponent.

Original: Teacher–student models.
Revised: Teacher–student models.

Original: In the context of neural networks, several works have considered the teacher–student setting [34], where the target function F takes the form of a neural network with the same activation as  network used for learning [42, 39,84, 76, 85,  10, 81].
Revised: In the context of neural networks, several works have considered the teacher–student setting [34], where the target function F takes the form of a neural network with the same activation as the network used for learning [43, 39, 85, 77, 86, 10, 82].

Original: In this case the problem does not involve non-parametric estimation as in our setup, but this line of work often involves studying optimization landscapes similar to ours for estimating hidden directions.
Revised: In this case the problem does not involve non-parametric estimation as in our setup, but this line of work often involves studying optimization landscapes similar to ours for estimating hidden directions.

Original: In particular, the population landscape appearing in [10] is similar to ours, based on Hermite coefﬁcients of link functions.
Revised: In particular, the population landscape appearing in [10] is similar to ours, based on Hermite coefﬁcients of link functions.

Original: The follow-up work [11] extends this  multiple student neuron directions, but still focuses on parametric rather than non-parametric statistical problems.
Revised: The follow-up work [11] extends this to multiple student neuron directions, but still focuses on parametric rather than non-parametric statistical problems.

Original: Kernels and random features.
Revised: Kernels and random features.

Original: In order to obtain non-parametric estimation guarantees for learning the target function f ⇤ of the single-index model, our work builds on the kernel methods literature for approximation and non-parametric regression [75, 13, 7], their links with neural networks [24, 5], and in particular on random feature approximation [69, 6, 71, 61].
Revised: In order to obtain non-parametric estimation guarantees for learning the target function f ⇤ of the single-index model, our work builds on the kernel methods literature for approximation and non-parametric regression [76, 13, 7], their links with neural networks [24, 5], and in particular on random feature approximation [70, 6, 72, 61].

Original: Non-convex  optimization landscapes.
Revised: Non-convex and non-smooth optimization landscapes.

Original: There is a vast literature studying tractable non-convex optimization landscapes, arising from high-dimensional statistics and statistical physics [58, 57, 37, 12, 78, 17, 39, 70, 54].
Revised: There is a vast literature studying tractable non-convex optimization landscapes, arising from high-dimensional statistics and statistical physics [59, 58, 37, 12, 79, 17, 39, 71, 55].

Original: A particular aspect of our setup is that the optimization landscape does not have the strict saddle property, which is often leveraged to establish global convergence [49].
Revised: A particular aspect of our setup is that the optimization landscape does not have the strict saddle property, which is often leveraged to establish global convergence [50].

Original: [60, 35] study concentration properties of the empirical landscape to the population one for nonconvex problems including generalized linear models.
Revised: [60, 35] study concentration properties of the empirical landscape to the population one for non-convex problems including generalized linear models.

Original: Our results rely on similar concentration analyses, but depart from these previous work by also allowing optimization of the link function, and by supporting the non-smoothness arising from ReLU activations.
Revised: Our results rely on similar concentration analyses, but depart from these previous work by also allowing optimization of the link function, and by supporting the non-smoothness arising from ReLU activations.

Original: On the algorithmic side, we consider gradient ﬂows on non-convex and non-smooth landscapes, which require careful technical treatment, but have been studied by previous works [32, 29, 48].
Revised: On the algorithmic side, we consider gradient ﬂows on non-convex and non-smooth landscapes, which require careful technical treatment, but have been studied by previous works [32, 29, 49].

Original: We refer the interested reader to Appendix G for more details on this technical issue.
Revised: We refer the interested reader to Appendix F for more details on this technical issue.

Original:  Preliminaries
Revised:  Preliminaries

Original: We focus on regression problems under a single-index model with Gaussian input data.
Revised: We focus on regression problems under a single-index model with Gaussian input data.

Original: Speciﬁcally, we assume
Revised: Speciﬁcally, we assume

Original: d -dimensional inputs x ⇠  d := N (0 , I d ) , and labels
Revised: d -dimensional inputs x ⇠  d := N (0 , I d ) , and labels

Original: [Equation]
Revised: [Equation]

Original: where ✓ ⇤ 2 S d  1 and ⇠ ⇠ N (0 ,  2 ) is an independent, additive Gaussian noise.
Revised: where ✓ ⇤ 2 S d  1 and ⇠ ⇠ N (0 ,  2 ) is an independent, additive Gaussian noise.

Original: The normalization ✓ ⇤ 2 S d  1 is to ensure that both f ⇤ and ✓ ⇤ are identiﬁable.
Revised: The normalization ✓ ⇤ 2 S d  1 is to ensure that both f ⇤ and ✓ ⇤ are identiﬁable.

Original: Shallow networks and random features.
Revised: Shallow networks and random features.

Original: We consider learning algorithms based on shallow neural networks of the form
Revised: We consider learning algorithms based on shallow neural networks of the form

Original: [Equation]
Revised: [Equation]

Original: with ✓ 2 S d  1 , where  ( u ) = max { 0 , u } is the ReLU activation, b i ⇠ N (0 , ⌧ 2 ) (we assume ⌧ > 1 ) are random “bias” scalars that are frozen throughout training, and " i are random signs with
Revised: with ✓ 2 S d  1 , where  ( u ) = max { 0 , u } is the ReLU activation, b i ⇠ N (0 , ⌧ 2 ) (we assume ⌧ > 1 ) are random “bias” scalars that are frozen throughout training, and " i are random signs with

Original: Rademacher distribution (i.e., uniform over {± 1 } ), independent from b i , and also frozen during training.
Revised: Rademacher distribution (i.e., uniform over {± 1 } ), independent from b i , and also frozen during training.

Original: The resulting vector of random features is thus  ( u ) =
Revised: The resulting vector of random features is thus  ( u ) =

Original: 1 p N (  ( " i u  b i ))
Revised: 1 p N (  ( " i u  b i ))

Original: i 2 [ N ] .
Revised: i 2 [ N ] .

Original: The choice of ReLU activation is motivated by its popularity amongst practitioners.
Revised: The choice of ReLU activation is motivated by its popularity among practitioners.

Original: As we shall see, the fact that  is non-smooth introduces some technical challenges, but its piece-line structure enables dedicated arguments both in terms of approximation as well as in the study of the optimization landscape.
Revised: As we shall see, the fact that  is non-smooth introduces some technical challenges, but its piece-wise linear structure enables dedicated arguments both in terms of approximation as well as in the study of the optimization landscape.

Original: In Appendix H we discuss how our main results are affected when replacing the ReLU by a smooth activation, especially when choosing it such that  0 is Lipschitz.
Revised: In Appendix G we discuss how our main results are affected when replacing the ReLU by a smooth activation, especially when choosing it such that  0 is Lipschitz.

Original: Empirical risk minimization.
Revised: Empirical risk minimization.

Original: The supervised learning task is to estimate F ⇤ (and therefore both f ⇤ and ✓ ⇤ ) from samples { ( x i , y i ) } i =1 ...n .
Revised: The supervised learning task is to estimate F ⇤ (and therefore both f ⇤ and ✓ ⇤ ) from samples { ( x i , y i ) } i =1 ...n .

Original: We will focus on mean-squared error with Tychonov regularisation, determined by the following losses.
Revised: We will focus on mean-squared error with Tychonov regularisation, determined by the following losses.

Original: Deﬁnition 3.1 (Population Risk) .
Revised: Deﬁnition 3.1 (Population risk) .

Original: We deﬁne the ` 2 -regularized population loss by
Revised: We deﬁne the ` 2 -regularized population loss by

Revised: L ( c,

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3.2 (Empirical Risk) .
Revised: Deﬁnition 3.2 (Empirical risk) .

Original: We deﬁne the ` 2 -regularized empirical loss by
Revised: We deﬁne the ` 2 -regularized empirical loss by

Original: [Equation]
Revised: [Equation]

Original: Hermite decompositions.
Revised: Hermite decomposition.

Original: Given that our data is normally distributed, we consider the family of (normalized)
Revised: Given that our data is normally distributed, we consider the family of (normalized)

Original: Hermite polynomials { h j } j 2 N , which form an orthonormal basis of L 2 (  ) , the space of squared-integrable function under the Gaussian measure  := N (0 , 1) .
Revised: Hermite polynomials { h j } j 2 N , which form an orthonormal basis of L 2 (  ) , the space of squared-integrable function under the Gaussian measure  := N (0 , 1) .

Original: We will denote by f ⇤ =
Revised: We will denote by f ⇤ =

Original: P j ↵ j h j the Hermite decomposition of the target link function.
Revised: P j ↵ j h j the Hermite decomposition of the target link function, which we assume is in

Revised: L 2 (  ) henceforth.

Original: We apply the following useful properties of Hermite polynomials [66]:
Revised: We apply the following useful properties of Hermite polynomials [67, Chapter 11.2]:

Original: [Equation]
Revised: [Equation]

Original: where h · , · i  d is the inner product in L 2 (  d ) and  the Kronecker delta.
Revised: where h · , · i  d is the inner product in L 2 (  d ) and  the Kronecker delta.

Original: We will assume throughout that k f ⇤ k 2  =
Revised: We will assume throughout that k f ⇤ k 2  =

Original: [-P j ↵ 2 j , k f 
 ⇤ k 2  =-]
Revised: [-P j ↵ 2 j , k f 
 ⇤ k 2  =-][Equation] [Equation]

Original: P j j ↵ 2 j , and k [-f 
-]
Revised: , and k [-f 
-][Equation] P

Original: ⇤ k 2  =
Revised: [Equation]

Original: P j j ( j  1) ↵ 2 j are all ﬁnite (see Assumption 5.2).
Revised: j ( j  1) ↵ 2 j are all ﬁnite (see Assumption 5.2).

Original: We will also consider the weighted Sobolev space H 2 (  ) , which contains functions f = 
Revised: We will also consider the weighted Sobolev space H 2 (  ) , which contains functions f = [Equation]

Original: P j ↵ j h j 2 L 2 (  ) such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: From random features to Hermite coefﬁcients.
Revised: Random features to Hermite coefﬁcients.

Original: To precisely characterize the landscape of L ( c, ✓ ) , we introduce notation to represent each random feature function in terms of Hermite polynomials h j .
Revised: To precisely characterize the landscape of L ( c, ✓ ) , we introduce notation to represent each random feature function in terms of Hermite polynomials h j .

Original: We deﬁne the linear integral operator T :
Revised: We deﬁne the linear integral operator T :

Original: L 2 (  ) !
Revised: L 2 (  ) !

Original: R N by
Revised: R N by

Original: [Equation]
Revised: [Equation]

Original: Note that T has rank N almost surely.
Revised: Note that T has rank N almost surely.

Original: The operator T has an adjoint T ⇤ : R N !
Revised: The operator T has an adjoint T ⇤ : R N !

Original: L 2 (  ) deﬁned as ( T ⇤ c )( u ) = 1 p N 
Revised: L 2 (  ) deﬁned as ( T ⇤ c )( u ) = 1 p N P Ni =1 c i

Original: [Equation]
Revised: [Equation]

Revised: " i u  b

Revised: [Equation]

Original:  c >  ( u ) .
Revised: = c >  ( u ) .

Original: Finally, for any j 2 N , let T j 2 R N deﬁned as T j = T h j .
Revised: Finally, for any j 2 N , let T j 2 R N deﬁned as T j = T h j .

Original: We can then write down the Hermite expansion of the student network:
Revised: We can then write down the Hermite expansion of the student network:

Original: [Equation]
Revised: [Equation]

Original: Denoting m := h ✓ , ✓ ⇤ i ,this allows us to rewrite the regularized (population) objective  as
Revised: Denoting m = h ✓ , ✓ ⇤ i , the regularized population objective can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where the term
Revised: where the term

Original: P j ↵ 2 j is a constant that can be ignored.
Revised: P j ↵ 2 j is a constant that can be ignored.

Original: Let Q :=
Revised: Let Q :=

Original: T T ⇤ 2 R N ⇥ N be a feature covariance matrix and Q  = Q +  I .
Revised: T T ⇤ 2 R N ⇥ N be a feature covariance matrix and Q  = Q +  I .

Original: Note that Q  is positive semi-deﬁnite for  > 0 .
Revised: Note that Q  is positive deﬁnite for  > 0 .

Original: We deﬁne the regularized projection ˆ P  = ˆ ⌃ (ˆ ⌃ +  I )  1 onto the random feature spaceˆ H for ˆ ⌃ =
Revised: We deﬁne the regularized projection ˆ P  = ˆ ⌃ (ˆ ⌃ +  I )  1 onto the random feature space for ˆ ⌃ =

Original: T ⇤ T .
Revised: T ⇤ T .

Original: Observe that ( ˆ P  f )( u ) =
Revised: Observe that ( ˆ P  f )( u ) =

Original: c ⇤>  ( u ) , where c
Revised: c ⇤>  ( u ) , where c

Original: ⇤ 2 R N is the solution to the following objective:
Revised: ⇤ 2 R N is the solution to the following objective:

Original: [Equation]
Revised: [Equation]

Original: Geometry on the sphere.
Revised: Geometry on the sphere.

Original: Because the direction ✓ is constrained to lie on the sphere, our optimization algorithms rely on spherical (Riemannian) gradients, which are deﬁned as follows:
Revised: Because the direction ✓ is constrained to lie on the sphere, our optimization algorithms rely on spherical (Riemannian) gradients, which are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where P ✓ ?
Revised: where ⇧ ✓ ?

Original: v = v  h ✓ , v i ✓ .
Revised: v = v  h ✓ , v i ✓ .

Original: We say that ( c, ✓ ) is a critical point of L if r S d  1 ✓
Revised: We say that ( c, ✓ ) is a critical point of L if r S d  1 ✓

Original: L ( c, ✓ ) = 0 and r c L ( c, ✓ ) = 0 .
Revised: L ( c, ✓ ) = 0 and r c L ( c, ✓ ) = 0 .

Original:  Univariate Approximation using Random Features
Revised:  Univariate Approximation using Random Features

Original: Before addressing the high-dimensionality of the learning problem, we ﬁrst focus on the nonparametric approximation aspects of the univariate link function.
Revised: Before addressing the high-dimensionality of the learning problem, we ﬁrst focus on the nonparametric approximation aspects of the univariate link function.

Original: As usual, we start by deriving approximation rates of the inﬁnitely-wide model, given by a RKHS, and then establish approximation rates for our random feature model.
Revised: As usual, we start by deriving approximation rates of the inﬁnitely-wide model, given by a RKHS, and then establish approximation rates for our random feature model.

Original: Univariate RKHS.
Revised: Univariate RKHS.

Original: If we ﬁx the direction ✓ , learning c alone may be seen as a random feature model [69] that approximates a kernel method with the following kernel:
Revised: If we ﬁx the direction ✓ , learning c alone may be seen as a random feature model [70] that approximates a kernel method with the following kernel.

Original: [Equation]
Revised: [Equation]

Original: where  ⌧ = N (0 , ⌧ 2 ) is the Gaussian measure on R with mean zero and variance ⌧ 2 , and " is a random sign with a Rademacher distribution.
Revised: where  ⌧ = N (0 , ⌧ 2 ) is the Gaussian measure on R with mean zero and variance ⌧ 2 , and " is a random sign with a Rademacher distribution.

Original: The kernel  corresponds to an RKHS H , given by
Revised: The kernel  corresponds to an RKHS H , given by

Original: [Equation]
Revised: [Equation]

Original: The following lemma characterizes the corresponding RKHS norm k · k H , and follows from Theo- rem C.8.
Revised: The following lemma characterizes the corresponding RKHS norm k · k H , and follows from The- orem B.8, by noting

Revised: that  ( u, v ) =

Revised: h ( u ) , ( v ) i L 2 (  ⌧ ) 2 , with ( u ) =

Revised: 1 p 2 (  ( u  · ) ,  (  u  · )) .

Original: Lemma 4.1 (RKHS Norm) .
Revised: Lemma 4.1 (RKHS norm) .

Original: The RKHS norm in H is given by
Revised: The RKHS norm in H is given by

Original: [Equation]
Revised: [Equation]

Original: The choice of ReLU for the activation function gives us more explicit control over the RKHS norm, based on Sobolev representations, as already exploited by several works [67, 5, 74].
Revised: The choice of ReLU for the activation function gives us more explicit control over the RKHS norm, based on Sobolev representations, as already exploited by several works [68, 5, 75].

Original: Lemma 4.2 (RKHS Sobolev Representation) .
Revised: Lemma 4.2 (RKHS norm bound) .

Original: Suppose ⌧ > 1 and f 2 H 2 (  ) has polynomial growth and satisﬁes
Revised: Let f 2 H 2 (  ) \ C 2 ( R ) and ⌧ > 1 . If f and f 0 both have polynomial growth and R [Equation]

Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original: Then f 2 H , and
Revised: | 2  ⌧ ( t ) dt < 1 , then f 2 H with

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix D.1.
Revised: The proof is in Appendix C.1.

Original: RKHS approximation.
Revised: RKHS approximation properties.

Original: Let A ( f,  ) be the (regularized)
Revised: Let A ( f,  ) be the (regularized)

Original: L 2 approximation error for functions in the space H with respect to the target function f and measure  .
Revised: L 2 approximation error for functions in the space H with respect to the target function f and measure  .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: We will now show that the approximation error of the RKHS corresponding to an inﬁnite number of random features can be bounded in terms of the regularization  and the  -norm of the second derivative of the target function.
Revised: We will now show that the approximation error of the RKHS corresponding to an inﬁnite number of random features can be bounded in terms of the regularization  and the  -norm of the second derivative of the target function.

Original: For that purpose, we ﬁrst introduce two possible regularity assumptions that are sufﬁcient to ensure a polynomial approximation error in  .
Revised: For that purpose, we consider the following ‘source’ condition to ensure a polynomial approximation error in  .

Original: Assumption 4.3 (Asymptotic smoothness) .
Revised: Assumption 4.3 (Containment in L 4 (  ) ) .

Original: For f 2 L 2 (  ) and c > 0 , ⇤ ( 
Revised: Let F = { f 2 H 2 (  ) | f 00 2 L 4 (  ) } . We assume f 2 F and deﬁne K := inf

Original: [Equation]
Revised: [Equation]

Original: where P  m is the orthogonal projection onto the harmonics of degree equal or greater than m , and Q ( c ) is a polynomial in c . We say that f is b - asymptotically smooth if there exists c 0 < 1 , b >and C such that ⇤ ( f 00 , c )  C k f 00 k 2  e  b 2 c 2 for any c  c 0 . Compactly supported functions are asymptotically smooth, as well as piece-wise smooth functions with knots ina bounded domain (such as the ReLU activation).
Revised: Assumption 4.3 provides a sufﬁcient condition for approximating f with functions in the RKHS.

Original: Another possible regularity assumption, which implies the above asymptotic smoothness, is the following hypercontractivity assumption. Assumption 4.4 (Hypercontractivity) . Given f 2 H 2 (  ) , there exists a constant C such that
Revised: The family of approximants { h M 2 H | M > 0 } we use in Lemma 4.4 are exactly equal to f on [  M, M ] and are linear outside of [  M, M ] . The L 4 assumption on f 00 ensures control over the

Original: [Equation]
Revised: RKHS norm of h M . Note that by Jensen’s inequality, L 4 (  ) ⇢ L 2 (  ) , so K is always well-deﬁned for f 00 2 L 4 (  ) .

Original: Sigmoidal functions and, more generally, functions with polynomial growth whose energy is concentrated around zero, satisﬁy the hypercontactivty assumption.
Revised: Sigmoidal functions, compactly supported smooth functions, and, more generally, functions with polynomial growth satisfy Assumption 4.3.

Original: Lemma 4.5 (RKHS approximation error, informal statement) .
Revised: Lemma 4.4 (RKHS approximation error) .

Original: Assume  < 1 and f 2 L 2 ( ) is in the weighted Sobolev space H 2 (  ) ,  andlet f =
Revised: Let  2 (0 , 1) and f 2 F . Then, there exists a universal constant C > 0 such that

Original: P 1 j =0 ↵ j h j be the Hermite expansion of f .

Original: If f is sufﬁciently regular, then there exists a constant  2 (0 , 1) depending only on ⌧ and the regularity properties of f such that

Original: [Equation]
Revised: [Equation]

Original: See Lemma D.1 for a formal statement.
Revised: where  = 1  1 / ⌧ 2 3+1 / ⌧ 2 and C f = max { k f k  , k f 0 k  , k f 00 k  } .

Original: Its proof appears in Appendix D.2.
Revised: The proof appears in Appendix C.2.

Original: This lemma allows us to control the RKHS approximation error of a target function in terms of their Hermite decompositions.
Revised: This lemma allows us to control the RKHS approximation error of a target function in terms of their Hermite decompositions.

Original: The main technical difﬁculty is that the RKHS integral operator ⌃ does not diagonalise in the Hermite basis; we address this with a dedicated argument exploiting the RKHS Sobolev representation of Lemma 4.2.
Revised: The main technical difﬁculty is that the RKHS integral operator ⌃ does not diagonalise in the Hermite basis; we address this with a dedicated argument exploiting the RKHS Sobolev representation of Lemma 4.2.

Original: The O and e
Revised: The assumption that f 00 2 L 4 (  ) (Assumption 4.3) is sufﬁcient for our purposes but not necessary for polynomialin-  approximation rates. In Section H, we show that the ReLU function  ( t ) = max(0 , t ) , which is Lipschitz but not in H 2 (  ) , as the target satisﬁes A (  ,  ) . ⌧ 2  2 / 3 using a direct argument.

Original: O terms in the RHS hide the constants appearing in the regularity assumptions of f (Assumptions 4.3 and 4.4).
Revised: Extending the class of functions approximable by H with polynomial-in-  rate is an interesting future direction.

Original: Random feature approximation.
Revised: Random feature approximation.

Original: We now consider (ﬁnite) random feature approximations to functions in the RKHS.
Revised: We now consider (ﬁnite) random feature approximations to functions in the RKHS.

Original: Lemma D.4 shows that the best possible loss of a linear combination of sufﬁciently many ﬁnite features is bounded above by the best approximation with inﬁnitely many features with high probability.
Revised: Lemma C.2 shows that the best possible loss of a linear combination of sufﬁciently many ﬁnite features is bounded above by the best approximation with inﬁnitely many features with high probability.

Original: That is, as long as N &   1 , the random feature approximation error behaves like the RKHS approximation error.
Revised: More speciﬁcally, as long as N &   1 , the random feature approximation error behaves like the RKHS approximation error.

Original: The proof leverages the ‘degrees of freedom’ of the kernel and closely tracks [6].
Revised: The proof leverages the ‘degrees of freedom’ of the kernel and closely tracks [6].

Original: In this context, the zero-mean assumption E [ f ] = 0 is necessary to obtain a tight enough bound of these degrees of freedom.

Original:  Population Landscape under Frozen Random Biases
Revised:  Population Landscape under Frozen Random Biases

Original: To understand optimization and generalization properties of gradient ﬂow on the empirical loss
Revised: To understand optimization and generalization properties of gradient ﬂow on the empirical loss

Original: L n ( c, ✓ ) , we ﬁrst the study optimization landscape of the population loss L ( c, ✓ ) .
Revised: L n ( c, ✓ ) , we ﬁrst the study optimization landscape of the population loss L ( c, ✓ ) .

Original: We characterize the critical points of the population loss and show that gradient ﬂow on a shallow neural network of sufﬁcient width N converges only when its direction vector ✓ is either parallel or orthogonal to the target direction ✓ ⇤ .
Revised: We characterize the critical points of the population loss and show that gradient ﬂow on a shallow neural network of sufﬁcient width N converges only when its direction vector ✓ is either parallel or orthogonal to the target direction ✓ ⇤ .

Original: Importantly, the sufﬁcient number of random features N depends on the ` 2 -regularization parameter  2 (0 , 1) ,the variance ⌧ 2 > 1 of the random biases, and the target link function f ⇤ , but not on the input dimension d .
Revised: Importantly, the sufﬁcient number of random features N depends on the ` 2 -regularization parameter  2 (0 , 1) , but not on the input dimension d .

Original: In Section 6, we further show that sufﬁciently large n , the number of training samples, guarantees similar properties for the empirical landscape and thus has favorable generalization properties for most initializations.
Revised: In Section 6, we further show that sufﬁciently large n , the number of training samples, guarantees similar properties for the empirical landscape and thus has favorable generalization properties for most initializations.

Original: One of the main measures of complexity for the target link function f
Revised: One of the main measures of complexity for the target link function f

Original: ⇤ is its information exponent (see e.g., [10]), deﬁned as follows.
Revised: ⇤ is its information exponent (see e.g., [10]), deﬁned as follows.

Original: Deﬁnition 5.1 (Information exponent) .
Revised: Deﬁnition 5.1 (Information exponent) .

Original: Let f : R !
Revised: Let f : R !

Original: R be any function such that f 2 L 2 (  ) .
Revised: R be any function such that f 2 L 2 (  ) .

Original: The information exponent of f , which we denote by s , is the index of the ﬁrst non-zero Hermite coefﬁcient.
Revised: The information exponent of f , which we denote by s , is the index of the ﬁrst non-zero Hermite coefﬁcient.

Original: That is, s :=
Revised: That is, s :=

Original: min { j 2 N : ↵ j 6 = 0 } .
Revised: min { j 2 N : ↵ j 6 = 0 } .

Original: We make the following regularity assumptions on the target link function f ⇤ to ensure small approximation error by random features, and benign population and empirical landscape.
Revised: We make the following regularity assumptions on the target link function f ⇤ to ensure small approximation error by random features, and benign population and empirical landscape.

Original: Assumption 5.2 (Regularity of f ⇤ ) .
Revised: Assumption 5.2 (Regularity of f ⇤ ) .

Original: We consider f ⇤ 2 L 2 (  )  with f ⇤ =
Revised: We consider f ⇤ 2 L 2 (  ) , with f ⇤ =

Original: P j ↵ j h j .
Revised: [Equation] j .

Original: Assume f
Revised: Assume 1)

Original:  ⇤ is Lipschitz, has higher-order derivatives f ( p ) ⇤ for any p 2 { 1 , 2 , 3 , 4 } which are continuous and satisfy k f ( p )
Revised: f ⇤ is Lipschitz, 2) [Equation]

Original: ⇤ k  < 1 , and that either asymptotic smoothness (Assumption 4.3)or hypercontractivity (Assumption 4.4) holds.
Revised: Assumption 4.3)

Original: We also assume f ⇤ is normalized so that k f ⇤ k  = 1 .
Revised: We also suppose w.l.o.g. that f ⇤ is normalized so that k f ⇤ k  = 1 .

Original: To analyze the critical points of L ( c, ✓ ) , we introduce the projected population loss ¯ L ( ✓ ) , which can be seen as a semiparametric least squares (SLS) objective [47].
Revised: To analyze the critical points of L ( c, ✓ ) , we introduce the projected population loss ¯ L ( ✓ ) , which can be seen as a semiparametric least squares (SLS) objective [48].

Original: [Equation]
Revised: [Equation]

Original: Theorem 5.3 (Critical points of the population loss) .
Revised: Theorem 5.3 (Critical points of the population loss) .

Original: Assume f ⇤ satisﬁes Assumption 5.2 and has information exponent s  1 .
Revised: Assume f ⇤ satisﬁes Assumption 5.2 and has information exponent s  1 .

Original: For ⌧ > 1 , and  2 (0 , 1) , there exist explicit constants C 1 , C 2 >depending only on ⌧ and the target link function f {+⇤ and a universal constant C > 
+}
Revised: For ⌧ > 1 , and  2 (0 , 1) , there exists  ⇤  1 depending only on ⌧ and the target link function f {+⇤ and a universal constant C > 
+}

Original: ⇤ such that if
Revised: such that if

Original: [Equation]
Revised: [Equation]

Original: <

Original: C

Original: [Equation]
Revised: [Equation]

Original: and [Equation] then with probability 1   over the random biases b j and signs " j , j = 1 . . .
Revised: then with probability 1   over the random biases b j and signs " j , j = 1 . . .

Original: N , the set of ﬁrst-order critical points C L := { ( c, ✓ ) : r S d  1 ✓
Revised: N , the set of ﬁrst-order critical points ⌦ := { ( c, ✓ ) : r S d  1 ✓

Original: [Equation]
Revised: [Equation]

Original: satisﬁes:

Original: (orientation relative to ✓ ⇤)
Revised: (orientation relative to ✓ ⇤

Original:  if ( c, ✓ ) 2 C L , then either ✓ 2 {± ✓ ⇤ } or h ✓ , ✓ ⇤ i = 0 ; and 2.
Revised: ) if ( c, ✓ ) 2 ⌦ , then either ✓ 2 {± ✓ ⇤ } or h ✓ , ✓ ⇤ i = 0 .

Original: (existence and uniqueness of c ) if r S d  1 ✓ ¯ L ( ✓ ) = 0 , then there exists a unique c 2 R N such that ( c, ✓ ) 2 C L .
Revised: (existence and uniqueness of c ) if r S d  1 ✓ ¯ L ( ✓ ) = 0 , then there exists a unique c 2 R N such that ( c, ✓ ) 2 ⌦ .

Original: Theorem 5.3 thus establishes a benign optimization landscape in the population limit, rejoining several known non-convex objectives with similar behavior, such as tensor decomposition [40] or matrix completion [38].
Revised: Theorem 5.3 thus establishes a benign optimization landscape in the population limit, rejoining several known non-convex objectives with similar behavior, such as tensor decomposition [40] or matrix completion [38].

Original: Importantly, this optimization landscape has the same topology as the one that arises from using the Hermite basis, the tailored choice for data generated by a single-index model in Gaussian space [33, Theorem 5], instead of random scalar features.
Revised: Importantly, this optimization landscape has the same topology as the one that arises from using the Hermite basis, the tailored choice for data generated by a single-index model in Gaussian space [33, Theorem 5], instead of random scalar features.

Original: We view this as an interesting robustness property of shallow neural networks, at least in the regime where biases are randomly frozen.
Revised: We view this as an interesting robustness property of shallow neural networks, at least in the regime where biases are randomly frozen.

Original:  Empirical Landscape and Generalization Guarantees
Revised:  Empirical Landscape and Generalization Guarantees

Original: Now that we have established that the population landscapeL ( c, ✓ ) has a relatively simple structure given N  O d (1) random features, we study the optimization properties of its ﬁnite-sample counterpart
Revised: Section 5 shows that the population landscape has a relatively simple structure given N = ⌦ d (1) random features. We now study the optimization properties of its ﬁnite-sample counterpart.

Original: Wespeciﬁcally consider the estimator ˆ F ( x ) :
Revised: We consider the estimator ˆ F ( x ) :

Original: = ˆ f ( h x, ˆ ✓ i ) , where ( ˆ f, ˆ ✓ ) are obtained by using gradient descent on c and ✓ to minimize L n ( c, ✓ ) .
Revised: = ˆ f ( h x, ˆ ✓ i ) , where ( ˆ f, ˆ ✓ ) are obtained by running gradient ﬂow on c and ✓ to minimize L n ( c, ✓ ) .

Original: Such strategy appears to be reasonable, in light of the properties of the population landscape studied in Section 5, namely that itsonly local minimisers are also global, and correspond to ˜ f
Revised: Such strategy appears to be reasonable in light of the properties of the population landscape, since its local minimizers are also global and correspond to ˜ f

Original: = ˆ P  f ⇤ and ˜ ✓ = ✓ ⇤ .
Revised: = ˆ P  f ⇤ and ˜ ✓ = ✓ ⇤ (Theorem 5.3).

Original: Given a sufﬁciently large sample size n , one expects the empirical landscape L n to concentrate around its expectation L and inherit its benign optimization properties.
Revised: For a sufﬁciently large sample size n , one expects the empirical landscape L n to concentrate around its expectation L and inherit its benign optimization properties.

Original: However, the presence of a degenerate saddle at ( c, m ) =
Revised: However, the presence of a degenerate saddle at ( c, m ) =

Original: (0 , 0) for m =
Revised: (0 , 0) for m =

Original: h ✓ , ✓ ⇤ i ﬂattens the landscape around the “equator” { ✓ : h ✓ , ✓ ⇤ i = 0 } , and a sufﬁciently large sample complexity is ultimately required to ensure that gradient descent iterates escape from it, given its dangerously close  initialization | h ✓ 0 , ✓ ⇤ i | = ⇥ (1 / p d ) .
Revised: h ✓ , ✓ ⇤ i ﬂattens the landscape around the “equator” { ✓ : h ✓ , ✓ ⇤ i = 0 } . Thus, more samples are required to ensure that gradient ﬂow escapes from the equatorial region despite its dangerously close random initialization | h ✓ 0 , ✓ ⇤ i | = ⇥ (1 / p d ) .

Original: Prior works have obtainedan optimal sample complexity of n =
Revised: Prior works have obtained sample complexity of n =

Original: O ( d s ) (recall that s is the information exponent of the teacher function f ⇤ ) for recoveringthe teacher direction ✓ ⇤ either by employing a learning algorithm that explicitly learns individual harmonic polynomials [33] or by assuming that f ⇤ is known a priori [10] 1 .
Revised: O ( d s ) , where we recall that s is the information exponent of the target function f ⇤ , for recovering ✓ ⇤ either by employing a learning algorithm that explicitly learns individual Hermite polynomials [33] or by assuming that f ⇤ is known a priori [10].

Original: The intuitive argument goes roughly as follows:
Revised: The intuition behind this sample complexity is roughly as follows.

Original: • The empirical optimization landscape (when regarded as a function only of the direction ✓ ) near the equator ( | m | ⌧ 1 ) is of the form L ( ✓ ) ⇣ m s .
Revised: • The empirical optimization landscape (when regarded as a function only of the direction ✓ ) near the equator ( | m | ⌧ 1 ) is of the form L ( ✓ ) ⇣ m s .

Original: • To certify that the optimization  does not converge to a suboptimal critical pointon the equator (i
Revised: • In order to certify that the optimization algorithm does not converge to a suboptimal critical point (i

Original: e. the gradient steps are sufﬁciently large, or kr L ( ✓ ) k = ⌦ ( ✏ ) ) one requires that m = ⌦ ( ✏ 1 / ( s  1)) .
Revised: e., kr L ( ✓ ) k  ✏ ) on the equator, one requires that m  ✏ 1 / ( s  1) .

Original: • A uniform concentration argument of the form kr L ( ✓ ) r L n ( ✓ ) k
Revised: • A uniform gradient convergence bound of the form kr L ( ✓ )  r L n ( ✓ ) k

Original: = O
Revised: = O

Original: ( p d/n ) and the fact that m = ⇥ ( d  1 / 2 ) at initialization together imply
Revised: ( p d/n ) and the fact that m = ⇥ (1 / p d ) at initialization together imply

Original: that n = O ( d s ) samples are sufﬁcient to escape from the “inﬂuence” of the equator.
Revised: that n = O ( d s ) samples are sufﬁcient to escape from the “inﬂuence” of the equator.

Original: In order to repurpose these arguments to our setting, the relative scaling of the top-layer weights c relative to the direction vector ✓ will turn out to be crucial, as has also been observed in the literature on lazy -vs- rich regimes [23, 83] in the context of overparametrized neural networks.
Revised: In order to repurpose these arguments to our setting, the relative scaling of the top-layer weights c relative to the direction vector ✓ is crucial, as has also been observed in the literature on lazy -vs- rich regimes [23, 84] in the context of overparametrized neural networks.

Original: We consider an idealized version ofa Gradient Descentalgorithm over the empirical loss L n in the inﬁnitesimally small learning rate regime.
Revised: We consider an idealized version of Gradient Descent over the empirical loss L n in the inﬁnitesimally small learning rate regime.

Original: This results in a gradient ﬂow ODE of the form:
Revised: This results in a gradient ﬂow ODE of the form:

Original: [Equation]
Revised: [Equation]

Original: where ⇣ is the relativetime scale between ﬁrst and second layer weights
Revised: where ⇣ is the relative scale between c and ✓ , and r S d  1 ✓ is the Riemannian gradient.

Original: Speciﬁcally, we study a setting where ⇣ ( t ) = 1 ( t > T 0 ) for an appropriately chosen time T 0 .
Revised: Speciﬁcally, we study a setting where ⇣ ( t ) = 1 ( t > T 0 ) for an appropriately chosen time T 0 .

Original: This choice produces a two-stage gradient-ﬂow.
Revised: This choice produces a two-stage gradient-ﬂow.

Original: During the ﬁrst phase, up until time T 0 , we only optimize the ﬁrst-layer parameter ✓ from a random initialization.
Revised: During the ﬁrst phase, up until time T 0 , we only optimize the ﬁrst-layer parameter ✓ from a random initialization.

Original: In the second phase, the parameters c and ✓ are jointly optimized.
Revised: In the second phase, the parameters c and ✓ are jointly optimized.

Original: Additionally, the ﬁrst phase only utilizes a small fraction N 0 N ⌧of the random features employed in the second phase.
Revised: Additionally, the ﬁrst phase only utilizes a small fraction N 0 N ⌧of the random features employed in the second phase.

Original: Our procedure implements this by randomly initializing c (0) 2 R N as a sparse vector with N 0 non-zero components.
Revised: Our procedure implements this by randomly initializing c (0) 2 R N as a sparse vector with N 0 non-zero components.

Original: The overall approach is described in Procedure 1.
Revised: The overall approach is described in Procedure 1.

Original: Our main result, proved in Appendix F, establishes that this gradient ﬂow efﬁciently ﬁnds an approximate minimiser of the population loss, with an error (explicitly quantiﬁed as a function of n )  that reveals the fundamental role of the information exponent s of f ⇤ .
Revised: Our main result, proved in Appendix E, establishes that this gradient ﬂow efﬁciently ﬁnds an approximate minimizer of the population loss, with an error (explicitly quantiﬁed as a function of n ) e that reveals the fundamental role of the information exponent s of f ⇤ .

Original: On top of the regularity conditions on the target link function of Assumption 5.2, the upper bound on  , and the lower bound onN from
Revised: On top of the regularity conditions on the target link function of Assumption 5.2, the upper bound on  , and the lower bound on

Original:  Theorem 5.3 (eq. 9), the main result imposes a (compatible) upper bound on N and an appropriate choice of initial norm ( ⇢ ) and sparsity ( N 0 /N ) for c (0) .
Revised: N from Eq. (9) of Theorem 5.3, the main result imposes a (compatible) upper bound on N and an appropriate choice of initial norm ( ⇢ ) and sparsity ( N 0 /N ) for c (0) .

Original: In this section, we are interested in behavior as n , d , and N grow asymptotically and hence treat the teacher function f ⇤ and terms derived from it (including Hermite coefﬁcients ↵ j and information exponent s ) along with the bias parameter ⌧ and regularity parameter   as constants and omit them from asymptotic notation.
Revised: In this section, we are interested in behavior as n , d , and N grow asymptotically and hence treat the target function f ⇤ and terms derived from it (including Hermite coefﬁcients ↵ j and information exponent s ), along with the bias parameter ⌧ and regularity parameter  , as constants and omit them from asymptotic notation.

Original: Theorem 6.1 (Gradient ﬂow ﬁnds approximate minimizers) .
Revised: Theorem 6.1 (Gradient ﬂow ﬁnds approximate minimizers) .

Original: For  > 0 and f ⇤ satisfying Assumption 5.2, suppose the following are true: (i) 
Revised: For  2 (0 , 1 / 4) and f ⇤ satisfying Assumption 5.2, suppose the following are true: (i)

Original: = O (1) and 
Revised: = O (1) and

Original: = ⌦ ( p  crit ) , where  crit := max {
Revised: = ⌦ ( p  crit ) , where  crit := max {

Original: [Equation]
Revised: [Equation]

Original: s +3) / 2 } ) , (iii) N = ⌦ [Equation] 1  log 1  ) & N = e
Revised: (vi)

Original: O (    1 crit ) , (iv) N 0 = ⇥ (log( 1  )) , (v) ⇢ = ⇥

Original: [Equation]
Revised: [Equation]

Original: p NN

Original: [Equation]

Original: + s ) /

Original: [Equation]

Original: [Equation]

Original: 2 +  N/N 0 )  1 ) ,

Original: [Equation]

Original: vi) T 0 = ˜ ⇥ ( d s/ 2  1 ) , and (vii) T 1 = ˜ ⇥
Revised: ˜ ⇥ ( d s/ 2  1 ) , and (vii) T 1 = ˜ ⇥

Original: (  4 n d + N ) .
Revised: (  4 n d + N ) .

Original: Then, if we run Procedure 1 for T = T 

Revised: Then, if we run Procedure 1 for T = T

Original: + T 1 time steps with the above parameters, with probability at least 12   we have
Revised: + T 1 time steps with the above parameters, with probability at least 12   we have

Original: [Equation]
Revised: [Equation]

Original: The empirical gradient ﬂow therefore escapes the inﬂuence of the degenerate saddle with sample complexity n = ˜ ⇥ ( d s ) when  = ⇥ (1) and s > 2 .
Revised: The empirical gradient ﬂow therefore escapes the inﬂuence of the degenerate saddle with sample complexity n = ˜ ⇥ ( d s ) when  = ⇥ (1) and s > 2 .

Original: This is an instance of gradient descent successfully optimizing a non-convex objective without the strict saddle property as soon as s > 2 , building from the simpler optimization landscapes of [33, 10].
Revised: This is an instance of gradient ﬂow successfully optimizing a non-convex objective without the strict saddle property as soon as s > 2 , building from the simpler optimization landscapes of [33, 10].

Original: This is in contrast withe.g. spiked tensor recovery problems [12] where the signal strength is substantially weaker, leading to complexity in the optimization landscape.
Revised: This is in contrast, for example, with spiked tensor recovery problems [12] where the signal strength is substantially weaker, leading to complexity in the optimization landscape.

Original: This sample complexity nearly matches the optimal n  d s  1 of [10], obtained in the case s > 2 and using SGD rather than batch gradient descent as is our case.
Revised: This sample complexity nearly matches the tight lower bound n  d s of [10], obtained in the case s > 2 and applies to SGD rather than batch gradient descent, as is our case.

Original: For s 2 { 1 , 2 } , the sample complexity becomes d 2 and d 2 . 5 , respectively, but we note that these may be improved to d s when using a smooth activation (see Appendix H).
Revised: For s 2 { 1 , 2 } , the sample complexity becomes d 2 and d 2 . 5 , respectively, but we note that these may be improved to d s when using a smooth activation (see Appendix G).

Original: For s = 2 , this is comparable to [27], which requires ⌦ ( d 2 ) samples, but still above the n  d log d of [10].
Revised: For s = 2 , this is comparable to [27], which requires ⌦ ( d 2 ) samples, but still above the n  d log d of [10].

Original: We emphasize that the “near-optimality” of our sample complexity n = ˜ ⇥ ( d s ) only concerns gradient-based methods, infact only small learning rate regimes; e.g. [10].
Revised: We emphasize that the “near-optimality” of our sample complexity n = ˜ ⇥ ( d s ) only pertains to gradient-based methods in small learning rate regimes [10].

Original: In fact, alternative methods have been shown to obtain better sample complexity of e
Revised: In fact, alternative methods have been shown to achieve a better sample complexity of e

Original: O ( d d s/ 2 e ) in the setting where f ⇤ is a certain degree- s polynomial with information exponent s [19], leveraging tensor factorization tools.
Revised: O ( d d s/ 2 e ) in the setting where f ⇤ is a certain degrees polynomial with information exponent s [19], leveraging tensor factorization tools.

Original: We leave it as an interesting open question to further understand the nature of this gap.
Revised: We leave it as an interesting open question to further understand the nature of this gap.

Original: We note that the dependence on d + N in the recovery guarantee can likely be improved to d using a more reﬁned norm-based landscape concentration analysis.
Revised: We note that the dependence on d + N in the recovery guarantee can likely be improved to d using a more reﬁned norm-based landscape concentration analysis.

Original: We also remark that if we chose the number of random features N = ⇥ ( 1  log 1  ) , then the requirement  = ⌦ ( p  crit ) for large enough sample size n imposes   n  1 / 5 .
Revised: We also remark that if we chose the number of random features N = ⇥ ( 1  log 1  ) , then the requirement  = ⌦ ( p  crit ) for large enough sample size n imposes   n  1 / 5 .

Original: This lower bound on  guarantees that critical points near initialization can be escaped, but may slow down learning.
Revised: This lower bound on  guarantees that critical points near initialization can be escaped, but may slow down learning.

Original: Nevertheless, this is sufﬁcient to obtain an excess risk that vanishes with n , with a rate independent ofthe dimension d .
Revised: Nevertheless, this is sufﬁcient to obtain an excess risk that vanishes with n , with a rate independent of d , as we now show.

Original: Corollary 6.2 (Excess risk of Algorithm 1) .
Revised: Corollary 6.2 (Excess risk of Algorithm 1) .

Original: Under the assumptions of Theorem 6.1, and further assuming n & d 3 , the choice of ⇠ n  1 / (  +5) yields an excess risk guarantee of the form
Revised: Under the assumptions of Theorem 6.1, and further assuming n & d 3 , an appropriate choice of  yields an excess risk guarantee of the form

Original: [Equation]
Revised: [Equation]

Original: where  is deﬁned from Lemma D.1 according to the regularity assumptions of f ⇤ . Since N = e ⌦ (   1 ) , this result indicates thatone needs to increase the width of the network to
Revised: where  is deﬁned as in Lemma 4.4. This result indicates that the joint training is consistent, with excess risk that vanishes with a rate with explicit dependence on the ambient dimension d and the non-parametric exponent  . However, it requires a regularisation strength  = ⇥

Original: N = e ⌦ ( n 1 / (  +5) )
Revised: [Equation]

Original:  d s/ (1+  ) in order to achieve vanishing excess error. With the joint training we
Revised: to ensure enough gradient concentration, which happens to be larger than the optimal regularisation of the univariate kernel ridge regression associated with learning f ⇤ . We are thus ‘over-regularising’ as a consequence of the joint training, resulting in a slower rate than what would be dictated by estimating ✓ ⇤ and f ⇤ separately. A simple mechanism to break this inefﬁciency is by considering a ﬁne-tuning step of the second-layer terms.

Original: Procedure
Revised: Procedure

Original: Require: N 0 , ⇢ , T 0 , T 1 , N,  
Revised: Require: N 0 , ⇢ , T 0 , T 1 , N , and  .

Original: Initialize ✓ (0)
Revised: Initialize ✓ (0)

Original: ⇠ Unif( S d  1 ) , c (0) ⇠ Unif( { c 2 R N ; k c k 2 = ⇢ ; k c k 0 = N 0 } ) .
Revised: ⇠ Unif( S d  1 ) , c (0) ⇠ Unif( { c 2 R N ; k c k 2 = ⇢ ; k c k 0 = N 0 } ) .

Original: Run Gradient Flow (10) with ⇣ ( t ) = 1 ( t > T 0 ) up to time T
Revised: Run Gradient Flow (10) with ⇣ ( t ) = 1 ( t > T 0 ) up to time T

Original: = T 0 + T 1 .
Revised: = T 0 + T 1 .

Original: Set ˆ ✓ = ✓ ( T ) , ˆ c = c ( T ) .
Revised: Set ˆ ✓ = ✓ ( T ) , ˆ c = c ( T ) .

Original: Procedure
Revised: Procedure

Original: Require: n 0 , ˆ ✓ from Algorithm 1.
Revised: Require: ˆ ✓ from Procedure 1, and  n 0 .

Original: Set ˆ c = arg
Revised: Set ˆ c = arg

Original: min c L 0 n 0 ( c, ˆ ✓ ) as in (13).
Revised: min c L 0 n 0 ( c, ˆ ✓ ) as in (13).

Original: are thus paying a dependency in the ambient dimension in terms of width, yet this width increase is driven by the non-parametric approximation error of f ⇤ , which is itself independent of d .

Original: A simple mechanism to break this inefﬁciency is by considering a ﬁne-tuning step of the second-layer terms.

Original: Fine-tuning the second layer.
Revised: Fine-tuning the second layer.

Original: After running Algorithm 1, we may include a ﬁnal ﬁne-tuning phase of training for second layer weights c alone, using a separate training sample ( x 0 i , y 0 i ) , i = 1 , . . . , n 0 and a possibly different regularization parameter  n 0 .
Revised: After running Algorithm 1, we may include a ﬁnal ﬁne-tuning phase of training for second layer weights c alone, using a separate training sample ( x 0 i , y 0 i ) , i = 1 , . . . , n 0 and a possibly different regularization parameter  n 0 .

Original: More precisely, we set 
Revised: More precisely, we set ˆ

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Original: where ˆ ✓ denotes the output of the previous gradient descent phase.
Revised: where ˆ ✓ denotes the output of the previous gradient descent phase.

Original: Note that this is a strongly convex optimization problem, and can thus be optimized efﬁciently using gradient methods or by solving a linear system.
Revised: Note that this is a strongly convex optimization problem, and can thus be optimized efﬁciently using gradient methods or by solving a linear system.

Original: While this may not be needed in practice, we use a different training sample for technical reasons, namely to break the dependence between the data and the kernel, which depends on the initial training sample through ˆ ✓ .
Revised: While this may not be needed in practice, we use a different training sample for technical reasons, namely to break the dependence between the data and the kernel, which depends on the initial training sample through ˆ ✓ .

Original: We note that such sample splitting strategies are commonly used in other contexts in the statistics literature (e.g., [15, 21]).
Revised: We note that such sample splitting strategies are commonly used in other contexts in the statistics literature (e.g., [15, 21]).

Original: We obtain the following guarantee.
Revised: We obtain the following guarantee.

Original: Proposition 6.3 (Excess risk of ﬁne-tuning) .
Revised: Proposition 6.3 (Excess risk of ﬁne-tuning) .

Original: Let  > 0 .
Revised: Let  2 (0 , 1 / 4) .

Original: Let m =
Revised: Let m =

Original: hˆ ✓  ,  ✓⇤ i , where ˆ ✓ is obtained from the previous gradient descent phase, and let ˆ c be the ridge regression estimator obtained from a fresh dataset D 0 of n 0 samples, N random features,2 and regularization parameter  n 0 :=
Revised: h ✓ ⇤ , ˆ ✓ i , where ˆ ✓ is obtained from the previous gradient descent phase, and let ˆ c be the ridge regression estimator obtained from a fresh dataset D 0 of n 0 samples, N random features, and regularization parameter  n 0 :=

Original: (  2 ⌧ 2 / k f 

Revised: (  2 ⌧ 2 / k f

Original: ⇤ k 2  n 0 ) 1 / (  +1) , and let ˆ F ( x ) =
Revised: ⇤ k 2  n 0 ) 1 / (  +1) , and let ˆ F ( x ) =

Original: ˆ c >  ( h ˆ ✓ , x i ) .
Revised: ˆ c >  ( h ˆ ✓ , x i ) .

Original: Assume
Revised: Assume

Original: [Equation]
Revised: [Equation]

Original: , and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then with probability at least 1   over the random features, we have
Revised: Then with probability at least 1   over the random features, we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over the n 0 fresh samples, and is conditioned on the previously obtained ˆ ✓ .
Revised: where the expectation is over the n 0 fresh samples, and is conditioned on the previously obtained ˆ ✓ .

Original: Decoupling the regularization parameters of the two phases (along with number of random features N ) allows us to keep a large  in the ﬁrst phase, leading to fast recovery as per Theorem 6.1, while obtaining vanishing excess risk through a decreasing  n 0 .
Revised: Decoupling the regularization parameters of the two phases (along with number of random features N ) allows us to keep a large  in the ﬁrst phase, leading to fast recovery as per Theorem 6.1, while obtaining vanishing excess risk through a decreasing  n 0 .

Original: This is illustrated in the result on the excess risk for Algorithm 2.
Revised: This is illustrated in the result on the excess risk for Algorithm 2.

Original: By comparing Corollaries 6.2 and 6.4, we observe that the ﬁne-tuning stage recovers the optimal sample complexity, where the non-parametric rate no longer depends on the ambient dimension d .

Original: Corollary 6.4 (Excess risk of Algorithm 2) .
Revised: Corollary 6.4 (Excess risk of Algorithm 2) .

Original: Let  > 0 .
Revised: Let  2 (0 , 1 / 4) .

Original: As in Theorem 6.1, assume d  C log(   1 ) , µ s = h h s , ⌃ h
Revised: As in Theorem 6.1, let µ s = h h s , ⌃ h

Original: s i > 0 , and assumption 5.2 on f ⇤ with a constant C f ⇤ . Let 
Revised: s i > 0 , and let f ⇤ satisfy Assumption 5.2.

Original: = ( s 2 ↵ 2 s / 2 C f ⇤ ) 2 /   , and assume the following on the sample sizes and number of random features for the ﬁrst phase ( n, N, N 0 ) and ﬁne-tuning phase ( n 0 , N 0 ):
Revised: Let  = ⇥ (1) , and assume the following on the sample sizes and number of random features for the ﬁrst phase ( n, N, N 0 ) and ﬁne-tuning phase ( n 0 , N 0 ):

Original: [Equation]
Revised: [Equation]

Original: and let ⇢ be as in Theorem 6.1.
Revised: and let ⇢ be as in Theorem 6.1.

Original: With probability at least 1 / 2  2  over the initial n samples, initialization, random features, we have
Revised: With probability at least 1 / 2  2  over the initial n samples, initialization, random features, we have

Original: [Equation]
Revised: [Equation]

Original: where the constants in e
Revised: where the constants in e

Original: O do not depend on d other than through logarithmic factors.
Revised: O do not depend on d other than through logarithmic factors.

Revised: By comparing Corollaries 6.2 and 6.4, we observe that the ﬁne-tuning stage recovers the optimal sample complexity, where the non-parametric rate is fully independent of the ambient dimension d , while in Corollary 6.2 there is still a dependence in the constants.

Original: We make the following additional remarks:
Revised: We make the following additional remarks:

Original: • The time-scale separation scheduling for ⇣ in Theorem 6.1 is sufﬁcient but possibly not necessary.
Revised: • The time-scale separation schedule for ⇣ in Theorem 6.1 is sufﬁcient but possibly not necessary.

Original: The analysis of vanilla dynamics ( ⇣ ( t ) ⌘ 1 ) is challenging, since during the initial phase of training there may be adverse interactioneffects between c and ✓ , which under naive analysis lead to suboptimal sample complexity of n  O ( d 2 s ) .
Revised: The analysis of vanilla dynamics ( ⇣ ( t ) ⌘ 1 ) is challenging, since during the initial phase of training there may be adverse interaction between c and ✓ , which under naive analysis lead to sub-optimal sample complexity of n  O ( d 2 s ) .

Original: Observe that this separate analysis of ‘weak’ and ‘strong’ recovery phases of learning appears in most contemporary related work [27, 1, 4, 10, 11].
Revised: Observe that this separate analysis of ‘weak’ and ‘strong’ recovery phases of learning appears in most contemporary related work [27, 1, 4, 10, 11].

Original: • The time discretisation to turn Procedure 1 into a proper algorithm should follow from standard time discretisation arguments, although the case   = ReLU requires special care due to the nonsmoothness of the loss.
Revised: • The time discretization to turn Procedure 1 into a proper algorithm should follow from standard time discretization arguments, although the case where  = ReLU requires special care due to the non-smoothness of the loss (see Appendix F for further discussion).

Original: In such setting, such discretisation arguments do not hold for vanilla gradient descent in the worst-case [50], although these may be recovered by  smoothing the objective prior to computing the gradient, or by using instead a smooth activation function (see Appendix H)
Revised: In such setting, such discretization arguments do not hold for vanilla gradient descent in the worst-case [51], although these may be recovered by appropriately smoothing the objective prior to computing the gradient, or by using instead a smooth activation function (see Appendix G).

Original:  Conclusions
Revised:  Conclusion

Original: This work studies the ability of shallow neural networks to learn single-index models with gradient descent.
Revised: This work studies the ability of shallow neural networks to learn single-index models with gradient descent.

Original: Our main results are positive, and demonstrate their ability to solve a semi-parametric problem with nearly optimal guarantees.
Revised: Our main results are positive, and demonstrate their ability to solve a semi-parametric problem with nearly optimal guarantees.

Original: Interestingly, this success story combines elements from the feature-learning regime,  the ability to efﬁciently identify the hidden direction in high-dimensions under a non-convex objective, and the lazy-regime, which offer better computational tradeoffs to solve low-dimensional non-parametric problems.
Revised: Interestingly, this success story combines elements from the feature-learning regime, i.e., the ability to efﬁciently identify the hidden direction in highdimensions under a non-convex objective, with ingredients from the lazy-regime, which offer better computational tradeoffs to solve low-dimensional non-parametric problems.

Original: Our technical analysis leverages tools from highdimensional probability (such as uniform gradient concentration) and RKHS approximation theory, and complements the growing body of theoretical work certifying the efﬁciency of gradient methods on non-convex objectives.
Revised: Our technical analysis leverages tools from high-dimensional probability (such as uniform gradient concentration) and RKHS approximation theory, and complements the growing body of theoretical work certifying the efﬁciency of gradient methods on non-convex objectives.

Original: We have followed the standard approach of ﬁrst establishing benign topological properties of the population loss, and then extend them to the empirical loss.
Revised: We have followed the standard approach of ﬁrst establishing benign topological properties of the population loss, and then extending them to the empirical loss.

Original: Nevertheless, there are  several  questions that our work has not addressed, such as
Revised: There are nonetheless several unanswered questions that our work has not addressed.

Revised: We provide a list of potentially interesting future directions below.

Revised: Weaker regularity and discrete-time analysis.

Original: • Extension to Multi-Index Models.
Revised: Our approximation rate for ReLU as the target (see Appendix H) suggests that the polynomial-in-  approximation rate may be extended to function classes beyond F ⇢ H 2 (  ) , such as Lipschitz functions with smooth tail behavior. Thus, it would be interesting to extend our empirical landscape concentration results to such functions satisfying weaker regularity assumptions, which currently rely on certain polynomial decay of the Hermite coefﬁcients (see Assumption 5.2).

Revised: Additionally, by using a smooth activation function (see

Revised: Appendix G)

Revised: , our Gradient Flow dynamics can be discretized and turned into GD with analogous sample and time complexity.

Revised: In that context, a natural goal is to compare quantitatively the differences between GD with multiple passes over the training data and SGD by adapting tools from [10, 11].

Revised: Trainable biases and untied directions.

Revised: Our proposed neural network architecture is non-standard, in the sense that its biases are frozen at initialization and all neurons share the same inner weight.

Revised: For the purposes of learning single-index models, removing these restrictions would not bring any statistical beneﬁts.

Revised: However, it would be interesting to extend our analysis to the general setting where the ﬁrst layer weights are not tied and biases are not frozen.

Original:  Multi-index models are natural extensions of single-index models where the hidden direction  is replaced by a hidden low-dimensional subspace.
Revised: Extension to multi-index models. Multi-index models are natural extensions of single-index models where the hidden direction ✓ ⇤ is replaced by a hidden low-dimensional subspace .

Original: They enjoy similar statistical guarantees as single-index models [33, 5], and thus a natural question is whether the same algorithmic tools developed here extend to the multi-index setting.
Revised: Typically, multi-index models enjoy similar statistical guarantees as single-index models [33, 5], and thus a natural question is whether the same algorithmic tools developed here extend to the multi-index setting.

Original: • Gradient dynamics without warm-start.
Revised: Gradient dynamics without warm-start.

Original: An unsatisfactory aspect of our results is the requirement that the algorithm starts by only optimizing ✓ for t < T 0 .
Revised: An unsatisfactory aspect of our results is the requirement that the algorithm starts by only optimizing ✓ for t < T 0 .

Original: It would be interesting to understand whether the vanilla dynamics could also succeed.
Revised: It would be interesting to understand whether the vanilla dynamics can also succeed provably.

Original: • Trainable biases and untied directions. Our proposed neural network architecture is nonstandard, in the sense that its biases are frozen at initialization and all neurons share the same inner weight. For single-index models, removing these restrictions would not bring any statistical beneﬁts. However, it would be interesting to extend our analysis to the general setting.
Revised: Acknowledgments and Disclosure of Funding

Original: Acknowledgements. We are thankful to Enric Boix-Adser ` a, Alex Damian, C ´ edric Gerbelot, 
Revised: We are thankful to Enric Boix-Adser`a, Alex Damian, C´edric Gerbelot, Daniel Hsu, Jason

Original: Daniel Hsu, Jason Lee, Theodor Misiakiewicz, Matus Telgarsky, Eric Vanden-Eijnden, and Denny Wu for useful discussions.
Revised: Lee, Theodor Misiakiewicz, Matus Telgarsky, Eric Vanden-Eijnden and Denny Wu for useful discussions.

Revised: We also thank the anonymous NeurIPS reviewers and area chair for helpful feedback.

Original: JB, AB and MJ are partially supported by NSF RI-1816753, NSF CAREER CIF 1845360, NSF CHS-1901091, NSF Scale MoDL DMS 2134216, Capital One and Sam Checklist
Revised: JB, AB and MJ are partially supported by NSF RI-1816753, NSF CAREER CIF 1845360, NSF CHS-1901091, NSF Scale MoDL DMS 2134216, Capital One and Samsung

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example: • Did you include the license to the code and datasets? [Yes] • Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Revised: Electronics.

Original: Did you include the license to the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers. Note that the
Revised: CS is supported by an NSF GRFP and by NSF grants CCF-1814873 and IIS-1838154.

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Conclusions.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: This is a purely theoretical work with no practical applications.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sectionsand 6.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Section E.2.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Supplemental zip ﬁle.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section B.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Section B. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section B. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]