Original: A DAPTIVE G ENERALIZATION AND O PTIMIZATION OF T HREE -L AYER N EURAL N ETWORKS
Revised: A DAPTIVE G ENERALIZATION AND O PTIMIZATION OF T HREE -L AYER N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While there has been substantial recent work studying generalization of neural networks, the ability of deep nets in automating the process of feature extraction still evades a thorough mathematical understanding.
Revised: While there has been substantial recent work studying generalization of neural networks, the ability of deep nets in automating the process of feature extraction still evades a thorough mathematical understanding.

Original: As a step toward this goal, we analyze learning and generalization of a three-layer neural network with ReLU activations in a regime that goes beyond the linear approximation of the network and is hence not captured by the common Neural Tangent Kernel.
Revised: As a step toward this goal, we analyze learning and generalization of a three-layer neural network with ReLU activations in a regime that goes beyond the linear approximation of the network and is hence not captured by the common Neural Tangent Kernel.

Original: We show that despite nonconvexity of the empirical loss, a variant of SGD converges in polynomially many iterations to a good solution that generalizes.
Revised: We show that despite nonconvexity of the empirical loss, a variant of SGD converges in polynomially many iterations to a good solution that generalizes.

Original: In particular, our generalization bounds are adaptive: they automatically optimize over a family of kernels that includes the Neural Tangent Kernel to provide the tightest bound.
Revised: In particular, our generalization bounds are adaptive: they automatically optimize over a family of kernels that includes the Neural Tangent Kernel to provide the tightest bound.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability of overparameterized neural networks trained by (stochastic) gradient descent to generalize well on test data (Krizhevsky et al., 2012; Silver et al., 2016; Hinton et al., 2012), even if they perfectly ﬁt the the training data, has intrigued theoretical researchers and led to many approaches for generalization bounds (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018; Dziugaite & Roy, 2017; Wei et al., 2019; Golowich et al., 2018; Arora et al., 2018b; Zhou et al., 2018;
Revised: The ability of overparameterized neural networks trained by (stochastic) gradient descent to generalize well on test data (Krizhevsky et al., 2012; Silver et al., 2016; Hinton et al., 2012), even if they perfectly ﬁt the the training data, has intrigued theoretical researchers and led to many approaches for generalization bounds (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018; Dziugaite & Roy, 2017; Wei et al., 2019; Golowich et al., 2018; Arora et al., 2018b; Zhou et al., 2018;

Original: Konstantinos et al., 2017).
Revised: Konstantinos et al., 2017).

Original: This generalization ability is closely tied to the optimization procedure – the trajectory of the training algorithm in a non-convex loss landscape – and the structure of the data.
Revised: This generalization ability is closely tied to the optimization procedure – the trajectory of the training algorithm in a non-convex loss landscape – and the structure of the data.

Original: Hence, several recent works study the training of neural networks.
Revised: Hence, several recent works study the training of neural networks.

Original: For instance, Safran & Shamir (2018) address the role of overparametrization in avoiding bad local minima, and Zhang et al.
Revised: For instance, Safran & Shamir (2018) address the role of overparametrization in avoiding bad local minima, and Zhang et al.

Original: (2016) empirically show that overparametrized networks trained by SGD can even perfectly ﬁt to random labels.
Revised: (2016) empirically show that overparametrized networks trained by SGD can even perfectly ﬁt to random labels.

Original: Within the popular framework of the Neural Tangent Kernel (NTK) (Jacot et al., 2018), which uses a linear approximation of the network at initialization, several works analyze the optimization trajectory and show global convergence of (S)GD to a global optimum of the empirical loss (AllenZhu et al., 2019; Li & Liang, 2018; Zou et al., 2018; Du et al., 2018).
Revised: Within the popular framework of the Neural Tangent Kernel (NTK) (Jacot et al., 2018), which uses a linear approximation of the network at initialization, several works analyze the optimization trajectory and show global convergence of (S)GD to a global optimum of the empirical loss (AllenZhu et al., 2019; Li & Liang, 2018; Zou et al., 2018; Du et al., 2018).

Original: Extending the viewpoint to generalization, Arora et al.
Revised: Extending the viewpoint to generalization, Arora et al.

Original: (2019a;b) exploit the kernel-like behaviour of two-layer networks close to their initialization to prove generalization for the ﬁnal network, showing that two-layer neural networks generalize as well as Kernel Ridgeless Regression (KRLR) with the NTK.
Revised: (2019a;b) exploit the kernel-like behaviour of two-layer networks close to their initialization to prove generalization for the ﬁnal network, showing that two-layer neural networks generalize as well as Kernel Ridgeless Regression (KRLR) with the NTK.

Original: Cao & Gu (2019) show a tighter bound with a Neural Tangent Random Feature Model .
Revised: Cao & Gu (2019) show a tighter bound with a Neural Tangent Random Feature Model .

Original: The kernel approach, however, has two main limitations: First, while KRLR can generalize well in speciﬁc high dimensional regimes (Liang et al., 2020), there is theoretical and empirical evidence that it can be inconsistent with noise (Rakhlin & Zhai, 2019).
Revised: The kernel approach, however, has two main limitations: First, while KRLR can generalize well in speciﬁc high dimensional regimes (Liang et al., 2020), there is theoretical and empirical evidence that it can be inconsistent with noise (Rakhlin & Zhai, 2019).

Original: Is there an approach for analyzing neural networks that shows they perform at least as well as KRLR, but is also robust to noise?
Revised: Is there an approach for analyzing neural networks that shows they perform at least as well as KRLR, but is also robust to noise?

Original: Second, importantly, neural networks are known to outperform traditional statistical methods in many regimes as they are able to automate the process of feature extraction from data, as opposed to kernel methods that work with a ﬁxed feature representation.
Revised: Second, importantly, neural networks are known to outperform traditional statistical methods in many regimes as they are able to automate the process of feature extraction from data, as opposed to kernel methods that work with a ﬁxed feature representation.

Original: This poses the question of other, adaptive, regimes beyond the linear network approximation.
Revised: This poses the question of other, adaptive, regimes beyond the linear network approximation.

Original: In this realm, Wu et al.
Revised: In this realm, Wu et al.

Original: (2018) show generalization bounds that, instead of the NTK norm, scale with respect to another functional norm.
Revised: (2018) show generalization bounds that, instead of the NTK norm, scale with respect to another functional norm.

Original: This norm corresponds to the minimum RKHS norm of the function among a family of kernels, i.e., their method in a sense picks the best kernel in this family.
Revised: This norm corresponds to the minimum RKHS norm of the function among a family of kernels, i.e., their method in a sense picks the best kernel in this family.

Original: However, this result ignores the computational aspect of the problem.
Revised: However, this result ignores the computational aspect of the problem.

Original: Are there particular nonlinear regimes beyond NTK for which a gradient-type polynomial-time algorithm, in a way, adaptively chooses a suitable kernel?
Revised: Are there particular nonlinear regimes beyond NTK for which a gradient-type polynomial-time algorithm, in a way, adaptively chooses a suitable kernel?

Original: Going beyond the NTK view, a line of work convexiﬁes the optimization problem via an approximation of SGD dynamics with a continuous time gradient ﬂow in the space of probability measures on the hidden units of the network, equipped with the Wasserstein metric (Mei et al., 2018; Chizat & Bach, 2018; Mei et al., 2019; Wei et al., 2018; Sirignano & Spiliopoulos, 2020; Javanmard et al., 2019;
Revised: Going beyond the NTK view, a line of work convexiﬁes the optimization problem via an approximation of SGD dynamics with a continuous time gradient ﬂow in the space of probability measures on the hidden units of the network, equipped with the Wasserstein metric (Mei et al., 2018; Chizat & Bach, 2018; Mei et al., 2019; Wei et al., 2018; Sirignano & Spiliopoulos, 2020; Javanmard et al., 2019;

Original: Lu et al., 2020).
Revised: Lu et al., 2020).

Original: Taking another perspective, Allen-Zhu et al.
Revised: Taking another perspective, Allen-Zhu et al.

Original: (2018) consider a three-layer network model that is not captured by the NTK approximation, and learn an underlying concept class by exploiting saddle-point escape theory for nonconvex SGD (Ge et al., 2015a).
Revised: (2018) consider a three-layer network model that is not captured by the NTK approximation, and learn an underlying concept class by exploiting saddle-point escape theory for nonconvex SGD (Ge et al., 2015a).

Original: However, evaluating the complexity measure of Allen-Zhu et al.
Revised: However, evaluating the complexity measure of Allen-Zhu et al.

Original: (2018) is rather involved, and only aligns well with functions that are described by a particular network.
Revised: (2018) is rather involved, and only aligns well with functions that are described by a particular network.

Original: Whether one can recover the NTK bound (e.g. the NTK norm) from these results is not clear, while computing these generalization bounds from a given dataset is another question.
Revised: Whether one can recover the NTK bound (e.g. the NTK norm) from these results is not clear, while computing these generalization bounds from a given dataset is another question.

Original: For the NTK setting, in contrast, Arora et al. (2019a) develop a purely data dependent generalization bound.
Revised: For the NTK setting, in contrast, Arora et al. (2019a) develop a purely data dependent generalization bound.

Original: Going to deeper than two layers, is it possible to prove a data-dependent complexity measure beyond the NTK regime that recovers the NTK result (Arora et al., 2019a) as a special case?
Revised: Going to deeper than two layers, is it possible to prove a data-dependent complexity measure beyond the NTK regime that recovers the NTK result (Arora et al., 2019a) as a special case?

Original: In this work, we address the above questions:
Revised: In this work, we address the above questions:

Original: • We consider a regime for 3-layer neural networks that is not captured by the NTK approximation and show that, despite nonconvexity, a variant of projected SGD ﬁnds a good solution, as measured by the regularized empirical loss, importantly, after polynomially many iterations.
Revised: • We consider a regime for 3-layer neural networks that is not captured by the NTK approximation and show that, despite nonconvexity, a variant of projected SGD ﬁnds a good solution, as measured by the regularized empirical loss, importantly, after polynomially many iterations.

Original: • We introduce a new function norm ∥ .
Revised: • We introduce a new function norm ∥ .

Original: ∥ ζ as the minimum RKHS norm with respect to a family of kernels K , which is upper bounded by the NTK norm up to constants.
Revised: ∥ ζ as the minimum RKHS norm with respect to a family of kernels K , which is upper bounded by the NTK norm up to constants.

Original: We show that for an arbitrary function f , the generalization gap of the trained network scales by ∥ f ∥ ζ .
Revised: We show that for an arbitrary function f , the generalization gap of the trained network scales by ∥ f ∥ ζ .

Original: This makes our generalization bound adaptive, in the sense that it scales with the best kernel in K .
Revised: This makes our generalization bound adaptive, in the sense that it scales with the best kernel in K .

Original: As a byproduct, our bounds are comparable with kernel regression bounds simultaneously with all kernels in K .
Revised: As a byproduct, our bounds are comparable with kernel regression bounds simultaneously with all kernels in K .

Original: We hope that our techniques motivate researchers to prove such adaptive generalization bounds for deeper networks, which can potentially result in stronger depth separation.
Revised: We hope that our techniques motivate researchers to prove such adaptive generalization bounds for deeper networks, which can potentially result in stronger depth separation.

Original: • We show generalization bounds with a new data-dependent complexity measure that generalizes the NTK-based complexity measure in (Arora et al., 2019a).
Revised: • We show generalization bounds with a new data-dependent complexity measure that generalizes the NTK-based complexity measure in (Arora et al., 2019a).

Original: Up to logarithmic factors, our bounds are upper bounded by the NTK-based bounds in (Arora et al., 2019a) and hence strictly improve over them (if one substitutes their Lipschitz loss with a smooth one) – see Appendix A.1 for a simple explicit example.
Revised: Up to logarithmic factors, our bounds are upper bounded by the NTK-based bounds in (Arora et al., 2019a) and hence strictly improve over them (if one substitutes their Lipschitz loss with a smooth one) – see Appendix A.1 for a simple explicit example.

Original: Importantly, our bound can also handle noisy distributions as opposed to (Arora et al., 2019a).
Revised: Importantly, our bound can also handle noisy distributions as opposed to (Arora et al., 2019a).

Original: Further Related work.
Revised: Further Related work.

Original: While the idea of a learning algorithm that combines multiple kernels has been employed for a while in the community (Sonnenburg et al., 2006; Rakotomamonjy et al., 2007; Duan et al., 2012), our understanding of the connections between deep learning and multiple kernel learning is yet in its infancy.
Revised: While the idea of a learning algorithm that combines multiple kernels has been employed for a while in the community (Sonnenburg et al., 2006; Rakotomamonjy et al., 2007; Duan et al., 2012), our understanding of the connections between deep learning and multiple kernel learning is yet in its infancy.

Original: Recently, Dou & Liang (2020) deﬁne a time-varying kernel based on the network weights and show that the limit of the gradient ﬂow converge to a suitable dynamic kernel, in the sense that the residual of the link function onto its RKHS could be in a smaller ranked space compared to the orthogonal complement of the RKHS.
Revised: Recently, Dou & Liang (2020) deﬁne a time-varying kernel based on the network weights and show that the limit of the gradient ﬂow converge to a suitable dynamic kernel, in the sense that the residual of the link function onto its RKHS could be in a smaller ranked space compared to the orthogonal complement of the RKHS.

Original: Ghorbani et al.
Revised: Ghorbani et al.

Original: (2019) analyze the difference between training a two layer ReLu network compared to its NTK or random feature simpliﬁcations, for a mixture of Gaussians input distribution and quadratic target functions.
Revised: (2019) analyze the difference between training a two layer ReLu network compared to its NTK or random feature simpliﬁcations, for a mixture of Gaussians input distribution and quadratic target functions.

Original: Ignoring the computational hardness imposed by nonconvexity, Bach (2017) prove a dimension dependent generalization bound beyond NTK.
Revised: Ignoring the computational hardness imposed by nonconvexity, Bach (2017) prove a dimension dependent generalization bound beyond NTK.

Original: In another line of work, Chizat & Bach (2020) study gradient ﬂow on losses with exponential tail and its relation to the max margin solution.
Revised: In another line of work, Chizat & Bach (2020) study gradient ﬂow on losses with exponential tail and its relation to the max margin solution.

Original: Wei et al.
Revised: Wei et al.

Original: (2019) show an interesting separation between the learning power of two layer ReLu nets and
Revised: (2019) show an interesting separation between the learning power of two layer ReLu nets and

Original: their
Revised: their

Original: NTK approximation, by showing a sample complexity gap for an artiﬁcially constructed distribution.
Revised: NTK approximation, by showing a sample complexity gap for an artiﬁcially constructed distribution.

Original: With a different approach, Allen-Zhu & Li (2020) analyze multi-layer networks with quadratic activations, and prove generalization bounds polynomial in the dimension and precision by assuming an underlying teacher network, which shows a remarkable algorithmic depth separation.
Revised: With a different approach, Allen-Zhu & Li (2020) analyze multi-layer networks with quadratic activations, and prove generalization bounds polynomial in the dimension and precision by assuming an underlying teacher network, which shows a remarkable algorithmic depth separation.

Original: The problem of depth separation for neural networks and more generally their expressive power has been investigated by several researchers before (Raghu et al., 2017; Daniely, 2017; Barron, 1994; Funahashi, 1989; Safran & Shamir, 2016; Safran et al., 2019).
Revised: The problem of depth separation for neural networks and more generally their expressive power has been investigated by several researchers before (Raghu et al., 2017; Daniely, 2017; Barron, 1994; Funahashi, 1989; Safran & Shamir, 2016; Safran et al., 2019).

Original: The assumption of an underlying teacher network that one seeks to recover is common, too (Li & Yuan, 2017; Zhong et al., 2017; Brutzkus & Globerson, 2017).
Revised: The assumption of an underlying teacher network that one seeks to recover is common, too (Li & Yuan, 2017; Zhong et al., 2017; Brutzkus & Globerson, 2017).

Original: Other works focus mainly on the algorithm and use other techniques, such as tensor factorization, to ﬁnd a global optimum (Tian, 2016; Bakshi et al., 2019; Janzamin et al., 2015;
Revised: Other works focus mainly on the algorithm and use other techniques, such as tensor factorization, to ﬁnd a global optimum (Tian, 2016; Bakshi et al., 2019; Janzamin et al., 2015;

Original: Zhong et al., 2017).
Revised: Zhong et al., 2017).

Original: Finally, many authors study the loss landscape under various assumptions (Freeman & Bruna, 2016; Nguyen & Hein, 2017; Soudry & Carmon, 2016; Soltanolkotabi et al., 2018;
Revised: Finally, many authors study the loss landscape under various assumptions (Freeman & Bruna, 2016; Nguyen & Hein, 2017; Soudry & Carmon, 2016; Soltanolkotabi et al., 2018;

Original: Ge et al., 2017), some of them consider the simpliﬁed case of deep linear networks (Arora et al., 2018a; Saxe et al., 2013; Bartlett et al., 2018; Kawaguchi, 2016).
Revised: Ge et al., 2017), some of them consider the simpliﬁed case of deep linear networks (Arora et al., 2018a; Saxe et al., 2013; Bartlett et al., 2018; Kawaguchi, 2016).

Original:  S ETUP AND APPROXIMATION BY KERNELS
Revised:  S ETUP AND APPROXIMATION BY KERNELS

Original: We analyze a 3-layer ReLu neural network from inputs x ∈ R d to outputs y ∈ R , of the form
Revised: We analyze a 3-layer ReLu neural network from inputs x ∈ R d to outputs y ∈ R , of the form

Original: [Equation]
Revised: [Equation]

Original: where a ∈ R m 2 is a vector of random signs, V (

Revised: where a ∈ R m 2 is a vector of random signs, V (

Original: ) ∈
Revised: ) ∈

Original: R m 2 × m 3 and W (0) ∈
Revised: R m 2 × m 3 and W (0) ∈

Original: R m 1 ×
Revised: R m 1 ×

Original: d are random weight initializations with i.i.d Gaussian entries V (0) j,k ∼ N (0 , κ 22 ) , W (0) j,k ∼ N (0 , κ 21 ) , and
Revised: d are random weight initializations with i.i.d Gaussian entries V (0) j,k ∼ N (0 , κ 22 ) , W (0) j,k ∼ N (0 , κ 21 ) , and

Original: W s ∈ R m 3 × m 1 is a random sign matrix, which is roughly a random projection and change of coordinates into a lower dimensional space.
Revised: W s ∈ R m 3 × m 1 is a random sign matrix, which is roughly a random projection and change of coordinates into a lower dimensional space.

Original: We refer to W s 1 √ m 1 σ (( W (0) + W ′ )
Revised: We refer to W s 1 √ m 1 σ (( W (0) + W ′ )

Original: x ) as the ﬁrst layer and 1 √ m 2 a T σ ( V (0) + V ′ )
Revised: x ) as the ﬁrst layer and 1 √ m 2 a T σ (( V (0) + V ′ )( . ))

Original: as the second layer.
Revised: as the second layer.

Original: The algorithm trains weight matrices V ′ , W ′ , and W s , a are ﬁxed.
Revised: The algorithm trains weight matrices V ′ , W ′ , and W s , a are ﬁxed.

Original: We assume that the outputs are a.s. bounded by a constant, | y i | ≤ B , and ∥ x i ∥ = 1 .
Revised: We assume that the outputs are a.s. bounded by a constant, | y i | ≤ B , and ∥ x i ∥ = 1 .

Original: We consider the square loss denoted by ℓ ( ., . ) .
Revised: We consider the square loss denoted by ℓ ( ., . ) .

Original: We denote the training (empirical) loss of a function f on our dataset { x i } ni =1 , { y i } ni =1 , and the expected loss with respect to the data distribution  byR n ( f ) = 1 n X n i =1 ℓ ( f ( x i ) , y i ) , and
Revised: We denote the training (empirical) loss of a function f on our dataset { x i } ni =1 , { y i } ni =1 , and the expected loss with respect to the data distribution (population loss) by

Original: R ( f ) =
Revised: [Equation]

Original: E ℓ ( f ( x ) , y ) , respectively.
Revised: , y ) , respectively.

Original: Sometimes, we refer to the vector of labels ( y i ) ni
Revised: Sometimes, we refer to the vector of labels ( y i ) ni

Original: =1 by y .
Revised: =1 by y .

Original: We denote the space of functions with bounded RKHS-norm of kernel K by H K and write ˜ O to hide the log factors.
Revised: We denote the space of functions with bounded RKHS-norm of kernel K by H K and write ˜ O to hide the log factors.

Original: 2. K ERNEL APPROXIMATIONS , DECOMPOSITION AND ADAPTIVITY
Revised: 2. K ERNEL APPROXIMATIONS , DECOMPOSITION AND ADAPTIVITY

Original: Kernel approximations of neural networks play an important role in our analysis.
Revised: Kernel approximations of neural networks play an important role in our analysis.

Original: First, a common approximation is the NTK.
Revised: First, a common approximation is the NTK.

Original: The Neural Tangent kernel for a 2-layer ReLu network is
Revised: The Neural Tangent kernel for a 2-layer ReLu network is

Revised: H ∞ ( x 1 , x 2 ) =

Revised: ⟨ x 1 , x 2 ⟩ · F(cid:16)

Revised: [Equation]

Revised: x 1 , x 2 ⟩ / (

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Original:  To introduce adaptivity, a key observation of our analysis is to approximate the second layer in the 3-layer network by a product kernel K ∞ ⊙ G that decomposes into a “ﬁxed” part K ∞ and an “adaptive” part G .
Revised: x [Equation] ) (cid:17) , for F 2 ( x ) = 14 + arcsin( x ) / (2 π ) . To introduce adaptivity, a key part of our analysis is to approximate the second layer in the 3-layer network by a product kernel K ∞ ⊙ G that decomposes into a “ﬁxed” part K ∞ and an “adaptive” part G .

Original: To deﬁne these kernels, for every i ∈ [ n ] , let ϕ (0) ( x i ) be the output of the ﬁrst layer of the network at initialization, ϕ (0) ( x i ) =
Revised: To deﬁne these kernels, for every i ∈ [ n ] , let ϕ (0) ( x i ) be the output of the ﬁrst layer of the network at initialization, ϕ (0) ( x i ) =

Original: [Equation]
Revised: [Equation]

Original: m 1 W s σ ( W (0) x i ) , and ϕ (0) ( x i ) + ϕ ′ ( x i ) be the same output for weights W (0) + W ′ .
Revised: m 1 W s σ ( W (0) x i ) , and ϕ (0) ( x i ) + ϕ ′ ( x i ) be that output for weights W (0) + W ′ .

Original: The adaptive kernel G captures the  learned weights:
Revised: The adaptive kernel G captures the dot product between the learned weights:

Original: [Equation]
Revised: [Equation]

Original: This form of G motivates the complexity measure we deﬁne in the next section, if one thinks of the entries of ϕ ′ as bounded NTK-norm functions.
Revised: This form of G motivates the complexity measure we deﬁne in the next section, if one thinks of the entries of ϕ ′ as bounded NTK-norm functions of the input.

Original: Next, we consider the second layer, where the part K ∞ arises from roughly stable activations.
Revised: Next, we consider the second layer, where the part K ∞ arises from roughly stable activations.

Original: To formalize this stability, let Sgn ( V x ) be the diagonal matrix whose diagonal contains the coordinate-wise signs of vector V x .
Revised: To formalize this stability, let Sgn ( V x ) be the diagonal matrix whose diagonal contains the coordinate-wise signs of vector V x .

Original: If we assume that Sgn
Revised: If we assume that Sgn

Original: [Equation]
Revised: [Equation]

Original: 0) ϕ (0) ( x i ) (cid:1) – we prove a rigorous statement in
Revised: x i ) [Equation] [Equation] [Equation]

Original:  Appendix A.12 – then
Revised: x i [Equation] – we prove a rigorous statement in Appendix A.12 – then

Original: [Equation]
Revised: [Equation]

Original: Focusing on the adaptive part ϕ ′ ( x ) of the ﬁrst layer, we write
Revised: Focusing on the adaptive part ϕ ′ ( x ) of the ﬁrst layer, we write

Original: [Equation]
Revised: [Equation]

Original: and can then view the second layer as a function in the RKHS of the product kernel ⟨ Υ( x i ) , Υ( x j ) ⟩ = ˜ K ∞ ( x i , x j ) G ( x i , x j ) .
Revised: and can then view the second layer as a function in the RKHS of the product kernel ⟨ Υ( x i ) , Υ( x j ) ⟩ =: ˜ K ∞ ( x i , x j ) G ( x i , x j ) .

Revised: This deﬁnes the new kernel ˜ K ∞ , which we simplify into the kernel K ∞ that is independent of initialization.

Original: We show that ˜ K ∞ concentrates aroundE
Revised: To do so, we ﬁrst observe that ˜ K ∞ concentrates around

Original: [Equation]
Revised: [Equation]

Original: Moreover, the Gaussian initialization and assumption ∥ x i ∥
Revised: Moreover, the Gaussian initialization and assumption ∥ x i ∥

Original: = 1 imply that ⟨ ϕ (0) ( x i ) , ϕ (0) ( x j ) ⟩ concentrates around m 3 F 3 ( ⟨ x i , x j ⟩ ) , for F 3 : [ − 1 , 1] → [0 , 12 ] deﬁned as
Revised: = 1 imply that ⟨ ϕ (0) ( x i ) , ϕ (0) ( x j ) ⟩ concentrates around m 3 F 3 ( ⟨ x i , x j ⟩ ) , for F 3 : [ − 1 , 1] → [0 , 12 ] deﬁned as

Original: [Equation]
Revised: [Equation]

Original: For general x 1 , x 2 not necessarily unit norm, we deﬁne K ∞ ( x 1 , x 2 ) =
Revised: For general x 1 , x 2 not necessarily unit norm, we deﬁne K ∞ ( x 1 , x 2 ) =

Original: K ∞ ( x 1 / ∥ x 1 ∥ , x 2 / ∥ x 2 ∥ ) .
Revised: K ∞ ( x 1 / ∥ x 1 ∥ , x 2 / ∥ x 2 ∥ ) .

Original: It is easy to check that the coefﬁcients in the Taylor series of F 2 and F 3 are nonnegative.
Revised: It is easy to check that the coefﬁcients in the Taylor series of F 2 and F 3 are nonnegative.

Original: Combining this with Schur’s Product Theorem implies K ∞ is PSD (Appendix A.4).
Revised: Combining this with Schur’s Product Theorem implies K ∞ is PSD (Appendix A.4).

Original: We also denote the data kernel matrix on ( x i ) ni
Revised: We also denote the data kernel matrix on ( x i ) ni

Original: =1 by K ∞ and H ∞ .
Revised: =1 by K ∞ and H ∞ .

Original: As Arora et al.
Revised: As Arora et al.

Original: (2019a)
Revised: (2019a)

Original: , we assume the data distribution is ( λ 0 , δ, n ) -non-degenerate with respect to H ∞ and K ∞ , i.e., with probability at least 1 − δ , the smallest eigenvalues of H ∞ and K ∞ are at least λ 0 > 0 .
Revised: , we assume the data distribution is ( λ 0 , δ, n ) -non-degenerate with respect to H ∞ and K ∞ , i.e., with probability at least 1 − δ , the smallest eigenvalues of H ∞ and K ∞ are at least λ 0 > 0 .

Original:  D ATA DEPENDENT COMPLEXITY MEASURE AND GENERALIZATION
Revised:  D ATA DEPENDENT COMPLEXITY MEASURE AND GENERALIZATION

Original: The emergence of G ⊙ K ∞ above gives rise to an adaptive kernel-like complexity measure that will determine generalization bounds.
Revised: The emergence of G ⊙ K ∞ above gives rise to an adaptive kernel-like complexity measure that will determine generalization bounds.

Original: Intuitively, this complexity measure reﬂects the two layers.
Revised: Intuitively, this complexity measure reﬂects the two layers.

Original: Here, we view G as the Gram matrix of some “ideal” ﬁrst-layer feature functions g k .
Revised: Here, we view G as the Gram matrix of some “ideal” ﬁrst-layer feature functions g k .

Original: We measure the complexity of the prediction function via the RKHS of A :=
Revised: We measure the complexity of the prediction function via the RKHS of A :=

Original: G ⊙ K ∞ , and allow a ﬂexible choice of the features g k .
Revised: G ⊙ K ∞ , and allow a ﬂexible choice of the features g k .

Revised: Here, g k ’s substitute the entries of ϕ ′ in Equation (3): G ( x i , x j ) :

Revised: =

Original:  To regularize this choice, we penalize the complexity of the features g via the NTK norm, so we seek g k ’s with the minimum possible norms.
Revised: P g k ( x i ) g k ( x j ) . To regularize this choice, we penalize the complexity of the features g via the NTK norm, so we seek g k ’s with the minimum possible norms.

Original: Alternatively, the features g k are ﬂexible but have bounded NTK norm.
Revised: Alternatively, the features g k are ﬂexible but have bounded NTK norm.

Original: For a labeling f ∗ ∈ R n of the n data points and ﬁxed G , this leads to the complexity
Revised: For a labeling f ∗ ∈ R n of the n data points and ﬁxed G , this leads to the complexity

Original: [Equation]
Revised: [Equation]

Original: where m  is the number of intermediate features.
Revised: where m 3 is the number of intermediate features (The choice of m 3 is discussed in Appendix A.13).

Original: Our data-dependent complexity measure implicitly selects the G  that leads to the tightest bound, trading off data ﬁt and function complexity:
Revised: Our data-dependent complexity measure implicitly selects the G (or equivalently the feature vectors g k ) that leads to the tightest bound, trading off data ﬁt and function complexity:

Original: [Equation]
Revised: [Equation]

Original: where we use a log factor ϖ =
Revised: where we use a log factor ϖ =

Original: O (log( n ) 3 + log(1 /λ 0 )) .
Revised: O (log( n ) 3 + log(1 /λ 0 )) .

Original: To make the relation to adaptive kernel spaces even more explicit, assume that the g k are bounded as P k ∥
Revised: To make the relation to adaptive kernel spaces even more explicit, assume that the g k are bounded as P k ∥

Original: g k ∥ 2 H ∞ ≤ 1 .
Revised: g k ∥ 2 H ∞ ≤ 1 .

Original: Then we deﬁne a family K of corresponding kernels of the form
Revised: Then we deﬁne a family K of corresponding kernels of the form

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: i.e., K :=
Revised: i.e., K :=

Original: { K { g } | { g } ﬁnite , P g ∈{ g } ∥
Revised: { K { g } | { g } ﬁnite , P g ∈{ g } ∥

Original: g ∥ 2 H ∞ ≤ 1 } .
Revised: g ∥ 2 H ∞ ≤ 1 } .

Original: With this notation, the complexity measure is
Revised: With this notation, the complexity measure is

Original: [Equation]
Revised: [Equation]

Original: Hence, this measure may be understood as searching for the most efﬁcient and effective feature representation within a family of RKHSs.
Revised: Hence, this measure may be understood as searching for the most efﬁcient and effective feature representation within a family of RKHSs.

Original: We may also relate this complexity measure to the NTK-based complexity measure y T H ∞− 1 y (Arora et al., 2019a).
Revised: We may also relate this complexity measure to the NTK-based complexity measure y T H ∞− 1 y (Arora et al., 2019a).

Original: For any labeling f ∗ , let ˜ f ∗ ∈ H H ∞ be the function with minimum NTK norm that maps x i ’s to f ∗ i ’s, so ∥
Revised: For any labeling f ∗ , let ˜ f ∗ ∈ H H ∞ be the function with minimum NTK norm that maps x i ’s to f ∗ i ’s, so ∥

Original: ˜ f ∗ ∥ H ∞ =
Revised: ˜ f ∗ ∥ H ∞ =

Original: f ∗ T H ∞− 1 f ∗ .
Revised: f ∗ T H ∞− 1 f ∗ .

Original: If we set { g } = { ˜ f ∗ / ∥ ˜ f ∗ ∥} , then one can show (Appendix A.5)
Revised: If we set { g } = { ˜ f ∗ / ∥ ˜ f ∗ ∥} , then one can show (Appendix A.5)

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: One can further set f ∗ =
Revised: One can further set f ∗ =

Original: y above and obtain
Revised: y above and obtain

Original: [Equation]
Revised: [Equation]

Original: 3. G ENERALIZATION
Revised: 3. G ENERALIZATION

Original: With the complexity ℑ in hand, we can now state our generalization result.
Revised: With the complexity ℑ in hand, we can now state our generalization result.

Original: It assumes optimization by a Projected Stochastic Gradient Descent ( PSGD ), described in detail in Section 4.
Revised: It assumes optimization by a Projected Stochastic Gradient Descent ( PSGD ), described in detail in Section 4.

Original: Theorem 1 Suppose we run Projected Stochastic Gradient Descent ( PSGD ) on the regularized empirical risk with parameters as in Section 4.
Revised: Theorem 1 Suppose we run Projected Stochastic Gradient Descent ( PSGD ) on the regularized empirical risk with parameters as in Section 4.

Original: Then, with high probability (e.g. 0 . 99 ) over the randomness of data, initialization and noise of the gradient steps, PSGD converges in poly ( B ∨ 1 /B, 1 /λ 0 , n ) iterations to a solution ( W PSGD , V PSGD ) with population risk bounded as (recall | y i | ≤ B a.s.)
Revised: Then, with high probability (e.g. 0 . 99 ) over the randomness of data, initialization and noise of the gradient steps, PSGD converges in poly ( B ∨ 1 /B, 1 /λ 0 , n ) iterations to a solution ( W PSGD , V PSGD ) with population risk bounded as (recall | y i | ≤ B a.s.)

Original: [Equation]
Revised: [Equation]

Original: As a side remark, the factor 2 in front of R n ( f ∗ ) in the deﬁnition of ℑ in (12) is not special and a similar generalization bound can be obtained for any γ > 1 .
Revised: As a side remark, the factor 2 in front of R n ( f ∗ ) in the deﬁnition of ℑ in (12) is not special and a similar generalization bound can be obtained for any γ > 1 .

Original: Substituting the upper bound on the complexity in Equation (15), one recovers an NTK-based generalization bound that scales with y T H ∞− 1 y/n up to log factors, which is roughly the square of the generalization bound presented in (Arora et al., 2019a).
Revised: Substituting the upper bound on the complexity in Equation (15), one recovers an NTK-based generalization bound that scales with y T H ∞− 1 y/n up to log factors, which is roughly the square of the generalization bound presented in (Arora et al., 2019a).

Original: The reason for the faster squared rate here is that we are considering smooth losses, while Arora et al.
Revised: The reason for the faster squared rate here is that we are considering smooth losses, while Arora et al.

Original: (2019a) work with a bounded Lipschitz loss.
Revised: (2019a) work with a bounded Lipschitz loss.

Original: Indeed, it is not hard to apply a more rigorous uniform convergence analysis from (Srebro et al., 2010) to also obtain a faster squared rate for the approach used in (Arora et al., 2019a).
Revised: Indeed, it is not hard to apply a more rigorous uniform convergence analysis from (Srebro et al., 2010) to also obtain a faster squared rate for the approach used in (Arora et al., 2019a).

Original: Since Equation (15) is an upper bound on our complexity, our result generalizes and tightens
Revised: Since Equation (15) is an upper bound on our complexity, our result generalizes and tightens

Original: the
Revised: the

Original: NTK bound (Arora et al., 2019a).
Revised: NTK bound (Arora et al., 2019a).

Original: To illustrate the ﬂexibility of our complexity measure, we show in Appendix A.1 a simple explicit example of functions represented as polynomial series where our bound improves upon the NTK bound (Arora et al., 2019a).
Revised: To illustrate the ﬂexibility of our complexity measure, we show in Appendix A.1 a simple explicit example of functions represented as polynomial series where our bound improves upon the NTK bound (Arora et al., 2019a).

Original: Notably, we only substitute low-rank matrices G in our complexity measure for this construction.
Revised: Notably, we only substitute low-rank matrices G in our complexity measure for this construction.

Original: We leave further investigation of our complexity measure for arbitrary G ’s to future work.
Revised: We leave further investigation of our complexity measure for arbitrary G ’s to future work.

Original: 3. U NDERLYING C ONCEPT CLASS
Revised: 3. U NDERLYING C ONCEPT CLASS

Original: Instead of data dependent generalization bounds, one may study the generalization gap with respect to some concept class.
Revised: Instead of data dependent generalization bounds, one may study the generalization gap with respect to some concept class.

Original: The complexity measure ℑ implicitly uses the following adaptive norm on the space of functions from R d , the inﬁmum of the RKHS norms for the family of kernels K :
Revised: The complexity measure ℑ implicitly uses the following adaptive norm on the space of functions from R d , the inﬁmum of the RKHS norms for the family of kernels K :

Original: [Equation]
Revised: [Equation]

Original: It is not hard to check that ∥ .
Revised: It is not hard to check that ∥ .

Original: ∥ ζ is in fact a norm, and that the inf is achieved by a particular set { g } .
Revised: ∥ ζ is in fact a norm, and that the inf is achieved by a particular set { g } .

Original: Similar to the derivation of the upper bound on the complexity measure in Equation (15), by setting { g }
Revised: Similar to the derivation of the upper bound on the complexity measure in Equation (15), by setting { g }

Original: = { f ∗ /
Revised: = { f ∗ /

Original: ∥ f ∗ ∥
Revised: ∥ f ∗ ∥

Original: H ∞ } , we obtain the following NTK upper bound:
Revised: H ∞ } , we obtain the following NTK upper bound:

Original: [Equation]
Revised: [Equation]

Original: This leads to a function-dependent generalization bound which bounds the risk of the learned network against an arbitrary function f with ∥ f ∥ ζ < ∞ .
Revised: This leads to a function-dependent generalization bound which bounds the risk of the learned network against an arbitrary function f with ∥ f ∥ ζ < ∞ .

Original: Theorem 2 For any measurable function f : R d → R , in the same setting as Theorem 1, the population risk of the trained network can be bounded as
Revised: Theorem 2 For any measurable function f : R d → R , in the same setting as Theorem 1, the population risk of the trained network can be bounded as

Original: [Equation]
Revised: [Equation]

Original: As in the data-dependent case, the factor 2 of the risk of f can be reduced to any constant γ > 1 .
Revised: As in the data-dependent case, the factor 2 of the risk of f can be reduced to any constant γ > 1 .

Original: 3. I NTERACTION OF LAYERS BEYOND THE LINEAR APPROXIMATION
Revised: 3. I NTERACTION OF LAYERS BEYOND THE LINEAR APPROXIMATION

Original: Here, we give a high level intuition on how the adaptivity is achieved in our regime compared to
Revised: Here, we give a high level intuition on how the adaptivity is achieved in our regime compared to

Original: NTK.
Revised: NTK.

Original: In the NTK approach, for every input x , the neural net f W,V ( x ) is approximated by its linear approximation at ( W (0) , V (0) )
Revised: In the NTK approach, for every input x , the neural net f W,V ( x ) is approximated by its linear approximation at ( W (0) , V (0) )

Original: (the initialized network), f W,V ( x ) = ⟨∇ W,V f W (0) ,V (0) ( x ) , ( W −
Revised: (the initialized network), f W,V ( x ) = ⟨∇ W,V f W (0) ,V (0) ( x ) , ( W −

Original: W (0) , V − V (0) ) ⟩ .
Revised: W (0) , V − V (0) ) ⟩ .

Original: The NTK approximation works as long as ( W, V ) are close enough to their initialization that the linear approximation remains accurate and the interaction of weights between layers is negligible.
Revised: The NTK approximation works as long as ( W, V ) are close enough to their initialization that the linear approximation remains accurate and the interaction of weights between layers is negligible.

Original: Speciﬁcally, the features ϕ ′
Revised: Speciﬁcally, the features ϕ ′

Original: ( x i ) behave almost linearly with respect to W − W (0) as ∥ W − W (0) ∥ is taken to be small and the sign pattern Sgn (cid:0) ( W (0) + W ′ )
Revised: ( x i ) behave almost linearly with respect to W − W (0) as ∥ W − W (0) ∥ is taken to be small and the sign pattern Sgn (cid:0) ( W (0) + W ′ )

Original: x i (cid:1) is proven not to change much compared to Sgn (cid:0)
Revised: x i (cid:1) is proven not to change much compared to Sgn (cid:0)

Original: W [Equation] 0) x i (cid:1) .
Revised: W (0) x i (cid:1) .

Original: Additionally, the NTK-type analysis needs the following two conditions to be satisﬁed: (1) the sign pattern of ϕ (0) ( x i ) + ϕ ′ ( x i ) with respect to V (0) + V ′ remains almost the same as the sign pattern of ϕ (0) ( x ) with respect to V (0) , and (2) the weight changes W ′ and V ′ should not interact, which means the “interaction” term,[Equation] m
Revised: Additionally, the NTK-type analysis needs the following two conditions to be satisﬁed: (1) the sign pattern of ϕ (0) ( x i ) + ϕ ′ ( x i ) with respect to V (0) + V ′ remains almost the same as the sign pattern of ϕ (0) ( x ) with respect to V (0) , and (2) the weight changes W ′ and V ′ should not interact, which means the “interaction” term,

Original: [Equation]
Revised: [Equation]

Original: x
Revised: ϕ ′

Original: [Equation]
Revised: [Equation]

Original: ϕ ′ ( x i ) ≈ 0 , should be negligible.
Revised: x i ) ≈ 0 , should be negligible.

Original: Therefore, the non-negligible terms for the NTK are: (1)
Revised: Therefore, the non-negligible terms for the NTK are: (1)

Original: [Equation]
Revised: [Equation]

Original: x
Revised: ϕ

Original: [Equation]
Revised: [Equation]

Original:  ) , which is almost linear in W ′ (recall that ϕ
Revised: x i ) , which is almost linear in W ′ (recall that ϕ

Original: ′ ( x i ) depends on W ′ ), and (2) 1 √ m 2 a T Sgn 
Revised: ′ ( x i ) depends on W ′ ), and (2) 1 √ m 2 a T Sgn [Equation] x

Original: [Equation]
Revised: [Equation]

Original:  which is linear in V ′ .
Revised: ) which is linear in V ′ .

Original: This approach has two important implications: (1) it convexiﬁes the optimization (for convex loss), as the approximation is now linear in W ; and (2) it simpliﬁes proving generalization, as it works with the class of functions in the RKHS space of some ﬁxed kernel.
Revised: This approach has two important implications: (1) it convexiﬁes the optimization (for convex loss), as the approximation is now linear in W ; and (2) it simpliﬁes proving generalization, as it works with the class of functions in the RKHS space of some ﬁxed kernel.

Original: However, this simpliﬁcation leaves no room for the ability of the neural network to learn intermediate feature representations which could result in better function classes.
Revised: However, this simpliﬁcation leaves no room for the ability of the neural network to learn intermediate feature representations which could result in better function classes.

Original: In our regime, however, we enforce the condition ∀ j, i : V ′ j ⊥ ϕ (0) ( x i )  , which implies the second (2) above is zero, while the interaction term is not negligible any more and the network behaves similar to a quadratic function with respect to ( W ′ , V ′ )
Revised: In our regime, however, we enforce the condition ∀ j, i : V ′ j ⊥ ϕ (0) ( x i ) ( ⋆ ) , which implies the second (2) above is zero, while the interaction term is not negligible any more and the network behaves similar to a quadratic function with respect to ( W ′ , V ′ )

Original: (for ﬁxed x i ).
Revised: (for ﬁxed x i ).

Revised: Condition ( ⋆ ) is critical both in proving the convergence of the algorithm as well as bounding the Rademacher complexity of the class of networks with bounded weights.

Original: Rather than working with a ﬁxed kernel, the interaction term enables us to use the ﬁrst layer for representing the input in a suitable feature space, which can be interpreted as picking a suitable kernel, then use the second layer to describe the output based on those features.
Revised: Rather than working with a ﬁxed kernel, the interaction term enables us to use the ﬁrst layer for representing the input in a suitable feature space, which can be interpreted as picking a suitable kernel, then use the second layer to describe the output based on those features.

Original: This is also indirectly encoded in our complexity measure.
Revised: This is also indirectly encoded in our complexity measure.

Revised: In addition to enforcing the orthogonality condition ( ⋆ ) (in the SGD variant), conditions for entering our regime are that the overparameterization m 1 , m 2 , m 3 and κ 1 , κ 2 are within a speciﬁc range with respect to each other.

Revised: We listed these relations in Appendix A.3.

Original: To illustrate the beneﬁt of going to this more involved regime, denote the class of neural nets with
Revised: To illustrate the beneﬁt of going to this more involved regime, denote the class of neural nets with

Original: Frobenius bounded norms ∥ W ′ ∥ ≤ γ 1 , ∥ V ′ ∥ ≤ γ 2 by G γ 1 ,γ 2 (and a bit more structure which we elaborate upon in the proofs); it turns out that G γ 1 ,γ 2 roughly includes H K ( O ( γ 1 γ 2 )) for every kernel K ∈ K , in the sense that each f ∈ H K ( O ( γ 1 γ 2 )) is well-approximated within G γ 1 ,γ 2 to arbitrarily small error on ﬁxed input (the error goes down with the size of the network).
Revised: Frobenius bounded norms ∥ W ′ ∥ ≤ γ 1 , ∥ V ′ ∥ ≤ γ 2 by G γ 1 ,γ 2 (and a bit more structure which we elaborate upon in the proofs); it turns out that G γ 1 ,γ 2 roughly includes H K ( O ( γ 1 γ 2 )) for every kernel K ∈ K , in the sense that each f ∈ H K ( O ( γ 1 γ 2 )) is well-approximated within G γ 1 ,γ 2 to arbitrarily small error on ﬁxed input (the error goes down with the size of the network).

Original: On the other hand, we show that the Rademacher Complexity (RC) of G γ 1 ,γ 2 behaves similar to the RC of the
Revised: On the other hand, we show that the Rademacher Complexity (RC) of G γ 1 ,γ 2 behaves similar to the RC of the

Original: NTK class H H ∞ ( O ( γ 1 γ 2 )) !
Revised: NTK class H H ∞ ( O ( γ 1 γ 2 )) !

Original: As our algorithm guarantees ﬁnding a network with sufﬁciently small empirical risk within G γ 1 ,γ 2 , this phenomenon underlies our adaptive generalization bounds.
Revised: As our algorithm guarantees ﬁnding a network with sufﬁciently small empirical risk within G γ 1 ,γ 2 , this phenomenon underlies our adaptive generalization bounds.

Original: Compared to previous work that provides an adaptive kernel analysis still for a two layer model (Dou & Liang, 2020) (although their analysis is for the gradient ﬂow and non-algorithmic), our model requires an additional layer so it can, in a sense, “simulate” the process of feature extraction in one layer to be used in the next layer.
Revised: Compared to previous work that provides an adaptive kernel analysis still for a two layer model (Dou & Liang, 2020) (although their analysis is for the gradient ﬂow and non-algorithmic), our model requires an additional layer so it can, in a sense, “simulate” the process of feature extraction in one layer to be used in the next layer.

Original: 3. C OMPARISON WITH K ERNEL FITTING
Revised: 3. C OMPARISON WITH K ERNEL FITTING

Original: Here we compare our generalization bounds with some kernel ﬁtting rates.
Revised: We compare our generalization bounds with some kernel ﬁtting rates.

Original: Given a kernel K with
Revised: Given a kernel K with

Original: K ( x, x ) ≤ 1 for every x : ∥ x ∥ ≤ 1 , suppose we want to ﬁt with the class H K ( B ′ ) of functions with K -RKHS norm bounded by B ′ .
Revised: K ( x, x ) ≤ 1 for every x : ∥ x ∥ ≤ 1 , suppose we want to ﬁt a function from H K ( B ′ ) , i.e. having KRKHS norm bounded by B ′ .

Original: In the realizable setting, when there is an underlying f ∗∗ ∈ H K ( B ′ ) with zero risk, Empirical Risk Minimization (ERM) enjoys a fast rate by exploiting the smoothness of the loss Srebro et al.
Revised: In the realizable setting, when there is an underlying f ∗∗ ∈ H K ( B ′ ) with zero risk, Empirical Risk Minimization (ERM) enjoys a fast rate using the smoothness of the loss (Srebro et al., 2010).

Original: With the the bound R ( H K ( B ′ )) ≤ O ( B ′ √ n ) on its Rademacher Complexity, one obtains:
Revised: The Rademacher Complexity bound R ( H K ( B ′ )) ≤ O ( B ′ √ n ) then implies:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 20) for the square loss, which is minimax optimal up to log factors.
Revised: for the square loss, which is minimax optimal up to log factors.

Original: To compare the neural network, we substitute f ∗∗ into Theorem 2.
Revised: To compare the neural network, we substitute f ∗∗ into Theorem 2.

Original: To relate the B in our bound to B ′ , assume for simplicity of exposition
Revised: To relate the B in our bound to B ′ , assume for simplicity of exposition

Original: that ∥ f ∥ ζ = ∥ f ∥ K ∗ for some K ∗ ∈ K (otherwise we can use a convergent sequence).
Revised: that ∥ f ∥ ζ = ∥ f ∥ K ∗ for some K ∗ ∈ K (otherwise we can use a convergent sequence).

Original: Observing that K { g } ( x, x ) ≤ 1 for every kernel K { g } ∈ K , we obtain that |
Revised: Observing that K { g } ( x, x ) ≤ 1 for every kernel K { g } ∈ K , we obtain that |

Original: f
Revised: f

Original: ∗∗ ( x ) | ≤ ∥ f ∗∗ ∥ K ∗ =
Revised: ∗∗ ( x ) | ≤ ∥ f ∗∗ ∥ K ∗ =

Original: ∥
Revised: ∥

Original: f ∗∗ ∥ ζ (Appendix A.7).
Revised: f ∗∗ ∥ ζ (Appendix A.6).

Original: Combining this fact with the realizability assumption, we can then upper bound the parameter B in Theorem 2 by ∥
Revised: Combining this fact with the realizability assumption, we can then upper bound the parameter B in Theorem 2 by ∥

Original: f ∗∗ ∥ ζ , and obtain
Revised: f ∗∗ ∥ ζ , and obtain

Original: [Equation]
Revised: [Equation]

Original: If we further take K to be in K , then Equation (21) combined with ∥
Revised: If we further take K to be in K , then Equation (21) combined with ∥

Original: f ∗∗ ∥ ζ ≤ ∥ f ∗∗ ∥ K ≤ B ′ implies:
Revised: f ∗∗ ∥ ζ ≤ ∥ f ∗∗ ∥ K ≤ B ′ implies:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: that is, for every kernel K ∈ K , our deep learning approach almost achieves the conventional kernel bound in Equation (20).
Revised: that is, for every kernel K ∈ K , our deep learning approach almost achieves the conventional kernel bound in Equation (20).

Original: Repeating the uniform risk bound stated in Theorem 1 in (Srebro et al., 2010) for H K ( B ′ )
Revised: Repeating the uniform risk bound stated in Theorem 1 in (Srebro et al., 2010) for H K ( B ′ )

Original: where
Revised: where

Original: B ′ is set to all powers of two, followed by a union bound, one can easily obtain adata-dependent fast rate of 
Revised: B ′ is set to all powers of two, followed by a union bound, one can easily obtain a fast rate of [Equation]

Original: [Equation]
Revised: [Equation]

Original: for the solution of KRLR in the general case (not realizable) for the square loss.
Revised: for the solution of KRLR in the general case (not realizable) for the square loss.

Original: On the other hand, for a B -bounded Lipschitz loss, we instead get a slow rate for KRLR:
Revised: On the other hand, for a B -bounded Lipschitz loss, we instead get a slow rate for KRLR:

Original: [Equation]
Revised: [Equation]

Original: where B is an a.s. bound on | y | as before.
Revised: where B is an a.s. bound on | y | as before.

Original: This bound is similar to Arora et al. (2019a).
Revised: This bound is similar to Arora et al. (2019a).

Original: Note that our data dependent generalization bound in Theorem 1 already achieves the fast rate for KRLR in (23) for any K ∈ K .
Revised: Note that our data dependent generalization bound in Theorem 1 already achieves the fast rate for KRLR in (23) for any K ∈ K .

Original: Finally, in the non-realizable case, we still have the following fast rate for ERM regarding the hypothesis class H K ( B ′ )
Revised: Finally, in the non-realizable case, we still have the following fast rate for ERM regarding the hypothesis class H K ( B ′ )

Original: (Srebro et al., 2010):
Revised: (Srebro et al., 2010):

Original: [Equation]
Revised: [Equation]

Original: where now f ∗∗ := argmin f ∈H K ( B ′ )
Revised: where now f ∗∗ := argmin f ∈H K ( B ′ )

Original: R ( f ) , while Theorem 2 also implies (again for every K ∈ K ):
Revised: R ( f ) , while Theorem 2 also implies (again for every K ∈ K ):

Original: [Equation]
Revised: [Equation]

Original:  A LGORITHM : P ROJECTED S TOCHASTIC G RADIENT D ESCENT
Revised:  A LGORITHM : P ROJECTED S TOCHASTIC G RADIENT D ESCENT

Original: Next, we discuss the algorithm that achieves the generalization bounds above.
Revised: Next, we discuss the algorithm that achieves the generalization bounds above.

Original: Two speciﬁc challenges are to address the interaction of layers, and to establish that the algorithm remains in a regime in which it converges to a good solution that generalizes.
Revised: Two speciﬁc challenges are to address the interaction of layers, and to establish that the algorithm remains in a regime in which it converges to a good solution that generalizes.

Original: In this section, we describe our algorithm PSGD , presented as pseudocode in Figure 1, which is roughly Stochastic Gradient Descent modiﬁed to project out a low-dimensional random subspace from the second-layer weights.
Revised: In this section, we describe our algorithm PSGD , presented as pseudocode in Figure 1, which is roughly Stochastic Gradient Descent modiﬁed to project out a low-dimensional random subspace from the second-layer weights.

Original: PSGD roughly runs SGD on a smoothed version of the following loss function ( ψ 1 , ψ 2 are deﬁned in Appendix A.2)
Revised: PSGD roughly runs SGD on a smoothed version of the following loss function ( ψ 1 , ψ 2 are deﬁned in Appendix A.2)

Original: [Equation]
Revised: [Equation]

Original: Compared to standard SGD, our algorithm makes two modiﬁcations: (1) it uses randomized smoothing to alleviate the non-smoothness of the ReLus, (2) it ensures that the weights in the second layer are orthogonal to the data features ϕ (0) ( x ) computed by the ﬁrst layer at initialization.
Revised: Compared to standard SGD, our algorithm makes two modiﬁcations: (1) it uses randomized smoothing to alleviate the non-smoothness of the ReLus, (2) it ensures that the weights in the second layer are orthogonal to the data features ϕ (0) ( x ) computed by the ﬁrst layer at initialization.

Original: This helps to control layer interactions as pointed out in Section 3.3.
Revised: This helps to control layer interactions as pointed out in Section 3.3.

Original: For smoothing, we add Gaussian smoothing matrices W ρ and V ρ to the weights with i.i.d.
Revised: For smoothing, we add Gaussian smoothing matrices W ρ and V ρ to the weights with i.i.d.

Original: entries from N (0 , β 21 /m 1 ) and N (0 , β 22 /m 2 ) respectively, for β 

Revised: entries from N (0 , β 21 /m 1 ) and N (0 , β 22 /m 2 ) respectively, for β

Original: =
Revised: =

Original: O p (( κ 1 √ m 1 ) − 1 ( κ 2 √ m 2 ) − 2 / 3 ) , β 1 =
Revised: O p (( κ 1 √ m 1 ) − 1 ( κ 2 √ [Equation] [Equation] [Equation]

Revised: [Equation]

Original: O p ( m 23 κ 2 √ m 2 ( κ 1 √ m 1 ) − 1 ) .
Revised: [Equation] p ( m 23 κ 2 √ m 2 ( κ 1 √ [Equation] 1 ) − 1 ) .

Original: To simplify the exposition, O p ( . ) is hiding the dependencies on the basic parameters B, n, 1 /λ 0 and log factors.
Revised: To simplify the exposition, O p ( . ) is hiding the dependencies on the basic parameters B, n, 1 /λ 0 and log factors.

Original: Our convergence proof uses the loss with respect to this smoothed network.
Revised: Our convergence proof uses the loss with respect to this smoothed network.

Original: For the projection, let Φ ⊥ ⊂
Revised: For the projection, let Φ ⊥ ⊂

Original: R m 2 × m 3 be the subspace of weights of the second layer whose rows are orthogonal to the ﬁrst-layer data representations ϕ
Revised: R m 2 × m 3 be the subspace of weights of the second layer whose rows are orthogonal to the ﬁrst-layer data representations ϕ

Original: (0) ( x i ) ’s ∀ i ∈ [ n ] at initialization
Revised: (0) ( x i ) ’s ∀ i ∈ [ n ] at initialization:

Original: [Equation]
Revised: [Equation]

Original: In summary, at point ( W ′ , V ′ ) , the algorithm samples a random ( x i , y i ) from the data, as well as smoothing matrices W ρ, 1 , V ρ, 1 , W ρ, 2 , V ρ, 2 .
Revised: In summary, at point ( W ′ , V ′ ) , the algorithm samples a random ( x i , y i ) from the data, as well as smoothing matrices W ρ, 1 , V ρ, 1 , W ρ, 2 , V ρ, 2 .

Original: It then computes an unbiased estimate for the gradient ( ˆ ∇ W , ˆ ∇ V ) , adds additional normalized Gaussian noise matrices Ξ 1 , Ξ 2 and moves in this direction with step size η = 1 / poly ( n, B ∨ 1 /B, 1 /λ 0 ) : 
Revised: It then computes an unbiased estimate for the gradient ( ˆ ∇ W , ˆ ∇ V ) , adds additional normalized Gaussian noise matrices Ξ 1 , Ξ 2 and moves in this direction with step size η = 1 / poly ( n, B ∨ 1 /B, 1 /λ 0 ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: Parameters.
Revised: Parameters.

Original: Our results apply to the overparametrization regime, when the size of the network, i.e. parameters m 1 , m 2 , m 3 is polynomially large in n, B ∨ 1 /B, 1 /λ 0 .
Revised: Our results apply to the overparametrization regime, when the size of the network, i.e. parameters m 1 , m 2 , m 3 is polynomially large in n, B ∨ 1 /B, 1 /λ 0 .

Original: This ensures that we can guarantee that the network has good enough function representation capacity, and PSGD is able to ﬁnd a good local direction in every iteration.
Revised: This ensures that we can guarantee that the network has good enough function representation capacity, and PSGD is able to ﬁnd a good local direction in every iteration.

Original: The regularization coefﬁcients ψ 1 , ψ 2 can be set with respect to any candidate ( f ∗ , G ) for our complexity measure (9).
Revised: The regularization coefﬁcients ψ 1 , ψ 2 can be set with respect to any candidate ( f ∗ , G ) for our complexity measure (9).

Original: In Appendix A.2, we introduce a simple doubling trick which handles when we do not have access to an optimal candidate solution.
Revised: In Appendix A.2, we introduce a simple doubling trick which handles when we do not have access to an optimal candidate solution.

Original: With such an f ∗ , deﬁne ν := max { R n ( f ∗ ) / 2 , B 2 /n } , and set ψ 1 = ν/ 4 , ψ 2 = ν/ (4 ζ ( f ∗ , G )) .
Revised: With such an f ∗ , deﬁne ν := max { R n ( f ∗ ) / 2 , B 2 /n } , and set ψ 1 = ν/ 4 , ψ 2 = ν/ (4 ζ ( f ∗ , G )) .

Original: We list the suitable regime for our overparameterization in Appendix A.3.
Revised: We list the suitable regime for our overparameterization in Appendix A.3.

Original: Algorithm Input: network architecture m 1 , m 2 , m 3 , initialization parameters κ 1 , κ 2 , smoothing parameters β 1 , β 2 , training set ( x i , y i ) ni =1 , label parameter B , ( f ∗ , G ) from the complexity measure 1: Gaussian initialization W (0) j,k ← N (0 , κ 1 ) , V (0) j,k ← N (0 , κ 2 ) 2: Deﬁne parameters ψ 1 , ψ 2 , ν , η , subspace Φ ⊥ , and objective L 1 as described in Section3: while L 1 ( W ′ , V ′ ) >R n ( f ∗ )
Revised: (0) j,k ← N (0 , κ 1 ) , V (0) j,k ← N (0 , κ 2 ) 2: Deﬁne parameters ψ 1 , ψ 2 , ν , η , subspace Φ ⊥ , and objective L 1 as described in Section3: while L 1 ( W ′ , V ′ ) >

Original:  + 2 ν do 4:
Revised: R n ( f ∗ ) + 2 ν do 4:

Original: Gaussian matrices W ρ, 1 j,k , W ρ, 2 j,k ← N (0 , β 21 m 1 ) , V ρ, 1 j,k , V ρ, 2 j,k ← N (0 , β 22 m 2 ) 5:
Revised: Gaussian matrices W ρ, 1 j,k , W ρ, 2 j,k ← N (0 , β 21 m 1 ) , V ρ, 1 j,k , V ρ, 2 j,k ← N (0 , β 22 m 2 ) 5:

Original: Sample data ( x i , y i ) uniformly at random 6:
Revised: Sample data ( x i , y i ) uniformly at random 6:

Original: Compute gradient estimates 7:(
Revised: Compute gradient estimates 7:

Original: [Equation]
Revised: [Equation]

Original: ρ, 1 ( x i ) , y i ) ∇ W f W ′

Original: + W ρ, 2 ,V ′

Original: + V ρ, 2 ( x i )

Original: + 2 ψ 1 W ′ , ˆ ∇ V

Original: = ˙ ℓ ( f W ′ + W ρ, 1 ,V ′

Original: + V ρ, 1 ( x i ) , y i ) ∇ V f W ′

Original: + W ρ, 2 ,V ′

Original: + V ρ, 2 ( x i )

Original: + 2 ψ 2 V ′ 8:

Original: Move as ( W ′ , V ′ )
Revised: Move as ( W ′ , V ′ )

Original: ← ( W ′ , V ′ )
Revised: ← ( W ′ , V ′ )

Original: + η(cid:16) ˆ ∇ W + Ξ 1 / ( √ m 1 ∥ Ξ 1 ∥ ) , Proj Φ ⊥ ( ˆ ∇ V + Ξ 2 / ∥
Revised: + η

Original: Ξ 2 ∥ ) (cid:17) 9: Return ( W ′ , V ′ )
Revised: [Equation] ) (cid:17) 9: Return ( W ′ , V ′ )

Original:  H IGH L EVEL I DEA OF THE PSGD A NALYSIS
Revised:  H IGH L EVEL I DEA OF THE PSGD A NALYSIS

Original: Aside from the network’s representation ability, a major challenge is the poly-time computation of some good weights that generalize.
Revised: Aside from the network’s representation ability, a major challenge is the poly-time computation of some good weights that generalize.

Original: On the one hand, we want the weights to remain close to their initialization so the ﬁnal network is in the class G γ 1 ,γ 2 for suitably chosen γ 1 , γ 2 , where recall that G γ 1 ,γ 2 is roughly the class of nets with bounded norms ∥ W ′ ∥ ≤ γ 1 ,
Revised: On the one hand, we want the weights to remain close to their initialization so the ﬁnal network is in the class G γ 1 ,γ 2 for suitably chosen γ 1 , γ 2 , where recall that G γ 1 ,γ 2 is roughly the class of nets with bounded norms ∥ W ′ ∥ ≤ γ 1 ,

Original: ∥ V ′ ∥ ≤ γ 2 .
Revised: ∥ V ′ ∥ ≤ γ 2 .

Original: On the other hand, we aim to reduce the nonconvex empirical loss R n ( f W ′ ,V ′ ) .
Revised: On the other hand, we aim to reduce the nonconvex empirical loss R n ( f W ′ ,V ′ ) .

Original: Aligned with these goals, we roughly run SGD on our smoothed objective consisting of the empirical risk regularized by the Frobenius norm of the weight matrices.
Revised: Aligned with these goals, we roughly run SGD on our smoothed objective consisting of the empirical risk regularized by the Frobenius norm of the weight matrices.

Original: We prove convergence by building on ideas from Allen-Zhu et al.
Revised: We prove convergence by building on ideas from Allen-Zhu et al.

Original: (2018), with a framework based on the classic result that SGD can escape saddle points in face of nonconvexity.
Revised: (2018), with a framework based on the classic result that SGD can escape saddle points in face of nonconvexity.

Original: Compared to (AllenZhu et al., 2018), we take a different approach driven by our purely data-dependent complexity measure in Theorem 1.
Revised: Compared to (AllenZhu et al., 2018), we take a different approach driven by our purely data-dependent complexity measure in Theorem 1.

Original: We augment this by a careful Rademacher complexity analysis in Appendix A. for Frobenius norm bounded networks in our regime, to precisely get the same behavior as the RC bound for the class of NTK-norm bounded functions (Arora et al., 2019a).
Revised: We augment this by a careful Rademacher complexity analysis in Appendix A. for Frobenius norm bounded networks in our regime, to precisely get the same behavior as the RC bound for the class of NTK-norm bounded functions (Arora et al., 2019a).

Original: We face two challenges in the proof: ﬁrst, to have an SGD-type algorithm, we cannot directly guide the optimization trajectory based on our complexity measure, so we have little control on the random trajectory of PSGD .
Revised: We face two challenges in the proof: ﬁrst, to have an SGD-type algorithm, we cannot directly guide the optimization trajectory based on our complexity measure, so we have little control on the random trajectory of PSGD .

Original: Therefore, we mainly rely on uniform convergence techniques in our convergence proof.
Revised: Therefore, we mainly rely on uniform convergence techniques in our convergence proof.

Original: In particular, we prove it is unlikely that PSGD reaches a “bad" saddle point if the objective is above some threshold.
Revised: In particular, we prove it is unlikely that PSGD reaches a “bad" saddle point if the objective is above some threshold.

Original: Second, to prove the last fact, we require a weak regularity condition that the value of the objective does not exceed a certain threshold during the algorithm.
Revised: Second, to prove the last fact, we require a weak regularity condition that the value of the objective does not exceed a certain threshold during the algorithm.

Original: This puts us in a different setting compared to the traditional result on escaping saddle points (Ge et al., 2015a), so we need to reﬁne their analysis for our setting.
Revised: This puts us in a different setting compared to the traditional result on escaping saddle points (Ge et al., 2015a), so we need to reﬁne their analysis for our setting.

Original: We give a high-level sketch of our proof in Section 5.
Revised: We give a high-level sketch of our proof in Section 5.

Original: Below, we outline the steps of the convergence proof at a high level.
Revised: Below, we outline the steps of the convergence proof at a high level.

Original: Construction of a good Network To study the loss landscape, similar to (Allen-Zhu et al., 2018), we show the existence of a good local update for PSGD at reasonable points ( W ′ , V ′ ) , using the ideal weight matrices ( W ∗ , V ∗ ) that we carefully construct from our complexity measure.
Revised: Construction of a good Network To study the loss landscape, similar to (Allen-Zhu et al., 2018), we show the existence of a good local update for PSGD at reasonable points ( W ′ , V ′ ) , using the ideal weight matrices ( W ∗ , V ∗ ) that we carefully construct from our complexity measure.

Original: Here, we sketch our proof for constructing ( W ∗ , V ∗ ) .
Revised: Here, we sketch our proof for constructing ( W ∗ , V ∗ ) .

Original: Let ( W ′ , V ′ ) be the current weights of the algorithm.
Revised: Let ( W ′ , V ′ ) be the current weights of the algorithm.

Original: Fix a sample i ∈ [ n ] .
Revised: Fix a sample i ∈ [ n ] .

Original: In Appendix A.12, we use G to construct W ∗ for the ﬁrst layer weights with decomposition W ∗ =
Revised: In Appendix A.12, we use G to construct W ∗ for the ﬁrst layer weights with decomposition W ∗ =

Original: P m 3 k =1 W ∗ k and O (1) bounded norm, such that ϕ ∗ ( x i ) k 
Revised: P m 3 k =1 W ∗ k and O (1) bounded norm, such that ϕ ∗ ( x i ) k :=

Original: = 1 √ m 1 W s Sgn (cid:16) ( W (0) + W ′ ) x i (cid:17)
Revised: [Equation]

Original: W ∗ x i .
Revised: W [Equation] x i .

Original: This decomposition ensures that for every k, k ′ ∈ [ m 3 ] , negating the matrix W ∗ k only negates ϕ ∗ ( x i ) k ′ when k ′ =
Revised: This decomposition ensures that for every k, k ′ ∈ [ m 3 ] , negating the matrix W ∗ k only negates ϕ ∗ ( x i ) k ′ when k ′ =

Original: k and has no effect on ϕ ∗ ( x i )
Revised: k and has no effect on ϕ ∗ ( x i )

Original: k ′ for k ′ ̸ = k .
Revised: k ′ for k ′ ̸ = k .

Original: This way, we can easily generate any arbitrary sign ﬂip of the entries of ϕ ∗ ( x i ) .
Revised: This way, we can easily generate any arbitrary sign ﬂip of the entries of ϕ ∗ ( x i ) .

Original: We use this property to generate a suitable random descent direction.
Revised: We use this property to generate a suitable random descent direction.

Original: Next, we want to construct a suitable weight matrix V ∗ for the second layer which maps the features ϕ ∗ ( x i ) into f ∗ i (recall the deﬁnition of the complexity measure).
Revised: Next, we want to construct a suitable weight matrix V ∗ for the second layer which maps the features ϕ ∗ ( x i ) into f ∗ i (recall the deﬁnition of the complexity measure).

Original: The key here is that we consider a regime where the norm of ϕ (0) ( x i ) is typically larger than that of ϕ ′ ( x i ) and ϕ ∗ ( x i ) , so  that the sign pattern in the second layer is determined by ϕ (0) ( x i ) in most of the rows.
Revised: The key here is that we consider a regime where the norm of ϕ (0) ( x i ) is typically larger than that of ϕ ′ ( x i ) and ϕ ∗ ( x i ) , so it is very likely that the sign pattern in the second layer is determined by ϕ (0) ( x i ) in most of the rows.

Original: In such a senario,  the interaction of V ′ with ϕ (0) ( x i ) becomes problematic both for the generalization and optimization.
Revised: In such a senario, having the condition V ′ j ⊥ ϕ (0) ( x i ) becomes vital as the interaction of V ′ with ϕ (0) ( x i ) becomes problematic both for the generalization and optimization.

Original: From the standpoint of generalization, without excluding this interaction, one can exploit the large size of ϕ (0) ( x i ) and build a network within the class G γ 1 ,γ 2 (i.e., with ∥ W ′ ∥ , ∥ V ′ ∥ suitably bounded) corresponding to a complex function which overﬁts the data.
Revised: From the standpoint of generalization, without excluding this interaction, one can exploit the large size of ϕ (0) ( x i ) and build a network within the class G γ 1 ,γ 2 (i.e., with ∥ W ′ ∥ , ∥ V ′ ∥ suitably bounded) corresponding to a complex function which overﬁts the data.

Original: Indeed, we utilize the large magnitude of ϕ (0) and its orthogonality to the rows of V ′ in our RC bound.
Revised: Indeed, we utilize the large magnitude of ϕ (0) and its orthogonality to the rows of V ′ in our RC bound.

Original: On the other hand, mainly because changing the weights of the ﬁrst layer cannot affect ϕ (0) ( x i ) , the interaction of V ′ with ϕ (0) ( x i ) is problematic in our convergence analysis for optimization part, particularly in proving the existence of a local descent direction.
Revised: On the other hand, mainly because changing the weights of the ﬁrst layer cannot affect ϕ (0) ( x i ) , the interaction of V ′ with ϕ (0) ( x i ) is problematic in our convergence analysis for optimization part, particularly in proving the existence of a local descent direction.

Original: This is the rationale behind our orthogonality constraint (24).
Revised: This is the rationale behind our orthogonality constraint (24).

Original: Finally, the ϕ ∗ ( x i ) ’s, the above control on the signs, and the fact that ⟨ ϕ (0)
Revised: Finally, the ϕ ∗ ( x i ) ’s, the above control on the signs, and the fact that ⟨ ϕ (0)

Original: ( x i 1 ) , ϕ (0) ( x i 2 ) ⟩ concentrates around m 3 E w ∼ N (0 ,κ 1 I )
Revised: ( x i 1 ) , ϕ (0) ( x i 2 ) ⟩ concentrates around m 3 E w ∼ N (0 ,κ 1 I )

Original: [ σ ( w T x i 1 ) σ ( w T x i 2 )] which recovers the structure of the kernel K ∞ (Section 2), give rise to the kernel G ⊙ K ∞ in the second layer.
Revised: [ σ ( w T x i 1 ) σ ( w T x i 2 )] which recovers the structure of the kernel K ∞ (Section 2), give rise to the kernel G ⊙ K ∞ in the second layer.

Original: Using this structure, we constructsuch V ∗ that maps ϕ ∗ ( x i ) ’s to f ∗ i ’s, which hassome additional good properties including O ( f ∗ T ( G ⊙ K ∞ ) − 1 f ∗ ) 
Revised: Using this structure, we construct V ∗ that maps ϕ ∗ ( x i ) ’s to f ∗ i ’s, which has additional good properties, including O ( f ∗ T ( G ⊙ K ∞ ) − 1 f ∗ ) - bounded norm, and rows that are orthogonal to ϕ (0) ( x i ) ’

Original: -bounded norm and rows that are orthogonal to ϕ (0) ( x i ) ’s. For more details, see Appendix A.12.
Revised: s. For more details, see Appendix A.12.

Original: Nonexistence of Bad Saddle Points Next, we want to exploitthe construction ( W ∗ , V ∗ )from the previous part to prove the existence of a good direction along which  the objectivedecreases locally.
Revised: Nonexistence of Bad Saddle Points Next, we want to exploit ( W ∗ , V ∗ ) to prove the existence of a good direction along which reduces the objective locally.

Original: Moving along ( W ∗ , V ∗ ) is the ﬁrst idea which failsin reducing the loss as the cross terms created between W ′ , V ∗ and V ′ , W ∗ cannot be bounded effectively.
Revised: Moving along ( W ∗ , V ∗ ) is the ﬁrst idea which fails as the cross terms created between W ′ , V ∗ and V ′ , W ∗ cannot be bounded effectively.

Original: Instead, we randomly perturb W ∗ and V ∗ in a coupled way and prove a reduction in expectation.
Revised: Instead, we randomly perturb W ∗ and V ∗ in a coupled way and prove a reduction in expectation.

Original: Next, we elaborate more upon this suitable random direction.
Revised: We elaborate more on this suitable random direction.

Original: Multiplying random signs Σ k onto W ∗ k , we deﬁne the sum W ∗ Σ
Revised: Multiplying random signs Σ k onto W ∗ k , we deﬁne the sum W ∗ Σ

Original: = P m 3 k =1 Σ k W ∗ k .
Revised: = P m 3 k =1 Σ k W ∗ k .

Original: We also multiply the same signs to the columns of V ∗ and project it back onto Φ ⊥ to obtain V ∗ Σ .
Revised: We also multiply the same signs to the columns of V ∗ and project it back onto Φ ⊥ to obtain V ∗ Σ .

Original: Then, we move in the random direction ( √ ηW ∗ Σ − ηW/ 2 , √ ηV ∗ Σ − ηV/ 2) ; this update creates a number of extra cross terms in the objective which we have to bound to prove a local reduction argument.
Revised: Then, we move in the random direction ( √ ηW ∗ Σ − ηW/ 2 , √ ηV ∗ Σ − ηV/ 2) ; this update creates additional cross terms in the objective that we must bound to prove a local reduction argument.

Original: A key point here is that we prove that with high probability, the norm of the matrices remainsuniformly bounded along the algorithm, i.e. ∥
Revised: A key point here is that we prove that with high probability, the norm of the weights remains bounded along the algorithm, i.e. ∥

Original: W ′ ∥
Revised: W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 .
Revised: ≤ C 2 .

Original: This norm restriction enables us to substitute terms that we do not have control over by their worst-case supremum counterparts.
Revised: This norm restriction enables us to substitute terms that we do not have control over by their worst-case supremum counterparts.

Original: We refer to Appendix A.13 for similar techniques.
Revised: We refer to Appendix A.13 for similar techniques.

Original: Convergence of PSGD
Revised: Convergence of PSGD

Original: Finally, we use the fact that SGD escapes good saddle points (Ge et al., 2015b).
Revised: Finally, we use the fact that SGD escapes good saddle points (Ge et al., 2015b).

Original: For proving the existence of a good random direction to escape saddle points above, we use that the norm of weights is uniformly bounded along all iterations; this bound, in fact, is looser than the bound that we show for the ﬁnal weights of the network.
Revised: For proving the existence of a good random direction to escape saddle points above, we use that the norm of weights is uniformly bounded along all iterations; this bound, in fact, is looser than the bound that we show for the ﬁnal weights of the network.

Original: Yet, this additional restriction cannot be addressed by the classical nonconvex theory of SGD.
Revised: Yet, this additional restriction cannot be addressed by the classical nonconvex theory of SGD.

Original: Consequently, we reﬁne and adapt the proof of (Ge et al., 2015b) to incorporate this additional constraint.
Revised: Consequently, we reﬁne and adapt the proof of (Ge et al., 2015b) to incorporate this additional constraint.

Original: At a high level, Ge et al.
Revised: At a high level, Ge et al.

Original: (2015b) work with a supermartingale based on the loss value.
Revised: (2015b) work with a supermartingale based on the loss value.

Original: To guarantee the additional norm restriction, it is initially tempting to apply Azuma-Hoeffding concentration to bound the upward deviations of this process.
Revised: To guarantee the additional norm restriction, it is initially tempting to apply Azuma-Hoeffding concentration to bound the upward deviations of this process.

Original: However, this fails as the process has a two-fold behavior, depending on how large the gradient is at the current point.
Revised: However, this fails as the process has a two-fold behavior, depending on how large the gradient is.

Original: At the core of our reﬁnement proof here, we instead directly bound the MGF of the martingale using Doobs maximal inequality.
Revised: At the core of our reﬁnement proof here, we instead directly bound the MGF of the martingale using Doobs maximal inequality.

Original: We refer to Section A.16 for more details.
Revised: We refer to Section A.16 for more details.

Original: A A PPENDIXC ONTENTS
Revised: A A PPENDIX

Revised: Below is the list of different sections of the proof.

Revised: We have a separate Appendix list for lower level

Original:  A.1 Stronger Generalization bounds for polynomials. . . . . . . . . . . . . . . . . . .
Revised: Lemmas in our proof in B. C ONTENTS A.1 Stronger Generalization bounds for polynomials

Original: A.2 The Doubling Trick. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 The Doubling Trick

Original: A.3 Amount of Overparameterization. . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.3 Amount of Overparameterization

Original: A.4 PSD property of K ∞. .
Revised: A.4 PSD property of K ∞

Original: A.5 Complexity upper bound. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.5 Complexity upper bound

Original: A.6 Structure of the proof, setting m 3 , and further deﬁnitions . . . . . . . . . . . . . .

Original: A.7 Complexity measure and the ζ -norm. . .
Revised: A.6 Complexity measure and the ζ -norm

Original: A.8 Core Generalization Result. .
Revised: A.7 Core Generalization Result

Original:  A.9 Proof of Theorem2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.8 Structure of the proof, setting m 3 , and further deﬁnitions A.9 Proof of Theorem

Original: A.10 Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.10 Optimization

Original: A.11 Rademacher Complexity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.11 Rademacher Complexity

Original: A.12 Constructing W ∗ , V ∗ . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.12 Constructing W ∗ , V ∗ . . .

Original: A.12.1 First Layer, Construction of W ∗ . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.12.1 First Layer, Construction of W ∗ . . . . . . . . . . . . . . . . . . . . . . .

Original: A.12.2 Second Layer, Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . .
Revised: A.12.2 Second Layer, Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . .

Original: A.12.3 Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.12.3 Construction of V ∗ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.13 Existence of a good direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.13 Existence of a good direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.14 Existence of a good direction Helper Lemmas . . . . . . . . . . . . . . . . . . . .
Revised: A.14 Existence of a good direction Helper Lemmas . . . . . . . . . . . . . . . . . . . .

Original: A.15 Bounding the worst-case Senario . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.15 Bounding the worst-case Senario . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.16 Convergence . . .
Revised: A.16 Convergence . . .

Original: A.17 Process from a higher view: deﬁnition of the ( X ) sequence . .
Revised: A.17 Process from a higher view: deﬁnition of the ( X ) sequence . .

Original: A.18 Bounding the MGF of X i ’s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.18 Bounding the MGF of X i ’s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.19 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.19 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.20 Gaussian Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.20 Gaussian Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.20.1 Setting β 1 and β 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.20.1 Setting β 1 and β 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.21 Basic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.21 Basic Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.21.1 Deﬁning the rare events E j . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.21.1 Deﬁning the rare events E j . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.21.2 Bounding the value of f ′ . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.21.2 Bounding the value of f ′ . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.21.3 Bounding the difference between Original and Smoothed Functions . . . .
Revised: A.21.3 Bounding the difference between Original and Smoothed Functions . . . .

Original: A. S TRONGER G ENERALIZATION BOUNDS FOR POLYNOMIALS
Revised: A. S TRONGER G ENERALIZATION BOUNDS FOR POLYNOMIALS

Original: In this section, we prove an explicit generalization bound for functions represented as a polynomial sum.
Revised: In this section, we prove an explicit generalization bound for functions represented as a polynomial sum.

Original: Note that the bounds in Arora et al.
Revised: Note that the bounds in Arora et al.

Original: (2019a) for polynomials assume the monomials with degree larger than one to have even powers, while here we do not impose this restriction.
Revised: (2019a) for polynomials assume the monomials with degree larger than one to have even powers, while here we do not impose this restriction.

Original: In addition, note that despite Arora et al. (2019a), our bounds remain meaningful in the noisy case (recall our
Revised: In addition, note that despite Arora et al. (2019a), our bounds remain meaningful in the noisy case (recall our

Original: Theorem 2).
Revised: Theorem 2).

Original: More speciﬁcally, we bound the ζ norm of such functions.
Revised: More speciﬁcally, we bound the ζ norm of such functions.

Original: Consider the target function s with the following power series formula:
Revised: Consider the target function s with the following power series formula:

Original: [Equation]
Revised: [Equation]

Original: where a p ∈ R and w p ∈ R d .
Revised: where a p ∈ R and w p ∈ R d .

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: 27) where x k denotes the k th entry of vector x here and
Revised: 27) where x k denotes the k th entry of vector x here and

Original: [Equation]
Revised: [Equation]

Original: and for all k ∈ [ d ] :
Revised: and for all k ∈ [ d ] :

Original: [Equation]
Revised: [Equation]

Original: Then, using the tailor series of x ( 14 + arcsin ( x ) 2 π )
Revised: Then, using the tailor series of x ( 14 + arcsin ( x ) 2 π )

Original: =
Revised: =

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: p =
Revised: p =

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: x p for | x | ≤ 1 , the RKHS space H ( H ∞ ) of NTK can be identiﬁed by square-summable sequences of reals ( a p ′ )
Revised: x p for | x | ≤ 1 , the RKHS space H ( H ∞ ) of NTK can be identiﬁed by square-summable sequences of reals ( a p ′ )

Original: ∞ p ′
Revised: ∞ p ′

Original: =1 with dot product
Revised: =1 with dot product

Original: [Equation]
Revised: [Equation]

Original: where λ ( p ′ ) :
Revised: where λ ( p ′ ) :

Original: Z ≥ 

Revised: Z ≥

Original: → Z ≥ 0 such that it maps zero to zero, the ﬁrst d positive integers are mapped to one, the next d 2 ones are mapped to 2 , etc.
Revised: → Z ≥ 0 such that it maps zero to zero, the ﬁrst d positive integers are mapped to one, the next d 2 ones are mapped to 2 , etc.

Original: Moreover, the RKHS mapping Ψ : R d → H ( H ∞ ) from the Euclidean space is:
Revised: Moreover, the RKHS mapping Ψ : R d → H ( H ∞ ) from the Euclidean space is:

Original: [Equation]
Revised: [Equation]

Original: where x ′ = x/ ∥ x ∥ and in the notation above we are presenting a sequence of sequences, by which we mean the inner sequences simply unfold.
Revised: where x ′ = x/ ∥ x ∥ and in the notation above we are presenting a sequence of sequences, by which we mean the inner sequences simply unfold.

Original: Using this identiﬁcation, one can see using the linear representations of g 1 , g k 2 in H :
Revised: Using this identiﬁcation, one can see using the linear representations of g 1 , g k 2 in H :

Original: [Equation]
Revised: [Equation]

Original: Summing above and noting the linear representation of g :
Revised: Summing above and noting the linear representation of g :

Original: [Equation]
Revised: [Equation]

Original: Now for { g } := { g ′ k } d +1 k
Revised: Now for { g } := { g ′ k } d +1 k

Original: =
Revised: =

Original: := { g 1 } ∪ { g k 2 } dk =1 , we consider the kernel K { g } .
Revised: := { g 1 } ∪ { g k 2 } dk =1 , we consider the kernel K { g } .

Original: Expanding the tailor series of F 2 (2 F 3 )( x )  = P ∞ p =0 µ p x p , we ﬁnd the identiﬁcation ( h kp ′ )
Revised: Expanding the tailor series of F 2 (2 F 3 )( x ) j = P ∞ p =0 µ p x p , we ﬁnd the identiﬁcation ( h kp ′ )

Original: k ∈ [ d +1] ,p ′ =
Revised: k ∈ [ d +1] ,p ′ =

Original: 0 ,..., ∞ with dot product=
Revised: 0 ,..., ∞ with dot product

Original: [Equation]
Revised: [Equation]

Original: with RKHS map
Revised: with RKHS map

Original: [Equation]
Revised: [Equation]

Original: Now [Equation] we compute the norm of function s with respect to K { g } , combining the above representation and dot product with Equation (27) and the fact that we work with unit norm x , so x ′ =
Revised: Now, we compute the norm of function s with respect to K { g } , combining the above representation and dot product with Equation (27) and the fact that we work with unit norm x , so x ′ =

Original: x :
Revised: x :

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging the above and Equation (30) into the deﬁnition of ∥ .
Revised: Plugging the above and Equation (30) into the deﬁnition of ∥ .

Original: ∥ ζ in ( ?? ) [Equation] we conclude
Revised: ∥ ζ in (17), we conclude

Original: [Equation]
Revised: [Equation]

Original: Note that if the odd exponents (except possibly one) in the deﬁnition of s in (26) are zero, then we could consider only the function g 1 and kernel K g 1 , which would have implied a bound of µ
Revised: Note that if the odd exponents (except possibly one) in the deﬁnition of s in (26) are zero, then we could consider only the function g 1 and kernel K g 1 , which would have implied a bound of µ

Original: P ∞
Revised: P ∞

Original: [Equation]
Revised: [Equation]

Original: A. T HE D OUBLING T RICK
Revised: A. T HE D OUBLING T RICK

Original: For the SGD optimization [Equation] we set the regularization coefﬁcients in the loss L 1 as
Revised: For the SGD optimization, we set the regularization coefﬁcients in the loss L 1 as

Original: [Equation]
Revised: [Equation]

Original: , (33) with ν := max { R n ( f ∗ ) / 2 , B 2 /n } .
Revised: 33) with ν := max { R n ( f ∗ ) / 2 , B 2 /n } .

Original: This assumes we know the f ∗ and G that minimize the adaptation within the complexity measure (12).
Revised: This assumes we know the f ∗ and G that minimize the adaptation within the complexity measure (12).

Original: Here, we explain how to use a simple doubling trick to get over the fact that we might not know these optimal solutions f ∗ and G .
Revised: Here, we explain how to use a simple doubling trick to get over the fact that we might not know these optimal solutions f ∗ and G .

Original: Theorem 1 Without explicitly knowing the exact value of the complexity measure, i.e., the optimal solution of Equation (12), one can still achieve the generalization bound in Theorem 1.
Revised: Theorem 1 Without explicitly knowing the exact value of the complexity measure, i.e., the optimal solution of Equation (12), one can still achieve the generalization bound in Theorem 1.

Original: Proof of Theorem
Revised: Proof of Theorem

Revised: Our core generalization result is in Theorem 3.

Original: The proof  is simply adding a doubling trick on top of the argument of Theorem 3.
Revised: The proof of Theorem 1 is simply adding a doubling trick on top of the argument of Theorem 3.

Revised: We also prove Theorem 2 as a consequence of Theorem 1 in Appendix 2.

Original: In the rest of the proofs, for simplicity, we refer to ( W PSGD , V PSGD ) by ( W ′ , V ′ ) .
Revised: In the rest of the proofs, for simplicity, we refer to ( W PSGD , V PSGD ) by ( W ′ , V ′ ) .

Original: Let f ∗∗ , G ∗ be the optimal solution to (10).
Revised: Let f ∗∗ , G ∗ be the optimal solution to (10).

Original: With a simple rescaling of G ∗ , we can assume ⟨ H ∞− 1 , G ∗ ⟩ = 1 .
Revised: With a simple rescaling of G ∗ , we can assume ⟨ H ∞− 1 , G ∗ ⟩ = 1 .

Original: (Note that the complexity does not change by such rescaling).
Revised: (Note that the complexity does not change by such rescaling).

Original: Now one can exploit the condition ∥
Revised: Now one can exploit the condition ∥

Original: y i ∥ ∞ ≤ B , and consider the setting f ∗ = 0 to get the following trivial upper bound on the complexity measure:
Revised: y i ∥ ∞ ≤ B , and consider the setting f ∗ = 0 to get the following trivial upper bound on the complexity measure:

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Using Equation (34) and the optimality of ( f ∗∗ , G ∗ ) :
Revised: Using Equation (34) and the optimality of ( f ∗∗ , G ∗ ) :

Original: [Equation]
Revised: [Equation]

Original: Combining the ﬁrst equation above with the deﬁnition of ν in Equation (51), we get
Revised: Combining the ﬁrst equation above with the deﬁnition of ν in Equation (43), we get

Original: [Equation]
Revised: [Equation]

Original: To initialize ψ 1 and ψ 2 , we use Equations (33) for any f ∗ and G , and as a result we get a generalization bound as in Equation (55).
Revised: To initialize ψ 1 and ψ 2 , we use Equations (33) for any f ∗ and G , and as a result we get a generalization bound as in Equation (47).

Original: However, to achieve the best possible rate characterized by our complexity measure in Theorem 1 without explicitly computing the answer of (12), we use a simple doubling trick; for every pair ( ζ ′ , ν ′ )
Revised: However, to achieve the best possible rate characterized by our complexity measure in Theorem 1 without explicitly computing the answer of (12), we use a simple doubling trick; for every pair ( ζ ′ , ν ′ )

Original: such that ζ ′ is a power of 2 between B 2 and 2 nB 2 , and ν ′ is a power of two between B 2 /n and B 2 , we initialize ψ 1 , ψ 2 as in Equation (33) and run the algorithm, then return the network that minimizes the upper bound of the risk given by Equation(54).
Revised: such that ζ ′ is a power of 2 between B 2 and 2 nB 2 , and ν ′ is a power of two between B 2 /n and B 2 , we initialize ψ 1 , ψ 2 as in Equation (33) and run the algorithm, then return the network that minimizes the upper bound of the risk given by Equation(46).

Original: Note that we do not need to know R n ( f ∗∗ ) to compute this upper bound.
Revised: Note that we do not need to know R n ( f ∗∗ ) to compute this upper bound.

Original: Now let ν ′ be the power of 2 within ν ( G ∗ , f ∗∗ ) /
Revised: Now let ν ′ be the power of 2 within ν ( G ∗ , f ∗∗ ) /

Original: 2 ≤ ν ′ <
Revised: 2 ≤ ν ′ <

Original: ν ( G ∗ , f ∗∗ ) .
Revised: ν ( G ∗ , f ∗∗ ) .

Original: If we are in the case
Revised: If we are in the case

Original: [Equation]
Revised: [Equation]

Original: 37) then for ζ ′ equal to the smallest power of two larger than B 2 , when we run PSGD with pair ( ν ′ , ζ ′ ) , by Theorem 3:
Revised: (37) then for ζ ′ equal to the smallest power of two larger than B 2 , when we run PSGD with pair ( ν ′ , ζ ′ ) , by Theorem 3:

Original: [Equation]
Revised: [Equation]

Original: Because we return the minimum upper bound on the risk (the tighter lower bound of Equation (54)) among all such powers of two, we certainly achieve the above rate in (38).
Revised: Because we return the minimum upper bound on the risk (the tighter lower bound of Equation (46)) among all such powers of two, we certainly achieve the above rate in (38).

Original: Otherwise, if f ∗∗ T A − 1 f ∗∗ ≥ B 2 , let ζ ′ be the power of two within f ∗ T A − 1 f ∗ ≤ ζ ′ ≤ 2 f ∗ T A − 1 f ∗ , then again it is easy to check that conditions of Theorem 3 are satisﬁed, hence we get the following generalization bound:
Revised: Otherwise, if f ∗∗ T A − 1 f ∗∗ ≥ B 2 , let ζ ′ be the power of two within f ∗ T A − 1 f ∗ ≤ ζ ′ ≤ 2 f ∗ T A − 1 f ∗ , then again it is easy to check that conditions of Theorem 3 are satisﬁed, hence we get the following generalization bound:

Original: [Equation]
Revised: [Equation]

Original: which proves the bound of Theorem 1.
Revised: which proves the bound of Theorem 1.

Original: A. A MOUNT OF O VERPARAMETERIZATION
Revised: A. A MOUNT OF O VERPARAMETERIZATION

Original: In this section, to provide high-level insight, we indicate the right order of magnitude that our overparameterization should be in, with respect to one another.
Revised: In this section, to provide high-level insight, we indicate the right order of magnitude that our overparameterization should be in, with respect to one another.

Original: Note that the exact coefﬁcients in these inequalities would depend on the basic parameters B, 1 /λ 0 , n , which we have avoided here for sake of simplicity.
Revised: Note that the exact coefﬁcients in these inequalities would depend on the basic parameters B, 1 /λ 0 , n , which we have avoided here for sake of simplicity.

Original: We refer the reader to our main proof (mostly Appendix parts A.12, A.13) for more details.
Revised: We refer the reader to our main proof (mostly Appendix parts A.12, A.13) for more details.

Original: [Equation]
Revised: [Equation]

Original: In addition [Equation] we set the smoothing parameters as
Revised: In addition, we set the smoothing parameters as

Original: [Equation]
Revised: [Equation]

Original: where Θ p only shows polynomial dependencies on the overparameterization
Revised: where Θ p only shows polynomial dependencies on the overparameterization

Original: [Equation]
Revised: [Equation]

Original: A. PSD PROPERTY OF K ∞
Revised: A. PSD PROPERTY OF K ∞

Original: The Schur product theorem states that for PSD matrices A and B , A ⊙ B is also PSD.
Revised: The Schur product theorem states that for PSD matrices A and B , A ⊙ B is also PSD.

Original: Now given an analytic function F whose tailor series coefﬁcients are all nonnegative, Suppose we apply F on some PSD matrix A entrywise, denoted by F ( A ) , under the condition that the entries of A are in the radius of convergence of F , then using Schur product theorem, it is straightforward that F ( A ) is also PSD.
Revised: Now given an analytic function F whose tailor series coefﬁcients are all nonnegative, Suppose we apply F on some PSD matrix A entrywise, denoted by F ( A ) , under the condition that the entries of A are in the radius of convergence of F , then using Schur product theorem, it is straightforward that F ( A ) is also PSD.

Original: Using the above property, one can then check that the tailor series of the deﬁned functions F 2 and Fare nonnegative, hence, the application of the function F 2 (2 F 3 ( x )) on the gram matrix of ( x i ) ni =1 is a PSD matrix, ( note that (cid:12)(cid:12)(cid:12) ⟨ x i , x j ⟩ (cid:12)(cid:12)(cid:12) ≤ 1 is in the convergence radius of F 2 (2 F 3 ( x )) .)
Revised: Using the above property, one can then check that the tailor series of the deﬁned functions F 2 and Fare nonnegative, hence, the application of the function F 2 (2 F 3 ( x )) on the gram matrix of ( x i ) ni =1 is a PSD matrix, ( note that (cid:12)(cid:12)(cid:12) ⟨ x i , x j ⟩ (cid:12)(cid:12)(cid:12) ≤ 1 is in the convergence radius of F 2 (2 F 3 ( x )) .)

Original: thus K ∞ is indeed a kernel.
Revised: thus K ∞ is indeed a kernel.

Original: A. C OMPLEXITY UPPER BOUND
Revised: A. C OMPLEXITY UPPER BOUND

Original: First we mention a simple fact that hadamard product respects matrix orderings.
Revised: First we mention a simple fact that hadamard product respects matrix orderings.

Original: Given PSD matrices
Revised: Given PSD matrices

Original: A, B, C such that A ⪯ B , the fact that A ⊙ C ⪯ B ⊙ C is an easy consequence of the Schur Product
Revised: A, B, C such that A ⪯ B , the fact that A ⊙ C ⪯ B ⊙ C is an easy consequence of the Schur Product

Original: Theorem; indeed, B − A is PSD by deﬁnition, so ( B − A )
Revised: Theorem; indeed, B − A is PSD by deﬁnition, so ( B − A )

Original: ⊙ C
Revised: ⊙ C

Original: = B ⊙ C −
Revised: = B ⊙ C −

Original: A ⊙ C is also PSD.
Revised: A ⊙ C is also PSD.

Original: Next, it is easy to check that the tailor series of arcsin( x ) has all nonnegative coefﬁcients.
Revised: Next, it is easy to check that the tailor series of arcsin( x ) has all nonnegative coefﬁcients.

Original: Therefore, for a PSD matrix X , as we discussed in Appendix A.4, applying arcsin entrywise on X , namely arcsin X , is also PSD.
Revised: Therefore, for a PSD matrix X , as we discussed in Appendix A.4, applying arcsin entrywise on X , namely arcsin X , is also PSD.

Original: Setting X equal to the entrywise application of 2 F 3 to the gram matrix of datapoints ( x i ) ni
Revised: Setting X equal to the entrywise application of 2 F 3 to the gram matrix of datapoints ( x i ) ni

Original: =1 , we realize the matrix arcsin (cid:16) 2 F(cid:16)(cid:16) ⟨ x i , x j ⟩ (cid:17) 1 ≤ i,j ≤ n (cid:17)(cid:17) is also PSD.
Revised: =1 , we realize the matrix arcsin (cid:16) 2 F(cid:16)(cid:16) ⟨ x i , x j ⟩ (cid:17) 1 ≤ i,j ≤ n (cid:17)(cid:17) is also PSD.

Original: Noting the deﬁnition of K ∞ in Equation (8), we conclude that for the data kernel matrix K ∞ we have
Revised: Noting the deﬁnition of K ∞ in Equation (8), we conclude that for the data kernel matrix K ∞ we have

Original: [Equation]
Revised: [Equation]

Original: where 1 is the all ones n -dimensional vector.
Revised: where 1 is the all ones n -dimensional vector.

Original: Combining the two mentioned facts, we can lower bound the matrix K = K ∞ ⊙ G for any matrix
Revised: Combining the two mentioned facts, we can lower bound the matrix K = K ∞ ⊙ G for any matrix

Original: G as
Revised: G as

Original: [Equation]
Revised: [Equation]

Original: Substituting the rank one matrix f ∗ f ∗ T for the n -dimensional vector f ∗ in Equation (13):
Revised: Substituting the rank one matrix f ∗ f ∗ T for the n -dimensional vector f ∗ in Equation (13):

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The inequality used in (13) then follows from Equation (41).
Revised: The inequality used in (13) then follows from Equation (41).

Original: A. S TRUCTURE OF THE PROOF , SETTING m 3 , AND FURTHER DEFINITIONS
Revised: A. C OMPLEXITY MEASURE AND THE ζ - NORM

Revised: This is a brief section regarding some basic properties of ℑ and ∥ . ∥ ζ .

Revised: First, note that the two versions of the complexity measure in Equations (10) and (11) are equivalent, as for any ﬁnite set of functions { g } , we can deﬁne the gram matrix with respect to the feature vectors of these functions on data, and for an arbitrary nonzero PSD G we can consider a Cholesky factorization for G as G =

Original: Almost all of our proofs in the rest are in the aim of proving Theorem 3.
Revised: ¯ X T ¯ X , then deﬁne the functions { g k } as the minimum-NTK norm functions which map the input to the features corresponding to ¯ X . This observation further implies we can suppose the factor matrix ¯ X is in R n × n , and there is a set of at most n functions { g k } nk = which corresponds to this G . Next, we show that for an arbitrary function f , its sup norm over the unit ball is bounded by its ζ norm: [Equation]

Revised: Note that for a kernel K which satisﬁes K ( x, x ) ≤ 1 , using Cauchy Schwarz we simply obtain

Revised: [Equation]

Revised: where recall that ∥ .

Revised: ∥ K is the norm corresponding to the RKHS space of K .

Revised: Hence, to show (42), it sufﬁce to show that for all kernels K ∈ K and unit norm x we have K ( x, x ) ≤ 1 .

Revised: To see this fact, note that the norm of each x ∈ R d in the NTK-space is H ∞ ( x, x )

Revised: = 12 .

Revised: Therefore, for each function g with bounded-NTK norm, again using Cauchy Schwarz:

Revised: [Equation]

Revised: As a result, for a family of functions { g } with P g ∈{ g } ∥ g ∥ 2 H ∞ ≤ 1 , we have on every unit norm x :

Revised: [Equation]

Revised: On the other hand, it is easy to check that for every unit norm x , we have K ∞ ( x, x ) ≤ 12 , so for every such { g } , we have by deﬁnition

Revised: [Equation]

Revised: , which completes the proof of Equation (42).

Revised: A. C ORE G ENERALIZATION R ESULT

Revised: In this section, we prove our core generalization result for the trained network, Theorem 3, which underlies our generalization bounds in Theorems 1 and 2.

Revised: Recall that in the rest of the proofs, we refer to the solution ( W PSGD , V PSGD ) returned by PSGD simply by ( W ′ , V ′ ) .

Revised: Theorem 3 Suppose we have a candidate pair ( f ∗ , G ) regarding our complexity measure in (10) that satisﬁes ⟨ H ∞− 1 , G ⟩ ≤ 1 , f ∗ T A − 1 f ∗ ≤ ζ (recall A

Revised: =

Revised: G ⊙ H ∞ ).

Revised: Then, for

Revised: [Equation]

Original: Throughout the proof, ( W ′ , V ′ ) represents the pair of matrices of the current iteration of PSGD, ( W ∗ , V ∗ ) are the “ideal” matrices that we construct in Appendix A.12, ( W ρ , V ρ ) and refers to the gaussian smoothing matrices.
Revised: 43) if we are given ν/ 2 ≤ ν ′ ≤ ν , and we set [Equation] [Equation] then for the solution ( W ′ , V ′ ) returned by PSGD we have the following generalization bound:

Original: Importantly, we assume the loss function ℓ ( f, y ) is zero at f = y for any label.
Revised: [Equation] [Equation] for constants c ′′ , c ′′′ and log factor ϖ = log( n ) 3 + log(1 /λ 0 ) . Proof of Theorem

Original: There arefour main parts in our proof:
Revised: Almost all of our proofs in the rest are in the aim of proving Theorem 3. Crucially, to prove this Theorem, we need to establish two big results: We need to show that the ﬁnal network has small training loss, and is within the class G γ 1 ,γ 2 for some suitable γ 1 , γ 2 . This is handled by Theorem 4 in Appendix A.10.

Revised: We deﬁne the class G γ 1 ,γ 2 roughly as the class of networks with norm bounds ∥ W − W (0) ∥

Revised: ≤ γ 1 , ∥ V − V (0) ∥

Revised: ≤ γ 2 where the rows of V − V (0) are orthogonal to the subspace Φ

Original: we construct a “good” underlying network, Appendix A. 2.
Revised: , plus an additional structure deﬁned in Appendix A.11.

Original:  we ﬁnd a “good” random direction and study the landscape of the objective, Appendix A. 3.
Revised: This task , on its own, has three main steps in our proof: (a) we construct a “good” underlying network, Appendix A. (b) we ﬁnd a “good” random direction and study the landscape of the objective, Appendix A. (c) we prove the convergence, Appendix A. 2.

Original: we bound the Rademacher Complexity of the class G γ 1 ,γ 2 , Appendix A. 4.
Revised: The Rademacher Complexity of the class G γ 1 ,γ 2 needs to be suitably bounded.

Original:  we prove the convergence, Appendix A.{+generalization bound in (Srebro et al., 2010). Speciﬁcally, for ﬁxed constants z 1 , z 3 and every integer i ≥ 0 , we use Theorem 1 of (Srebro et al., 2010) for the class G z 1 ,γ i , γ i = 
 i × B/z 3 with conﬁdence probability 1 − 2 − i δ 3 , which, with a union bound, implies+}
Revised: This is handled by Theorem 5 in Appendix A.11. With access to these results, here we show how Theorem 3 follows by a simple application of the {+generalization bound in (Srebro et al., 2010). Speciﬁcally, for ﬁxed constants z 1 , z 3 and every integer i ≥ 0 , we use Theorem 1 of (Srebro et al., 2010) for the class G z 1 ,γ i , γ i = 
 i × B/z 3 with conﬁdence probability 1 − 2 − i δ 3 , which, with a union bound, implies+}

Revised: that with probability at least 1 − δ 3 , for every i and f W ′ ,V ′ ∈

Revised: G z 1 ,γ i :

Revised: [Equation]

Revised: where ℓ ( f W ′ ,V ′ ( x ) , y ) is a.s. bounded by b for function within the class G z 1 ,γ i , and K is a universal constant.

Revised: In the following, we aim to further bound the Rademacher complexity R and parameter b .

Revised: Applying the AM-GM inequality with respect to ratio z 4 > 0 for the second term:

Revised: [Equation]

Revised: Now let γ ∗ be the smallest number of the form 2 i B/z 3 (for some i ) which is not smaller than z 2 √ ζ .

Revised: This deﬁnition implies

Revised: [Equation]

Revised: Now Theorem 5 in Appendix A.11 bounds the Rademacher complexity:

Revised: [Equation]

Revised: On the other hand, from Theorem 4 by setting z 1 , z 2 = √ 40 , we get f W ′ ,

Revised: V ′ ∈

Revised: G z 1 ,γ ∗ .

Revised: Moreover, from Lemma 34, for f W ′ ,V ′ ∈ G z 1 ,γ ∗ , we have for every ∥ x ∥ ≤ 1 :

Revised: [Equation]

Revised: [Equation]

Revised: so the loss ℓ ( f W ′ ,V ′ ( x ) , y ) can be bounded by ( B + 2 z 1 γ ∗ ) 2 using the 1 smoothness property.

Revised: Therefore, for the class G z 1 ,γ ∗ we can set b =

Revised: ( B + 2 z 1 γ ∗ ) 2 .

Revised: Combining this with Equation (52) and plugging into Equation (50):

Revised: [Equation]

Revised: Furthermore, by deﬁnition of γ ∗ , we have 2 i ≤ 2 z 2 z 3 √ ζ/B :

Revised: [Equation]

Revised: Now applying the upper bound on γ ∗ :

Revised: [Equation]

Revised: If ζ > B 2 , in the third term above we substitute B by √ ζ .

Revised: Finally, similar to the bound we stated in

Revised: Equation (15), note that we have the following trivial bound for ζ :

Revised: [Equation]

Revised: [Equation]

Revised: i.e. there is no point in considering larger ζ ’s, which implies log(2 z 2 z 3 √ ζ/B ) =

Revised: O (log( n ) + log(1 /λ 0 )) .

Revised: Plugging this above and picking z 4 = 1 / 3 show the proof of Equation (46).

Revised: Furthermore, applying Equation (70) in Theorem 4 to the R n ( f W ′ ,V ′ ) term in Equation (46) further gives the second Equation (47).

Original: {+Remark 
+}
Revised: {+Remark 
+}

Revised: In the same setting of Theorem 3, if we have ν/ 2 ≤ ν ′ but not generally upper bounded by ν , then PSGD leads to the following generalization bound:

Revised: [Equation]

Revised: using a similar argument as we did for Theorem 3.

Revised: A. S TRUCTURE OF THE PROOF , SETTING m 3 , AND FURTHER DEFINITIONS

Revised: Throughout the proof, ( W ′ , V ′ ) represents the pair of matrices of the current iteration of PSGD , ( W ∗ , V ∗ ) are the “ideal” matrices that we construct in Appendix A.12, ( W ρ , V ρ ) and refers to the gaussian smoothing matrices.

Revised: Importantly, note that our squared loss ℓ ( f, y ) is zero at f = y .

Original: We have tried to make the lower level proofs into sub-lemmas and create a manageable hierarchy as much as we could, to make the document more clear and readable.
Revised: We have tried to make the lower level proofs into sub-lemmas and create a manageable hierarchy as much as we could, to make the document more clear and readable.

Original: Similar to the conditions in Theorem 3, through out most of the proofs we assume that we are given a pair ( f ∗ , G ) with a slightly more general setting of Theorem 3:
Revised: Similar to the conditions in Theorem 3, through out most of the proofs we assume that we are given a pair ( f ∗ , G ) with a slightly more general setting of Theorem 3:

Original: [Equation]
Revised: [Equation]

Original: Particularly, ζ 1 , ζ 2 appear in Appendix A.13.
Revised: Particularly, ζ 1 , ζ 2 appear in Appendix A.13.

Original: Because we are allowed to rescale G , we do not really gain much by assuming this more general setting, though we pick to work with the general setting as the abstraction makes the proof more straightforward to understand.
Revised: Because we are allowed to rescale G , we do not really gain much by assuming this more general setting, though we pick to work with the general setting as the abstraction makes the proof more straightforward to understand.

Original: We refer to the parameters B, 1 /λ 0 , n as the “basic parameters”, m 1 , m 2 , m 3 , 1 /κ 1 , κ 2 as the “overparameterization”, and β 1 , β 2 as the “smoothing parameters.”
Revised: We refer to the parameters B, 1 /λ 0 , n as the “basic parameters”, m 1 , m 2 , m 3 , 1 /κ 1 , κ 2 as the “overparameterization”, and β 1 , β 2 as the “smoothing parameters.”

Original: By the phrase “having enough overparameterization” we mean it sufﬁces to pick the overparameterization m 1 , m 2 , m 3 , 1 /κ 1 , 1 /κ 2 only polynomially large in the basic parameters.
Revised: By the phrase “having enough overparameterization” we mean it sufﬁces to pick the overparameterization m 1 , m 2 , m 3 , 1 /κ 1 , 1 /κ 2 only polynomially large in the basic parameters.

Original: Throughout the proof, we denote the change in the output of the ﬁrst layer at W (0) + W ′ + W ρ compared to the initialization value by ϕ (2) ( x i ) , i.e.
Revised: Throughout the proof, we denote the change in the output of the ﬁrst layer at W (0) + W ′ + W ρ compared to the initialization value by ϕ (2) ( x i ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: while recall that ϕ
Revised: while recall that ϕ

Original: ′ ( x i ) has a similar deﬁnition except without the smoothing matrix W ρ .
Revised: ′ ( x i ) has a similar deﬁnition except without the smoothing matrix W ρ .

Original: Although our model is a three layer network, throughout the proof, we refer to the parts W s 1 √ m 1 σ (( W (0)
Revised: Although our model is a three layer network, throughout the proof, we refer to the parts W s 1 √ m 1 σ (( W (0)

Original: +
Revised: +

Original: W ′ )
Revised: W ′ )

Original: x ) and 
Revised: x ) and 1 √ m 2 a T σ (cid:16) ( V (

Original: [Equation]
Revised: [Equation]

Original:  as the “ﬁrst layer” and “second layer,” respectively.
Revised: ) + V ′ )( . ) (cid:17) as the “ﬁrst layer” and “second layer,” respectively.

Original: Also, we sometimes refer to the binary sign pattern of vector x multiplied to matrix W by D W,x
Revised: Also, we sometimes refer to the binary sign pattern of vector x multiplied to matrix W by D W,x

Original: ( D W,x :=
Revised: ( D W,x :=

Original: Sgn ( Wx ) ), i.e. the j th diagonal entry of D W,x is one if W Tj, x ≥ 0 , and is zero otherwise.
Revised: Sgn ( Wx ) ), i.e. the j th diagonal entry of D W,x is one if W Tj, x ≥ 0 , and is zero otherwise.

Original: To refer to the j th row of W as a vector, we sometimes drop the comma in W j, and write it as W j .
Revised: To refer to the j th row of W as a vector, we sometimes drop the comma in W j, and write it as W j .

Original: For brevity, we denote the Frobenius norm ∥ W ∥ F of matrix by ∥ W ∥ , and the Euclidean norm of a vector x by ∥ x ∥ .
Revised: For brevity, we denote the Frobenius norm ∥ W ∥ F of matrix by ∥ W ∥ , and the Euclidean norm of a vector x by ∥ x ∥ .

Original: For matrices W 1 , W 2 we denote their natural dot product by ⟨ W 1 , W 2 ⟩ := tr ( W T 1 W 2 ) .
Revised: For matrices W 1 , W 2 we denote their natural dot product by ⟨ W 1 , W 2 ⟩ := tr ( W T 1 W 2 ) .

Original: We refer to the smallest eigenvalue of a matrix by λ min ( . ) .
Revised: We refer to the smallest eigenvalue of a matrix by λ min ( . ) .

Original: We write R ( . ) for the
Revised: We write R ( . ) for the

Original: Rademacher complexity of a function class.
Revised: Rademacher complexity of a function class.

Original: We refer to the smoothed version of the network by f ′ W ′ ,V ′ ( x ) , deﬁned by
Revised: We refer to the smoothed version of the network by f ′ W ′ ,V ′ ( x ) , deﬁned by

Original: [Equation]
Revised: [Equation]

Original: In the proof, we mainly work with the loss over the smoothed network f ′ , deﬁned as
Revised: In the proof, we mainly work with the loss over the smoothed network f ′ , deﬁned as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Our algorithm, PSGD can be regarded roughly as an SGD over L .
Revised: Our algorithm, PSGD can be regarded roughly as an SGD over L .

Original: Similar to what we discussed in section 3, let the functions { g k } m 3 k
Revised: Similar to what we discussed in section 3, let the functions { g k } m 3 k

Original: =1 be some feature representation whose gram matrix is equal to G and ⟨ H ∞ , G ⟩ = P m 3 k =

Revised: =1 be some feature representation whose gram matrix is equal to G and ⟨ H ∞ , G ⟩ = P m 3 k =

Original: ∥
Revised: ∥

Original: g k ∥ 2 H ∞ .
Revised: g k ∥ 2 H ∞ .

Original: In such setting, it is not hard to check that we can assume each g k is the minimum norm NTK function which maps ( x i ) ni
Revised: In such setting, it is not hard to check that we can assume each g k is the minimum norm NTK function which maps ( x i ) ni

Original: =1 to ( g k ( x i )) ni
Revised: =1 to ( g k ( x i )) ni

Original: =1 ’s.
Revised: =1 ’s.

Original: Indeed, if this is not the case for some g k , we can project the RKHS representation of g k onto the span of the representations of ( x i ) ni =1 , which can only decrease the complexity measure.
Revised: Indeed, if this is not the case for some g k , we can project the RKHS representation of g k onto the span of the representations of ( x i ) n i =1 , which can only decrease the complexity measure.

Original: Hence we can represent g k ∈ H H ∞ as a linear combination of basic functions H ∞ ( x i , . )
Revised: Hence we can represent g k ∈ H H ∞ as a linear combination of basic functions H ∞ ( x i , . )

Original: on data points:
Revised: on data points:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Here, the sum of squared- H ∞ norms of V k is bounded as X
Revised: Here, the sum of squared- H ∞ norms of V k is bounded as X

Original: [Equation]
Revised: [Equation]

Original: For each i ∈ [ n ] , we refer to the feature representation vector ( g k ( x i ))
Revised: For each i ∈ [ n ] , we refer to the feature representation vector ( g k ( x i ))

Original: m 3 k
Revised: m 3 k

Original: =1 on x i as ¯ x i .
Revised: =1 on x i as ¯ x i .

Original: Note that we have the relationship
Revised: Note that we have the relationship

Original: [Equation]
Revised: [Equation]

Original: where H ∞ i, is the i th row of H ∞ .
Revised: where H ∞ i, is the i th row of H ∞ .

Original: In the analysis, we work with a bound ξ on the quantity max k ∥V k ∥ which should be bounded polynomially by other basic parameters; in particular, it is deﬁned in
Revised: In the analysis, we work with a bound ξ on the quantity max k ∥V k ∥ which should be bounded polynomially by other basic parameters; in particular, it is deﬁned in

Original: Lemma 10 and is used to bound a cross term in Lemma 14.
Revised: Lemma 10 and is used to bound a cross term in Lemma 14.

Original: However, max k ∥V k ∥ might not be effectively bounded for an arbitrary feature representation.
Revised: However, max k ∥V k ∥ might not be effectively bounded for an arbitrary feature representation.

Original: Fortunately, we can remedy this by a simple trick; for every natural number s , one can substitute every g k by s copies of g k / √ s , without changing the gram matrix G .
Revised: Fortunately, we can remedy this by a simple trick; for every natural number s , one can substitute every g k by s copies of g k / √ s , without changing the gram matrix G .

Original: Therefore, for any δ , one can increase the multiset of functions ( g k ) to a bigger set (˜ g k ) , by adding at most O ( ζ 1 /δ ) functions, making sure of the following for the new functions:
Revised: Therefore, for any δ , one can increase the multiset of functions ( g k ) to a bigger set (˜ g k ) , by adding at most O ( ζ 1 /δ ) functions, making sure of the following for the new functions:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (This is because P k ∥
Revised: (This is because P k ∥

Original: ˜ g k ∥ 2 H ∞ ≤ 1 ).
Revised: ˜ g k ∥ 2 H ∞ ≤ 1 ).

Original: Furthermore, observe that for each gram matrix G , we have an n -dimensional feature representation ( g k )
Revised: Furthermore, observe that for each gram matrix G , we have an n -dimensional feature representation ( g k )

Original: nk =1 for G according to the Cholesky factorization.
Revised: nk =1 for G according to the Cholesky factorization.

Original: Combining these facts, we conclude, to guarantee Equation (46), in the worst case, we need m 3 to be as large as n + O ( ζ 1 /δ ) .
Revised: Combining these facts, we conclude, to guarantee Equation (63), in the worst case, we need m 3 to be as large as n + O ( ζ 1 /δ ) .

Original: Finally, observing the following inequality
Revised: Finally, observing the following inequality

Original: [Equation]
Revised: [Equation]

Original: [Equation] in order to guarantee max k ∥V k ∥ ≤ ξ we need to take m 3 as large as n + O ( ζ 1 / ( ξ 2 λ 0 )) , which is indeed bounded polynomially by the basic parameters because of the same condition for 1 /ξ .
Revised: in order to guarantee max k ∥V k ∥ ≤ ξ we need to take m 3 as large as n + O ( ζ 1 / ( ξ 2 λ 0 )) , which is indeed bounded polynomially by the basic parameters because of the same condition for 1 /ξ .

Original: This computation also brings into sight an important point: “Although each gram matrix G is representable by n features, in order for the algorithm to be able to ﬁnd a suitable network, m 3 might need to be larger than n .”
Revised: This computation also brings into sight an important point: “Although each gram matrix G is representable by n features, in order for the algorithm to be able to ﬁnd a suitable network, m 3 might need to be larger than n .”

Original: Moreover, for every 1 ≤ k, i ≤ n , we deﬁne the matrices
Revised: Moreover, for every 1 ≤ k, i ≤ n , we deﬁne the matrices

Original: Z ki ∈ R md as
Revised: Z ki ∈ R md as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where in the above notation, j is enumerating the columns of the matrix.
Revised: where in the above notation, j is enumerating the columns of the matrix.

Original: We also deﬁne the following matrices which we use in our construction later:
Revised: We also deﬁne the following matrices which we use in our construction later:

Original: [Equation]
Revised: [Equation]

Original: ) and W + as
Revised: and W + as

Original: [Equation]
Revised: [Equation]

Original: Finally, to avoid unnecessary complication, we often argue high probability bounds without an explicit representation of their dependency on the chance of failure (which is a negligible logarithmic factor).
Revised: Finally, to avoid unnecessary complication, we often argue high probability bounds without an explicit representation of their dependency on the chance of failure (which is a negligible logarithmic factor).

Original: We also ignore all constants and log factors, and mainly work with the notation ≲ which ignores constants; we write a ≲ b ± c as a short form for b − O ( c ) ≤ a ≤ b + O ( c ) .
Revised: We also ignore all constants and log factors, and mainly work with the notation ≲ which ignores constants; we write a ≲ b ± c as a short form for b − O ( c ) ≤ a ≤ b + O ( c ) .

Original: As there are several hierarchies of new parameters that are deﬁned based on lower-level ones, we rename the new parameters and continue viewing them as black-box.
Revised: As there are several hierarchies of new parameters that are deﬁned based on lower-level ones, we rename the new parameters and continue viewing them as black-box.

Original: This makes the proofs more readable, since we also do not care about the exact dependency of the underlying parameters most of the time, rather we are interested in their orders of magnitude, for example that a given parameter goes to zero polynomially fast with respect to the overparameterization, etc.
Revised: This makes the proofs more readable, since we also do not care about the exact dependency of the underlying parameters most of the time, rather we are interested in their orders of magnitude, for example that a given parameter goes to zero polynomially fast with respect to the overparameterization, etc.

Original: Due to the large number of symbols that we have to work with, we might use a symbol more than once, of course when it is clear from the context which one we are refering to.
Revised: Due to the large number of symbols that we have to work with, we might use a symbol more than once, of course when it is clear from the context which one we are refering to.

Original: A. C OMPLEXITY MEASURE AND THE ζ - NORM

Original: This is a brief section regarding some basic properties of ℑ and ∥ .

Original: ∥ ζ .

Original: First, note that the two versions of the complexity measure in Equations (10) and (11) are equivalent, as for any ﬁnite set of functions { g } , we can deﬁne the gram matrix with respect to the feature vectors of these functions on data, and for an arbitrary nonzero PSD G we can consider a Cholesky factorization for G as G =

Original: ¯ X T ¯ X , then deﬁne the functions { g k } as the minimum-NTK norm functions which map the input to the features corresponding to ¯ X .

Original: This observation further implies we can suppose the factor matrix ¯ X is in R n × n , and there is a set of at most n functions { g k } n k = which corresponds to this G .

Original: Next, we show that for an arbitrary function f , its sup norm over the unit ball is bounded by its ζ norm:

Original: [Equation]

Original: Note that for a kernel K which satisﬁes K ( x, x ) ≤ 1 , using Cauchy Schwarz we simply obtain

Original: [Equation]

Original: where recall that ∥ .

Original: ∥ K is the norm corresponding to the RKHS space of K .

Original: Hence, to show (50), it sufﬁce to show that for all kernels K ∈ K and unit norm x we have K ( x, x ) ≤ 1 .

Original: To see this fact, note that the norm of each x ∈ R d in the NTK-space is H ∞ ( x, x )

Original: = 12 .

Original: Therefore, for each function g with bounded-NTK norm, again using Cauchy Schwarz:

Original: [Equation]

Original: As a result, for a family of functions { g } with P g ∈{ g } ∥ g ∥ 2 H ∞ ≤ 1 , we have on every unit norm x :

Original: [Equation]

Original: On the other hand, it is easy to check that for every unit norm x , we have K ∞ ( x, x ) ≤ 12 , so for every such { g } , we have by deﬁnition

Original: K { g } ( x ) ≤ 1 , which completes the proof of Equation (50).

Original: A. C ORE G ENERALIZATION R ESULT

Original: In this section, we prove our core generalization result for the trained network, Theorem 3, which underlies our generalization bounds in Theorems 1 and 2.

Original: Recall that in the rest of the proofs, we refer to the solution ( W PSGD , V PSGD ) returned by PSGD simply by ( W ′ , V ′ ) .

Original: Theorem 3 Suppose we have a candidate pair ( f ∗ , G ) regarding our complexity measure in (10) that satisﬁes ⟨ H ∞− 1 , G ⟩ ≤ 1 , f ∗ T A − 1 f ∗ ≤ ζ (recall A

Original: =

Original: G ⊙ H ∞ ).

Original: Then, for

Original: [Equation]

Original: 51) if we are given ν/ 2 ≤ ν ′ ≤ ν , and we set

Original: [Equation]

Original: [Equation]

Original: then for the solution ( W ′ , V ′ ) returned by PSGD we have the following generalization bound:

Original: [Equation]

Original: [Equation]

Original: for constants c ′′ , c ′′′ and log factor ϖ = log( n ) 3 + log(1 /λ 0 ) .

Original: Proof of Theorem

Original: To prove Theorem 3, we use a generalization bound from Srebro et al.

Original: Crucially, in order to apply this bound, we need to establish two big results:

Original: We need to show that the ﬁnal network has small training loss, and is within the class G γ 1 ,γ 2 for some suitable γ 1 , γ 2 .

Original: This is handled by Theorem 4 in Appendix A.10.

Original: We deﬁne the class G γ 1 ,γ 2 roughly as the class of networks with norm bounds ∥ W − W (0) ∥

Original: ≤ γ 1 , ∥ V − V (0) ∥

Original: ≤ γ 2 where the rows of V − V (0) are orthogonal to the subspace Φ , plus an additional structure deﬁned in Appendix A.11.

Original: The Rademacher Complexity of the class G γ 1 ,γ 2 needs to be suitably bounded.

Original: This is handled by Theorem 5 in Appendix A.11.

Original: With access to these results, we show how Theorem 3 follows.

Original: For ﬁxed constants z 1 , z 3 and every integer i ≥ 0 , we use Theorem 1 in Srebro et al.

Original: [-(2010) for the class G z 1 ,γ i , γ i = 
-]
Revised: [-(2010) for the class G z 1 ,γ i , γ i = 
-]

Original: i × B/z 3 with conﬁdence probability 1 − 2 − i δ 3 , which, with a union bound, implies

Original: that with probability at least 1 − δ 3 for every i and f W ′ ,V ′ ∈

Original: G z 1 ,γ i :

Original: [Equation]

Original: where ℓ ( f W ′ ,V ′ ( x ) , y ) is a.s. bounded by b for function within the class G z 1 ,γ i , and K is a universal constant.

Original: In the following, we aim to further bound the Rademacher complexity R and parameter b .

Original: Applying the AM-GM inequality with respect to ratio z 4 > 0 for the second term:

Original: [Equation]

Original: Now let γ ∗ be the smallest number of the form 2 i B/z 3 (for some i ) which is not smaller than z 2 √ ζ .

Original: This deﬁnition implies

Original: [Equation]

Original: Now Theorem 5 in Appendix A.11 bounds the Rademacher complexity:

Original: [Equation]

Original: On the other hand, from Theorem 4 by setting z 1 , z 2 = √ 40 , we get f W ′ ,

Original: V ′ ∈

Original: G z 1 ,γ ∗ .

Original: Moreover, from Lemma 34, for f W ′ ,V ′ ∈ G z 1 ,γ ∗ , we have for every ∥ x ∥ ≤ 1 :

Original: [Equation]

Original: [Equation]

Original: so the loss ℓ ( f W ′ ,V ′ ( x ) , y ) can be bounded by ( B + 2 z 1 γ ∗ ) 2 using the 1 smoothness property.

Original: Therefore, for the class G z 1 ,γ ∗ we can set b =

Original: ( B + 2 z 1 γ ∗ ) 2 .

Original: Combining this with Equation (60) and plugging into Equation (58):

Original: [Equation]

Original: Furthermore, by deﬁnition of γ ∗ , we have 2 i ≤ 2 z 2 z 3 √ ζ/B :

Original: [Equation]

Original: Now applying the upper bound on γ ∗ :

Original: [Equation]

Original: If ζ > B 2 , in the third term above we substitute B by √ ζ .

Original: Finally, similar to the bound we stated in

Original: Equation (15), note that we have the following trivial bound for ζ :

Original: [Equation]

Original: [Equation]

Original: i.e. there is no point in considering larger ζ ’s, which implies log(2 z 2 z 3 √ ζ/B ) =

Original: O (log( n ) + log(1 /λ 0 )) .

Original: Plugging this above and picking z 4 = 1 / 3 show the proof of Equation (54).

Original: Furthermore, applying Equation (70) in Theorem 4 to the R n ( f W ′ ,V ′ ) term in Equation (54) further gives the second Equation (55).

Original: [-Remark 
-]
Revised: [-Remark 
-]

Original: In the same setting of Theorem 3, if we have ν/ 2 ≤ ν ′ but not generally upper bounded by ν , then PSGD leads to the following generalization bound:

Original: [Equation]

Original: using a similar argument as we did for Theorem 3.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: In this section, we prove Theorem 2, stated below.
Revised: In this section, we prove Theorem 2, stated below.

Original: Theorem 2 For any function f : R d → R , in the same setting as Theorem 1, the population risk of the trained network ( W ′ , V ′ ) can be bounded as
Revised: Theorem 2 For any function f : R d → R , in the same setting as Theorem 1, the population risk of the trained network ( W ′ , V ′ ) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem
Revised: Proof of Theorem

Original: Theorem 2 is a simple consequence of Theorem 1; for the given function f , we apply Theorem 2 with the smaller coefﬁcient γ = 43 for R n ( f ∗ ) , regarding the complexity upper bound, by setting f ∗ :
Revised: Theorem 2 is a simple consequence of Theorem 1; for the given function f , we apply Theorem 2 with the smaller coefﬁcient γ = 43 for R n ( f ∗ ) , regarding the complexity upper bound, by setting f ∗ :

Original: = ( f ( x i ))
Revised: = ( f ( x i ))

Original: ni
Revised: ni

Original: =1 :
Revised: =1 :

Original: [Equation]
Revised: [Equation]

Original: On the other hand, because f ∗ T K − 1 f ∗ is the minimum-RKHS norm of a function with respect to kernel K which maps x i ’s to f ∗ i and f is one such function, we have f ∗ T K − 1 f ∗ ≤ ∥ f ∥ K .
Revised: On the other hand, because f ∗ T K − 1 f ∗ is the minimum-RKHS norm of a function with respect to kernel K which maps x i ’s to f ∗ i and f is one such function, we have f ∗ T K − 1 f ∗ ≤ ∥ f ∥ K .

Original: This inequality implies
Revised: This inequality implies

Original: [Equation]
Revised: [Equation]

Original: so we obtain
Revised: so we obtain

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, it remains to bound R n ( f ) by R ( f ) .
Revised: Therefore, it remains to bound R n ( f ) by R ( f ) .

Original: As we showed in Appendix A.7, for every input x we have f ( x ) ≤ ∥ f ∥ ζ , so for every data ( x, y ) , by the fact that | y | ≤ B a.s. and α smoothness of the loss, we have ℓ ( f ( x ) , y ) ≤ α ( ∥
Revised: As we showed in Appendix A.6, for every input x we have f ( x ) ≤ ∥ f ∥ ζ , so for every data ( x, y ) , by the fact that | y | ≤ B a.s. and α smoothness of the loss, we have ℓ ( f ( x ) , y ) ≤ α ( ∥

Original: f ∥ ζ + B ) 2 .
Revised: f ∥ ζ + B ) 2 .

Original: Moreover, note that the random variable ℓ ( f ( x ) , y ) has mean R ( f ) .
Revised: Moreover, note that the random variable ℓ ( f ( x ) , y ) has mean R ( f ) .

Original: It is easy to check that in this setting, the variance of ℓ ( f ( x ) , y ) is at most R ( f ) α (
Revised: It is easy to check that in this setting, the variance of ℓ ( f ( x ) , y ) is at most R ( f ) α (

Original: ∥ f ∥
Revised: ∥ f ∥

Original: ζ + B ) 2 .
Revised: ζ + B ) 2 .

Original: Therefore, an application of the
Revised: Therefore, an application of the

Original: Bernstein inequality, we have with high probability over the dataset
Revised: Bernstein inequality, we have with high probability over the dataset

Revised: R n

Original: [Equation]
Revised: [Equation]

Original: Plugging this back to Equation (68) completes the proof.
Revised: Plugging this back to Equation (68) completes the proof.

Original: As a result, the learned network can compete with any function that has reasonably small ∥ f ∥ ζ :
Revised: As a result, the learned network can compete with any function that has reasonably small ∥ f ∥ ζ :

Original: [Equation]
Revised: [Equation]

Original: A. O PTIMIZATION
Revised: A. O PTIMIZATION

Original: In this section, we glue together
Revised: In this section, we glue together

Original: • the existence of a good random direction that we prove in Appendix A. • the convergence analysis of PSGD that we do based on the work Ge et al.
Revised: • the existence of a good random direction that we prove in Appendix A. • the convergence analysis of PSGD that we do based on the work Ge et al.

Original: (2015b) in Appendix A.16.
Revised: (2015b) in Appendix A.16.

Original: Theorem 

Revised: Theorem

Original: In the same setting of Theorem 3, assume the network ( W ′ , V ′ ) returned by PSGD , has sufﬁcient polynomially large “overparameterization”.
Revised: In the same setting of Theorem 3, assume the network ( W ′ , V ′ ) returned by PSGD , has sufﬁcient polynomially large “overparameterization”.

Original: Then, for the solution ( W ′ , V ′ ) returned by
Revised: Then, for the solution ( W ′ , V ′ ) returned by

Original: PSGD we have
Revised: PSGD we have

Original: [Equation]
Revised: [Equation]

Original: which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Original: Moreover, for every i ∈ [ n ] , j ∈ [ m 1 ] , j / ∈ P for P deﬁned in 1, we have that sign(( W (0)
Revised: Moreover, for every i ∈ [ n ] , j ∈ [ m 1 ] , j / ∈ P for P deﬁned in 1, we have that sign(( W (0)

Original: j + W ′ j ) T x i ) and sign( W (0)
Revised: j + W ′ j ) T x i ) and sign( W (0)

Original: j T x i ) are the same.
Revised: j T x i ) are the same.

Original: Proof of Theorem
Revised: Proof of Theorem

Original: Let Υ ∈ R m 2 ( m 3 − n ) × m 2 m 3 be a matrix whose rows are an orthonormal basis for the space of matrices whose rows are orthogonal to span ( { ϕ (0)
Revised: Let Υ ∈ R m 2 ( m 3 − n ) × m 2 m 3 be a matrix whose rows are an orthonormal basis for the space of matrices whose rows are orthogonal to span ( { ϕ (0)

Original: ( x i ) } ni =1 ) , i.e. Φ ⊥ , as deﬁned in (24).
Revised: ( x i ) } ni =1 ) , i.e. Φ ⊥ , as deﬁned in (24).

Original: Then, we consider a linear change of coordinates for the subspace Φ ⊥ , regarding the second layer weights, as v ′ =
Revised: Then, we consider a linear change of coordinates for the subspace Φ ⊥ , regarding the second layer weights, as v ′ =

Original: Υ vec ( V ′ ) where vec ( . ) splits out the vectorized version of a matrix.
Revised: Υ vec ( V ′ ) where vec ( . ) splits out the vectorized version of a matrix.

Original: For consistent notation, we also denote w ′
Revised: For consistent notation, we also denote w ′

Original: =
Revised: =

Original: W ′ , so we now have a new coordinate system ( w ′ , v ′ )
Revised: W ′ , so we now have a new coordinate system ( w ′ , v ′ )

Original: ∈
Revised: ∈

Original: R m 2 ( m 3 − n ) × m 1 d for pairs of weights ( W ′ , V ′ )
Revised: R m 2 ( m 3 − n ) × m 1 d for pairs of weights ( W ′ , V ′ )

Original: such that V ′ ∈
Revised: such that V ′ ∈

Original: Φ ⊥ .
Revised: Φ ⊥ .

Original: We also deﬁne the loss function
Revised: We also deﬁne the loss function

Original: [Equation]
Revised: [Equation]

Original: ′ ) , with respect to the change of coordinate.
Revised: ′ ) , with respect to the change of coordinate.

Original: Now it is easy to see that running PSGD on L in the normal coordinates is equivalent to running stochastic gradient descent on L Π with respect to ( w ′ , v ′ ) .
Revised: Now it is easy to see that running PSGD on L in the normal coordinates is equivalent to running stochastic gradient descent on L Π with respect to ( w ′ , v ′ ) .

Original: Moreover, because multiplying to matrix Υ is an orthonormal change of coordinates for Φ ⊥ and because V ′ is already in ϕ ⊥ at each step of
Revised: Moreover, because multiplying to matrix Υ is an orthonormal change of coordinates for Φ ⊥ and because V ′ is already in ϕ ⊥ at each step of

Original: PSGD , then ∥ v ′ ∥ =
Revised: PSGD , then ∥ v ′ ∥ =

Original: ∥ V ′ ∥ , so the conditions ∥
Revised: ∥ V ′ ∥ , so the conditions ∥

Original: W ′ ∥
Revised: W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 are equivalent to ∥ w ′ ∥ ≤
Revised: ≤ C 2 are equivalent to ∥ w ′ ∥ ≤

Original: C 1 , ∥ v ′ ∥
Revised: C 1 , ∥ v ′ ∥

Original: ≤ C 2 .
Revised: ≤ C 2 .

Original: Furthermore, by our construction, the random matrix V ∗ Σ is in the subspace Φ ⊥ , so the norm bounds ∥ W ∗ ∥ ≤ ζ 1 , ∥ V ∗ ∥ ≤ ζ 2 are equivalent to ∥ w ∗ ∥ ≤ ζ 1 , ∥ v ∗ ∥ ≤ ζ 2 for w ∗ :
Revised: Furthermore, by our construction, the random matrix V ∗ Σ is in the subspace Φ ⊥ , so the norm bounds ∥ W ∗ ∥ ≤ ζ 1 , ∥ V ∗ ∥ ≤ ζ 2 are equivalent to ∥ w ∗ ∥ ≤ ζ 1 , ∥ v ∗ ∥ ≤ ζ 2 for w ∗ :

Original: = W ∗ Σ and v ∗ :
Revised: = W ∗ Σ and v ∗ :

Original: = Υ vec ( V ∗ ) .
Revised: = Υ vec ( V ∗ ) .

Original: Now we apply the result of Theorem 6 on L Π with parameter ν set as ν ′ , ζ 2 := ζ and ζ 1 := 1 , and ∆ := R n ( f ∗ ) , as deﬁned in Theorem 3.
Revised: Now we apply the result of Theorem 6 on L Π with parameter ν set as ν ′ , ζ 2 := ζ and ζ 1 := 1 , and ∆ := R n ( f ∗ ) , as deﬁned in Theorem 3.

Original: More speciﬁcally, based on our arguments above regarding the natural isometry in the change of coordinate, any pair ( w ′ , v ′ ) in the domain ∥ w ′ ∥
Revised: More speciﬁcally, based on our arguments above regarding the natural isometry in the change of coordinate, any pair ( w ′ , v ′ ) in the domain ∥ w ′ ∥

Original: ≤ C 1 , ∥ v ′ ∥ ≤
Revised: ≤ C 1 , ∥ v ′ ∥ ≤

Original: C 2 , L Π ( w ′ , v ′ ) ≥ R n ( f ∗ ) + ν ′ translates into a pair ( W ′ , V ′ ) in the domain ∥ W ′ ∥
Revised: C 2 , L Π ( w ′ , v ′ ) ≥ R n ( f ∗ ) + ν ′ translates into a pair ( W ′ , V ′ ) in the domain ∥ W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥ ≤
Revised: ≤ C 1 , ∥ V ′ ∥ ≤

Original: C 2 , L ( W ′ , V ′ ) ≥ R n ( f ∗ ) + ν ′ , for which by Theorem 6 there exists ( W ∗ Σ , V ∗ Σ ) such that
Revised: C 2 , L ( W ′ , V ′ ) ≥ R n ( f ∗ ) + ν ′ , for which by Theorem 6 there exists ( W ∗ Σ , V ∗ Σ ) such that

Original: [Equation]
Revised: [Equation]

Original: Translating back to the change of coordinates:
Revised: Translating back to the change of coordinates:

Original: [Equation]
Revised: [Equation]

Original: Now we apply Lemma 44 to translate this into an argument about the landscape of L Π .
Revised: Now we apply Lemma 44 to translate this into an argument about the landscape of L Π .

Original: As a result, applying the bounds in Equations (107) and (127), we obtain that for ( w ′ , v ′ )
Revised: As a result, applying the bounds in Equations (107) and (127), we obtain that for ( w ′ , v ′ )

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: we should either have
Revised: we should either have

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: Next, we want to apply Theorem 7 by setting
Revised: Next, we want to apply Theorem 7 by setting

Original: [Equation]
Revised: [Equation]

Original: and lipschitz parameters ρ 1 , ρ 2 , ρ 3 = poly ( B, C 1 , C 2 , m 1 , m 2 , m 3 ) set as described in Appendix B.1, Theorem 9.
Revised: and lipschitz parameters ρ 1 , ρ 2 , ρ 3 = poly ( B, C 1 , C 2 , m 1 , m 2 , m 3 ) set as described in Appendix B.1, Theorem 9.

Original: Also, note that as prescribed by Theorem 7, we set
Revised: Also, note that as prescribed by Theorem 7, we set

Original: [Equation]
Revised: [Equation]

Original: where l = O (1) depends on our desired chance of success for the algorithm, speciﬁed in Theorem 7.
Revised: where l = O (1) depends on our desired chance of success for the algorithm, speciﬁed in Theorem 7.

Original: Finally, note that Theorem 7 needs to work with a bounded noise on the gradient whose covariance matrix is bounded between two multipliers of identity.
Revised: Finally, note that Theorem 7 needs to work with a bounded noise on the gradient whose covariance matrix is bounded between two multipliers of identity.

Original: The point of injecting extra noise to SGD in PSGD is in fact because of this covariance condition that we need in Theorem 7.
Revised: The point of injecting extra noise to SGD in PSGD is in fact because of this covariance condition that we need in Theorem 7.

Original: On the other hand, note that in general, because of the gaussian smoothing that we use, the noise vector is not supported on a bounded domain, which makes it a bit harder to apply Hoeffding type concentration.
Revised: On the other hand, note that in general, because of the gaussian smoothing that we use, the noise vector is not supported on a bounded domain, which makes it a bit harder to apply Hoeffding type concentration.

Original: To remedy this, we introduce a coupling between our unbounded noise vector for L ( W ′ , V ′ ) and another noise random variable whose support is bounded, which with high probability is equal to the real noise, along all iterations.
Revised: To remedy this, we introduce a coupling between our unbounded noise vector for L ( W ′ , V ′ ) and another noise random variable whose support is bounded, which with high probability is equal to the real noise, along all iterations.

Original: In Corollary, we further translate this coupling for the objective
Revised: In Corollary, we further translate this coupling for the objective

Original: L Π after change of cooridnates, and write down the exact dependencies of the parameters Q , σ 1 and σ 2 , which are all polynomial in the basic parameters and the overparameterization.
Revised: L Π after change of cooridnates, and write down the exact dependencies of the parameters Q , σ 1 and σ 2 , which are all polynomial in the basic parameters and the overparameterization.

Original: Hence, the conditions of Theorem 7 are satisﬁed, so we conclude that after at most poly ( ρ 1 , ρ 2 , ρ 3 , Q, ℵ , C 1 , C 2 , 1 /γ, log( σ 1 /σ 2 ))
Revised: Hence, the conditions of Theorem 7 are satisﬁed, so we conclude that after at most poly ( ρ 1 , ρ 2 , ρ 3 , Q, ℵ , C 1 , C 2 , 1 /γ, log( σ 1 /σ 2 ))

Original: [Equation]
Revised: [Equation]

Original: poly ( n, B ∨ 1 /B, 1 /γ 0 ) number of iterations, PSGD reach a point w t in some iteration t with
Revised: poly ( n, B ∨ 1 /B, 1 /γ 0 ) number of iterations, PSGD reach a point w t in some iteration t with

Original: L Π ( w t ) ≤ ℵ ℓ .
Revised: L Π ( w t ) ≤ ℵ ℓ .

Original: Translating back this w t = ( w ′ t , v ′ t ) by multiplying the v ′ t part to Υ T , we get a pair ( W ′ t , V ′ t ) with objective value bounded as
Revised: Translating back this w t = ( w ′ t , v ′ t ) by multiplying the v ′ t part to Υ T , we get a pair ( W ′ t , V ′ t ) with objective value bounded as

Original: [Equation]
Revised: [Equation]

Original: But note that we obviously have the condition
Revised: But note that we obviously have the condition

Original: ∥
Revised: ∥

Original: W ′ ∥
Revised: W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 through the whole iterations, for the choice of C 1 , C 2 in Equation (74).
Revised: ≤ C 2 through the whole iterations, for the choice of C 1 , C 2 in Equation (74).

Original: Therefore, using Lemma 34, for every i ∈ [ n ] :
Revised: Therefore, using Lemma 34, for every i ∈ [ n ] :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From Equations (76), as also stated in Theorem 9, we know that for all i ∈ [ n ] , ℓ ( ., y i ) is
Revised: From Equations (76), as also stated in Theorem 9, we know that for all i ∈ [ n ] , ℓ ( ., y i ) is

Original: O ( C 1 C 2 ) + B 2 -Lipschitz at points f W ′ ,V ′
Revised: O ( C 1 C 2 ) + B 2 -Lipschitz at points f W ′ ,V ′

Original: ( x i ) and f ′ W ′ ,V ′ ( x i ) , so we can bound the difference | ℓ ( f ′ W ′ ,V ′
Revised: ( x i ) and f ′ W ′ ,V ′ ( x i ) , so we can bound the difference | ℓ ( f ′ W ′ ,V ′

Original: ( x i ) , y i ) − ℓ ( f W ′ ,V ′ ( x i ) , y i ) | by ( O ( C 1 C 2 ) + B 2 )
Revised: ( x i ) , y i ) − ℓ ( f W ′ ,V ′ ( x i ) , y i ) | by ( O ( C 1 C 2 ) + B 2 )

Original: | f ′ W ′ ,V ′ ( x i ) , y i ) − f W ′ ,V ′ ( x i ) | , which in turn can become arbitrarily small having enough overparameterization using Lemma 35, in particular, we force it to be smaller than O ( ν ′ /
Revised: | f ′ W ′ ,V ′ ( x i ) , y i ) − f W ′ ,V ′ ( x i ) | , which in turn can become arbitrarily small having enough overparameterization using Lemma 35, in particular, we force it to be smaller than O ( ν ′ /

Original: ( B 2 + C 1 C 2 ))
Revised: ( B 2 + C 1 C 2 ))

Original: (recall ν ′ ≥ ν/ 2 ≥ B 2 / (2 n ) ).
Revised: (recall ν ′ ≥ ν/ 2 ≥ B 2 / (2 n ) ).

Original: As a result, we get | ℓ ( f ′ W ′ ,V ′ ( x i ) , y i ) − ℓ ( f W ′ ,V ′ ( x i ) , y i ) |
Revised: As a result, we get | ℓ ( f ′ W ′ ,V ′ ( x i ) , y i ) − ℓ ( f W ′ ,V ′ ( x i ) , y i ) |

Original: = O ( ν ) for every i ∈ [ n ] , which in turn implies | L ( W ′ , V ′ )
Revised: = O ( ν ) for every i ∈ [ n ] , which in turn implies | L ( W ′ , V ′ )

Original: − L 1 ( W ′ , V ′ )
Revised: − L 1 ( W ′ , V ′ )

Original: | ≤ ν by picking small constants, where recall that the objective L 1 is the same as L but without the smoothing.
Revised: | ≤ ν by picking small constants, where recall that the objective L 1 is the same as L but without the smoothing.

Original: Now applying this bound to Equation (75), we get
Revised: Now applying this bound to Equation (75), we get

Original: L 
Revised: L 1 ( W [Equation] , V [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: f

Original:  Therefore, as PSGD check the values of L 1 in the loop, it terminates at such pair ( W t , V t ) .
Revised: [Equation] + [Equation] Therefore, as PSGD check the values of L 1 in the loop, it terminates at such pair ( W t , V t ) .

Original: From this point onward, we refer to the returned ( W ′ t , V ′ t ) as just ( W ′ , V ′ ) .
Revised: From this point onward, we refer to the returned ( W ′ t , V ′ t ) as just ( W ′ , V ′ ) .

Original: Opening the deﬁnition of L 1 ( W ′ , V ′ ) , we clearly get
Revised: Opening the deﬁnition of L 1 ( W ′ , V ′ ) , we clearly get

Original: [Equation]
Revised: [Equation]

Original: Furthermore, noting the setting of ψ 1 , ψ 2 in Theorem 3 and the fact that ν ′ ≥ ν/ 2 ≥ R n ( f ∗ ) / 8 , we get
Revised: Furthermore, noting the setting of ψ 1 , ψ 2 in Theorem 3 and the fact that ν ′ ≥ ν/ 2 ≥ R n ( f ∗ ) / 8 , we get

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: The fact that for every i ∈ [ n ] , j ∈ [ m 1 ] , j / ∈ P we have that sign(( W (0)
Revised: The fact that for every i ∈ [ n ] , j ∈ [ m 1 ] , j / ∈ P we have that sign(( W (0)

Original: j + W ′ j ) T x i ) and sign( W (0)
Revised: j + W ′ j ) T x i ) and sign( W (0)

Original: j T x i ) are the same follows from Lemma 1.
Revised: j T x i ) are the same follows from Lemma 1.

Original: A. R ADEMACHER C OMPLEXITY
Revised: A. R ADEMACHER C OMPLEXITY

Original: In this section we show the proof for our Rademacher Complexity bound, which is used in Theorem 3.
Revised: In this section we show the proof for our Rademacher Complexity bound, which is used in Theorem 3.

Original: Theorem 5 Let G γ 1 ,γ 2 be the class of neural nets with weights ( W, V ) in our three layer setting, such that ∥ W − W (0) ∥ ≤ γ 1 , ∥ V − V (0) ∥
Revised: Theorem 5 Let G γ 1 ,γ 2 be the class of neural nets with weights ( W, V ) in our three layer setting, such that ∥ W − W (0) ∥ ≤ γ 1 , ∥ V − V (0) ∥

Original: ≤ γ 2 , where for every j ∈ [ m 2 ] , i ∈ [ n ] : V ′ j ⊥ ϕ
Revised: ≤ γ 2 , where for every j ∈ [ m 2 ] , i ∈ [ n ] : V ′ j ⊥ ϕ

Original: (0) ( x i ) , and for every i ∈ [ n ] , j ∈ [ m 1 ] , j / ∈ P for P ⊆
Revised: (0) ( x i ) , and for every i ∈ [ n ] , j ∈ [ m 1 ] , j / ∈ P for P ⊆

Original: [ m 1 ] deﬁned in Lemma 1, it satisﬁes sign(( W (0)
Revised: [ m 1 ] deﬁned in Lemma 1, it satisﬁes sign(( W (0)

Original: j
Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: sign( W (0) x i ) .
Revised: sign( W (0) x i ) .

Original: Then, for large enough overparameterization, we have the following bound on the Rademacher complexity:
Revised: Then, for large enough overparameterization, we have the following bound on the Rademacher complexity:

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem
Revised: Proof of Theorem

Original: Here, we do not have the smoothing matrices W ρ , V ρ anymore.
Revised: Here, we do not have the smoothing matrices W ρ , V ρ anymore.

Original: In this section, unlike the optimization section that we used { x ′ i } ni =1 to denote the output of the ﬁrst layer by incorporating also the smoothing matrices, here we deﬁne it without them:
Revised: In this section, unlike the optimization section that we used { x ′ i } ni =1 to denote the output of the ﬁrst layer by incorporating also the smoothing matrices, here we deﬁne it without them:

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne the matrices
Revised: Now deﬁne the matrices

Original: [Equation]
Revised: [Equation]

Original: To bound the Z
Revised: To bound the Z

Original: i + part, note that substituting C 1 by γ 1 in lemma 6 and assuming conditions
Revised: i + part, note that substituting C 1 by γ 1 in lemma 6 and assuming conditions

Original: [Equation]
Revised: [Equation]

Original: (we can use this result because we do not have the smoothing matrix W ρ here), we get with high probability over the initialization for every i ∈ [ n ] :
Revised: (we can use this result because we do not have the smoothing matrix W ρ here), we get with high probability over the initialization for every i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: Now using the fact that V j − V (0) j is orthogonal to ϕ (0)
Revised: Now using the fact that V j − V (0) j is orthogonal to ϕ (0)

Original: [Equation]
Revised: [Equation]

Original: x i
Revised: x i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Next, using the upper bound on ∥ x ′ i − ϕ (0) ( x i ) ∥ :
Revised: Next, using the upper bound on ∥ x ′ i − ϕ (0) ( x i ) ∥ :

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: applying the ﬁrst argument of Lemma
Revised: applying the ﬁrst argument of Lemma

Original: [Equation]
Revised: [Equation]

Original: we have with high probability over the randomness of V (0) :
Revised: we have with high probability over the randomness of V (0) :

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can write:
Revised: Therefore, we can write:

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term, for every j ∈ [ m 2 ] , deﬁne H j to be the set of i ’s in [ n ] where the j th column of
Revised: For the ﬁrst term, for every j ∈ [ m 2 ] , deﬁne H j to be the set of i ’s in [ n ] where the j th column of

Original: Z ′ i is non-zero, i.e.
Revised: Z ′ i is non-zero, i.e.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, we use the crucial assumption that ( V − V (0) )
Revised: Here, we use the crucial assumption that ( V − V (0) )

Original: jT ϕ (0) ( x i ) = 0 , so we can drop the ϕ (0) ( x i ) term when x ′ i is multiplied to V − V (0) .
Revised: jT ϕ (0) ( x i ) = 0 , so we can drop the ϕ (0) ( x i ) term when x ′ i is multiplied to V − V (0) .

Original: Using this trick and applying Cauchy Schwarz, we bound the ﬁrst term as:
Revised: Using this trick and applying Cauchy Schwarz, we bound the ﬁrst term as:

Original: [Equation]
Revised: [Equation]

Original: Further using Jensen’s inequality:
Revised: Further using Jensen’s inequality:

Original: [Equation]
Revised: [Equation]

Original: Using Equation (111) of Lemma 6 (note that we do not have the smoothing matrix W ρ here, so we are allowed to use this result), we obtain
Revised: Using Equation (111) of Lemma 6 (note that we do not have the smoothing matrix W ρ here, so we are allowed to use this result), we obtain

Original: [Equation]
Revised: [Equation]

Original: where Z ki ’s are deﬁned in Equation (318).
Revised: where Z ki ’s are deﬁned in Equation (318).

Original: Plugging this back in (82):
Revised: Plugging this back in (82):

Original: [Equation]
Revised: [Equation]

Original: Now for every ﬁxed dataset, with high probability over the randomness of W s , for every k 1 ̸ = k 2 : (cid:12)(cid:12)(cid:12) ⟨ X
Revised: Now for every ﬁxed dataset, with high probability over the randomness of W s , for every k 1 ̸ = k 2 : (cid:12)(cid:12)(cid:12) ⟨ X

Original: [Equation]
Revised: [Equation]

Original: But note that because ⟨ x i 1 , x i 2 ⟩
Revised: But note that because ⟨ x i 1 , x i 2 ⟩

Original: [Equation]
Revised: [Equation]

Original: 1 , the variables W sk
Revised: 1 , the variables W sk

Original: [Equation]
Revised: [Equation]

Original: j W sk
Revised: j W sk

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: { W
Revised: { W

Original: [Equation]
Revised: [Equation]

Original: 0 } 1 { W (0) j, x i 2 ≥ 0 } are subgaussian with parameter one with respect to the randomness of W s .
Revised: 0 } 1 { W (0) j, x i 2 ≥ 0 } are subgaussian with parameter one with respect to the randomness of W s .

Original: Hence, with high probability over the randomness of W s , we get
Revised: Hence, with high probability over the randomness of W s , we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, with high probability over the randomness of W (0) and W ′ and the dataset, we get
Revised: Therefore, with high probability over the randomness of W (0) and W ′ and the dataset, we get

Original: Equation (84).
Revised: Equation (84).

Original: In order to get rid of the high probability argument on the dataset, we use the stronger
Revised: In order to get rid of the high probability argument on the dataset, we use the stronger

Original: Equation (319) in Lemma 46 which uniformly bounds ⟨ Z k 1 ( x ) , Z k 2 ( x ′ ) ⟩ by log( m 1 ) d/m 1 for any x, x ′ , which in turn gives
Revised: Equation (319) in Lemma 46 which uniformly bounds ⟨ Z k 1 ( x ) , Z k 2 ( x ′ ) ⟩ by log( m 1 ) d/m 1 for any x, x ′ , which in turn gives

Original: [Equation]
Revised: [Equation]

Original: with high probability, independent of the choice of dataset.
Revised: with high probability, independent of the choice of dataset.

Original: This bounds is slightly worse comapred to (84), but still efﬁcient for our purpose.
Revised: This bounds is slightly worse comapred to (84), but still efﬁcient for our purpose.

Original: Furthermore, a similar bound to Equation (84) can be obtained in a more adversarial situation when we also take maximum against the choice of the dataset.
Revised: Furthermore, a similar bound to Equation (84) can be obtained in a more adversarial situation when we also take maximum against the choice of the dataset.

Original: Note that the entries of P i ∈
Revised: Note that the entries of P i ∈

Original: H
Revised: H

Original: j ϵ i Z ki for 1 ≤ k ≤ m 3 can differ only in a sign.
Revised: j ϵ i Z ki for 1 ≤ k ≤ m 3 can differ only in a sign.

Original: Therefore, their norms are all equal.
Revised: Therefore, their norms are all equal.

Original: Now suppose that C j is the random variable of the norm of these variables:
Revised: Now suppose that C j is the random variable of the norm of these variables:

Original: [Equation]
Revised: [Equation]

Original: Then, by substituting r k = 1 C j
Revised: Then, by substituting r k = 1 C j

Original: P i ∈
Revised: P i ∈

Original: H
Revised: H

Original: j ϵ
Revised: j ϵ

Original: i Z
Revised: i Z

Original: [Equation]
Revised: [Equation]

Original: in Lemma 40, we get
Revised: in Lemma 40, we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now recall from Equation (80), we have
Revised: Now recall from Equation (80), we have

Original: [Equation]
Revised: [Equation]

Original: Hence, we can apply Corollary 5.1 with ϕ (2) ( x i ) and C 1 substituted by ϕ ′ ( x i ) and γ 1 respectively, to argue with high probability over the initialization, there exists a set ˜ P
Revised: Hence, we can apply Corollary 5.1 with ϕ (2) ( x i ) and C 1 substituted by ϕ ′ ( x i ) and γ 1 respectively, to argue with high probability over the initialization, there exists a set ˜ P

Original: i such that for every i ∈ [ n ] and j / ∈ ˜
Revised: i such that for every i ∈ [ n ] and j / ∈ ˜

Original: P
Revised: P

Original: i , sign of V jT x ′ i is the same as V (0)
Revised: i , sign of V jT x ′ i is the same as V (0)

Original: j T ϕ (0) ( x i ) , and moreover,|
Revised: j T ϕ (0) ( x i ) , and moreover,

Original: [Equation]
Revised: [Equation]

Original: Now let
Revised: Now let

Original: [Equation]
Revised: [Equation]

Original: Note that for j / ∈ ˜ P = S i ˜ P
Revised: Note that for j / ∈ ˜ P = S i ˜ P

Original: i , we have H j = ¯ H j .
Revised: i , we have H j = ¯ H j .

Original: Now note that the norm of each P i ∈
Revised: Now note that the norm of each P i ∈

Original: H
Revised: H

Original: j ϵ i Z ki is at most one.
Revised: j ϵ i Z ki is at most one.

Original: for each index 1 ≤ ℓ ≤ m 1 d , as the random variables P i ∈
Revised: for each index 1 ≤ ℓ ≤ m 1 d , as the random variables P i ∈

Original: ¯ H j ϵ
Revised: ¯ H j ϵ

Original: i ( Z ki ) ℓ are P i ∈
Revised: i ( Z ki ) ℓ are P i ∈

Original: ¯ H j ( Z ki ) 2 ℓ ≤ P i ∈ [ n ] ( Z ki ) 2 ℓ subgaussian, we have with probability at least 1 − 1 n over the randomness of ϵ i ’s, for every 1 ≤ ℓ ≤ m 1 d and every 1 ≤ j ≤ m 2 :
Revised: ¯ H j ( Z ki ) 2 ℓ ≤ P i ∈ [ n ] ( Z ki ) 2 ℓ subgaussian, we have with probability at least 1 − 1 n over the randomness of ϵ i ’s, for every 1 ≤ ℓ ≤ m 1 d and every 1 ≤ j ≤ m 2 :

Original: [Equation]
Revised: [Equation]

Original: X i ∈ ¯
Revised: X i ∈ ¯

Original: H
Revised: H

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: , which implies for every j ∈ [ m 2 ] :
Revised: , which implies for every j ∈ [ m 2 ] :

Original: [Equation]
Revised: [Equation]

Original: Name this event B , so
Revised: Name this event B , so

Original: [Equation]
Revised: [Equation]

Original: Note that although H j might depend on the randomness of ϵ i ’s, ¯ H j does not, and if j / ∈ ˜ P , we obtain
Revised: Note that although H j might depend on the randomness of ϵ i ’s, ¯ H j does not, and if j / ∈ ˜ P , we obtain

Original: C j = ∥
Revised: C j = ∥

Original: X i ∈ ¯
Revised: X i ∈ ¯

Original: H j [Equation]
Revised: H j ϵ

Original: i Z ki ∥ ≤ p n log( m 1 dm 2 n ) .
Revised: i Z ki ∥ ≤ p n log( m 1 dm 2 n ) .

Original: Moreover, note that we also have the following worse case bound:
Revised: Moreover, note that we also have the following worse case bound:

Original: [Equation]
Revised: [Equation]

Original: Applying the last two inequalities into Equations (298) and (86):
Revised: Applying the last two inequalities into Equations (298) and (86):

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: (cid:16) trace (( W − W (
Revised: (cid:16) trace (( W − W (

Original: [Equation]
Revised: [Equation]

Original: X i ∈ H
Revised: X i ∈ H

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: Next, we analyze the term 1 n E ϵ sup W ∈ S trace ( V (0) ( P ni =

Revised: Next, we analyze the term 1 n E ϵ sup W ∈ S trace ( V (0) ( P ni =

Original: ϵ i Z ′ i )) .
Revised: ϵ i Z ′ i )) .

Original: Noting that ∥ ϕ (0) ( x i ) ∥ ≲ κ√ m 3 with high probability and using Equation (87):
Revised: Noting that ∥ ϕ (0) ( x i ) ∥ ≲ κ√ m 3 with high probability and using Equation (87):

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: But using Lemma 30:
Revised: But using Lemma 30:

Original: [Equation]
Revised: [Equation]

Original: Applying a similar bound as we did in Equation (89) on ∥ x ′ i ∥ :
Revised: Applying a similar bound as we did in Equation (89) on ∥ x ′ i ∥ :

Original: [Equation]
Revised: [Equation]

Original: Substituting above, we get
Revised: Substituting above, we get

Original: [Equation]
Revised: [Equation]

Original: Finally, Substituting Equations (88) into (82), then combining it with (90) into (81), we obtain a bound on Rademacher complexity which holds w.h.p over both the randomness of the initialization and the dataset:
Revised: Finally, Substituting Equations (88) into (82), then combining it with (90) into (81), we obtain a bound on Rademacher complexity which holds w.h.p over both the randomness of the initialization and the dataset:

Original: [Equation]
Revised: [Equation]

Original: Having enough overparameterization, we have for every dataset ( x, y ) (i.e. worst-case Rademacher complexity):
Revised: Having enough overparameterization, we have for every dataset ( x, y ) (i.e. worst-case Rademacher complexity):

Original: [Equation]
Revised: [Equation]

Original: Note that for the bound (93) to hold, the overparameterization should be picked poly large in γ 1 , γ 2 , as well as in other basic parameters.
Revised: Note that for the bound (93) to hold, the overparameterization should be picked poly large in γ 1 , γ 2 , as well as in other basic parameters.

Original: However, noting Equations (59) and (66) in the proof of
Revised: However, noting Equations (51) and (58) in the proof of

Original: Theorem 3, we set γ 1 = 1 , γ 2 ≥ Ω( B, n, 1 /γ 0 ) in Theorem 3, so γ 1 γ 2 is at most poly in the basic parameters.
Revised: Theorem 3, we set γ 1 = 1 , γ 2 ≥ Ω( B, n, 1 /γ 0 ) in Theorem 3, so γ 1 γ 2 is at most poly in the basic parameters.

Original: Therefore, again the overparameterization can be picked polynomially large in the basic only parameters (i.e. independent of γ 1 , γ 2 or ζ ).
Revised: Therefore, again the overparameterization can be picked polynomially large in the basic only parameters (i.e. independent of γ 1 , γ 2 or ζ ).

Original: A. C ONSTRUCTING W ∗ , V ∗
Revised: A. C ONSTRUCTING W ∗ , V ∗

Original: This section consists of two subsections; First, we prove a structural result for the ﬁrst layer weights ( W ′ , V ′ ) that the algorithm visits, then construct a weight matrix W ∗ for the ﬁrst layer with some good properties.
Revised: This section consists of two subsections; First, we prove a structural result for the ﬁrst layer weights ( W ′ , V ′ ) that the algorithm visits, then construct a weight matrix W ∗ for the ﬁrst layer with some good properties.

Original: Second, we do the same thing for the second layer (however, the structure of the ﬁrst and second layers are completely different).
Revised: Second, we do the same thing for the second layer (however, the structure of the ﬁrst and second layers are completely different).

Original: Through out this section, we assume we have the norm bounds ∥ W ′ ∥
Revised: Through out this section, we assume we have the norm bounds ∥ W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 .
Revised: ≤ C 2 .

Original: Notably, we rely on a number of basic Lemmas more related to the representation power of the network, which we defer their proof into a later Appendix B.2 and refer to them here as needed.
Revised: Notably, we rely on a number of basic Lemmas more related to the representation power of the network, which we defer their proof into a later Appendix B.2 and refer to them here as needed.

Original: A.12. F IRST L AYER , C ONSTRUCTION OF W ∗
Revised: A.12. F IRST L AYER , C ONSTRUCTION OF W ∗

Original: Lemma 1 Suppose m 1 ≥ 16 n 2 m 23 /λ 20 .
Revised: Lemma 1 Suppose m 1 ≥ 16 n 2 m 23 /λ 20 .

Original: Let P i = { j ∈ [ m 1 ] | | W (0)
Revised: Let P i = { j ∈ [ m 1 ] | | W (0)

Original: j x i | ≤ c 2 / √ m 1 } and P = ∪ P i .
Revised: j x i | ≤ c 2 / √ m 1 } and P = ∪ P i .

Original: During SGD iterations, suppose we have ∥ W ′ ∥
Revised: During SGD iterations, suppose we have ∥ W ′ ∥

Original: F ≤ C 1 .
Revised: F ≤ C 1 .

Original: Then, for a value c 2 satisfying
Revised: Then, for a value c 2 satisfying

Original: [Equation]
Revised: [Equation]

Original: with high probability ∀ i : | P i | ≲ c 2 √ m 1 /κ 1 , and for j / ∈ P , during the whole algorithm we have
Revised: with high probability ∀ i : | P i | ≲ c 2 √ m 1 /κ 1 , and for j / ∈ P , during the whole algorithm we have

Original: [Equation]
Revised: [Equation]

Original: So the signs of neurons outside P never changes.
Revised: So the signs of neurons outside P never changes.

Original: In particular, we can set c 2 as small as c 2 =
Revised: In particular, we can set c 2 as small as c 2 =

Original: C 1 √ nm 3 / √ λ 0 .
Revised: C 1 √ nm 3 / √ λ 0 .

Original: In the rest of the proof (i.e. other sections), we set c 2 to this value.
Revised: In the rest of the proof (i.e. other sections), we set c 2 to this value.

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Deﬁne the matrix
Revised: Deﬁne the matrix

Original: [Equation]
Revised: [Equation]

Original: Let P i be the set of indices j such that 1 { W (0) j T x i ≥ c 2 / √ m 1 } is zero.
Revised: Let P i be the set of indices j such that 1 { W (0) j T x i ≥ c 2 / √ m 1 } is zero.

Original: First of all, note that by
Revised: First of all, note that by

Original: Bernstein inequality:
Revised: Bernstein inequality:

Original: [Equation]
Revised: [Equation]

Original: Now suppose that until the current iteration of the algorithm the assumption has been true, i.e. the signs of the neurons outside of P have never changed.
Revised: Now suppose that until the current iteration of the algorithm the assumption has been true, i.e. the signs of the neurons outside of P have never changed.

Original: As a result, due to the speciﬁc update of the
Revised: As a result, due to the speciﬁc update of the

Original: SGD for both of the terms E Z ℓ ( f V ′ ,W ′ ( x ) , y ) and ∥ W ′ ∥
Revised: SGD for both of the terms E Z ℓ ( f V ′ ,W ′ ( x ) , y ) and ∥ W ′ ∥

Original: 2 F , if we deﬁne W ′ | P to be the restriction of W ′ to indices that are not in P (i.e. the columns in P are equal to zero), then we can write
Revised: 2 F , if we deﬁne W ′ | P to be the restriction of W ′ to indices that are not in P (i.e. the columns in P are equal to zero), then we can write

Original: [Equation]
Revised: [Equation]

Original: An issue here is that we also have some injected noise by PSGD into W ′ which violates Equation (94).
Revised: An issue here is that we also have some injected noise by PSGD into W ′ which violates Equation (94).

Original: To handle the injected noise as well, we deﬁne the subspace Φ ′ of R m 1 × d matrices to be the set of vectors with arbitrary rows for j ∈ [ m 1 ] with j ∈ P , while restricted to the other rows j / ∈ P in should be in the span of ( ˜ Z ik ) i,k .
Revised: To handle the injected noise as well, we deﬁne the subspace Φ ′ of R m 1 × d matrices to be the set of vectors with arbitrary rows for j ∈ [ m 1 ] with j ∈ P , while restricted to the other rows j / ∈ P in should be in the span of ( ˜ Z ik ) i,k .

Original: Then, we decompose W ′ into subspaces Φ ′ and Φ ′⊥ respectively as W ′ =
Revised: Then, we decompose W ′ into subspaces Φ ′ and Φ ′⊥ respectively as W ′ =

Original: W ′ (1) + W ′ (2) , where W ′ (1) ∈
Revised: W ′ (1) + W ′ (2) , where W ′ (1) ∈

Original: Φ ′ , W ′ (2) ∈ Φ ′⊥ .
Revised: Φ ′ , W ′ (2) ∈ Φ ′⊥ .

Original: Here, we want to prove ∥ W ′ (1) j ∥ ≤ c 2 / (4 √ m 1 ) .
Revised: Here, we want to prove ∥ W ′ (1) j ∥ ≤ c 2 / (4 √ m 1 ) .

Original: We handle the ∥ W ′ (2) j ∥ part in Appendix 45.
Revised: We handle the ∥ W ′ (2) j ∥ part in Appendix 45.

Original: So instead of W ′ | P in Equation (94) we consider W ′ (1)
Revised: So instead of W ′ | P in Equation (94) we consider W ′ (1)

Original: | P :
Revised: | P :

Original: [Equation]
Revised: [Equation]

Original: k,i ˜ Z ik .
Revised: k,i ˜ Z ik .

Original: We handle the other part W ′ (2) in Appendix 45.
Revised: We handle the other part W ′ (2) in Appendix 45.

Original: Now exactly similar to the drivation in Lemma 38, we can state with high probability
Revised: Now exactly similar to the drivation in Lemma 38, we can state with high probability

Original: [Equation]
Revised: [Equation]

Original: Note that we are exploiting the fact that the norm ∥ W ′ ∥ remains bounded by C 1 .
Revised: Note that we are exploiting the fact that the norm ∥ W ′ ∥ remains bounded by C 1 .

Original: Now using a
Revised: Now using a

Original: Hoeffding bound for matrix H ∞′ deﬁned below
Revised: Hoeffding bound for matrix H ∞′ deﬁned below

Original: [Equation]
Revised: [Equation]

Original: we write:
Revised: we write:

Original: [Equation]
Revised: [Equation]

Original: Now opening Equation (96) and using the property c 2 ≤ k 1 λ 0 √ m 1 / (2 n 2 ) , we get
Revised: Now opening Equation (96) and using the property c 2 ≤ k 1 λ 0 √ m 1 / (2 n 2 ) , we get

Original: [Equation]
Revised: [Equation]

Original: For the last line to hold
Revised: For the last line to hold

Original: [Equation]
Revised: [Equation]

Original: we need enough overparameterization.
Revised: we need enough overparameterization.

Original: This implies X
Revised: This implies X

Original: [Equation]
Revised: [Equation]

Original: Now again, exactly similar to the derivation in Lemma
Revised: Now again, exactly similar to the derivation in Lemma

Original: [Equation]
Revised: [Equation]

Original: for j / ∈ P we have
Revised: for j / ∈ P we have

Original: [Equation]
Revised: [Equation]

Original: which completes most of the proof.
Revised: which completes most of the proof.

Original: For the rest
Revised: For the rest

Original: [Equation]
Revised: [Equation]

Original: we are left to show that for the other part W ′ (2) , we have ∥ W ′ (2) j ∥ ≤ c 2 / (4 √ m 1 ) , which we do in Appendix 45.
Revised: we are left to show that for the other part W ′ (2) , we have ∥ W ′ (2) j ∥ ≤ c 2 / (4 √ m 1 ) , which we do in Appendix 45.

Original: Lemma 2 Under condition m 3 n/ √ m 1 ≤ λ 0 / 4 , there exist matrices { W ∗ k } m 3 k =1 ∈ R m 

Revised: Lemma 2 Under condition m 3 n/ √ m 1 ≤ λ 0 / 4 , there exist matrices { W ∗ k } m 3 k =1 ∈ R m

Original: × d s.t.
Revised: × d s.t.

Original: for every k ̸ = k ′ ∈
Revised: for every k ̸ = k ′ ∈

Original: [ m 3 ] and i ∈ [ n ] :
Revised: [ m 3 ] and i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for k 1 ̸ = k 2 :
Revised: Furthermore, for k 1 ̸ = k 2 :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: we want to compute the norm of the projection P ( W + k ) of W + k onto the subspace spanned by all
Revised: we want to compute the norm of the projection P ( W + k ) of W + k onto the subspace spanned by all

Original: Z ik ′ for k ′ ̸
Revised: Z ik ′ for k ′ ̸

Original: = k and i ∈ [ n ] :
Revised: = k and i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst and third terms are vectors and the middle term is a matrix.
Revised: where the ﬁrst and third terms are vectors and the middle term is a matrix.

Original: Now note that for each k ′ , k 1 , k 2 ̸
Revised: Now note that for each k ′ , k 1 , k 2 ̸

Original: = k , by Hoeffding inequality:
Revised: = k , by Hoeffding inequality:

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Now Equation (100) implies for small enough m
Revised: Now Equation (100) implies for small enough m

Original: [Equation]
Revised: [Equation]

Original: as long as λ 0 ≥ 2 n/m 1 .
Revised: as long as λ 0 ≥ 2 n/m 1 .

Original: Moreover, deﬁne ˜
Revised: Moreover, deﬁne ˜

Original: A to be the block version of
Revised: A to be the block version of

Original: [Equation]
Revised: [Equation]

Original: i.e. for k 1 = k 2 they are the same but for k 1 ̸ = k 

Revised: i.e. for k 1 = k 2 they are the same but for k 1 ̸ = k

Original: ˜
Revised: ˜

Original: A is zero.
Revised: A is zero.

Original: Then λ min ( ˜ A ) ≥ λ 0 / 2 , because the eigenvalues of each block is at least λ 0 / 2 using Equation (103).
Revised: Then λ min ( ˜ A ) ≥ λ 0 / 2 , because the eigenvalues of each block is at least λ 0 / 2 using Equation (103).

Original: But note that
Revised: But note that

Original: [Equation]
Revised: [Equation]

Original: So as long as m 3 n/ √ m 1 ≤ λ 0 / 4 , we have λ min ( A ) ≥ λ 0 / 4 .
Revised: So as long as m 3 n/ √ m 1 ≤ λ 0 / 4 , we have λ min ( A ) ≥ λ 0 / 4 .

Original: Combining this fact with Equation (102) and plugging it into Equation (99), we obtain
Revised: Combining this fact with Equation (102) and plugging it into Equation (99), we obtain

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne W ∗ k = W + k − P
Revised: Now deﬁne W ∗ k = W + k − P

Original: ( W + k ) .
Revised: ( W + k ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Furthermore, note that W ∗ k 2 is orthogonal to W + k 1 for k 1 ̸ = k 2 , so
Revised: Furthermore, note that W ∗ k 2 is orthogonal to W + k 1 for k 1 ̸ = k 2 , so

Original: [Equation]
Revised: [Equation]

Original: But note that
Revised: But note that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can bound Equation (104) as:
Revised: we can bound Equation (104) as:

Original: [Equation]
Revised: [Equation]

Original: Lemma 3 There exists a matrix W +2 k such that for every j ∈ P , W +

Revised: Lemma 3 There exists a matrix W +2 k such that for every j ∈ P , W +

Original: k j
Revised: k j

Original: = 0 , and |
Revised: = 0 , and |

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Deﬁne W +2 k to be equal to W + k for j / ∈ P and equal to zero vector otherwise.
Revised: Deﬁne W +2 k to be equal to W + k for j / ∈ P and equal to zero vector otherwise.

Original: Then, by
Revised: Then, by

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Combining this with Lemma 37, the desired result follows.
Revised: Combining this with Lemma 37, the desired result follows.

Original: Lemma 4 Under condition m 3 n/ √ m 1 ≤ λ 0 / 4 , there exist matrix W ∗
Revised: Lemma 4 Under condition m 3 n/ √ m 1 ≤ λ 0 / 4 , there exist matrix W ∗

Original: k ’s exactly satisfying the same conditions in Lemma 2 but with respect to W +2 k instead of W + k , and moreover, for j ∈ P we have
Revised: k ’s exactly satisfying the same conditions in Lemma 2 but with respect to W +2 k instead of W + k , and moreover, for j ∈ P we have

Original: W ∗ k j = 0 .
Revised: W ∗ k j = 0 .

Original: Proof of Lemma
Revised: Proof of Lemma

Original: We can repeat the exact same procedure of Lemma 2 for W +2 k .
Revised: We can repeat the exact same procedure of Lemma 2 for W +2 k .

Original: Using the bound in Equation (97), we have (cid:16)
Revised: Using the bound in Equation (97), we have (cid:16)

Original: [Equation]
Revised: [Equation]

Original: so as long as
Revised: so as long as

Original: [Equation]
Revised: [Equation]

Original: with similar argument as in Lemma 2, we get
Revised: with similar argument as in Lemma 2, we get

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: Thus, using the same argument as before the proof is complete.
Revised: Thus, using the same argument as before the proof is complete.

Original: Lemma 5 Suppose
Revised: Lemma 5 Suppose

Original: [Equation]
Revised: [Equation]

Original: During SGD, suppose we are currently at ( V ′ , W ′ ) with W ′ ≤ C 1 .
Revised: During SGD, suppose we are currently at ( V ′ , W ′ ) with W ′ ≤ C 1 .

Original: For any matrix W 1 , we denote the signs of the ﬁrst layer imposed by W 1 by D W 1 ,x i .
Revised: For any matrix W 1 , we denote the signs of the ﬁrst layer imposed by W 1 by D W 1 ,x i .

Original: Then with high probability, there exists
Revised: Then with high probability, there exists

Original: W ∗ =
Revised: W ∗ =

Original: P k ∈ [ m 3 ] W ∗ k such that W ∗ k ’s is orthogonal to all other Z ik ′ ’s for k ′ ̸
Revised: P k ∈ [ m 3 ] W ∗ k such that W ∗ k ’s is orthogonal to all other Z ik ′ ’s for k ′ ̸

Original: = k , and for every i ∈ [ n ] , we have:
Revised: = k , and for every i ∈ [ n ] , we have:

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have
Revised: Moreover, we have

Original: [Equation]
Revised: [Equation]

Original: Particularly, for any diagonal sign matrix Σ ∈ R m 3 × m 3 , we have
Revised: Particularly, for any diagonal sign matrix Σ ∈ R m 3 × m 3 , we have

Original: [Equation]
Revised: [Equation]

Original: which, by having enough overparameterization
Revised: which, by having enough overparameterization

Original: [Equation]
Revised: [Equation]

Original: implies
Revised: implies

Original: [Equation]
Revised: [Equation]

Original: ) where
Revised: ) where

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: From Lemma 3, we have
Revised: From Lemma 3, we have

Original: [Equation]
Revised: [Equation]

Original: Combining this with the result of Lemma 4, we get:
Revised: Combining this with the result of Lemma 4, we get:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, based on the property that W ∗ k j = 0 for j ∈ P and its orthogonal property from
Revised: On the other hand, based on the property that W ∗ k j = 0 for j ∈ P and its orthogonal property from

Original: Lemma 4, for j ∈ P we get
Revised: Lemma 4, for j ∈ P we get

Original: [Equation]
Revised: [Equation]

Original: which combined with Equation (110) completes the proof.
Revised: which combined with Equation (110) completes the proof.

Original: From the above, Equation (109) is also clear.
Revised: From the above, Equation (109) is also clear.

Original: Finally, note that by Lemma 38 we have
Revised: Finally, note that by Lemma 38 we have

Original: [Equation]
Revised: [Equation]

Original: which Combined with Lemma 4 implies
Revised: which Combined with Lemma 4 implies

Original: [Equation]
Revised: [Equation]

Original: while the other claims follows from Lemma 39 and Lemma 4, combined with Equation (98):
Revised: while the other claims follows from Lemma 39 and Lemma 4, combined with Equation (98):

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we move on to construct V ∗ for the second layer
Revised: we move on to construct V ∗ for the second layer

Original: [Equation]
Revised: [Equation]

Original: A.12. S ECOND L AYER , C ONSTRUCTION OF V ∗
Revised: A.12. S ECOND L AYER , C ONSTRUCTION OF V ∗

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we present a couple of lemmas that step by step lead to the construction of V ∗ .
Revised: we present a couple of lemmas that step by step lead to the construction of V ∗ .

Original: we remind the reader that ϕ (0) ( x i ) is the output of the ﬁrst layer at initialization weights, ϕ ′ ( x i ) and ϕ (2) ( x i ) are the changes in the output of the ﬁrst layer when W ′ and W ′
Revised: we remind the reader that ϕ (0) ( x i ) is the output of the ﬁrst layer at initialization weights, ϕ ′ ( x i ) and ϕ (2) ( x i ) are the changes in the output of the ﬁrst layer when W ′ and W ′

Original: + W ρ are added, respectively, and ﬁnally ϕ ∗ ( x i ) is the optimal features that are generated by the matrix W ∗ but
Revised: + W ρ are added, respectively, and ﬁnally ϕ ∗ ( x i ) is the optimal features that are generated by the matrix W ∗ but

Original: with the sign pattern of W (0) + W ′ , i.e.
Revised: with the sign pattern of W (0) + W ′ , i.e.

Original: [Equation]
Revised: [Equation]

Original: We also deﬁne x ′ i as
Revised: We also deﬁne x ′ i as

Original: [Equation]
Revised: [Equation]

Original: To begin, we state a lemma to bound the magnitude of ∥ ϕ ′ ( x i ) ∥ , given that the norm of W ′ is bounded by C 1 and the sign pattern Sgn
Revised: To begin, we state a lemma to bound the magnitude of ∥ ϕ ′ ( x i ) ∥ , given that the norm of W ′ is bounded by C 1 and the sign pattern Sgn

Original: [Equation]
Revised: [Equation]

Original: satisﬁes condition stated for the set of indices P in Lemma 1.
Revised: satisﬁes condition stated for the set of indices P in Lemma 1.

Original: Later on, we exploit this Lemma in Lemma 33 to state bounds for ∥ ϕ (2) ( x i ) ∥ .
Revised: Later on, we exploit this Lemma in Lemma 33 to state bounds for ∥ ϕ (2) ( x i ) ∥ .

Original: Lemma 6 Let the matrix W ′ with norm bound ∥ W ′ ∥
Revised: Lemma 6 Let the matrix W ′ with norm bound ∥ W ′ ∥

Original: ≤ C 1 , such that the signs of ( W (0) j + W ′ j ) x i and W (0)
Revised: ≤ C 1 , such that the signs of ( W (0) j + W ′ j ) x i and W (0)

Original: j x i can be different only for j ∈ P , for P deﬁned in Lemma 1.
Revised: j x i can be different only for j ∈ P , for P deﬁned in Lemma 1.

Original: (Note that for W ′ at every step of the algorithm, this is automatically satisﬁed by Lemma 1)
Revised: (Note that for W ′ at every step of the algorithm, this is automatically satisﬁed by Lemma 1)

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Particularly for large enough m 1 compared to n, m 3 , λ 0 , κ 1 , C 1 , we have
Revised: Particularly for large enough m 1 compared to n, m 3 , λ 0 , κ 1 , C 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: where the last line follows from the bound on | P | from Lemma 1.
Revised: where the last line follows from the bound on | P | from Lemma 1.

Original: On the other hand, because by Hoeffding we know that ⟨ Z ik , Z ik ′ ⟩ ≲ 1 / √ m 1 by Lemma 40, we get m
Revised: On the other hand, because by Hoeffding we know that ⟨ Z ik , Z ik ′ ⟩ ≲ 1 / √ m 1 by Lemma 40, we get m

Original: [Equation]
Revised: [Equation]

Original: Combining this with Equation (111), we get
Revised: Combining this with Equation (111), we get

Original: [Equation]
Revised: [Equation]

Original: Next, we prove a structural lemma regarding the sign pattern in the second layer when we feed in x ′ i to it, with the important message that the dominance of sign patterns are speciﬁed by ϕ (0) ( x i ) .
Revised: Next, we prove a structural lemma regarding the sign pattern in the second layer when we feed in x ′ i to it, with the important message that the dominance of sign patterns are speciﬁed by ϕ (0) ( x i ) .

Original: Lemma 7 Suppose we have m 3 κ 2 1 ≳ C 2 1 , κ 2 √ m 2 ≥ C 2 , and m 1 satisﬁes the condition on
Revised: Lemma 7 Suppose we have m 3 κ 2 1 ≳ C 2 1 , κ 2 √ m 2 ≥ C 2 , and m 1 satisﬁes the condition on

Original: Lemma 6.
Revised: Lemma 6.

Original: If we have the condition ∥ ϕ (2) ( x i ) ∥
Revised: If we have the condition ∥ ϕ (2) ( x i ) ∥

Original: ≲ C 1 , which happens under the high probability event E c deﬁned in Lemma 33, then for every i ∈ [ n ] , there exist a subset ˜ P i which might depend on
Revised: ≲ C 1 , which happens under the high probability event E c deﬁned in Lemma 33, then for every i ∈ [ n ] , there exist a subset ˜ P i which might depend on

Original: W (0) , V (0) , W ′ , V ′ , such that
Revised: W (0) , V (0) , W ′ , V ′ , such that

Original: [Equation]
Revised: [Equation]

Original: Moreover, for every i ∈ [ n ] , for j / ∈ ˜ P
Revised: Moreover, for every i ∈ [ n ] , for j / ∈ ˜ P

Original: i ,:
Revised: i ,:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: By assumption
Revised: By assumption

Original: [Equation]
Revised: [Equation]

Original: we know that during the algorithm
Revised: we know that during the algorithm

Original: [Equation]
Revised: [Equation]

Original: we have ∥ V ′ ∥
Revised: we have ∥ V ′ ∥

Original: ≤ C 2 .
Revised: ≤ C 2 .

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: we know by
Revised: we know by

Original: Lemma 33 that under E c :
Revised: Lemma 33 that under E c :

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the set
Revised: Deﬁne the set

Original: [Equation]
Revised: [Equation]

Original: and P ′ =
Revised: and P ′ =

Original: ∪ P
Revised: ∪ P

Original: i .
Revised: i .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: so by Bernstein, with high probability:
Revised: so by Bernstein, with high probability:

Original: [Equation]
Revised: [Equation]

Original: so with high prob.
Revised: so with high prob.

Original: [Equation]
Revised: [Equation]

Original: On the other hand, Note that
Revised: On the other hand, Note that

Original: [Equation]
Revised: [Equation]

Original: is subGaussian with parameter σ 2 =
Revised: is subGaussian with parameter σ 2 =

Original: O
Revised: O

Original: (1 /m 1 P j σ ( W (0) j x i ) 2 ) .
Revised: (1 /m 1 P j σ ( W (0) j x i ) 2 ) .

Original: Furthermore, note that if we compute the variance of ϕ (0) k
Revised: Furthermore, note that if we compute the variance of ϕ (0) k

Original: ( x i ) with respect to the randomness of W s :
Revised: ( x i ) with respect to the randomness of W s :

Original: [Equation]
Revised: [Equation]

Original: which itself concentrates around 1 / 2 κ 21 ∥ x i ∥ 2 = 1 / 2 κ 21 by another Bernstein, i.e. ℵ = 1 / 2 κ 21 (1 ±
Revised: which itself concentrates around 1 / 2 κ 21 ∥ x i ∥ 2 = 1 / 2 κ 21 by another Bernstein, i.e. ℵ = 1 / 2 κ 21 (1 ±

Original: O (1 / √ m 1 )) .
Revised: O (1 / √ m 1 )) .

Original: Therefore, by concentration of subexponential variables (Bernstein), it is not hard to see that the squared norm of the vector ϕ (0) ( x i ) is ( m 3 κ 41 , κ 2 ) -subexponential and concentrates around m 3 ℵ , i.e.
Revised: Therefore, by concentration of subexponential variables (Bernstein), it is not hard to see that the squared norm of the vector ϕ (0) ( x i ) is ( m 3 κ 41 , κ 2 ) -subexponential and concentrates around m 3 ℵ , i.e.

Original: [Equation]
Revised: [Equation]

Original: ) with high probability.
Revised: ) with high probability.

Original: Combining this with the fact that ∥ ϕ (2) ( x i ) ∥ ≲ C 1 implies with high probability:
Revised: Combining this with the fact that ∥ ϕ (2) ( x i ) ∥ ≲ C 1 implies with high probability:

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne P ′′
Revised: Now deﬁne P ′′

Original: i = { j | | V ′
Revised: i = { j | | V ′

Original: j x ′ i
Revised: j x ′ i

Original: | ≥ | V (0) j ϕ (0) ( x i ) | / 3 } .
Revised: | ≥ | V (0) j ϕ (0) ( x i ) | / 3 } .

Original: If j ∈ P ′′ i − P
Revised: If j ∈ P ′′ i − P

Original: i , then by Equation (117), with high probability
Revised: i , then by Equation (117), with high probability

Original: [Equation]
Revised: [Equation]

Original: = | V ′
Revised: = | V ′

Original: j x ′ i |
Revised: j x ′ i |

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: But note that ∥ V ′ ∥
Revised: But note that ∥ V ′ ∥

Original: 2 F ≤ C 22 by our assumption, which implies
Revised: 2 F ≤ C 22 by our assumption, which implies

Original: [Equation]
Revised: [Equation]

Original: Now combining Equations (114) and (118), we ﬁnally obtain
Revised: Now combining Equations (114) and (118), we ﬁnally obtain

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne the set
Revised: Now deﬁne the set

Original: [Equation]
Revised: [Equation]

Original: Note that for every j ∈ [ m 2 ] , V (0) j ϕ (0) ( x i ) is gaussian with variance ∥ ϕ
Revised: Note that for every j ∈ [ m 2 ] , V (0) j ϕ (0) ( x i ) is gaussian with variance ∥ ϕ

Original: (0) ( x i ) ∥ over the randomness of V (0) j , so
Revised: (0) ( x i ) ∥ over the randomness of V (0) j , so

Original: [Equation]
Revised: [Equation]

Original: Therefore, if we deﬁne the set
Revised: Therefore, if we deﬁne the set

Original: [Equation]
Revised: [Equation]

Original: then for large enough m 2 , by Bernstein with high prob.
Revised: then for large enough m 2 , by Bernstein with high prob.

Original: [Equation]
Revised: [Equation]

Original: Now note that ϕ (0) ( x i ) is ﬁxed during the algorithm.
Revised: Now note that ϕ (0) ( x i ) is ﬁxed during the algorithm.

Original: On the other hand, by random matrix theory, we know that with high probability, the eigenvalues of the matrix V (0) are in ( κ 2 ( √ m 2 − √ m 3 ) , κ 2 ( √ m 2 + √ m 3 )) .
Revised: On the other hand, by random matrix theory, we know that with high probability, the eigenvalues of the matrix V (0) are in ( κ 2 ( √ m 2 − √ m 3 ) , κ 2 ( √ m 2 + √ m 3 )) .

Original: Therefore, even if the vector ϕ (2) ( x i ) is picked adversarialy (because it keeps changing during the algorithm), we get that with high probability over the randomness of
Revised: Therefore, even if the vector ϕ (2) ( x i ) is picked adversarialy (because it keeps changing during the algorithm), we get that with high probability over the randomness of

Original: V (0) :
Revised: V (0) :

Original: [Equation]
Revised: [Equation]

Original: Moreover, because ∥ ϕ (2) ( x i ) ∥ ≲ C 1 and from Equation (116), with high probability over the randomness of W (0) :
Revised: Moreover, because ∥ ϕ (2) ( x i ) ∥ ≲ C 1 and from Equation (116), with high probability over the randomness of W (0) :

Original: [Equation]
Revised: [Equation]

Original: This means that for j ∈ P
Revised: This means that for j ∈ P

Original: ′′′ i − Q
Revised: ′′′ i − Q

Original: i , combining these inequalities we conclude with high probability
Revised: i , combining these inequalities we conclude with high probability

Original: [Equation]
Revised: [Equation]

Original: which combined with (121) implies
Revised: which combined with (121) implies

Original: [Equation]
Revised: [Equation]

Original: Balancing this term with the one in Equation (120), we set α :
Revised: Balancing this term with the one in Equation (120), we set α :

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Deﬁning ˜ P
Revised: Deﬁning ˜ P

Original: i
Revised: i

Original: =
Revised: =

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: i ∪ P
Revised: i ∪ P

Original: ′′′ i , we ﬁnally get
Revised: ′′′ i , we ﬁnally get

Original: [Equation]
Revised: [Equation]

Original: Clearly by the deﬁnition of P ′′ i and P
Revised: Clearly by the deﬁnition of P ′′ i and P

Original: ′′′ i the proof is complete.
Revised: ′′′ i the proof is complete.

Original: Corollary 5.1 Under the condition ∥
Revised: Corollary 5.1 Under the condition ∥

Original: ϕ (2) ( x i ) ∥ ≲ C 1 (which happens under the event E c deﬁned in
Revised: ϕ (2) ( x i ) ∥ ≲ C 1 (which happens under the event E c deﬁned in

Original: Lemma 33), setting c 3 :=
Revised: Lemma 33), setting c 3 :=

Original: C 2 / 

Revised: C 2 /

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 3 in the previous Lemma, we obtain ∀ i ∈ [ n ] :
Revised: 3 in the previous Lemma, we obtain ∀ i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: Also for j / ∈ ˜ P
Revised: Also for j / ∈ ˜ P

Original: i , the conditions in (113) and (119) becomes the same as
Revised: i , the conditions in (113) and (119) becomes the same as

Original: [Equation]
Revised: [Equation]

Original: Hence, for every i ∈ [ n ] and for j / ∈ ˜ P
Revised: Hence, for every i ∈ [ n ] and for j / ∈ ˜ P

Original: i , with high probability:
Revised: i , with high probability:

Original: [Equation]
Revised: [Equation]

Original: Next, we state concentration result for the gram matrix of ϕ (0) ( x i ) ’s.
Revised: Next, we state concentration result for the gram matrix of ϕ (0) ( x i ) ’s.

Original: Lemma 

Revised: Lemma

Original: For every i 1 , i 2 ∈ [ n ] , with high probability over the randomness of W (0) and V (0) we have
Revised: For every i 1 , i 2 ∈ [ n ] , with high probability over the randomness of W (0) and V (0) we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: First, we compute the expectation:
Revised: First, we compute the expectation:

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: But σ ( W (0) j 1 x i 1 ) σ ( W (0)
Revised: But σ ( W (0) j 1 x i 1 ) σ ( W (0)

Original: j 2 x i 2 ) is ( m 1 κ 41 , κ 21 ) -sub-exponential, so X j ∈ [ m 1 ] σ ( W (0)
Revised: j 2 x i 2 ) is ( m 1 κ 41 , κ 21 ) -sub-exponential, so X j ∈ [ m 1 ] σ ( W (0)

Original: j x i 1 ) σ ( W (0)
Revised: j x i 1 ) σ ( W (0)

Original: j x i
Revised: j x i

Original: [Equation]
Revised: [Equation]

Original: ) , which means with high probability:
Revised: ) , which means with high probability:

Original: [Equation]
Revised: [Equation]

Original: On the other side, we know that ϕ (0) k
Revised: On the other side, we know that ϕ (0) k

Original: ( x i 1 ) is subgaussian with parameters σ= 1 /m
Revised: ( x i 1 ) is subgaussian with parameters σ= 1 /m

Original: P j ( W (0)
Revised: P j ( W (0)

Original: j x i 1 ) 2 := ℵ 1 and σ 2 = 1 /m 1 P j ( W (0)
Revised: j x i 1 ) 2 := ℵ 1 and σ 2 = 1 /m 1 P j ( W (0)

Original: j x i 2 ) 2 := ℵ 2 respectively.
Revised: j x i 2 ) 2 := ℵ 2 respectively.

Original: On the other hand, we know that by Bernstein w.h.p
Revised: On the other hand, we know that by Bernstein w.h.p

Original: [Equation]
Revised: [Equation]

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: Hence, ϕ (0) k ( x i 1 ) ϕ (2) k
Revised: Hence, ϕ (0) k ( x i 1 ) ϕ (2) k

Original: ( x i 2 ) is ( ℵ 1 ℵ 2 , √ℵ 1 ℵ 2 ) -subexponential, and so ⟨ ϕ (0) ( x i 1 ) , ϕ (0) ( x i 2 ) ⟩ is ( m 3 ℵ 1 ℵ 2 , √ℵ 1 ℵ 2 ) -subexponential.
Revised: ( x i 2 ) is ( ℵ 1 ℵ 2 , √ℵ 1 ℵ 2 ) -subexponential, and so ⟨ ϕ (0) ( x i 1 ) , ϕ (0) ( x i 2 ) ⟩ is ( m 3 ℵ 1 ℵ 2 , √ℵ 1 ℵ 2 ) -subexponential.

Original: Therefore, applying another Bernstein on the top, we get ⟨ ϕ (0) ( x i 1 ) , ϕ (0)[Equation] i
Revised: Therefore, applying another Bernstein on the top, we get ⟨ ϕ (0) ( x i 1 ) , ϕ (0)

Original: [Equation]
Revised: [Equation]

Original: Now we deﬁne the matrix L i ∈ R m 3 × m 2 , with its j th column L i,j equal to
Revised: Now we deﬁne the matrix L i ∈ R m 3 × m 2 , with its j th column L i,j equal to

Original: [Equation]
Revised: [Equation]

Original: First, we state the following lemma which characterize a concentration result for the gram matrix of ( L i ) ni =1 .
Revised: First, we state the following lemma which characterize a concentration result for the gram matrix of ( L i ) ni =1 .

Original: Lemma 9 With high probability, we have the following approximation:
Revised: Lemma 9 With high probability [Equation] we have the following approximation:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: By Hoeffding:
Revised: By Hoeffding:

Original: [Equation]
Revised: [Equation]

Original: where recall
Revised: where recall

Original: [Equation]
Revised: [Equation]

Original: measures the angle between two unit vectors based on their dot product.
Revised: measures the angle between two unit vectors based on their dot product.

Original: Now notice that according to Lemma 8, with high probability:
Revised: Now notice that according to Lemma 8, with high probability:

Original: [Equation]
Revised: [Equation]

Original: where recall F 3 : [ − 1 , +1]
Revised: where recall F 3 : [ − 1 , +1]

Original: → [ − 1 / 2 , 1 / 2] is deﬁned as:
Revised: → [ − 1 / 2 , 1 / 2] is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: It is easy to see F 3 has the property that for unit vectors x 1 , x 2 and w sampled as standard normal:
Revised: It is easy to see F 3 has the property that for unit vectors x 1 , x 2 and w sampled as standard normal:

Original: [Equation]
Revised: [Equation]

Original: But because | F 3 ( . ) | =
Revised: But because | F 3 ( . ) | =

Original: O (1)
Revised: O (1)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now notice that the derivative of F 2 , i.e. 1 / 2 π √ 1 − x 2 is increasing in the interval (0 , 1) , so for a ﬁxed δ , the maximum of | F 2 ( x ) − F 2 ( x − δ ) | happens at x = 1 .
Revised: Now notice that the derivative of F 2 , i.e. 1 / 2 π √ 1 − x 2 is increasing in the interval (0 , 1) , so for a ﬁxed δ , the maximum of | F 2 ( x ) − F 2 ( x − δ ) | happens at x = 1 .

Original: On the other hand, by writing the ﬁrst order approximation of arcsin(1 − t 2 ) around t = 0 and upper bounding its derivative in the interval [0 , 1] , we get that for 0 ≤ δ ≤ 1 :
Revised: On the other hand, by writing the ﬁrst order approximation of arcsin(1 − t 2 ) around t = 0 and upper bounding its derivative in the interval [0 , 1] , we get that for 0 ≤ δ ≤ 1 :

Original: [Equation]
Revised: [Equation]

Original: Therefore, F 2 ( x ± δ ) = F 2 ( x ) ± O ( √ δ ) .
Revised: Therefore, F 2 ( x ± δ ) = F 2 ( x ) ± O ( √ δ ) .

Original: Hence:
Revised: Hence:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: L i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ϕ ∗ ( x i 1 ) , ϕ∗
Revised: ϕ ∗ ( x i 1 ) , ϕ

Original: [Equation]
Revised: [Equation]

Original: x i
Revised: x i

Original: [Equation][Equation]
Revised: [Equation]

Original: F

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.