Original: Then, there exists weight matrix V ∗ which only depends on the random initializations W (0) , V (0) (e
Revised: Then, there exists weight matrix V ∗ which only depends on the random initializations W (0) , V (0) (e

Original: [Equation]
Revised: [Equation]

Original: g. not on V ′ and W ′ )
Revised: g. not on V ′ and W ′ )

Original: for the second layer
Revised: for the second layer

Original: [Equation]
Revised: [Equation]

Original: such that having enough overparameterization
Revised: such that having enough overparameterization

Original: [Equation]
Revised: [Equation]

Original: 127) and for every j ∈ [ m 2 ] :[Equation]
Revised: [Equation] and for every j ∈ [ m 2 ] :

Original: [Equation]
Revised: [Equation]

Original: and further under the high probability event E c deﬁned in Lemma 33:
Revised: and further under the high probability event E c deﬁned in Lemma 33:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: be the minimum norm vector which maps L i ’s to f ∗ ( x i ) ’s. As a result, for the matrix
Revised: be the minimum norm vector which maps L i ’s to f ∗ ( x i ) ’s. As a result, for the matrix

Revised: L = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: it is easy to see ∥
Revised: ⟩ (cid:17) i

Original: [Equation]
Revised: [Equation]

Original:  f 
Revised: it is easy to see ∥ V ∗ ∥ 2 F = ( f ∗ ( x i )) ni =1 T L − 1 ( f ∗ ( x i ))

Original: [Equation]
Revised: ni =1 .

Original: Now combining Lemmas 5 and 41, we get
Revised: Now combining Lemmas 5 and 41, we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 132) and ⟨| ϕ ∗ 
Revised: (131) and ⟨| ϕ ∗ ( x i 1 ) , ϕ ∗ ( x i

Original: [Equation]
Revised: [Equation]

Original: x
Revised: ) ⟩ − ⟨ ¯

Original: [Equation]
Revised: [Equation]

Original: , ϕ
Revised: i

Original: [Equation]
Revised: [Equation]

Original: −
Revised: [Equation] X

Original: [Equation]
Revised: [Equation]

Original: Now by Lemma 9:
Revised: Now by Lemma 9:

Original: [Equation]
Revised: [Equation]

Original: + other cross term .
Revised: + other cross term .

Original: By applying ⟨ ¯ x i 1 , ¯ x i 2 ⟩ ≤ ∥ ¯ x i 1 ∥∥ ¯ x i 2 ∥
Revised: By applying ⟨ ¯ x i 1 , ¯ x i 2 ⟩ ≤ ∥ ¯ x i 1 ∥∥ ¯ x i 2 ∥

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Therefore,∥
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Note that ℜ 2 naturally goes to zero (with poly dependence) as ℜ → 0 and m 1 , m 2 , m 3 are large enough.
Revised: Note that ℜ 2 naturally goes to zero (with poly dependence) as ℜ → 0 and m 1 , m 2 , m 3 are large enough.

Original: Now
Revised: Now

Original: if all of the eigenvalues of the matrix
Revised: if all of the eigenvalues of the matrix

Original: A are Ω(1 /n 2 ) , then if we overparameterize enough such that ℜ 

Revised: A are Ω(1 /n 2 ) , then if we overparameterize enough such that ℜ

Original: = O (1 /n 2 ) with small enough constant so that ℜ 2 is less than half of the smallest eigenvalue of A , then for the i th eigenvalue
Revised: = O (1 /n 2 ) with small enough constant so that ℜ 2 is less than half of the smallest eigenvalue of A , then for the i th eigenvalue

Original: λ i of A and L we can write
Revised: λ i of A and L we can write

Original: [Equation]
Revised: [Equation]

Original: so
Revised: so

Original: [Equation]
Revised: [Equation]

Original: which implies the property
Revised: which implies the property

Original: [Equation]
Revised: [Equation]

Original: However, A might have very small eigenvalues.
Revised: However, A might have very small eigenvalues.

Original: To remedie this, we use Lemma 42; we can substitute f ∗ with some ¯ f ∗ such that
Revised: To remedie this, we use Lemma 42; we can substitute f ∗ with some ¯ f ∗ such that

Original: [Equation]
Revised: [Equation]

Original: where ¯ f ∗ is on the subspace of eigenvectors of A whose eigenvalues are larger than Ω(1 /n 2 ) .
Revised: where ¯ f ∗ is on the subspace of eigenvectors of A whose eigenvalues are larger than Ω(1 /n 2 ) .

Original: But it is easy to check that in the context of Theorem 3, such substitution results in a ¯ f ∗ T A − 1 ¯ f ∗ ≤ f ∗ T A − 1 f ∗ ≤ ζ and ¯ v ( ¯ f ∗ ) parameter (as deﬁned in (43)) with respect to ¯ f ∗ which satisﬁes ¯ ν/ 2 ≤ ν .
Revised: But it is easy to check that in the context of Theorem 3, such substitution results in a ¯ f ∗ T A − 1 ¯ f ∗ ≤ f ∗ T A − 1 f ∗ ≤ ζ and ¯ v ( ¯ f ∗ ) parameter (as deﬁned in (41)) with respect to ¯ f ∗ which satisﬁes ¯ ν/ 2 ≤ ν .

Original: This enables us to use remark ?? with respect to ¯ f ∗ . Note that the algorithm is with respect to the setting ν , however we want to exploit generalization bound with respect to ¯ f ∗ whose parameter is ¯ ν as it enables us to use our analysis in this Lemma.
Revised: Note that the algorithm is with respect to the setting ν , however we want to exploit generalization bound with respect to ¯ f ∗ whose parameter is ¯ ν as it enables us to use our analysis in this Lemma.

Original: Furthermore, note that using Equation (134) we can further upper bound the empirical risk of ¯ f ∗ with that of f ∗ , which makes it straightforward to derive a similar generalization bound as in (47) with respect to f ∗ , of course with a change of constants.
Revised: Furthermore, note that using Equation (133) we can further upper bound the empirical risk of ¯ f ∗ with that of f ∗ , which makes it straightforward to derive a similar generalization bound as in (45) with respect to f ∗ , of course with a change of constants.

Revised: Note that ¯ f ∗ is just the sum of A -eigenbasis directions in f ∗ whose eigenvalues are larger than Ω(1 /n 2 ) .

Revised: Hence, given a pair ( f ∗ , G ) , as we also point out in remark 1, we can construct the suitable pair ( ¯ f ∗ , G ) algorithmically and then use that pair to initialize the parameters of the algorithm (namely ζ and ν ).

Revised: Otherwise, if we are not explicitly given a pair ( f ∗ , G ) and instead want to run the doubling trick described in Theorem 1, we do not even have any additional computation; since using Theorem 1, within the framework of the doubling trick, the risk of the ﬁnal network is competitive with respect to any choice of ( ¯ f ∗ , G ) .

Revised: Note that as we mentioned in Lemma 42, the constant 2 is arbitrary and can be reduced to any number less than 2 , and it is easy to see that one can pick choice of constants along the way such that we end up with a factor two behind the risk (ﬁrst) term in the deﬁnition of our complexity measure.

Original: Therefore, without loss of generality we can use substitute f ∗ by ¯ f ∗ and still obtain Equation (133).
Revised: Therefore, without loss of generality we can use substitute f ∗ by ¯ f ∗ and still obtain Equation (132).

Original: On the other hand, the deﬁnition of V ∗ implies
Revised: On the other hand, the deﬁnition of V ∗ implies

Original: [Equation]
Revised: [Equation]

Original: But note that by Corollary 5.1, under the high probability event E c deﬁned in Lemma 33, D V (0) ,x i and D V (0) + V ′ ,x i can only be different in the index set ˜ P i and
Revised: But note that by Corollary 5.1, under the high probability event E c deﬁned in Lemma 33, D V (0) ,x i and D V (0) + V ′ ,x i can only be different in the index set ˜ P i and

Original: [Equation]
Revised: [Equation]

Original: Therefore, for all i ∈ [ n ] :
Revised: Therefore, for all i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: which proves the ﬁrst claim.
Revised: which proves the ﬁrst claim.

Original: On the other hand [Equation] we get:
Revised: On the other hand, we get:

Original: [Equation]
Revised: [Equation]

Original: But because λ min ( L ) ≳
Revised: But because λ min ( L ) ≳

Original: 1 /n 2 , we get
Revised: 1 /n 2 , we get

Revised: V T L V ≳

Original: [Equation]{+∥V∥ 
+}
Revised: {+∥V∥ 
+}

Original:  which implies [Equation]
Revised: /n 2 , which implies ∥V∥ 2 ≲ n p ζ 2 .

Original: But now using Equation (131) [Equation] we can write
Revised: But now using Equation (130), we can write

Original: [Equation]
Revised: [Equation]

Original: which proves the other part.
Revised: which proves the other part.

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: A. E XISTENCE OF A GOOD DIRECTION
Revised: A. E XISTENCE OF A GOOD DIRECTION

Original: Our aim in this section is to show that if the objective value is above certain threshold, there exists a good random direction which reduces the objective in expectation.
Revised: Our aim in this section is to show that if the objective value is above certain threshold, there exists a good random direction which reduces the objective in expectation.

Original: Particularly our aim is to prove the following theorem (informal):
Revised: Particularly our aim is to prove the following theorem (informal):

Original: Theorem 6 For a given pair ( f ∗ , G ) with
Revised: Theorem 6 For a given pair ( f ∗ , G ) with

Original: [Equation]
Revised: [Equation]

Original: recall the ideal random matrices ( W ∗ Σ , V ∗ Σ ) constructed in Appendix A.12, where Σ is a random diagonal sign matrix.
Revised: recall the ideal random matrices ( W ∗ Σ , V ∗ Σ ) constructed in Appendix A.12, where Σ is a random diagonal sign matrix.

Original: Speciﬁcally, W ∗ Σ is deﬁned in Equation (108), and V ∗ Σ is the projection of the rows of matrix V ∗ onto the orthogonal subspace spanned by ( ϕ (0) ( x i ))
Revised: Speciﬁcally, W ∗ Σ is deﬁned in Equation (107), and V ∗ Σ is the projection of the rows of matrix V ∗ onto the orthogonal subspace spanned by ( ϕ (0) ( x i ))

Original: ni =1 .
Revised: ni =1 .

Original: Using the parameter setting for i = 1 ,
Revised: Using the parameter setting for i = 1 ,

Original: [Equation]
Revised: [Equation]

Original: with respect to an arbitrary parameter ν > 0 , then for every pair ( W ′ , V ′ )
Revised: with respect to an arbitrary parameter ν > 0 , then for every pair ( W ′ , V ′ )

Original: such that ∥ W ′ ∥ ≤
Revised: such that ∥ W ′ ∥ ≤

Original: C 1 , ∥ V ′ ∥
Revised: C 1 , ∥ V ′ ∥

Original: ≤ C 2 and
Revised: ≤ C 2 and

Original: [Equation]
Revised: [Equation]

Original: for parameters m 1 , m 2 , m 3 , 1 /κ 1 , 1 /κ 2 polynomially large enough in B, 1 /λ 0 , n, C 1 , C 2 and small enough step size η , we have
Revised: for parameters m 1 , m 2 , m 3 , 1 /κ 1 , 1 /κ 2 polynomially large enough in B, 1 /λ 0 , n, C 1 , C 2 and small enough step size η , we have

Original: [Equation]
Revised: [Equation]

Original: In order to prove the above theorem, we ﬁrst state and prove the following lemma which is the core of Theorem 6.
Revised: In order to prove the above theorem, we ﬁrst state and prove the following lemma which is the core of Theorem 6.

Original: Lemma 11 For matrices ( W ∗ , V ∗ ) constructed in Appendix A.12, speciﬁcally for their random coupling ( W ∗ Σ , V ∗ Σ ) as denoted above, we have:
Revised: Lemma 11 For matrices ( W ∗ , V ∗ ) constructed in Appendix A.12, speciﬁcally for their random coupling ( W ∗ Σ , V ∗ Σ ) as denoted above, we have:

Original: [Equation]
Revised: [Equation]

Original: where ℘ goes to zero with polynomially large overparameterization (the exact dependence is revealed via the proof).
Revised: where ℘ goes to zero with polynomially large overparameterization (the exact dependence is revealed via the proof).

Original: Proof of Lemma
Revised: Proof of Lemma

Original: For brevity, we use the notation D ′ ,ρ here to refer to the diagonal binary sign matrix when the input is multiplied by the sum of weight and smoothing matrices.
Revised: For brevity, we use the notation D ′ ,ρ here to refer to the diagonal binary sign matrix when the input is multiplied by the sum of weight and smoothing matrices.

Original: It will be clear in the context of the equation that what the “input” and the “weight” matrices are.
Revised: It will be clear in the context of the equation that what the “input” and the “weight” matrices are.

Original: This notation is also deﬁned and used in Lemma 28).
Revised: This notation is also deﬁned and used in Lemma 28).

Original: Here, we bound multiple cross terms that are created as a result of moving in the random direction.
Revised: Here, we bound multiple cross terms that are created as a result of moving in the random direction.

Original: To simplify the presentation and avoid confusing recursions in the proof, we have made a sublemma for each of these cross terms and has deferred its proof to Appendix A.14.
Revised: To simplify the presentation and avoid confusing recursions in the proof, we have made a sublemma for each of these cross terms and has deferred its proof to Appendix A.14.

Original: We use difference sub-indices of the symbol ℜ to illustrate terms that go to zero by growing the overparameterization in our architecture.
Revised: We use difference sub-indices of the symbol ℜ to illustrate terms that go to zero by growing the overparameterization in our architecture.

Original: We start by using Lemma 28,
Revised: We start by using Lemma 28,

Original: [Equation]
Revised: [Equation]

Original: Now using the notation introduced in Lemma
Revised: Now using the notation introduced in Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma
Revised: By Lemma

Original: [Equation]
Revised: [Equation]

Original: we have the following bound for ϕ (2) ′
Revised: we have the following bound for ϕ (2) ′

Original: ( x i ) :
Revised: ( x i ) :

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Combining this with Lemma
Revised: Combining this with Lemma

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i [Equation] Moreover, using the notation ϕ ∗′ ( x i ) introduced in Lemma 15 and the bound in Lemma 17, we can rewrite the second term as:
Revised: Moreover, using the notation ϕ ∗′ ( x i ) introduced in Lemma 15 and the bound in Lemma 17, we can rewrite the second term as:

Original: [Equation]
Revised: [Equation]

Original: Now we write the gradient-lipshitz inequality for ℓ at point
Revised: Now we write the gradient-lipshitz inequality for ℓ at point

Original: [Equation]
Revised: [Equation]

Original: and regarding the following vector, where ℘ 1 is the sum of all the noise terms above and goes to zero by over parameterization:
Revised: and regarding the following vector, where ℘ 1 is the sum of all the noise terms above and goes to zero by over parameterization:

Original: [Equation]
Revised: [Equation]

Original: Hence, using the 1 smoothness of ℓ ( ., y i ) :
Revised: Hence, using the 1 smoothness of ℓ ( ., y i ) :

Original: [Equation]
Revised: [Equation]

Original: But note that
Revised: But note that

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: using the notation of Lemma 15 and the result of Lemma 16:
Revised: using the notation of Lemma 15 and the result of Lemma 16:

Original: [Equation]
Revised: [Equation]

Original: Moreover, using again the result on ϕ (2) ′
Revised: Moreover, using again the result on ϕ (2) ′

Original: ( x i ) from Lemma 18 and the fact that ϕ (0) ( x i ) is orthogonal to the rows of V ∗ Σ :
Revised: ( x i ) from Lemma 18 and the fact that ϕ (0) ( x i ) is orthogonal to the rows of V ∗ Σ :

Original: [Equation]
Revised: [Equation]

Original: Combining the last Equation with Lemma 14:
Revised: Combining the last Equation with Lemma 14:

Original: [Equation]
Revised: [Equation]

Original: Combining Equations (147) and (148):
Revised: Combining Equations (146) and (147):

Original: [Equation]
Revised: [Equation]

Original: Combining Equations (142) and (149), plugging into (141), and reopening the deﬁnition of p (1)Σ :
Revised: Combining Equations (141) and (148), plugging into (140), and reopening the deﬁnition of p (1)Σ :

Original: [Equation]
Revised: [Equation]

Original: Now note that we can easily bound the magnitude of the term η E W ρ ,V ρ a T D ′ ,ρ V ∗ Σ ϕ ∗ ( x i ) as:
Revised: Now note that we can easily bound the magnitude of the term η E W ρ ,V ρ a T D ′ ,ρ V ∗ Σ ϕ ∗ ( x i ) as:

Original: [Equation]
Revised: [Equation]

Original: while using Lemma 34:
Revised: while using Lemma 34:

Original: [Equation]
Revised: [Equation]

Original: which is O ( C 1 C 2 ) for enough overparameterization and smoothing parameters β 1 , β 2 as deﬁned in A.20.1.
Revised: which is O ( C 1 C 2 ) for enough overparameterization and smoothing parameters β 1 , β 2 as deﬁned in A.20.1.

Original: Furthermore, from Equations (132) and (127), we easily see that
Revised: Furthermore, from Equations (131) and (126), we easily see that

Original: [Equation]
Revised: [Equation]

Original: Now taking η small enough so that the bound η √ 2 ζ 2 (1 + ℜ ) pP k ∥V k ∥ 2 H ∞ and η℘ 1 both also be bounded of order O ( C 1 C 2 ) , we observe that the term inside the argument of ℓ ( ., y i ) Equation in (150) is O ( C 1 C 2 ) .
Revised: Now taking η small enough so that the bound η √ 2 ζ 2 (1 + ℜ ) pP k ∥V k ∥ 2 H ∞ and η℘ 1 both also be bounded of order O ( C 1 C 2 ) , we observe that the term inside the argument of ℓ ( ., y i ) Equation in (149) is O ( C 1 C 2 ) .

Original: Hence, we can use the lipschitz parameter of ℓ in the interval [ − O
Revised: Hence, we can use the lipschitz parameter of ℓ in the interval [ − O

Original: ( C 1 C 2 ) , O ( C 1 C 2 )] , given by Lemma 9 to take out the noise term:
Revised: ( C 1 C 2 ) , O ( C 1 C 2 )] , given by Lemma 9 to take out the noise term:

Original: [Equation]
Revised: [Equation]

Original: Now by applying Lemma 19 and writing the Lipchitz property of ℓ at point (1 − η ) f ′ W ′ ,V ′ ( x i ) =
Revised: Now by applying Lemma 19 and writing the Lipchitz property of ℓ at point (1 − η ) f ′ W ′ ,V ′ ( x i ) =

Original: O ( C 1 C 2 ) :
Revised: O ( C 1 C 2 ) :

Original: [Equation]
Revised: [Equation]

Original: where the last line is just deﬁnition.
Revised: where the last line is just deﬁnition.

Original: Now Convexity of ℓ ﬁnishes the proof.
Revised: Now Convexity of ℓ ﬁnishes the proof.

Original: Next, using Lemma 11 we prove Theorem 6.
Revised: Next, using Lemma 11 we prove Theorem 6.

Original: Restating Theorem 

Revised: Restating Theorem

Original: In the same setting as Theorem 6 and having enough overparameterization such that ℘ ≤ ν 8 ( ℘ deﬁned in Lemma 11) and polynomially small enough step size η , we have
Revised: In the same setting as Theorem 6 and having enough overparameterization such that ℘ ≤ ν 8 ( ℘ deﬁned in Lemma 11) and polynomially small enough step size η , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem
Revised: Proof of Theorem

Original: First, note that taking expectation w.r.t Σ :
Revised: First, note that taking expectation w.r.t Σ :

Original: [Equation]
Revised: [Equation]

Original: which by orthogonality of W ∗ k ’s:
Revised: which by orthogonality of W ∗ k ’s:

Original: [Equation]
Revised: [Equation]

Original: Similarly for V ′ :
Revised: Similarly for V ′ :

Original: [Equation]
Revised: [Equation]

Original: Now using Lemma 11:
Revised: Now using Lemma 11:

Original: [Equation]
Revised: [Equation]

Original: which by the choice of ψ i ’s is equal to
Revised: which by the choice of ψ i ’s is equal to

Original: [Equation]
Revised: [Equation]

Original: Moreover, using the condition ℘ ≤ 
Revised: Moreover, using the condition ℘ ≤ ν/

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and picking η as small as
Revised: and picking η as small as

Original: [Equation]
Revised: [Equation]

Original: we ﬁnally get
Revised: we ﬁnally get

Original: [Equation]
Revised: [Equation]

Original: A. E XISTENCE OF A GOOD DIRECTION H ELPER L EMMAS
Revised: A. E XISTENCE OF A GOOD DIRECTION H ELPER L EMMAS

Original: In this section, we state and prove the core lemmas that are used in the proof of Lemma 11.
Revised: In this section, we state and prove the core lemmas that are used in the proof of Lemma 11.

Original: Notably, through all of this section, we assume the norm bounds ∥ W ′ ∥
Revised: Notably, through all of this section, we assume the norm bounds ∥ W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 and that as our usual assumption, the rows of V ′ are orthogonal to ϕ (0) ( x i ) ’s
Revised: ≤ C 2 and that as our usual assumption, the rows of V ′ are orthogonal to ϕ (0) ( x i ) ’s

Original: for all i ∈ [ n ] .
Revised: for all i ∈ [ n ] .

Original: A notation that we use throughout the proofs is V ∗ Σ which refers to the projectiono of V ∗ Σ onto the orthogonal subspace to ( ϕ (0) ( x i ))
Revised: A notation that we use throughout the proofs is V ∗ Σ which refers to the projectiono of V ∗ Σ onto the orthogonal subspace to ( ϕ (0) ( x i ))

Original: ni =1 .
Revised: ni =1 .

Original: Lemma 12 Let P ( . ) be the projection operator onto the subspace spanned by ( ϕ (0) ( x i ))
Revised: Lemma 12 Let P ( . ) be the projection operator onto the subspace spanned by ( ϕ (0) ( x i ))

Original: ni =1 .
Revised: ni =1 .

Original: Also, we denote the projection of rows of V ∗ Σ onto the orthogonal subspace to ( ϕ (0) ( x i ))
Revised: Also, we denote the projection of rows of V ∗ Σ onto the orthogonal subspace to ( ϕ (0) ( x i ))

Original: ni =1 by V ∗ Σ j .
Revised: ni =1 by V ∗ Σ j .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: with high probability
Revised: with high probability

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: By Equation (129), we have ∥
Revised: By Equation (128), we have ∥

Original: V ∗ j ∥ ∞ ≤ ϱ 3 ξ/ √ m 2 .
Revised: V ∗ j ∥ ∞ ≤ ϱ 3 ξ/ √ m 2 .

Original: Now suppose that u 1 , ..., u n are an orthonormal basis for the subspace span ( ϕ (0) ( x i ))
Revised: Now suppose that u 1 , ..., u n are an orthonormal basis for the subspace span ( ϕ (0) ( x i ))

Original: ni =1 .
Revised: ni =1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Also, by Hoeffding, with high probability:
Revised: Also, by Hoeffding, with high probability:

Original: [Equation]
Revised: [Equation]

Original: which implies the second part.
Revised: which implies the second part.

Original: Lemma 

Revised: Lemma

Original: The ﬁrst cross term goes away because of the deﬁnition of V ∗ Σ .
Revised: The ﬁrst cross term goes away because of the deﬁnition of V ∗ Σ .

Original: (inside the expectations is zero almost surely)
Revised: (inside the expectations is zero almost surely)

Original: [Equation]
Revised: [Equation]

Original: Lemma 14 Second cross term:
Revised: Lemma 14 Second cross term:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: This time we use Equation (129) in Lemma 10 and Lemma 12:
Revised: This time we use Equation (128) in Lemma 10 and Lemma 12:

Original: [Equation]
Revised: [Equation]

Original: Now according to Lemma
Revised: Now according to Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: which completes the proof
Revised: which completes the proof

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: We get an additional term ϕ ∗′ ( x i ) as a result of smoothing which we deﬁne as
Revised: We get an additional term ϕ ∗′ ( x i ) as a result of smoothing which we deﬁne as

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have the following inequality almost surely (over the randomness of W ρ ):
Revised: Moreover, we have the following inequality almost surely (over the randomness of W ρ ):

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: According to Lemma 1, for j / ∈ P , for every i ∈ [ n ] we have
Revised: According to Lemma 1, for j / ∈ P , for every i ∈ [ n ] we have

Original: [Equation]
Revised: [Equation]

Original: Now note that as long as the sign patterns for j / ∈ P does not change, ϕ ∗′ ( x i ) will be zero.
Revised: Now note that as long as the sign patterns for j / ∈ P does not change, ϕ ∗′ ( x i ) will be zero.

Original: Therefore by union bound
Revised: Therefore by union bound

Original: [Equation]
Revised: [Equation]

Original: But ( W ρj ) x i is Gaussian with variance β 21 /m 1 .
Revised: But ( W ρj ) x i is Gaussian with variance β 21 /m 1 .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: which proves the ﬁrst part.
Revised: which proves the ﬁrst part.

Original: For the second part, according to Equation (106) in Lemma 5, for every k ∈ [ m 3 ] :
Revised: For the second part, according to Equation (105) in Lemma 5, for every k ∈ [ m 3 ] :

Original: [Equation]
Revised: [Equation]

Original: which implies the second part.
Revised: which implies the second part.

Original: Lemma 16 Fourth Extra term:
Revised: Lemma 16 Fourth Extra term:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Note that with high probability over the randomness of V (0)
Revised: Note that with high probability over the randomness of V (0)

Original: [Equation]
Revised: [Equation]

Original: we have ∥ V (0) ∥
Revised: we have ∥ V (0) ∥

Original: F ≲ √ m 2 m 3 κ 2 .
Revised: F ≲ √ m 2 m 3 κ 2 .

Original: Now according to Lemma 15 and using the fact that ∥ V ′ ∥
Revised: Now according to Lemma 15 and using the fact that ∥ V ′ ∥

Original: F ≤ C 2 :
Revised: F ≤ C 2 :

Original: [Equation]
Revised: [Equation]

Original: Lemma 17 Fifth extra term:
Revised: Lemma 17 Fifth extra term:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Similar to the previous Lemma, the inner expectation can be bounded as:
Revised: Similar to the previous Lemma [Equation] the inner expectation can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: We have another extra term as a product of the movement − η/ 2 W ′ in the ﬁrst layer: ϕ (2) ′ ( x i
Revised: We have another extra term as a product of the movement − η/ 2 W ′ in the ﬁrst layer: ϕ [Equation]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: First we prove the following approximation argument (for all k ∈ [ m 3 ] ):
Revised: First we prove the following approximation argument (for all k ∈ [ m 3 ] ):

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: By the independence of W ρj ’s, the ﬁrst term can be upper bounded as
Revised: By the independence of W ρj ’s [Equation] the ﬁrst term can be upper bounded as

Original: [Equation]
Revised: [Equation]

Original: For the second term, note that for every j / ∈ P , the j th entries of D W (0) + W ρ ,x i and
Revised: For the second term, note that for every j / ∈ P , the j th entries of D W (0) + W ρ ,x i and

Original: D W (0) + W ρ + W ′ ,x i are different only if W ρj can make a sign change in the j th row, i.e. | ( W (0) j
Revised: D W (0) + W ρ + W ′ ,x i are different only if W ρj can make a sign change in the j th row, i.e. | ( W (0) j

Original: +
Revised: [Equation]

Original: W ′ j ) x i | ≤ | W ρj x i | should happen.
Revised: W ′ j ) x i | ≤ | W ρj x i | should happen.

Original: We denote this event for every j / ∈ P by ˜ E j .
Revised: We denote this event for every j / ∈ P by ˜ E j .

Original: Furthermore, if this happens for some j , then the value of ( W (0) + W ρ )
Revised: Furthermore, if this happens for some j , then the value of ( W (0) + W ρ )

Original: j x i is upper bounded by | W ′
Revised: j x i is upper bounded by | W ′

Original: j x i | .
Revised: j x i | .

Original: Now similar to our discussion in Lemma 15 and using the result of Lemma 1:
Revised: Now similar to our discussion in Lemma 15 and using the result of Lemma 1:

Original: [Equation]
Revised: [Equation]

Original: But note that ( W ρj )
Revised: But note that ( W ρj )

Original: x
Revised: x

Original: i is Gaussian with variance β 21 /m 1 .
Revised: i is Gaussian with variance β 21 /m 1 .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: So ﬁnally we can write
Revised: So ﬁnally we can write

Original: [Equation]
Revised: [Equation]

Original: which completes the proof for Equation (159).
Revised: which completes the proof for Equation (158).

Original: This immediately implies
Revised: This immediately implies

Original: E W
Revised: E W

Original: [Equation]
Revised: [Equation]

Revised: ϕ

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Now we ﬁrst prove Equation (157):
Revised: Now we ﬁrst prove Equation (156):

Original: [Equation]
Revised: [Equation]

Original: To prove Equation (158):
Revised: To prove Equation (157):

Original: [Equation]
Revised: [Equation]

Original: Lemma 19 Closeness condition:
Revised: Lemma 19 Closeness condition:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Note that by Corollary 5.1 and according to the proof of Equation 130 in Lemma 10, if for every j / ∈ ˜ P we don’t have a sign change in D V (0)
Revised: Note that by Corollary 5.1 and according to the proof of Equation 129 in Lemma 10, if for every j / ∈ ˜ P we don’t have a sign change in D V (0)

Original: + V ρ + V ′ ,x i V ∗ ϕ ∗ ( x i ) , then get
Revised: + V ρ + V ′ ,x i V ∗ ϕ ∗ ( x i ) , then get

Original: [Equation]
Revised: [Equation]

Original: Also, note that we need the event E c (deﬁned in Lemma 33) to happen in order to be able to use
Revised: Also, note that we need the event E c (deﬁned in Lemma 33) to happen in order to be able to use

Original: Corrolary 5.1.
Revised: Corrolary 5.1.

Original: Hence, given a W ρ for which E c happens, we upper bound the probability of sign change with respect to the randomness of V ρ .
Revised: Hence, given a W ρ for which E c happens, we upper bound the probability of sign change with respect to the randomness of V ρ .

Original: We deﬁne the following event with respect to the randomness of V ρ when conditioned on a W ρ for which E c happens ( P i ’s are deﬁned in Lemma 7):
Revised: We deﬁne the following event with respect to the randomness of V ρ when conditioned on a W ρ for which E c happens ( P i ’s are deﬁned in Lemma 7):

Original: [Equation]
Revised: [Equation]

Original: Now from the result in Corollary 5.1 we have ≤ 1 { sign change in j / ∈ P i } ≤ 1 { SC } .
Revised: Now from the result in Corollary 5.1 we have ≤ 1 { sign change in j / ∈ P i } ≤ 1 { SC } .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: But note that ( V ρ j ) x ′ i is Gaussian with variance β 2 2 ∥ x ′ i ∥ 2 /m 2 .
Revised: But note that ( V ρ j ) x ′ i is Gaussian with variance β 2 2 ∥ x ′ i ∥ 2 /m 2 .

Original: Hence
Revised: Hence

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now let D be a sign matrix random variable such that if E c and SC c both happens, then it is equal to the valid sign matrix D V (0)
Revised: Now let D be a sign matrix random variable such that if E c and SC c both happens, then it is equal to the valid sign matrix D V (0)

Original: + V ρ + V ′ ,x i ,
Revised: + V ρ + V ′ ,x i ,

Original: and otherwise it is equal to an arbitrary valid sign matrix in the case when both E c and SC c happen.
Revised: and otherwise it is equal to an arbitrary valid sign matrix in the case when both E c and SC c happen.

Original: Now using Equation (117) we have with high probabilityover the
Revised: Now using Equation (116) we have with high probability

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now note that for any sign matrix D ′ , we have the following bound:
Revised: Now note that for any sign matrix D ′ , we have the following bound:

Original: [Equation]
Revised: [Equation]

Original: Also, applying a union bound and using Lemmas
Revised: Also, applying a union bound and using Lemmas

Original: [Equation]
Revised: [Equation]

Original: Hence, also applying Lemma
Revised: Hence, also applying Lemma

Original: [Equation]
Revised: [Equation]

Original: we further write
Revised: we further write

Original: [Equation]
Revised: [Equation]

Original: Lemma 20 Suppose we have m 3 κ 21 ≥ C 21 .
Revised: Lemma 20 Suppose we have m 3 κ 21 ≥ C 21 .

Original: Then, for the following basic term we have:
Revised: Then, for the following basic term we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]

Original: ℜ

Original: [Equation]
Revised: [Equation]

Original: and ℜ 6 is deﬁned in Lemma 22.
Revised: and ℜ 6 is deﬁned in Lemma 22.

Original: Proof of Lemma
Revised: Proof of Lemma

Original: First, note that by orthogonality of ϕ (0) ( x i ) to the rows of V ′ :
Revised: First, note that by orthogonality of ϕ (0) ( x i ) to the rows of V ′ :

Original: [Equation]
Revised: [Equation]

Original: But note that for the second term:
Revised: But note that for the second term:

Original: [Equation]
Revised: [Equation]

Original: Now conditioned on x ′ i , by the result of Lemma 31 we know there exists a set of indices
Revised: Now conditioned on x ′ i , by the result of Lemma 31 we know there exists a set of indices

Original: O ⊆
Revised: O ⊆

Original: [ m 2 ] , s.t.
Revised: [ m 2 ] , s.t.

Original: | O | ≤
Revised: | O | ≤

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: 3( √ m 2 κ 2 ) 2 / 3 m 2 and for j / ∈
Revised: 3( √ m 2 κ 2 ) 2 / 3 m 2 and for j / ∈

Original: O we have
Revised: O we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Now for j ∈ [ m 2 ] , deﬁne the event
Revised: Now for j ∈ [ m 2 ] , deﬁne the event

Original: [Equation]
Revised: [Equation]

Original: and R = ∪ j R j .
Revised: and R = ∪ j R j .

Original: First, note that using Gaussian tail bound, R is a rare event:
Revised: First, note that using Gaussian tail bound, R is a rare event:

Original: [Equation]
Revised: [Equation]

Original: Now for j / ∈ O and under R c , clearly we have that the signs of ( V (0) j + V ρj ) x ′ i
Revised: Now for j / ∈ O and under R c , clearly we have that the signs of ( V (0) j + V ρj ) x ′ i

Original: and ( V (0)
Revised: and ( V (0)

Original: j + V ρj + V ′ j ) x
Revised: j + V ρj + V ′ j ) x

Original: ′ i are the same.
Revised: ′ i are the same.

Original: Therefore, applying Lemma 33, we can argue under R c :
Revised: Therefore, applying Lemma 33, we can argue under R c :

Revised: [Equation]

Revised: ,V

Original: [Equation]
Revised: [Equation]

Original: Hence, overall, using Cauchy-Shwartz
Revised: Hence, overall, using Cauchy-Shwartz

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: using Lemma
Revised: using Lemma

Original: [Equation]
Revised: [Equation]

Original: we have with high probability over the randomness of initialization
Revised: we have with high probability over the randomness of initialization

Original: [Equation]
Revised: [Equation]

Original: Hence:
Revised: Hence:

Original: [Equation]
Revised: [Equation]

Original: Combining Equations (165) and (166) into Equation (164):
Revised: Combining Equations (164) and (165) into Equation (163):

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: for the ﬁrst term in (163)
Revised: for the ﬁrst term in (162)

Original: [Equation]
Revised: [Equation]

Original: using Equation (166) and Lemmas 33 and Lemma 30 we have
Revised: using Equation (165) and Lemmas 33 and Lemma 30 we have

Original: [Equation]
Revised: [Equation]

Original: )( C 1 + √ m 3 β 1 ) .
Revised: ( C 1 + √ m 3 β 1 ) .

Original: Substituting Equations (167) and (168) into Equation (163), we ﬁnally get
Revised: Substituting Equations (166) and (167) into Equation (162), we ﬁnally get

Original: [Equation]
Revised: [Equation]

Original: Now by picking η small enough so that the second term is dominated by the third term we get:
Revised: Now by picking η small enough so that the second term is dominated by the third term we get:

Original: [Equation]
Revised: [Equation]

Original: Now we aim to bound the term E W ρ ,V ρ [ 1 √ m 2 a T D V (0)
Revised: Now we aim to bound the term E W ρ ,V ρ [ 1 √ m 2 a T D V (0)

Original: + V ρ + V ′ ,x i ( V (0)
Revised: + V ρ + V ′ ,x i ( V (0)

Original: + V ρ ) ϕ
Revised: + V ρ ) ϕ

Original: (0) ( x i ) .
Revised: (0) ( x i ) .

Original: First assume that we are in the event E c deﬁned in Lemma 33, i.e. we have ∥ ϕ (2) ( x i ) ∥ ≲ C 1 .
Revised: First assume that we are in the event E c deﬁned in Lemma 33, i.e. we have ∥ ϕ (2) ( x i ) ∥ ≲ C 1 .

Original: Conditioned on such W ρ , we now work with the randomness of the initialization and V ρ .
Revised: Conditioned on such W ρ , we now work with the randomness of the initialization and V ρ .

Original: Note that the random matrix V (0) + V ρ jointly over the randomness of V ρ and the initialization is also Gaussian, and its variance is
Revised: Note that the random matrix V (0) + V ρ jointly over the randomness of V ρ and the initialization is also Gaussian, and its variance is

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from the fact that κ 2 ≥ 1 √ m 2 and β 2 ≤ 1 .
Revised: where the inequality follows from the fact that κ 2 ≥ 1 √ m 2 and β 2 ≤ 1 .

Original: Therefore, applying
Revised: Therefore, applying

Original: Lemma 22 for the random matrix V (0) in the Lemma as V (0)
Revised: Lemma 22 for the random matrix V (0) in the Lemma as V (0)

Original: + V ρ here, the bound does not change up to constants because of the inequality (173).
Revised: + V ρ here, the bound does not change up to constants because of the inequality (172).

Original: Hence, with high probability, lets say with prob.
Revised: Hence, with high probability, lets say with prob.

Original: 1 − δ 1 this time over both the randomness of initialization and V ρ :
Revised: 1 − δ 1 this time over both the randomness of initialization and V ρ :

Original: [Equation]
Revised: [Equation]

Original: This means that with probability at least 1 − √ δ 1 over the random initialization, then we have (174) with prob.
Revised: This means that with probability at least 1 − √ δ 1 over the random initialization, then we have (173) with prob.

Original: at least 1 −√ δ 1 over the randomenss of V ρ .
Revised: at least 1 −√ δ 1 over the randomenss of V ρ .

Original: We name the latter high probability statement as ( ⋆ ).
Revised: We name the latter high probability statement as ( ⋆ ).

Original: Moreover, note that by Lemma 32 and assuming m 3 log( m 2 ) < m 2 , we have the following [Table] [Table] [Table]
Revised: Moreover, note that by Lemma 32 and assuming m 3 log( m 2 ) < m 2 , we have the following almost surely bound (also note that V ρj

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: (0) ( x i ) is Gaussian with std β

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Furthermore, note because each variable | V ρj ϕ (0) ( x i ) | is β 2 √ m 2 ∥ ϕ (0) ( x i )
Revised: Furthermore, note because each variable | V ρj ϕ (0) ( x i ) | is β 2 √ m 2 ∥ ϕ (0) ( x i )

Original: ∥
Revised: ∥

Original: subGaussian.
Revised: subGaussian.

Original: Therefore,
Revised: Therefore,

Original: L is subGaussian with parameter ∥ ϕ
Revised: L is subGaussian with parameter ∥ ϕ

Original: (0) ( x i ) ∥
Revised: (0) ( x i ) ∥

Original: β 2 with respect to the randomness of V ρ .
Revised: β 2 with respect to the randomness of V ρ .

Original: Now the point is that the high probability argument in ( ⋆ ) is much stronger than what one can get from the subGaussian ineqaulity with parameter ∥ ϕ (0) ( x i ) ∥
Revised: Now the point is that the high probability argument in ( ⋆ ) is much stronger than what one can get from the subGaussian ineqaulity with parameter ∥ ϕ (0) ( x i ) ∥

Original: β 2 (with the corresponding expectation term ∥ ϕ (0) ( x i ) ∥
Revised: β 2 (with the corresponding expectation term ∥ ϕ (0) ( x i ) ∥

Original: ( √ m 2 + β 2 ) ).
Revised: ( √ m 2 + β 2 ) ).

Original: However, the disadvantage of ( ⋆ ) is that it only works for a ﬁxed δ 1 .
Revised: However, the disadvantage of ( ⋆ ) is that it only works for a ﬁxed δ 1 .

Original: In other words, at least it is not obvious from this argument that why for a ﬁxed W (0) in a high probaiblity region of the random initialization, whether we can send δ 1 to zero by growing the constant behind ℜ 6 with logarithmic rate log(1 /δ ) .
Revised: In other words, at least it is not obvious from this argument that why for a ﬁxed W (0) in a high probaiblity region of the random initialization, whether we can send δ 1 to zero by growing the constant behind ℜ 6 with logarithmic rate log(1 /δ ) .

Original: This makes our job hard for bounding the expectation with respect to V ρ if we only wish to rely on ( ⋆ ).
Revised: This makes our job hard for bounding the expectation with respect to V ρ if we only wish to rely on ( ⋆ ).

Original: Therefore, we combine it with the inequality that we get from the subGaussian parameter that we introudced above.
Revised: Therefore, we combine it with the inequality that we get from the subGaussian parameter that we introudced above.

Original: More rigorously, we deﬁne the thresholding parameter
Revised: More rigorously, we deﬁne the thresholding parameter

Original: [Equation]
Revised: [Equation]

Original: for which we have
Revised: for which we have

Original: [Equation]
Revised: [Equation]

Original: we divide the range of values for L into three parts:
Revised: we divide the range of values for L into three parts:

Original: [Equation]
Revised: [Equation]

Original: Now by choosing δ 1 ≲ 1 / ℧ , we conclude with high probability over initialization and conditioned on W ρ ’s such that E c happens we have
Revised: Now by choosing δ 1 ≲ 1 / ℧ , we conclude with high probability over initialization and conditioned on W ρ ’s such that E c happens we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we integrate also with respect to W ρ .
Revised: Finally [Equation] we integrate also with respect to W ρ .

Original: To control the random variable when E happens, we use the bound in (179) and the fact that E is a rare event due to Lemma 33:
Revised: To control the random variable when E happens, we use the bound in (178) and the fact that E is a rare event due to Lemma 33:

Original: [Equation]
Revised: [Equation]

Original: Substituting this into (172) the proof is ﬁnally complete.
Revised: Substituting this into (171) the proof is ﬁnally complete.

Original: Lemma 21 Third cross term: with high probability over initialization, we have
Revised: Lemma 21 Third cross term: with high probability over initialization, we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Note that the way we deﬁned the matrix W ∗ and as a result ϕ ∗ ( x i ) only depends on the randomness of W (0) , not on W ′ or the randomness of V (0) .
Revised: Note that the way we deﬁned the matrix W ∗ and as a result ϕ ∗ ( x i ) only depends on the randomness of W (0) , not on W ′ or the randomness of V (0) .

Original: Now using Equation (131) and Jensen inequality we can write (for vector v , the notation v 2 ⊙ is another vector with each entry as the second power of the corresponding entry in v ):
Revised: Now using Equation (130) and Jensen inequality we can write (for vector v , the notation v 2 ⊙ is another vector with each entry as the second power of the corresponding entry in v ):

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now under the event E c deﬁned in Lemma 33 we get that ∥ ϕ (2) ( x i ) ∥ ≲ C 1 , so we can bound the above as 
Revised: Now under the event E c deﬁned in Lemma 33 we get that ∥ ϕ (2) ( x i ) ∥ ≲ C 1 , so we can bound the above as [Equation] +

Original: [Equation]
Revised: [Equation]

Original: Now deﬁning
Revised: Now deﬁning

Original: [Equation]
Revised: [Equation]

Original: to bound the ﬁrst term, we want to apply Lemma 23 using the same trick that we did in the proof of Lemma 20.
Revised: to bound the ﬁrst term, we want to apply Lemma 23 using the same trick that we did in the proof of Lemma 20.

Original: Note that L 2 is the same term as Γ 2 x ′ ,V ′ in Lemma 23 except that it is deﬁned with respect to V (0)
Revised: Note that L 2 is the same term as Γ 2 x ′ ,V ′ in Lemma 23 except that it is deﬁned with respect to V (0)

Original: + V ρ instead of V (0) .
Revised: + V ρ instead of V (0) .

Original: On the other hand, note that V (0)
Revised: On the other hand, note that V (0)

Original: + V ρ has Gaussian entries with variance κ 2 2 + β 22 m 

Revised: + V ρ has Gaussian entries with variance κ 2 2 + β 22 m

Original: and we know κ 2 2 ≤ κ 2 2 + β 22 m 2 ≤ 2 κ 2 2 , which means the argument of Lemmaholds true here up to constants:
Revised: and we know κ 2 2 ≤ κ 2 2 + β 22 m 2 ≤ 2 κ 2 2 , which means the argument of Lemmaholds true here up to constants:

Original: [Equation]
Revised: [Equation]

Original: This holds with probability say 1 − δ 2 over the randomness of both V (0) and V ρ .
Revised: This holds with probability say 1 − δ 2 over the randomness of both V (0) and V ρ .

Original: Therefore, with probability 1 −√ δ 2 over the initialization, then with probability at least 1 −√ δ 2 over the randomness of V ρ we have the above.
Revised: Therefore, with probability 1 −√ δ 2 over the initialization, then with probability at least 1 −√ δ 2 over the randomness of V ρ we have the above.

Original: Moreover, with a simple Cauchy-Swuartz we get the following almost surely bound:
Revised: Moreover, with a simple Cauchy-Swuartz we get the following almost surely bound:

Original: [Equation]
Revised: [Equation]

Original: Now the variable ∥ V ρ ∥ 2 is subexponential with parameter ( β 41 m 23 , β 21 m 3 ) .
Revised: Now the variable ∥ V ρ ∥ 2 is subexponential with parameter ( β 41 m 23 , β 21 m 3 ) .

Original: Furthermore, with high probability we have ∥ V (0) ∥ 2 F ≲ m 2 m 3 κ 22 .
Revised: Furthermore, with high probability we have ∥ V (0) ∥ 2 F ≲ m 2 m 3 κ 22 .

Original: Therefore, taking
Revised: Therefore, taking

Original: [Equation]
Revised: [Equation]

Original: then one can easily see by the subexponential tail:
Revised: then one can easily see by the subexponential tail:

Original: [Equation]
Revised: [Equation]

Original: Hence, we can apply the same trick as Lemma 20 as
Revised: Hence, we can apply the same trick as Lemma 20 as

Original: [Equation]
Revised: [Equation]

Original: Now taking δ 2 ≲ ℜ 47 / ℧ 22 , we ﬁnally get that conditioned on W ρ ’s where E happens, then
Revised: Now taking δ 2 ≲ ℜ 47 / ℧ 22 , we ﬁnally get that conditioned on W ρ ’s where E happens, then

Original: [Equation]
Revised: [Equation]

Original: On the other hand, to handle the case when E happens, we can use the bound in (182) as it does not depend on the occurrence of E as well:
Revised: On the other hand, to handle the case when E happens, we can use the bound in (181) as it does not depend on the occurrence of E as well:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Plugging this back into (181) we ﬁnally get
Revised: Plugging this back into (180) we ﬁnally get

Revised: LHS

Original: [Equation]
Revised: [Equation]

Original: A. B OUNDING THE WORST - CASE S ENARIO
Revised: A. B OUNDING THE WORST - CASE S ENARIO

Original: Lemma 22 Suppose m 3 ≥ log( m 2 ) and √ m 3 κ 1 ≳ C 1 .
Revised: Lemma 22 Suppose m 3 ≥ log( m 2 ) and √ m 3 κ 1 ≳ C 1 .

Original: We deﬁne the sign matrices D x ′ V (0) + V ′ ,x i and D x ′ V (0) ,x i with respect to the multiplications
Revised: We deﬁne the sign matrices D x ′ V (0) + V ′ ,x i and D x ′ V (0) ,x i with respect to the multiplications

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: Then, with high probability:
Revised: Then, with high probability:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Consider a cover for the euclidean ball of radius C 1 in R m 3 with precision ϵ , 
Revised: Consider a cover for the euclidean ball of radius C 1 in R m 3 with precision ϵ , i.e.

Original: i
Revised: B C 1 ( ϵ ) . So for every x ′ ∈ R m

Original: [Equation]
Revised: [Equation]

Original: e. B C 1 ( ϵ ) . So for every x ′ ∈ R m 3 , there exists an x ∈ B C 1 ( ϵ ) such that ∥ x − x ′ ∥ ≤ ϵ , and | B C 1 ( ϵ ) | ≲ ( 1 ϵ ) m 3 .
Revised: there exists an x ∈ B C 1 ( ϵ ) such that ∥ x − x ′ ∥ ≤ ϵ , and | B C 1 ( ϵ ) | ≲ ( 1 ϵ ) m 3 .

Original: Now ﬁx x ′ and x .
Revised: Now ﬁx x ′ and x .

Original: We have Γx
Revised: We have Γ

Original: [Equation]
Revised: [Equation]

Original: = [Equation] Now by a union bound, because each variable V (0) j ϕ (0) ( x i ) is Gaussian with parameter κ 2 ∥ ϕ (0) ( x i ) ∥ and using Equation (116), with high probability we have for every j ∈ [ m 2 ] :
Revised: Now by a union bound, because each variable V (0) j ϕ (0) ( x i ) is Gaussian with parameter κ 2 ∥ ϕ (0) ( x i ) ∥ and using Equation (115), with high probability we have for every j ∈ [ m 2 ] :

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Hoeffding over the randomness of the Bernoulli variables a j , for a ﬁxed x ′ with high probability:
Revised: Therefore, by Hoeffding over the randomness of the Bernoulli variables a j , for a ﬁxed x ′ with high probability:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, We know that the VC-dimension of the class of binary functions with respect to halfspaces in R m 3 is m 3 + 1 .
Revised: On the other hand, We know that the VC-dimension of the class of binary functions with respect to halfspaces in R m 3 is m 3 + 1 .

Original: Therefore, the set of different sign patterns in matrices D x ′ V (0) ,x i is bounded by m m 3 +1, i [Equation] e. for
Revised: Therefore, the set of different sign patterns in matrices D x ′ V (0) ,x i is bounded by m m 3 +1, i.e. for

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, by taking a union bound over all sign matrices in D , we get with high probability
Revised: Therefore, by taking a union bound over all sign matrices in D , we get with high probability

Original: [Equation]
Revised: [Equation]

Original: Now for a threshold r which satisﬁes
Revised: Now for a threshold r which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: we deﬁne
Revised: we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Now by Equation (116)and the assumption of the Lemma √ m 3 κ 1 ≳ C 1 , we have
Revised: Now by Equation (115)and the assumption of the Lemma √ m 3 κ 1 ≳ C 1 , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence, V (0) j
Revised: Hence, V (0) j

Original: ( ϕ (0) ( x i ) + x ) is Gaussian with standard deviation at least Ω( κ 2 √ m 3 κ 1 ) .
Revised: ( ϕ (0) ( x i ) + x ) is Gaussian with standard deviation at least Ω( κ 2 √ m 3 κ 1 ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: On the other hand, note that |J x,r | is the sum of m 2 Bernoulli random variables, so it is subGaussian with parameter m 2 .
Revised: On the other hand, note that |J x,r | is the sum of m 2 Bernoulli random variables, so it is subGaussian with parameter m 2 .

Original: Therefore, with high probability
Revised: Therefore, with high probability

Original: [Equation]
Revised: [Equation]

Original: Now taking maximum over all x ∈ B C 1 ( ϵ ) and exploiting the subGaussian tail of the random variables, we get with high probability
Revised: Now taking maximum over all x ∈ B C 1 ( ϵ ) and exploiting the subGaussian tail of the random variables, we get with high probability

Original: [Equation]
Revised: [Equation]

Original: Moreover, consider a threshold 1 < θ , such that e − θ 2 / 8 ≤ m 2 /m 3 , and deﬁne the following set of indices
Revised: Moreover [Equation] consider a threshold 1 < θ , such that e − θ 2 / 8 ≤ m 2 /m 3 , and deﬁne the following set of indices

Original: [Equation]
Revised: [Equation]

Original: Then, using Lemma 29 and noting the fact that the standard deviation of Gaussians in V (0) is κ 2 and that ∥ ϕ (2) ( x i ) ∥
Revised: Then, using Lemma 29 and noting the fact that the standard deviation of Gaussians in V (0) is κ 2 and that ∥ ϕ (2) ( x i ) ∥

Original: ≤ C 1 , with high probability:
Revised: ≤ C 1 , with high probability:

Original: [Equation]
Revised: [Equation]

Original:  Now note that for each j ∈ [ m 2 ] , ∥ V (0)
Revised: [Equation] [Equation] Now note that for each j ∈ [ m 2 ] , ∥ V (0)

Original: j ∥ 2 is subexponential with parameters ( m 3 κ 42 , κ 22 ) , which means that with high probability:
Revised: j ∥ 2 is subexponential with parameters ( m 3 κ 42 , κ 22 ) , which means that with high probability:

Original: [Equation]
Revised: [Equation]

Original: But with condition m 3 ≥ log( m 2 ) , we can further upper bound it as
Revised: But with condition m 3 ≥ log( m 2 ) , we can further upper bound it as

Original: [Equation]
Revised: [Equation]

Original: Now for ﬁxed x, x ′ , for j ∈ J x,r we have
Revised: Now for ﬁxed x, x ′ , for j ∈ J x,r we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for j / ∈ J (2) x ′ ,θ :
Revised: On the other hand, for j / ∈ J (2) x ′ ,θ :

Original: [Equation]
Revised: [Equation]

Original: Therefore, for j ∈ J x,r − J (2) x ′ ,θ :
Revised: Therefore, for j ∈ J x,r − J (2) x ′ ,θ :

Original: [Equation]
Revised: [Equation]

Original: In a similar fashion, if j / ∈ J x,r , then using assumption (185):
Revised: In a similar fashion, if j / ∈ J x,r , then using assumption (184):

Original: [Equation]
Revised: [Equation]

Original: Hence, using the fact that ϕ (0) ( x i ) is orthogonal to V ′ j :
Revised: Hence, using the fact that ϕ (0) ( x i ) is orthogonal to V ′ j :

Original: [Equation]
Revised: [Equation]

Original: Now by triangle inequality and Equations (194), (192)
Revised: Now by triangle inequality and Equations (193) [Equation]

Original: [Equation]
Revised: [Equation]

Original: (191) and the fact that ∥ V ′ ∥
Revised: (190) and the fact that ∥ V ′ ∥

Original: F ≤ C
Revised: F ≤ C

Original: [Equation]
Revised: [Equation]

Original: we can write:
Revised: we can write:

Original: [Equation]
Revised: [Equation]

Original: ,θ )
Revised: θ )

Original: [Table]

Original: [Table]

Original: + V ′ j )( ϕ (

Original: [Table]
Revised: [Table]

Original: ) ( x i ) +x ′ ) ≥ 0 } − [Table] { V [Table]
Revised: +

Original: j ( ϕ

Original: [Table]
Revised: [Table]

Original: ) ( x

Original: [Table]
Revised: [Table]

Original: ) + x ′ ) ≥[Table] [Table] |
Revised: ≥

Original: [Table]
Revised: [Table]

Original: j ϕ

Original: [Table]

Original: ) | + 1 √ m
Revised: + 1 √ m

Original: X j ∈J (2) x ′ ,θ
Revised: X j ∈J (2) x ′ ,θ

Original: [Table]
Revised: [Table]

Original: 1 { ( V (

Original: [Table]
Revised: [Table]

Original: j + V′ j )( ϕ (0) ( x i ) + x ′ ) ≥ 0 } − 1 {
Revised: V

Original: [Table]
Revised: [Table]

Original: j 
Revised: j + V ′ j )

Original: [Table]
Revised: [Table]

Original: ϕ
Revised: ϕ

Original: [Table]
Revised: [Table]

Original: ) + x ′ [Table] j ϕ
Revised: −

Original: [Table]
Revised: [Table]

Original: | ≤ 1 √ m|J x,r − J (2) x ′ ,θ | max j ∈J x,r −J (2) x ′ ,θ | V (0) j ϕ (0) ( x i ) | + 1 √ m
Revised: ≤ 1 √ m|J x,r − J (2) x ′ ,θ | max j ∈J x,r −J (2) x ′ ,θ | V (0) j ϕ (0) ( x i ) | + 1 √ m

Original: X j/ ∈ ( J x,r ∪J (2) x ′ ,θ ) 1 {∥ V ′ j ∥ ≳
Revised: X j/ ∈ ( J x,r ∪J (2) x ′ ,θ ) 1 {∥ V ′ j ∥ ≳

Original: | V (0) j
Revised: | V (0) j

Original: ( ϕ (0) ( x i ) + x ′ )
Revised: ( ϕ (0) ( x i ) + x ′ )

Original: | 2 C+ r 4 C 1 }| V (0)
Revised: | 2 C+ r 4 C 1 }| V (0)

Original: j ϕ (0) ( x i ) | + 1 √ m| J (2) x ′ ,θ | max j ∈ m 2 | V (0) j ϕ (0) ( x i ) | ≤ 1 √ m|J x,r − J (2) x ′ ,θ | max j ∈J x,r −J (2) x ′ ,θ | V (0) j ϕ (0) ( x i ) | + 1 √ m
Revised: j ϕ ( [Table] ) ( x i ) | + 1 √ m| J (2) x ′ ,θ | max j ∈ m 2 | V (0) j ϕ (0) ( x i ) | ≤ 1 √ m|J x,r − J (2) x ′ ,θ | max j ∈J x,r −J (2) x ′ ,θ | V (0) j ϕ (0) ( x i ) | + 1 √ m

Original: X j/ ∈ ( J x,r ∪J (2) x ′ ,θ ) 1 {∥ V ′ j ∥ ≳
Revised: X j/ ∈ ( J x,r ∪J (2) x ′ ,θ ) 1 {∥ V ′ j ∥ ≳

Original: | V (0) j
Revised: | V (0) j

Original: ( ϕ (0) ( x i ) + x ′ )
Revised: ( ϕ (0) ( x i ) + x ′ )

Original: | 2 C+ r 4 C 1 } ( | V (0)
Revised: | 2 C+ r 4 C 1 } ( | V (0)

Original: j ( ϕ (0) ( x i ) + x ′ )
Revised: j ( ϕ (0) ( x i ) + x ′ )

Original: | + θκ 2 C 1 ) + 1 √ m| J (2) x ′ ,θ | max j ∈ m 2 | V (0)
Revised: | + θκ 2 C 1 ) + 1 √ m| J (2) x ′ ,θ | max j ∈ m 2 | V (0)

Original: j ϕ (0) ( x i ) | ≲ 1 √ m|J x,r − J (2) x ′ ,θ | max j ∈J x,r −J (2) x ′ ,θ | V (0)
Revised: j ϕ (0) ( x i ) | ≲ 1 √ m|J x,r − J (2) x ′ ,θ | max j ∈J x,r −J (2) x ′ ,θ | V (0)

Original: j ϕ (0) ( x i ) | + 1 √ m
Revised: j ϕ (0) ( x i ) | + 1 √ m

Original: X j/ ∈ ( J x,r ∪J (2) x ′ ,θ ) 1 {∥ V ′ j ∥ ≳
Revised: X j/ ∈ ( J x,r ∪J (2) x ′ ,θ ) 1 {∥ V ′ j ∥ ≳

Original: r C } ( C 1 ∥ V ′ j
Revised: r C } ( C 1 ∥ V ′ j

Original: ∥ + θκ 2 C 1 ) + 1 √ m| J (2) x ′ ,θ | max j ∈ m 2 | V (0)
Revised: ∥ + θκ 2 C 1 ) + 1 √ m| J (2) x ′ ,θ | max j ∈ m 2 | V (0)

Original: j ϕ (0) ( x i ) | ≲ 1 √ m|J x,r − J (2) x ′ ,θ | ( r + √ m 3 κ 2 ϵ + θκ 2 C 1 )
Revised: j ϕ (0) ( x i ) | ≲ 1 √ m|J x,r − J (2) x ′ ,θ | ( r + √ m 3 κ 2 ϵ + θκ 2 C 1 )

Original: + C 1 √
Revised: + C 1 √

Original: m r # (cid:16)
Revised: m r # (cid:16)

Original: j : ∥ V ′ j ∥ ≳
Revised: j : ∥ V ′ j ∥ ≳

Original: r C(cid:17)
Revised: r C(cid:17)

Original: ∥ V ′ ∥
Revised: ∥ V ′ ∥

Original: 2 F
Revised: 2 F

Original: [Equation]
Revised: [Equation]

Original: Now using Equations (208), (190), (183)
Revised: Now using Equations (207), (189), (182)

Original: [Equation]
Revised: [Equation]

Original: and (185)
Revised: and (184)

Original: [Equation]
Revised: [Equation]

Original: and the bound on |J (2) x ′ ,θ | from Lemma
Revised: and the bound on |J (2) x ′ ,θ | from Lemma

Original: [Equation]
Revised: [Equation]

Original: we write
Revised: we write

Original: [Equation]
Revised: [Equation]

Original: Now setting
Revised: Now setting

Original: [Equation]
Revised: [Equation]

Original: By this choice
Revised: By this choice

Original: [Equation]
Revised: [Equation]

Original: from (195) we obtain|
Revised: from (194) we obtain

Original: [Equation]
Revised: [Equation]

Original: Now we set
Revised: Now we set

Original: [Equation]
Revised: [Equation]

Original: which also satisﬁes the condition of Lemma 29 and combining with Equation (184)
Revised: which also satisﬁes the condition of Lemma 29 and combining with Equation (183)

Original: [Equation]
Revised: [Equation]

Original: we get that with high probability |
Revised: we get that with high probability |

Original: [Equation]
Revised: [Equation]

Original: where ∥ x ′ ∥ ≲ C 2 and ∥ V ′ ∥
Revised: where ∥ x ′ ∥ ≲ C 2 and ∥ V ′ ∥

Original: F ≤ C 2 , ∀ j : V ′ j ϕ (0) ( x i )
Revised: F ≤ C 2 , ∀ j : V ′ j ϕ (0) ( x i )

Original: = 0 .
Revised: = 0 .

Original: We also need to satisfy condition (185), which regarding this choice for θ = θ ∗ becomes
Revised: We also need to satisfy condition (184), which regarding this choice for θ = θ ∗ becomes

Original: [Equation]
Revised: [Equation]

Original: for which it sufﬁces to set
Revised: for which it sufﬁces to set

Original: [Equation]
Revised: [Equation]

Original: Substituting this choice of ϵ above and picking the overparameterization large enough to dominate the magnitude of C 1 , C 2 so that log(1 /ϵ ∗ ) ≲ log( m 2 ) , the proof is complete.
Revised: Substituting this choice of ϵ above and picking the overparameterization large enough to dominate the magnitude of C 1 , C 2 so that log(1 /ϵ ∗ ) ≲ log( m 2 ) , the proof is complete.

Original: Lemma 

Revised: Lemma

Original: Under the following condition
Revised: Under the following condition

Original: [Equation]
Revised: [Equation]

Original: with high probability we have
Revised: with high probability we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Similar to Lemma 22, deﬁne the helper functions Γ x ′ and Γ x ′ ,V ′ as
Revised: Similar to Lemma 22, deﬁne the helper functions Γ x ′ and Γ x ′ ,V ′ as

Original: [Equation]
Revised: [Equation]

Original: First we bound sup x ′ Γ x ′ .
Revised: First we bound sup x ′ Γ x ′ .

Original: To this end, note that because V (0) j ∈ R m 3 and the VC-dimension of half-planes is m 3 + 1 , then by Sauer’s Lemma, the set
Revised: To this end, note that because V (0) j ∈ R m 3 and the VC-dimension of half-planes is m 3 + 1 , then by Sauer’s Lemma, the set

Original: [Equation]
Revised: [Equation]

Original: of all sign pattern matrices has cardinality at most |D|
Revised: of all sign pattern matrices has cardinality at most |

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Now note that with high probability, the entries of the matrix V (0) are all less than
Revised: Now note that with high probability, the entries of the matrix V (0) are all less than

Original: O ( κp log( m 2 m 3 )) .
Revised: O ( κp log( m 2 m 3 )) .

Original: On the other hand, for each ﬁxed sign pattern D x ′ V (0) ,x i , we have for the sum with respect to this sign pattern:
Revised: On the other hand, for each ﬁxed sign pattern D x ′ V (0) ,x i , we have for the sum with respect to this sign pattern:

Original: [Equation]
Revised: [Equation]

Original: is ( m 3 κ 42 log 2 ( m 2 m 3 ) , κ 22 log( m 2 m 3 )) sub-exponential with respect to the randomness of a , because each entry of the vector
Revised: is ( m 3 κ 42 log 2 ( m 2 m 3 ) , κ 22 log( m 2 m 3 )) sub-exponential with respect to the randomness of a , because each entry of the vector

Original: [Equation]
Revised: [Equation]

Original: subGaussian.
Revised: subGaussian.

Original: Therefore, with high probability we have
Revised: Therefore, with high probability we have

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: if we take a union bound over all sign matrices in D and using the fact that m 2 > m 3 :
Revised: if we take a union bound over all sign matrices in D and using the fact that m 2 > m 3 :

Original: [Equation]
Revised: [Equation]

Original: which impliessup
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Γ

Original: [Equation]

Original: Moreover, deﬁning J v,x similar to Lemma 22 and using the similar approach we get with high probability
Revised: Moreover, deﬁning J v,x similar to Lemma 22 and using the similar approach we get with high probability

Original: [Equation]
Revised: [Equation]

Original: Now for simplifying the analysis, we assume that for indices j ∈ J x,r we can change the sign pattern with no cost on V ′ , i.e. we can pick any subset of them.
Revised: Now for simplifying the analysis, we assume that for indices j ∈ J x,r we can change the sign pattern with no cost on V ′ , i.e. we can pick any subset of them.

Original: Therefore, we ﬁrst compute a high probability upper bound on the following quantity:
Revised: Therefore, we ﬁrst compute a high probability upper bound on the following quantity:

Original: [Equation]
Revised: [Equation]

Original: If we form the matrix V (0) ( J x,r ) be the matrix which only keeps the rows with indices in J x,r , then the above quantity can be computed as
Revised: If we form the matrix V (0) ( J x,r ) be the matrix which only keeps the rows with indices in J x,r , then the above quantity can be computed as

Original: [Equation]
Revised: [Equation]

Original: where λ max is the maximum singular value of the matrix.
Revised: where λ max is the maximum singular value of the matrix.

Original: Now by random matrix theory, we know for a ﬁxed x and arbitrary t ≥ 0 , the following argument holds:
Revised: Now by random matrix theory, we know for a ﬁxed x and arbitrary t ≥ 0 , the following argument holds:

Original: [Equation]
Revised: [Equation]

Original: Therefore, as |D| ≤ m m 3 +1, we get with high probability
Revised: Therefore, as |D| ≤ m m 3 +1, we get with high probability

Original: [Equation]
Revised: [Equation]

Original: Therefore, with high probability
Revised: Therefore, with high probability

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: as in Equation (193) in the proof of Lemma
Revised: as in Equation (192) in the proof of Lemma

Original: [Equation]
Revised: [Equation]

Original: for j / ∈ J x,r we have:
Revised: for j / ∈ J x,r we have:

Original: [Equation]
Revised: [Equation]

Original: Picking
Revised: Picking

Original: [Equation]
Revised: [Equation]

Original: we get for j / ∈ J x,r
Revised: we get for j / ∈ J x,r

Original: [Equation]
Revised: [Equation]

Original: Now similar to the derivation in (194) we have
Revised: Now similar to the derivation in (193) we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence, because ∥ V ′ ∥ F ≤ C 2 , the number of indices for which 1 { ( V (0) j
Revised: Hence, because ∥ V ′ ∥ F ≤ C 2 , the number of indices for which 1 { ( V (0) j

Original: [Equation]
Revised: [Equation]

Original: )( ϕ (0) ( x i ) + x ′ ) ≥ 0 } ̸ = 1 { V (0) j ( ϕ (0) ( x i ) + x ′ ) ≥ 0 } is at most l
Revised: ( ϕ (0) ( x i ) + x ′ ) ≥ 0 } ̸ = 1 { V (0) j ( ϕ (0) ( x i ) + x ′ ) ≥ 0 } is at most l

Original: = ( C 1 C 2 ) 2 r.
Revised: = ( C 1 C 2 ) 2 r.

Original: Therefore, we bound the following quantity to use in the analysis:
Revised: Therefore, we bound the following quantity to use in the analysis:

Original: [Equation]
Revised: [Equation]

Original: But if we deﬁne for m 2 < j ≤ 2 m 2 ,
Revised: But if we deﬁne for m 2 < j ≤ 2 m 2 ,

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Now note that each entry of P j ∈ S V (0) j is √ lκ 2 subGaussian.
Revised: Now note that each entry of P j ∈ S V (0) j is √ lκ 2 subGaussian.

Original: Hence, the quantity ∥ P j ∈ S V (0) j ∥is ( m 3 l 2 κ 42 , lκ 22 ) subexponential.
Revised: Hence, the quantity ∥ P j ∈ S V (0) j ∥is ( m 3 l 2 κ 42 , lκ 22 ) subexponential.

Original: Therefore, we have with high probability
Revised: Therefore, we have with high probability

Original: [Equation]
Revised: [Equation]

Original: log

Original: [Equation]

Original: l

Original: [Equation]

Original: s log

Original: [Equation]

Original: l

Original: [Equation]

Original: l (cid:19)

Original: [Equation]

Original: Hence sup|
Revised: Hence sup

Original: [Equation]
Revised: [Equation]

Original: Now using Equation , we can write|
Revised: Now using Equation , we can write

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]

Original: Combining this with (207):
Revised: Combining this with (206):

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: r

Original: [Equation]
Revised: [Equation]

Original: Now setting
Revised: Now setting

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Now under the condition
Revised: Now under the condition

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal term is dominated by the ﬁrst term, which ﬁnally completes the proof.
Revised: The ﬁnal term is dominated by the ﬁrst term, which ﬁnally completes the proof.

Original: A. C ONVERGENCE
Revised: A. C ONVERGENCE

Original: The goal of this section is to prove Theorem 7.
Revised: The goal of this section is to prove Theorem 7.

Original: Theorem 7 Letting ℵ = 4 B 2 , by Corollary 8.1, we have L Π (0) ≤ ℵ .
Revised: Theorem 7 Letting ℵ = 4 B 2 , by Corollary 8.1, we have L Π (0) ≤ ℵ .

Original: We deﬁne the domain
Revised: We deﬁne the domain

Original: D l := {∥ w ′ ∥ ≤ C 1 :=
Revised: D l := {∥ w ′ ∥ ≤ C 1 :=

Original: ℵ +4 l ψ 1 , ∥ v ′ ∥ ≤ C 2 :
Revised: ℵ +4 l ψ 1 , ∥ v ′ ∥ ≤ C 2 :

Original: = ℵ +4 l ψ 2 } .
Revised: = ℵ +4 l ψ 2 } .

Original: For a large enough constant l = O (1) and function L Π ( w := ( w ′ , v ′ ))
Revised: For a large enough constant l = O (1) and function L Π ( w := ( w ′ , v ′ ))

Original: : R N → R .
Revised: : R N → R .

Original: Moreover, suppose L Π is ρ 1 lipschitz, ρ 2 gradient lipschitz, and ρ 3 hessian lipschitz in the domain D l  ,  in the sense that their ﬁrst, second, and third directional derivatives in an arbitrary unit direction is bounded by the corresponding parameters.
Revised: Moreover, suppose L Π is ρ 1 lipschitz, ρ 2 gradient lipschitz, and ρ 3 hessian lipschitz in the domain D l ( ρ 1 , ρ 2 , ρ 3 ≥ 1 ), in the sense that their ﬁrst, second, and third directional derivatives in an arbitrary unit direction is bounded by the corresponding parameters.

Original: Suppose we have access to the gradient of L Π at each point in D l plus a zero mean noise vector £ such that σ [Equation] ≤ E ££ T ≤ σ 22 I and ∥ £ ∥ ≤ Q almost surely.
Revised: Suppose we have access to the gradient of L Π at each point in D l plus a zero mean noise vector £ such that σ 21 I ≤ E ££ T ≤ σ 22 I and ∥ £ ∥ ≤ Q almost surely.

Original: Also, suppose for a threshold ℵ ℓ ≤ ℵ , if L Π ( w ) ≥ ℵ ℓ for any w ∈ D l , then weeither have 
Revised: Also, suppose for a threshold ℵ ℓ ≤ ℵ , if L Π ( w ) ≥ ℵ ℓ and w ∈ D l , then we have at least one of the following conditions holds:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: Then starting from w=

Original: [Equation]
Revised: [Equation]

Original: Then starting from w= 0 , with probability at least 0 .after at most poly ( ρ 1 , ρ 2 , ρ 3 , Q, ℵ , C 1 , C 2 , 1 /γ, log( σ 1 /σ 2 )) number of iterations, we reach a point w t such that L Π ( w t ) ≤ ℵ ℓ .
Revised: , with probability at least 0 .after at most poly ( ρ 1 , ρ 2 , ρ 3 , Q, ℵ , C 1 , C 2 , 1 /γ, log( σ 1 /σ 2 )) number of iterations, we reach a point w t such that L Π ( w t ) ≤ ℵ ℓ .

Original: Proof Our proof here is a reﬁned version of that in Ge et al.
Revised: Proof Our proof here is a reﬁned version of that in Ge et al.

Original: (2015a).
Revised: (2015a).

Original: As we mentioned in section 5, the key fact that we are using in the other parts of our proof is a uniform upper bound ∥ w ′ ∥ ≤ C 1 , ∥ v ′ ∥ ≤ C 2 which is unjustiﬁed by only naively using Ge et al.
Revised: As we mentioned in section 5, the key fact that we are using in the other parts of our proof is a uniform upper bound ∥ w ′ ∥ ≤ C 1 , ∥ v ′ ∥ ≤ C 2 which is unjustiﬁed by only naively using Ge et al.

Original: (2015a).
Revised: (2015a).

Original: Here, ﬁrst we restate a reﬁned version of Lemmas 14 and 16 in Ge et al.
Revised: Here, ﬁrst we restate a reﬁned version of Lemmas 14 and 16 in Ge et al.

Original: (2015a) in Lemmas 24 and 26 respectively, and then use them to also bound the upward deviations of L Π .
Revised: (2015a) in Lemmas 24 and 26 respectively, and then use them to also bound the upward deviations of L Π .

Original: Moreover, to avoid writing repeated proofs and overwhelm the reader, we mostly treat the arguments in Lemma 16 of Ge et al.
Revised: Moreover, to avoid writing repeated proofs and overwhelm the reader, we mostly treat the arguments in Lemma 16 of Ge et al.

Original: (2015a)
Revised: (2015a)

Original: as blackbox and use them for our purpose here.
Revised: as blackbox and use them for our purpose here.

Original: A point to mention before we start, unlike Lemma 14 of Ge et al.
Revised: A point to mention before we start, unlike Lemma 14 of Ge et al.

Original: (2015a) where the dependency on other parameters than the step size η is more explicit, Lemma 16 hides the dependencies on all the other parameters (which is polynomial).
Revised: (2015a) where the dependency on other parameters than the step size η is more explicit, Lemma 16 hides the dependencies on all the other parameters (which is polynomial).

Original: Here, we follow the same style.
Revised: Here, we follow the same style.

Original: We refer to the trajectory of the steps of algorithm by ( w t ) t ≥ 0 .
Revised: We refer to the trajectory of the steps of algorithm by ( w t ) t ≥ 0 .

Original: In Lemmas of this section, To avoid introducing new notation and complicating things, we refer to the current point of the algorithm by w 0 , while for the next point of the algorithm we use w 1 (in Lemma 24), and w T (in Lemma 26) respectively.
Revised: In Lemmas of this section, To avoid introducing new notation and complicating things, we refer to the current point of the algorithm by w 0 , while for the next point of the algorithm we use w 1 (in Lemma 24), and w T (in Lemma 26) respectively.

Original: Also, similar to Ge et al.
Revised: Also, similar to Ge et al.

Original: (2015a), ˜ O and ˜Ω below means we are looking at the dependency on η .
Revised: (2015a), ˜ O and ˜Ω below means we are looking at the dependency on η .

Original: Lemma 24 Suppose L Π ( w 0 ) ≤ ℵ +2 l , and consider a parameter χ > 1 which can be set arbitrarily.
Revised: Lemma 24 Suppose L Π ( w 0 ) ≤ ℵ +2 l , and consider a parameter χ > 1 which can be set arbitrarily.

Original: For every point w 0 such that L Π ( w 0 ) ≤ ℵ +2 , ∥∇ L Π ( w 0 )
Revised: For every point w 0 such that L Π ( w 0 ) ≤ ℵ + 2 l , ∥∇ L Π [Equation] w [Equation]

Original: ∥ ≥q η ( Q 2 + σ 22 N ) ρ 2 ρ 21 (2 χ + 12 ) , then for w 1 := w 0 − η ( ∇ L Π ( w 0 ) + £ ) and random variable R 1 (depending on w 0 ) deﬁned as
Revised: ≥ [Equation] then for w 1 := w 0 − η ( ∇ L Π ( w 0 ) + £ ) and random variable R 1 (depending on w 0 ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (229) we have R 1 = ˜Ω(1) a.s., and almost surely:
Revised: [Equation] we have R 1 = ˜Ω(1) a.s., and almost surely:

Original: [Equation]
Revised: [Equation]

Original: (the expectation is over the randomness of £ ).
Revised: (the expectation is over the randomness of £ ).

Original: Proof This lemma is a tuned version of Lemma 14 in Ge et al.
Revised: Proof This lemma is a tuned version of Lemma 14 in Ge et al.

Original: (2015a).
Revised: (2015a).

Original: First, note that the condition
Revised: First, note that the condition

Original: L Π ( w 0 ) ≤ ℵ + 2 l assures the smoothness coefﬁcients ρ 1 , ρ 2 and ρ 3 for L Π by Corollary ?? .
Revised: L Π ( w 0 ) ≤ ℵ + 2 l assures the smoothness coefﬁcients ρ 1 , ρ 2 and ρ 3 for L Π by Corollary ?? .

Original: We follow similar to Ge et al.
Revised: We follow similar to Ge et al.

Original: (2015a) (picking η < 1 / (2 /ρ 2 ) ):
Revised: (2015a) (picking η < 1 / (2 ρ 2 ) ):

Original: [Equation]
Revised: [Equation]

Revised: where we used the fact that ρ 1 ≥ 1 .

Original: On ther other hand, L Π is ρ 1 Lipcshitz, so we have almost surely
Revised: On ther other hand, L Π is ρ 1 Lipcshitz, so we have almost surely

Original: [Equation]
Revised: [Equation]

Original: 231) (To be completely precise, we should justify that we can write the Lipschitz inequality at point w 0 , we also need to make sure that w 1 remains in the domain that we have the Lipschitz parameter in, i.e. D l .
Revised: (To be completely precise, we should justify that we can write the Lipschitz inequality at point w 0 , we also need to make sure that w 1 remains in the domain that we have the Lipschitz parameter in, i.e. D l .

Original: To see why this is true, see the next Corollary).
Revised: To see why this is true, see the next Corollary).

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Taking η ≤(8 ρ 21 ρ 2 χ
Revised: Taking η ≤

Original: [Equation]
Revised: [Equation]

Original: − 1 , (233) we get from Equation (230):
Revised: (232) we get from Equation (229):

Original: [Equation]
Revised: [Equation]

Original: Combining Equations (232) and (234), we see that
Revised: Combining Equations (231) and (233), we see that

Original: [Equation]
Revised: [Equation]

Original: Hence, if we deﬁne
Revised: Hence, if we deﬁne

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 235) and from Equation (234), that
Revised: 234) and from Equation (233), that

Original: [Equation]
Revised: [Equation]

Original: Moreover, because the function is ρ 1 -Lipshitz at the domain point w 0 , we get from Equation (231):
Revised: Moreover, because the function is ρ 1 -Lipshitz at the domain point w 0 , we get from Equation (230):

Original: [Equation]
Revised: [Equation]

Original: by taking
Revised: by taking

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: This combined with Equation (235) anda triangle inequality implies:
Revised: This, combined with Equation (234) and triangle inequality implies:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Lemma 25 As long as the value of the function at some w is bounded by ℵ + 2 l ( L Π ( w ) ≤ ℵ + 2 l ), then η can be picked small enough (polynomially in other parameters) so that the change of the function by a step isa.s. bounded by l .
Revised: Lemma 25 As long as the value of the function at some w is bounded by ℵ + 2 l ( L Π ( w ) ≤ ℵ + 2 l ), then η can be picked small enough (polynomially in other parameters), namely η ≤ l/ ( ∥∇ L Π ( w 0 ) ∥ + Q ) , so that the change of the function by a step is bounded by l .

Original: Proof Let ψ = min { ψ 1 , ψ 2 } .
Revised: Proof Let ψ = min { ψ 1 , ψ 2 } .

Original: First, note that as the function is bounded by ℵ + 2 l , we have the lipschitz parameter ρ 1 , hence ∥∇ L Π ( w ) ∥ ≤ ρ 1 .
Revised: First, note that as the function is bounded by ℵ + 2 l , we have the lipschitz parameter ρ 1 , hence ∥∇ L Π ( w ) ∥ ≤ ρ 1 .

Original: Therefore, the change in w in a step is bounded as
Revised: Therefore, the change in w in a step is bounded as

Original: [Equation]
Revised: [Equation]

Original: So by picking
Revised: So by picking

Original: [Equation]
Revised: [Equation]

Original: we guarantee that the value of w after a step remains in the ball of radius q [Equation] +3 l ψ , hence we still have the smoothing parameters even after one step.
Revised: we guarantee that the value of w after a step remains in the ball of radius q ℵ +3 l ψ , hence we still have the smoothing parameters even after one step.

Original: Therefore, now we can use the Lipcshitz parameter ρto bound the value of the function after one step as it is written in Equation Equation (231).
Revised: Therefore, now we can use the Lipcshitz parameter ρto bound the value of the function after one step as it is written in Equation Equation (230).

Original: Using this Equation, it is enough to pick η as small as:
Revised: Using this Equation, it is enough to pick η as small as:

Original: [Equation]
Revised: [Equation]

Original: so that the change in the function would be  most l as desired.
Revised: so that the change in the function would be at most l as desired.

Original: Lemma 

Revised: Lemma

Original: For a ﬁxed point w 0 s.t.
Revised: For a ﬁxed point w 0 s.t.

Original: L Π ( w 0 ) ≤ ℵ + 2 l , suppose we pick η small enough such that
Revised: L Π ( w 0 ) ≤ ℵ + 2 l , suppose we pick η small enough such that

Original: [Equation]
Revised: [Equation]

Original: Then, note that for ∥∇ L Π ( w 0 )
Revised: Then, note that for ∥∇ L Π ( w 0 )

Original: ∥ ≤ § ( η ) , condition 228 implies:
Revised: ∥ ≤ § ( η ) , condition 227 implies:

Original: [Equation]
Revised: [Equation]

Original: Then, using the notation E T for the high probability event corresponding to Equations (36) and (44) in Ge et al.
Revised: Then, using the notation E T for the high probability event corresponding to Equations (36) and (44) in Ge et al.

Original: (2015a), for small enough η (polynomially small w.r.t other parameters), for R 2 deﬁned as
Revised: (2015a), for small enough η (polynomially small w.r.t other parameters), for R 2 deﬁned as

Original: [Equation]
Revised: [Equation]

Original: we have almost surely
Revised: we have almost surely

Original: [Equation]
Revised: [Equation]

Original: Note that in the expectations above w 0 is assumed ﬁxed.
Revised: Note that in the expectations above w 0 is assumed ﬁxed.

Original: Furthermore, we can assume P ( E T ) ≥ 1 − ˜ O
Revised: Furthermore, we can assume P ( E T ) ≥ 1 − ˜ O

Original: ( η 5 ) .
Revised: ( η 5 ) .

Original: Proof This Lemma is a tuned version of Lemma 16 in Ge et al.
Revised: Proof This Lemma is a tuned version of Lemma 16 in Ge et al.

Original: (2015a).
Revised: (2015a).

Original: We change a couple of things here.
Revised: We change a couple of things here.

Original: First, we consider an implicit coupling that if w t exits D l we do not move it anymore, i.e. w t ′ =
Revised: First, we consider an implicit coupling that if w t exits D l we do not move it anymore, i.e. w t ′ =

Original: w t , ∀ t ′ ≥ t , which means the noise vectors also becomes zero, i.e. £ t ′ = 0 , ∀ t ′ ≥ t .
Revised: w t , ∀ t ′ ≥ t , which means the noise vectors also becomes zero, i.e. £ t ′ = 0 , ∀ t ′ ≥ t .

Original: This way, the sequence of noise vectors remain bounded by Q , because if w t is inside D l , then by assumption ∥ £ t ∥ ≤ Q , while otherwise £ t = 0 .
Revised: This way, the sequence of noise vectors remain bounded by Q , because if w t is inside D l , then by assumption ∥ £ t ∥ ≤ Q , while otherwise £ t = 0 .

Revised: We denote the event that the sequence w 0 , . . .

Original: We call this event E T for T  deﬁned in Lemma 16 of Ge et al.
Revised: w T remain in D l by E T , where T is deﬁned in Lemma 16 of Ge et al.

Original: (2015a).
Revised: (2015a).

Original: Note that we also have the smoothing parameters ρ 1 , ρ 2 , ρ 3 for all ( w t ) because of this coupling.
Revised: Note that we also have the smoothing parameters ρ 1 , ρ 2 , ρ 3 for all ( w t ) because of this coupling.

Original: In fact, we will use a more strict coupling; we consider the event E T to be the high probability event corresponding to the bounds in Equations (44) and (36) of Ge et al.
Revised: In fact, we will use a more strict coupling; we consider the event E T to be the high probability event corresponding to the bounds in Equations (44) and (36) of Ge et al.

Original: (2015a) holding for all t ≤ T ; We will see that E T ⊆
Revised: (2015a) holding for all t ≤ T ; We will see that E T ⊆

Original: E T at the end of this proof, but for now we assume it is true.
Revised: E T at the end of this proof, but for now we assume it is true.

Original: An important point to note here is that in Ge et al.
Revised: An important point to note here is that in Ge et al.

Original: (2015a), P ( E T ) is bounded by O ( η 2 ) .
Revised: (2015a), P ( E T ) is bounded by O ( η 2 ) .

Original: However, the exponent dependency of η in this bound comes from Azuma-Hoeffding type inequalities, particularly used in Equations (60) and (42) in Ge et al.
Revised: However, the exponent dependency of η in this bound comes from Azuma-Hoeffding type inequalities, particularly used in Equations (60) and (42) in Ge et al.

Original: (2015a), in which by considering larger constants one can easily get higher exponents.
Revised: (2015a), in which by considering larger constants one can easily get higher exponents.

Original: Unlike their analysis though, a bit stronger dependence of η 5 is sufﬁcient for our later use.
Revised: For our analysis, a bit stronger dependence of η 5 is required.

Original: Also, because the distribution of our noise depends on the point w , our sequence of noise vectors ( £ t ) is a martingale instead of being i.i.d, so we apply Azuma-Hoeffding inequality instead of the simple Hoeffdings in Lemma 16 of Ge et al.
Revised: Also, because the distribution of our noise depends on the point w , our sequence of noise vectors ( £ t ) is a martingale instead of being i.i.d, so we apply Azuma-Hoeffding inequality instead of the simple Hoeffdings in Lemma 16 of Ge et al.

Original: (2015a).
Revised: (2015a).

Original: (because we are also samplingalso a random ( x i , y i ) to compute the estimate of  gradient, this could be simpliﬁed to the case where we compute the actual gradient and the injecting an i.i.d noise vector in each step, but it is an overhead to compute the actual gradient, so here we choose to analyze the more complicated case.)
Revised: (because we are also sampling a random ( x i , y i ) to compute the estimate of the gradient, this could be simpliﬁed to the case where we compute the actual gradient and then inject an i.i.d noise vector in each step, but it is an overhead to compute the actual gradient, so here we choose to analyze the more complicated case.)

Original: Next, notice the deﬁnition of Λ and ˜Λ right after Equation (66) in Ge et al.
Revised: Next, notice the deﬁnition of Λ and ˜ Λ right after Equation (66) in Ge et al.

Original: (2015a), which in our notation translates to
Revised: (2015a), which in our notation translates to

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for ( ˜ w t ) which is a coupled sequence with ( w t ) as deﬁned in Ge et al.
Revised: for ( ˜ w t ) which is a coupled sequence with ( w t ) as deﬁned in Ge et al.

Original: (2015a).
Revised: (2015a).

Original: Note that we apply the coupling for the sequence ˜ w t as well, i.e. if w t +1 =
Revised: Note that we apply the coupling for the sequence ˜ w t as well, i.e. if w t +1 =

Original: w t , we also set ˜ w t +

Revised: w t , we also set ˜ w t +

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: w t .
Revised: w t .

Original: To show Equation (238), we want to use Equation (67) in Ge et al.
Revised: To show Equation (237), we want to use Equation (67) in Ge et al.

Original: (2015a), though we only use the expansion for the ﬁrst term which is under 1 {E T } , i.e.
Revised: (2015a), though we only use the expansion for the ﬁrst term which is under 1 {E T } , i.e.

Original: [Equation]
Revised: [Equation]

Original: First of all, as it is mentioned in Lemma 16 of (Ge et al., 2015a), in the case where the noise vector σ 12 I ≤ E ££ T ≤ σ 22 I instead of having E ££ T = σ 2 I for a ﬁxed σ , in order to still get a negative term of order η in Equation (68) of (Ge et al., 2015a), we just need the size of T max to be as large as O ( 1 γη (log d + log σ 2 σ 1 )) , and it does not change the order of η in any other part of Lemma 16.
Revised: First of all, as it is mentioned in Lemma 16 of (Ge et al., 2015a), in the case where the noise vector σ 12 I ≤ E ££ T ≤ σ 22 I instead of having E ££ T = σ 2 I for a ﬁxed σ , in order to still get a negative term of order η in Equation (68) of (Ge et al., 2015a), we just need the size of T max to be as large as O ( 1 γη (log d + log σ 2 σ 1 )) , and it does not change the order of η in any other part of Lemma 16.

Original: Now similar to Equation (68) of (Ge et al., 2015a), if w.l.o.g we assume the smallest eigenvalue γcorresponds to i
Revised: Now similar to Equation (68) of (Ge et al., 2015a), if w.l.o.g we assume the smallest eigenvalue γcorresponds to i

Original: = 1 :
Revised: = 1 :

Original: [Equation]
Revised: [Equation]

Original:  where in the last line we use the fact that P ( E T ) ≤ 1 / 2 plus the additional log( σ 2 /σ 1 ) factor.
Revised: 243) where in the last line we use the fact that P ( E T ) ≤ 1 / 2 plus the additional log( σ 2 /σ 1 ) factor.

Original: Second, note that our threshold § ( η ) for the size of gradient in Lemmas 24 and 26 has the same order of η compared to that of Lemmas 14 and 16 in Ge et al.
Revised: Second, note that our threshold § ( η ) for the size of gradient in Lemmas 24 and 26 has the same order of η compared to that of Lemmas 14 and 16 in Ge et al.

Original: (2015a).
Revised: (2015a).

Original: Therefore, the arguments in
Revised: Therefore, the arguments in

Original: Lemma 16 that considers the order of η and treat the other parameters as constants is true here as well.
Revised: Lemma 16 that considers the order of η and treat the other parameters as constants is true here as well.

Original: Hence, we still have Equation (69) of Ge et al.
Revised: Hence, we still have Equation (69) of Ge et al.

Original: (2015a) which is under the event E T .
Revised: (2015a) which is under the event E T .

Original: Applying it to Equation (241),
Revised: Applying it to Equation (240),

Original: Hence, ﬁnally by a similar derivation of Equation (67) in Ge et al.
Revised: Hence, ﬁnally by a similar derivation of Equation (67) in Ge et al.

Original: (2015a):
Revised: (2015a):

Original: [Equation]
Revised: [Equation]

Original: Furthermore, combining Equations (36) and (44)  we get with high probability (we use the ﬁnal high probability parameter of Lemma 16 which is the result of a union bound over all the high probability arguments which is equivalent to the occurrence of E T ), i.e. when E T happens, 
Revised: [Equation] Next, we turn to prove the second bound (238). Combining Equations (36) and (44) in (Ge et al., 2015a), we get with high probability (we use the ﬁnal high probability parameter of Lemmawhich is the result of a union bound over all the high probability arguments which is equivalent to the occurrence of E T ), i.e. when E T happens, [Equation]

Original: [Equation]
Revised: [Equation]

Original: Picking η small enough such that for the bound above we have
Revised: Picking η small enough such that for the bound above we have

Original: [Equation]
Revised: [Equation]

Original: we get for every ¯ w in the line connecting w 0 to w T :
Revised: we get for every ¯ w in the line connecting w 0 to w T :

Original: [Equation]
Revised: [Equation]

Original: which implies that L Π has the smoothing parameters
Revised: which implies that L Π has the smoothing parameters

Original: ρ 1 , ρ 2 , ρ 3 along w 0 to w T .
Revised: ρ 1 , ρ 2 , ρ 3 along w 0 to w T .

Original: Therefore, by the ρ 2 -gradient smoothness property of L Π :
Revised: Therefore, by the ρ 2 -gradient smoothness property of L Π :

Original: [Equation]
Revised: [Equation]

Original: Combining the assumption of the Lemma ∥∇ L Π ( w0 )
Revised: Combining the assumption of the Lemma ∥∇ L Π ( w

Original:  ∥ ≤˜ O
Revised: [Equation] ∥ ≤

Original: η 12 ) , we get

Original: [Equation]
Revised: [Equation]

Revised: , we get ∥∇ L Π ( ¯ w ) ∥ =

Revised: ˜ O ( η 1 / 2 log(1 /η )) .

Original: (The last ˜ O also hides the dependency on ρ 2 ).
Revised: (The last ˜ O also hides the dependency on ρ 2 ).

Original: Now integrating over the derivative along the direction from w 0 to w T :
Revised: Now integrating over the derivative along the direction from w 0 to w T :

Original: [Equation]
Revised: [Equation]

Original: w T ) T ( w T − w 0 ) dt.

Original: Therefore, using (246) one more time, under the event E T :
Revised: Therefore, using (245) one more time, under the event E T :

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: which
Revised: which

Original: Now comparing Equations (245) and (247), it is obvious that one can pick η small enough (again polynomially small in the other parameters) such that for some random variable R 2 , which also depends on η Equations (238) and (239) hold. 
Revised: Now comparing Equations (244) and (246), it is clear that one can pick η small enough (again polynomially small in the other parameters) such that for some random variable R 2 , which also depends on η , so that equations (237) and (238) hold. It remains to show E T ⊆

Revised: E T .

Revised: This is desirable as up until now we have only proved (237) and (238) for the coupled sequence (which does not move outside the ball D l ), but we know that under the event E T , the coupled sequence and the original sequence are the same, which automatically implies the conclusion for the original sequence.

Original: Furthermore, note that the bound in (247) is an a.s. upper bound on the change of the function value under the event E T for every 1 ≤ t ≤ T .
Revised: Notice that the bound in (246) is an a.s. upper bound on the change of the function value under the event E T for every 1 ≤ t ≤ T .

Original: Therefore, by picking η small enough (polynomially) s.t.
Revised: Therefore, by picking η small enough (polynomially) s.t.

Original: the quantity O ( η log(1 /η ) 2 ) in Equation (247) is bounded by l , we again make sure that the value of function during these steps changes by at most  compared to w 0 , i.e. for every 1 ≤ t ≤ T :
Revised: the quantity O ( η log(1 /η ) 2 ) in Equation (246) is bounded by l , we again make sure that the value of function during these steps changes by at most l compared to w 0 , i.e. for every 1 ≤ t ≤ T :

Original: [Equation]
Revised: [Equation]

Original: ,therefore remains bounded by ℵ + 3 l .
Revised: hence, remains bounded by ℵ + 3 l .

Original: Thisalso impliesthat E T ⊆
Revised: This implies E T ⊆

Original: E T as promised.
Revised: E T as promised.

Original: A. P ROCESS FROM A HIGHER VIEW : DEFINITION OF THE ( X ) SEQUENCE
Revised: A. P ROCESS FROM A HIGHER VIEW : DEFINITION OF THE ( X ) SEQUENCE

Original: The goal  is to ﬁnd a w ∗ with L Π ( w ∗ ) ≤ ℵ ℓ using Lemmas 24 and 26.
Revised: The goal here is to ﬁnd a w ∗ with L Π ( w ∗ ) ≤ ℵ ℓ using Lemmas 24 and 26 (recall the deﬁnition of ℵ ℓ from Theorem 7). The main result of this section is Lemma 27.

Original: For this purpose we deﬁne a coupled sequence this way: First, as done in Ge et al.
Revised: For this purpose we deﬁne a useful coupling: to begin, as done in Ge et al.

Original: (2015a), deﬁne a sequence of times
Revised: (2015a), deﬁne a sequence of times

Original: τ i inductively in the following way: To deﬁne τ
Revised: τ i inductively in the following way: To deﬁne τ

Original: i +1 based on τ
Revised: i +1 based on τ

Original: i , if the condition
Revised: i , if the condition

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: does not hold, then just set τ i +

Revised: does not hold, then just set τ i +

Original: =
Revised: =

Original: τ i ⋆ (1) .
Revised: τ i ⋆ (1) .

Original: Otherwise, using the conditions (228), we are either in the situation of Lemma 24 or Lemma 26, setting  w 0 =
Revised: Otherwise, using the conditions (227), we are either in the situation of Lemma 24 or Lemma 26 by setting the value of w 0 in these Lemmas as w 0 =

Original: w τ i .
Revised: w τ i .

Original: In the ﬁrst case, deﬁne τ i +

Revised: In the ﬁrst case, deﬁne τ i +

Original: =
Revised: =

Original: τ i +⋆ (2) .
Revised: τ i + 1 ⋆ (2) .

Original: In the latter case, Let E T be the same high probability event that we consider in Lemma (26), which happens when the aggregate behavior of the noise vectors does not behave oddly, so that we remain close to the starting point w 0 .
Revised: In the latter case, Let E T be the same high probability event that we consider in Lemma (26), which happens when the aggregate behavior of the noise vectors is normal, as a result of which w remains close to the starting point w 0 .

Original: Note that from Lemma 26, we know P ( E T ) ≥ 1 − O ( η 2 ) .
Revised: Note that from Lemma 26, we know P ( E T ) ≥ 1 − O ( η 5 ) .

Original: Now if the event E T happens, deﬁne τ i +1 :
Revised: Now if the event E T happens, deﬁne τ i +1 :

Original: =
Revised: =

Original: τ i + T ⋆ (3) , for T also from Lemma 26 and deﬁned originally in Lemma 16 of Ge et al.
Revised: τ i + T ⋆ (3) , for T also from Lemma 26 and deﬁned originally in Lemma 16 of Ge et al.

Original: (2015a), while otherwise deﬁne τ i +

Revised: (2015a), while otherwise, deﬁne τ i +

Original: =
Revised: =

Original: τ i ⋆ (4) and, moreover, deﬁneall the rest of τ i ′ ’s equal to τ i : τ i ′ =
Revised: τ i ⋆ (4) . Moreover, if E T does not happen, deﬁne the rest of τ i ′ ’s equal to τ i : τ i ′ =

Original: τ i for every i ′ ≥
Revised: τ i for every i ′ ≥

Original: i .
Revised: i .

Original: At the same time, we deﬁne the event G i  , where G i happens in the case ⋆ (4) , and G i +1 happens in case ⋆ (4) .
Revised: At the same time, we deﬁne the monotone increasing events {G i } , where G i happens in the case ⋆ (4) , and G i +1 happens in case ⋆ (4) .

Original: Also, G i happens if any of the previous G i ′ ’s happen for i ′ <
Revised: Also, G i happens if any of the previous G i ′ ’s happen for i ′ <

Original: i ; in other words, G i is included in G i +1 .
Revised: i ; in other words, G i is included in G i +1 .

Original: We use these events to bound the probability that the process remains above ℵ ℓ .
Revised: We use these events to bound the probability that the process remains above ℵ ℓ .

Original: Moreover, deﬁne the sequence of random variables ( X i ) as X i :=
Revised: Moreover, deﬁne the sequence of random variables ( X i ) as X i :=

Original: L Π ( w τ i ) .
Revised: L Π ( w τ i ) .

Original: Note that by Lemma 25 and Equation (248) in Lemma 26, we have ℵ ℓ − l ≤ X i ≤ ℵ + 3 l .
Revised: Note that by Lemma 25 and Equation (247) in Lemma 26, we have ℵ ℓ − l ≤ X i ≤ ℵ + 3 l .

Original: The key idea behind deﬁning X
Revised: The key idea behind deﬁning X

Original: i ’s is that we want to bound the MGF of L Π ( w t ) , without worrying about falling out of the assumptions of Lemmas 24 and 26.
Revised: i ’s is that we want to bound the MGF of L Π ( w t ) , without worrying about falling out of the assumptions of Lemmas 24 and 26.

Original: With the deﬁnition of ( X i ) and G i , we are ready to state the theorem which roughly says the sequence τ i will most likely stop after a number of steps.
Revised: With the deﬁnition of ( X i ) and G i , we are ready to state the theorem which roughly says the sequence τ i will most likely stop after a number of steps.

Original: Lemma 27 Let Q R := S∞ i =
Revised: Lemma 27 Let Q R := S

Original: [Equation]
Revised: [Equation]

Original: Then, for some
Revised: Then, for some

Original: [Equation]
Revised: [Equation]

Original: we have P ( Q R ) ≤ δ 1 .
Revised: we have P ( Q R ) ≤ δ 1 .

Original: In other words, after R iterations of PSGD , the deﬁned sequence ( X i ) above has either been in situation ⋆ (1) or ⋆ (4) .
Revised: In other words, after R iterations of PSGD , the deﬁned sequence ( X i ) above has either been in situation ⋆ (1) or ⋆ (4) .

Revised: Here, θ depends polynomially in the other parameters.

Original: Proof By Equations (229) and (245) in Lemmas 24 and 26, there exist a constant θ depending polynomially on all parameters except η such that
Revised: Proof By Equations (228) and (244) in Lemmas 24 and 26, there exist a constant θ depending polynomially on all parameters except η such that

Original: [Equation]
Revised: [Equation]

Original: Now for some constant C that we specify later [Equation] deﬁne the random time ı as the largest i where τ i ≤ C/η 2 .
Revised: Now for some constant C that we specify later, deﬁne the random time ı as the largest i where τ i ≤ C/η 2 .

Original: Using the fact that G i − 1 ⊆
Revised: Using the fact that G i − 1 ⊆

Original: G i , for every i we have a.s.:
Revised: G i , for every i we have a.s.:

Original: [Equation]
Revised: [Equation]

Original: Now summing this for i = 1 to ı [Equation] taking expectation from both sides and using (251):
Revised: Now summing this for i = 1 to ı , taking expectation from both sides and using (250):

Original: [Equation]
Revised: [Equation]

Original: Now using Lemma [Equation] we know that in except when E T happens (in which we stop the time sequence τ i ), the increments of X i are at most l .
Revised: Now using Lemma 25, we know that in except when E T happens (in which we stop the time sequence τ i ), the increments of X i are at most l .

Original: Therefore, the value of X i ’s always remain bounded by ℵ + 3 l , hence:
Revised: Therefore, the value of X i ’s always remain bounded by ℵ + 3 l , hence:

Original: [Equation]
Revised: [Equation]

Original: Also, by restricting the integration of the second term to the part S ∞ i =
Revised: Also, by restricting the integration of the second term to the part S ∞ i =

Original: [Equation]
Revised: [Equation]

Original: ¯ G i ∩ { τ
Revised: ¯ G i ∩ { τ

Original: i ≥ 2 C/η 2 } (cid:17) of the sample space, we know that under the event { ı ≥ i } , ¯ G i automatically happens  (it is easy to check).
Revised: i ≥ 2 C/η 2 } (cid:17) of the sample space, we know that under the event { ı ≥ i } , ¯ G i automatically happens when τ i +1 ̸ = τ i (it is easy to check).

Original: Therefore:
Revised: Therefore:

Original: [Equation] 
Revised: [Equation] [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ı
Revised: ı

Revised: [Equation]

Revised: [Equation]

Revised: { ∞ [ i

Original: [Equation]
Revised: [Equation]

Revised: (cid:16)

Original: Now by the deﬁnition of ı , τ ı +1 ≥ C/η 2 :
Revised: [Equation] { τ i ≥ [Equation] /η

Original: [Equation] 
Revised: [Equation] (cid:17) } ı X i = ( τ i +1 − τ i ) = ℵ + 3 l − θη 2 E 1 { ∞ [ i = (cid:16)

Original: } ( C/η 2 ) ℵ + 3 l − Cθ P (cid:16) ∞ [ i = (cid:16) ¯ G i ∩ { τ i ≥ C/η 2 } (cid:17)(cid:17) . 
Revised: ¯ G i ∩ { τ i ≥ C/η 2 } (cid:17) } τ ı +1 . Now by the deﬁnition of ı , τ ı +1 ≥ C/η 2 . Hence, we can write

Revised: [Equation]

Original: But note that X i ’s are a.s. bounded between 0 and ℵ + 3 l , which implies the LHS above is at least − ( ℵ + 3 l ) .
Revised: But note that X i ’s are a.s. bounded between 0 and ℵ + 3 l , which implies the LHS above is at least − ( ℵ + 3 l ) .

Original: Therefore, we ﬁnally get:
Revised: Therefore, we ﬁnally get:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by picking C∗ =  2(2 ℵ + 6 l ) /θ :
Revised: Picking C = C ∗ := 2(2 ℵ + 6 l ) /θ :

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Note that the differences between τ i ’s is at most T max =
Revised: Note that the differences between τ i ’s is at most T max =

Original: ˜ O (1 /η )
Revised: ˜ O (1 /η )

Original: Ge et al.
Revised: Ge et al.

Original: (2015a).
Revised: (2015a).

Original: Hence, again for η polynomially small in other parameters, (252) implies that for R = 2 C ∗ η 2 , there exist ˜ R = poly ( . )
Revised: Hence, again for η polynomially small in other parameters, (251) implies that for R = 2 C ∗ /η 2 , there exists ˜ R = poly ( . )

Original: such that after ˜ R iterations on the main sequence ( w t ) , the corresponding sequence ( τ i ) has either been in ⋆ (1) or ⋆ (4) with chance at least 1 / 2 .
Revised: such that after ˜ R iterations on the main sequence ( w t ) , the corresponding sequence ( τ i ) has either been in ⋆ (1) or ⋆ (4) with chance at least 1 / 2 .

Original: Repeating this argument log(1 /δ 1 ) times (using the markov property of the process) we conclude the proof.
Revised: Repeating this argument log(1 /δ 1 ) times (using the markov property of the process) we conclude the proof.

Original: A. B OUNDING THE MGF OF X i ’ S
Revised: A. B OUNDING THE MGF OF X i ’ S

Original: Next, we want to exploit X i ’s to bound the upward deviation of L Π ( w t ) .
Revised: Next, we want to exploit X i ’s to bound the upward deviation of L Π ( w t ) .

Original: For a ﬁx θ the goal here is to bound E [exp { θX i } ]
Revised: For a ﬁx θ the goal here is to bound E [exp { θX i } ]

Original: (this is a different θ !).
Revised: (this is a different θ !).

Original: More precisely, let F t be the sub-sigma ﬁeld generated by variables w t from time zero to t , and F i :=
Revised: More precisely, let F t be the sub-sigma ﬁeld generated by variables w t from time zero to t , and F i :=

Original: F τ i be the sigma ﬁeld of the stop time τ i .
Revised: F τ i be the sigma ﬁeld of the stop time τ i .

Original: Then, obviously, X i is measurable w.r.t F i .
Revised: Then, obviously, X i is measurable w.r.t F i .

Original: We prove the following theorem:
Revised: We prove the following theorem:

Original: Theorem 8 For any θ > 0 , the sequence ( E e θ ( X i − X 0 ) ) ∞
Revised: Theorem 8 For any θ > 0 , the sequence ( E e θ ( X i − X 0 ) ) ∞

Original: i =1 is a supermartingale with respect to the ﬁlteration ( F i ) ,
Revised: i =1 is a supermartingale with respect to the ﬁlteration ( F i ) ,

Original: Proof We proceed inductively by jointly conditioning on the previous X i and whether G i has happened or not, and whether we are in situation ⋆ (2) or ⋆ (3) .
Revised: Proof We proceed inductively by jointly conditioning on the previous X i and whether G i has happened or not, and whether we are in situation ⋆ (2) or ⋆ (3) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Now by the a.s. bounds of Lemmas 24 and 26:
Revised: Now by the a.s. bounds of Lemmas 24 and 26:

Original: [Equation]
Revised: [Equation]

Original: Where R 1 and R 2 are r.v. deﬁned in Lemmas 24 and 26 and are clearly F i measurable.
Revised: Where R 1 and R 2 are r.v. deﬁned in Lemmas 24 and 26 and are clearly F i measurable.

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Now we mention the following fact:
Revised: Now we mention the following fact:

Original: Fact For a σ subGaussian random variable X we have E [exp { θX } ] ≤ exp { θ 2 σ 2 } .
Revised: Fact For a σ subGaussian random variable X we have E [exp { θX } ] ≤ exp { θ 2 σ 2 } .

Original: Using the a.s. bounds of Lemmas 24 and 26, we get that conditioned on { ¯ G i , w τ i s.t.
Revised: Using the a.s. bounds of Lemmas 24 and 26, we get that conditioned on { ¯ G i , w τ i s.t.

Original: ⋆ (2) } ,
Revised: ⋆ (2) } ,

Original: X i +1 − E X
Revised: X i +1 − E X

Original: i +

Revised: i +

Original: is a.s. bounded by 2 ηθ R 1 / ( ρ 2 χ ) , and conditioned on { ¯ G i , w τ i s.t.
Revised: is a.s. bounded by 2 ηθ R 1 / ( ρ 2 χ ) , and conditioned on { ¯ G i , w τ i s.t.

Original: ⋆ (3) } ,
Revised: ⋆ (3) } ,

Original: X i +1 − E X
Revised: X i +1 − E X

Original: i +1 is bounded by 2 ηθ R 2 / ( ρ 2 χ ) .
Revised: i +1 is bounded by 2 ηθ R 2 / ( ρ 2 χ ) .

Original: Therefore, using the above fact
Revised: Therefore, using the above fact

Original: [Equation]
Revised: [Equation]

Original: which implies in the notation of conditional expectation on sigma ﬁeld:
Revised: which implies in the notation of conditional expectation on sigma ﬁeld:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: Now we write:

Original: [Equation]
Revised: [Equation]

Original: Now we write:{+}| F i ] + E [exp { θ ( X i +1 − E [ X i +1 | F i ]) } exp { θ ( E [ X i +1 | F i ] − X i ) } exp { θ ( X i − X 0 ) } 1 { ¯ G i ∩ ⋆ (2) }| F i ] + E [exp { θ ( X i +1 − E [ X i +1 | F i ]) } exp { θ ( E [ X i +
 | F i ] − X i ) } exp { θ ( X i − X 0 ) } 1 { ¯ G i ∩ ⋆ (3) }|F i ] ≤ exp { θ ( X i − X 0 ) } 1 {G i } + exp { θ ( X i − X 0 ) } E [exp { θ 2 R 21 η 2 / ( ρ 2 χ ) } exp {− θ ( R 21 η 2 ) } 1 { ¯ G i ∩ ⋆ (2) }| F i ] + exp { θ ( X i − X 0 ) } E [exp { θ 2 R 22 η 2 / ( ρ 2 χ ) } exp {− θ ( R 22 η 2 ) } 1 { ¯ G i ∩ ⋆ (3) }+}
Revised: {+}| F i ] + E [exp { θ ( X i +1 − E [ X i +1 | F i ]) } exp { θ ( E [ X i +1 | F i ] − X i ) } exp { θ ( X i − X 0 ) } 1 { ¯ G i ∩ ⋆ (2) }| F i ] + E [exp { θ ( X i +1 − E [ X i +1 | F i ]) } exp { θ ( E [ X i +
 | F i ] − X i ) } exp { θ ( X i − X 0 ) } 1 { ¯ G i ∩ ⋆ (3) }|F i ] ≤ exp { θ ( X i − X 0 ) } 1 {G i } + exp { θ ( X i − X 0 ) } E [exp { θ 2 R 21 η 2 / ( ρ 2 χ ) } exp {− θ ( R 21 η 2 ) } 1 { ¯ G i ∩ ⋆ (2) }| F i ] + exp { θ ( X i − X 0 ) } E [exp { θ 2 R 22 η 2 / ( ρ 2 χ ) } exp {− θ ( R 22 η 2 ) } 1 { ¯ G i ∩ ⋆ (3) }+}

Original: [Equation]{+| F i ] ≤ exp { θ ( X i − X 0 ) } E h(cid:16) 
 {G i } + exp { θ 2 R 21 η 2 / ( ρ 2 χ ) − θ ( R 21 η 2 ) } 1 { ¯ G i ∩ ⋆ (2) } + exp { θ 2 R 22 η 2 / ( ρ 2 χ ) − θ ( R 22 η 2 ) } 1 { ¯ G i ∩ ⋆ (3) } (cid:17) | F i i .+}
Revised: {+| F i ] ≤ exp { θ ( X i − X 0 ) } E h(cid:16) 
 {G i } + exp { θ 2 R 21 η 2 / ( ρ 2 χ ) − θ ( R 21 η 2 ) } 1 { ¯ G i ∩ ⋆ (2) } + exp { θ 2 R 22 η 2 / ( ρ 2 χ ) − θ ( R 22 η 2 ) } 1 { ¯ G i ∩ ⋆ (3) } (cid:17) | F i i .+}

Original: Now setting θ := 1 and picking χ ≥ 1 /ρ 2 :
Revised: Now setting θ := 1 and picking χ ≥ 1 /ρ 2 :

Revised: LHS

Original: [Equation]
Revised: [Equation]

Revised: E h

Revised: [Equation]

Revised: i i .

Revised: =

Revised: [Equation]

Original: Now by hypothesis of Induction we have
Revised: Now by hypothesis of Induction we have

Original: [Equation]
Revised: [Equation]

Original: which ﬁnishes the proof of step of induction.
Revised: which ﬁnishes the proof of step of induction.

Original: Now using Doob’s Maximal inequality for positive supermartingales and R deﬁned in (250):
Revised: Now using Doob’s Maximal inequality for positive supermartingales and R deﬁned in (249):

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Finally with the developed tools [Equation] we are ready to prove Theorem 
Revised: Finally with the developed tools, we are ready to prove Theorem 7.

Original: [Equation] Proof of Theorem 7 Starting from w 0 = 0 with L Π ( w 0 ) ≤ ℵ , we use Equation (257) to get P (sup 1 ≤ i ≤ R exp { X i − X 0 } ≥ Ω(log(1 /δ 1 )))
Revised: Proof of Theorem 7 Starting from w 0 = 0 with L Π ( w 0 ) ≤ ℵ , we use Equation (256) to get P (sup 1 ≤ i ≤ R exp { X i − X 0 } ≥ Ω(log(1 /δ 1 )))

Original: ≤ δ 1 .
Revised: ≤ δ 1 .

Original: Therefore, setting l = Θ(log(1 /δ 1 )) and a union bound implies with probability at least 1 − 2 δ 1 we should have gotten into situation ⋆ (1) or ⋆ (4) without the value of X i exceeding ℵ + 2 l .
Revised: Therefore, setting l = Θ(log(1 /δ 1 )) and a union bound implies with probability at least 1 − 2 δ 1 we should have gotten into situation ⋆ (1) or ⋆ (4) without the value of X i exceeding ℵ + 2 l .

Original: On the other hand, using Lemma 26 we know that
Revised: On the other hand, using Lemma 26 we know that

Original: E T happens with probability at least 1 − ˜ O
Revised: E T happens with probability at least 1 − ˜ O

Original: ( η 5 ) for every 1 ≤ t ≤ R which is equal to τ i for some i
Revised: ( η 5 ) for every 1 ≤ t ≤ R which is equal to τ i for some i

Original: and when we are in the situation of Lemma 26.
Revised: and when we are in the situation of Lemma 26.

Original: As a result, the chance that even one of E T ’s happen along R iterations is at most
Revised: As a result, the chance that even one of E T ’s happen along R iterations is at most

Original: [Equation]
Revised: [Equation]

Original: But picking η small enough with respect to log( δ 1 ) and other parameters, we conclude that with probability at least 1 − 3 δ , after R rounds, we should have got into situation ⋆ (1) and not ⋆ (4) and not exceeding ℵ + 2 l , which means that X i =
Revised: But picking η small enough with respect to log( δ 1 ) and other parameters, we conclude that with probability at least 1 − 3 δ , after R rounds, we should have gotten into situation ⋆ (1) and not ⋆ (4) and not exceeding ℵ + 2 l , which means that X i =

Original: L Π ( w τ i ) has gotten under the threshold ℵ ℓ .
Revised: L Π ( w τ i ) has gotten under the threshold ℵ ℓ .

Original: Note that as soon as that happens, we terminate the algorithm.
Revised: Note that as soon as that happens, we terminate the algorithm.

Original: We elaborate on this more in Appendix A.10.
Revised: We elaborate on this more in Appendix A.10.

Original: A. G AUSSIAN S MOOTHING
Revised: A. G AUSSIAN S MOOTHING

Original: In this section, we describe our smoothing scheme and the approximation that it provides which enables us to keep the signs from the case η = 0 .
Revised: In this section, we describe our smoothing scheme and the approximation that it provides which enables us to keep the signs from the case η = 0 .

Original: Recall that we use Gaussian smoothing matrices
Revised: Recall that we use Gaussian smoothing matrices

Original: V ρj,k ∼ N (0 , β 21 /m 1 ) and W ρj,k ∼ N (0 , β 22 /m 2 ) .
Revised: V ρj,k ∼ N (0 , β 21 /m 1 ) and W ρj,k ∼ N (0 , β 22 /m 2 ) .

Original: Here, we will particularly specify lower bounds for β 1 and β 2 in order for our sign approximation to be precise.
Revised: Here, we will particularly specify lower bounds for β 1 and β 2 in order for our sign approximation to be precise.

Original: On the other hand, we normally prefer the smoothing noise to be as low as possible so the primary and smoothed functions are close, so we set β 1 , β 2 equal to their lower bounds, and use this setting in the other parts.
Revised: On the other hand, we normally prefer the smoothing noise to be as low as possible so the primary and smoothed functions are close, so we set β 1 , β 2 equal to their lower bounds, and use this setting in the other parts.

Original: To begin ﬁx one of the inputs x i .
Revised: To begin ﬁx one of the inputs x i .

Original: In order to reduce and simplify the amount of notations, we refer to the sign pattern matrix (diagonal sign matrix) of both the ﬁrst and second layers by D with the appropriate indices.
Revised: In order to reduce and simplify the amount of notations, we refer to the sign pattern matrix (diagonal sign matrix) of both the ﬁrst and second layers by D with the appropriate indices.

Original: More speciﬁcally, for the ﬁrst layer, we refer to Sgn ( W (0) + W ′ + W ρ )
Revised: More speciﬁcally, for the ﬁrst layer, we refer to Sgn ( W (0) + W ′ + W ρ )

Original: x i by
Revised: x i by

Original: D ′ ,ρ and Sgn ( W (0) + (1 − η/ 2) W ′
Revised: D ′ ,ρ and Sgn ( W (0) + (1 − η/ 2) W ′

Original: + W ρ + √ ηW ∗ )
Revised: + W ρ + √ ηW ∗ )

Original: x i by D ′ ,ρ .
Revised: x i by D ′ ,ρ .

Original: Similarly, for the second layer, of course depending on the input vector, we refer to the sign matrix with respect to the matrices
Revised: Similarly, for the second layer, of course depending on the input vector, we refer to the sign matrix with respect to the matrices

Original: V (0) + V ′
Revised: V (0) + V ′

Original: + V ρ and V (0)
Revised: + V ρ and V (0)

Original: + (1 − η/ 2) V ′
Revised: + (1 − η/ 2) V ′

Original: + V ρ + √ ηV ∗ by by D ′ .ρ
Revised: + V ρ + √ ηV ∗ by by D ′ .ρ

Original: and D ′ ,ρ,η , respectively.
Revised: and D ′ ,ρ,η , respectively.

Original: We introduce two new notations as well for the output of the ﬁrst layer with respect to different matrix and sign patterns:
Revised: We introduce two new notations as well for the output of the ﬁrst layer with respect to different matrix and sign patterns:

Original: [Equation]
Revised: [Equation]

Original: For further brevity, we sometimes refer to x ′ (2) by x ′ .
Revised: For further brevity, we sometimes refer to x ′ (2) by x ′ .

Original: Now we are ready to mention our approximation theorem regarding the smoothing and the sign changes.
Revised: Now we are ready to mention our approximation theorem regarding the smoothing and the sign changes.

Original: Lemma 

Revised: Lemma

Original: Under the conditions κ 1 √ m 3 ≳
Revised: Under the conditions κ 1 √ m 3 ≳

Original: C 1 + β 1 √ m 3 and m 2 ≥ m 3 log( m 2 ) [Equation] then for every i ∈ [ n ] :[Equation]
Revised: C 1 + β 1 √ m 3 and m 2 ≥ m 3 log( m 2 ) , then for every i ∈ [ n ] :

Original: Proof of Lemma

Original: We can bound the Left hand side above as

Original: [Equation]

Original: − η ) W ′ + W ρ + √ ηW ∗ ) x

Original: i − a

Original: [Equation]

Original: D ′ ,ρ

Original: [Equation]

Original: V (0)

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: + V

Original: [Equation]

Original: ηV ∗

Original: [Equation]

Original: [Equation]

Original: ρ

Original: [Equation]

Original: η

Original: [Equation]
Revised: [Equation]

Original: W

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: η ) W ′ + W ρ

Original: [Equation]

Original: √ ηW ∗ )

Original: x

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: + [Equation] E W ρ ,V ρ a T D ′ ,ρ ( V (0) + (
Revised: h

Original: [Equation]
Revised: [Equation]

Original: η ) V ′ + V ρ + √ ηV ∗ ) W s D ′ ,ρ,η ( W (0) +
Revised: Proof of Lemma

Original: [Equation] [Equation] η ) W ′ + W ρ + √ ηW ∗ ) x i
Revised: We can bound the Left hand side above as

Original: [Equation]
Revised: [Equation]

Original: We bound A 1 and A 2 separately.
Revised: We bound A 1 and A 2 separately.

Original: First, we start with A 1 .
Revised: First, we start with A 1 .

Original: Let ˆ P i be the set of indices j for which 1 {| ( V (0)
Revised: Let ˆ P i be the set of indices j for which 1 {| ( V (0)

Original: j 
Revised: j + V

Original: [Equation]
Revised: [Equation]

Original: } happens.
Revised: x ′ | ≤ R ∗ κ 2 ∥ x ′ ∥} happens.

Original: Then, from
Revised: Then, from

Original: Lemma 31, we have | ˆ P i | ≲ R ∗ m 2 .
Revised: Lemma 31, we have | ˆ P i | ≲ R ∗ m 2 .

Original: Now for j ∈ [ m 2 ] , we write 1 { sign change in the j th neuron } × | amount of change |
Revised: Now for j ∈ [ m 2 ] , we write 1 { sign change in the j th neuron } × | amount of change |

Original: [Equation]
Revised: [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Moreover, note that
Revised: Moreover, note that

Original: [Equation]
Revised: [Equation]

Original: Also [Equation] because ∥ V ′ j ∥ ≤ ∥ V ′ ∥ ≤ 2 C 2 plus using Equation (129)
Revised: Also, because ∥ V ′ j ∥ ≤ ∥ V ′ ∥ ≤ 2 C 2 plus using Equation (128)

Original: [Equation]
Revised: [Equation]

Original: we can further upper bound the above indicator as:
Revised: we can further upper bound the above indicator as:

Original: [Equation]
Revised: [Equation]

Original: Taking √ η ≤ ϱ/ (2 C 2 √ m 2 ) , we can further upper bound as
Revised: Taking √ η ≤ ϱ/ (2 C 2 √ m 2 ) [Equation] we can further upper bound as

Original: [Equation]
Revised: [Equation]

Original: Therefore, conditioned on x ′ :
Revised: Therefore, conditioned on x ′ :

Original: [Equation]
Revised: [Equation]

Original:  change in the j th neuron } × [Equation] amount of change 
Revised: sign change in the j th neuron } × | amount of change | | x ′ ] ≤

Original: [Equation] ] ≤ [Equation] Now notice that for j / ∈ ˆ Pi , we have | − V
Revised: P

Original: [Equation]
Revised: [Equation]

Original: j [Equation] [Equation] V ′ j
Revised: ηϱ

Original: [Equation]
Revised: [Equation]

Original: ≥ R [Equation] ∥ x
Revised: Now notice that for j / ∈ ˆ P i , we have

Original: [Equation]
Revised: [Equation]

Original: ∥ . Also, note that the variable V ρj x ′ is gaussian with variance ∥ x ′ ∥ β 2 / √ m 

Revised: Also, note that the variable V ρj x ′ is gaussian with variance ∥ x ′ ∥ β 2 / √ m

Original: Therefore, conditioned on x ′ , for j / ∈ ˆ P i , we have (note that x ′ does not depend on the randomness of V ρ ):P [Equation] [Equation]
Revised: Therefore, conditioned on x ′ , for j / ∈ ˆ P i , we have (note that x ′ does not depend on the randomness of V ρ ):

Original: [Equation]

Original: min

Original: [Equation]
Revised: [Equation]

Original: ∥ x ′

Original: [Equation]

Original: ϕ (

Original: [Equation]

Original: x i

Original: [Equation]

Original: ∥ , ∥ x

Original: [Equation]

Original: ∥} / √ m 2 , − V

Original: [Equation]

Original: j x ′ − V

Original: j x ′ ))

Original: ≲ exp { min {| − V (0)

Original: j x ′ − V

Original: j x ′ − 2 √ ηϱ 2 ∥ x ′ ∥ / √ m 2 | , | − V (0)

Original: j x ′ − V

Original: j x ′ |} / ( √ 2 ∥ x ′ ∥

Original: β 2 / √ m 2 ) }× (

Original: [Equation]
Revised: [Equation]

Original: This equation follows from the fact that
Revised: This equation follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, note that for √ η ≲ C2 / 3 2 ( √ m 2 κ 2 )
Revised: On the other hand, note that for √ η ≲ C

Original: [Equation]
Revised: [Equation]

Original: 3 ϱwe have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: On the other side
Revised: On the other side

Original: [Equation]
Revised: [Equation]

Original: for j ∈ ˆ P
Revised: for j ∈ ˆ P

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: Therefore

Original: [Equation] overall using the fact that ∥ V ∗ j ∥ ≤ ϱ 2 / √ m [Equation] we can write
Revised: Therefore, overall using the fact that ∥ V ∗ j ∥ ≤ ϱ 2 / √ m 2 , we can write

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Next, we bound A 2 .
Revised: Next [Equation] we bound A 2 .

Original: First we bound E W ρ ∥ x ′ (1) − x ′ (2) ∥ .
Revised: First we bound E W ρ ∥ x ′ (1) − x ′ (2) ∥ .

Original: Recalling the setting c 2 = 2 √ nm 3 C 1 / √ λand the deﬁnition of in P from Lemma 1, we obtain that for j / ∈ P , we have for all i ∈ [ n ] :
Revised: Recalling the setting c 2 = 2 √ nm 3 C 1 / √ λand the deﬁnition of in P from Lemma 1, we obtain that for j / ∈ P , we have for all i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: which means for j / ∈ P :
Revised: which means for j / ∈ P :

Original: [Equation]
Revised: [Equation]

Original: Also, we have
Revised: Also, we have

Original: [Equation]
Revised: [Equation]

Original: Now using Equation (105) in Lemma 5, we can write for every i ∈ [ n ] : val j := 1 { sign change in the j th neuron } × (cid:12)(cid:12)(cid:12) amount of change(cid:12)(cid:12)(cid:12)
Revised: Now using Equation (104) in Lemma 5, we can write for every i ∈ [ n ] : val j := 1 { sign change in the j th neuron } × [Equation] amount of change

Original: [Equation]
Revised: [Equation]

Original: Using the fact that ∥ W ′ j ∥ ≤ ∥ W ′ ∥
Revised: Using the fact that ∥ W ′ j ∥ ≤ ∥ W ′ ∥

Original: F ≤ C1 , and Equation (105) ( ∥ W ∗ j ∥ ≤ ϱ q mm 1 ) and picking √ η ≤ ϱ √ m 3 C
Revised: F ≤ C

Original: [Equation]
Revised: [Equation]

Original:  √ m 1 , we obtain
Revised: and Equation (104) ( ∥ W ∗ j ∥ ≤ ϱ q mm 1 ) and picking √ η ≤ ϱ √ m 3 C 1 √ m 1 , we obtain

Original: [Equation]
Revised: [Equation]

Original: Now for j / ∈ P , because W ρj x i is Gaussian with std β 1 √ m 1 :
Revised: Now for j / ∈ P , because W ρj x i is Gaussian with std β 1 √ m 1 :

Original: [Equation]
Revised: [Equation]

Original: Now from Equation (266) and by picking √ η ≲ c 2 ϱ √ m 3 so that
Revised: Now from Equation (265) and by picking √ η ≲ c 2 ϱ √ m 3 so that

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for j ∈ P we have
Revised: On the other hand, for j ∈ P we have

Revised: E val

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne the following random variable with respect to the randomness of W ρ :V al
Revised: Now deﬁne the following random variable with respect to the randomness of W ρ :

Original: [Equation]
Revised: [Equation]

Original: 1 { sign change in the j th neuron } × | amount of change | then for every k ∈ [ m 3 ] , we have
Revised: { sign change in the j th neuron } × | amount of change | then for every k ∈ [ m 3 ] , we have

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: But Combining Equations (268) and (269):
Revised: But Combining Equations (267) and (268):

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Now we can write(cid:12)(cid:12)(cid:12)
Revised: Now we can write

Original: [Equation]
Revised: [Equation]

Original: | [Equation] Now by Equation (127) in Lemma 10 (i.e. ∥ V ∗ ∥ F ≲ √ ζ 2 ) and the fact that ∥ V ′ ∥
Revised: Now by Equation (126) in Lemma 10 (i.e. ∥ V ∗ ∥ F ≲ √ ζ 2 ) and the fact that ∥ V ′ ∥

Original: F ≤ C 2 , and by taking
Revised: F ≤ C 2 , and by taking

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: so we can bound the above as
Revised: so we can bound the above as

Original: [Equation]
Revised: [Equation]

Original: Furthermore, using Lemma 32 and noting the fact that the entries of V (0) are normal with standard deviation κ 2 , we get with high probability over the randomness of V (0) :
Revised: Furthermore, using Lemma 32 and noting the fact that the entries of V (0) are normal with standard deviation κ 2 , we get with high probability over the randomness of V (0) :

Original: ≲
Revised: ≲

Original: [Equation]
Revised: [Equation]

Original: Now note that V ρj ( x ′ (2) − x ′ (1) ) is normal with standard deviation β 2 √ m 2 ∥ x ′ (2) − x ′ (1) ∥ .
Revised: Now note that V ρj ( x ′ (2) − x ′ (1) ) is normal with standard deviation β 2 √ m 2 ∥ x ′ (2) − x ′ (1) ∥ .

Original: Hence, taking expectation with respect to V ρ :
Revised: Hence, taking expectation with respect to V ρ :

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: Combining Equation (265) and (272) and applying it to Equation (261) implies with high probability over the random initialization:
Revised: Combining Equation (264) and (271) and applying it to Equation (260) implies with high probability over the random initialization:

Original: [Equation]
Revised: [Equation]

Original: Now notice that under E c , using the assumption κ 1 √ m 3 ≳ C 1 + √ m 3 β 1 and Lemma 33 we have
Revised: Now notice that under E c , using the assumption κ 1 √ m 3 ≳ C 1 + √ m 3 β 1 and Lemma 33 we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (273) which implies:
Revised: 272) which implies:

Original: [Equation]
Revised: [Equation]

Original: ( x i ) ∥ 2 / ∥ x ′ ∥ , ∥ x ′ − ϕ

Original: (0) ( x i ) ∥}

Original: ≤ E W ρ 1 { E c }∥ x

Original: ′ − ϕ (0) ( x i ) ∥ 2 / ∥ x ′ ∥

Original: + 1 { E }∥ x ′ − ϕ (0) ( x i )

Original: ∥ ≲ ( C 1 + √ m 3 β 1 ) 2 / ( κ 1 √ m 3 ) + E W ρ 1 { E }∥ x ′ − ϕ

Original: (0) ( x i ) ∥

Original: ≲ ( C 1 + √ m 3 β 1 ) 2 / ( κ 1 √ m 3 ) + h √ m 3 m 1 β 1 + Ci exp {− C 21 / (8 m 3 β 21 ) } .

Original: Substituting this above and further applying the result of Lemma 33 and Equation (270) and the assumption that m 2 ≥ m 3 log( m 2 ) :[Equation] i
Revised: Substituting this above and further applying the result of Lemma 33 and Equation (269) and the assumption that m 2 ≥ m 3 log( m 2 ) :

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: A.20. S ETTING β 1 AND β
Revised: A.20. S ETTING β 1 AND β

Original: As we mentioned, to minimize the amount of deviation of the smoothed function compared to the original one, we prefer to choose β 1 , β 2 as small as possible.
Revised: As we mentioned, to minimize the amount of deviation of the smoothed function compared to the original one, we prefer to choose β 1 , β 2 as small as possible.

Original: (The beneﬁt of such choice, indeed, can be observed more explicitly in other parts of the proof, e.g. Appendix A.14.)
Revised: (The beneﬁt of such choice, indeed, can be observed more explicitly in other parts of the proof, e.g. Appendix A.14.)

Original: Observing the bound in
Revised: Observing the bound in

Original: Equation ( 260) and noting that we can easily make the exponential terms orders of magnitude smaller than the poly terms, it is easy to ﬁnd the following optimal setting for the smoothing parameters:
Revised: Equation (259) and noting that we can easily make the exponential terms orders of magnitude smaller than the poly terms, it is easy to ﬁnd the following optimal setting for the smoothing parameters:

Original: [Equation]
Revised: [Equation]

Original: Using this setting, we still can make ℜ 8 arbitrarily small.
Revised: Using this setting, we still can make ℜ 8 arbitrarily small.

Original: Here, we remind the reader that O p only cares about the non-logarithmic dependencies on the overparameterization, i.e. m 1 , m 2 , m 3 , κ 1 , κ 2 .
Revised: Here, we remind the reader that O p only cares about the non-logarithmic dependencies on the overparameterization, i.e. m 1 , m 2 , m 3 , κ 1 , κ 2 .

Original: A. B ASIC T OOLS
Revised: A. B ASIC T OOLS

Original: In this section, we introduce and prove some lemmas that we use in our analysis as basic tools.
Revised: In this section, we introduce and prove some lemmas that we use in our analysis as basic tools.

Original: Lemma 

Revised: Lemma

Original: Suppose V (0) ∈
Revised: Suppose V (0) ∈

Original: R m 2 × m 3 has standard normal entries and a is a random sign vector.
Revised: R m 2 × m 3 has standard normal entries and a is a random sign vector.

Original: Suppose theta > 1 , R < 1 are given thresholds, such that
Revised: Suppose theta > 1 , R < 1 are given thresholds, such that

Original: [Equation]
Revised: [Equation]

Original: Then, for the following quantities:
Revised: Then, for the following quantities:

Original: [Equation]
Revised: [Equation]

Original: with high probability we have
Revised: with high probability we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Suppose B 1 ( ϵ ) is a cover for the Euclidean ball in R m 3 with precision ϵ .
Revised: Suppose B 1 ( ϵ ) is a cover for the Euclidean ball in R m 3 with precision ϵ .

Original: We know | B 1 ( ϵ ) | ≲ (1 /ϵ ) m 3 .
Revised: We know | B 1 ( ϵ ) | ≲ [Equation] 1 /ϵ ) m 3 .

Original: Now for a ﬁxed ∥ x ∥ = 1 , we have
Revised: Now for a ﬁxed ∥ x ∥ = 1 , we have

Original: P

Original: [Equation]

Original: [Equation]

Original: ≲

Original: [Equation]
Revised: [Equation]

Original: Therefore, using Bernstein, with high probability we have #
Revised: Therefore, using Bernstein, with high probability we have #

Original: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: Hence, using union bound, we have with high probability sup
Revised: Hence, using union bound, we have with high probability sup

Original: [Equation]
Revised: [Equation]

Original: p
Revised: m

Original: [Equation]
Revised: [Equation]

Original:  log
Revised: Rm 3 log

Original: [Equation]
Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original:  By pickingϵ ≲ R/ (
Revised: 1 /ϵ ) . By picking

Original: [Equation]
Revised: [Equation]

Original: The assumption implies m 2 R ≥ m 3 log(1 /ϵ ) , which implies
Revised: The assumption implies m 2 R ≥ m 3 log(1 /ϵ ) , which implies

Original: [Equation]
Revised: [Equation]

Original: On the other hand, note that with high probability we have
Revised: On the other hand, note that with high probability we have

Original: [Equation]
Revised: [Equation]

Original: Now for ∥ x ′ ∥ = 1 which is not in the cover, if x is the closest point to it in the cover, i.e. x ∈ B 1 ( ϵ ) and ∥ x − x ′ ∥ ≤ ϵ , then for every j ∈ [ m 2 ] we have
Revised: Now for ∥ x ′ ∥ = 1 which is not in the cover, if x is the closest point to it in the cover, i.e. x ∈ B 1 ( ϵ ) and ∥ x − x ′ ∥ ≤ ϵ , then for every j ∈ [ m 2 ] we have

Original: [Equation]
Revised: [Equation]

Original: by picking a small enough constant.
Revised: by picking a small enough constant.

Original: Therefore, for a j that | V (0)
Revised: Therefore, for a j that | V (0)

Original: j x | ≥ 2 R , then
Revised: j x | ≥ 2 R , then

Original: [Equation]
Revised: [Equation]

Original: Therefore, we get that with high probability, for every ∥ x ′ ∥
Revised: Therefore, we get that with high probability, for every ∥ x ′ ∥

Original: = 1 :
Revised: = 1 :

Original: [Equation]
Revised: [Equation]

Original: For the second part, note that for ∥ x ∥ = 1 , by the tail bound for normal vars:
Revised: For the second part, note that for ∥ x ∥ = 1 , by the tail bound for normal vars:

Original: [Equation]
Revised: [Equation]

Original: Hence, again using Bernstein
Revised: Hence, again using Bernstein

Original: [Equation]
Revised: [Equation]

Original: we have with high probability
Revised: we have with high probability

Original: [Equation]
Revised: [Equation]

Original: By picking
Revised: By picking

Original: [Equation]
Revised: [Equation]

Original: and using the assumption m 2 e − θ 2 / 8 ≲ m 3 , all terms are dominated by the third term so we can bound the above as
Revised: and using the assumption m 2 e − θ 2 / 8 ≲ m 3 , all terms are dominated by the third term so we can bound the above as

Original: [Equation]
Revised: [Equation]

Original: Now for ∥ x ′ ∥
Revised: Now for ∥ x ′ ∥

Original: = 1 not in the cover
Revised: = 1 not in the cover

Original: [Equation]
Revised: [Equation]

Original: for the new ϵ we can write
Revised: for the new ϵ we can write

Original: [Equation]
Revised: [Equation]

Original: Hence, with high prob.
Revised: Hence, with high prob.

Original: [Equation]
Revised: [Equation]

Original: log [Equation]
Revised: 3 ) + log(log( m 2 ))) .

Original: Lemma 30 For x ∈ R d and W (0) ∈
Revised: Lemma 30 For x ∈ R d and W (0) ∈

Original: R m × d which has standard normal entries (and a is a random sign vector), we have with high probability:
Revised: R m × d which has standard normal entries (and a is a random sign vector), we have with high probability:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: For the ﬁrst part, we ﬁrst compute an upper bound on
Revised: For the ﬁrst part, we ﬁrst compute an upper bound on

Original: [Equation]
Revised: [Equation]

Original: To do so, we use Dudley’s chaining.
Revised: To do so, we use Dudley’s chaining.

Original: Note that the for x 1 , x 2 ∈ R d , the variable σ ( W (0) j x 1 ) − σ ( W (0) j x 2 ) is subGaussian with parameter ∥ x 1 − x 2 ∥ , so the variable f ( x 1 ) − f ( x 2 ) is also subGaussian with parameter ∥ x 1 − x 2 ∥ .
Revised: Note that the for x 1 , x 2 ∈ R d , the variable σ ( W (0) j x 1 ) − σ ( W (0) j x 2 ) is subGaussian with parameter ∥ x 1 − x 2 ∥ , so the variable f ( x 1 ) − f ( x 2 ) is also subGaussian with parameter ∥ x 1 − x 2 ∥ .

Original: Hence, by Dudley’s integral:
Revised: Hence, by Dudley’s integral:

Original: [Equation]
Revised: [Equation]

Original: Now for a ﬁxed x , note that
Revised: Now for a ﬁxed x , note that

Original: [Equation]
Revised: [Equation]

Original: Hence, the function f ( x ) is 1-lipchitz with respect to W and l 2 norm, so is the function sup f ( x ) .
Revised: Hence, the function f ( x ) is 1-lipchitz with respect to W and l 2 norm, so is the function sup f ( x ) .

Original: Hence, by Gaussian concentration, sup f ( x ) is 1 -subGaussian around its mean, so we ﬁnally get with high probability
Revised: Hence, by Gaussian concentration, sup f ( x ) is 1 -subGaussian around its mean, so we ﬁnally get with high probability

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: For
Revised: For

Original: [Equation]
Revised: [Equation]

Original: we have with high probability over the randomness of V (0) :
Revised: we have with high probability over the randomness of V (0) :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Note that obviously the condition of Lemma 29 is satisﬁed with this choice of R = R ∗ .
Revised: Note that obviously the condition of Lemma 29 is satisﬁed with this choice of R = R ∗ .

Original: Therefore, with high probability we have for an arbitrary x ′ :
Revised: Therefore, with high probability we have for an arbitrary x ′ :

Original: [Equation]
Revised: [Equation]

Original: On the other hand, note that for j ∈ [ m 2 ] such that | V ′
Revised: On the other hand, note that for j ∈ [ m 2 ] such that | V ′

Original: j x ′ | ≥
Revised: j x ′ | ≥

Original: Rκ 2 ∥ x ′ ∥ , we have
Revised: Rκ 2 ∥ x ′ ∥ , we have

Original: [Equation]
Revised: [Equation]

Original: which implies 
Revised: which implies ∥

Original: [Equation]
Revised: [Equation]

Original: Therefore, there are at most C 22 R 2 κ 22 .
Revised: Therefore, there are at most C 22 R 2 κ 22 .

Original: Therefore, setting aside m 2 R + C 22 R 2 κ 22 of j ’s, for the rest we have
Revised: Therefore, setting aside m 2 R + C 22 R 2 κ 22 of j ’s, for the rest we have

Original: [Equation]
Revised: [Equation]

Original: Setting R ∗ as deﬁned above balances the terms m 2 R and C 22 R 2 κ 22 , which completes the proof.
Revised: Setting R ∗ as deﬁned above balances the terms m 2 R and C 22 R 2 κ [Equation] which completes the proof.

Original: Lemma 32 If V (0) ∈ R m 2 × m 3 is a matrix with standard normal entries, then with high probability
Revised: Lemma 32 If V (0) ∈ R m 2 × m 3 is a matrix with standard normal entries, then with high probability

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Let B 1 ( ϵ ) be a cover for the unit Euclidean ball with precision ϵ , for which we have | B 1 ( ϵ ) | ≲ ( 1 ϵ ) m 3 .
Revised: Let B 1 ( ϵ ) be a cover for the unit Euclidean ball with precision ϵ , for which we have | B 1 ( ϵ ) | ≲ ( 1 ϵ ) m 3 .

Original: Now for a ﬁxed x ∈ B 1 ( ϵ ) , note that because V (0) j x is a standard normal variable, the random variable | V (0) j x ′ |− E | V (0) j x ′ | is O (1) -subGaussian, which means
Revised: Now for a ﬁxed x ∈ B 1 ( ϵ ) , note that because V (0) j x is a standard normal variable, the random variable | V (0) j x ′ |− E | V (0) j x ′ | is O (1) -subGaussian, which means

Original: [Equation]
Revised: [Equation]

Original: is also O (1) − subGaussian.
Revised: is also O (1) − subGaussian.

Original: Now from the tail of maximum of subGaussian variables:
Revised: Now from the tail of maximum of subGaussian variables:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, note that E | V (0) j x ′ | )
Revised: On the other hand, note that E | V (0) j x ′ | )

Original: =
Revised: =

Original: O (1) , which implies w.h.p:
Revised: O (1) , which implies w.h.p:

Original: [Equation]
Revised: [Equation]

Original: Moreover, note that again by the tail of subGaussian variables, we have w.h.p:
Revised: Moreover, note that again by the tail of subGaussian variables, we have w.h.p:

Original: [Equation]
Revised: [Equation]

Original: which implies with high prob for every j ∈ [ m 2 ] :
Revised: which implies with high prob for every j ∈ [ m 2 ] :

Original: [Equation]
Revised: [Equation]

Original: Now by picking
Revised: Now by picking

Original: [Equation]
Revised: [Equation]

Original: we get with high probability
Revised: we get with high probability

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for an arbitrary x ′ with ∥ x ′ ∥ = 1 , if x ∈ B 1 ( ϵ ) is the representative of x ′ , we have by deﬁnition ∥ x ′ − x ∥ ≤ ϵ , which combined with (A.21) implies
Revised: On the other hand, for an arbitrary x ′ with ∥ x ′ ∥ = 1 , if x ∈ B 1 ( ϵ ) is the representative of x ′ , we have by deﬁnition ∥ x ′ − x ∥ ≤ ϵ , which combined with (A.21) implies

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Combining Equations (275) and (276), we conclude the result [Equation]
Revised: Combining Equations (274) and (275), we conclude the result.

Original: A.21. D EFINING THE RARE EVENTS E j
Revised: A.21. D EFINING THE RARE EVENTS E j

Original: Lemma 33 For x ′ (2) deﬁned in Equation (259) we have
Revised: Lemma 33 For x ′ (2) deﬁned in Equation (258) we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, for the events
Revised: Moreover, for the events

Original: [Equation]
Revised: [Equation]

Original: we have under E c :
Revised: we have under E c :

Original: [Equation]
Revised: [Equation]

Original: Furthermore, E happens rarely:
Revised: Furthermore, E happens rarely:

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we have the following almost surely bound:
Revised: we have the following almost surely bound:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: We start by writing
Revised: We start by writing

Original: [Equation]
Revised: [Equation]

Original: Now notice that by Lemma 1, we know for every j / ∈ ˜ P :
Revised: Now notice that by Lemma 1, we know for every j / ∈ ˜ P :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: In addition, by Equations in (105) from Lemma 5, for every j ∈ [ m 1 ] :
Revised: In addition, by Equations in (104) from Lemma 5, for every j ∈ [ m 1 ] :

Original: [Equation]
Revised: [Equation]

Original: so by picking
Revised: so by picking

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Combining this with Equations in (279), we see that the signs of ( W (0)
Revised: Combining this with Equations in (278), we see that the signs of ( W (0)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: and W (0)
Revised: and W (0)

Original: j x i are the same for j / ∈ ˜ P .
Revised: j x i are the same for j / ∈ ˜ P .

Original: Moreover, the matrix (1 − η ) W ′ + √ ηW ∗ satisﬁes
Revised: Moreover, the matrix (1 − η ) W ′ + √ ηW ∗ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: by picking √ η ≤ C 1 / √ ζ 2 .
Revised: by picking √ η ≤ C 1 / √ ζ 2 .

Original: Hence, the conditions of Lemma 6 are satisﬁed and we get:
Revised: Hence, the conditions of Lemma 6 are satisﬁed and we get:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Combining Equations (277) and (281):
Revised: Combining Equations (276) and (280):

Original: [Equation]
Revised: [Equation]

Original: In exactly similar fashion, one can derive
Revised: In exactly similar fashion [Equation] one can derive

Original: [Equation]
Revised: [Equation]

Original: Now ﬁrst of all, note
Revised: Now ﬁrst of all [Equation] note

Original: [Equation]
Revised: [Equation]

Original: which proves the ﬁrst part of the claims.
Revised: which proves the ﬁrst part of the claims.

Original: For the second part, note that by the Gaussian tail bound
Revised: For the second part, note that by the Gaussian tail bound

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: Plugging this into (282) ﬁnishes the proof.
Revised: Plugging this into (281) ﬁnishes the proof.

Original: Also, under E c by Equation (282) we have
Revised: Also, under E c by Equation (281) we have

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: exploiting Equation (283):
Revised: exploiting Equation (282):

Original: [Equation]
Revised: [Equation]

Original: A.21. B OUNDING THE VALUE OF f ′
Revised: A.21. B OUNDING THE VALUE OF f ′

Original: The following Lemma provides a reasonable bound on the value of the smoothed function.
Revised: The following Lemma provides a reasonable bound on the value of the smoothed function.

Original: Lemma 34 We have the following general bound on the values of the smoothed function: With high probability over the initialization, for ∥ W ′ ∥ ≤ C 1 ,
Revised: Lemma 34 We have the following general bound on the values of the smoothed function: With high probability over the initialization, for ∥ W ′ ∥ ≤ C 1 ,

Original: ∥ V ′ ∥
Revised: ∥ V ′ ∥

Original: ≤ C 2 and ∀ i ∈ [ n ] (having small enough choices of β 1 , β 2 described in Appendix A.20):
Revised: ≤ C 2 and ∀ i ∈ [ n ] (having small enough choices of β 1 , β 2 described in Appendix A.20):

Original: [Equation]
Revised: [Equation]

Original: which is O ( C 1 C 2 ) for large enough overparameterization as described in AppendixA.3.
Revised: which is O ( C 1 C 2 ) for large enough overparameterization as described in AppendixA.3.

Original: Moreover, we have the following almost surely bound (with respect to the randomness of W ρ and V ρ ):
Revised: Moreover, we have the following almost surely bound (with respect to the randomness of W ρ and V ρ ):

Original: [Equation]
Revised: [Equation]

Original: Notably, with slightly higher overparameterization
Revised: Notably, with slightly higher overparameterization

Original: [Equation]
Revised: [Equation]

Original: the high probability bound in (34) holds even if we take supremum over x .
Revised: the high probability bound in (34) holds even if we take supremum over x .

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Using Lemmas 30 and 33 and using the fact that ϕ (0) ( x i ) is orthogonal to the rows of V ′ (recall x ′ i =
Revised: Using Lemmas 30 and 33 and using the fact that ϕ (0) ( x i ) is orthogonal to the rows of V ′ (recall x ′ i =

Original: ϕ (0) ( x i ) + ϕ (2) ( x i ) ):
Revised: ϕ (0) ( x i ) + ϕ (2) ( x i ) ):

Original: [Equation]
Revised: [Equation]

Original: Note that above, if we apply the stronger worst-case norm bound of the ﬁrst layer’s output presented in Lemma 46, we would get sup x, ∥ x ∥ =1 | f W ′
Revised: Note that above, if we apply the stronger worst-case norm bound of the ﬁrst layer’s output presented in Lemma 46, we would get sup x, ∥ x ∥ =1 | f W ′

Original: + W
Revised: + W

Original: ρ ,V ′ + V ρ ( x ) | is bounded by the RHS, which in turn proves a stronger uniform bound on f ′ .
Revised: ρ ,V ′ + V ρ ( x ) | is bounded by the RHS, which in turn proves a stronger uniform bound on f ′ .

Original: Similarly, this time by taking expectation with respect to W ρ and V ρ :
Revised: Similarly, this time by taking expectation with respect to W ρ and V ρ :

Original: [Equation]
Revised: [Equation]

Original: Corollary 8.

Revised: Corollary 8.

Original: If we set C 1 = C 2 = 0 above
Revised: If we set C 1 = C 2 = 0 above

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: the point being these terms go to zero by an order of O (( √ m 2 κ 2 ) − 23 ) .
Revised: the point being these terms go to zero by an order of O (( √ m 2 κ 2 ) − 23 ) .

Original: Therefore, taking ( 
Revised: Therefore, taking ( √

Original: A.21. B OUNDING THE DIFFERENCE BETWEEN O RIGINAL AND S MOOTHED F UNCTIONS
Revised: A.21. B OUNDING THE DIFFERENCE BETWEEN O RIGINAL AND S MOOTHED F UNCTIONS

Original: The following Lemma bounds the difference between the smoothed function and original function of the network.
Revised: The following Lemma bounds the difference between the smoothed function and original function of the network.

Original: Lemma 35 Bound on the smoothing change under the assumption m 2 ≥ m 3 log( m 2 ) : with high probability over the initialization, for any ( W ′ , V ′ ) with ∥ W ′ ∥ ≤ C 1 ,
Revised: Lemma 35 Bound on the smoothing change under the assumption m 2 ≥ m 3 log( m 2 ) : with high probability over the initialization, for any ( W ′ , V ′ ) with ∥ W ′ ∥ ≤ C 1 ,

Original: ∥ V ′ ∥ ≤ C 2 :
Revised: ∥ V ′ ∥ ≤ C 2 :

Original: [Equation]
Revised: [Equation]

Original: f W ′

Original: [Equation]
Revised: [Equation]

Original: V ′ ( x i )

Original: [Equation]
Revised: [Equation]

Original: f ′ W ′ ,V ′ ( xi ) |
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: σ means we apply Relu activation to the vector in front of it (entrywise):
Revised: σ means we apply Relu activation to the vector in front of it (entrywise):

Original: [Equation]
Revised: [Equation]

Original: Now for the ﬁrst term above, using the previous notation of x ′ i representing the output of the ﬁrst layer and using Lemma 33:
Revised: Now for the ﬁrst term above, using the previous notation of x ′ i representing the output of the ﬁrst layer and using Lemma 33:

Original: [Equation]
Revised: [Equation]

Original: For the second term
Revised: For the second term

Original: [Equation]
Revised: [Equation]

Original: by starting off with a simple triangle inequality:
Revised: by starting off with a simple triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: ( x i ))
Revised: x i ))

Original: (cid:12)(cid:12)(cid:12) ≤ E W ρ ,V ρ 1 √ mm
Revised: (cid:12)(cid:12)(cid:12) ≤ E W ρ ,

Original: [Equation]

Original:  X j = (cid:12)(cid:12)(cid:12) ( V (0) j + V ′ j )( x ′ i − ϕ (0) ( x i ) − ϕ
Revised: V ρ 1 √ mm 2 X j = (cid:12)(cid:12)(cid:12) ( V (0) j + V ′ j ) ( x ′ i − ϕ (0) ( x i ) − ϕ

Original: (2) ̸ ρ ( x i ))
Revised: (2) ̸ ρ ( x i ))

Original: (cid:12)(cid:12)(cid:12) ≤ E W ρ ,V ρ 1 √ [Equation] (cid:12)(cid:12)(cid:12) V (0)
Revised: (cid:12)(cid:12)(cid:12) ≤ E W ρ ,V ρ 1 √ mm 2 X j = (cid:12)(cid:12)(cid:12) V (0)

Original: j ( x ′ i [Equation] (0) ( x i ) − 
Revised: j ( x ′ i − ϕ (0) ( x i ) − ϕ

Original: [Equation]

Original: (2) ̸ ρ ( x i ))
Revised: (2) ̸ ρ ( x i ))

Original: (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) V ′ j
Revised: (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12) V ′ j

Original: ( x ′ i − ϕ (0) ( x i ) − ϕ
Revised: ( x ′ i − ϕ (0) ( x i ) − ϕ

Original: (2) ̸ ρ ( x i )) | (cid:12)(cid:12)(cid:12) ≤ 
Revised: (2) ̸ ρ ( x i )) | (cid:12)(cid:12)(cid:12) ≤ C

Original: [Equation]
Revised: [Equation]

Original:  ∥ x ′
Revised: E W ρ ∥ x ′

Original: [Equation]
Revised: [Equation]

Original: ϕ ( [Equation] ( x i )
Revised: ϕ (0) ( x i )

Original: [Equation]
Revised: [Equation]

Original: ϕ (2) ̸
Revised: ϕ (2) ̸

Original: [Equation] ( x i ) [Equation] E W ρ 1 √ m
Revised: ρ ( x i ) ∥

Original: [Equation]
Revised: [Equation]

Original:  X j = (cid:12)(cid:12)(cid:12) V (0) [Equation] ( x ′ i − ϕ (0) ( x i ) − ϕ
Revised: E W ρ 1 √ mm 2 X j = (cid:12)(cid:12)(cid:12) V (0) j ( x ′ i − ϕ (0) ( x i ) − ϕ

Original: (2) ̸ ρ ( x i ))
Revised: (2) ̸ ρ ( x i ))

Original: (cid:12)(cid:12)(cid:12) .
Revised: (cid:12)(cid:12)(cid:12) .

Original: Now using Lemma 32:
Revised: Now using Lemma 32:

Original: [Equation]
Revised: [Equation]

Original: Combining Equations (285) and (286) we conclude the proof.
Revised: Combining Equations (284) and (285) we conclude the proof.

Original: B A PPENDIX C ONTENTS
Revised: B A PPENDIX C ONTENTS

Original: B.1 Smoothness coefﬁcients . .
Revised: [Table]

Original: B.1.
Revised: [Table]

Original: Computing the Lipschitz Coefﬁcients of f ′ W ′ ,V ′ .
Revised: Computing the Lipschitz Coefﬁcients of [Table]

Original: B.2 Representation Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  . .
Revised: [Table] ′ ,V ′ . . . . . . . . . . . . . . . [Table] Representation Lemmas . .

Original: B.2.
Revised: B.

Revised: [Table]

Original: Representation Toolbox . .
Revised: Representation Toolbox . .

Original: B.2.
Revised: [Table]

Original: Some Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Some Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.2.
Revised: [Table]

Original: Bound on the norm of ¯ x i ’s . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Bound on the norm of ¯ x i ’s . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: [Table]

Original: B.3 Coupling for ˆ ∇ W , ˆ ∇ V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: 3 Coupling for ˆ ∇ W , [Table] ∇ V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B

Revised: [Table]

Original: B.4 Handling the Injected Noise by PSGD . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 4 Handling the Injected Noise by PSGD . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.4.
Revised: [Table]

Original: Bounding the Norm of the ﬁrst Layer’s Output in the Worst Case . . . . . .
Revised: Bounding the Norm of the ﬁrst Layer’s Output in the Worst Case . . . . . .

Original: B.
Revised: [Table]

Original: S MOOTHNESS COEFFICIENTS
Revised: S MOOTHNESS COEFFICIENTS

Original: Recall that for a function f ∈ C 3 on R d , we say it is µ 1 lipschitz, µ 2 gradient lipschitz, and µhessian lipschitz at point x if for every unit direction v , | ddλ f ( x + λv ) | ≤ µ 1 , | d 2 dλ 2 f ( x + λv ) | ≤ µ 2 , and | d 2 dλ 2 f ( x + λv ) | ≤ µ 3 .
Revised: Recall that for a function f ∈ C 3 on R d , we say it is µ 1 lipschitz, µ 2 gradient lipschitz, and µhessian lipschitz at point x if for every unit direction v , | ddλ f ( x + λv ) | ≤ µ 1 , | d 2 dλ 2 f ( x + λv ) | ≤ µ 2 , and | d 2 dλ 2 f ( x + λv ) | ≤ µ 3 .

Original: The aim of this section is to bound the lipschitz coefﬁcients of the loss ℓ ( , y ) and objective L ( W ′ , V ′ ) in a bounded domain ∥ W ′ ∥
Revised: The aim of this section is to bound the lipschitz coefﬁcients of the loss ℓ ( , y ) and objective L ( W ′ , V ′ ) in a bounded domain ∥ W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 .
Revised: ≤ C 2 .

Original: The following is our main Theorem in this regard:
Revised: The following is our main Theorem in this regard:

Original: Theorem 9 For given values C 1 , C 2 > 0 , in the domain ∥ W ′ ∥ ≤ C 1 , ∥ V ′ ∥
Revised: Theorem 9 For given values C 1 , C 2 > 0 , in the domain ∥ W ′ ∥ ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 , for any label | y | ≤ B , the loss function ℓ ( ., y ) is O
Revised: ≤ C 2 , for any label | y | ≤ B , the loss function ℓ ( ., y ) is O

Original: (( C 1 C 2 + B 2 )) -lipschitz (having enough overparameterization) and 1 gradient-lipschitz x = f ′ W ′ ,V ′ .
Revised: (( C 1 C 2 + B 2 )) -lipschitz (having enough overparameterization) and 1 gradient-lipschitz x = f ′ W ′ ,V ′ .

Original: Moreover, the loss function L ( W ′ , V ′ ) is ( O ( C 1 C 2 )+ B )Ψ 1 + 2( C 1 + C 2 ) lipschitz, Ψ 21 +( O ( C 1 C 2 )+ B )Ψ 2 +4 gradient lipschitz, and 3Ψ 2 Ψ 

Revised: Moreover, the loss function L ( W ′ , V ′ ) is ( O ( C 1 C 2 )+ B )Ψ 1 + 2( C 1 + C 2 ) lipschitz, Ψ 21 +( O ( C 1 C 2 )+ B )Ψ 2 +4 gradient lipschitz, and 3Ψ 2 Ψ

Original: +( O ( C 1 C 2 )+ B )Ψhessian lipschitz, where Ψ 1 , Ψ 2 , Ψ 3 are deﬁned in Lemma 36.
Revised: +( O ( C 1 C 2 )+ B )Ψhessian lipschitz, where Ψ 1 , Ψ 2 , Ψ 3 are deﬁned in Lemma 36.

Original: Proof of Lemma
Revised: Proof of Lemma

Original: As in the proof of Lemma 36, let ( ˜ W, ˜ V ) be a unit direction, i.e.
Revised: As in the proof of Lemma 36, let ( ˜ W, ˜ V ) be a unit direction, i.e.

Original: ∥ [Equation] ∥ [Equation]
Revised: ∥ ˜ W ∥ 2 + ∥ ˜ V ∥ 2 = 1 .

Original: Then, using Lemma 34, we know that for every i ∈ [ n ] : | f ′ W ′ ,V ′ ( x i )
Revised: Then, using Lemma 34, we know that for every i ∈ [ n ] : | f ′ W ′ ,V ′ ( x i )

Original: |
Revised: |

Original: =
Revised: =

Original: O ( C 1 C 2 ) , so by 1 -smoothness of the loss and B -boundedness of the labels, we get that ℓ ( ., y ) is ( O ( C 1 C 2 ) + B ) lipshcitz at point f ′ W ′ ,V ′ .
Revised: O ( C 1 C 2 ) , so by 1 -smoothness of the loss and B -boundedness of the labels, we get that ℓ ( ., y ) is ( O ( C 1 C 2 ) + B ) lipshcitz at point f ′ W ′ ,V ′ .

Original: The gradient smoothness parameter of the square loss ℓ is bounded byand its third derivative is zero.
Revised: The gradient smoothness parameter of the square loss ℓ is bounded byand its third derivative is zero.

Original: Now using these coefﬁcients, we can easily compute the coefﬁcients for L as well by simple differentiation:
Revised: Now using these coefﬁcients, we can easily compute the coefﬁcients for L as well by simple differentiation:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Moreover, note that
Revised: Moreover, note that

Original: [Equation]
Revised: [Equation]

Original: and similarly for ∥ V ′ + λ ˜ V ∥2 .
Revised: and similarly for ∥ V ′ + λ ˜ V ∥

Original: Combining these results ﬁnishes the proof.
Revised: Combining these results ﬁnishes the proof.

Original: Above, we used parameters Ψ 1 , Ψ 2 , Ψ 3 , the lipschitz coefﬁcients of f ′ in domain
Revised: Above, we used parameters Ψ 1 , Ψ 2 , Ψ 3 , the lipschitz coefﬁcients of f ′ in domain

Original: ∥ W ′ ∥ ≤
Revised: ∥ W ′ ∥ ≤

Original: C 1 , ∥ V ′ ∥
Revised: C 1 , ∥ V ′ ∥

Original: ≤ C 2 , which we bound in Lemma 36 below.
Revised: ≤ C 2 , which we bound in Lemma 36 below.

Original: B.1. C OMPUTING THE L IPSCHITZ C OEFFICIENTS OF f ′ W ′ ,V′
Revised: B.1. C OMPUTING THE L IPSCHITZ C OEFFICIENTS OF f ′ W ′ ,V

Original: In this section, we bound the lipschitz coefﬁcients of f ′ W ′ ,V ′ in the domain
Revised: In this section, we bound the lipschitz coefﬁcients of f ′ W ′ ,V ′ in the domain

Original: ∥ W ′ ∥
Revised: ∥ W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ Cby poly ( m 1 , m 2 , m 3 , β 1 , β 2 ) functions.
Revised: ≤ Cby poly ( m 1 , m 2 , m 3 , β 1 , β 2 ) functions [Equation]

Original: Lemma 

Revised: Lemma

Original: For every point ( W ′ , V ′ ) in the domain ∥ W ′ ∥
Revised: For every point ( W ′ , V ′ ) in the domain ∥ W ′ ∥

Original: ≤ C 1 , ∥ V ′ ∥
Revised: ≤ C 1 , ∥ V ′ ∥

Original: ≤ C 2 , we have the following bounds on the lipschitz coefﬁcients of f ′ W ′ ,V ′
Revised: ≤ C 2 , we have the following bounds on the lipschitz coefﬁcients of f ′ W ′ ,V ′

Original: ( ( ˜ W, ˜ V ) is a unit direction with ∥ ˜ W ∥ 2 + ∥ ˜ V ∥ 2 = 1 ):[Equation] (cid:12)(cid:12)(cid:12) [Equation]
Revised: ( ( ˜ W, ˜ V ) is a unit direction with ∥ ˜ W ∥ 2 + ∥ ˜ V ∥ 2 = 1 ):

Original: (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: be the density function of the law of W ρ and V ρ which is a joint Gaussian.
Revised: be the density function of the law of W ρ and V ρ which is a joint Gaussian.

Original: Then to compute the derivative and second derivative of the function in the unit direction ( ˜ W, ˜ V ) , s.t.
Revised: Then to compute the derivative and second derivative of the function in the unit direction ( ˜ W, ˜ V ) , s

Original: ∥
Revised: [Equation]

Original:  ˜ W ∥ 2 F + ∥ ˜ V ∥ 2 F , we can write the value of the smoothed function as an integration with density ρ , change variable, and then take derivatives:
Revised: t. ∥ ˜ W ∥ 2 F + ∥ ˜ V ∥ 2 F , we can write the value of the smoothed function as an integration with density ρ , change variable, and then take derivatives:

Original: [Equation]
Revised: [Equation]

Original: But one can easily see that for ﬁxed V ′ and ˜ V , the set of functions f W + ,V + ( x i )
Revised: But one can easily see that for ﬁxed V ′ and ˜ V , the set of functions f W + ,V + ( x i )

Original: ρ ( W + −
Revised: ρ ( W + −

Original: W ′ + λ ˜ W ) , V + − ( V ′ + λ ˜ V )) for a small neighborhood of λ can simultaneously be upper bounded by an integrable function.
Revised: W ′ + λ ˜ W ) , V + − ( V ′ + λ ˜ V )) for a small neighborhood of λ can simultaneously be upper bounded by an integrable function.

Original: Hence, the Leibnitz rule holds here because of dominated convergence theorem, and we can change the order of integration and derivation:
Revised: Hence, the Leibnitz rule holds here because of dominated convergence theorem, and we can change the order of integration and derivation:

Original: [Equation]
Revised: [Equation]

Original: Similarly we can compute the second derivative: 
Revised: Similarly we can compute the second derivative: d

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: = d dλ
Revised: = d dλ

Original: Z fW + ,V + ( x i ) D ( ˜ W [Equation] ˜ V )
Revised: Z f

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:16)

Original: [Equation]

Original: ( W ′

Original: [Equation]

Original: (cid:17)

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:16)

Original: V

Original: [Equation]
Revised: [Equation]

Original: ( V ′

Original: [Equation]

Original: ) (cid:17) )

Original: E [Equation] [Equation] W
Revised: λ

Original: [Equation]

Original: V

Original: [Equation]

Original: V

Original: [Equation]
Revised: [Equation]

Original: Similarly for the third derivative:
Revised: Similarly for the third derivative:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]

Original: Now for ﬁrst derivative, exactly similar to the derivation in ( [Table] we can write
Revised: Now for ﬁrst derivative, exactly similar to the derivation in (283), we can write

Original: [Table]
Revised: [Table]

Original: ∥

Original: [Equation]

Original: X

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: E(cid:12)(cid:12)

Original: [Equation]

Original: where the last line follows because ∥ V ′ ∥
Revised: where the last line follows because ∥ V ′ ∥

Original: ≲ C 2 .
Revised: ≲ C 2 .

Original: But notice that because ∥ ˜ V ∥ F ≤ 1 , ∥ ˜ W ∥ F ≤ 1 , then
Revised: But notice that because ∥ ˜ V ∥ F ≤ 1 , ∥ ˜ W ∥ F ≤ 1 , then

Original: D ˜ V , V ρ
Revised: D [Equation]

Original: E and
Revised: E and

Original: D ˜ W, W ρ
Revised: D [Equation]

Original: E are Gaussian variables with variances at most β 22 /m 2 and β 21 /m 1 .
Revised: E are Gaussian variables with variances at most β 22 /m 2 and β 21 /m 1 .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Similarly, using the same derivation as in (33), one can also get the following a.s. bound (over the randomness of W ρ ):
Revised: Similarly, using the same derivation as in (33), one can also get the following a.s. bound (over the randomness of W ρ ):

Original: [Equation]
Revised: [Equation]

Original: therefore
Revised: therefore

Original: [Equation]
Revised: [Equation]

Original: Moreover, for every j ∈ [ m 2 ] :
Revised: Moreover, for every j ∈ [ m 2 ] :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: using Equation (292) we bound
Revised: using Equation (291) we bound

Original: [Equation]
Revised: [Equation]

Original: Now applying these bounds (291), (294), (295)
Revised: Now applying these bounds (290), (293), (294)

Original: [Equation]
Revised: [Equation]

Original: and (296) to (290) and using the fact that
Revised: and (295) to (289) and using the fact that

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: κ [Equation] C(cid:17) + β 21 √ m(cid:17)(cid:17) .
Revised: κ 1 + C(cid:17) + β 21 √ m(cid:17)(cid:17) .

Original: To make it easier for handling the second and third derivatives
Revised: To make it easier for handling the second and third derivatives

Original: [Equation]
Revised: [Equation]

Original: we ﬁrst bound the expectations of f 2 W ′ + W ρ ,V ′
Revised: we ﬁrst bound the expectations of f 2 W ′ + W ρ ,V ′

Original: + V
Revised: + V

Original: ρ ( x i ) ( x i ) which enables us to use Cauchy-schwartz.
Revised: ρ ( x i ) ( x i ) which enables us to use Cauchy-schwartz.

Original: Again using similar derivation as in (284) and Equation (292):
Revised: Again using similar derivation as in (283) and Equation (291):

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: x

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now note that the cross terms have expectation zero, so we get=
Revised: Now note that the cross terms have expectation zero, so we get

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: Now applying Cauchy-shwartz and Equation (297) to above and combining it with Equations (289): d
Revised: Now applying Cauchy-shwartz and Equation (296) to above and combining it with Equations (288): d

Original: [Equation]
Revised: [Equation]

Original: Similarly for the third derivative:
Revised: Similarly for the third derivative:

Original: [Equation]
Revised: [Equation]

Original: But note that
Revised: But note that

Original: [Equation]
Revised: [Equation]

Original: Now note that m 1 β 21 ⟨ ˜ W, W ρ ⟩ + m 2 β 22 ⟨ ˜ V , V ρ ⟩ is a normal variable with variance m
Revised: Now note [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: ∥ ˜

Original: [Equation]
Revised: [Equation]

Original: Therefore, by the bound on the moments of normal random variables:
Revised: Therefore, by the bound on the moments of normal random variables:

Original: [Equation]
Revised: [Equation]

Original: Plugging this into Equation (298) and also using Equation (297):
Revised: Plugging this into Equation (297) and also using Equation (296):

Original: [Equation]
Revised: [Equation]

Original: i .
Revised: i .

Original: [Figure]

Original: B. R EPRESENTATION L EMMAS
Revised: B. R EPRESENTATION L EMMAS

Original: In this section, we prove lemmas mostly related to the representation power of the network, which we mainly use in Appendix A.12.
Revised: In this section, we prove lemmas mostly related to the representation power of the network, which we mainly use in Appendix A.12.

Original: B.2. R EPRESENTATION T OOLBOX
Revised: B.2. R EPRESENTATION T OOLBOX

Original: Lemma 37 Recall the deﬁnitions of W + k , and ¯ x i,k from Equations (66), and (62).
Revised: Lemma 37 Recall the deﬁnitions of W + k , and ¯ x i,k from Equations (64), and (60).

Original: For all k, i ∈ [ n ] : | ¯ x i,k − trace ( W + k Z ik ) | ≤ p n/ ( m 1 λ 0 )
Revised: For all k, i ∈ [ n ] : | ¯ x i,k − trace ( W + k Z ik ) | ≤ p n/ ( m 1 λ 0 )

Original: ∥V k ∥ H ∞ .
Revised: ∥V k ∥ H ∞ .

Original: Proof of Lemma
Revised: Proof of Lemma

Original: [Equation]
Revised: [Equation]

Original: But note ⟨
Revised: But note ⟨

Original: [Equation]
Revised: [Equation]

Original:  Now note that ⟨ x i ′ , x i ⟩ ≤ 1 , and 1 { W (0) j T x i } 1 { W (0)
Revised: ik , [Equation] Now note that ⟨ x i ′ , x i ⟩ ≤ 1 , and 1 { W (0) j T x i } 1 { W (0)

Original: j T x i ′ } is a Bernoulli with
Revised: j T x i ′ } is a Bernoulli with

Original: E 1 { W (0)
Revised: E 1 { W (0)

Original: j T x i } 1 { W (0)
Revised: j T x i } 1 { W (0)

Original: j T x i ′ } = 1 / 4 + arcsin( ⟨ x, y ⟩ ) / 2 π.
Revised: j T x i ′ } = 1 / 4 + arcsin( ⟨ x, y ⟩ ) / 2 π.

Original: Therefore, by Hoeffding inequality we get |⟨
Revised: Therefore, by Hoeffding inequality we get |⟨

Original: Z ik , Z i 
Revised: Z ik , Z i ′

Original: [Equation]
Revised: k ⟩ − H ∞

Original: i,i 
Revised: i,i ′ |

Original: [Equation] [Equation]
Revised: = O (1 /

Original: [Equation]
Revised: [Equation]

Original:  Hence, because obviously ∥ H ∞ ∥ 2 ≤ 1 , we get trace ( W + k Z ik ) =
Revised: m 1 ) . Hence, because obviously ∥ H ∞ ∥ 2 ≤ 1 , we get trace ( W + k Z ik ) =

Original: n X
Revised: n X

Original: i

Original: [Equation]
Revised: [Equation]

Original: V
Revised: V

Original: [Equation]

Original: [Equation]

Original: √ m

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Lemma 38 (Bounding the rows norm)
Revised: Lemma 38 (Bounding the rows norm)

Original: For every 1 ≤ j ≤ m 1 , we have
Revised: For every 1 ≤ j ≤ m 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for every k ∈ [ m 3 ] , we have
Revised: Furthermore, for every k ∈ [ m 3 ] , we have

Original: [Equation]
Revised: [Equation]

Original: For the ease of notation, because here we want to work with row sub indices of the matrix W + k , we refer to it by W k + .
Revised: For the ease of notation, because here we want to work with row sub indices of the matrix W + k , we refer to it by W k + .

Original: Proof of Lemma
Revised: Proof of Lemma

Original: For a ﬁxed 1 ≤ j ≤ m 1 we have with high probability over the randomness of the sign matrix W s :
Revised: For a ﬁxed 1 ≤ j ≤ m 1 we have with high probability over the randomness of the sign matrix W s :

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for every k ∈ [ m 3 ]
Revised: Furthermore, for every k ∈ [ m 3 ]

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 39 With high probability over the initialization
Revised: Lemma 39 With high probability over the initialization

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Recall from the deﬁnition of W k + in Equation (66):
Revised: Recall from the deﬁnition of W k + in Equation (64):

Original: [Equation]
Revised: [Equation]

Original: B.2. S OME L INEAR A LGEBRA
Revised: B.2. S OME L INEAR A LGEBRA

Original: Lemma 

Revised: Lemma

Original: For n ≤ s , let r 1 , . .
Revised: For n ≤ s , let r 1 , . .

Original: , r n be s -dimensional vectors that are approximately normalized and orthogonal to one another, i.e. given some δ > 0 , for every 1 ≤ i ̸ = j ≤ n :
Revised: , r n be s -dimensional vectors that are approximately normalized and orthogonal to one another, i.e. given some δ > 0 , for every 1 ≤ i ̸ = j ≤ n :

Original: [Equation]
Revised: [Equation]

Original: Then, for any vector v we have
Revised: Then, for any vector v we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: First, note that n X i = ⟨ v, r i ⟩ 2 ≤ (1 + δ ) n X i = ⟨ v, r i ⟩ 2 ∥ r i ∥= (1 + δ )
Revised: First, note that n X i = ⟨ v, r i ⟩ 2 ≤ (1 + δ ) n X i = ⟨ v, r i ⟩ 2 ∥ r i ∥= (1 + δ )

Original: ∥
Revised: ∥

Original: X i ⟨ v, r i ⟩ r i ∥ 2 − 2(1 + δ )
Revised: X i ⟨ v, r i ⟩ r i ∥ 2 − 2(1 + δ )

Original: X 1 ≤ i ̸ =
Revised: X 1 ≤ i ̸ =

Original: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r
Revised: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r

Original: i , r j ⟩ = (1 + δ ) ∥ v 1 ∥ 2 − 2(1 + δ )
Revised: i , r j ⟩ = (1 + δ ) ∥ v 1 ∥ 2 − 2(1 + δ )

Original: X 1 ≤ i ̸ =
Revised: X 1 ≤ i ̸ =

Original: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r
Revised: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r

Original: i , r j ⟩ .
Revised: i , r j ⟩ .

Original: Next, we write ⟨ v 1 , v − v 1 ⟩ = ⟨ v, n X i = ⟨ v, r i ⟩ r i ⟩ − ⟨ n X i = ⟨ v, r i ⟩ r i , n X i = ⟨ v, r i ⟩ r i ⟩ =
Revised: Next, we write ⟨ v 1 , v − v 1 ⟩ = ⟨ v, n X i = ⟨ v, r i ⟩ r i ⟩ − ⟨ n X i = ⟨ v, r i ⟩ r i , n X i = ⟨ v, r i ⟩ r i ⟩ =

Original: X i ⟨ v, r i ⟩ 2 −
Revised: X i ⟨ v, r i ⟩ 2 −

Original: X i ⟨ v, r i ⟩ 2 −
Revised: X i ⟨ v, r i ⟩ 2 −

Original: X i ̸
Revised: X i ̸

Original: = j ⟨ v, r i ⟩⟨ v, r j ⟩⟨ r
Revised: = j ⟨ v, r i ⟩⟨ v, r j ⟩⟨ r

Original: i , r j ⟩ = −
Revised: i , r j ⟩ = −

Original: X i ̸
Revised: X i ̸

Original: = j ⟨ v, r i ⟩⟨ v, r j ⟩⟨ r
Revised: = j ⟨ v, r i ⟩⟨ v, r j ⟩⟨ r

Original: i , r j ⟩ .
Revised: i , r j ⟩ .

Original: Therefore n X i = ⟨ v, r i ⟩ 2 ≤ (1 + δ )( ∥ v 1 ∥ 2 + ∥ v − v 1 ∥ 2 ) − 2(1 + δ )
Revised: Therefore n X i = ⟨ v, r i ⟩ 2 ≤ (1 + δ )( ∥ v 1 ∥ 2 + ∥ v − v 1 ∥ 2 ) − 2(1 + δ )

Original: X 1 ≤ i ̸ =
Revised: X 1 ≤ i ̸ =

Original: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r
Revised: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r

Original: i , r j ⟩ .
Revised: i , r j ⟩ .

Original: ≤ (1 + δ )( ∥ v 1 ∥ 2 + ∥ v − v 1 ∥ 2 + 2 ⟨ v 1 , v − v 1 ⟩ ) + 2(1 + δ )
Revised: ≤ (1 + δ )( ∥ v 1 ∥ 2 + ∥ v − v 1 ∥ 2 + 2 ⟨ v 1 , v − v 1 ⟩ ) + 2(1 + δ )

Original: X 1 ≤ i ̸ =
Revised: X 1 ≤ i ̸ =

Original: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r
Revised: j ≤ n ⟨ v, r i ⟩⟨ v, r i ⟩⟨ r

Original: i , r j ⟩ ≤ (1 + δ ) ∥ v ∥ 2 + 2(1 + δ )
Revised: i , r j ⟩ ≤ (1 + δ ) ∥ v ∥ 2 + 2(1 + δ )

Original: X 1 ≤ i ̸ =
Revised: X 1 ≤ i ̸ =

Original: j ≤ n ∥ v ∥
Revised: j ≤ n ∥ v ∥

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: In the following lemma, we state a trivial bound on the norm of ¯ x i based on ζ 1 .
Revised: In the following lemma, we state a trivial bound on the norm of ¯ x i based on ζ 1 .

Original: B.2. B OUND ON THE NORM OF ¯ x i ’ S
Revised: B.2. B OUND ON THE NORM OF ¯ x i ’ S

Original: Lemma 

Revised: Lemma

Original: For every i ∈ [ n ] , we have
Revised: For every i ∈ [ n ] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: By deﬁnition:
Revised: By deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: Now consider the Cholskey factorization H ∞ =
Revised: Now consider the Cholskey factorization H ∞ =

Original: KK T .
Revised: KK T .

Original: Because of the assumption ∥ x i ∥ = 1 , we know that the diagonal of H ∞ is all 1 / 2 .
Revised: Because of the assumption ∥ x i ∥ = 1 , we know that the diagonal of H ∞ is all 1 / 2 .

Original: Hence, for the i th row of K we have ∥ K i ∥ = 1 / 2 .
Revised: Hence, for the i th row of K we have ∥ K i ∥ = 1 / 2 .

Original: Now by Cauchy-Swartz, we have
Revised: Now by Cauchy-Swartz, we have

Original: [Equation]
Revised: [Equation]

Original: Summing over i and noting Equation (61) completes the proof.
Revised: Summing over i and noting Equation (59) completes the proof.

Original: Lemma 

Revised: Lemma

Original: In the context of Lemma 10, for ζ 2 ≤ 2 nB 2 , one can substitute f ∗ by  f ∗1 such that
Revised: In the context of Lemma 10, for ζ 2 ≤ 2 nB 2 , one can substitute f ∗ by ¯ f ∗ such that

Original: [Equation]
Revised: [Equation]

Original: and furthermore, ¯ f ∗ is in the subspace of eigenvectors of A with eigenvaue larger than Ω( 1 n 2 ) .
Revised: and furthermore, ¯ f ∗ is in the subspace of eigenvectors of A with eigenvaue larger than Ω( 1 n 2 ) .

Revised: Moreover, the constant 2 is arbitrary and can be changed to any constant more than one, with the cost of an additional constant behind the second term.

Original: Proof of Lemma
Revised: Proof of Lemma

Original: For an arbitrary i ∈ [ n ] , deﬁne
Revised: For an arbitrary i ∈ [ n ] and some given vector ¯ f ∗ (we will specify later), we deﬁne

Original: [Equation]
Revised: [Equation]

Original: and suppose the slope of ℓ ( ., y i ) at point f ∗ i is equal to c .
Revised: and suppose the slope of ℓ ( ., y i ) at point f ∗ i is equal to c .

Original: Then, using the convexity, the fact that ℓ ( y i , y i ) = 0 , and the 1 -smoothness of ℓ ( ., y i ) , it is not hard to see the following poincare inequality between the value and derivative of ℓ ( ., y i ) at point f ∗ i :
Revised: Then, using the convexity, the fact that ℓ ( y i , y i ) = 0 , and the 1 -smoothness of ℓ ( ., y i ) , it is not hard to see the following poincare inequality between the value and derivative of ℓ ( ., y i ) at point f ∗ i :

Original: [Equation]
Revised: [Equation]

Original: where from now on, for brevity, we refer to ℓ ( f ∗ i , y i ) by ℓ .
Revised: where from now on, for brevity, we refer to ℓ ( f ∗ i , y i ) by ℓ .

Original: Also, from the deﬁnition of δ and again using 1 smoothness property, it is easy to see that
Revised: Also, from the deﬁnition of δ and again using 1 smoothness property, it is easy to see that

Original: [Equation]
Revised: [Equation]

Original: Plugging Equation (302) into (303) and using AM-GM inequality:
Revised: Plugging Equation (301) into (302) and using AM-GM inequality:

Original: [Equation]
Revised: [Equation]

Original: Summing above for i ∈ [ n ] , we obtain
Revised: Summing above for i ∈ [ n ] , we obtain

Original: [Equation]
Revised: [Equation]

Original: Now we write an eigendecomposition for A as A = P ni =
Revised: Now we write an eigendecomposition for A as A = P ni =

Original: λ i u
Revised: λ i u

Original: i
Revised: i

Original: u
Revised: u

Original: Ti for orthonormal basis { u i } , and let f ∗ =
Revised: Ti for orthonormal basis { u i } , and let f ∗ =

Original: P i γ i
Revised: P i γ i

Original: u
Revised: u

Original: i be the representation of f ∗ in this basis.
Revised: i be the representation of f ∗ in this basis.

Original: Then, from our assumption, for arbitrary ω >
Revised: Then, from our assumption, for arbitrary ω >

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: or equivalently
Revised: or equivalently

Original: [Equation]
Revised: [Equation]

Original: where notice that P i : λ i ≤ ω γ 

Revised: where notice that P i : λ i ≤ ω γ

Original: i is the squared norm of the projection of f ∗ onto the directions whose eigenvalue is at most ω .
Revised: i is the squared norm of the projection of f ∗ onto the directions whose eigenvalue is at most ω .

Original: Now taking ω = 1 12 n 2 and deﬁning ¯ f ∗ by keeping only the directions  for which λ i > ω  completes the proof.
Revised: Now taking ω = 1 12 n 2 and deﬁning ¯ f ∗ by keeping only the directions in the expansion of f ∗ in the eigenbasis of A , for which λ i > ω , completes the proof.

Original: B. C OUPLING FOR ˆ ∇ W , ˆ ∇ V
Revised: B. C OUPLING FOR ˆ ∇ W , ˆ ∇ V

Original: In general, because the gaussian smoothing matrices ( W ρ , V ρ ) can become unbounded, the gradient estimates ( ˆ ∇ W , ˆ ∇ V ) = ∇ W,V ℓ ( f W ′ + W ρ ,V ′ + V ρ ( x i ) , y i ) also become unbounded.
Revised: In general, because the gaussian smoothing matrices ( W ρ , V ρ ) can become unbounded, the gradient estimates ( ˆ ∇ W , ˆ ∇ V ) = ∇ W,V ℓ ( f W ′ + W ρ ,V ′ + V ρ ( x i ) , y i ) also become unbounded.

Original: However, in analyzing the stochastic behavior of SGD and showing that it can escape saddle points, it is convenient to assume the gradient’s noise vector is almost surely bounded.
Revised: However, in analyzing the stochastic behavior of SGD and showing that it can escape saddle points, it is convenient to assume the gradient’s noise vector is almost surely bounded.

Original: The goal of this section is to introduce a coupling between ( W ρ , V ρ ) and another random variable that is a.s. bounded polynomially in other parameters.
Revised: The goal of this section is to introduce a coupling between ( W ρ , V ρ ) and another random variable that is a.s. bounded polynomially in other parameters.

Original: As that the coupled random variables take different values is exponentially small while the number of iterations in our algorithm is only polynomially large, without any concern we instead work with this new random varaible, and with an overload of notation we also denote it by ( W ρ , V ρ ) .
Revised: As that the coupled random variables take different values is exponentially small while the number of iterations in our algorithm is only polynomially large, without any concern we instead work with this new random varaible, and with an overload of notation we also denote it by ( W ρ , V ρ ) .

Original: Lemma 

Revised: Lemma

Original: For an arbitrary parameter χ >> 1 , On any pair for ( W ′ , V ′ ) with ∥ W ′ ∥ ≤ C 1 , ∥
Revised: For an arbitrary parameter χ >> 1 , On any pair for ( W ′ , V ′ ) with ∥ W ′ ∥ ≤ C 1 , ∥

Original: V ′ ∥
Revised: V ′ ∥

Original: ≤ C 2 , there exist a mean zero random vector ¯Λ with respect to the randomness of the uniformly picked data point ( x i , y i ) and the smoothing matrices W ρ, 1 , W ρ, 2 , V ρ, 1 , and V ρ, 2 which deﬁne ˆ ∇ W ′ ,V ′
Revised: ≤ C 2 , there exist a mean zero random vector ¯Λ with respect to the randomness of the uniformly picked data point ( x i , y i ) and the smoothing matrices W ρ, 1 , W ρ, 2 , V ρ, 1 , and V ρ, 2 which deﬁne ˆ ∇ W ′ ,V ′

Original: (meaning it is a function of those variables), such that with probability at least
Revised: (meaning it is a function of those variables), such that with probability at least

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and ﬁnally ¯Λ is a.s. polynomially bounded, i.e. almost surely we have
Revised: and ﬁnally ¯Λ is a.s. polynomially bounded, i.e. almost surely we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Remember that x
Revised: Remember that x

Original: ′ i was the output of the ﬁrst layer (by considering the smoothing matrix
Revised: ′ i was the output of the ﬁrst layer (by considering the smoothing matrix

Original: W ρ ).
Revised: W ρ ).

Original: Now with high probability over the initialization,
Revised: Now with high probability over the initialization,

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: using the ﬁnal bound in Lemma 33:
Revised: using the ﬁnal bound in Lemma 33:

Original: [Equation]
Revised: [Equation]

Original: Denoting
Revised: Denoting

Original: [Equation]
Revised: [Equation]

Original: i ) , y i ) by ˜ ∇ W ′ ,V ′ , then combining Equations (307) and (308) and using the 1 gradient lipschitzness property of the square loss, :
Revised: ) , y i ) by ˜ ∇ W ′ ,V ′ , then combining Equations (306) and (307) and using the 1 gradient lipschitzness property of the square loss, ∥ ˜ ∇ W ′ ,V ′ ∥ F = [Equation]

Revised: d ( ℓ

Original: [Equation]
Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: y i

Revised: [Equation]

Original: Finally, applying Cauchy-Swartz to the second a
Revised: Finally, applying Cauchy-Swartz to the second a

Original: [Equation]
Revised: [Equation]

Original: s. bound in Lemma 34 we have:
Revised: s. bound in Lemma 34 we have:

Original: [Equation]
Revised: [Equation]

Original: Combining this with (309):
Revised: Combining this with (308):

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: using the lipschitz bound in Theorem 9:
Revised: using the lipschitz bound in Theorem 9:

Original: [Equation]
Revised: [Equation]

Original: Now we deﬁne the following events
Revised: Now we deﬁne the following events

Original: [Equation]
Revised: [Equation]

Original: where recall we assume χ >> 1 .
Revised: where recall we assume χ >> 1 .

Original: Then, as we know the variable ∥ W ρ ∥ 2 F has mean dβ 21 and is subexponential with parameters ( dβ 41 /m 1 , β 21 /m 1 ) .
Revised: Then, as we know the variable ∥ W ρ ∥ 2 F has mean dβ 21 and is subexponential with parameters ( dβ 41 /m 1 , β 21 /m 1 ) .

Original: Hence, by a union bound and Bernstein (Note that W ρ, 1 , W ρ, 2 are independent):
Revised: Hence, by a union bound and Bernstein (Note that W ρ, 1 , W ρ, 2 are independent):

Original: [Equation]
Revised: [Equation]

Original: Similarly for ∥ V ρ ∥ F :
Revised: Similarly for ∥ V ρ ∥ F :

Original: [Equation]
Revised: [Equation]

Original: Moreover, because of the subexponential tails of ∥ W ρ ∥ 2 F and ∥ V ρ ∥ 2 F , for each of W ρ, 1 or W ρ 2 ,
Revised: Moreover, because of the subexponential tails of ∥ W ρ ∥ 2 F and ∥ V ρ ∥ 2 F , for each of W ρ, 1 or W ρ 2 ,

Original: V ρ 1 , or V ρ 2 :
Revised: V ρ 1 , or V ρ 2 :

Original: [Equation]
Revised: [Equation]

Original: Now Deﬁning Ξ =
Revised: Now Deﬁning Ξ =

Original: Ξ 1 ∪ Ξ 2 and combining the above equations:
Revised: Ξ 1 ∪ Ξ 2 and combining the above equations:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Applying these equations to (310) with Cauchy-Schwartz to get the upper bounds E 1 { Ξ 1 }∥
Revised: Applying these equations to (309) with Cauchy-Schwartz to get the upper bounds E 1 { Ξ 1 }∥

Original: W ρ, 1 ∥∥ W ρ, 2 ∥ ≤ E 1 { Ξ 1 }∥
Revised: W ρ, 1 ∥∥ W ρ, 2 ∥ ≤ E 1 { Ξ 1 }∥

Original: W ρ ∥ 2 and ( E W ρ ∥ W ρ ∥ ) 2 ≤ E W ρ ∥ W ρ ∥ 2 (for terms with only one W ρ,i or V ρ,i
Revised: W ρ ∥ 2 and ( E W ρ ∥ W ρ ∥ ) 2 ≤ E W ρ ∥ W ρ ∥ 2 (for terms with only one W ρ,i or V ρ,i

Original: [Equation]
Revised: [Equation]

Original: we simply write them as W ρ and V ρ ):
Revised: we simply write them as W ρ and V ρ ):

Original: [Equation]
Revised: [Equation]

Original: But note that
Revised: But note that

Original: [Equation]
Revised: [Equation]

Original: Applying this to Equation (311), we get that if we deﬁne
Revised: Applying this to Equation (310), we get that if we deﬁne

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: On the other hand, note that using again Equation (310), we have the following a.s. bound:
Revised: On the other hand, note that using again Equation (309), we have the following a.s. bound:

Original: [Equation]
Revised: [Equation]

Original: Deﬁning
Revised: Deﬁning

Original: [Equation]
Revised: [Equation]

Original: we get that with probability at least 1 − P (Ξ) :
Revised: we get that with probability at least 1 − P (Ξ) :

Original: [Equation]
Revised: [Equation]

Original: and also note that
Revised: and also note that

Revised: E

Original: [Equation]
Revised: [Equation]

Original: Finally by the a.s. bound for Λ 1 , we have a.s.:
Revised: Finally by the a.s. bound for Λ 1 , we have a.s.:

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.

Original: Corollary 9.1 It is easy to check that running PSGD with unbiased gradient estimate ˆ ∇ W ′ ,V ′ is equivalent to running SGD after our change of coordinates, with unbiased gradient estimate ˆ ∇ w ′ ,v ′ :
Revised: Corollary 9.1 It is easy to check that running PSGD with unbiased gradient estimate ˆ ∇ W ′ ,V ′ is equivalent to running SGD after our change of coordinates, with unbiased gradient estimate ˆ ∇ w ′ , [Equation]

Original: =
Revised: =

Original: ¯ Υ ∇ W ′ ,V ′ , where ¯ Υ is the matrix for our change of coordinate, which is equal to Υ deﬁned in Appendix A.10 for the coordinates in V ′ and simply identity for the coordinates in W ′ .
Revised: ¯ Υ ∇ W ′ ,V ′ , where ¯ Υ is the matrix for our change of coordinate, which is equal to Υ deﬁned in Appendix A.10 for the coordinates in V ′ and simply identity for the coordinates in W ′ .

Original: Therefore, projecting both sides in Equation (312) of Lemma 43 onto Φ ⊥ by multiplying Υ implies that with high probability for all iterations of the algorithm
Revised: Therefore, projecting both sides in Equation (311) of Lemma 43 onto Φ ⊥ by multiplying Υ implies that with high probability for all iterations of the algorithm

Original: [Equation]
Revised: [Equation]

Original:  where ¯ £ :=
Revised: (311) where ¯ £ :=

Original: ¯Υ¯Λ (using the properties of ¯Λ in Lemma 43) is a mean zero noise vector with almost surely bounded norm, i.e. ∥ ¯ £ ∥ ≤ Q ′ for some Q ′ =
Revised: ¯Υ¯Λ (using the properties of ¯Λ in Lemma 43) is a mean zero noise vector with almost surely bounded norm, i.e. ∥ ¯ £ ∥ ≤ Q ′ for some Q ′ =

Original: poly ( m 1 , m 2 , m 3 , C 1 , C 2 ) .
Revised: poly ( m 1 , m 2 , m 3 , C 1 , C 2 ) .

Original: (we dropped the χ parameters by considering constant high probability argument).
Revised: (we dropped the χ parameters by considering constant high probability argument).

Original: Finally, note that injecting noise (Ξ 1 / ∥ Ξ 1 ∥ , Ξ 2 / ( √ m 1 ∥
Revised: Finally, note that injecting noise (Ξ 1 / ∥ Ξ 1 ∥ , Ξ 2 / ( √ m 1 ∥

Original: Ξ 2 ∥ )) by PSGD results in adding an extra zero mean noise (˜Ξ 1 , ˜Ξ 2 ) :
Revised: Ξ 2 ∥ )) by PSGD results in adding an extra zero mean noise (˜Ξ 1 , ˜Ξ 2 ) :

Original: =
Revised: =

Original: (¯ΥΞ 1 / ∥ Ξ 1 ∥ , ¯ΥΞ 2 / ( √ m 1 ∥
Revised: (¯ΥΞ 1 / ∥ Ξ 1 ∥ , ¯ΥΞ 2 / ( √ m 1 ∥

Original: Ξ 2 ∥ )) to the gradient ∇ w ′ ,v ′ L Π ( w ′ , v ′ ) .
Revised: Ξ 2 ∥ )) to the gradient ∇ w ′ ,v ′ L Π ( w ′ , v ′ ) .

Original: Therefore, overall running SGD on L Π (which is equivalent to PSGD on L) observe an unbiased noise vector deﬁned as £ :
Revised: Therefore, overall running SGD on L Π (which is equivalent to PSGD on L) observe an unbiased noise vector deﬁned as £ :

Original: =
Revised: =

Original: ¯ £ + (˜Ξ 1 , ˜Ξ 2 ) .
Revised: ¯ £ + (˜Ξ 1 , ˜Ξ 2 ) .

Original: Now it is easy to check that the moment matrix of ˜Ξand ˜Ξ 2 are σ
Revised: Now it is easy to check that the moment matrix of ˜Ξand ˜Ξ 2 are σ

Original: ′ 12 I and σ ′ 

Revised: ′ 12 I and σ ′

Original: I for
Revised: I for

Original: [Equation]
Revised: [Equation]

Original: which implies the moment matrix of £ is upper bounded by
Revised: which implies the moment matrix of £ is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: and lower bounded by
Revised: and lower bounded by

Original: [Equation]
Revised: [Equation]

Original: i.

Original: [Equation]
Revised: [Equation]

Original: (Note that we look at the new coordinates ( w ′ , v ′ ) as a vectors, so the term E ££ T makes sense.)
Revised: (Note that we look at the new coordinates ( w ′ , v ′ ) as a vectors, so the term E ££ T makes sense.)

Original: Moreover, ∥ ˜Ξ 1 ∥ = 1 / √ m 1 , ∥ ˜Ξ 2 ∥ ≤ 1 almost surely, which implies the following almost surely bound for £ :
Revised: Moreover, ∥ ˜Ξ 1 ∥ = 1 / √ m 1 , ∥ ˜Ξ 2 ∥ ≤ 1 almost surely, which implies the following almost surely bound for £ :

Original: [Equation]
Revised: [Equation]

Original: Lemma 44 Let g ( x ) be a second order differentiable function over R N such that at point x , there exist a random direction y and deterministic direction z and ﬁxed positive real r with:
Revised: Lemma 44 Let g ( x ) be a second order differentiable function over R N such that at point x , there exist a random direction y and deterministic direction z and ﬁxed positive real r with:

Original: [Equation]
Revised: [Equation]

Original: Then, for the gradient and Hessian at point x [Equation] we have either
Revised: Then, for the gradient and Hessian at point x , we have either

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma
Revised: Proof of Lemma

Original: We write the second order tailor approximation of g around x :
Revised: We write the second order tailor approximation of g around x :

Original: [Equation]
Revised: [Equation]

Original: Now substituting w with ηz + √ ηy and taking expectation with respect to y
Revised: Now substituting w with ηz + √ ηy and taking expectation with respect to y

Original: [Equation]
Revised: [Equation]

Original: as we send η → 0 and using the fact that E y = 0 :
Revised: as we send η → 0 and using the fact that E y = 0 :

Original: [Equation]
Revised: [Equation]

Original: Combining the assumption with the above Equation
Revised: Combining the assumption with the above Equation

Original: [Equation]
Revised: [Equation]

Original: we get that for small enough η
Revised: we get that for small enough η

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: .e.

Original: [Equation]
Revised: [Equation]

Original: which means we should either have
Revised: which means we should either have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ≤
Revised: ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: which implies 
Revised: which implies λ

Original: [Equation]
Revised: [Equation]

Original: B. H ANDLING THE I NJECTED N OISE BY PSGD
Revised: B. H ANDLING THE I NJECTED N OISE BY PSGD

Original: In this section, we prove that having SGD injecting noise into our gradient estimates mostly does not change the sign pattern of the ﬁrst layer, namely among the set of rows in P deﬁned in Lemma 1.
Revised: In this section, we prove that having SGD injecting noise into our gradient estimates mostly does not change the sign pattern of the ﬁrst layer, namely among the set of rows in P deﬁned in Lemma 1.

Original: Lemma 45 Having enough overparameterization, with high probability, at every iteration of the
Revised: Lemma 45 Having enough overparameterization, with high probability, at every iteration of the

Original: PSGD for W ′ (2) deﬁned in  Lemma 1, we have for every j ∈ [ m 1 ] : ∥ W ′ (2) j ∥ ≤ c 2 / (4 √ m 1 ) .
Revised: PSGD for W ′ (2) deﬁned in the proof of Lemma 1, we have for every j ∈ [ m 1 ] : ∥ W ′ (2) j ∥ ≤ c 2 / (4 √ m 1 ) .

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Let Φ ′ be the subspace of the ﬁrst layer weight matrices which is zero in rows j ∈ P
Revised: Let Φ ′ be the subspace of the ﬁrst layer weight matrices which is zero in rows j ∈ P

Original: ( P is deﬁned in Lemma 1), while in other rows it is the span of Z ki ’s, i.e. using our notation ˜ Z ik introduced in the proof of Lemma 1, we can write Φ ′ is span ( Z ki ) i,k .
Revised: ( P is deﬁned in Lemma 1), while in other rows it is the span of Z ki ’s, i.e. using our notation ˜ Z ik introduced in the proof of Lemma 1, we can write Φ ′ is span ( Z ki ) i,k .

Original: Recall from Lemma 1 that we decompose the ﬁrst layer weight W ′ as W ′ (1) + W ′ (2) , namely the parts in the subspace Φ ′ and subspace Φ ′⊥ respectively.
Revised: Recall from Lemma 1 that we decompose the ﬁrst layer weight W ′ as W ′ (1) + W ′ (2) , namely the parts in the subspace Φ ′ and subspace Φ ′⊥ respectively.

Original: Moreover, let
Revised: Moreover, let

Original: Ξ 1 / ( √ m 1 ∥
Revised: Ξ 1 / ( √ m 1 ∥

Original: Ξ 1 ∥ )
Revised: Ξ 1 ∥ )

Original: =
Revised: =

Original: Ξ (1) + Ξ (2) be the decomposition of the injected noise at some iteration of PSGD into subspaces Φ ′ and Φ ′⊥ respectively.
Revised: Ξ (1) + Ξ (2) be the decomposition of the injected noise at some iteration of PSGD into subspaces Φ ′ and Φ ′⊥ respectively.

Original: Now recall that the current W ′ is the value of the previous iteration moved by the gradient plus the injected noise:
Revised: Now recall that the current W ′ is the value of the previous iteration moved by the gradient plus the injected noise:

Original: [Equation]
Revised: [Equation]

Original: where W ′ is the weight of the previous iteration and W ′− , (1) , W ′− , (2) are again its decomposition to Φ ′ and Φ ′⊥ , where ˜ ∇ W ′ ,V ′ is deﬁned in Lemma 43.
Revised: where W ′ is the weight of the previous iteration and W ′− , (1) , W ′− , (2) are again its decomposition to Φ ′ and Φ ′⊥ , where ˜ ∇ W ′ ,V ′ is deﬁned in Lemma 43.

Original: Applying Lemma 24 for the previous iteration of the algorithm, we get ˜ ∇ W ′ ∈
Revised: Applying Lemma 24 for the previous iteration of the algorithm, we get ˜ ∇ W ′ ∈

Original: Φ ′ since the bad events E deﬁned in Lemma 33 occurs only with probability exponentially small (hence union bound across all the iterations rules it out).
Revised: Φ ′ since the bad events E deﬁned in Lemma 33 occurs only with probability exponentially small (hence union bound across all the iterations rules it out).

Original: Hence, the decomposition for the current iteration becomes
Revised: Hence, the decomposition for the current iteration becomes

Original: [Equation]
Revised: [Equation]

Original: We handle the W ′ (1) part in Lemma 1 and prove that as long as ∥ W ′ (1) ∥ 2 ≤ ∥ W ′ ∥
Revised: We handle the W ′ (1) part in Lemma 1 and prove that as long as ∥ W ′ (1) ∥ 2 ≤ ∥ W ′ ∥

Original: 2 remains bounded by C 2 1 , then the sign pattern of the ﬁrst layer, when only considering the W ′ (1) part, is speciﬁed by the initialization except within set P ; here we handle the W ′ (2) part as well.
Revised: 2 remains bounded by C 2 1 , then the sign pattern of the ﬁrst layer, when only considering the W ′ (1) part, is speciﬁed by the initialization except within set P ; here we handle the W ′ (2) part as well.

Original: Note that for every row j
Revised: Note that for every row j

Original: [Equation]
Revised: [Equation]

Original: [ m [Equation] ] , the variable
Revised: [ m 1 ] , the variable

Original: [Equation]
Revised: [Equation]

Original: is ( O (1 / ( m 41 d )) , O (1 / ( m 21 d )))
Revised: is ( O (1 / ( m 41 d )) , O (1 / ( m 21 d )))

Original: -subexponential with mean 1 /m 1 .
Revised: -subexponential with mean 1 /m 1 .

Original: Therefore, with probability that is exponentially small in m 1 , ∥
Revised: Therefore, with probability that is exponentially small in m 1 , ∥

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: √ m
Revised: Ξ 1 /

Original: [Equation]
Revised: [Equation]

Original:  ∥ Ξ
Revised: √ m 1 ∥ Ξ

Original: [Equation]
Revised: [Equation]

Original:  (cid:17) j ∥ is bounded by O (1 /m 1 ) .
Revised: ∥ ) (cid:17) j ∥ is bounded by O (1 /m 1 ) .

Original: It is not hard to see the same argument holds for the projection of Ξ 1 / ( √ m 1 ∥ Ξ 1 ∥ ) onto Φ ⊥ , i.e. Ξ (2) .
Revised: It is not hard to see the same argument holds for the projection of Ξ 1 / ( √ m 1 ∥ Ξ 1 ∥ ) onto Φ ⊥ , i.e. Ξ (2) .

Original: Applying a union bound for all iterations, again using the fact that we run PSGD for poly iterations while the chance of error is exponentially small in m 1 , we can then argue that with high probability over the noise of gradients, at every iteration and for every j ∈ [ m 1 ] :
Revised: Applying a union bound for all iterations, again using the fact that we run PSGD for poly iterations while the chance of error is exponentially small in m 1 , we can then argue that with high probability over the noise of gradients, at every iteration and for every j ∈ [ m 1 ] :

Original: [Equation]
Revised: [Equation]