Original: We handle the W ′ (1) part in Lemma 1 and prove that as long as ∥ W ′ (1) ∥ 2 ≤ ∥ W ′ ∥
Revised: We handle the W ′ (1) part in Lemma 1 and prove that as long as ∥ W ′ (1) ∥ 2 ≤ ∥ W ′ ∥

Original: 2 remains bounded by C 2 1 , then the sign pattern of the ﬁrst layer, when only considering the W ′ (1) part, is speciﬁed by the initialization except within set P ; here we handle the W ′ (2) part as well.
Revised: 2 remains bounded by C 2 1 , then the sign pattern of the ﬁrst layer, when only considering the W ′ (1) part, is speciﬁed by the initialization except within set P ; here we handle the W ′ (2) part as well.

Original: Note that for every row j
Revised: Note that for every row j

Original: [Equation]
Revised: [Equation]

Original: [ m 1 ] , the variable
Revised: [ m 1 ] , the variable

Original: [Equation]
Revised: [Equation]

Original: is ( O (1 / ( m 41 d )) , O (1 / ( m 21 d )))
Revised: is ( O (1 / ( m 41 d )) , O (1 / ( m 21 d )))

Original: -subexponential with mean 1 /m 1 .
Revised: -subexponential with mean 1 /m 1 .

Original: Therefore, with probability that is exponentially small in m 1 , ∥
Revised: Therefore, with probability that is exponentially small in m 1 , ∥

Original: [Equation]
Revised: [Equation]

Original: Ξ 1 /
Revised: Ξ 1 /

Original: [Equation]
Revised: [Equation]

Original: √ m 1 ∥ Ξ
Revised: √ m 1 ∥ Ξ

Original: [Equation]
Revised: [Equation]

Original: ∥ ) (cid:17) j ∥ is bounded by O (1 /m 1 ) .
Revised: ∥ ) (cid:17) j ∥ is bounded by O (1 /m 1 ) .

Original: It is not hard to see the same argument holds for the projection of Ξ 1 / ( √ m 1 ∥ Ξ 1 ∥ ) onto Φ ⊥ , i.e. Ξ (2) .
Revised: It is not hard to see the same argument holds for the projection of Ξ 1 / ( √ m 1 ∥ Ξ 1 ∥ ) onto Φ ⊥ , i.e. Ξ (2) .

Original: Applying a union bound for all iterations, again using the fact that we run PSGD for poly iterations while the chance of error is exponentially small in m 1 , we can then argue that with high probability over the noise of gradients, at every iteration and for every j ∈ [ m 1 ] :
Revised: Applying a union bound for all iterations, again using the fact that we run PSGD for poly iterations while the chance of error is exponentially small in m 1 , we can then argue that with high probability over the noise of gradients, at every iteration and for every j ∈ [ m 1 ] :

Original: [Equation]
Revised: [Equation]

Original: But applying trinagle inequality to Equation (315) and writing it in a telescope form, particularly for the j th row, and further using the assumption in 316, we get that ∥ W ′ (2) j ∥ grows at most to
Revised: But applying trinagle inequality to Equation (315) and writing it in a telescope form, particularly for the j th row, and further using the assumption in 316, we get that ∥ W ′ (2) j ∥ grows at most to

Original: O (1 / ( m 1 ψ 1 )) ; as we set 1 /ψ 1 =
Revised: O (1 / ( m 1 ψ 1 )) ; as we set 1 /ψ 1 =

Original: O ( poly ( n )) , assuming polynomially large enough m 1 concludes the claim.
Revised: O ( poly ( n )) , assuming polynomially large enough m 1 concludes the claim.

Original: B.4. B OUNDING THE N ORM OF THE FIRST L AYER ’ S O UTPUT IN THE W ORST C ASE
Revised: B.4. B OUNDING THE N ORM OF THE FIRST L AYER ’ S O UTPUT IN THE W ORST C ASE

Original: Lemma 46 Suppose W ′ satisﬁes the assumption of Lemma 1, i.e. ∥ W ′ ∥ ≤ C 1 , and ∥ W ′ j ∥ ≤ c 2 / (2 √ m 1 ) except possible for indices in P , also deﬁned in 1.
Revised: Lemma 46 Suppose W ′ satisﬁes the assumption of Lemma 1, i.e. ∥ W ′ ∥ ≤ C 1 , and ∥ W ′ j ∥ ≤ c 2 / (2 √ m 1 ) except possible for indices in P , also deﬁned in 1.

Original: Then, with high probability over initialization
Revised: Then, with high probability over initialization

Original: [Equation]
Revised: [Equation]

Original: which is O ( C 1 ) for large enough overparameterization.
Revised: which is O ( C 1 ) for large enough overparameterization.

Original: Proof of Lemma
Revised: Proof of Lemma

Original: Note the because the VC-dimension of the class of binary functions with respect to halfspaces in R d is d + 1 , the number of different sign patterns D W (0) ,x for different x can be at most m d + .
Revised: Note the because the VC-dimension of the class of binary functions with respect to halfspaces in R d is d + 1 , the number of different sign patterns D W (0) ,x for different x can be at most m d + .

Original: Now similar to Equation (317), for k ∈ [ m 3 ] deﬁne
Revised: Now similar to Equation (317), for k ∈ [ m 3 ] deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, for k 1 ̸ = k 2 , as ∥ x ∥ = 1 :
Revised: Then, for k 1 ̸ = k 2 , as ∥ x ∥ = 1 :

Original: [Equation]
Revised: [Equation]

Original: But for each ﬁxed D W (0) ,x , using Hoeffding bound, we have with probability 1 − δ :
Revised: But for each ﬁxed D W (0) ,x , using Hoeffding bound, we have with probability 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: Applying the above for all possible sign patterns with δ < O (1 /m 1 d +1 ) and a union bound, we have with high probability
Revised: Applying the above for all possible sign patterns with δ < O (1 /m 1 d +1 ) and a union bound, we have with high probability

Original: [Equation]
Revised: [Equation]

Original: We can even state the following stronger bound with respect to two adversarially picked vectors x, x ′ :
Revised: We can even state the following stronger bound with respect to two adversarially picked vectors x, x ′ :

Original: [Equation]
Revised: [Equation]

Original: because each D W (0) ,x ′ j,j has at most m d + cases as we discussed above, then D W (0) ,xj,j D W (0) ,x ′ j,j has at most m d + possible cases, and applying a similar Hoeffding bound for each of them and a union bound as we did will imply (318).
Revised: because each D W (0) ,x ′ j,j has at most m d + cases as we discussed above, then D W (0) ,xj,j D W (0) ,x ′ j,j has at most m d + possible cases, and applying a similar Hoeffding bound for each of them and a union bound as we did will imply (318).

Original: We will use this generalized version in another section.
Revised: We will use this generalized version in another section.

Original: Now combining Equation (318) with the fact that ∥ W ′ ∥ ≤ C 1 and applying Lemma 40:
Revised: Now combining Equation (318) with the fact that ∥ W ′ ∥ ≤ C 1 and applying Lemma 40:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, setting m 2 = m 1 , m 

Revised: On the other hand, setting m 2 = m 1 , m

Original: = d , and R =
Revised: = d , and R =

Original: c 2 / (2 √ m 1 κ 1 ) in Lemma 29, we get with high probability
Revised: c 2 / (2 √ m 1 κ 1 ) in Lemma 29, we get with high probability

Original: [Equation]
Revised: [Equation]

Original: Noting that ∥ W ′ j ∥ ≤ c 2 / (2 √ m 1 ) for j / ∈ P , we conclude that with high probability, for any x , 1 { ( W (0)
Revised: Noting that ∥ W ′ j ∥ ≤ c 2 / (2 √ m 1 ) for j / ∈ P , we conclude that with high probability, for any x , 1 { ( W (0)

Original: + W ′ )
Revised: + W ′ )

Original: Tj x ≥ 0 } and 1 { W (0)
Revised: Tj x ≥ 0 } and 1 { W (0)

Original: j T x ≥ 0 } can be different in at most √ m 1 c 2 / (2 κ 1 ) of the j ’s outside of [ m 1 ] \ P .
Revised: j T x ≥ 0 } can be different in at most √ m 1 c 2 / (2 κ 1 ) of the j ’s outside of [ m 1 ] \ P .

Original: Therefore, as we have also | P | ≲ nc 2 √ m 1 /κ 1 from Lemma 1, we conclude that with high probability, for any x , there are at most O ( nc 2 √ m 1 /κ 1 ) sign changes by adding W ′ to W (0) .
Revised: Therefore, as we have also | P | ≲ nc 2 √ m 1 /κ 1 from Lemma 1, we conclude that with high probability, for any x , there are at most O ( nc 2 √ m 1 /κ 1 ) sign changes by adding W ′ to W (0) .

Original: This further implies:
Revised: This further implies:

Original: [Equation]
Revised: [Equation]

Original: Combining this with (319), we conclude with high probability:
Revised: Combining this with (319), we conclude with high probability:

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.