Original: Suppose Assumption F.3 is satisﬁed with R ≤ O ( 1 L 4 .
Revised: Suppose Assumption F.3 is satisﬁed with R ≤ O ( 1 L 4 .

Original: 5 log 3 m ) and m ≥ Ω( nL log( nL )) , with probability at least 1 − e − Ω(log 2 m ) over the randomness of W (0) and a , for all i ∈ [ n ] , (cid:96) ∈
Revised: 5 log 3 m ) and m ≥ Ω( nL log( nL )) , with probability at least 1 − e − Ω(log 2 m ) over the randomness of W (0) and a , for all i ∈ [ n ] , (cid:96) ∈

Original: [ L ] .
Revised: [ L ] .

Original: Deﬁneu :
Revised: Deﬁne

Revised: [Equation]

Revised: [Equation]

Revised: u

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It satisﬁes that
Revised: It satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that we can invoke Corollary F.13 to give an upper bound on (cid:107) u (cid:107) 2 with probability 1 − e − Ω(log 2 m ) ,
Revised: Note that we can invoke Corollary F.13 to give an upper bound on (cid:107) u (cid:107) 2 with probability 1 − e − Ω(log 2 m ) ,

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can use Lemma F.9 to give an upper bound on (cid:107) u 0 (cid:107) 2 with probability 1 − e − Ω(log 2 m ) ,
Revised: Similarly, we can use Lemma F.9 to give an upper bound on (cid:107) u 0 (cid:107) 2 with probability 1 − e − Ω(log 2 m ) ,

Original: [Equation]
Revised: [Equation]

Original: Finally, by triangle inequality, we get
Revised: Finally, by triangle inequality, we get

Original: [Equation]
Revised: [Equation]

Original: with the desired probability.
Revised: with the desired probability.

Original: F. B OUNDS ON INITIALIZATION
Revised: F. B OUNDS ON INITIALIZATION

Original: In the following lemma, we generalize the Lemma C.2 in Brand et al.
Revised: In the following lemma, we generalize the Lemma C.2 in Brand et al.

Original: (2021) into multiple layer neural networks.
Revised: (2021) into multiple layer neural networks.

Original: Lemma F.15 (Bounds on initialization, multiple layer version of Lemma C.2 in Brand et al. (2021)) .
Revised: Lemma F.15 (Bounds on initialization, multiple layer version of Lemma C.2 in Brand et al. (2021)) .

Original: Suppose m = Ω( nL log( nL )) , then with probability 1 − O ( nL ) · e − Ω( k/L 2 ) , we have the following
Revised: Suppose m = Ω( nL log( nL )) , then with probability 1 − O ( nL ) · e − Ω( k/L 2 ) , we have the following

Original: • f ( W, x i ) =
Revised: • f ( W, x i ) =

Original: O (1) , ∀ i ∈ [ n ] .
Revised: O (1) , ∀ i ∈ [ n ] .

Original: • With probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) J (cid:96), 0 ,i (cid:107) =
Revised: • With probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) J (cid:96), 0 ,i (cid:107) =

Original: (cid:101) O
Revised: (cid:101) O

Original: ( (cid:112) L/m ) , ∀ (cid:96)
Revised: ( (cid:112) L/m ) , ∀ (cid:96)

Original: ∈
Revised: ∈

Original: [ L ] , ∀ i ∈ [ n ] .
Revised: [ L ] , ∀ i ∈ [ n ] .

Original: Remark F.16.
Revised: Remark F.16.

Original: The bound of m in Brand et al.
Revised: The bound of m in Brand et al.

Original: (2021) has a linear dependence on n because they need to bound (cid:107) W (0) (cid:107) 2 .
Revised: (2021) has a linear dependence on n because they need to bound (cid:107) W (0) (cid:107) 2 .

Original: If we do not need to bound (cid:107) W (0) (cid:107) 2 , then m does not depend linearly on n .
Revised: If we do not need to bound (cid:107) W (0) (cid:107) 2 , then m does not depend linearly on n .

Original: Proof.
Revised: Proof.

Original: We will prove the two parts of the statement separately.
Revised: We will prove the two parts of the statement separately.

Original: Part 1: By deﬁnition, for any i ∈ [ n ] , we have
Revised: Part 1: By deﬁnition, for any i ∈ [ n ] , we have

Original: [Equation]
Revised: [Equation]

Original: We shall make use Lemma F.7 here: with probability at least 1 − O ( nL ) · exp( − Ω( k/L 2 )) , we have (cid:107) h
Revised: We shall make use Lemma F.7 here: with probability at least 1 − O ( nL ) · exp( − Ω( k/L 2 )) , we have (cid:107) h

Original: i,L (cid:107) 2 ∈ [0 . 9 , 1 . 1] .
Revised: i,L (cid:107) 2 ∈ [0 . 9 , 1 . 1] .

Original: Recall that a ∈ R m has each of its entry being a Rademacher random variable scaled by 1 /m , hence it’s 1 /m -subgaussian.
Revised: Recall that a ∈ R m has each of its entry being a Rademacher random variable scaled by 1 /m , hence it’s 1 /m -subgaussian.

Original: Using the concentration of subgaussian (Lemma A.5), we have that
Revised: Using the concentration of subgaussian (Lemma A.5), we have that

Original: [Equation]
Revised: [Equation]

Original: exp( − t
Revised: exp( − t

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:107) h
Revised: 2 (cid:107) h

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: L (cid:107) 2 ) , ﬁnally by noticing that t = Θ(1) and (cid:107) h
Revised: L (cid:107) 2 ) , ﬁnally by noticing that t = Θ(1) and (cid:107) h

Original: i,L (cid:107) 2 ∈ [0 . 9 , 1 . 1] , we conclude that
Revised: i,L (cid:107) 2 ∈ [0 . 9 , 1 . 1] , we conclude that

Original: Pr[
Revised: Pr[

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: O
Revised: O

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − Ω
Revised: − Ω

Original: [Equation]
Revised: [Equation]

Original: m )) , union bounding over all
Revised: m )) , union bounding over all

Original: i , we conclude that
Revised: i , we conclude that

Original: [Equation]
Revised: [Equation]

Original: Part 2: We will combine Lemma F.7 and F.9, with probability at least 1 − O ( nL ) · e − Ω( k/L 2 ) , we have (cid:107) h i,(cid:96) (cid:107) 2 ∈
Revised: Part 2: We will combine Lemma F.7 and F.9, with probability at least 1 − O ( nL ) · e − Ω( k/L 2 ) , we have (cid:107) h i,(cid:96) (cid:107) 2 ∈

Original: 0 . 9 , 1 . 1] and with probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) a (cid:62) D i,L . . .
Revised: 0 . 9 , 1 . 1] and with probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) a (cid:62) D i,L . . .

Original: W (cid:96) (cid:107) 2 ≤ (cid:101) O
Revised: W (cid:96) (cid:107) 2 ≤ (cid:101) O

Original: ( (cid:112) L/m ) .
Revised: ( (cid:112) L/m ) .

Original: Combine them together, we know that with probability at least 1 − e − Ω(log 2 m ) , (cid:107) J (cid:96), 0 ,i (cid:107) ≤ (cid:101) O
Revised: Combine them together, we know that with probability at least 1 − e − Ω(log 2 m ) , (cid:107) J (cid:96), 0 ,i (cid:107) ≤ (cid:101) O

Original: ( (cid:112) L/m ) .
Revised: ( (cid:112) L/m ) .

Original: Remark F.17.
Revised: Remark F.17.

Original: By utilizing the structure of vector a , we can show that with a weaker probability ( 1 − e − Ω(log 2 m ) ), f ( W, x i ) has even smaller magnitude ( (cid:101) O
Revised: By utilizing the structure of vector a , we can show that with a weaker probability ( 1 − e − Ω(log 2 m ) ), f ( W, x i ) has even smaller magnitude ( (cid:101) O

Original: ( 1 √ m ) ).
Revised: ( 1 √ m ) ).

Original: For our purpose, f ( W, x i ) =
Revised: For our purpose, f ( W, x i ) =

Original: O (1) sufﬁces.
Revised: O (1) sufﬁces.

Original: F. B OUNDS ON SMALL PERTURBATION
Revised: F. B OUNDS ON SMALL PERTURBATION

Original: In the following, we generalize the Lemma C.4 in Brand et al.
Revised: In the following, we generalize the Lemma C.4 in Brand et al.

Original: (2021) into multiple layer neural network.
Revised: (2021) into multiple layer neural network.

Original: For the simplicity of notation, we set all m (cid:96) to be m .
Revised: For the simplicity of notation, we set all m (cid:96) to be m .

Original: Lemma F.18 (multiple layer version of Lemma C.4 in Brand et al. (2021)) .
Revised: Lemma F.18 (multiple layer version of Lemma C.4 in Brand et al. (2021)) .

Original: Suppose R ≤
Revised: Suppose R ≤

Original: O ( 1 L 4 .
Revised: O ( 1 L 4 .

Original: 5 log 3 m ) and m = Ω( nL log( nL )) .
Revised: 5 log 3 m ) and m = Ω( nL log( nL )) .

Original: With probability at least 1 − e − Ω(log 2 m ) over the random initialization of W (0)
Revised: With probability at least 1 − e − Ω(log 2 m ) over the random initialization of W (0)

Original: = { W 1 (0) , W 2 (0) , · · · W L (0) } , the following holds for any set of weights
Revised: = { W 1 (0) , W 2 (0) , · · · W L (0) } , the following holds for any set of weights

Original: W 1 , · · · , W L satisfying
Revised: W 1 , · · · , W L satisfying

Original: [Equation]
Revised: [Equation]

Original: • (cid:107) J W (cid:96) − J W (cid:96) (0) (cid:107) F
Revised: • (cid:107) J W (cid:96) − J W (cid:96) (0) (cid:107) F

Original: =
Revised: =

Original: cid:101) O
Revised: cid:101) O

Original: ( n 1 / 2 (cid:112) L/m ) .
Revised: ( n 1 / 2 (cid:112) L/m ) .

Original: • (cid:107) J W
Revised: • (cid:107) J W

Original: (cid:96) (cid:107) F
Revised: (cid:96) (cid:107) F

Original: = (cid:101) O ( n 1 / 2 (cid:112) L/m ) .
Revised: = (cid:101) O ( n 1 / 2 (cid:112) L/m ) .

Original: Proof.
Revised: Proof.

Original: Part 1.
Revised: Part 1.

Original: To simplify the notation
Revised: To simplify the notation

Original: [Equation]
Revised: [Equation]

Original: we ignore the subscripts i below
Revised: we ignore the subscripts i below

Original: [Equation]
Revised: [Equation]

Original: Consider the following computation:
Revised: Consider the following computation:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: + ( W k (0) + ∆ W k ) (cid:62) ( D k (0) + ∆ D k ) (cid:33) a u 0 :=
Revised: + ( W k (0) + ∆ W k ) (cid:62) ( D k (0) + ∆ D k ) (cid:33) a u 0 :=

Original: D (cid:96) (0) (cid:32) L (cid:89) k
Revised: D (cid:96) (0) (cid:32) L (cid:89) k

Original: = (cid:96) +
Revised: = (cid:96) +

Original: W k (0) (cid:62) D k (0) (cid:33)
Revised: W k (0) (cid:62) D k (0) (cid:33)

Original: a
Revised: a

Original: By Lemma F.14, we know that with probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) u − u 0 (cid:107) 2 ≤ (cid:101) O ( (cid:112) L/m ) .
Revised: By Lemma F.14, we know that with probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) u − u 0 (cid:107) 2 ≤ (cid:101) O ( (cid:112) L/m ) .

Original: By Lemma F.7, we know that with probability at least 1 − O ( nL ) ·
Revised: By Lemma F.7, we know that with probability at least 1 − O ( nL ) ·

Original: e − k/L 2 , we have (cid:107) h (cid:96) − 1 (0) (cid:107) 2 ∈
Revised: e − k/L 2 , we have (cid:107) h (cid:96) − 1 (0) (cid:107) 2 ∈

Original: By Corollary F.13, we know that with probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) u (cid:107) 2 ≤ (cid:101) O
Revised: By Corollary F.13, we know that with probability at least 1 − e − Ω(log 2 m ) , we have (cid:107) u (cid:107) 2 ≤ (cid:101) O

Original: ( (cid:112) L/m ) .
Revised: ( (cid:112) L/m ) .

Original: By Lemma F.10, we know that with probability at least 1 − e − Ω( mR 2 / 3 L ) , we have (cid:107) ∆ h (cid:96) − 1 (cid:107) 2 ≤ (cid:101) O ( RL 2 . 5 ) .
Revised: By Lemma F.10, we know that with probability at least 1 − e − Ω( mR 2 / 3 L ) , we have (cid:107) ∆ h (cid:96) − 1 (cid:107) 2 ≤ (cid:101) O ( RL 2 . 5 ) .

Original: Note that due to the choice of R , we know that (cid:107) ∆ h (cid:96) − 1 (cid:107) 2 ≤ 1 .
Revised: Note that due to the choice of R , we know that (cid:107) ∆ h (cid:96) − 1 (cid:107) 2 ≤ 1 .

Original: Taking a union bound over all events, with probability at least 1 − e − Ω(log 2 m ) , we achieve the following bound:
Revised: Taking a union bound over all events, with probability at least 1 − e − Ω(log 2 m ) , we achieve the following bound:

Original: [Equation]
Revised: [Equation]

Original: Part 2.
Revised: Part 2.

Original: Note that the squared Frobenious norm is just the sum of all squared (cid:96) 2 norm of rows, hence 
Revised: Note that the squared Frobenious norm is just the sum of all squared (cid:96) 2 norm of rows, hence (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Part 3.
Revised: Part 3.

Original: We will prove by triangle inequality:
Revised: We will prove by triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: F. 
Revised: F. P UTTING IT ALL TOGETHER

Original: P UTTING IT ALL TOGETHER

Original: In this section, we will prove the following core theorem that analyzes the convergence behavior of
Revised: In this section, we will prove the following core theorem that analyzes the convergence behavior of

Original: Algorithm 2:
Revised: Algorithm 2:

Original: Theorem F.19 (Formal version of Theorem 1.1) .
Revised: Theorem F.19 (Formal version of Theorem 1.1) .

Original: Suppose the width of the neural network satisﬁes m = Ω( λ − 2 L n 2 L 2 ) , then with probability at least 1 − e − Ω(log 2 m ) over the randomness of the initialization of the neural network and the randomness of the algorithm, Algorithm 2 satisﬁes
Revised: Suppose the width of the neural network satisﬁes m = Ω( λ − 2 L n 2 L 2 ) , then with probability at least 1 − e − Ω(log 2 m ) over the randomness of the initialization of the neural network and the randomness of the algorithm, Algorithm 2 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Before moving on, we introduce several deﬁnitions and prove some useful facts related to them.
Revised: Before moving on, we introduce several deﬁnitions and prove some useful facts related to them.

Original: Deﬁnition F.20 (function J ) .
Revised: Deﬁnition F.20 (function J ) .

Original: We deﬁne
Revised: We deﬁne

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Revised: D

Revised: [Equation]

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: Fact F.21.
Revised: Fact F.21.

Original: Let J (cid:96) denote the function be deﬁned as Deﬁnition F.20.
Revised: Let J (cid:96) denote the function be deﬁned as Deﬁnition F.20.

Original: For any t ∈ { 0 , . . .
Revised: For any t ∈ { 0 , . . .

Original: , T } , we have
Revised: , T } , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For i ∈ [ n ] , consider the i -th coordinate.
Revised: For i ∈ [ n ] , consider the i -th coordinate.

Original: [Equation]
Revised: [Equation]

Original: Thus, we complete the proof.
Revised: Thus, we complete the proof.

Original: Fact F.22.
Revised: Fact F.22.

Original: For any t ∈ { 0 , . . .
Revised: For any t ∈ { 0 , . . .

Original: , T } , we have J (cid:96) ( W 1 ( t ) , . . .
Revised: , T } , we have J (cid:96) ( W 1 ( t ) , . . .

Original: , W L ( t ))
Revised: , W L ( t ))

Original: =
Revised: =

Original: J (cid:96),t .
Revised: J (cid:96),t .

Original: Proof.
Revised: Proof.

Original: In order to simplify the notation, we drop the term t below.
Revised: In order to simplify the notation, we drop the term t below.

Original: We note that for i ∈ [ n ] , the i -th row of matrix J (cid:96),t is deﬁned as
Revised: We note that for i ∈ [ n ] , the i -th row of matrix J (cid:96),t is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where D i,(cid:96) =
Revised: k D i,k ) ah (cid:62) i,(cid:96) − 1 , where D i,(cid:96) =

Original: diag( φ (cid:48)
Revised: diag( φ (cid:48)

Original: ( W (cid:96) h i,(cid:96) − 1 )) and h i,(cid:96) − 1 = φ ( W (cid:96) − 1 ( φ ( W (cid:96) − 2 . . .
Revised: ( W (cid:96) h i,(cid:96) − 1 )) and h i,(cid:96) − 1 = φ ( W (cid:96) − 1 ( φ ( W (cid:96) − 2 . . .

Original: ( φ ( W 1 x i ))))) , this is essentially the same as h i ( W 1 , . .
Revised: ( φ ( W 1 x i ))))) , this is essentially the same as h i ( W 1 , . .

Original: , W (cid:96) − 1 ) and D i,(cid:96) ( W (cid:96) ) .
Revised: , W (cid:96) − 1 ) and D i,(cid:96) ( W (cid:96) ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: We state the range we require for parameter R :
Revised: We state the range we require for parameter R :

Original: Deﬁnition F.23.
Revised: Deﬁnition F.23.

Original: We choose R so that
Revised: We choose R so that

Original: [Equation]
Revised: [Equation]

Original: Remark F.24.
Revised: Remark F.24.

Original: Recall that the sparsity parameter s is directly related to R : s =
Revised: Recall that the sparsity parameter s is directly related to R : s =

Original: O ( mR 2 / 3 L ) , hence to ensure the sparsity is small
Revised: O ( mR 2 / 3 L ) , hence to ensure the sparsity is small

Original: , we shall pick R as small as possible.
Revised: , we shall pick R as small as possible.

Original: Speciﬁcally, if we pick R to be n √ mLλ L , then s =
Revised: Speciﬁcally, if we pick R to be n √ mLλ L , then s =

Original: O ( λ − 1 L n √ mL ) , as long as m (cid:29) λ − 2 L n 2 L then s ≈ O ( √ m ) .
Revised: O ( λ − 1 L n √ mL ) , as long as m (cid:29) λ − 2 L n 2 L then s ≈ O ( √ m ) .

Original: Next, we pick the value of m :
Revised: Next, we pick the value of m :

Original: Deﬁnition F.25.
Revised: Deﬁnition F.25.

Original: We choose m to be
Revised: We choose m to be

Original: [Equation]
Revised: [Equation]

Original: Remark F.26.
Revised: Remark F.26.

Original: The choice of m here makes sure that, as long as we pick R matching its lower bound, then the sparsity s =
Revised: The choice of m here makes sure that, as long as we pick R matching its lower bound, then the sparsity s =

Original: O ( m 0 . 8 ) , this matches the other sparsity parameter k , which is also in the order of O ( m 0 . 8 ) .
Revised: O ( m 0 . 8 ) , this matches the other sparsity parameter k , which is also in the order of O ( m 0 . 8 ) .

Original: We use induction to prove the following two claims recursively.
Revised: We use induction to prove the following two claims recursively.

Original: Deﬁnition F.27 (Induction hypothesis 1) .
Revised: Deﬁnition F.27 (Induction hypothesis 1) .

Original: Let t ∈
Revised: Let t ∈

Original: [ T ] be a ﬁxed integer.
Revised: [ T ] be a ﬁxed integer.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: holds for any (cid:96) ∈
Revised: holds for any (cid:96) ∈

Original: [ L ] .
Revised: [ L ] .

Original: Deﬁnition F.28 (Induction Hypothesis 2) .
Revised: Deﬁnition F.28 (Induction Hypothesis 2) .

Original: Let t ∈
Revised: Let t ∈

Original: [ T ] be a ﬁxed integer.
Revised: [ T ] be a ﬁxed integer.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Suppose the above two claims hold up to t , we prove they continue to hold for time t
Revised: Suppose the above two claims hold up to t , we prove they continue to hold for time t

Original: + 1 .
Revised: + 1 .

Original: The second claim is more delicate, we are going to prove it ﬁrst and we deﬁne
Revised: The second claim is more delicate, we are going to prove it ﬁrst and we deﬁne

Original: [Equation]
Revised: [Equation]

Original: where J (cid:96) is deﬁned as Deﬁnition F.20.
Revised: where J (cid:96) is deﬁned as Deﬁnition F.20.

Original: Lemma F.29.
Revised: Lemma F.29.

Original: Let g (cid:63)(cid:96) :
Revised: Let g (cid:63)(cid:96) :

Original: = ( J (cid:96),t J (cid:62) (cid:96),t ) − 1 · 1 L ( f t − y ) .
Revised: = ( J (cid:96),t J (cid:62) (cid:96),t ) − 1 · 1 L ( f t − y ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: f

Original: [Equation]

Original: − y

Original: [Equation]

Original: ≤

Original: [Equation]

Original: − y −

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider the following computation:
Revised: Consider the following computation:

Original: [Equation]
Revised: [Equation]

Original: (cid:107) [Equation]
Revised: =

Original: [Equation]
Revised: [Equation]

Original:  (cid:107)
Revised: = (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: −

Original: [Equation]
Revised: [Equation]

Revised: (cid:96)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)= (cid:107) f t − y −
Revised: (cid:107)= (cid:107) f t − y −

Original: L (cid:88) (cid:96) [Equation]
Revised: L (cid:88) (cid:96) =

Original: [Equation] (cid:96),t [Equation] (cid:96),t 
Revised: J (cid:96),t J (cid:62) (cid:96),t g (cid:96),t +

Original: [Equation] (cid:96) =
Revised: L (cid:88) (cid:96) =

Original: [Equation] (cid:96),t [Equation] (cid:96),t −
Revised: J (cid:96),t J (cid:62) (cid:96),t g (cid:96),t −

Original: L (cid:88) (cid:96) =J (cid:96) [Equation]
Revised: L (cid:88) (cid:96) =

Original: t,t + [Equation] (cid:62) (cid:96),t g (cid:96),t (cid:107)≤ (cid:107) f t − y −
Revised: J (cid:96),t,t +1 J (cid:62) (cid:96),t g (cid:96),t (cid:107)≤ (cid:107) f t − y −

Original: L (cid:88) (cid:96) =
Revised: L (cid:88) (cid:96) =

Original: J (cid:96),t J (cid:62) (cid:96),t g (cid:96),t (cid:107) 2 +
Revised: J (cid:96),t J (cid:62) (cid:96),t g (cid:96),t (cid:107) 2 +

Original: L (cid:88) (cid:96) = (cid:107) ( J (cid:96),t − J (cid:96),t,t +1 ) J (cid:62) (cid:96),t g (cid:96),t (cid:107)≤ (cid:107) f t − y −
Revised: L (cid:88) (cid:96) = (cid:107) ( J (cid:96),t − J (cid:96),t,t +1 ) J (cid:62) (cid:96),t g (cid:96),t (cid:107)≤ (cid:107) f t − y −

Original: L (cid:88) (cid:96) =
Revised: L (cid:88) (cid:96) =

Original: J (cid:96),t J (cid:62) (cid:96),t

Original: [Equation]

Original:  (cid:96),t (cid:107) 2 +L (cid:88) (cid:96) = (cid:107) ( [Equation] (cid:96),t −
Revised: J (cid:96),t J (cid:62) (cid:96),t g (cid:96),t (cid:107) 2 +

Original: [Equation]

Original:  (cid:96),t,t +1 ) [Equation] (cid:62) (cid:96),t [Equation]
Revised: L (cid:88) (cid:96) = (cid:107) ( J (cid:96),t − J (cid:96),t,t +1 ) J (cid:62) (cid:96),t g (cid:63)(cid:96) (cid:107) 2 +

Original: [-(cid:63)(cid:96) (cid:107) 
 +-] (cid:107) ( [Equation] (cid:96),t − J (cid:96),t,t +1 ) J (cid:62) (cid:96),t ( g (cid:96),t − g (cid:63)(cid:96) )
Revised: [-(cid:63)(cid:96) (cid:107) 
 +-]L (cid:88) (cid:96) = (cid:107) ( J (cid:96),t − J (cid:96),t,t +1 ) J (cid:62) (cid:96),t ( g (cid:96),t − g (cid:63)(cid:96) )

Original: (cid:107) 2 ,
Revised: (cid:107) 2 ,

Original: The second step follows from thedeﬁnition ofJ (cid:96),t,t +1 and simple calculus.
Revised: The second step follows from the of

Original: Claim F.30 (1st term in Eq.
Revised: Claim F.30 (1st term in Eq.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: since g (cid:96),t is an (cid:15) 

Revised: since g (cid:96),t is an (cid:15)

Original: ( (cid:15) 0 ≤ 16 ) approximate solution to regression problem
Revised: ( (cid:15) 0 ≤ 16 ) approximate solution to regression problem

Original: [Equation]
Revised: [Equation]

Original: Claim F.31 (2nd term in Eq.
Revised: Claim F.31 (2nd term in Eq.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the second term in Eq.
Revised: For the second term in Eq.

Original: [Equation] for any (cid:96) ∈
Revised: (8), for any (cid:96) ∈

Original: [ L ]
Revised: [ L ]

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We bound these term separately.
Revised: We bound these term separately.

Original: First,
Revised: First,

Original: [Equation]
Revised: [Equation]

Original: where by Fact F.22, we know (cid:107) J
Revised: where by Fact F.22, we know (cid:107) J

Original: (cid:96) ( W t ) − J
Revised: (cid:96) ( W t ) − J

Original: [Equation]
Revised: [Equation]

Original: W 0 )
Revised: W 0 )

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = (cid:107) J W

Original: [Equation]
Revised: [Equation]

Original:  J W
Revised: − J W

Original: [Equation]
Revised: [Equation]

Original: t ) − J W

Original: [Equation]
Revised: [Equation]

Original: 0) (cid:107) ≤ (cid:101) O 
Revised: ≤ (cid:101) O [Equation]

Original: ( n 1 / 2 (cid:112) L/m ) , the last inequality is by Lemma F.18.
Revised: n 1 / 2 (cid:112) L/m ) , the last inequality is by Lemma F.18.

Original: For the second term, we have
Revised: For the second term, we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: =

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: ( t ) − W (cid:96) (0) (cid:107) F + s · (cid:107) W (cid:96) ( t + 1) − W (cid:96) (0) (cid:107) F ≤ O ( R ) .
Revised: F ≤ O ( R ) .

Original: This means the perturbation of (1 − s ) W (cid:96) ( t ) + sW (cid:96) ( t + 1) with respect to W (cid:96) (0) is small, for any (cid:96) ∈
Revised: This means the perturbation of (1 − s ) W (cid:96) ( t ) + sW (cid:96) ( t + 1) with respect to W (cid:96) (0) is small, for any (cid:96) ∈

Original: [ L ] , hence (cid:107) J (cid:96) ( W 0 )
Revised: [ L ] , hence (cid:107) J (cid:96) ( W 0 )

Original: − J (cid:96) ((1 − s ) W t + sW t +1 ) (cid:107) =
Revised: − J (cid:96) ((1 − s ) W t + sW t +1 ) (cid:107) =

Original: (cid:101) O
Revised: (cid:101) O

Original: ( n 1 / 2 (cid:112) L/m ) .
Revised: ( n 1 / 2 (cid:112) L/m ) .

Original: Furthermore, we have
Revised: Furthermore, we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from σ min ( J (cid:96),t )
Revised: where the second inequality follows from σ min ( J (cid:96),t )

Original: = (cid:113) λ min ( J (cid:62) (cid:96),t J  (cid:96),t ) ≥ (cid:112) λ
Revised: = (cid:113) λ min ( J (cid:96) [Equation] J (cid:62) (cid:96),t ) ≥ (cid:112) λ

Original: L / 2 (see
Revised: L / 2 (see

Original: Lemma E.4).
Revised: Lemma E.3).

Original: Combining Eq.
Revised: Combining Eq.

Original: (10), (11) and (12), we have
Revised: (10), (11) and (12), we have

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from choice of m (Deﬁnition F.25).
Revised: where the last step follows from choice of m (Deﬁnition F.25).

Original: Claim F.32 (3rd term in Eq.
Revised: Claim F.32 (3rd term in Eq.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: We can show
Revised: We can show

Original: [Equation]
Revised: [Equation]

Original: Moreover, one has
Revised: Moreover [Equation] one has

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst step comes from λ min ( J (cid:96),t J (cid:62) (cid:96),t ) =
Revised: The ﬁrst step comes from λ min ( J (cid:96),t J (cid:62) (cid:96),t ) =

Original: λ min ( G (cid:96),t ) ≥ λ L / 2 (see Lemma E.4) and the last step comes from g t is an (cid:15) 0 approximate solution.
Revised: λ min ( G (cid:96),t ) ≥ λ L / 2 (see Lemma E.3) and the last step comes from g t is an (cid:15) 0 approximate solution.

Original: The fourth step follows from Eq.
Revised: The fourth step follows from Eq.

Original: (15) and the fact that (cid:107) ( J (cid:96),t J (cid:62) (cid:96),t ) − 1 (cid:107) ≤ 2 /λ L .
Revised: (15) and the fact that (cid:107) ( J (cid:96),t J (cid:62) (cid:96),t ) − 1 (cid:107) ≤ 2 /λ L .

Original: The last step follows from g t is an (cid:15) 

Revised: The last step follows from g t is an (cid:15)

Original: ( (cid:15) 0 ≤ (cid:112) λ L /n ) approximate solution to the regression.
Revised: ( (cid:15) 0 ≤ (cid:112) λ L /n ) approximate solution to the regression.

Original: Consequently, we have
Revised: Consequently, we have

Original: [Equation]
Revised: [Equation]

Revised: ,t

Revised: [Equation]

Revised: (cid:96)

Revised: [Equation]

Revised: t +

Revised: [Equation]

Original: The second step follows from Eq.
Revised: The second step follows from Eq.

Original: (11) and (15) and the fact that (cid:107) J (cid:96),t (cid:107) ≤  O ( (cid:112) nL/m ) (see
Revised: (11) and (15) and the fact that (cid:107) J (cid:96),t (cid:107) ≤ (cid:101) O ( (cid:112) nL/m ) (see

Original: Lemma F.18).
Revised: Lemma F.18)

Revised: [Equation]

Revised: Finally

Revised: [Equation]

Revised: we have

Revised: [Equation]

Original: The last step follows from choice of m (Deﬁnition F.25).
Revised: The last step follows from choice of m (Deﬁnition F.25).

Original: Lemma F.33 (Putting it all together) .
Revised: Lemma F.33 (Putting it all together) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Combining Eq.
Revised: Combining Eq.

Original: (8), (9), (13), and (16), we have proved the second claim, i.e.,
Revised: (8), (9), (13), and ( ?? ), we have proved the second claim, i [Equation] e.,

Original: [Equation]
Revised: [Equation]

Original: F. W EIGHTS ARE NOT MOVING TOO FAR
Revised: F. W EIGHTS ARE NOT MOVING TOO FAR

Original: Lemma F.34.
Revised: Lemma F.34.

Original: Let R be chosen as in Deﬁnition F.23, then the following holds:
Revised: Let R be chosen as in Deﬁnition F.23, then the following holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It remains to show that W t does not move far away from W 0 .
Revised: It remains to show that W t does not move far away from W 0 .

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: 18) where the third step follows from Eq.
Revised: where the third step follows from Eq.

Original: (15) and the last step follows from the obvious fact that 1 / √ nλ L ≤ 1 /λ L .
Revised: (15) and the last step follows from the obvious fact that 1 / √ nλ L ≤ 1 /λ L .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The third step uses the fact that (cid:107) J (cid:96),k (cid:107) ≤ (cid:107) J (cid:96),k (cid:107) F ≤ O ( (cid:112) nLm ) by Lemma F.18, and the last step uses the fact that both (cid:107) f 0 (cid:107) 2 and (cid:107) y (cid:107) 2 are in the order of O (1) .
Revised: The third step uses the fact that (cid:107) J (cid:96),k (cid:107) ≤ (cid:107) J (cid:96),k (cid:107) F ≤ O ( (cid:112) nLm ) by Lemma F.18, and the last step uses the fact that both (cid:107) f 0 (cid:107) 2 and (cid:107) y (cid:107) 2 are in the order of O (1) .

Original: Consequently, we have
Revised: Consequently, we have

Revised: [Equation]

Revised: [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) k

Revised: [Equation]

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: [Equation]

Original: By the choice of R (Deﬁnition F.23)
Revised: By the choice of R (Deﬁnition F.23)

Original: , we know this is upper bounded by R .
Revised: , we know this is upper bounded by R .

Original: This concludes our proof.
Revised: This concludes our proof.

Original: G P ROOF OF L EMMA F. 
Revised: G P ROOF OF L EMMA F.

Original: In this section, we prove a technical lemma (Lemma F.7) involving truncated gaussian distribution, which correlates to the shifted ReLU activation we use.
Revised: In this section, we prove a technical lemma (Lemma F.7) involving truncated gaussian distribution, which correlates to the shifted ReLU activation we use.

Original: Deﬁnition G.1 (Truncated Gaussian distribution) .
Revised: Deﬁnition G.1 (Truncated Gaussian distribution) .

Original: Suppose X ∼ N (0 , σ 2 ) .
Revised: Suppose X ∼ N (0 , σ 2 ) .

Original: Let b ∈ R .
Revised: Let b ∈ R .

Original: Then, we say a random variable Y follows from a truncated Gaussian distribution N b (0 , σ 2 )
Revised: Then, we say a random variable Y follows from a truncated Gaussian distribution N b (0 , σ 2 )

Original: if Y = X | X ≥ b .
Revised: if Y = X | X ≥ b .

Original: The probability density function for N b (0 , σ 2 ) is as follows:
Revised: The probability density function for N b (0 , σ 2 ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: where Φ( · ) is the standard Gaussian distribution’s CDF.
Revised: where Φ( · ) is the standard Gaussian distribution’s CDF.

Original: Fact G.2 (Properties of truncated Gaussian distribution) .
Revised: Fact G.2 (Properties of truncated Gaussian distribution) .

Original: For b ∈ R , suppose X ∼ N b (0 , σ 2 ) .
Revised: For b ∈ R , suppose X ∼ N b (0 , σ 2 ) .

Original: Let β :=
Revised: Let β :=

Original: b/σ .
Revised: b/σ .

Original: Then, we have
Revised: Then, we have

Original: • E [ X ] = σφ ( β ) 1 − Φ( β ) , where φ ( x ) := 1 √ 2 π e − x 2 / 2 .
Revised: • E [ X ] = σφ ( β ) 1 − Φ( β ) , where φ ( x ) := 1 √ 2 π e − x 2 / 2 .

Original: • Var [ X ] = σ 2 (1 + βφ ( β ) / (1 − Φ( β )) − ( φ ( β ) / (1 − Φ( β ))) 2 ) .
Revised: • Var [ X ] = σ 2 (1 + βφ ( β ) / (1 − Φ( β )) − ( φ ( β ) / (1 − Φ( β ))) 2 ) .

Original: • X/σ ∼ N b/σ (0 , 1) .
Revised: • X/σ ∼ N b/σ (0 , 1) .

Original: • When σ = 1 , X is C ( b + 1) -subgaussian, where C > 0 is an absolute constant.
Revised: • When σ = 1 , X is C ( b + 1) -subgaussian, where C > 0 is an absolute constant.

Original: Lemma G.3 (Concentration inequality for b -truncated chi-square distribution) .
Revised: Lemma G.3 (Concentration inequality for b -truncated chi-square distribution) .

Original: For b ∈ R , n > 0 , let X ∼ χ 2 b,n ; that is, X = (cid:80) ni
Revised: For b ∈ R , n > 0 , let X ∼ χ 2 b,n ; that is, X = (cid:80) ni

Original: =1 Y 2 i where Y 1 , . . .
Revised: =1 Y 2 i where Y 1 , . . .

Original: , Y n ∼ N b (0 , 1) are independent b -truncated
Revised: , Y n ∼ N b (0 , 1) are independent b -truncated

Original: Gaussian random variables.
Revised: Gaussian random variables.

Original: Then, there exist two constants C 1 , C 2 such that for any t > 0 ,
Revised: Then, there exist two constants C 1 , C 2 such that for any t > 0 ,

Original: [Equation]
Revised: [Equation]

Original: In particular, we have
Revised: In particular, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since we know that Y i ∼ N b (0 , 1) is C ( b + 1) -subgaussian, it implies that
Revised: Since we know that Y i ∼ N b (0 , 1) is C ( b + 1) -subgaussian [Equation]

Original:  Y 2 i is a subexponential random variable with parameters (4 √ 2 C 2 ( b +1) 2 , 4 C 2 ( b +1) 2 ) .
Revised: it implies that Y 2 i is a subexponential random variable with parameters (4 √ 2 C 2 ( b +1) 2 , 4 C 2 ( b +1) 2 ) .

Original: Hence, by the standard concentration of sub-exponential random variables, we have
Revised: Hence, by the standard concentration of sub-exponential random variables, we have

Original: [Equation]
Revised: [Equation]

Original: otherwise

Original: [Equation]

Original: Fact G.4.
Revised: Fact G.4.

Original: Let h ∈ R p be ﬁxed vectors and h (cid:54) = 0 , let b > 0 be a ﬁxed scalar, W ∈ R m × p be random matrix with i.i.d.
Revised: Let h ∈ R p be ﬁxed vectors and h (cid:54) = 0 , let b > 0 be a ﬁxed scalar, W ∈ R m × p be random matrix with i

Original:  entries W i,j ∼ N (0 , 2 m ) and vector v ∈ R m deﬁned as v i =
Revised: .i d. entries W i,j ∼ N (0 , 2 m ) and vector v ∈ R m deﬁned as v i =

Original: φ (( Wh ) i ) =
Revised: φ (( Wh ) i ) =

Original: 1 [( Wh )
Revised: 1 [( Wh )

Original: i ≥ b ]( Wh ) i .
Revised: i ≥ b ]( Wh ) i .

Original: Then • | v i | follows i.i
Revised: Then • | v i | follows i.i

Original: d. from the following distribution: with probability 1 − e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) , | v i | = 0 , and with probability e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) , | v i | follows from truncated Gaussian distribution N b (0 , 2 m (cid:107) h (cid:107) 22 ) .
Revised: d. from the following distribution: with probability 1 − e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) , | v i | = 0 , and with probability e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) , | v i | follows from truncated Gaussian distribution N b (0 , 2 m (cid:107) h (cid:107) 22 ) .

Original: • m (cid:107) v (cid:107) 22 2 (cid:107) h (cid:107) [-
-] is in distribution identical to χ 2 b (cid:48) ,ω ( b (cid:48) -truncated chi-square distribution of order ω ) where ω follows from binomial distribution B ( m, e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) ) and b (cid:48)
Revised: • m (cid:107) v (cid:107) 22 2 (cid:107) h (cid:107) [-
-]22 is in distribution identical to χ 2 b (cid:48) ,ω ( b (cid:48) -truncated chi-square distribution of order ω ) where ω follows from binomial distribution B ( m, e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) ) and b (cid:48)

Original: = √ m/(cid:107)
Revised: = √ m/(cid:107)

Original: h (cid:107) 2 b .
Revised: h (cid:107) 2 b .

Original: Proof.
Revised: Proof.

Original: We assume each vector W i is generated by ﬁrst generating a gaussian vector g ∼ N (0 , 2 m I ) and then setting W i = ± g where the sign is chosen with half-half probability.
Revised: We assume each vector W i is generated by ﬁrst generating a gaussian vector g ∼ N (0 , 2 m I ) and then setting W i = ± g where the sign is chosen with half-half probability.

Original: Now, |(cid:104) W i , h (cid:105)| = |(cid:104) g, h (cid:105)| only depends on g , and is in distribution identical to N b (0 , 2 m (cid:107) h (cid:107) 22 ) .
Revised: Now, |(cid:104) W i , h (cid:105)| = |(cid:104) g, h (cid:105)| only depends on g , and is in distribution identical to N b (0 , 2 m (cid:107) h (cid:107) 2 2 ) .

Original: Next, after the sign is determined, the indicator 1 [( W i h )
Revised: Next, after the sign is determined, the indicator 1 [( W i h )

Original: i ≥ b ] is 1 with probability e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) and 0 with probability 1 − e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) .
Revised: i ≥ b ] is 1 with probability e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) and 0 with probability 1 − e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) .

Original: Therefore, | v i | satisﬁes the aforementioned distribution.
Revised: Therefore, | v i | satisﬁes the aforementioned distribution.

Original: As for (cid:107) v (cid:107) 22 , letting ω ∈ { 0 , 1 , . . .
Revised: As for (cid:107) v (cid:107) 22 , letting ω ∈ { 0 , 1 , . . .

Original: , m } be the variable indicates how many indicators are 1, then ω ∼ B ( m, e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) ) and m (cid:107) v (cid:107) 22 2 (cid:107) h (cid:107) 22 ∼ χ 2 b (cid:48) ,ω , where b (cid:48)
Revised: , m } be the variable indicates how many indicators are 1, then ω ∼ B ( m, e − b 2 m/ (4 (cid:107) h (cid:107) 2 ) ) and m (cid:107) v (cid:107) 22 2 (cid:107) h (cid:107) 22 ∼ χ 2 b (cid:48) ,ω , where b (cid:48)

Original: = √ m/(cid:107)
Revised: = √ m/(cid:107)

Original: h (cid:107) 2 b .
Revised: h (cid:107) 2 b .

Original: Fact G.5 (Gaussian tail bound) .
Revised: Fact G.5 (Gaussian tail bound) .

Original: For any b > 0 , we have
Revised: For any b > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where C is an absolute constant.
Revised: where C is an absolute constant.

Original: We prove a truncated Gaussian version of Lemma 7.1 of Allen-Zhu et al. (2019a).
Revised: We prove a truncated Gaussian version of Lemma 7.1 of Allen-Zhu et al. (2019a).

Original: Lemma G.6.
Revised: Lemma G.6.

Original: Let b > 0 be a ﬁxed scalar.
Revised: Let b > 0 be a ﬁxed scalar.

Original: Let the activation function be deﬁned as φ ( x ) := √ c b 1 [ x > (cid:112) 2 /mb ] x , where c b :=
Revised: Let the activation function be deﬁned as φ ( x ) := √ c b 1 [ x > (cid:112) 2 /mb ] x , where c b :=

Original: (2(1 − Φ( b ) + bφ ( b )))
Revised: (2(1 − Φ( b ) + bφ ( b )))

Original: − 1 / 2 .
Revised: − 1 / 2 .

Original: Let (cid:15) ∈ (0 , 1) , then over the randomness of W (0) , with probability at least 1 − O ( nL ) · exp( − Ω( m exp( − b 2 / 2) (cid:15) 2 /L 2 )) , we have
Revised: Let (cid:15) ∈ (0 , 1) , then over the randomness of W (0) , with probability at least 1 − O ( nL ) · exp( − Ω( m exp( − b 2 / 2) (cid:15) 2 /L 2 )) , we have

Original: [Equation]
Revised: [Equation]

Revised: ∈ [ L ] .

Original: Proof.
Revised: Proof.

Original: We only prove for a ﬁxed i ∈ [ n ] and (cid:96) ∈ { 0 , 1 , 2 , . . .
Revised: We only prove for a ﬁxed i ∈ [ n ] and (cid:96) ∈ { 0 , 1 , 2 , . . .

Original: , L } because we can apply union bound at the end.
Revised: , L } because we can apply union bound at the end.

Original: Below, we drop the subscript i for notational convenience, and write h i,(cid:96) and x i as h (cid:96) and x respectively.
Revised: Below, we drop the subscript i for notational convenience, and write h i,(cid:96) and x i as h (cid:96) and x respectively.

Original: According to Fact G.4, ﬁxing any h (cid:96) − 1 (cid:54) = 0 and letting W (cid:96) be the only source of randomness, we have
Revised: According to Fact G.4, ﬁxing any h (cid:96) − 1 (cid:54) = 0 and letting W (cid:96) be the only source of randomness, we have

Original: [Equation]
Revised: [Equation]

Original: , with ω ∼ B ( m, 1 − Φ( b (cid:48) )) , where b (cid:48) := b/ (cid:107) h (cid:96) − 1 (cid:107) 2 .
Revised: , with ω ∼ B ( m, 1 − Φ( b [Equation] )) , where b (cid:48) : = b/ (cid:107) h (cid:96) − 1 (cid:107) 2 .

Original: We ﬁrst consider the (cid:96) = 1 case.
Revised: We ﬁrst consider the (cid:96) = 1 case.

Original: Then, we have (cid:107) h (cid:96) − 1 (cid:107) 

Revised: Then, we have (cid:107) h (cid:96) − 1 (cid:107)

Original: = 1 , and b (cid:48)
Revised: = 1 , and b (cid:48)

Original: =
Revised: =

Original: b .
Revised: b .

Original: Let P b := 1 − Φ( b ) .
Revised: Let P b := 1 − Φ( b ) .

Original: By
Revised: By

Original: Chernoff bound, for any δ ∈ (0 , 1) , we have
Revised: Chernoff bound, for any δ ∈ (0 , 1) , we have

Original: [Equation]
Revised: [Equation]

Original: In the following proof, we condition on this event.
Revised: In the following proof, we condition on this event.

Original: By Fact G.5,
Revised: By Fact G.5,

Original: [Equation]
Revised: [Equation]

Original: By Lemma G.3, we have
Revised: By Lemma G.3, we have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Let c − 1 b := 2( P b + bφ ( b )) be the normalization constant.
Revised: Let c − 1 b := 2( P b + bφ ( b )) be the normalization constant.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: We want 2 tc b /m
Revised: We want 2 tc b /m

Original: =
Revised: =

Original: O ( δ ) , i.e., t =
Revised: O ( δ ) , i.e., t =

Original: O ( δc − 1 b m ) .
Revised: O ( δc − 1 b m ) .

Original: Then, we have ωt = m Ω(1) >
Revised: Then, we have ωt = m Ω(1) >

Original: b 2 .
Revised: b 2 .

Original: Hence, by
Revised: Hence, by

Original: Lemma G.3, we actually have
Revised: Lemma G.3, we actually have

Original: [Equation]
Revised: [Equation]

Original: By taking δ = (cid:15)/L , we get that
Revised: By taking δ = (cid:15)/L , we get that

Original: [Equation]
Revised: [Equation]

Original: ] with probability at least
Revised: with probability at least

Original: [Equation]
Revised: [Equation]

Original: where the last step follows from 1 c b b 2 = P b + bφ ( b ) b=
Revised: where the last step follows from 1 c b b 2 = P b + bφ ( b ) b

Original: Θ( P b ) .
Revised: [Equation]

Original: We can inductively prove the (cid:96) > 1 case.
Revised: We can inductively prove the (cid:96) > 1 case.

Original: Since the blowup of the norm of h 1 is 1 ± (cid:15)/L , the concentration bound is roughly the same for h (cid:96) for (cid:96) ≥ 2 .
Revised: Since the blowup of the norm of h 1 is 1 ± (cid:15)/L , the concentration bound is roughly the same for h (cid:96) for (cid:96) ≥ 2 .

Original: Thus, by carefully choosing the parameters, we can achieve (cid:107) h (cid:96) (cid:107) 22 ∈ [(1 − (cid:15)/L ) (cid:96) , (1 + (cid:15)/L ) (cid:96) ] with high probability.
Revised: Thus, by carefully choosing the parameters, we can achieve (cid:107) h (cid:96) (cid:107) 22 ∈ [(1 − (cid:15)/L ) (cid:96) , (1 + (cid:15)/L ) (cid:96) ] with high probability.

Original: In this end, by a union bound over all the layers (cid:96) ∈
Revised: In this end, by a union bound over all the layers (cid:96) ∈

Original: [ L ] and all the input data i ∈ [ n ] , we get that
Revised: [ L ] and all the input data i ∈ [ n ] , we get that

Original: [Equation]
Revised: [Equation]

Original: ] with probability at least
Revised: ] with probability at least

Original: [Equation]
Revised: [Equation]

Original: which completes the proof of the lemma.
Revised: which completes the proof of the lemma.