Original: Proof.
Revised: Proof.

Original: Without loss of generality, assume j = 1 .
Revised: Without loss of generality, assume j = 1 .

Original: And suppose (cid:104) w ( t, 0) 1 ,r , ν (cid:105) ≥ d − 1 / 3 for some r ∈ [ m ] .
Revised: And suppose (cid:104) w ( t, 0) 1 ,r , ν (cid:105) ≥ d − 1 / 3 for some r ∈ [ m ] .

Original: By Eq.
Revised: By Eq.

Original: (C.11), we have
Revised: (C.11), we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Recall that from Condition D
Revised: Recall that from Condition D

Original: [Table]
Revised: [Table]

Original: 1 and Condition 4.
Revised: 1 and Condition 4.

Original: [Table]
Revised: [Table]

Original: (cid:88) t (cid:48)
Revised: (cid:88) t (cid:48)

Original: = (cid:88)
Revised: = (cid:88)

Original: ( k,i ) ∈ Ψ (cid:96) (cid:48)
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48)

Original: ( t (cid:48) ,J ) k,i σ (cid:48)
Revised: ( t (cid:48) [Equation] J ) k,i σ (cid:48)

Original: ( (cid:104) w ( t (cid:48) ,J ) j,r , ν k (cid:105)
Revised: ( (cid:104) w ( t (cid:48) ,J ) j,r , ν k (cid:105)

Original: y k,i ) (cid:107) ν (cid:107) 22 .
Revised: y k,i ) (cid:107) ν (cid:107) 22 .

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: (cid:96) (cid:48)
Revised: (cid:96) (cid:48)

Original: ( t (cid:48) ,J ) k
Revised: ( t (cid:48) [Equation] J ) k

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ( (cid:104) w ( t (cid:48) ,J ) j,r , ν k (cid:105)
Revised: (cid:48) ( (cid:104) w ( t (cid:48) [Equation] J ) [Equation]

Revised: r

Original: y k,i ) (cid:107) ν (cid:107) 22 ,
Revised: [Equation] ν k (cid:105) [Equation] k,i ) (cid:107) ν (cid:107) 22 ,

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: On the other hand, from Eq.
Revised: On the other hand, from Eq.

Original: (F.1) and Lemma D.2 we have
Revised: (F.1) and Lemma D.2 we have

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation] [Equation][Equation]
Revised: [Equation] [Equation]

Original: r , z k
Revised: r , z k

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: r

Original: [Equation]
Revised: [Equation]

Original:  z k
Revised: r , z k

Original: [Equation]
Revised: [Equation]

Original: − η
Revised: − η

Original: Kn
Revised: Kn

Original: [Equation]
Revised: [Equation]

Original: where the third line is a direct comparison of the sizes of the ﬁrst and second term of the second line.
Revised: where the third line is a direct comparison of the sizes of the ﬁrst and second term of the second line.

Original: The second last equality is due to Eq.
Revised: The second last equality is due to Eq.

Original: (F.2).
Revised: (F.2).

Original: Lemma F.3.
Revised: Lemma F.3.

Original: Under Condition D.1 and Condition 4.1, for any j ∈ {− 1 , 1 } , r ∈ [ m ] , if at time t we have j (cid:104) w ( t, 0) j,r , ν (cid:105) <
Revised: Under Condition D.1 and Condition 4.1, for any j ∈ {− 1 , 1 } , r ∈ [ m ] , if at time t we have j (cid:104) w ( t, 0) j,r , ν (cid:105) <

Original: d −
Revised: d −

Original: [Equation]
Revised: [Equation]

Original: 3 , then for any k ∈ [ K ] j (cid:104) w ( t, 0) j,r , z k (cid:105) <
Revised: 3 , then for any k ∈ [ K ] j (cid:104) w ( t, 0) j,r , z k (cid:105) <

Original: d −
Revised: d −

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Without loss of generality, consider j = 1 .
Revised: Without loss of generality, consider j = 1 .

Original: Let k ∈
Revised: Let k ∈

Original: [ K ] , r ∈ [ m ] and suppose (cid:104) w ( t, 0) 1 ,r , ν (cid:105) <
Revised: [ K ] , r ∈ [ m ] and suppose (cid:104) w ( t, 0) 1 ,r , ν (cid:105) <

Original: d − 1 / 3 at time t .
Revised: d − 1 / 3 at time t .

Original: By Eq.
Revised: By Eq.

Original: (C.11), we have
Revised: (C.11), we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the fourth line is by Lemma D.2.
Revised: where the fourth line is by Lemma D.2.

Original: Meanwhile, using Eq.
Revised: Meanwhile, using Eq.

Original: (F.1) and Lemma D.2 we obtain(cid:104) [Equation] z
Revised: (F.1) and Lemma D.2 we obtain

Original: [Equation]
Revised: [Equation]

Original: where the second line is by Eq.
Revised: where the second line is by Eq.

Original: (F.3).
Revised: (F.3).

Original: The last inequality is because the ﬁrst term in the second last line is (cid:101)
Revised: The last inequality is because the ﬁrst term in the second last line is (cid:101)

Original: O
Revised: O

Original: ( d − 1 / 2 ) and Θ( σ 2 s d ) =
Revised: ( d − 1 / 2 ) and Θ( σ 2 s d ) =

Original: o (1) .
Revised: o (1) .

Original: This proves our lemma.
Revised: This proves our lemma.

Original: Lemma F.4.
Revised: Lemma F.4.

Original: Under Condition D.1 and Condition 4.1, for any j ∈ {− 1 , 1 } , r ∈ [ m ] and k ∈ [ K ] if at time t we have j (cid:104) w ( t, 0) j,r , z k (cid:105) =
Revised: Under Condition D.1 and Condition 4.1, for any j ∈ {− 1 , 1 } , r ∈ [ m ] and k ∈ [ K ] if at time t we have j (cid:104) w ( t, 0) j,r , z k (cid:105) =

Original: O ( σ 2 s d ) j (cid:104) w ( t, 0) j,r , ν (cid:105) , then j
Revised: O ( σ 2 s d ) j (cid:104) w ( t, 0) j,r , ν (cid:105) , then j

Original: [Equation]
Revised: [Equation]

Original: where w ( t,J ) j,r on both hand sides refer to the weights w ( t, 0) j,r after J steps inner-loop updates using samples from S tr k .
Revised: where w ( t,J ) j,r on both hand sides refer to the weights w ( t, 0) j,r after J steps inner-loop updates using samples from S tr k .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, let j = 1 .
Revised: Without loss of generality, let j = 1 .

Original: Using samples from S tr k , similar to Eq.
Revised: Using samples from S tr k , similar to Eq.

Original: (C.5) we have
Revised: (C.5) we have

Original: [Equation]
Revised: [Equation]

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: Recall (cid:107) ν
Revised: Recall (cid:107) ν

Original: (cid:107) 22 = 1 and (cid:107) z k (cid:107) 

Revised: (cid:107) 22 = 1 and (cid:107) z k (cid:107)

Original: =
Revised: =

Original: O ( σ 2 s d ) by Condition 4.1.
Revised: O ( σ 2 s d ) by Condition 4.1.

Original: Since we are given (cid:104) w ( t, 0) 1 ,r , z k (cid:105) =
Revised: Since we are given (cid:104) w ( t, 0) 1 ,r , z k (cid:105) =

Original: O ( σ 2 s d ) (cid:104) w ( t, 0) 1 ,r , ν (cid:105) , we conclude that (cid:104)
Revised: O ( σ 2 s d ) (cid:104) w ( t, 0) 1 ,r , ν (cid:105) , we conclude that (cid:104)

Original: [Equation]
Revised: [Equation]

Original: ) (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (cid:105) .

Original: The following corollary is an immediate result of Lemma F.2 and Lemma F.4.
Revised: The following corollary is an immediate result of Lemma F.2 and Lemma F.4.

Original: Corollary F.5.
Revised: Corollary F.5.

Original: Fix j ∈ {− 1 , 1 } .
Revised: Fix j ∈ {− 1 , 1 } .

Original: Then for any r ∈ [ m ] , if at time t we have j (cid:104) w ( t, 0) j,r , ν (cid:105) ≥ d − 1 / 3 , then for any k ∈ [ K ] ,
Revised: Then for any r ∈ [ m ] , if at time t we have j (cid:104) w ( t, 0) j,r , ν (cid:105) ≥ d − 1 / 3 , then for any k ∈ [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: where w ( t,J ) j,r on both hand sides refer to the weights w ( t, 0) j,r after J steps inner-loop updates using samples from S tr k .
Revised: where w ( t,J ) j,r on both hand sides refer to the weights w ( t, 0) j,r after J steps inner-loop updates using samples from S tr k .

Original: Lemma F.6.
Revised: Lemma F.6.

Original: Under Condition D.1 and Condition 4.1, for any j ∈ {− 1 , 1 } , r ∈ [ m ] , and k ∈ [ K ] , if we are using the samples from S tr k , then for any t ≥ 

Revised: Under Condition D.1 and Condition 4.1, for any j ∈ {− 1 , 1 } , r ∈ [ m ] , and k ∈ [ K ] , if we are using the samples from S tr k , then for any t ≥

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Without loss of generality, consider j = 1 .
Revised: Without loss of generality, consider j = 1 .

Original: Let r ∈
Revised: Let r ∈

Original: [ m ] and k ∈ [ K ] .
Revised: [ m ] and k ∈ [ K ] .

Original: Using samples from
Revised: Using samples from

Original: S tr k , by Eq.
Revised: S tr k , by Eq.

Original: (F.4)
Revised: (F.4)

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: for any τ ∈ [ J − 1] , where we have used the fact that − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 .
Revised: for any τ ∈ [ J − 1] , where we have used the fact that − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 .

Original: Repeating J times we get
Revised: Repeating J times we get

Original: [Equation]
Revised: [Equation]

Revised: ν (cid:105) ≤ (cid:104) w ( t, 0) 1 ,r , ν (cid:105) + Jγ ≤ Λ ( t ) 1 + Jγ ≤ 2 max (cid:8) Λ ( t

Revised: [Equation]

Revised: 1 , Jγ (cid:9) .

Original: F. P HASE I
Revised: F. P HASE I

Original: Let T (2) j be the ﬁrst iteration such that Λ ( t ) j ≥ m − 1 / 2 (1 + γ )
Revised: Let T (2) j be the ﬁrst iteration such that Λ ( t ) j ≥ m − 1 / 2 (1 + γ )

Original: − J
Revised: − J

Original: =
Revised: =

Original: (cid:101) Θ(1) and let T (2) =
Revised: (cid:101) Θ(1) and let T (2) =

Original: max j T (2) j .
Revised: max j T (2) j .

Original: Without loss of generality, let us assume that T (2) − 1 ≤ T (2)1 .
Revised: Without loss of generality, let us assume that T (2) − 1 ≤ T (2)1 .

Original: Recall that we assumed T (2) ≥ T (2) − 1 .
Revised: Recall that we assumed T (2) ≥ T (2) − 1 .

Original: Lemma F.7 (Restatement of Lemma 6.2) .
Revised: Lemma F.7 (Restatement of Lemma 6.2) .

Original: Under Condition D.1 and Condition 4.1, if one uses the train-validation method, then for any t ≤ T (2) ,
Revised: Under Condition D.1 and Condition 4.1, if one uses the train-validation method, then for any t ≤ T (2) ,

Original: For any k ∈ [ K ] and r = arg max r (cid:48) ∈
Revised: For any k ∈ [ K ] and r = arg max r (cid:48) ∈

Original: [ m ] (cid:104) w ( t, 0) 1 ,r (cid:48) , ν (cid:105) (cid:104) w ( t,J ) 1 ,r , ν k (cid:105) =
Revised: [ m ] (cid:104) w ( t, 0) 1 ,r (cid:48) , ν (cid:105) (cid:104) w ( t,J ) 1 ,r , ν k (cid:105) =

Original: Ω(1) (cid:104) w ( t, 0) 1 ,r , ν (cid:105) (1 + Θ(1) γ ) J .
Revised: Ω(1) (cid:104) w ( t, 0) 1 ,r , ν (cid:105) (1 + Θ(1) γ ) J .

Original: For any r ∈ [ m ] , ( k, i ) ∈ Ψ , and j ∈ {− 1 , 1 } (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) ≤ O (cid:16) max (cid:110) (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) , C (cid:111)(cid:17) .
Revised: For any r ∈ [ m ] , ( k, i ) ∈ Ψ , and j ∈ {− 1 , 1 } (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) ≤ O (cid:16) max (cid:110) (cid:104) w ( t, 0) j,r , ξ k,i (cid:105) , C (cid:111)(cid:17) .

Original: Proof.
Revised: Proof.

Original: The proof of the ﬁrst part relies on the following hypothesis which we will verify inductively later.
Revised: The proof of the ﬁrst part relies on the following hypothesis which we will verify inductively later.

Original: − (cid:96) (cid:48)
Revised: − [Equation]

Original: ( t,τ ) k,i = Θ(1) , for all τ ∈ [ J ] , k ∈ [ K ] , i ∈ [ n ] such that y k,i = 1 .
Revised: = Θ(1) , for all τ ∈ [ J ] , k ∈ [ K ] , i ∈ [ n ] such that y k,i = 1 .

Original: (F.5) σ (cid:48)
Revised: (F.5) σ (cid:48)

Original: ( (cid:104) w ( t,τ ) 1 ,r , ν k (cid:105) )
Revised: ( (cid:104) w ( t,τ ) 1 ,r , ν k (cid:105) )

Original: =
Revised: =

Original: Θ(1) (cid:104) w ( t,τ ) 1 ,r , ν k
Revised: Θ(1) (cid:104) w ( t,τ ) 1 ,r , ν k

Original: (cid:105) , for all τ ∈
Revised: (cid:105) , for all τ ∈

Original: [ J ] , k ∈ [ K ] if (cid:104) w ( t,τ ) 1 ,r , ν k (cid:105) > 0 .
Revised: [ J ] , k ∈ [ K ] if (cid:104) w ( t,τ ) 1 ,r , ν k (cid:105) > 0 .

Original: (F.6)
Revised: (F.6)

Original: By Eq.
Revised: By Eq.

Original: (C.5), we know (cid:104) w ( t,τ ) 1 ,r , ν k (cid:105) is increasing in τ for any r ∈ [ m ] and k ∈
Revised: (C.5), we know (cid:104) w ( t,τ ) 1 ,r , ν k (cid:105) is increasing in τ for any r ∈ [ m ] and k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Moreover, we have
Revised: Moreover, we have

Original: [Equation]
Revised: [Equation]

Original: w [Equation] which shows that (cid:104) w ( t, 0) 1 ,r , ν k (cid:105) is increasing in t for all r ∈ [ m ] ,
Revised: which shows that (cid:104) w ( t, 0) 1 ,r , ν k (cid:105) is increasing in t for all r ∈ [ m ] ,

Original: k ∈
Revised: k ∈

Original: [ K ] , since (cid:104) ν k , ν k (cid:48) (cid:105)
Revised: [ K ] , since (cid:104) ν k , ν k (cid:48) (cid:105)

Original: = (cid:107) ν (cid:107)  

Revised: = (cid:107) ν (cid:107) 2

Original: + (cid:104)
Revised: + (cid:104)

Original: z k , z k (cid:48) (cid:105) > 0 .
Revised: z k , z k (cid:48) (cid:105) > 0 .

Original: And Eq.
Revised: And Eq.

Original: (C.11) shows that (cid:104) w ( t, 0) 1 ,r , ν (cid:105) is increasing in t for all r ∈ [ m ] .
Revised: (C.11) shows that (cid:104) w ( t, 0) 1 ,r , ν (cid:105) is increasing in t for all r ∈ [ m ] .

Original: Let r ( t ) =
Revised: Let r ( t ) =

Original: [Equation]
Revised: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ν (cid:105) .

Original: Recall that under Lemma D.2, we have (cid:104) w [Equation] [Equation] [Equation]
Revised: Recall that under Lemma D.2, we have (cid:104) w (0 , 0) 1 ,r (0) , ν (cid:105)

Original: r [Equation] 0) , ν [Equation]
Revised: =

Original: = Ω( σ 0 ) > 0 .
Revised: Ω( σ 0 ) > 0 .

Original: Since this is increasing in t , we have
Revised: Since this is increasing in t , we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by deﬁnition of r ( t ) .
Revised: where the ﬁrst inequality is by deﬁnition of r ( t ) .

Original: By Lemma F.1, we have for any k ∈
Revised: By Lemma F.1, we have for any k ∈

Original: [ K ] ,
Revised: [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: And by monotonicity in τ , we have (cid:104) w [Equation] t [Equation]
Revised: And by monotonicity in τ , we have (cid:104) w ( t,τ ) 1 ,r ( t ) , ν k (cid:105) > 0 for all τ ∈

Original: [Equation]

Original: t

Original: [Equation]

Original: [-, ν k [Equation] > 
 for all τ ∈-] [ J ] .
Revised: [-, ν k [Equation] > 
 for all τ ∈-] [ J ] .

Original: From Eq.
Revised: From Eq.

Original: (C.5) we can compute
Revised: (C.5) we can compute

Original: [Equation]
Revised: [Equation]

Original: where we have used Hypothesis F.5 F.6 and (cid:107) ν k (cid:107) 2 = Θ(1) to get the second line.
Revised: where we have used Hypothesis F.5 F.6 and (cid:107) ν k (cid:107) 2 = Θ(1) to get the second line.

Original: Applying this repeatedly to get
Revised: Applying this repeatedly to get

Original: [Equation]
Revised: [Equation]

Original: where we have used Lemma F.1 to get the second equality.
Revised: where we have used Lemma F.1 to get the second equality.

Original: This proves the ﬁrst part of the lemma.
Revised: This proves the ﬁrst part of the lemma.

Original: Consider some ( k, i ) ∈ Ψ .
Revised: Consider some ( k, i ) ∈ Ψ .

Original: Without loss of generality we will prove the second part of the lemma for j = 1 , since the proof for j = − 1 is the same.
Revised: Without loss of generality we will prove the second part of the lemma for j = 1 , since the proof for j = − 1 is the same.

Original: By Eq.
Revised: By Eq.

Original: (C.3), for any r ∈ [ m ] we have
Revised: (C.3), for any r ∈ [ m ] we have

Original: [Equation]
Revised: [Equation]

Original: , where we used − (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2.
Revised: where we used − (cid:96) (cid:48) ( · ) ≤ 1 , σ (cid:48) ( · ) ≤ 1 and Lemma D.2.

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: The second inequality is by deﬁnition of C .
Revised: The second inequality is by deﬁnition of C .

Original: This proves the second part of the lemma.
Revised: This proves the second part of the lemma.

Original: Remark F.8 .
Revised: Remark F.8 .

Original: Eq.
Revised: Eq.

Original: (F.8) is not a surprising result.
Revised: (F.8) is not a surprising result.

Original: Since ξ k,i comes from the validation set and the inner-loop updates only uses samples from the training set, we expect that after J steps of inner-loop updates, the inner product should remain as the same order as before the J steps inner-loop updates.
Revised: Since ξ k,i comes from the validation set and the inner-loop updates only uses samples from the training set, we expect that after J steps of inner-loop updates, the inner product should remain as the same order as before the J steps inner-loop updates.

Original: Theorem F.9 (Restatement of Lemma 6.3) .
Revised: Theorem F.9 (Restatement of Lemma 6.3) .

Original: Under Condition D.1 and Condition 4.1, if one uses the train-validation method, then for any t ≤ T (2) , ( k, i ) ∈ Ψ ,
Revised: Under Condition D.1 and Condition 4.1, if one uses the train-validation method, then for any t ≤ T (2) , ( k, i ) ∈ Ψ ,

Original: [Equation]
Revised: [Equation]

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will only prove our ﬁrst statement that involves (cid:98) Ξ ( t ) k,i , because the proof of the second statement that involves (cid:98) Γ ( t ) k,i will be exactly the same.
Revised: We will only prove our ﬁrst statement that involves (cid:98) Ξ ( t ) k,i , because the proof of the second statement that involves (cid:98) Γ ( t ) k,i will be exactly the same.

Original: Similar to the proof of Theorem E.2, we will give lower bound on the growth of Λ ( t ) 1 and upper bound on the growth of (cid:98) Ξ ( t ) k,i .
Revised: Similar to the proof of Theorem E.2, we will give lower bound on the growth of Λ ( t ) 1 and upper bound on the growth of (cid:98) Ξ ( t ) k,i .

Original: We can upper bound the growth of Λ ( t ) 1 using Lemma F.7.
Revised: We can upper bound the growth of Λ ( t ) 1 using Lemma F.7.

Original: We ﬁrst note that since (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) > 0 , we have −(cid:104) w ( t,J ) 1 ,r ( t ) , ν k
Revised: We ﬁrst note that since (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) > 0 , we have −(cid:104) w ( t,J ) 1 ,r ( t ) , ν k

Original: (cid:105) < 0 .
Revised: (cid:105) < 0 .

Original: Thus, if ( k, i ) ∈ Ψ − we have σ (cid:48) ( (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105)
Revised: Thus, if ( k, i ) ∈ Ψ − we have σ (cid:48) ( (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105)

Original: y k,i )
Revised: y k,i )

Original: = 0 .
Revised: = 0 .

Original: Then by Eq.
Revised: Then by Eq.

Original: (C.11)
Revised: (C.11), we get Λ ( t +1)≥ (cid:104) w ( t +1 , 0) 1 ,r ( t ) , ν (cid:105) = (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105)

Original: , we get
Revised: − η

Original: [Equation]
Revised: Kn(cid:88)

Original: t +
Revised: ( k,i ) ∈ Ψ (cid:96)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we used σ (cid:48)
Revised: where we used σ (cid:48)

Original: ( (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105)
Revised: [Equation] ( t,J ) 1 ,r ( t ) , ν k (cid:105)

Original: y k,i )
Revised: y k,i )

Original: = 

Revised: =

Original: if ( k, i ) ∈ Ψ − , and Hypothesis F.5 F.6 to get the second equality.
Revised: if ( k, i ) ∈ Ψ − , and Hypothesis F.5 F.6 to get the second equality.

Original: We used Lemma F.7 to get the third equality.
Revised: We used Lemma F.7 to get the third equality.

Original: Next, let ( k, i ) ∈ Ψ and r ∈ [ m ] .
Revised: Next, let ( k, i ) ∈ Ψ and r ∈ [ m ] .

Original: We will derive an upper bound on the growth of (cid:98) Ξ ( t ) k,i .
Revised: We will derive an upper bound on the growth of (cid:98) Ξ ( t ) k,i .

Original: By Eq.
Revised: By Eq.

Original: (C.9), we have
Revised: (C.9), we have

Original: [Equation]
Revised: [Equation]

Original: where we used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 and Lemma D.2 in the ﬁrst inequality.
Revised: where we used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 and Lemma D.2 in the ﬁrst inequality.

Original: We used σ (cid:48)
Revised: We used σ (cid:48)

Original: ( x ) ≤ 2 | x | for any x ∈ R to get the second inequality.
Revised: ( x ) ≤ 2 | x | for any x ∈ R to get the second inequality.

Original: And we used Lemma F.7 to get the last inequality.
Revised: And we used Lemma F.7 to get the last inequality.

Original: Abusing notation, let r ( t ) =
Revised: Abusing notation, let r ( t ) =

Original: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .
Revised: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .

Original: By deﬁnition, we have Ξ ( t ) k,i = (cid:104) w ( t, 0) 1 ,r ( t ) , ξ k,i (cid:105) .
Revised: By deﬁnition, we have Ξ ( t ) k,i = (cid:104) w ( t, 0) 1 ,r ( t ) , ξ k,i (cid:105) .

Original: Let us compare the size of the second and third term on the right hand side of Eq.
Revised: Let us compare the size of the second and third term on the right hand side of Eq.

Original: (F.10).
Revised: (F.10).

Original: The third term outweighs the second term at least by a factor of Θ( Jγσ 2 ξ d/ ( Kn 2 ))
Revised: The third term outweighs the second term at least by a factor of Θ( Jγσ 2 ξ d/ ( Kn 2 ))

Original: =
Revised: =

Original: log( d ) 0 .
Revised: log( d ) 0 .

Original: 9 , due to the presence of the constant C in the third term.
Revised: 9 , due to the presence of the constant C in the third term.

Original: Therefore, the second term is dominated by the third term for all r ∈ [ m ] and for all t ≥ 0 .
Revised: Therefore, the second term is dominated by the third term for all r ∈ [ m ] and for all t ≥ 0 .

Original: So Eq.
Revised: So Eq.

Original: (F.10) becomes
Revised: (F.10) becomes

Original: [Equation]
Revised: [Equation]

Original: Since the above inequality holds for all r ∈ [ m ] on the left hand side [Equation] it holds for r ( t + 1) in particular.
Revised: Since the above inequality holds for all r ∈ [ m ] on the left hand side, it holds for r ( t + 1) in particular.

Original: This gives
Revised: This gives

Original: [Equation]
Revised: [Equation]

Original: And it is straightforward to check by deﬁnition of (cid:98) Ξ ( t ) k,i that
Revised: And it is straightforward to check by deﬁnition of (cid:98) Ξ ( t ) k,i that

Original: [Equation]
Revised: [Equation]

Original: We are now ready to prove our theorem.
Revised: We are now ready to prove our theorem.

Original: Compare Eq.
Revised: Compare Eq.

Original: (F.9) and Eq.
Revised: (F.9) and Eq.

Original: (F.11)
Revised: (F.11)

Original: , we can use Lemma D. to get an upper bound on (cid:98) Ξ ( T (2) )
Revised: , we can use Lemma D. to get an upper bound on (cid:98) Ξ ( T (2) )

Original: k,i .
Revised: k,i .

Original: We take A = Ω(1) ηγ J , B
Revised: We take A = Ω(1) ηγ J , B

Original: = O (1) ησ 

Revised: = O (1) ησ

Original: ξ d Kn 2 , G = 2 , D
Revised: ξ d Kn 2 , G = 2 , D

Original: = m − 1 / 2 γ − J in Lemma D.5.
Revised: = m − 1 / 2 γ − J in Lemma D.5.

Original: We see that since Λ (0)1 = Θ( d − 1 / 2 ) > G − A/B = 2 − log( d ) 1 .
Revised: We see that since Λ (0)1 = Θ( d − 1 / 2 ) > G − A/B = 2 − log( d ) 1 .

Original: under Condition D.1,
Revised: under Condition D.1,

Original: Lemma D.5 tells us that
Revised: Lemma D.5 tells us that

Original: [Equation]
Revised: [Equation]

Original: This proves our main theorem.
Revised: This proves our main theorem.

Original: It remains to verify our previous hypothesis.
Revised: It remains to verify our previous hypothesis.

Original: Note that if we suppose all of our hypothesis hold at time t < T (2) , then Theorem F.9 implies that for all ( k, i ) ∈ Ψ we have (cid:98) Ξ ( t +1) k,i ≤ O ( (cid:98) Ξ (0) k,i ) .
Revised: Note that if we suppose all of our hypothesis hold at time t < T (2) , then Theorem F.9 implies that for all ( k, i ) ∈ Ψ we have (cid:98) Ξ ( t +1) k,i ≤ O ( (cid:98) Ξ (0) k,i ) .

Original: By
Revised: By

Original: Lemma D.2, it is clear that Hypothesis F.5 and F.6 hold at initialization.
Revised: Lemma D.2, it is clear that Hypothesis F.5 and F.6 hold at initialization.

Original: Proof of Hypothesis F.5.
Revised: Proof of Hypothesis F.5.

Original: The proof will be similar to the proof of Hypothesis E.2.
Revised: The proof will be similar to the proof of Hypothesis E.2.

Original: Let k ∈
Revised: Let k ∈

Original: [ K ] and i ∈ [ n ] be such that y k,i = 1 .
Revised: [ K ] and i ∈ [ n ] be such that y k,i = 1 .

Original: Recall Eq.
Revised: Recall Eq.

Original: (C.7) and that
Revised: (C.7) and that

Original: [Equation]
Revised: [Equation]

Original: Assume our Hypothesis hold at time t .
Revised: Assume our Hypothesis hold at time t .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by deﬁnition of (cid:98) Ξ ( t +
Revised: where the ﬁrst inequality is by deﬁnition of (cid:98) Ξ ( t +1)

Original: [Equation]

Original:  and the second inequality is because (cid:98) Ξ ( t +1)
Revised: k,i and the second inequality is because (cid:98) Ξ ( t +1)

Original: k,i ≤
Revised: k,i ≤

Original: O ( (cid:98) Ξ (0) k,i ) .
Revised: O ( (cid:98) Ξ (0) k,i ) .

Original: The third equality is because m = polylog( d ) and σ (cid:0) O ( (cid:98) Ξ (0)
Revised: The third equality is because m = polylog( d ) and σ (cid:0) O ( (cid:98) Ξ (0)

Original: k,i ) (cid:1) =
Revised: k,i ) (cid:1) =

Original: (cid:101) O
Revised: (cid:101) O

Original: ( d − 1 ) .
Revised: ( d − 1 ) .

Original: For any r ∈ [ m ] , we have
Revised: For any r ∈ [ m ] , we have

Original: [Equation]
Revised: [Equation]

Original: where the second line is by deﬁnition of Λ ( t +1).
Revised: where the second line is by deﬁnition of Λ ( t +1).

Original: The third line is by Lemma F.1.
Revised: The third line is by Lemma F.1.

Original: The fourth line is because t + 1 ≤ T (2) and Λ ( t ) 1 is increasing in t .
Revised: The fourth line is because t + 1 ≤ T (2) and Λ ( t ) 1 is increasing in t .

Original: The last line is by deﬁnition of T (2) .
Revised: The last line is by deﬁnition of T (2) .

Original: Thus m 
Revised: Thus m (cid:88) [Equation] = [Equation]

Revised: [Equation]

Revised: O

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: o (1) .

Revised: (F.14)

Original: Combining Eq.
Revised: Combining Eq.

Original: (C.7), Eq.
Revised: (C.7), Eq.

Original: (F.12), Eq.
Revised: (F.12), Eq.

Original: (F.13), and Eq.
Revised: (F.13), and Eq.

Original: (F.14), we have 
Revised: (F.14), we have − (cid:96) (cid:48)

Original: [Equation]
Revised: ( t +1 , 0) k, i ≥ − (cid:96) (cid:48) ( o (1) + o (1))

Original: ) = Ω(1) .
Revised: = Ω(1) .

Original: Now, suppose that − (cid:96) (cid:48) ( t +1 ,τ ) k,i = Ω(1) for some 0 ≤ τ ≤ J − 1 .
Revised: Now, suppose that − (cid:96) (cid:48) ( t +1 ,τ ) k,i = Ω(1) for some 0 ≤ τ ≤ J − 1 .

Original: We wish to show that − (cid:96) (cid:48)
Revised: We wish to show that − (cid:96) (cid:48)

Original: ( t +1 ,τ +1)
Revised: ( t +1 ,τ +1)

Original: k,i = Ω(1) .
Revised: k,i = Ω(1) .

Original: By Eq.
Revised: By Eq.

Original: (F.8), we know that for i > n 1 , we have
Revised: (F.8), we know that for i > n 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where the second and third line are by deﬁnition of Ξ ( t +1) k,i and (cid:98) Ξ ( t +1)
Revised: where the second and third line are by deﬁnition of Ξ ( t +1) k,i and (cid:98) Ξ ( t +1)

Original: k,i respectively.
Revised: k,i respectively.

Original: This givesm
Revised: This gives

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is because (cid:98) Ξ ( t +1)
Revised: where the second inequality is because (cid:98) Ξ ( t +1)

Original: k,i ≤ O ( (cid:98) Ξ (0) k,i ) .
Revised: k,i ≤ O ( [Equation] Ξ (0) k,i ) .

Original: If i ∈ [ n 1 ] , then by Eq.
Revised: If i ∈ [ n 1 ] , then by Eq.

Original: (C.4) we can bound
Revised: (C.4) we can bound

Original: [Equation]
Revised: [Equation]

Original: where we used − (cid:96) (cid:48) ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 and Lemma D.2 in the ﬁrst inequality.
Revised: where we used − (cid:96) [Equation] ( · ) ≤ 1 and σ (cid:48) ( · ) ≤ 1 and Lemma D.2 in the ﬁrst inequality.

Original: And we used σ (cid:48)
Revised: And we used σ (cid:48)

Original: ( x ) ≤ 2 | x | for any x ∈ R to get the second inequality.
Revised: ( x ) ≤ 2 | x | for any x ∈ R to get the second inequality.

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: where the third line is by deﬁnition of (cid:98) Ξ ( t +1)
Revised: where the third line is by deﬁnition of (cid:98) Ξ ( t +1)

Original: k,i .
Revised: k,i .

Original: The fourth line is because (cid:98) Ξ ( t +1)
Revised: The fourth line is because (cid:98) Ξ ( t +1)

Original: k,i ≤ O ( (cid:98) Ξ (0) k,i ) .
Revised: k,i ≤ O ( [Equation] Ξ (0) k,i ) .

Original: And the last equality is by direct calculation.
Revised: And the last equality is by direct calculation.

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original:  gives m(cid:88)
Revised: This gives m

Original: [Equation]
Revised: [Equation]

Revised: ) ≤ m (cid:88) r = σ (cid:0) (cid:101) O

Revised: ( d − 1 / 2 ) (cid:1) = m · σ (cid:0) (cid:101)

Revised: O ( d − 1 / 2 ) (cid:1)

Original: = o (1) .
Revised: = o (1) .

Original: Using Eq.
Revised: Using Eq.

Original: (C.5), we have [Equation] (cid:12)(cid:12) 
Revised: (C.5), we have (cid:104) w ( t +1 ,τ +1) 1 ,r , ν k (cid:105) ≤ (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) + γ Θ(1) (cid:12)(cid:12) (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) (cid:12)(cid:12) ≤ (cid:12)(cid:12) (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) (cid:12)(cid:12) (1 + Θ(1) γ ) ≤ (cid:12)(cid:12) (cid:104) w ( t +1 , 0) 1 ,r , ν k (cid:105) (cid:12)(cid:12) (1 + Θ(1) γ ) τ + ≤ (cid:12)(cid:12) (cid:104) w ( t +1 , 0) 1 ,r , ν k (cid:105) (cid:12)(cid:12) (1 + Θ(1) γ ) J ≤ O ( m − 1 / 2 (1 + γ ) − J )(1 + Θ(1) γ ) J

Original: [Equation] [Equation] F.16) where we used (cid:96) (cid:48)
Revised: = O ( m − 1 / 2 ) , (F.16) where we used (cid:96) (cid:48)

Original: ( · ) ≤ 1 , σ (cid:48)
Revised: ( · ) ≤ 1 , σ (cid:48)

Original: ( x ) ≤ 2 | x | for any x ∈ R and (cid:107) ν k (cid:107) 2 = Θ(1) in the ﬁrst inequality.
Revised: ( x ) ≤ 2 | x | for any x ∈ R and (cid:107) ν k (cid:107) 2 = Θ(1) in the ﬁrst inequality.

Original: The ﬁfth inequality is by t + 1 ≤ T (2) and the deﬁnition of T (2) .
Revised: The ﬁfth inequality is by t + 1 ≤ T (2) and the deﬁnition of T (2) .

Original: Then we have 
Revised: Then we have m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The inequality is because σ ( O ( m − 1 / 2 ))
Revised: The inequality is because σ ( O ( m − 1 / 2 ))

Original: =
Revised: =

Original: O ( m − 1 ) .
Revised: O ( m − 1 ) .

Original: Combining Eq.
Revised: Combining Eq.

Original: (C.7), Eq.
Revised: (C.7), Eq.

Original: (F.12), Eq.
Revised: (F.12), Eq.

Original: (F.15), and Eq.
Revised: (F.15), and Eq.

Original: (F.17) we obtain
Revised: (F.17) we obtain

Original: [Equation]
Revised: [Equation]

Original: This proves the hypothesis.
Revised: This proves the hypothesis.

Original: Proof of Hypothesis F.6.
Revised: Proof of Hypothesis F.6.

Original: By Eq.
Revised: By Eq.

Original: (F.16), we have that (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) <
Revised: (F.16), we have that (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) <

Original: O ( m − 1 / 2 ) .
Revised: O ( m − 1 / 2 ) .

Original: Thus if (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) > 0 , then 
Revised: Thus if (cid:104) w ( t +1 ,τ ) 1 ,r , ν k (cid:105) > 0 , then σ (cid:48) ( (cid:104) [Equation] ( t + [Equation]

Revised: ν

Original: [Equation]
Revised: [Equation]

Original:  by deﬁnition of σ (cid:48) ( · ) .
Revised: (cid:104) [Equation] [Equation] (cid:105) , by deﬁnition of σ (cid:48) ( · ) .

Original: F. P HASE II
Revised: F. P HASE II

Original: We have shown that when the feature inner product has grown to (cid:101) Θ(1) , the noise inner products remain at (cid:101) Θ( d − 1 / 2 ) , which is exactly the opposite of Theorem E.2.
Revised: We have shown that when the feature inner product has grown to (cid:101) Θ(1) , the noise inner products remain at (cid:101) Θ( d − 1 / 2 ) , which is exactly the opposite of Theorem E.2.

Original: Next we will show that this difference is maintained at least when Λ ( t ) j has grown to O (log( d ) 0 . 1 ) .
Revised: Next we will show that this difference is maintained at least when Λ ( t ) j has grown to O (log( d ) 0 . 1 ) .

Original: For ( k, i ) ∈ Ψ , let(cid:101)
Revised: For ( k, i ) ∈ Ψ , let

Original: [Equation]
Revised: [Equation]

Original: k,i
Revised: k,i

Original: [Equation]
Revised: [Equation]

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: Let T (3) j be the ﬁrst iteration such that Λ ( t ) j ≥ log( d ) 0 .
Revised: Let T (3) j be the ﬁrst iteration such that Λ ( t ) j ≥ log( d ) 0 .

Original: Let T (3) = max (cid:110) T (3) − 1 , T (3)1 (cid:111) .
Revised: Let T (3) = max (cid:110) T (3) − 1 , T (3)1 (cid:111) .

Original: And recall
Revised: And recall

Original: T = poly( d ) is the total number of iterations.
Revised: T = poly( d ) is the total number of iterations.

Original: We ﬁrst present two claims under Condition D.1 and
Revised: We ﬁrst present two claims under Condition D.1 and

Original: Condition 4.1.
Revised: Condition 4.1.

Original: Claim F.10.
Revised: Claim F.10.

Original: For any t ≥ T (2) , any ( k, i ) ∈ Ψ , we have (cid:101) Γ ( t ) k,i ≤ (cid:101)
Revised: For any t ≥ T (2) , any ( k, i ) ∈ Ψ , we have (cid:101) Γ ( t ) k,i ≤ (cid:101)

Original: Θ( d − 1 / 2 ) .
Revised: Θ( d − 1 / 2 ) .

Original: Claim F.11.
Revised: Claim F.11.

Original: For any t ≤ T (3) j , ( k, i ) ∈
Revised: For any t ≤ T (3) j , ( k, i ) ∈

Original: Ψ j , we have (cid:101) Ξ ( t ) k,i ≤
Revised: Ψ j , we have (cid:101) Ξ ( t ) k,i ≤

Original: O
Revised: O

Original: ( d − 1 / 4 ) .
Revised: ( d − 1 / 4 ) .

Original: Remark F.12 .
Revised: Remark F.12 .

Original: Under Claim F.10, it holds that (cid:101) Γ ( t ) k,i
Revised: Under Claim F.10, it holds that (cid:101) Γ ( t ) k,i

Original: =
Revised: =

Original: o
Revised: o

Original: (1) (cid:101) Ξ ( t ) k (cid:48) ,i (cid:48) for all ( k, i ) , ( k (cid:48) , i (cid:48) )
Revised: (1) (cid:101) Ξ ( t ) k (cid:48) ,i (cid:48) for all ( k, i ) , ( k (cid:48) , i (cid:48) )

Original: ∈ Ψ .
Revised: ∈ Ψ .

Original: Remark F.13 .
Revised: Remark F.13 .

Original: By Theorem F.9, Claim F.10 holds at time T (2) .
Revised: By Theorem F.9, Claim F.10 holds at time T (2) .

Original: Also by Theorem F.9, Claim F. holds for all t ≤ T (2) .
Revised: Also by Theorem F.9, Claim F. holds for all t ≤ T (2) .

Original: Note that from Eq.
Revised: Note that from Eq.

Original: (C.11), the growth of j (cid:104) w ( t, 0) j,r , ν (cid:105) depends on the size of j (cid:104) w ( t,J ) j,r ,
Revised: (C.11), the growth of j (cid:104) w ( t, 0) j,r , ν (cid:105) depends on the size of j (cid:104) w ( t,J ) j,r ,

Original: ν k (cid:105) .
Revised: ν k (cid:105) .

Original: We ﬁrst need a lemma that can gives a lower bound on j (cid:104) w ( t,J ) j,r , ν k (cid:105) for t ≥ T (2) .
Revised: We ﬁrst need a lemma that can gives a lower bound on j (cid:104) w ( t,J ) j,r , ν k (cid:105) for t ≥ T (2) .

Original: The following lemma will be of a similar form to Lemma E.7.
Revised: The following lemma will be of a similar form to Lemma E.7.

Original: Lemma F.14.
Revised: Lemma F.14.

Original: For any T (3) j ≥ t ≥ T (2) , let r ( t ) =
Revised: For any T (3) j ≥ t ≥ T (2) , let r ( t ) =

Original: arg max r ∈ [ m ] j (cid:104) w ( t, 0) j,r , ν (cid:105) .
Revised: arg max r ∈ [ m ] j (cid:104) w ( t, 0) j,r , ν (cid:105) .

Original: Assume Claim F. holds, then j (cid:104) w ( t,J ) j,r ( t ) , ν k (cid:105) ≥ Ω( m − 1 / 2 ) for any k ∈
Revised: Assume Claim F. holds, then j (cid:104) w ( t,J ) j,r ( t ) , ν k (cid:105) ≥ Ω( m − 1 / 2 ) for any k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Proof.
Revised: Proof.

Original: For simplicity, take j = 1 .
Revised: For simplicity, take j = 1 .

Original: We shall see that our proof does not depend on the choice of j and thus taking j = 1 is not a loss of generality.
Revised: We shall see that our proof does not depend on the choice of j and thus taking j = 1 is not a loss of generality.

Original: By Eq.
Revised: By Eq.

Original: (C.5)
Revised: (C.5)

Original: , we know that (cid:104) w ( t,τ ) 1 ,r ( t ) , ν k (cid:105) is an increasing function of τ .
Revised: , we know that (cid:104) w ( t,τ ) 1 ,r ( t ) , ν k (cid:105) is an increasing function of τ .

Original: Thus if (cid:104) w ( t, 0) 1 ,r ( t ) , ν k (cid:105) =
Revised: Thus if (cid:104) w ( t, 0) 1 ,r ( t ) , ν k (cid:105) =

Original: Ω( m − 1 / 2 ) , then we must have (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) ≥ Ω( m − 1 / 2 ) .
Revised: Ω( m − 1 / 2 ) , then we must have (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) ≥ Ω( m − 1 / 2 ) .

Original: By Eq.
Revised: By Eq.

Original: (C.11) and the deﬁnition of r ( t ) , we also have that (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) is an increasing function in t .
Revised: (C.11) and the deﬁnition of r ( t ) , we also have that (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) is an increasing function in t .

Original: By deﬁnition of T (2) , for any t ≥ T (2) , it holds that (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) ≥ m − 1 / 2 (1 + γ ) − J .
Revised: By deﬁnition of T (2) , for any t ≥ T (2) , it holds that (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) ≥ m − 1 / 2 (1 + γ ) − J .

Original: Then by
Revised: Then by

Original: Lemma F.2, we have (cid:104) w ( t, 0) 1 ,r ( t ) , ν k (cid:105) ≥ O (1) m − 1 / 2 (1 + γ ) − J .
Revised: Lemma F.2, we have (cid:104) w ( t, 0) 1 ,r ( t ) , ν k (cid:105) ≥ O (1) m − 1 / 2 (1 + γ ) − J .

Original: It sufﬁces to prove that if
Revised: It sufﬁces to prove that if

Original: [Equation]
Revised: [Equation]

Original: 18) for τ ∈ [ J ] , then (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) ≥ Ω( m − 1 / 2 ) .
Revised: 18) for τ ∈ [ J ] , then (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) ≥ Ω( m − 1 / 2 ) .

Original: Let us suppose that F.18 is true for some τ .
Revised: Let us suppose that F.18 is true for some τ .

Original: Under our assumption that (cid:101) Ξ ( t ) k,i ≤ O ( d − 1 / 4 ) , we can apply the same proof as Hypothesis F.5 to get that − (cid:96) (cid:48) ( t,τ (cid:48) )
Revised: Under our assumption that (cid:101) Ξ ( t ) k,i ≤ O ( d − 1 / 4 ) , we can apply the same proof as Hypothesis F.5 to get that − (cid:96) (cid:48) ( t,τ (cid:48) )

Original: k,i = Ω(1) for any τ (cid:48) ∈ [ τ ] .
Revised: k,i = Ω(1) for any τ (cid:48) ∈ [ τ ] .

Original: Then using the same derivation of Eq.
Revised: Then using the same derivation of Eq.

Original: (F.7), we obtain (cid:104) w ( t,τ ) 1 ,r ( t ) , ν k (cid:105)
Revised: (F.7), we obtain (cid:104) w ( t,τ ) 1 ,r ( t ) , ν k (cid:105)

Original: = Ω(1) (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) (1+Θ(1) γ )
Revised: = Ω(1) (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) (1+Θ(1) γ )

Original: τ ≥ m − 1 / 2 (1+ γ ) − τ Ω(1)(1+Θ(1) γ ) τ
Revised: τ ≥ m − 1 / 2 (1+ γ ) − τ Ω(1)(1+Θ(1) γ ) τ

Original: = Ω(1)
Revised: = Ω(1)

Original: m − 1 / 2 .
Revised: m − 1 / 2 .

Original: This concludes the lemma.
Revised: This concludes the lemma.

Original: The next lemma shows that as long as the noise vectors have not been learnt well by our network, the loss from different samples are essentially due to the feature and hence are of similar sizes.
Revised: The next lemma shows that as long as the noise vectors have not been learnt well by our network, the loss from different samples are essentially due to the feature and hence are of similar sizes.

Original: Lemma F.15.
Revised: Lemma F.15.

Original: Let j ∈ {− 1 , 1 } .
Revised: Let j ∈ {− 1 , 1 } .

Original: Under Condition D.1 and Condition 4.1, while (cid:101) Ξ ( t ) k,i ≤ O ( d − 1 / 4 ) and (cid:101) Γ ( t ) k,i ≤ O ( d − 1 / 2 ) for all ( k, i ) ∈
Revised: Under Condition D.1 and Condition 4.1, while (cid:101) Ξ ( t ) k,i ≤ O ( d − 1 / 4 ) and (cid:101) Γ ( t ) k,i ≤ O ( d − 1 / 2 ) for all ( k, i ) ∈

Original: Ψ j , and mσ 2 s d Λ ( t )
Revised: Ψ j , and mσ 2 s d Λ ( t )

Original: j ≤ O (1) , we have (cid:96) (cid:48) ( t,J ) k 1 ,i 1 = Θ(1) (cid:96) (cid:48)
Revised: j ≤ O (1) , we have (cid:96) (cid:48) ( t,J ) k 1 ,i 1 = Θ(1) (cid:96) (cid:48)

Original: ( t,J ) k 2 ,i 

Revised: ( t,J ) k 2 ,i

Original: if ( k 1 , i 1 ) , ( k 2 , i 2 ) ∈
Revised: if ( k 1 , i 1 ) , ( k 2 , i 2 ) ∈

Original: Ψ j .
Revised: Ψ j .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, assume y k 1 ,i 

Revised: Without loss of generality, assume y k 1 ,i

Original: = y k 2 ,i 

Revised: = y k 2 ,i

Original: = j = 1 .
Revised: = j = 1 .

Original: We can essentially use the proof of Hypothesis F.5 to get that
Revised: We can essentially use the proof of Hypothesis F.5 to get that

Original: [Equation]
Revised: [Equation]

Original: for q ∈ { 1 , 2 } .
Revised: for q ∈ { 1 , 2 } .

Original: Moreover, since j (cid:104) w ( t,τ ) j,r , ν k (cid:105) is increasing in t and τ for any j ∈ {− 1 , 1 } , r ∈ [ m ] , k ∈
Revised: Moreover, since j (cid:104) w ( t,τ ) j,r , ν k (cid:105) is increasing in t and τ for any j ∈ {− 1 , 1 } , r ∈ [ m ] , k ∈

Original: [ K ] , we get that (cid:104) w ( t,J ) − 1 ,r , ν k (cid:105) ≤ (cid:104) w (0 , 0) − 1 ,r , ν k (cid:105) ≤
Revised: [ K ] , we get that (cid:104) w ( t,J ) − 1 ,r , ν k (cid:105) ≤ (cid:104) w (0 , 0) − 1 ,r , ν k (cid:105) ≤

Original: O ( d − 1 / 2 ) .
Revised: O ( d − 1 / 2 ) .

Original: This implies m (cid:88)
Revised: This implies m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: for q ∈ { 1 , 2 } .
Revised: for q ∈ { 1 , 2 } .

Original: By Eq.
Revised: By Eq.

Original: (C.7), it sufﬁces to show that
Revised: (C.7), it sufﬁces to show that

Original: [Equation]
Revised: [Equation]

Original: We divide the weights into two sets.
Revised: We divide the weights into two sets.

Original: Let G = (cid:8) r ∈
Revised: Let G = (cid:8) r ∈

Original: [ m ] : (cid:104) w ( t, 0) 1 ,r , ν (cid:105) ≥ d − 1 / 3 (cid:9) , and H = (cid:8) r ∈
Revised: [ m ] : (cid:104) w ( t, 0) 1 ,r , ν (cid:105) ≥ d − 1 / 3 (cid:9) , and H = (cid:8) r ∈

Original: [ m ] : r / ∈ G (cid:9) .
Revised: [ m ] : r / ∈ G (cid:9) .

Original: Let us ﬁrst consider r ∈ G .
Revised: Let us ﬁrst consider r ∈ G .

Original: By Corollary F.5, we have that
Revised: By Corollary F.5, we have that

Original: [Equation]
Revised: [Equation]

Original: for q ∈ { 1 , 2 } , which implies
Revised: for q ∈ { 1 , 2 } , which implies

Original: [Equation]
Revised: [Equation]

Original: Moreover, by Lemma F.6 we get
Revised: Moreover, by Lemma F.6 we get

Original: [Equation]
Revised: [Equation]

Original: By a direct calculation, we have mσ
Revised: By a direct calculation [Equation] we have mσ

Original: 2 s dJγ = o (1) and by assumption
Revised: 2 s dJγ = o (1) and by assumption

Original: mσ 2 s d Λ ( t ) 1 ≤ O (1) .
Revised: mσ 2 s d Λ ( t ) 1 ≤ O (1) .

Original: We can then calculate (cid:88) r
Revised: We can then calculate (cid:88) r

Original: [Equation]
Revised: [Equation]

Original: (F.20) where in the second line we used σ ( x + ε ) ≤ σ ( x ) + O (1) ε for ε ≥ 0 .
Revised: (F.20) where in the second line we used σ ( x + ε ) ≤ σ ( x ) + O (1) ε for ε ≥ 0 .

Original: The third line is due to |G| ≤ m .
Revised: The third line is due to |G| ≤ m .

Original: If r ∈ H , then by Lemma F.3 we have (cid:104) w ( t, 0) 1 ,r , z k q (cid:105) <
Revised: If r ∈ H , then by Lemma F.3 we have (cid:104) w ( t, 0) 1 ,r , z k q (cid:105) <

Original: d − 1 / 3 .
Revised: d − 1 / 3 .

Original: Then by Eq.
Revised: Then by Eq.

Original: (C.5) we have(cid:12)(cid:12) [Equation] [Equation] r
Revised: (C.5) we have

Original: [Equation]
Revised: [Equation]

Original: (cid:105) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original:  (cid:12)(cid:12) (cid:104) w ( t
Revised: (cid:105) (cid:12)(cid:12) (1 + O (1) γ ) , for τ ∈

Original: [Equation]
Revised: [ J − 1] .

Original: q (cid:105) (cid:12)(cid:12) (1 + O (1) γ ) , for τ ∈ [ J − 1] .

Original: Hence (cid:12)(cid:12)
Revised: Hence (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: t,J

Original: [Equation]
Revised: [Equation]

Original: ( t [Equation]
Revised: ) J ≤ (cid:101) O

Original: [Equation]
Revised: [Equation]

Original: q (cid:105) (cid:12)(cid:12) (1 + O (1) γ ) J ≤ (cid:101) O

Original: ( d − 1 / 3 ) .

Original: (F.21)
Revised: (F.21)

Original: Combining the above results, we ﬁnally have m (cid:88) r =
Revised: Combining the above results, we ﬁnally have m (cid:88) r =

Original: [Equation]
Revised: [Equation]

Original: r

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second line is due to Eq.
Revised: where the second line is due to Eq.

Original: (F.20) and Eq.
Revised: (F.20) and Eq.

Original: (F.21).
Revised: (F.21).

Original: Since the above holds for q ∈ { 1 , 2 } , we get
Revised: Since the above holds for q ∈ { 1 , 2 } , we get

Original: [Equation]
Revised: [Equation]

Original: And using Eq.
Revised: And using Eq.

Original: (C.7), we conclude that (cid:96) (cid:48)
Revised: (C.7), we conclude that (cid:96) (cid:48)

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: (1) (cid:96) (cid:48)
Revised: Θ(1) (cid:96) (cid:48)

Original: ( t,J ) k 2 ,
Revised: ( t,J ) k 2 ,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Lemma F.16.
Revised: Lemma F.16.

Original: Under Claim F.11, for any t ≤ T (3)
Revised: Under Claim F.11, for any t ≤ T (3)

Original: j we have − (cid:96) (cid:48) (
Revised: j we have − (cid:96) (cid:48) (

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Without loss of generality, let us consider j = 1 .
Revised: Without loss of generality, let us consider j = 1 .

Original: Let ( k, i ) ∈
Revised: Let ( k, i ) ∈

Original: Ψ
Revised: Ψ

Original: + .
Revised: + .

Original: By Eq.
Revised: By Eq.

Original: (C.7), Eq.
Revised: (C.7), Eq.

Original: (F.19) and Eq.
Revised: (F.19) and Eq.

Original: (F.22)
Revised: (F.22)

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁrst line G = (cid:8) r ∈ [ m ] : (cid:104) w ( t, 0) 1 ,r , ν (cid:105) ≥ d − 1 / 3 (cid:9) .
Revised: where in the ﬁrst line G = (cid:8) r ∈ [ m ] : (cid:104) w ( t, 0) 1 ,r , ν (cid:105) ≥ d − 1 / 3 (cid:9) .

Original: Then recall that t ≤ T (3) implies Λ ( t ) 1 < log( d ) 0 .
Revised: Then recall that t ≤ T (3) implies Λ ( t ) 1 < log( d ) 0 .

Original: Using Lemma F.6, we have that for all r ∈ [ m ]
Revised: Using Lemma F.6, we have that for all r ∈ [ m ]

Original: [Equation]
Revised: [Equation]

Original: Then by a direct calculation using Condition D.1, we get
Revised: Then by a direct calculation using Condition D.1, we get

Original: [Equation]
Revised: [Equation]

Original: Remark F.17 .
Revised: Remark F.17 .

Original: The − 1 / 8 factor on d can be replaced by any c < 0 .
Revised: The − 1 / 8 factor on d can be replaced by any c < 0 .

Original: We choose − 1 / 8 for convenience in the later proof.
Revised: We choose − 1 / 8 for convenience in the later proof.

Original: Proof of Claim F.11.
Revised: Proof of Claim F.11.

Original: Consider T (2) ≤ t ≤ T (3)1 .
Revised: Consider T (2) ≤ t ≤ T (3)1 .

Original: Let (cid:101) Ξ ( t )= max (cid:8)(cid:101) Ξ ( t )
Revised: Let (cid:101) Ξ ( t )= max (cid:8)(cid:101) Ξ ( t )

Original: k,i : ( k, i )
Revised: k,i : ( k, i )

Original: ∈ Ψ + (cid:9) .
Revised: ∈ Ψ + (cid:9) .

Original: By
Revised: By

Original: Eq.
Revised: Eq.

Original: (F.8) and the deﬁnition of (cid:101) Ξ ( t ) k,i , we have that for any ( k, i ) ∈ Ψ + ,
Revised: (F.8) and the deﬁnition of (cid:101) Ξ ( t ) k,i , we have that for any ( k, i ) ∈ Ψ + ,

Original: [Equation]
Revised: [Equation]

Original: And if ( k, i ) ∈ Ψ − ,
Revised: And if ( k, i ) ∈ Ψ − ,

Original: [Equation]
Revised: [Equation]

Original: where we note the second inequality in Remark F.12.
Revised: where we note the second inequality in Remark F.12.

Original: It is convenient to deﬁne
Revised: It is convenient to deﬁne

Original: [Equation]
Revised: [Equation]

Original: because by Lemma F.15, (cid:96) (cid:48) ( t,J ) k,i = Θ(1) (cid:96) (cid:48)
Revised: because by Lemma F.15, (cid:96) (cid:48) ( t,J ) k,i = Θ(1) (cid:96) (cid:48)

Original: ( t,J )for all ( k, i )
Revised: ( t,J )for all ( k, i )

Original: ∈ Ψ
Revised: ∈ Ψ

Original: + .
Revised: + .

Original: Then by our assumption
Revised: Then by our assumption

Original: that t ≤ T (3) and Lemma F.16 we have that − (cid:96) (cid:48) ( t,J )= ω
Revised: that t ≤ T (3) and Lemma F.16 we have that − (cid:96) (cid:48) ( t,J )= ω

Original: ( d − 1 / 8 ) .
Revised: ( d − 1 / 8 ) .

Original: By Eq.
Revised: By Eq.

Original: (C.9)
Revised: (C.9)

Original: we have that for any r ∈ [ m ] , and ( k, i )
Revised: we have that for any r ∈ [ m ] , and ( k, i )

Original: ∈ Ψ + (cid:104)
Revised: ∈ Ψ + (cid:104)

Original: w
Revised: w

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we used Eq.
Revised: where we used Eq.

Original: (F.23) to get the second inequality.
Revised: (F.23) to get the second inequality.

Original: We used Eq.
Revised: We used Eq.

Original: (F.23) and Eq.
Revised: (F.23) and Eq.

Original: (F.24) to get the fourth inequality.
Revised: (F.24) to get the fourth inequality.

Original: The last equality is because − Θ (cid:16) ησ
Revised: The last equality is because − Θ (cid:16) ησ

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: And maximizing the left hand side over r ∈ [ m ] and ( k, i )
Revised: And maximizing the left hand side over r ∈ [ m ] and ( k, i )

Original: ∈ Ψ
Revised: ∈ Ψ

Original: + we get
Revised: + we get

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: By Theorem F.9, we have that (cid:101)
Revised: By Theorem F.9, we have that (cid:101)

Original: Ξ ( T (2) )
Revised: Ξ ( T (2) )

Original: =
Revised: =

Original: d − 1 / 3 .
Revised: d − 1 / 3 .

Original: Let T (cid:48) be the ﬁrst iteration such that (cid:101) Ξ ( T (cid:48) )
Revised: Let T (cid:48) be the ﬁrst iteration such that (cid:101) Ξ ( T (cid:48) )

Original: ≥
Revised: ≥

Original: d − 1 / 4 .
Revised: d − 1 / 4 .

Original: Then, using Eq.
Revised: Then, using Eq.

Original: (F.25) we get
Revised: (F.25) we get

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is by deﬁnition of T (cid:48) .
Revised: where the second inequality is by deﬁnition of T (cid:48) .

Original: The third inequality is due to − (cid:96) (cid:48) ( · ) ≤ 1 .
Revised: The third inequality is due to − (cid:96) (cid:48) ( · ) ≤ 1 .

Original: We have
Revised: We have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where we get the third inequality by taking the logarithm of the second inequality.
Revised: where we get the third inequality by taking the logarithm of the second inequality.

Original: In the fourth inequality, we used log(1 + x ) ≤ O (1) x for x ∈ [0 , 1) .
Revised: In the fourth inequality, we used log(1 + x ) ≤ O (1) x for x ∈ [0 , 1) .

Original: By the second line of Eq.
Revised: By the second line of Eq.

Original: (F.9)
Revised: (F.9)

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: To get the second inequality, we use Lemma F.15 and absorb all the constants into Ω( m − 1 / 2 ) .
Revised: To get the second inequality, we use Lemma F.15 and absorb all the constants into Ω( m − 1 / 2 ) .

Original: Summing over t , we obtain
Revised: Summing over t , we obtain

Original: [Equation]
Revised: [Equation]

Original: We used − (cid:96) (cid:48) ( · ) ≤ 1 to get the last inequality.
Revised: We used − (cid:96) (cid:48) ( · ) ≤ 1 to get the last inequality.

Original: By a direct computation, we have
Revised: By a direct computation, we have

Original: [Equation]
Revised: [Equation]

Original: Combining this fact with Eq.
Revised: Combining this fact with Eq.

Original: (F.26) and Eq.
Revised: (F.26) and Eq.

Original: (F.27) we conclude that T (3) − 1 < T (cid:48) − 1 and hence
Revised: (F.27) we conclude that T (3) − 1 < T (cid:48) − 1 and hence

Original: T (3) < T (cid:48) .
Revised: T (3) < T (cid:48) .

Original: Now let (cid:101) Ξ ( t ) − 1 = max (cid:8)(cid:101) Ξ ( t ) k,i : ( k, i
Revised: Now let (cid:101) Ξ ( t ) − 1 = max (cid:8)(cid:101) Ξ ( t ) k,i : ( k, i

Original: ) ∈
Revised: ) ∈

Original: Ψ − (cid:9) .
Revised: Ψ − (cid:9) .

Original: If we redeﬁne T (cid:48) to be the ﬁrst iteration such that (cid:101) Ξ ( T (cid:48) )
Revised: If we redeﬁne T (cid:48) to be the ﬁrst iteration such that (cid:101) Ξ ( T (cid:48) )

Original: − 1 ≥ d − 1 / 4 , then using the exactly same argument, we obtain T (3) − 1 < T (cid:48) .
Revised: − 1 ≥ d − 1 / 4 , then using the exactly same argument, we obtain T (3) − 1 < T (cid:48) .

Original: This proves our Claim.
Revised: This proves our Claim.

Original: Theorem F.18 (Restatement of Lemma 6.5) .
Revised: Theorem F.18 (Restatement of Lemma 6.5) .

Original: Under Claim F.10, we have T (3) <
Revised: Under Claim F.10, we have T (3) <

Original: T .
Revised: T .

Original: Proof.
Revised: Proof.

Original: We will only prove T (3) <
Revised: We will only prove T (3) <

Original: T , since the proof of T (3) − 1 < T will be the same.
Revised: T , since the proof of T (3) − 1 < T will be the same.

Original: For the sake of contradiction, let us assume that T (3) ≥ T .
Revised: For the sake of contradiction, let us assume that T (3) ≥ T .

Original: Then, for t ≤ T , we have that Λ ( t ) 1 ≤ log( d ) 0 .
Revised: Then, for t ≤ T , we have that Λ ( t ) 1 ≤ log( d ) 0 .

Original: From the proof of Lemma F.16, we have
Revised: From the proof of Lemma F.16, we have

Original: [Equation]
Revised: [Equation]

Original: for all ( k, i ) ∈ Ψ
Revised: for all ( k, i ) ∈ Ψ

Original: + .
Revised: + .

Original: By the second line of Eq.
Revised: By the second line of Eq.

Original: (F.9)
Revised: (F.9)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is by deﬁnition of T (2) .
Revised: where the second inequality is by deﬁnition of T (2) .

Original: The third inequality is because
Revised: The third inequality is because

Original: T = poly( d ) , T (2) =
Revised: T = poly( d ) , T (2) =

Original: polylog( d ) , η = (polylog( d )) − 1 , and m = polylog( d ) .
Revised: polylog( d ) , η = (polylog( d )) − 1 , and m = polylog( d ) .

Original: And poly( d ) exp( −O (log( d ) 0 . 6 ))
Revised: And poly( d ) exp( −O (log( d ) 0 . 6 ))

Original: = Ω(poly
Revised: = Ω(poly

Original: ( d )) .
Revised: ( d )) .

Original: But this is a contradiction since we assumed T ≤ T (3) which means Λ ( T )≤ log( d ) 0 .
Revised: But this is a contradiction since we assumed T ≤ T (3) which means Λ ( T )≤ log( d ) 0 .

Original: Thus, we must have T > T (3)1 .
Revised: Thus, we must have T > T (3)1 .

Original: And the proof of T > T (3) − 1 is the same.
Revised: And the proof of T > T (3) − 1 is the same.

Original: This proves our theorem.
Revised: This proves our theorem.

Original: This will be useful for our convergence result later.
Revised: This will be useful for our convergence result later.

Original: Theorem F.18 implies that before training ends, feature will be memorized by the neurons with an inner product of size at least log( d ) 0 .
Revised: Theorem F.18 implies that before training ends, feature will be memorized by the neurons with an inner product of size at least log( d ) 0 .

Original: As we shall see later, this already guarantees a small test loss.
Revised: As we shall see later, this already guarantees a small test loss.

Original: Our next theorem makes sure that the feature inner product will not grow too big by the end of the training.
Revised: Our next theorem makes sure that the feature inner product will not grow too big by the end of the training.

Original: Theorem F.19.
Revised: Theorem F.19.

Original: Under Claim F.10, we have Λ ( T ) j ≤ log( d ) 1 . 2 for j ∈ {− 1 , 1 } .
Revised: Under Claim F.10, we have Λ ( T ) j ≤ log( d ) 1 . 2 for j ∈ {− 1 , 1 } .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, let j = 1 .
Revised: Without loss of generality, let j = 1 .

Original: Let t (cid:48) be the ﬁrst iteration such that Λ ( t ) 1 ≥ log( d ) 1 . 1 .
Revised: Let t (cid:48) be the ﬁrst iteration such that Λ ( t ) 1 ≥ log( d ) 1 . 1 .

Original: Let r ( t ) =
Revised: Let r ( t ) =

Original: arg max r ∈ [ m ] (cid:8) (cid:104) w ( t, 0) 1 ,r , ν (cid:105) (cid:9) .
Revised: arg max r ∈ [ m ] (cid:8) (cid:104) w ( t, 0) 1 ,r , ν (cid:105) (cid:9) .

Original: Using Eq.
Revised: Using Eq.

Original: (C.11) we have (cid:104) w [Equation] t +
Revised: (C.11) we have (cid:104) w ( t +1 , 0) 1 ,r ( t +1) , ν (cid:105) =

Original: [Equation]

Original: 0) 1 ,r ( t +1) , ν

Original: [Equation]

Original: =

Original: [Equation]

Original:  w [Equation] [Equation] r
Revised: (cid:104) w ( t, 0) 1 ,r

Original: [Equation] + [Equation] ν 
Revised: ( t +1) , ν (cid:105) − η

Original: [Equation]

Original: − η

Original: Kn(cid:88)
Revised: Kn(cid:88)

Original: ( k [Equation] i ) ∈ Ψ (cid:96) (cid:48)
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48)

Original: ( t,J ) k,i σ (cid:48)
Revised: ( t,J ) k,i σ (cid:48)

Original: ( (cid:104) w ( t,J ) 1 ,r
Revised: ( (cid:104) w ( t,J ) 1 ,r

Original: ( t +1) , ν k (cid:105)
Revised: ( t +1) , ν k (cid:105)

Original: y k,i ) ≤ (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105)
Revised: y k,i ) ≤ (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105)

Original: − η
Revised: − η

Original: Kn(cid:88)
Revised: Kn(cid:88)

Original: ( k,i ) ∈ Ψ (cid:96) (cid:48) ( t,J ) k [Equation]
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i σ (cid:48)

Original: i σ (cid:48) ( (cid:104) w ( t,J ) [Equation] r ( t ) 
Revised: ( (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105)

Original: [Equation]
Revised: y k,i ) =

Original: ν k (cid:105) y k,i ) = (cid:104) w (  t [Equation]
Revised: (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105)

Original: [Equation] r ( t ) [Equation] ν (cid:105) − η
Revised: − η

Original: Kn(cid:88)
Revised: Kn(cid:88)

Original: ( k [Equation] i
Revised: ( k,i )

Original: [Equation] ∈ Ψ + (cid:96) (cid:48)( t [Equation] J ) k
Revised: ∈ Ψ + (cid:96) (cid:48)

Original: [Equation]

Original: i σ (cid:48) ( (cid:104) w ( t [Equation] J ) 
Revised: ( t,J ) k,i σ (cid:48)

Original: [Equation] r ( t ) [Equation] ν k (cid:105) ) ≤ (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) − η
Revised: ( (cid:104) w ( t,J ) 1 ,r ( t ) , ν k (cid:105) ) ≤ (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) − η

Original: Kn(cid:88)
Revised: Kn(cid:88)

Original: ( k,i ) ∈
Revised: ( k,i ) ∈

Original: [Equation]
Revised: [Equation]

Original: where the second line is by deﬁnition of r ( t ) .
Revised: where the second line is by deﬁnition of r ( t ) .

Original: We get the third line using the second line of Eq.
Revised: We get the third line using the second line of Eq.

Original: (F.9).
Revised: (F.9).

Original: We have used σ (cid:48) ( · ) ≤ 1 to get the last line.
Revised: We have used σ (cid:48) ( · ) ≤ 1 to get the last line.

Original: By deﬁnition of r ( t ) , this means
Revised: By deﬁnition of r ( t ) , this means

Original: [Equation]
Revised: [Equation]

Original: And by deﬁnition of t (cid:48) , we have
Revised: And by deﬁnition of t (cid:48) , we have

Original: [Equation]
Revised: [Equation]

Original: where we have used − (cid:96) (cid:48) ( · ) ≤ 1 to get the second inequality.
Revised: where we have used − (cid:96) (cid:48) ( · ) ≤ 1 to get the second inequality.

Original: Hence if T ≤ t (cid:48) , then we are done.
Revised: Hence if T ≤ t (cid:48) , then we are done.

Original: So let us suppose that T > t (cid:48) .
Revised: So let us suppose that T > t (cid:48) .

Original: Since Λ ( t ) 1 is increasing in t , we know that for all t ≥ t (cid:48) , Λ ( t ) 1 ≥
Revised: Since Λ ( t ) 1 is increasing in t , we know that for all t ≥ t (cid:48) , Λ ( t ) 1 ≥

Original: [Equation]
Revised: [Equation]

Original: ≥log
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: By Corollary F.5, we have that (cid:104) w ( t, 0) 1 ,r ( t) , ν k (cid:105) =
Revised: By Corollary F.5, we have that (cid:104) w ( t [Equation] [Equation] r ( t

Revised: [Equation]

Original: Θ(1) (cid:104) w ( t, 0) 1 ,r ( t ) , ν (cid:105) for any k ∈
Revised: k (cid:105) = Θ( [Equation] (cid:104)

Revised: [Equation]

Revised: ( t, 0)

Revised: [Equation]

Revised: r ( t )

Revised: [Equation]

Revised: ν (cid:105) for any k ∈

Original: [ K ] .
Revised: [ K ] .

Original: Moreover, for t ≥ t (cid:48)
Revised: Moreover, for t ≥ t (cid:48)

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (C.7), for ( k, i
Revised: (C.7), for ( k, i

Original: ) ∈
Revised: ) ∈

Original: Ψ
Revised: Ψ

Original: + we have
Revised: + we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to Eq.
Revised: where the second inequality is due to Eq.

Original: (F.19).
Revised: (F.19).

Original: The last inequality is by property of the function (cid:96) (cid:48) ( · ) .
Revised: The last inequality is by property of the function (cid:96) (cid:48) ( · ) .

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: where the second last inequality is because ηT = O (poly( d ) / polylog( d ))
Revised: where the second last inequality is because ηT = O (poly( d ) / polylog( d ))

Original: =
Revised: =

Original: O (poly( d )) and
Revised: O (poly( d )) and

Original: O (poly( d )) exp (cid:0) − Θ(1) log( d ) 1 . 1 (cid:1)
Revised: O (poly( d )) exp (cid:0) − Θ(1) log( d ) 1 . 1 (cid:1)

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: This proves our theorem.
Revised: This proves our theorem.

Original: Proof of Claim F.10.
Revised: Proof of Claim F.10.

Original: Suppose our claim holds for all t ∈
Revised: Suppose our claim holds for all t ∈

Original: [ T (2) , T (cid:48) − 1] where T (cid:48) ≤ T .
Revised: [ T (2) , T (cid:48) − 1] where T (cid:48) ≤ T .

Original: Then by Theorem F.19, we have that Λ ( T (cid:48) )
Revised: Then by Theorem F.19, we have that Λ ( T (cid:48) )

Original: j ≤ log( d ) 1 . 2 for j ∈ {− 1 , 1 } .
Revised: j ≤ log( d ) 1 . 2 for j ∈ {− 1 , 1 } .

Original: Let ( k, i
Revised: Let ( k, i

Original: ) ∈ Ψ − .
Revised: ) ∈ Ψ − .

Original: Denote r ( t ) =
Revised: Denote r ( t ) =

Original: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .
Revised: arg max r ∈ [ m ] (cid:104) w ( t, 0) 1 ,r , ξ k,i (cid:105) .

Original: From Eq.
Revised: From Eq.

Original: (C.9), we get that
Revised: (C.9), we get that

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: where we also used Lemma D
Revised: where we also used Lemma D

Original: [Equation]
Revised: [Equation]

Original: 2 in the ﬁrst line.
Revised: 2 in the ﬁrst line.

Original: The third line is a direct calculation using Condition
Revised: The third line is a direct calculation using Condition

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: We have for t ≥ T (2)
Revised: We have for t ≥ T (2)

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by deﬁnition of T (2) and the monotonicity of Λ ( t ) 1 .
Revised: where the ﬁrst inequality is by deﬁnition of T (2) and the monotonicity of Λ ( t ) 1 .

Original: And similarly
Revised: And similarly

Original: [Equation]
Revised: [Equation]

Original: Summing them up we obtain
Revised: Summing them up we obtain

Original: [Equation]
Revised: [Equation]

Original: Summing over t we get
Revised: Summing over t we get

Original: [Equation]
Revised: [Equation]

Original: [Equation][Equation] [Equation] [Equation]
Revised: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: ( k

Original: [Equation]
Revised: [Equation]

Original: i )

Original: [Equation]
Revised: [Equation]

Original: (cid:96)

Original: [Equation]

Original: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: Plugging Eq.
Revised: Plugging Eq.

Original: (F.29) into Eq.
Revised: (F.29) into Eq.

Original: (F.28)
Revised: (F.28)

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: r ,
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:104) w ( T (2)

Original: [Equation]

Original: 0) 1 ,r , ξ [Equation] η T (cid:48)
Revised: w

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: 2) (cid:88)

Original: [Equation]
Revised: [Equation]

Original: i (cid:48)

Original: [Equation]

Original: (cid:96) [Equation] [Equation] ) k (cid:48)
Revised: r ,

Original: [Equation]
Revised: [Equation]

Original: i (cid:48) ≤

Original: [Equation]
Revised: [Equation]

Original: w ( T (2) [Equation] 0) 1 ,r , ξ
Revised: r

Original: [Equation]
Revised: [Equation]

Original: i(cid:105) [Equation] (cid:101)
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( d
Revised: ( d

Original: [Equation]
Revised: [Equation]

Original: 1 / 2 ) (cid:101)
Revised: 2 ) (cid:101)

Original: [Equation]
Revised: [Equation]

Original: where the third inequality is by Theorem F.9.
Revised: where the third inequality is by Theorem F.9.

Original: Since this relation holds for all r ∈ [ m ] , we get that max r ∈ [ m ] (cid:104) w ( T (cid:48) , 0) 1 ,r , ξ k,i (cid:105) ≤ (
Revised: Since this relation holds for all r ∈ [ m ] , we get that max r ∈ [ m ] (cid:104) w ( T (cid:48) , 0) 1 ,r , ξ k,i (cid:105) ≤ (

Original: cid:101) Θ( d − 1 / 2 ) .
Revised: cid:101) Θ( d − 1 / 2 ) .

Original: And hence (cid:101) Γ ( T (cid:48) )
Revised: And hence (cid:101) Γ ( T (cid:48) )

Original: k,i ≤ (cid:101) Θ( d − 1 / 2 ) .
Revised: k,i ≤ (cid:101) Θ( d − 1 / 2 ) .

Original: This proves our claim.
Revised: This proves our claim.

Original: Lemma F.20 (Restatement of Lemma 6.4) .
Revised: Lemma F.20 (Restatement of Lemma 6.4) .

Original: Let T = poly( d ) be the total number of iterations.
Revised: Let T = poly( d ) be the total number of iterations.

Original: Under Condition D.1 and Condition 4.1, we have
Revised: Under Condition D.1 and Condition 4.1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Theorem F.19, we have Λ ( T ) j ≤ log( d ) 1 . 2 for j ∈ {− 1 , 1 } .
Revised: By Theorem F.19, we have Λ ( T ) j ≤ log( d ) 1 . 2 for j ∈ {− 1 , 1 } .

Original: Using the same derivation of
Revised: Using the same derivation of

Original: Eq.
Revised: Eq.

Original: (F.29), we have
Revised: (F.29), we have

Original: [Equation]
Revised: [Equation]

Original: And we can bound the minimum over t by
Revised: And we can bound the minimum over t by

Original: [Equation]
Revised: [Equation]

Original: where the second equality is because T (2)
Revised: where the second equality is because T (2)

Original: =
Revised: =

Original: polylog( d ) =
Revised: polylog( d ) =

Original: o (1) T , and hence T − T (2) = Ω(1) T .
Revised: o (1) T , and hence T − T (2) = Ω(1) T .

Original: Lastly, we have the following theorem for the convergence of training similar to Theorem E.10.
Revised: Lastly, we have the following theorem for the convergence of training similar to Theorem E.10.

Original: Theorem F.21 (Restatement of Theorem 4.5 (1)) .
Revised: Theorem F.21 (Restatement of Theorem 4.5 (1)) .

Original: Let T = poly( d ) be the total number of iterations.
Revised: Let T = poly( d ) be the total number of iterations.

Original: Under Condition D.1 and Condition 4.1, we have
Revised: Under Condition D.1 and Condition 4.1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Denote − (cid:96) (cid:48) ( t,J ) k,i = − (cid:96) (cid:48)
Revised: Denote − (cid:96) (cid:48) ( t,J ) k,i = − (cid:96) (cid:48)

Original: ( α ( t, J, k, i )) , i.e. the argument of − (cid:96) (cid:48) ( t,J ) k,i is some function α
Revised: ( α ( t, J, k, i )) , i.e. the argument of − (cid:96) (cid:48) ( t,J ) k,i is some function α

Original: ( t, J, k, i ) .
Revised: ( t, J, k, i ) .

Original: Then the training loss at time t is related to − (cid:96) (cid:48)
Revised: Then the training loss at time t is related to − (cid:96) (cid:48)

Original: ( t,J ) k,i by
Revised: ( t,J ) k,i by

Original: [Equation]
Revised: [Equation]

Original: Let t (cid:48) =
Revised: Let t (cid:48) =

Original: arg min T − 1 t ∈
Revised: arg min T − 1 t ∈

Original: [ T (2) (cid:80) ( k,i ) ∈
Revised: [ T [Equation] 2) (cid:80) ( k,i ) ∈

Original: Ψ − (cid:96) (cid:48) ( t,J ) k,i .
Revised: Ψ − (cid:96) (cid:48) ( t,J ) k,i .

Original: By deﬁnition of (cid:96) (cid:48) ( · ) and (cid:96) ( · ) we can bound − (cid:96) (cid:48)
Revised: By deﬁnition of (cid:96) (cid:48) ( · ) and (cid:96) ( · ) we can bound − (cid:96) (cid:48)

Original: ( x ) ≥ exp( − x ) / 2 ≥ (cid:96) ( x ) / 2 for x > 0 .
Revised: ( x ) ≥ exp( − x ) / 2 ≥ (cid:96) ( x ) / 2 for x > 0 .

Original: This can be used to bound our loss by
Revised: This can be used to bound our loss by

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by Lemma F.20.
Revised: where the last inequality is by Lemma F.20.

Original: This proves our theorem.
Revised: This proves our theorem.

Original: Theorem F.22 (Restatement of Theorem 4.5 (2)) .
Revised: Theorem F.22 (Restatement of Theorem 4.5 (2)) .

Original: Let W = W ( T ) be our trained weights using the train-validation method.
Revised: Let W = W ( T ) be our trained weights using the train-validation method.

Original: Then the test loss is small
Revised: Then the test loss is small

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Sample a new example x from the distribution, since we use the convolutional structure, we can assume that the ﬁrst patch x (1) =
Revised: Sample a new example x from the distribution, since we use the convolutional structure, we can assume that the ﬁrst patch x (1) =

Original: y · ( ν + z ) and the second patch x (2) = ξ .
Revised: y · ( ν + z ) and the second patch x (2) = ξ .

Original: Clearly (cid:104) ξ k,i , ξ (cid:105) follows the Gaussian distribution with mean zero and standard deviation σ ξ · (cid:107) ξ k,i (cid:107) 2 .
Revised: Clearly (cid:104) ξ k,i , ξ (cid:105) follows the Gaussian distribution with mean zero and standard deviation σ ξ · (cid:107) ξ k,i (cid:107) 2 .

Original: By
Revised: By

Original: Lemma D.2, we further know that (cid:107) ξ k,i (cid:107) 2 ≤ 2 σ ξ √ d .
Revised: Lemma D.2, we further know that (cid:107) ξ k,i (cid:107) 2 ≤ 2 σ ξ √ d .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Similarly we have that
Revised: Similarly we have that

Original: [Equation]
Revised: [Equation]

Original: Denote E to be the event that |(cid:104) ξ k,i , ξ (cid:105)| ≤ d − 1 / 4 and |(cid:104) w (0 , 0)
Revised: Denote E to be the event that |(cid:104) ξ k,i , ξ (cid:105)| ≤ d − 1 / 4 and |(cid:104) w (0 , 0)

Original: j,r , ξ (cid:105)| ≤ d 3 / 2 for all k ∈
Revised: j,r , ξ (cid:105)| ≤ d 3 / 2 for all k ∈

Original: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j ∈ {− 1 , 1 } .
Revised: [ K ] , i ∈ [ n ] , r ∈ [ m ] and j ∈ {− 1 , 1 } .

Original: Applying union bound, we have that the P ( E ) ≥ 1 − 2( nK + m ) exp( − d 1 / 4 ) .
Revised: Applying union bound, we have that the P ( E ) ≥ 1 − 2( nK + m ) exp( − d 1 / 4 ) .

Original: We can divide L test ( W ) into two parts:
Revised: We can divide L test ( W ) into two parts:

Original: [Equation]
Revised: [Equation]

Original: In the following, we bound I 

Revised: In the following, we bound I

Original: and I 2 respectively.
Revised: and I 2 respectively.

Original: Bounding I 1 : When event E holds, for any j ∈ {− 1 , 1 } , we have (cid:104) w ( T, 0) j,r , ξ (cid:105) =
Revised: Bounding I 1 : When event E holds, for any j ∈ {− 1 , 1 } , we have (cid:104) w ( T, 0) j,r , ξ (cid:105) =

Original: (cid:104) w (0 , 0)
Revised: (cid:104) w (0 , 0)

Original: j,r , ξ (cid:105) − η
Revised: j,r , ξ (cid:105) − η

Original: Kn
Revised: Kn

Original: T − 1 (cid:88) t = (cid:88)
Revised: T − 1 (cid:88) t = (cid:88)

Original: ( k,i ) ∈ Ψ (cid:96) (cid:48)
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48)

Original: ( t,J ) k,i σ (cid:48)
Revised: ( t,J ) k,i σ (cid:48)

Original: ( (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) )
Revised: ( (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) )

Original: (cid:104) ξ , ξ k,i (cid:105) ≤ (cid:104) w (0 , 0) j,r , ξ (cid:105) − η
Revised: (cid:104) ξ , ξ k,i (cid:105) ≤ (cid:104) w (0 , 0) j,r , ξ (cid:105) − η

Original: Kn
Revised: Kn

Original: T − 1 (cid:88) t = (cid:88)
Revised: T − 1 (cid:88) t = (cid:88)

Original: ( k,i ) ∈ Ψ (cid:96) (cid:48)
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48)

Original: ( t,J ) k,i σ (cid:48)
Revised: ( t,J ) k,i σ (cid:48)

Original: ( (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) )
Revised: ( (cid:104) w ( t,J ) j,r , ξ k,i (cid:105) )

Original: d − 1 /≤ (cid:104) w (0 , 0) j,r , ξ (cid:105)
Revised: d − 1 /≤ (cid:104) w (0 , 0) j,r , ξ (cid:105)

Original: − η
Revised: − η

Original: Kn
Revised: Kn

Original: T − 1 (cid:88) t = (cid:88)
Revised: T − 1 (cid:88) t = (cid:88)

Original: ( k,i ) ∈ Ψ (cid:96) (cid:48)
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48)

Original: ( t,J ) k,i d − 1 /≤ (cid:104) w (0 , 0)
Revised: ( t,J ) k,i d − 1 /≤ (cid:104) w (0 , 0)

Original: j,r , ξ (cid:105) −
Revised: j,r , ξ (cid:105) −

Original: ηd − 1 / 4 Kn
Revised: ηd − 1 / 4 Kn

Original: T (2) − 1 (cid:88) t
Revised: T (2) − 1 (cid:88) t

Original: =
Revised: =

Original: (cid:88) ( k,i )
Revised: (cid:88) ( k,i )

Original: ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i − ηd − 1 /
Revised: ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i − ηd − 1 /

Original: Kn
Revised: Kn

Original: T − 1 (cid:88) t = T (2) (cid:88)
Revised: T − 1 (cid:88) t = T (2) (cid:88)

Original: ( k,i ) ∈ Ψ (cid:96) (cid:48)
Revised: ( k,i ) ∈ Ψ (cid:96) (cid:48)

Original: ( t,J ) k,i ≤ (cid:104)
Revised: ( t,J ) k,i ≤ (cid:104)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the second line is by Eq.
Revised: where the second line is by Eq.

Original: (F.30).
Revised: (F.30).

Original: The third line is by 0 ≤ σ (cid:48) ( · ) ≤ 1 .
Revised: The third line is by 0 ≤ σ (cid:48) ( · ) ≤ 1 .

Original: The third last line is due to − (cid:96) (cid:48) ( · ) ≤ 1 , T (2) =
Revised: The third last line is due to − (cid:96) (cid:48) ( · ) ≤ 1 , T (2) =

Original: polylog( d ) and (cid:80) T − 1 t
Revised: polylog( d ) and (cid:80) T − 1 t

Original: =
Revised: =

Original: T (2) (cid:80) ( k,i )
Revised: T (2) (cid:80) ( k,i )

Original: ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i = (cid:101) O (1) (recall Eq.
Revised: ∈ Ψ (cid:96) (cid:48) ( t,J ) k,i = (cid:101) O (1) (recall Eq.

Original: (F.29)).
Revised: (F.29)).

Original: The second last line is by Eq.
Revised: The second last line is by Eq.

Original: (F.31).
Revised: (F.31).

Original: We note that Eq.
Revised: We note that Eq.

Original: (F.33) holds for all j ∈ {− 1 , 1 } , r ∈ [ m ] , and y ∈ {− 1 , 1 } .
Revised: (F.33) holds for all j ∈ {− 1 , 1 } , r ∈ [ m ] , and y ∈ {− 1 , 1 } .

Original: Now without loss of generality we assume y = 1 .
Revised: Now without loss of generality we assume y = 1 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where the second line is by Lemma Eq.
Revised: where the second line is by Lemma Eq.

Original: (F.33)
Revised: (F.33)

Original: , the third line is by monotonicity of j (cid:104) w ( t, 0) j,r , ν (cid:105) and j (cid:104) w ( t, 0) j,r , z (cid:105) in t , the fourth line is by Lemma D.2.
Revised: , the third line is by monotonicity of j (cid:104) w ( t, 0) j,r , ν (cid:105) and j (cid:104) w ( t, 0) j,r , z (cid:105) in t , the fourth line is by Lemma D.2.

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: where the second and the third line are due to the fact that σ ( · ) ≥ 0 , the fourth line is by Theorem
Revised: where the second and the third line are due to the fact that σ ( · ) ≥ 0 , the fourth line is by Theorem

Original: F.18, the ﬁfth line is by deﬁnition of T (3) , the last line is by log(1 + x ) ≤ x, ∀ x ≥ 0 .
Revised: F.18, the ﬁfth line is by deﬁnition of T (3) , the last line is by log(1 + x ) ≤ x, ∀ x ≥ 0 .

Original: Therefore we have that
Revised: Therefore we have that

Original: [Equation]
Revised: [Equation]

Original: Bounding I 2 : Next we bound the second term I 2 .
Revised: Bounding I 2 : Next we bound the second term I 2 .

Original: When event E holds, for any j ∈ {− 1 , 1 }
Revised: When event E holds, for any j ∈ {− 1 , 1 }

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Revised: J )

Original: [Equation]
Revised: [Equation]

Original: i = (cid:101) O (1) (recall Eq.
Revised: i = (cid:101) O (1) (recall Eq.

Original: (F.29)).
Revised: (F.29)).

Original: The second last line is by |(cid:104) w (0 , 0)
Revised: The second last line is by |(cid:104) w (0 , 0)

Original: j,r , ξ (cid:105)| ≤ (cid:107) w (0 , 0) j,r (cid:107) 2 (cid:107) ξ (cid:107) 

Revised: j,r , ξ (cid:105)| ≤ (cid:107) w (0 , 0) j,r (cid:107) 2 (cid:107) ξ (cid:107)

Original: = (cid:101) O
Revised: = (cid:101) O

Original: ( (cid:107) ξ (cid:107) 2 ) .
Revised: ( (cid:107) ξ (cid:107) 2 ) .

Original: We note that Eq.
Revised: We note that Eq.

Original: (F.33) holds for all j ∈ {− 1 , 1 } , r ∈ [ m ] , and y ∈ {− 1 , 1 } .
Revised: (F.33) holds for all j ∈ {− 1 , 1 } , r ∈ [ m ] , and y ∈ {− 1 , 1 } .

Original: Now again without loss of generality we assume y = 1 and have that
Revised: Now again without loss of generality we assume y = 1 and have that

Original: [Equation]
Revised: [Equation]

Original: where the second line is by Eq.
Revised: where the second line is by Eq.

Original: (F.35).
Revised: (F.35).

Original: The third line is by monotonicity of j (cid:104) w ( t, 0) j,r , ν (cid:105) and j (cid:104) w ( t, 0) j,r , z (cid:105) in t .
Revised: The third line is by monotonicity of j (cid:104) w ( t, 0) j,r , ν (cid:105) and j (cid:104) w ( t, 0) j,r , z (cid:105) in t .

Original: The fourth line is by Lemma D.2.
Revised: The fourth line is by Lemma D.2.

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: where the second line is due to the fact that σ ( · ) ≥ 0 , the fourth line is by by the property of cross-entropy loss, i.e., log(1 + exp( x )) ≤ 1 + x for all x ≥ 0 .
Revised: where the second line is due to the fact that σ ( · ) ≥ 0 , the fourth line is by by the property of cross-entropy loss, i.e., log(1 + exp( x )) ≤ 1 + x for all x ≥ 0 .

Original: Then we further have that
Revised: Then we further have that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by Cauchy-Schwartz inequality.
Revised: where the ﬁrst inequality is by Cauchy-Schwartz inequality.

Original: The second inequality is by Eq.
Revised: The second inequality is by Eq.

Original: (F.36).
Revised: (F.36).

Original: The third inequality is by the fact that (cid:113) 8 + (cid:101) O
Revised: The third inequality is by the fact that (cid:113) 8 + (cid:101) O

Original: (1) E [ (cid:107) ξ (cid:107) 2 2 ] ≤ O (poly( d )) .
Revised: (1) E [ (cid:107) ξ (cid:107) 2 2 ] ≤ O (poly( d )) .

Original: Plugging the bounds of
Revised: Plugging the bounds of

Original: I 1 , I 2 into Eq.
Revised: I 1 , I 2 into Eq.

Original: (F.32) gives that
Revised: (F.32) gives that

Original: [Equation]
Revised: [Equation]

Original: which completes the proof
Revised: which completes the proof

Original: [Equation]
Revised: [Equation]

Original: G A UXILIARY LEMMA
Revised: G A UXILIARY LEMMA

Original: Lemma G.1 (Restatement of Lemma B.2 from Cao et al.
Revised: Lemma G.1 (Restatement of Lemma B.2 from Cao et al.

Original: Suppose that δ > 0 and d = Ω (cid:0) log(4 Kn/δ ) (cid:1) .
Revised: Suppose that δ > 0 and d = Ω (cid:0) log(4 Kn/δ ) (cid:1) .

Original: Then with probability at least 1 − δ
Revised: Then with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Kn
Revised: Kn

Original: [Equation]
Revised: [Equation]

Original: δ (cid:1) , for all k, k (cid:48) ∈
Revised: δ (cid:1) , for all k, k (cid:48) ∈

Original: [ K ] and i, i (cid:48) ∈ [ n ] with ( k, i ) (cid:54) =
Revised: [ K ] and i, i (cid:48) ∈ [ n ] with ( k, i ) (cid:54) =

Original: ( k (cid:48) , i (cid:48) ) .
Revised: ( k (cid:48) , i (cid:48) ) .

Original: Lemma G.2 (Restatement of Lemma B.3 from Cao et al.
Revised: Lemma G.2 (Restatement of Lemma B.3 from Cao et al.

Original: Suppose that δ > 0 , d = Ω(log( mKn/δ )) and m = Ω(log(1 /δ )) .
Revised: Suppose that δ > 0 , d = Ω(log( mKn/δ )) and m = Ω(log(1 /δ )) .

Original: Then with probability at least 1 − δ
Revised: Then with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: for all r ∈ [ m ] , j ∈ {− 1 , 1 } , k ∈ [ K ] and i ∈ [ n ] .
Revised: for all r ∈ [ m ] , j ∈ {− 1 , 1 } , k ∈ [ K ] and i ∈ [ n ] .

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: for all j ∈ {− 1 ,

Original: [Table]

Original: [Table]

Original: and

Original: [Table]
Revised: [Table]

Original: [ n ] .