Original: = 0 , 1 , 2 , ... , is always bounded by 4Γ d 1 .
Revised: = 0 , 1 , 2 , ... , is always bounded by 4Γ d 1 .

Original: 5 a (cid:63)max (cid:107) 

Revised: 5 a (cid:63)max (cid:107)

Original: i +1 R (cid:107) 5 exp( − 2 i R ) <
Revised: i +1 R (cid:107) 5 exp( − 2 i R ) <

Original: ε/ 2 i +1 .
Revised: ε/ 2 i +1 .

Original: Therefore (cid:107)∇

Original: [Equation]

Original: trunc (

Original: [Equation]

Original:  ) (cid:107) [Equation] ∞ (cid:88) i = ε/ 2 i 
Revised: Therefore (cid:107)∇ G ( B ) − ∇ G trunc ( B ) (cid:107) ≤ ∞ (cid:88) i = ε/ 2 i +1 ≤ ε.

Original: [Equation]

Original: The bound for the Hessian follows from the same argument.
Revised: The bound for the Hessian follows from the same argument.

Original: Finally, we combine this truncation with a result of Mei et al.
Revised: Finally, we combine this truncation with a result of Mei et al.

Original: (2016) that proves universal convergence of gradient and Hessian.
Revised: (2016) that proves universal convergence of gradient and Hessian.

Original: For completeness here we state a version of their theorem with bounded gradient/Hessian:
Revised: For completeness here we state a version of their theorem with bounded gradient/Hessian:

Original: Theorem E.3 (Theorem 1 in Mei et al.
Revised: Theorem E.3 (Theorem 1 in Mei et al.

Original: Let f ( θ ) be a function from R p
Revised: Let f ( θ ) be a function from R p

Original: → R and ˆ f be its empirical version.
Revised: → R and ˆ f be its empirical version.

Original: If the norm of the gradient and Hessian of a function is always bounded by τ , for variables in a ball of radius r in p dimensions, there exists a universal constant C 0 such that for
Revised: If the norm of the gradient and Hessian of a function is always bounded by τ , for variables in a ball of radius r in p dimensions, there exists a universal constant C 0 such that for

Original: C = C 0 max { log rτ/δ, 1 } , the following hold:
Revised: C = C 0 max { log rτ/δ, 1 } , the following hold:

Original: (a) The sample gradient converges to the population gradient.
Revised: (a) The sample gradient converges to the population gradient.

Original: Namely if N ≥ Cp log p we have
Revised: Namely if N ≥ Cp log p we have

Original: [Equation]
Revised: [Equation]

Original: (b) The sample Hessian converges to the empirical Hessian.
Revised: (b) The sample Hessian converges to the empirical Hessian.

Original: Namely if N ≥ Cp log p we have
Revised: Namely if N ≥ Cp log p we have

Original: [Equation]
Revised: [Equation]

Original: As an immediate corollary of this theorem and Lemma E.2, we have
Revised: As an immediate corollary of this theorem and Lemma E.2, we have

Original: Corollary E.4.
Revised: Corollary E.4.

Original: In the setting of Theorem 2.7, for every B whose rows have norm at most 2, we have with high probability,
Revised: In the setting of Theorem 2.7, for every B whose rows have norm at most 2, we have with high probability,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: On the other hand, for all such matrices B , by Lemma E.2 we know the gradient and Hessian of G is close to the gradient and Hessian of G trunc .
Revised: On the other hand, for all such matrices B , by Lemma E.2 we know the gradient and Hessian of G is close to the gradient and Hessian of G trunc .

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Now, the gradient and Hessian for individual samples for estimating G trunc are bounded by some poly ( d, 1 /ε ) , therefore by Theorem E.

Revised: Now, the gradient and Hessian for individual samples for estimating G trunc are bounded by some poly ( d, 1 /ε ) , therefore by Theorem E.

Original: we know the gradient and Hessian of (cid:98) G are close to those of G trunc .
Revised: we know the gradient and Hessian of (cid:98) G are close to those of G trunc .

Original: When N ≥ poly ( d, 1 /ε ) for a large enough polynomial, we have with high probability, for all B with all rows (cid:107)
Revised: When N ≥ poly ( d, 1 /ε ) for a large enough polynomial, we have with high probability, for all B with all rows (cid:107)

Original: b i (cid:107) ≤ 2 ,
Revised: b i (cid:107) ≤ 2 ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The corollary then follows from triangle inequality.
Revised: The corollary then follows from triangle inequality.

Original: Finally we handle the case when B has a row with large norm.
Revised: Finally we handle the case when B has a row with large norm.

Original: We will show that in this case ∇ (cid:98) G ( B ) must also be large, so B cannot be an approximate local minimum.
Revised: We will show that in this case ∇ (cid:98) G ( B ) must also be large, so B cannot be an approximate local minimum.

Original: Lemma E.5.
Revised: Lemma E.5.

Original: If b i is the row with largest norm and (cid:107) b i (cid:107) ≥ 2 , then when N ≥ poly ( d, a (cid:63)max /a (cid:63)min ) 
Revised: If b i is the row with largest norm and (cid:107) b i (cid:107) ≥ 2 , then when N ≥ poly ( d, a (cid:63)max /a (cid:63)min ) for some ﬁxed polynomial, we have with high probability (cid:104)∇ (cid:98) G ( B ) , b i (cid:105) ≥ cλ (cid:107) b i (cid:107) 4 for some universal constant c > 0 .

Original: Proof.
Revised: Proof.

Original: The proof of this Lemma is very similar to Claim C.8.
Revised: The proof of this Lemma is very similar to Claim C.8.

Original: Note that by equation (2.7) there are three terms in (cid:98) G ( B ) :(1) sign(ˆ σ 4 )
Revised: Note that by equation (2.7) there are three terms in (cid:98) G ( B ) :(1) sign(ˆ σ 4 )

Original: ˆ E (cid:104)
Revised: ˆ E (cid:104)

Original: y · (cid:80) j,k ∈ [ d ] ,j (cid:54) =
Revised: y · (cid:80) j,k ∈ [ d ] ,j (cid:54) =

Original: k φ ( b j , b k , x ) (cid:105) , (2) − µ sign(ˆ σ 4 )ˆ E (cid:104) y · (cid:80) j ∈
Revised: k φ ( b j , b k , x ) (cid:105) , (2) − µ sign(ˆ σ 4 )ˆ E (cid:104) y · (cid:80) j ∈

Original: [ d ] ϕ ( b j , x ) (cid:105) , (3) λ (cid:80) mi
Revised: [ d ] ϕ ( b j , x ) (cid:105) , (3) λ (cid:80) mi

Original: =1 ( (cid:107)
Revised: =1 ( (cid:107)

Original: b i (cid:107) 2 − 1) 2 .
Revised: b i (cid:107) 2 − 1) 2 .

Original: Here ˆ E is the empirical average over the samples.
Revised: Here ˆ E is the empirical average over the samples.

Original: Note that the ﬁrst two terms are homogeneous degree 4 polynomials over B , and the third term does not depend on the sample.
Revised: Note that the ﬁrst two terms are homogeneous degree 4 polynomials over B , and the third term does not depend on the sample.

Original: By argument similar to Corollary E.4, we know for any B where b i has the largest row norm, with the number of samples we choose the gradient of the ﬁrst two terms is ca (cid:63)min (cid:107) b i (cid:107) 3 close to the gradient of their expectations, where c < 0 .
Revised: By argument similar to Corollary E.4, we know for any B where b i has the largest row norm, with the number of samples we choose the gradient of the ﬁrst two terms is ca (cid:63)min (cid:107) b i (cid:107) 3 close to the gradient of their expectations, where c < 0 .

Original: 01 is a small constant.
Revised: 01 is a small constant.

Original: By Theorem 2.6, we know the expectation of the ﬁrst two terms are equal to A 1( B ) = 2 √ 

Revised: By Theorem 2.6, we know the expectation of the ﬁrst two terms are equal to A 1( B ) = 2 √

Original: | ˆ σ 4 | · (cid:80) i ∈ [ d ] a (cid:63)i (cid:80) j,k ∈ [ d ] ,j (cid:54) =
Revised: | ˆ σ 4 | · (cid:80) i ∈ [ d ] a (cid:63)i (cid:80) j,k ∈ [ d ] ,j (cid:54) =

Original: k (cid:104) b (cid:63)i , b
Revised: k (cid:104) b (cid:63)i , b

Original: j (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105) 

Revised: j (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105)

Original: and A 2( B ) =
Revised: and A 2( B ) =

Original: − | ˆ σ 4 | µ √ 6 (cid:80) i,j ∈
Revised: − | ˆ σ 4 | µ √ 6 (cid:80) i,j ∈

Original: [ d ] a (cid:63)i (cid:104) b (cid:63)i , b j (cid:105) 4 .
Revised: [ d ] a (cid:63)i (cid:104) b (cid:63)i , b j (cid:105) 4 .

Original: Here the gradient of the ﬁrst term always have positive correlation with b i , so we can ignore it.
Revised: Here the gradient of the ﬁrst term always have positive correlation with b i , so we can ignore it.

Original: For the second term, we know the gradient
Revised: For the second term, we know the gradient

Original: [Equation]
Revised: [Equation]

Original: Taking the inner-product with b i , and use the fact that b (cid:63)i form an orthonormal basis, we know
Revised: Taking the inner-product with b i , and use the fact that b (cid:63)i form an orthonormal basis, we know

Original: [Equation]
Revised: [Equation]

Original: On the other hand, when (cid:107) b i (cid:107) ≥ 2 , we have for the third term
Revised: On the other hand, when (cid:107) b i (cid:107) ≥ 2 , we have for the third term

Original: [Equation]
Revised: [Equation]

Original: Since λ is larger than a (cid:63)max , we know the negative contribution from A 2 and the difference between the empirical version and G are both negligible.
Revised: Since λ is larger than a (cid:63)max , we know the negative contribution from A 2 and the difference between the empirical version and G are both negligible.

Original: Therefore we have (cid:104)∇ (cid:98) G ( B ) , b i (cid:105) ≥ cλ (cid:107) b i (cid:107) 4 as desired.
Revised: Therefore we have (cid:104)∇ (cid:98) G ( B ) , b i (cid:105) ≥ cλ (cid:107) b i (cid:107) 4 as desired.

Original: Now we are ready to prove Theorem 2.7:
Revised: Now we are ready to prove Theorem 2.7:

Original: Proof.
Revised: Proof.

Original: By Lemma E.5, any point B with ∇ (cid:98) G ( B ) ≤ ε must have (cid:107) b i (cid:107) ≤ 2 for all i .
Revised: By Lemma E.5, any point B with ∇ (cid:98) G ( B ) ≤ ε must have (cid:107) b i (cid:107) ≤ 2 for all i .

Original: Now by
Revised: Now by

Original: Corollary E.4, we know the point B we have must satisfy (cid:107)∇ G ( B ) (cid:107) ≤ ε ; ∇ 2 G ( B ) (cid:23) − τ 0 Id .
Revised: Corollary E.4, we know the point B we have must satisfy [Equation]

Original: By point 3 in Theorem 2.3, this implies the guarantee on B .
Revised: By point 3 in Theorem 2.3, this implies the guarantee on B .

Original: F S PURIOUS L OCAL MINIMUM FOR FUNCTION P (cid:48)
Revised: F S PURIOUS L OCAL MINIMUM FOR FUNCTION P (cid:48)

Original: In this section we give an example where the function P (cid:48) does have spurious local minimum.
Revised: In this section we give an example where the function P (cid:48) does have spurious local minimum.

Original: In this example, d = 4 , and the true vectors are the standard basis vectors b (cid:63)i = e i .
Revised: In this example, d = 4 , and the true vectors are the standard basis vectors b (cid:63)i = e i .

Original: We will set a (cid:63) 1 = 1 , and a (cid:63) 2 = a (cid:63) 3 = a (cid:63) 4 = 2 + δ (where δ > 0 is an arbitrary positive constant).
Revised: We will set a (cid:63) 1 = 1 , and a (cid:63) 2 = a (cid:63) 3 = a (cid:63) 4 = 2 + δ (where δ > 0 is an arbitrary positive constant).

Original: The spurious local minimum that we consider is b
Revised: The spurious local minimum that we consider is b

Original: 1 =
Revised: 1 =

Original: b 2 = e 1 =
Revised: b 2 = e 1 =

Original: b (cid:63) 1 , b 3 = e 2 = b (cid:63) 2 , b 4 = √ 22 e 3 + √ 22 e 4 .
Revised: b (cid:63) 1 , b 3 = e 2 = b (cid:63) 2 , b 4 = √ 22 e 3 + √ 22 e 4 .

Original: That is,
Revised: That is,

Revised: B =  

Original: {+ 1  1  0  
+}
Revised: {+ 1  1  0  
+}

Revised: [Equation]

Revised: [Equation]

Revised:   

Original: [Equation]
Revised: [Equation]

Original: The objective P (cid:48) ( B ) = 1 and the only non-zero term is a (cid:63) 1 (cid:104) b (cid:63) 1 , b 1 (cid:105) 2 (cid:104) b (cid:63) 1 , b 2 (cid:105) 2 .
Revised: The objective P (cid:48) ( B ) = 1 and the only non-zero term is a (cid:63) 1 (cid:104) b (cid:63) 1 , b 1 (cid:105) 2 (cid:104) b (cid:63) 1 , b 2 (cid:105) 2 .

Original: In order to improve the objective locally, we need to change either b 1 or b 2 , otherwise the term a (cid:63) 1 (cid:104) b (cid:63) 1 , b 1 (cid:105) 2 (cid:104) b (cid:63) 1 , b 2 (cid:105) 2 is still 1, and all other terms ( a (cid:63) i (cid:104) b (cid:63) i , b j (cid:105) 2 (cid:104) b (cid:63) i , b k (cid:105) 2 ) are non-negative.
Revised: In order to improve the objective locally, we need to change either b 1 or b 2 , otherwise the term a (cid:63) 1 (cid:104) b (cid:63) 1 , b 1 (cid:105) 2 (cid:104) b (cid:63) 1 , b 2 (cid:105) 2 is still 1, and all other terms ( a (cid:63)i (cid:104) b (cid:63)i , b j (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105) 2 ) are non-negative.

Original: Assume we have a local perturbation B (cid:48) , where b (cid:48)
Revised: Assume we have a local perturbation B (cid:48) , where b (cid:48)

Original: =
Revised: =

Original: (cid:112) 1 − ε 21 e 1 + ε 1 u 1 , b (cid:48) 

Revised: (cid:112) 1 − ε 21 e 1 + ε 1 u 1 , b (cid:48)

Original: = (cid:112) 1 − ε 22 e 1 + ε 2 u 2 .
Revised: = (cid:112) 1 − ε 22 e 1 + ε 2 u 2 .

Original: Here u 1 , u 2 are unit vectors that are orthogonal to e 1 .
Revised: Here u 1 , u 2 are unit vectors that are orthogonal to e 1 .

Original: Also, since this is a local perturbation, we make sure ε 1 , ε 2 ≤ ε , and b 3 (2) ≥ 1 − ε , [ b 4 (3)] 2 , [ b 4 (4)] 2 ≥ 0 .
Revised: Also, since this is a local perturbation, we make sure ε 1 , ε 2 ≤ ε , and b 3 (2) ≥ 1 − ε , [ b 4 (3)] 2 , [ b 4 (4)] 2 ≥ 0 .

Original: 5 − ε .
Revised: 5 − ε .

Original: We will show that when ε is small enough, the objective function P (cid:48)
Revised: We will show that when ε is small enough, the objective function P (cid:48)

Original: ( B (cid:48) ) ≥ 1 .
Revised: ( B (cid:48) ) ≥ 1 .

Original: To see this, notice that the term a (cid:63) 1 (cid:104) b (cid:63) 1 , b 1 (cid:105) 2 (cid:104) b (cid:63) 1 , b 2 (cid:105) 2 is now equal to (1 − ε 21 )(1 − ε 22 ) .
Revised: To see this, notice that the term a (cid:63) 1 (cid:104) b (cid:63) 1 , b 1 (cid:105) 2 (cid:104) b (cid:63) 1 , b 2 (cid:105) 2 is now equal to (1 − ε 21 )(1 − ε 22 ) .

Original: On the other hand, for b 1 , we have 4 (cid:88) i = a (cid:63)i4 (cid:88) k
Revised: On the other hand, for b 1 , we have 4 (cid:88) i = a (cid:63)i

Revised: [Equation]

Original:  = (cid:104) b (cid:63)i , b 1 (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105) 2 = ε 21 (2 + δ ) 4 (cid:88) i = 4 (cid:88)
Revised: k = (cid:104) b (cid:63)i , b 1 (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105) 2 = ε 21 (2 + δ ) 4 (cid:88) i = 4 (cid:88)

Original: k
Revised: k

Original: = (cid:104) b (cid:63)i , u 1 (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105)= ε 21 (2 + δ ) 4 (cid:88)
Revised: = (cid:104) b (cid:63)i , u 1 (cid:105) 2 (cid:104) b (cid:63)i , b k (cid:105)= ε 21 (2 + δ ) 4 (cid:88)

Original: i = (cid:104) b (cid:63)i , u 1 (cid:105) 2 ( 4 (cid:88)k = (cid:104) b (cid:63)i , b k (cid:105) 2 ) ≥ ε 21 (2 + δ ) 4 (cid:88)
Revised: i = (cid:104) b (cid:63)i , u 1 (cid:105) 2 ( 4 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Similarly we have the same equation for b 2 .
Revised: Similarly we have the same equation for b 2 .

Original: Note that all the terms we analyzed are disjoint, therefore
Revised: Note that all the terms we analyzed are disjoint, therefore

Original: [Equation]
Revised: [Equation]

Original: By removing higher order terms of ε , it is easy to see that P (cid:48)
Revised: By removing higher order terms of ε , it is easy to see that P (cid:48)

Original: ( B (cid:48) ) ≥ 1 when ε is small enough.
Revised: ( B (cid:48) ) ≥ 1 when ε is small enough.

Original: Therefore B is a local minima of P (cid:48) .
Revised: Therefore B is a local minima of P (cid:48) .