Original: L EARNING O NE - HIDDEN - LAYER N EURAL N ETWORKS WITH L ANDSCAPE D ESIGN
Revised: L EARNING O NE - HIDDEN - LAYER N EURAL N ETWORKS WITH L ANDSCAPE D ESIGN

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the problem of learning a one-hidden-layer neural network: we assume the input x ∈ R d is from Gaussian distribution and the label y
Revised: We consider the problem of learning a one-hidden-layer neural network: we assume the input x ∈ R d is from Gaussian distribution and the label y

Original: = a (cid:62) σ ( Bx ) + ξ , where a is a nonnegative vector in R m with m ≤ d , B ∈ R m ×
Revised: = a (cid:62) σ ( Bx ) + ξ , where a is a nonnegative vector in R m with m ≤ d , B ∈ R m ×

Original: d is a full-rank weight matrix, and ξ is a noise vector.
Revised: d is a full-rank weight matrix, and ξ is a noise vector.

Original: We ﬁrst give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.
Revised: We ﬁrst give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.

Original: Inspired by the formula, we design a non-convex objective function G ( · ) whose landscape is guaranteed to have the following properties:
Revised: Inspired by the formula, we design a non-convex objective function G ( · ) whose landscape is guaranteed to have the following properties:

Original: All local minima of G are also global minima.
Revised: All local minima of G are also global minima.

Original: All global minima of G correspond to the ground truth parameters.
Revised: All global minima of G correspond to the ground truth parameters.

Original: The value and gradient of G can be estimated using samples.
Revised: The value and gradient of G can be estimated using samples.

Original: With these properties, stochastic gradient descent on G provably converges to the global minimum and learn the ground-truth parameters.
Revised: With these properties, stochastic gradient descent on G provably converges to the global minimum and learn the ground-truth parameters.

Original: We also prove ﬁnite sample complexity results and validate the results by simulations.
Revised: We also prove ﬁnite sample complexity results and validate the results by simulations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Scalable optimization has been playing crucial roles in the success of deep learning, which has immense applications in artiﬁcial intelligence.
Revised: Scalable optimization has played an important role in the success of deep learning, which has immense applications in artiﬁcial intelligence.

Original: Remarkably, optimization issues are often addressed through designing new models that make the resulting training objective functions easier to be optimized.
Revised: Remarkably, optimization issues are often addressed through designing new models that make the resulting training objective functions easier to be optimized.

Original: For example, over-parameterization Livni et al.
Revised: For example, over-parameterization (Livni et al., 2014), batch-normalization (Ioffe &

Original: (2014), batch-normalization Ioffe & Szegedy (2015), and residual networks He et al. (2016a;b) are often considered as ways to improve the optimization landscape of the resulting objective functions.
Revised: Szegedy, 2015), and residual networks (He et al., 2016a;b) are often considered as ways to improve the optimization landscape of the resulting objective functions.

Original: How do we design models and objective functions that allow efﬁcient optimization with guarantees?
Revised: How do we design models and objective functions that allow efﬁcient optimization with guarantees?

Original: Towards understanding this question in a principled way, this paper studies learning neural networks with one hidden layer.
Revised: Towards understanding this question in a principled way, this paper studies learning neural networks with one hidden layer.

Original: Roughly speaking, we will show that when the input is from Gaussian distribution and under certain simplifying assumptions on the weights, we can design an objective function G ( · ) , such that
Revised: Roughly speaking, we will show that when the input is from Gaussian distribution and under certain simplifying assumptions on the weights, we can design an objective function G ( · ) , such that

Original: [a] all local minima of G ( · ) are global minima [b] all the global minima are the desired solutions, namely, the ground-truth parameters (up to permutation and some ﬁxed transformation).
Revised: [a] all local minima of G ( · ) are global minima [b] all the global minima are the desired solutions, namely, the ground-truth parameters (up to permutation and some ﬁxed transformation).

Original: We note that designing such objective functions is challenging because 1) the natural (cid:96) 2 loss objective does have bad local minimum, and 2) due to the permutation invariance 1 , the objective function inherently has to contain an exponential number of isolated local minima.
Revised: We note that designing such objective functions is challenging because 1) the natural (cid:96) 2 loss objective does have bad local minimum, and 2) due to the permutation invariance 1 , the objective function inherently has to contain an exponential number of isolated local minima.

Original: 1. S ETUP AND KNOWN ISSUES WITH PROPER LEARNING
Revised: 1. S ETUP AND KNOWN ISSUES WITH PROPER LEARNING

Original: We aim to learn a neural network with a one-hidden-layer using a non-convex objective function.
Revised: We aim to learn a neural network with a one-hidden-layer using a non-convex objective function.

Original: We assume input x comes from Gaussian distribution and the label y comes from the model
Revised: We assume input x comes from Gaussian distribution and the label y comes from the model

Original: [Equation]
Revised: [Equation]

Original: where a (cid:63) ∈ R m , B (cid:63) ∼ R m × d are the ground-truth parameters, σ ( · ) is a element-wise non-linear function, and ξ is a noise vector with zero mean.
Revised: where a (cid:63) ∈ R m , B (cid:63) ∼ R m × d are the ground-truth parameters, σ ( · ) is a element-wise non-linear function, and ξ is a noise vector with zero mean.

Original: Here we can without loss of generality assume x comes from spherical Gaussian distribution N (0 , Id d × d ) .
Revised: Here we can without loss of generality assume x comes from spherical Gaussian distribution N (0 , Id d × d ) .

Original: For technical reasons, we will further assume m ≤ d and that a (cid:63) has non-negative entries.
Revised: For technical reasons, we will further assume m ≤ d and that a (cid:63) has non-negative entries.

Original: The most natural learning objective is perhaps the (cid:96) 2 loss function, given the additive noise.
Revised: The most natural learning objective is perhaps the (cid:96) 2 loss function, given the additive noise.

Original: Concretely, we can parameterize with training parameters a ∈ R d , B ∼ R m × d of the same dimension as a (cid:63) and B (cid:63) correspondingly,
Revised: Concretely, we can parameterize with training parameters a ∈ R m , B ∼ R m × d of the same dimension as a (cid:63) and B (cid:63) correspondingly,

Original: [Equation]
Revised: [Equation]

Original: and then use stochastic gradient descent to optimize the (cid:96) 2 loss function.
Revised: and then use stochastic gradient descent to optimize the (cid:96) 2 loss function.

Revised: In many parts of our paper, we consider σ to be the ReLU function σ ( x ) = max { x, 0 } .

Revised: In such settings we assume rows of B (cid:63) have norm 1 because a (cid:63) and rows of B (cid:63) can be scaled simultaneously without changing the model or the objective.

Original: When we have enough training examples, we are effectively minimizing the following population risk with stochastic updates,
Revised: When we have enough training examples, we are effectively minimizing the following population risk with stochastic updates,

Original: [Equation]
Revised: [Equation]

Original: However, empirically stochastic gradient descent cannot converge to the ground-truth parameters in the synthetic setting above when σ ( x ) =
Revised: However, empirically stochastic gradient descent cannot converge to the ground-truth parameters in the synthetic setting above when σ ( x ) =

Original: ReLU ( x ) = max { x, 0 } , even if we have access to an inﬁnite number of samples, and B (cid:63) is a orthogonal matrix.
Revised: ReLU ( x ) = max { x, 0 } , even if we have access to an inﬁnite number of samples, and B (cid:63) is a orthogonal matrix.

Revised: We also show this phenomena generalizes to the case when σ ( x ) is the sigmoid function and the learned network also have the same architecture.

Original: Such empirical results have been reported in Livni et al.
Revised: Such empirical results have been reported in Livni et al.

Original: (2014) previously,
Revised: (2014) previously,

Original: and we also provide our version in Figure 1  of Section 4.
Revised: and we also provide our version in Figure 1 and Figure 2 of Section 4.

Original: This is consistent with observations and theory that over-parameterization is crucial for training neural networks successfully Livni et al. ; Hardt et al. ; Soudry & Carmon (2016).
Revised: This is consistent with observations and theory that over-parameterization is important for training neural networks successfully (Livni et al., 2014; Hardt et al., 2016; Soudry & Carmon, 2016).

Original: These empirical ﬁndings suggest that the population risk f ( a, B ) has spurious local minima with inferior error compared to that of the global minimum.
Revised: These empirical ﬁndings suggest that the population risk f ( a, B ) has spurious local minima with inferior error compared to that of the global minimum.

Original: This phenomenon occurs even if we assume we know a (cid:63) or a (cid:63) = 1 is merely just the all one’s vector.
Revised: This phenomenon occurs even if we assume we know a (cid:63) or a (cid:63) = 1 is merely just the all one’s vector.

Original: Empirically, such landscape issues seem to be alleviated by over-parameterization.
Revised: Empirically, such landscape issues seem to be alleviated by over-parameterization.

Original: By contrast, our method described in the next section does not require over-parameterization and might be suitable for applications that demand the recovery of the true parameters.
Revised: By contrast, our method described in the next section does not require over-parameterization and might be suitable for applications that demand the recovery of the true parameters.

Original: 1. O UR CONTRIBUTIONS
Revised: 1. O UR CONTRIBUTIONS

Original: Towards learning with the same number of training parameters as the ground-truth model, we ﬁrst study the landscape of the population risk f ( · ) and give an analytic formula for it — as an explicit function of the ground-truth parameters and training parameters with the randomness of the data being marginalized out.
Revised: Towards learning with the same number of training parameters as the ground-truth model, we ﬁrst study the landscape of the population risk f ( · ) and give an analytic formula for it — as an explicit function of the ground-truth parameters and training parameters with the randomness of the data being marginalized out.

Original: The formula in equation (2.3) shows that f ( · ) is implicitly attempting to solve simultaneously a ﬁnite number of low-rank tensor decomposition problems with commonly shared components.
Revised: The formula in equation (2.3) shows that f ( · ) is implicitly attempting to solve simultaneously an inﬁnite number of low-rank tensor decomposition problems with commonly shared components.

Original: Inspired by the formula, we design a new training model whose associated loss function — named f (cid:48) and formally deﬁned in equation (2.5) — corresponds to the loss function for decomposing a matrix (2-nd order tensor) and a 4-th order tensor (Theorem 2.2).
Revised: Inspired by the formula, we design a new training model whose associated loss function — named f (cid:48) and formally deﬁned in equation (2.5) — corresponds to the loss function for decomposing a matrix (2-nd order tensor) and a 4-th order tensor (Theorem 2.2).

Original: Empirically, stochastic gradient descent on f (cid:48) learns the network as shown Section 4.
Revised: Empirically, stochastic gradient descent on f (cid:48) learns the network as shown Section 4.

Original: Despite the empirical success of f (cid:48) , we still lack a provable guarantee on the landscape of f (cid:48) .
Revised: Despite the empirical success of f (cid:48) , we still lack a provable guarantee on the landscape of f (cid:48) .

Original: The second contribution of the paper is to design a more sophisticated objective G ( · ) whose landscape is provably nice — all the local minima of G ( · ) are proven to be global, and they correspond to the permutation of the true parameters.
Revised: The second contribution of the paper is to design a more sophisticated objective G ( · ) whose landscape is provably nice — all the local minima of G ( · ) are proven to be global, and they correspond to the permutation of the true parameters.

Original: See Theorem 2.3.
Revised: See Theorem 2.3.

Original: Moreover, the value and the gradient of G can be estimated using samples, and there are no constraints in the optimization.
Revised: Moreover, the value and the gradient of G can be estimated using samples, and there are no constraints in the optimization.

Original: These allow us to use straightforward SGD (see guarantees in Ge et al.
Revised: These allow us to use straightforward SGD (see guarantees in Ge et al.

Original: ; Jin et al.
Revised: ; Jin et al.

Original: to optimize G ( · ) and converge to a local minimum, which is also a global minimum (Corollary 2.4).
Revised: to optimize G ( · ) and converge to a local minimum, which is also a global minimum (Corollary 2.4).

Original: Finally, we also prove a ﬁnite-sample complexity result.
Revised: Finally, we also prove a ﬁnite-sample complexity result.

Original: We will show that with a polynomial number of samples, the empirical version of G share almost the same landscape properties as G itself (Theorem 2.7).
Revised: We will show that with a polynomial number of samples, the empirical version of G share almost the same landscape properties as G itself (Theorem 2.7).

Original: Therefore, we can also use an empirical version of G as a surrogate in the optimization.
Revised: Therefore, we can also use an empirical version of G as a surrogate in the optimization.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: The work of Arora et al.
Revised: The work of Arora et al.

Original: (2014) is one of the early results on provable algorithms for learning deep neural networks, where the authors give an algorithm for learning deep generative models with sparse weights.
Revised: (2014) is one of the early results on provable algorithms for learning deep neural networks, where the authors give an algorithm for learning deep generative models with sparse weights.

Original: Livni et al.
Revised: Livni et al.

Original: (2014), Zhang et al.
Revised: (2014), Zhang et al.

Original: (2016; 2017b), and Daniely et al.
Revised: (2016; 2017b), and Daniely et al.

Original: (2016) study the learnability of special cases of neural networks using ideas from kernel methods.
Revised: (2016) study the learnability of special cases of neural networks using ideas from kernel methods.

Original: Janzamin et al.
Revised: Janzamin et al.

Original: (2015) give a polynomial-time algorithm for learning one-hidden-layer neural networks with twice-differential activation function and known input distributions, using the ideas from tensor decompositions.
Revised: (2015) give a polynomial-time algorithm for learning one-hidden-layer neural networks with twice-differentiable activation function and known input distributions.

Revised: Their approach uses the idea of score function to estimate the high order tensors related to the true components, and then apply tensor decompositions to recover the true parameters.

Revised: When applied to Gaussian input distribution, the score function becomes Hermite polynomials.

Original: A series of recent papers study the theoretical properties of non-convex optimization algorithms for one-hidden-layer neural networks.
Revised: A series of recent papers study the theoretical properties of non-convex optimization algorithms for one-hidden-layer neural networks.

Original: Brutzkus & Globerson (2017) and Tian (2017) analyze the landscape of the population risk for one-hidden-layer neural networks with Gaussian inputs under the assumption that the weights vector associated to each hidden variable (that is, the ﬁlters) have disjoint supports.
Revised: Brutzkus & Globerson (2017) and Tian (2017) analyze the landscape of the population risk for one-hidden-layer neural networks with Gaussian inputs under the assumption that the weights vector associated to each hidden variable (that is, the ﬁlters) have disjoint supports.

Original: Li & Yuan (2017) proves that stochastic gradient descent recovers the ground-truth parameters when the parameters are known to be close to the identity matrix.
Revised: Li & Yuan (2017) proves that stochastic gradient descent recovers the ground-truth parameters when the parameters are known to be close to the identity matrix.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017a) study the optimization landscape of learning one-hidden-layer neural networks with a speciﬁc activation function, and they design a speciﬁc objective function that can recover a single column of the weight matrix.
Revised: (2017a) study the optimization landscape of learning one-hidden-layer neural networks with a speciﬁc activation function, and they design a speciﬁc objective function that can recover a single column of the weight matrix.

Original: Zhong et al.
Revised: Zhong et al.

Original: (2017) study the convergence of non-convex optimization from a good initializer that is produced by tensor methods.
Revised: (2017) study the convergence of non-convex optimization from a good initializer that is produced by tensor methods.

Original: Our algorithm works for a large family of activation functions (including ReLU) and any full-rank weight matrix.
Revised: Our algorithm works for a large family of activation functions (including ReLU) and any full-rank weight matrix.

Original: To our best knowledge, we give the ﬁrst global convergence result for gradient-based methods for our general setting.
Revised: To our best knowledge, we give the ﬁrst global convergence result for gradient-based methods for our general setting.

Original: The optimization landscape properties have also been investigated on simpliﬁed neural networks models.
Revised: The optimization landscape properties have also been investigated on simpliﬁed neural networks models.

Original: Kawaguchi (2016) shows that the landscape of deep neural nets does not have bad local minima but has degenerate saddle points.
Revised: Kawaguchi (2016) shows that the landscape of deep neural nets does not have bad local minima but has degenerate saddle points.

Original: Hardt & Ma (2017) show that re-parametrization using identity connection as in residual networks He et al.
Revised: Hardt & Ma (2017) show that re-parametrization using identity connection as in residual networks He et al.

Original: (2016a) can remove the degenerate saddle points in the optimization landscape of deep linear residual networks.
Revised: (2016a) can remove the degenerate saddle points in the optimization landscape of deep linear residual networks.

Original: Soudry & Carmon (2016) show that an over-parameterized neural network does not have bad differentiable local minimum.
Revised: Soudry & Carmon (2016) show that an over-parameterized neural network does not have bad differentiable local minimum.

Original: Hardt et al.
Revised: Hardt et al.

Original: (2016) analyze the power of over-parameterization in a linear recurrent network (which is equivalent to a linear dynamical system.)
Revised: (2016) analyze the power of over-parameterization in a linear recurrent network (which is equivalent to a linear dynamical system.)

Original: The optimization landscape has also been analyzed for other machine learning problems, including
Revised: The optimization landscape has also been analyzed for other machine learning problems, including

Original: SVD/PCA phase retrieval/synchronization, orthogonal tensor decomposition, dictionary learning, matrix completion, matrix sensing Baldi & Hornik (1989); Srebro & Jaakkola (2013);
Revised: SVD/PCA phase retrieval/synchronization, orthogonal tensor decomposition, dictionary learning, matrix completion, matrix sensing Baldi & Hornik (1989); Srebro & Jaakkola (2013);

Original: Ge et al.
Revised: Ge et al.

Original: ; Sun et al.
Revised: ; Sun et al.

Original: ; Bandeira et al.
Revised: ; Bandeira et al.

Original: Ge et al.
Revised: Ge et al.

Original: ; Bhojanapalli et al.
Revised: ; Bhojanapalli et al.

Original: Ge et al.
Revised: Ge et al.

Original: Our analysis techniques build upon that for tensor decomposition in Ge et al.
Revised: Our analysis techniques build upon that for tensor decomposition in Ge et al.

Original:  we add two additional regularization terms to deal with spurious local minimum caused by the weights a (cid:63) and to remove the constraints
Revised: — we add two additional regularization terms to deal with spurious local minimum caused by the weights a (cid:63) and to remove the constraints.

Original: Notations: We use (cid:107)·(cid:107) to denote the Euclidean norm of a vector and spectral norm of a matrix.
Revised: Notations: We use (cid:107)·(cid:107) to denote the Euclidean norm of a vector and spectral norm of a matrix.

Original: We use (cid:107)·(cid:107) F to denote the Frobenius/Euclidean norm of a matrix or high-order tensor.
Revised: We use (cid:107)·(cid:107) F to denote the Frobenius/Euclidean norm of a matrix or high-order tensor.

Original: For a vector x , let (cid:107) x (cid:107) 0 denotes its inﬁnity norm and for a matrix A , let | A | 0 be a shorthand for (cid:107) vec ( A ) (cid:107) 0 where vec ( A ) is the vectorization of A .
Revised: For a vector x , let (cid:107) x (cid:107) 0 denotes its inﬁnity norm and for a matrix A , let | A | 0 be a shorthand for (cid:107) vec ( A ) (cid:107) 0 where vec ( A ) is the vectorization of A .

Original: We use A ⊗ B to denote the Kronecker product of A and B , and A ⊗ k is a shorthand for A ⊗· · ·⊗ A where A appears k times.
Revised: We use A ⊗ B to denote the Kronecker product of A and B , and A ⊗ k is a shorthand for A ⊗· · ·⊗ A where A appears k times.

Original: For vectors a ⊗ b and a ⊗ k denote the tensor product.
Revised: For vectors a ⊗ b and a ⊗ k denote the tensor product.

Original: We denote the identity matrix in dimension d × d by Id d × d , or Id when the dimension is clear from the context.
Revised: We denote the identity matrix in dimension d × d by Id d × d , or Id when the dimension is clear from the context.

Original: We will deﬁne other notations when we ﬁrst use them.
Revised: We will deﬁne other notations when we ﬁrst use them.

Original:  M AIN R ESULTS 2. C ONNECTING (cid:96) 2 P OPULATION R ISK WITH T ENSOR D ECOMPOSITION
Revised:  M AIN R ESULTS 2. C ONNECTING (cid:96) 2 P OPULATION R ISK WITH T ENSOR D ECOMPOSITION

Original: We ﬁrst show that a natural (cid:96) 2 loss for the one-hidden-layer neural network can be interpreted as simultaneously decomposing tensors of different orders.
Revised: We ﬁrst show that a natural (cid:96) 2 loss for the one-hidden-layer neural network can be interpreted as simultaneously decomposing tensors of different orders.

Original: A straightforward approach of learning the model (1.1) is to parameterize the prediction by
Revised: A straightforward approach of learning the model (1.1) is to parameterize the prediction by

Original: [Equation]
Revised: [Equation]

Original: where a ∈ R d , B ∼ R m ×
Revised: where a ∈ R d , B ∼ R m ×

Original: d are the training parameters.
Revised: d are the training parameters.

Original: Naturally, we can use (cid:96) 2 as the empirical loss, which means the population risk is
Revised: Naturally, we can use (cid:96) 2 as the empirical loss, which means the population risk is

Original: [Equation]
Revised: [Equation]

Original: Throughout the paper, we use b (cid:63) 1 (cid:62) , . . ., b
Revised: Throughout the paper, we use b (cid:63) 1 (cid:62) , . . .

Original: [Equation] (cid:62) to denote the row vectors of B (cid:63) and similarly for B .
Revised: , b (cid:63)m (cid:62) to denote the row vectors of B (cid:63) and similarly for B .

Original: That is, we have 
Revised: That is, we have B = 

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised:   and B [Equation] =  

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  .
Revised:  .

Original: Let a i and a (cid:63)i ’s be the coordinates of a and a (cid:63) respectively.
Revised: Let a i and a (cid:63)i ’s be the coordinates of a and a (cid:63) respectively.

Original: We give the following analytic formula for the population risk deﬁned above.
Revised: We give the following analytic formula for the population risk deﬁned above.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Assume vectors b i , b (cid:63)i ’s are unit vectors.
Revised: Assume vectors b i , b (cid:63)i ’s are unit vectors.

Original: Then, the population risk f deﬁned in equation (2.2) satisﬁes that
Revised: Then, the population risk f deﬁned in equation (2.2) satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: where ˆ σ k is the k -th Hermite coefﬁcient of the function σ .
Revised: where ˆ σ k is the k -th Hermite coefﬁcient of the function σ .

Original: See section A.1 for a short introduction of Hermite polynomial basis.
Revised: See section A.1 for a short introduction of Hermite polynomial basis.

Original: Connection to tensor decomposition:
Revised: Connection to tensor decomposition:

Original: We see from equation (2.3) that the population risk of f is essentially an average of inﬁnite number of loss functions for tensor decomposition.
Revised: We see from equation (2.3) that the population risk of f is essentially an average of inﬁnite number of loss functions for tensor decomposition.

Original: For a ﬁxed k ∈ N , we have that the k -th summand in equation (2.3) is equal to (up to the scaling factor ˆ σ 2 k )[Equation] k (cid:44)
Revised: For a ﬁxed k ∈ N , we have that the k -th summand in equation (2.3) is equal to (up to the scaling factor ˆ σ 2 k )

Original: [Equation]
Revised: [Equation]

Original: where T k = (cid:80) i ∈ [ m ] a (cid:63)i b (cid:63)i ⊗ k is a k -th order tensor in ( R d ) ⊗ k .
Revised: where T k = (cid:80) i ∈ [ m ] a (cid:63)i b (cid:63)i ⊗ k is a k -th order tensor in ( R d ) ⊗ k .

Original: We note that the objective f k naturally attempts to decompose the k -order
Revised: We note that the objective f k naturally attempts to decompose the k -order

Original: rank- m tensor T k into m rank-1 components a 1 b ⊗ k i , . . .
Revised: rank- m tensor T k into m rank-1 components a 1 b ⊗ k i , . . .

Original: , a m b ⊗ km .
Revised: , a m b ⊗ km .

Original: The proof of Theorem 2.1 follows from using techniques in Hermite Fourier analysis, which is deferred to Section A.2.
Revised: The proof of Theorem 2.1 follows from using techniques in Hermite Fourier analysis, which is deferred to Section A.2.

Original: Issues with optimizing f :.
Revised: Issues with optimizing f :.

Original: It turns out that optimizing the population risk using stochastic gradient descent is empirically difﬁcult.
Revised: It turns out that optimizing the population risk using stochastic gradient descent is empirically difﬁcult.

Original: Figure 1 shows that in a synthetic setting where the noise is zero, the test error empirically doesn’t converge to zero for sufﬁciently long time with various learning rate schemes, even if we are using fresh samples in iteration.
Revised: Figure 1 shows that in a synthetic setting where the noise is zero, the test error empirically doesn’t converge to zero for sufﬁciently long time with various learning rate schemes, even if we are using fresh samples in iteration.

Original: This suggests that the landscape of the population risk has some spurious local minimum that is not a global minimum.
Revised: This suggests that the landscape of the population risk has some spurious local minimum that is not a global minimum.

Original: See Section 4 for more details on the experiment setup.
Revised: See Section 4 for more details on the experiment setup.

Original: An empirical ﬁx:.
Revised: An empirical ﬁx:.

Original: Inspired by the connection to tensor decomposition objective described earlier in the subsection, we can design a new objective function that takes exactly the same form as the tensor decomposition objective function f 2 + f 4 .
Revised: Inspired by the connection to tensor decomposition objective described earlier in the subsection, we can design a new objective function that takes exactly the same form as the tensor decomposition objective function f 2 + f 4 .

Original: Concretely, let’s deﬁne ˆ y (cid:48)
Revised: Concretely, let’s deﬁne ˆ y (cid:48)

Original: = a (cid:62) γ ( Bx ) where
Revised: = a (cid:62) γ ( Bx ) where

Original: γ = ˆ σ 2 h 2 + ˆ σ 4 h 4 and h 2 ( t ) = 1 √ 2 ( t 2 − 1) and h 4 ( t ) = 1 √ 24 ( t 4 − 6 t 2 + 3) are the 2nd and 4th normalized probabilists’ Hermite polynomials Wikipedia (2017a).
Revised: γ = ˆ σ 2 h 2 + ˆ σ 4 h 4 and h 2 ( t ) = 1 √ 2 ( t 2 − 1) and h 4 ( t ) = 1 √ 24 ( t 4 − 6 t 2 + 3) are the 2nd and 4th normalized probabilists’ Hermite polynomials Wikipedia (2017a).

Original: We abuse the notation slightly by using the same notation to denote the its element-wise application on a vector.
Revised: We abuse the notation slightly by using the same notation to denote the its element-wise application on a vector.

Original: Now for each example we use (cid:107) ˆ y (cid:48) − y (cid:107) 2 as loss function.
Revised: Now for each example we use (cid:107) ˆ y (cid:48) − y (cid:107) 2 as loss function.

Original: The corresponding population risk is
Revised: The corresponding population risk is

Original: [Equation]
Revised: [Equation]

Original: Now by an extension of Theorem 2.1, we have that the new population risk is equal to the ˆ σ 22 f 2 + ˆ σ 24 f 4 .
Revised: Now by an extension of Theorem 2.1, we have that the new population risk is equal to the ˆ σ 22 f 2 + ˆ σ 24 f 4 .

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Let f (cid:48) be deﬁned as in equation (2.5) and f 2 and f 4 be deﬁned in equation (2.4) .
Revised: Let f (cid:48) be deﬁned as in equation (2.5) and f 2 and f 4 be deﬁned in equation (2.4) .

Original: Assume b i , b (cid:63)i ’s are unit vectors.
Revised: Assume b i , b (cid:63)i ’s are unit vectors.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: It turns out stochastic gradient descent on the objective f (cid:48)
Revised: It turns out stochastic gradient descent on the objective f (cid:48)

Original: ( a, B ) (with projection to the set of matrices B with row norm 1) converges empirically to the ground truth ( a (cid:63) , B (cid:63) ) or one of its equivalent permutations.
Revised: ( a, B ) (with projection to the set of matrices B with row norm 1) converges empirically to the ground truth ( a (cid:63) , B (cid:63) ) or one of its equivalent permutations.

Original: (See Figure 3.)
Revised: (See Figure 3.)

Original: However, we don’t know of any existing work for analyzing the landscape of the objective f (cid:48) (or f k for any k ≥ 3 ).
Revised: However, we don’t know of any existing work for analyzing the landscape of the objective f (cid:48) (or f k for any k ≥ 3 ).

Original: We conjecture that the landscape of f (cid:48) doesn’t have any spurious local minimum under certain mild assumptions on ( a (cid:63) , B (cid:63) ) .
Revised: We conjecture that the landscape of f (cid:48) doesn’t have any spurious local minimum under certain mild assumptions on ( a (cid:63) , B (cid:63) ) .

Original: Despite recent attempts on other loss functions for tensor decomposition Ge & Ma (2017), we believe that analyzing f (cid:48) is technically challenging and its resolution will be potentially enlightening for the understanding landscape of loss function with permutation invariance.
Revised: Despite recent attempts on other loss functions for tensor decomposition Ge & Ma (2017), we believe that analyzing f (cid:48) is technically challenging and its resolution will be potentially enlightening for the understanding landscape of loss function with permutation invariance.

Original: See Section 4 for more experimental results.
Revised: See Section 4 for more experimental results.

Original: 2. L ANDSCAPE DESIGN FOR ORTHOGONAL B (cid:63)
Revised: 2. L ANDSCAPE DESIGN FOR ORTHOGONAL B (cid:63)

Original: The population risk deﬁned in equation (2.5) — though works empirically for randomly generated ground-truth ( a (cid:63) , B (cid:63) ) — doesn’t have any theoretical guarantees.
Revised: The population risk deﬁned in equation (2.5) — though works empirically for randomly generated ground-truth ( a (cid:63) , B (cid:63) ) — doesn’t have any theoretical guarantees.

Original: It’s also possible that when ( a (cid:63) , B (cid:63) ) are chosen adversarially or from a different distribution, SGD no longer converges to the true parameters.
Revised: It’s also possible that when ( a (cid:63) , B (cid:63) ) are chosen adversarially or from a different distribution, SGD no longer converges to the true parameters.

Original: To solve this problem, we design another objective function G ( · ) , such that the optimizer of G ( · ) still corresponds to the ground-truth, and G () has provably nice landscape — all local minima of
Revised: To solve this problem, we design another objective function G ( · ) , such that the optimizer of G ( · ) still corresponds to the ground-truth, and G () has provably nice landscape — all local minima of

Original: G () are global minima.
Revised: G () are global minima.

Original: In this subsection, for simplicity, we work with the case when B (cid:63) is an orthogonal matrix and state our main result.
Revised: In this subsection, for simplicity, we work with the case when B (cid:63) is an orthogonal matrix and state our main result.

Original: The discussion of the general case is deferred to the end of this Section and
Revised: The discussion of the general case is deferred to the end of this Section and

Original: Section C.
Revised: Section C.

Original: We deﬁne our objective function G ( B ) as
Revised: We deﬁne our objective function G ( B ) as

Original: [Equation]
Revised: [Equation]

Original: where ϕ ( · , · ) is deﬁned as
Revised: where ϕ ( · , · ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and φ ( · , · , · ) is deﬁned as
Revised: and φ ( · , · , · ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The rationale behind of the choices of φ and ϕ will only be clearer and relevant in later sections.
Revised: The rationale behind of the choices of φ and ϕ will only be clearer and relevant in later sections.

Original: For now, the only relevant property of them is that both are smooth functions whose derivatives are easily computable.
Revised: For now, the only relevant property of them is that both are smooth functions whose derivatives are easily computable.

Original: We remark that we can sample G ( · ) using the samples straightforwardly — it’s deﬁned as an average of functions of examples and the parameters.
Revised: We remark that we can sample G ( · ) using the samples straightforwardly — it’s deﬁned as an average of functions of examples and the parameters.

Original: We also note that only parameter B appears in the loss function.
Revised: We also note that only parameter B appears in the loss function.

Original: We will infer the value of a (cid:63) using straightforward linear regression after we get the (approximately) accurate value of B (cid:63) .
Revised: We will infer the value of a (cid:63) using straightforward linear regression after we get the (approximately) accurate value of B (cid:63) .

Original: Due to technical reasons, our method only works for the case when a (cid:63)i > 0 for every i .
Revised: Due to technical reasons, our method only works for the case when a (cid:63)i > 0 for every i .

Original: We will assume this throughout the rest of the paper.
Revised: We will assume this throughout the rest of the paper.

Original: The general case is left for future work.
Revised: The general case is left for future work.

Original: Let a (cid:63) max = max a (cid:63)i , a (cid:63) min =
Revised: Let a (cid:63) max = max a (cid:63)i , a (cid:63) min =

Original: min
Revised: min

Original: a (cid:63)i , and κ (cid:63) = max a (cid:63)i / min a (cid:63)i .
Revised: a (cid:63)i , and κ (cid:63) = max a (cid:63)i / min a (cid:63)i .

Original: Our result will depend on the value of κ (cid:63) .
Revised: Our result will depend on the value of κ (cid:63) .

Original: Essentially we treat κ (cid:63) as an absolute constant that doesn’t scale in dimension.
Revised: Essentially we treat κ (cid:63) as an absolute constant that doesn’t scale in dimension.

Original: The following theorem characterizes the properties of the landscape of G ( · ) .
Revised: The following theorem characterizes the properties of the landscape of G ( · ) .

Original: Theorem 2.3.
Revised: Theorem 2.3.

Original: Let c be a sufﬁciently small universal constant (e.g. c = 0 .
Revised: Let c be a sufﬁciently small universal constant (e.g. c = 0 .

Original: 01 sufﬁces)
Revised: 01 sufﬁces)

Original: and suppose the activation function σ satisﬁes ˆ σ 4 (cid:54)
Revised: and suppose the activation function σ satisﬁes ˆ σ 4 (cid:54)

Original: = 0 .
Revised: = 0 .

Original: Assume µ ≤ c/κ (cid:63) , λ ≥ c − 1 a (cid:63) max , and B (cid:63) is an orthogonal matrix.
Revised: Assume µ ≤ c/κ (cid:63) , λ ≥ c − 1 a (cid:63) max , and B (cid:63) is an orthogonal matrix.

Original: The function G ( · ) deﬁned as in equation (2.7) satisﬁes that
Revised: The function G ( · ) deﬁned as in equation (2.7) satisﬁes that

Original: A matrix B is a local minimum of G if and only if B can be written as B = DPB (cid:63) where P is a permutation matrix and D is a diagonal matrix with D ii ∈ {± 1 ±
Revised: A matrix B is a local minimum of G if and only if B can be written as B = DPB (cid:63) where P is a permutation matrix and D is a diagonal matrix with D ii ∈ {± 1 ±

Original: O ( µa (cid:63) max /λ ) } .Furthermore, this means that all local minima of G are also global.
Revised: O ( µa (cid:63) max /λ ) } .Furthermore, this means that all local minima of G are also global.

Original: Any saddle point B has a strictly negative curvature in the sense that λ min ( ∇ 2 G ( B )) ≥ − τ 0 where τ [-
-]
Revised: Any saddle point B has a strictly negative curvature in the sense that λ min ( ∇ 2 G ( B )) ≥ − τ 0 where τ [-
-]0 =

Original: = c min { µa (cid:63) min / ( κ (cid:63) d ) , λ } 3.
Revised: c min { µa (cid:63) min / ( κ (cid:63) d ) , λ } 3.

Original: Suppose B is an approximate local minimum in the sense that B satisﬁes (cid:107)∇ G ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 G ( B )) ≥ − τ
Revised: Suppose B is an approximate local minimum in the sense that B satisﬁes (cid:107)∇ G ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 G ( B )) ≥ − τ

Original: Then B can be written as B = PDB (cid:63) + EB (cid:63) where P is a permutation matrix, D is a diagonal matrix satisfying the same bound as in bullet 1, and | E | ∞ ≤ O ( ε/ (ˆ σ 4 a (cid:63) min )) .
Revised: Then B can be written as B = PDB (cid:63) + EB (cid:63) where P is a permutation matrix, D is a diagonal matrix satisfying the same bound as in bullet 1, and | E | ∞ ≤ O ( ε/ (ˆ σ 4 a (cid:63) min )) .

Original: As a direct consequence, B is O d ( ε ) -close to a global minimum in Euclidean distance, where O d ( · ) hides polynomial dependency on d and other parameters.
Revised: As a direct consequence, B is O d ( ε ) -close to a global minimum in Euclidean distance, where O d ( · ) hides polynomial dependency on d and other parameters.

Original: The theorem above implies that we can learn B (cid:63) (up to permutation of rows and sign-ﬂip) if we take λ to be sufﬁciently large and optimize G ( · ) using stochastic gradient descent.
Revised: The theorem above implies that we can learn B (cid:63) (up to permutation of rows and sign-ﬂip) if we take λ to be sufﬁciently large and optimize G ( · ) using stochastic gradient descent.

Original: In this case, the diagonal matrix D in bullet 1 is sufﬁciently close to identity (up to sign ﬂip) and therefore a local minimum B is close to B (cid:63) up to permutation of rows and sign ﬂip.
Revised: In this case, the diagonal matrix D in bullet 1 is sufﬁciently close to identity (up to sign ﬂip) and therefore a local minimum B is close to B (cid:63) up to permutation of rows and sign ﬂip.

Original: The sign of each b (cid:63)i can be recovered easily after we recover a (see Lemma 2.5 below.)
Revised: The sign of each b (cid:63)i can be recovered easily after we recover a (see Lemma 2.5 below.)

Original: SGD converges to a local minimum Ge et al.
Revised: SGD converges to a local minimum Ge et al.

Original: (2015) (under the additional property as established in bullet 2 above), which is also a global minimum for the function G ( · ) .
Revised: (2015) (under the additional property as established in bullet 2 above), which is also a global minimum for the function G ( · ) .

Original: We will prove the theorem in Section B as a direct corollary of Theorem B.1.
Revised: We will prove the theorem in Section B as a direct corollary of Theorem B.1.

Original: The technical bullet 2 and 3 of the theorem is to ensure that we can use SGD to converge to a local minimum as stated below.
Revised: The technical bullet 2 and 3 of the theorem is to ensure that we can use SGD to converge to a local minimum as stated below.

Original: Corollary 2.4.
Revised: Corollary 2.4.

Original: In the setting of Theorem 2.3, we can use stochastic gradient descent to optimize function G ( · ) (with fresh samples at each iteration) and converge to an approximate global minimum
Revised: In the setting of Theorem 2.3, we can use stochastic gradient descent to optimize function G ( · ) (with fresh samples at each iteration) and converge to an approximate global minimum

Original: B that is ε -close to a global minimum in time poly( d, 1 /ε ) .
Revised: B that is ε -close to a global minimum in time poly( d, 1 /ε ) .

Original: After approximately recovering the matrix B (cid:63) , we can also recover the coefﬁcient a (cid:63) easily.
Revised: After approximately recovering the matrix B (cid:63) , we can also recover the coefﬁcient a (cid:63) easily.

Original: Note that ﬁxing B , we can ﬁt a using simply linear regression.
Revised: Note that ﬁxing B , we can ﬁt a using simply linear regression.

Original: For the ease of analysis, we analyze a slightly different algorithm.
Revised: For the ease of analysis, we analyze a slightly different algorithm.

Original: The lemma below is proved in Section D.
Revised: The lemma below is proved in Section D.

Original: Lemma 2.5.
Revised: Lemma 2.5.

Original: Given a matrix B whose rows have unit norm, and are δ -close to B (cid:63) in Euclidean distance up to permutation and sign ﬂip with δ ≤ 1 / (2 κ (cid:63) ) .
Revised: Given a matrix B whose rows have unit norm, and are δ -close to B (cid:63) in Euclidean distance up to permutation and sign ﬂip with δ ≤ 1 / (2 κ (cid:63) ) .

Original: Then, we can give estimates a, B (cid:48) (using e.g., Algorithm 1) such that there exists a permutation P where (cid:107) a − Pa (cid:63) (cid:107) ∞
Revised: Then, we can give estimates a, B (cid:48) (using e.g., Algorithm 1) such that there exists a permutation P where (cid:107) a − Pa (cid:63) (cid:107) ∞

Original: ≤ δa (cid:63) max and B (cid:48) is row-wise δ -close to PB (cid:63) .
Revised: ≤ δa (cid:63) max and B (cid:48) is row-wise δ -close to PB (cid:63) .

Original: The key step towards analyzing objective G ( B ) is the following theorem that gives an analytic formula for G ( · ) . 
Revised: The key step towards analyzing objective G ( B ) is the following theorem that gives an analytic formula for G ( · ) . [Equation]

Original: Theorem 2.6.
Revised: Theorem 2.6.

Original: The function G ( · ) satisﬁes
Revised: The function G ( · ) satisﬁes

Original: [Equation]
Revised: [Equation]

Revised: i,j

Revised: [Equation]

Revised: [ d ]

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Theorem 2.6 is proved in Section A.
Revised: Theorem 2.6 is proved in Section A.

Original: We will motivate our design choices with a brief overview in
Revised: We will motivate our design choices with a brief overview in

Original: Section 3 and formally analyze the landscape of G in Section B (see Theorem B.1).
Revised: Section 3 and formally analyze the landscape of G in Section B (see Theorem B.1).

Original: Finite sample complexity bounds.
Revised: Finite sample complexity bounds.

Original: Extending Theorem 2.3, we can characterize the landscape of the empirical risk (cid:98) G , which implies that stochastic gradient on (cid:98) G also converges approximately to the ground-truth parameters with polynomial number of samples.[Equation]
Revised: Extending Theorem 2.3, we can characterize the landscape of the empirical risk (cid:98) G , which implies that stochastic gradient on (cid:98) G also converges approximately to the ground-truth parameters with polynomial number of samples.

Original: Theorem 2.7.
Revised: Theorem 2.7.

Original: In the setting of Theorem 2.3, suppose we use N empirical samples to approximate
Revised: In the setting of Theorem 2.3, suppose we use N empirical samples to approximate

Original: G and obtain empirical risk (cid:98) G .
Revised: G and obtain empirical risk (cid:98) G .

Original: There exists a ﬁxed polynomial poly ( d, 1 /ε )
Revised: There exists a ﬁxed polynomial poly ( d, 1 /ε )

Original: such that if N ≥ poly ( d, 1 /ε ) , then with high probability the landscape of (cid:98) G has the properties to that of G in bullet 2 and 3 of Theorem 2.3.
Revised: such that if N ≥ poly ( d, 1 /ε ) , then with high probability the landscape of (cid:98) G has the properties to that of G in bullet 2 and 3 of Theorem 2.3.

Original: All of the results above assume that B (cid:63) is orthogonal.
Revised: All of the results above assume that B (cid:63) is orthogonal.

Original: Since the local minimum are preserved by linear transformation of the input space, these results can be extended to the general case when B (cid:63) is not orthogonal but full rank (with some additional technicality) or the case when the dimension is larger than the number of neurons ( m < d ).
Revised: Since the local minimum are preserved by linear transformation of the input space, these results can be extended to the general case when B (cid:63) is not orthogonal but full rank (with some additional technicality) or the case when the dimension is larger than the number of neurons ( m < d ).

Original: See Section C.
Revised: See Section C.

Original:  O VERVIEW : L ANDSCAPE D ESIGN AND A NALYSIS
Revised:  O VERVIEW : L ANDSCAPE D ESIGN AND A NALYSIS

Original: In this section, we present a general overview of ideas behind the design of objective function G ( · ) .
Revised: In this section, we present a general overview of ideas behind the design of objective function G ( · ) .

Original: Inspired by the formula (2.3), in Section 3.1, we envision a family of possible objective functions for which we have unbiased estimators via samples.
Revised: Inspired by the formula (2.3), in Section 3.1, we envision a family of possible objective functions for which we have unbiased estimators via samples.

Original: In Section 3.2, we pick a speciﬁc function that feeds our needs: a) it has no spurious local minimum; b) the global minimum corresponds to the ground-truth parameters.
Revised: In Section 3.2, we pick a speciﬁc function that feeds our needs: a) it has no spurious local minimum; b) the global minimum corresponds to the ground-truth parameters.

Original: 3. W HICH OBJECTIVE CAN BE ESTIMATED BY SAMPLES ?
Revised: 3. W HICH OBJECTIVE CAN BE ESTIMATED BY SAMPLES ?

Original: Recall that in equation (2.2) of Theorem 2.1 we give an analytic formula for the straightforward population risk f .
Revised: Recall that in equation (2.2) of Theorem 2.1 we give an analytic formula for the straightforward population risk f .

Original: Although the population risk f doesn’t perform well empirically, the lesson that we learn from it help us design better objective functions.
Revised: Although the population risk f doesn’t perform well empirically, the lesson that we learn from it help us design better objective functions.

Original: One of the key fact that leads to the proof of Theorem 2.1 is that for any continuous and bounded function γ , we have that
Revised: One of the key fact that leads to the proof of Theorem 2.1 is that for any continuous and bounded function γ , we have that

Original: [Equation]
Revised: [Equation]

Original: Here ˆ σ k and ˆ γ k are the k -th Hermite coefﬁcient of the function σ and γ .
Revised: Here ˆ σ k and ˆ γ k are the k -th Hermite coefﬁcient of the function σ and γ .

Original: That is, letting h k the k -th normalized probabilists’ Hermite polynomials Wikipedia (2017a) and (cid:104)· , ·(cid:105) be the standard inner product between functions, we have ˆ σ k
Revised: That is, letting h k the k -th normalized probabilists’ Hermite polynomials Wikipedia (2017a) and (cid:104)· , ·(cid:105) be the standard inner product between functions, we have ˆ σ k

Original: = (cid:104)
Revised: = (cid:104)

Original: h k , σ (cid:105) .
Revised: h k , σ (cid:105) .

Original: Note that γ can be chosen arbitrarily to extract different terms.
Revised: Note that γ can be chosen arbitrarily to extract different terms.

Original: For example, by choosing γ = h k , we obtain that
Revised: For example, by choosing γ = h k , we obtain that

Original: [Equation]
Revised: [Equation]

Original: That is, we can always access functions forms that involves weighted sum of the powers of (cid:104) b (cid:63)i ,
Revised: That is, we can always access functions forms that involves weighted sum of the powers of (cid:104) b (cid:63)i ,

Original: b i (cid:105) , as in RHS of equation (3.1).
Revised: b i (cid:105) , as in RHS of equation (3.1).

Original: Using a bit more technical tools in Fourier analysis (see details in
Revised: Using a bit more technical tools in Fourier analysis (see details in

Original: Section A), we claim that most of the symmetric polynomials over variables (cid:104) b (cid:63) i , b j (cid:105) can be estimated by samples:
Revised: Section A), we claim that most of the symmetric polynomials over variables (cid:104) b (cid:63)i , b j (cid:105) can be estimated by samples:

Original: Claim 3.1 (informal) .
Revised: Claim 3.1 (informal) .

Original: For any polynomial p () over a single variable, there exits a corresponding function φ p such that
Revised: For any polynomial p () over a single variable, there exits a corresponding function φ p such that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Moreover, for an any polynomial q ( · , · ) over two variables, there exists corresponding φ q such that
Revised: Moreover, for an any polynomial q ( · , · ) over two variables, there exists corresponding φ q such that

Original: [Equation]
Revised: [Equation]

Original: We will not prove these two general claims.
Revised: We will not prove these two general claims.

Original: Instead, we only focus on the formulas in Theorem A. and Theorem A.6, which are two special cases of the claims above.
Revised: Instead, we only focus on the formulas in Theorem A. and Theorem A.6, which are two special cases of the claims above.

Original: Motivated by Claim A.3, in the next subsection, we will pick an objective function which has no spurious local minimum among those functional forms on the right-hand sides of equation (3.2) and (3.3).
Revised: Motivated by Claim A.3, in the next subsection, we will pick an objective function which has no spurious local minimum among those functional forms on the right-hand sides of equation (3.2) and (3.3).

Original: 3. W HICH OBJECTIVE HAS NO SPURIOUS LOCAL MINIMA ?
Revised: 3. W HICH OBJECTIVE HAS NO SPURIOUS LOCAL MINIMA ?

Original: As discussed brieﬂy in the introduction, one of the technical difﬁculties to design and analyze objective functions for neural networks comes from the permutation invariance — if a matrix B is a good solution, then any permutation of the rows of B still gives an equally good solution (if we also permute the coefﬁcients in a accordingly).
Revised: As discussed brieﬂy in the introduction, one of the technical difﬁculties to design and analyze objective functions for neural networks comes from the permutation invariance — if a matrix B is a good solution, then any permutation of the rows of B still gives an equally good solution (if we also permute the coefﬁcients in a accordingly).

Original: We only know of a very limited number of objective functions that guarantee to enjoy permutation invariance and have no spurious local minima Ge et al.
Revised: We only know of a very limited number of objective functions that guarantee to enjoy permutation invariance and have no spurious local minima Ge et al.

Original: We start by considering the objective function used in Ge et al.
Revised: We start by considering the objective function used in Ge et al.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Note that here we overload the notation by using b (cid:63)i ’s to denote a set of ﬁxed vectors that we wanted to recover and using b i ’s to denote the variables.
Revised: Note that here we overload the notation by using b (cid:63)i ’s to denote a set of ﬁxed vectors that we wanted to recover and using b i ’s to denote the variables.

Original: Careful readers may notice that P ( B ) doesn’t fall into the family of functions that we described in the previous section (that is, RHS equation of (3.2) and (3.3)), because it lacks the weighting a (cid:63)i ’s. We will ﬁx this issue later in the subsection.
Revised: Careful readers may notice that P ( B ) doesn’t fall into the family of functions that we described in the previous section (that is, RHS equation of (3.2) and (3.3)), because it lacks the weighting a (cid:63)i ’s. We will ﬁx this issue later in the subsection.

Original: Before that we ﬁrst summarize the nice properties of the landscape of P ( B ) .
Revised: Before that we ﬁrst summarize the nice properties of the landscape of P ( B ) .

Original: For the simplicity of the discussion, let’s assume B (cid:63) forms an orthonormal matrix in the rest of the subsection.
Revised: For the simplicity of the discussion, let’s assume B (cid:63) forms an orthonormal matrix in the rest of the subsection.

Original: Then, any permutation and sign-ﬂip of the rows of B (cid:63) leads to a global minimum of
Revised: Then, any permutation and sign-ﬂip of the rows of B (cid:63) leads to a global minimum of

Original: P ( · ) — when B = SQB (cid:63) with a permutation matrix Q and a sign matrix S (diagonal with ± 1 ),
Revised: P ( · ) — when B = SQB (cid:63) with a permutation matrix Q and a sign matrix S (diagonal with ± 1 ),

Original: we have that P ( B ) = 

Revised: we have that P ( B ) =

Original: because one of (cid:104) b (cid:63)i , b j (cid:105) 2 and (cid:104) b (cid:63)i , b k (cid:105) 2 has to be zero for all i, j, k 7 ).
Revised: because one of (cid:104) b (cid:63)i , b j (cid:105) 2 and (cid:104) b (cid:63)i , b k (cid:105) 2 has to be zero for all i, j, k 7 ).

Original: It turns out that these permutations/sign-ﬂips of B (cid:63) are also the only local minima 8 of function
Revised: It turns out that these permutations/sign-ﬂips of B (cid:63) are also the only local minima 8 of function

Original: P ( · ) .
Revised: P ( · ) .

Original: To see this, notice that P ( B ) is a degree-4 polynomial of B .
Revised: To see this, notice that P ( B ) is a degree-4 polynomial of B .

Original: Thus if we pick an index s and ﬁx every row except for b s , then P ( B ) is a quadratic function over unit vector b s – reduces to an smallest eigenvector problem.
Revised: Thus if we pick an index s and ﬁx every row except for b s , then P ( B ) is a quadratic function over unit vector b s – reduces to a smallest eigenvector problem.

Original: Eigenvector problems are known to have no spurious local minimum.
Revised: Eigenvector problems are known to have no spurious local minimum.

Original: Thus the corresponding function (w.r.t b s ) has no spurious local minimum.
Revised: Thus the corresponding function (w.r.t b s ) has no spurious local minimum.

Original: It turns out the same property still holds when we treat all the rows as variables and add the row-wise norm constraints.
Revised: It turns out the same property still holds when we treat all the rows as variables and add the row-wise norm constraints.

Original: However, there are two issues with using objective function P ( B ) .
Revised: However, there are two issues with using objective function P ( B ) .

Original: The obvious one is that it doesn’t involve the coefﬁcients a (cid:63)i ’s and thus doesn’t fall into the forms of equation (3.3).
Revised: The obvious one is that it doesn’t involve the coefﬁcients a (cid:63)i ’s and thus doesn’t fall into the forms of equation (3.3).

Original: Optimistically, we would hope that for nonnegative a (cid:63)i ’s the weighted version of P below would also enjoy the similar landscape property
Revised: Optimistically, we would hope that for nonnegative a (cid:63)i ’s the weighted version of P below would also enjoy the similar landscape property

Original: [Equation]
Revised: [Equation]

Original: When a (cid:63)i ’s are positive, indeed the global minimum of P (cid:48) are still just all the permutations of the
Revised: When a (cid:63)i ’s are positive, indeed the global minimum of P (cid:48) are still just all the permutations of the

Original: B (cid:63) .
Revised: B (cid:63) .

Original: However, when max a (cid:63)i > 2 min a (cid:63)i , we found that P (cid:48) starts to have spurious local minima .
Revised: However, when max a (cid:63)i > 2 min a (cid:63)i , we found that P (cid:48) starts to have spurious local minima .

Original: It seems that spurious local minimum often occurs when a row of B is a linear combination of a smaller number of rows of B (cid:63) .
Revised: It seems that spurious local minimum often occurs when a row of B is a linear combination of a smaller number of rows of B (cid:63) .

Original: See Section F for a concrete example.
Revised: See Section F for a concrete example.

Original: {+Iterations × 10 
+}
Revised: {+Iterations × 10 
+}

Revised: O b j e c t i v e V a l ue 0.

Original: {+Iterations × 10 
+}
Revised: {+Iterations × 10 
+}

Revised: P a r a m e t e r E rr o r 0.

Original: To remove such spurious local minima, we add a regularization term below that pushes each row of
Revised: To remove such spurious local minima, we add a regularization term below that pushes each row of

Original: B to be close to one of the rows of B (cid:63) ,
Revised: B to be close to one of the rows of B (cid:63) ,

Original: [Equation]
Revised: [Equation]

Original: We see that for each ﬁxed j , the part in R ( B ) that involves b j has the form − µ (cid:80) i a (cid:63)i (cid:104) b (cid:63)i , b j (cid:105) 4 = − µ (cid:104) (cid:80) i a (cid:63)i b (cid:63)i ⊗ 4 , b ⊗ 4 j (cid:105)
Revised: We see that for each ﬁxed j , the part in R ( B ) that involves b j has the form − µ (cid:80) i a (cid:63)i (cid:104) b (cid:63)i , b j (cid:105) 4 = − µ (cid:104) (cid:80) i a (cid:63)i b (cid:63)i ⊗ 4 , b ⊗ 4 j (cid:105)

Original: This is commonly used objective function for decomposing tensor (cid:80) i a (cid:63)i b (cid:63)i ⊗ 4 .
Revised: This is commonly used objective function for decomposing tensor (cid:80) i a (cid:63)i b (cid:63)i ⊗ 4 .

Original: It’s known that for orthogonal b (cid:63)i ’s, the only local minima are ± b (cid:63) 1 , . .
Revised: It’s known that for orthogonal b (cid:63)i ’s, the only local minima are ± b (cid:63) 1 , . .

Original: ± b (cid:63)d Ge et al.
Revised: ± b (cid:63)d Ge et al.

Original: Therefore, intuitively R ( B ) pushes each of the b i ’s towards one of the b (cid:63)i ’s. 10 Choosing µ to be small enough, it turns out that P (cid:48)
Revised: Therefore, intuitively R ( B ) pushes each of the b i ’s towards one of the b (cid:63)i ’s. 10 Choosing µ to be small enough, it turns out that P (cid:48)

Original: ( B ) + R ( B ) doesn’t have any spurious local minimum as we will show in Section B.j . [Figure] [Figure]
Revised: ( B ) + R ( B ) doesn’t have any spurious local minimum as we will show in Section B.

Original: Another issue with the choice of P (cid:48)
Revised: Another issue with the choice of P (cid:48)

Original: ( B ) + R ( B ) is that we are still having a constraint minimization problem.
Revised: ( B ) + R ( B ) is that we are still having a constraint minimization problem.

Original: Such row-wise norm constraints only make sense when the ground-truth B (cid:63) is orthogonal and thus has unit row norm.
Revised: Such row-wise norm constraints only make sense when the ground-truth B (cid:63) is orthogonal and thus has unit row norm.

Original: A straightforward generalization of P ( B ) to non-orthogonal case requires some special constraints that also depend on the covariance matrix B (cid:63) B (cid:63) (cid:62) , which in turn requires a specialized procedure to estimate.
Revised: A straightforward generalization of P ( B ) to non-orthogonal case requires some special constraints that also depend on the covariance matrix B (cid:63) B (cid:63) (cid:62) , which in turn requires a specialized procedure to estimate.

Original: Instead, we move the constraints into the objective function by considering adding another regularization term that approximately enforces the constraints
Revised: Instead, we move the constraints into the objective function by considering adding another regularization term that approximately enforces the constraints.

Original: [Figure]

Original: It turns out the following regularizer sufﬁces for the orthogonal case: S ( B )
Revised: It turns out the following regularizer sufﬁces for the orthogonal case: S ( B )

Original: =
Revised: =

Original: λ (cid:80) i
Revised: λ (cid:80) i

Original: ( (cid:107) b
Revised: ( (cid:107) b

Original: i (cid:107) 2 − 1) 2 .
Revised: i (cid:107) 2 − 1) 2 .

Original: Moreover, we can extend this easily to the non-orthogonal case (see Section C) without estimating any statistics of B (cid:63) in advance.
Revised: Moreover, we can extend this easily to the non-orthogonal case (see Section C) without estimating any statistics of B (cid:63) in advance.

Original: We note that S ( B ) is not the Lagrangian multiplier and it does change the global minima slightly.
Revised: We note that S ( B ) is not the Lagrangian multiplier and it does change the global minima slightly.

Original: We will take λ to be large enough so that (cid:107) b i (cid:107) has to be close to 1.
Revised: We will take λ to be large enough so that (cid:107) b i (cid:107) has to be close to 1.

Original: As a summary, we ﬁnally use the unconstrained objective
Revised: As a summary, we ﬁnally use the unconstrained objective

Original: [Equation]
Revised: [Equation]

Original: Since R ( B ) and S ( B ) are degree-4 polynomials of B , the analysis of G ( B ) is much more delicate, and we cannot use much linear algebra as we could for P (cid:48)
Revised: Since R ( B ) and S ( B ) are degree-4 polynomials of B , the analysis of G ( B ) is much more delicate, and we cannot use much linear algebra as we could for P (cid:48)

Original: ( B ) .
Revised: ( B ) .

Original: See Section B for details.
Revised: See Section B for details.

Original: Finally we note that a feature of this objective G ( · ) is that it only takes B as variables.
Revised: Finally we note that a feature of this objective G ( · ) is that it only takes B as variables.

Original: We will estimate the value of a (cid:63) after we recover the value of B .
Revised: We will estimate the value of a (cid:63) after we recover the value of B .

Original: (see Section D).
Revised: (see Section D).

Original: [Figure]

Original: i v

Original: [Figure]

Original: V

Original: [Figure]

Original: P a r a m e t e r E rr o r 0.

Original:  S IMULATION
Revised:  S IMULATION

Original: In this section, we provide simple simulation results that verify that minimizing G ( B ) with SGD recovers a permutation of B (cid:63) ; however, minimizing Equation (2.2) with SGD results in ﬁnding spurious local minima.
Revised: In this section, we provide simple simulation results that verify that minimizing G ( B ) with SGD recovers a permutation of B (cid:63) ; however, minimizing Equation (2.2) with SGD results in ﬁnding spurious local minima.

Original: Based on the formula for the population risk in Equation (2.3), we also  veriﬁed empirically the conjecture that SGD would successfully recover B (cid:63) using the activation functions γ ( z )=
Revised: Based on the formula for the population risk in Equation (2.3), we also [Figure] [Figure] veriﬁed empirically the conjecture that SGD would successfully recover B (cid:63) using the activation functions γ ( z )

Original:  ˆ σ 2 h 2 ( z ) + ˆ σ 

Revised: = ˆ σ 2 h 2 ( z ) + ˆ σ

Original: h 4 ( z ) , 11 even if the data were generated via a model with ReLU activation.
Revised: h 4 ( z ) , 11 even if the data were generated via a model with ReLU activation.

Original: (See Section 2.1 for the rationale behind such conjectures.)
Revised: (See Section 2.1 for the rationale behind such conjectures [Figure]

Original: For all of our experiments, we chose B (cid:63)=
Revised: For all of our experiments [Figure] we chose B (cid:63)

Original:  Id d × d with dimension d = 50 and a (cid:63) = 1 for simplicity, and the data is generated from a one-hidden-layer network with ReLU or Sigmoid activation without noise.
Revised: = Id d × d with dimension d = 50 and a (cid:63) = 1 for simplicity, and the data is generated from a one-hidden-layer network with ReLU or Sigmoid activation without noise.

Original: We use stochastic gradient descent with fresh samples at each iteration, and we plot the (expected) population error (that is, the error on a fresh batch of examples).
Revised: We use stochastic gradient descent with fresh samples at each iteration, and we plot the (expected) population error (that is, the error on a fresh batch of examples).

Original: [Figure] To test whether SGD converges to a matrix B which is equivalent to B (cid:63) up to permutation of rows, we use a surrogate error metric to evaluate whether B (cid:63) − 1 B is close to a permutation matrix.
Revised: To test whether SGD converges to a matrix B which is equivalent to B (cid:63) up to permutation of rows, we use a surrogate error metric to evaluate whether B (cid:63) − 1 B is close to a permutation matrix.

Original: Given a matrix Q with row norm 1, let
Revised: Given a matrix Q with row norm 1, let

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then we have that if e ( Q ) ≤ ε for some ε < 1 / 3 , then it implies that Q is √ 2 ε -close to a permutation matrix in inﬁnity norm.
Revised: Then we have that if e ( Q ) ≤ ε for some ε < 1 / 3 , then it implies that Q is √ 2 ε -close to a permutation matrix in inﬁnity norm.

Original: On the other direction, we know that if e ( Q ) > ε , then Q is not ε -close to any permutation matrix in inﬁnity norm.
Revised: On the other direction, we know that if e ( Q ) > ε , then Q is not ε -close to any permutation matrix in inﬁnity norm.

Original: The latter statement also holds when Q doesn’t have row norm 1 .
Revised: The latter statement also holds when Q doesn’t have row norm 1 .

Original: Figure 1 shows that without over-parameterization, using ReLU as an activation function, SGD doesn’t converge to zero test error and the ground-truth parameters.
Revised: Figure 1 shows that without over-parameterization, using ReLU as an activation function, SGD doesn’t converge to zero test error and the ground-truth parameters.

Original: We decreased step-size by a  factor of 4 every 5000 number of iterations after the error plateaus at 10000 iterations.
Revised: We decreased step-size by a 8 Iterations 10-0. O b j e c t i v e V a l ue    8 Iterations 10 0. P a r a m e t e r E rr o r factor of 4 every 5000 number of iterations after the error plateaus at 10000 iterations.

Original: For the ﬁnal 5000 iterations, the step-size is less than 10 − 9 , so we can be conﬁdent that the non-zero objective value is not due to the variance of SGD.
Revised: For the ﬁnal 5000 iterations, the step-size is less than 10 − 9 , so we can be conﬁdent that the non-zero objective value is not due to the variance of SGD.

Original: We see that none of the ﬁve runs of SGD converged to a global minimum.
Revised: We see that none of the ﬁve runs of SGD converged to a global minimum.

Original: Figure 2 shows the result for sigmoid activation which is quantitatively similar.
Revised: Figure 2 shows the result for sigmoid activation which is quantitatively similar.

Original: Figure 3 shows that using ˆ σ 2 h 2 + ˆ σ 4 h 4 as the activation function, SGD with projection to the set of matrices B with row norm 1 converges to the ground-truth parameters.
Revised: Figure 3 shows that using ˆ σ 2 h 2 + ˆ σ 4 h 4 as the activation function, SGD with projection to the set of matrices B with row norm 1 converges to the ground-truth parameters.

Original: We also plot the loss function which converges the value of a global minimum.
Revised: We also plot the loss function which converges the value of a global minimum.

Original: (We subtracted the constant term in equation (2.6) so that the global minimum has loss 0.)
Revised: (We subtracted the constant term in equation (2.6) so that the global minimum has loss 0.)

Original: Figure 4 shows that using our objective function G ( B ) , the iterate converges to a permutation of the ground truth matrix B (cid:63) .
Revised: Figure 4 shows that using our objective function G ( B ) , the iterate converges to a permutation of the ground truth matrix B (cid:63) .

Original: The fact that the parameter error goes up and down is not surprising, because the algorithm ﬁrst gets close to a saddle point and then breaks ties and converges to a one of the global minima.
Revised: The fact that the parameter error goes up and down is not surprising, because the algorithm ﬁrst gets close to a saddle point and then breaks ties and converges to a one of the global minima.

Original: Finally we note that using the loss function G ( · ) seems to require signiﬁcantly larger batch (and sample complexity) to reduce the variance in the gradients estimation.
Revised: Finally we note that using the loss function G ( · ) seems to require signiﬁcantly larger batch (and sample complexity) to reduce the variance in the gradients estimation.

Original: We used batch sizein the experiment for G ( · ) .
Revised: We used batch sizein the experiment for G ( · ) .

Original: However, in contrast, for the ˆ σ 2 h 2 + ˆ σ 4 h 4 we used batch size 8192 and for relu we used batch size 256.
Revised: However, in contrast, for the ˆ σ 2 h 2 + ˆ σ 4 h 4 we used batch size 8192 and for relu we used batch size 256.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we ﬁrst give an analytic formula for the population risk of the standard (cid:96) 2 loss, which empirically may converge to a spurious local minimum.
Revised: In this paper we ﬁrst give an analytic formula for the population risk of the standard (cid:96) 2 loss, which empirically may converge to a spurious local minimum.

Original: We then design a novel population loss that is guaranteed to have no spurious local minimum.
Revised: We then design a novel population loss that is guaranteed to have no spurious local minimum.

Original: Designing objective functions with well-behaved landscape is an intriguing and potentially fruitful direction.
Revised: Designing objective functions with well-behaved landscape is an intriguing and potentially fruitful direction.

Original: We hope that our techniques can be useful for characterizing and designing the optimization landscape for other settings.
Revised: We hope that our techniques can be useful for characterizing and designing the optimization landscape for other settings.

Original: We conjecture that the objective αf 2 + βf 4 12 has no spurious local minimum when α, β are reasonable constants and the ground-truth parameters are in general position.
Revised: We conjecture that the objective αf 2 + βf 4 12 has no spurious local minimum when α, β are reasonable constants and the ground-truth parameters are in general position.

Original: We provided empirical evidence to support the conjecture.
Revised: We provided empirical evidence to support the conjecture.

Original: Our results assume that the input distribution is Gaussian.
Revised: Our results assume that the input distribution is Gaussian.

Original: Extending them to other input distributions is a very interesting open problem.
Revised: Extending them to other input distributions is a very interesting open problem.

Revised: A CKNOWLEDGEMENT

Revised: We thank Chi Jin for discussions at the beginning of this work.

Revised: R.G. is funded by NSF CCF1704860.

Original: A A NALYTIC F ORMULA FOR P OPULATION R ISKS A. B ASICS ON H ERMITE P OLYNOMIALS
Revised: A A NALYTIC F ORMULA FOR P OPULATION R ISKS A. B ASICS ON H ERMITE P OLYNOMIALS

Original: In this section, we brieﬂy review Hermite polynomials and Fourier analysis on Gaussian space.
Revised: In this section, we brieﬂy review Hermite polynomials and Fourier analysis on Gaussian space.

Original: Let
Revised: Let

Original: H j be the probabilists’ Hermite polynomial Wikipedia (2017a), and let h j = 1 √ j !
Revised: H j be the probabilists’ Hermite polynomial Wikipedia (2017a), and let h j = 1 √ j !

Original: H
Revised: H

Original: j be the normalized Hermite polynomials.
Revised: j be the normalized Hermite polynomials.

Original: The normalized Hermite polynomial forms a complete orthonormal basis in the function space L 2 ( R , e − x 2 / 2 ) in the following sense 13 .
Revised: The normalized Hermite polynomial forms a complete orthonormal basis in the function space L 2 ( R , e − x 2 / 2 ) in the following sense 13 .

Original: For two functions f, g that map R to R , deﬁne the inner product (cid:104) f, g (cid:105) with respect to the Gaussian measure as
Revised: For two functions f, g that map R to R , deﬁne the inner product (cid:104) f, g (cid:105) with respect to the Gaussian measure as

Original: [Equation]
Revised: [Equation]

Original: The polynomials h 0 , . . .
Revised: The polynomials h 0 , . . .

Original: , h m , . .
Revised: , h m , . .

Original: . are orthogonal to each other under this inner product:
Revised: . are orthogonal to each other under this inner product:

Original: [Equation]
Revised: [Equation]

Original: Here δ ij = 1 if i = j and otherwise δ ij = 0 .
Revised: Here δ ij = 1 if i = j and otherwise δ ij = 0 .

Original: Given a function σ ∈ L 2 ( R , e − x 2 / 2 ) , let the k -th
Revised: Given a function σ ∈ L 2 ( R , e − x 2 / 2 ) , let the k -th

Original: Hermite coefﬁcient of σ be deﬁned as
Revised: Hermite coefﬁcient of σ be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Since h 0 , . . .
Revised: Since h 0 , . . .

Original: , h m , . .
Revised: , h m , . .

Original: . , forms a complete orthonormal basis, we have the expansion that
Revised: . , forms a complete orthonormal basis, we have the expansion that

Original: [Equation]
Revised: [Equation]

Original: We will leverage several other nice properties of the Hermite polynomials in our proofs.
Revised: We will leverage several other nice properties of the Hermite polynomials in our proofs.

Original: The following claim connects the Hermite polynomial to the coefﬁcients of Taylor expansion of a certain exponential function.
Revised: The following claim connects the Hermite polynomial to the coefﬁcients of Taylor expansion of a certain exponential function.

Original: It can also serve as a deﬁnition of Hermite polynomials.
Revised: It can also serve as a deﬁnition of Hermite polynomials.

Revised: space, namely,

Revised: [Equation]

Original: Claim A.1 ((O’Donnell, 2014, Equation 11.8)) .
Revised: Claim A.1 ((O’Donnell, 2014, Equation 11.8)) .

Original: We have that for t, z ∈ R ,
Revised: We have that for t, z ∈ R ,

Original: [Equation]
Revised: [Equation]

Original: The following Claims shows that the expectation E
Revised: The following Claims shows that the expectation E

Original: [ h n ( x ) h m ( y )] can be computed easily when x, y are (correlated) Gaussian random variables.
Revised: [ h n ( x ) h m ( y )] can be computed easily when x, y are (correlated) Gaussian random variables.

Original: Claim A.2 ((O’Donnell, 2014, Section 11.2)) .
Revised: Claim A.2 ((O’Donnell, 2014, Section 11.2)) .

Original: Let ( x, y ) be ρ -correlated standard normal variables (that is, both x , y have marginal distribution N (0 , 1) and E
Revised: Let ( x, y ) be ρ -correlated standard normal variables (that is, both x , y have marginal distribution N (0 , 1) and E

Original: [ xy ] = ρ ).
Revised: [ xy ] = ρ ).

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: As a direct corollary, we can compute E x ∼N(0 , Id d d ) (cid:2) σ (u (cid:62) x ) γ ( v (cid:62) x ) (cid:3) by expanding in the Hermite basis and applying the Claim above.
Revised: As a direct corollary, we can compute E x ∼N (

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: γ

Original: [Equation]
Revised: [Equation]

Revised: by expanding in the Hermite basis and applying the Claim above.

Original: Claim A.3 .
Revised: Claim A.3 .

Original: Let σ, γ be two functions from R to R such that σ 2 , γ 2 ∈ L 2 ( R , e − x 2 / 2 ) .
Revised: Let σ, γ be two functions from R to R such that σ 2 , γ 2 ∈ L 2 ( R , e − x 2 / 2 ) .

Original: Then, for any unit vectors u, v ∈ R d , we have that
Revised: Then, for any unit vectors u, v ∈ R d , we have that

Original: [Equation]
Revised: [Equation]

Original: Proof of Claim A.3.
Revised: Proof of Claim A.3.

Original: Let s =
Revised: Let s =

Original: u (cid:62) x and t = v (cid:62) x .
Revised: u (cid:62) x and t = v (cid:62) x .

Original: Then s, t are two spherical standard normal random variables that are (cid:104) u, v (cid:105) -correlated, and we have that
Revised: Then s, t are two spherical standard normal random variables that are (cid:104) u, v (cid:105) -correlated, and we have that

Original: [Equation]
Revised: [Equation]

Revised: t )] .

Original: We expand σ ( s ) and γ ( t ) in the Fourier basis and obtain that
Revised: We expand σ ( s ) and γ ( t ) in the Fourier basis and obtain that

Original: [Equation]
Revised: [Equation]

Revised: Claim A.2)

Original: A. A NALYTIC F ORMULA FOR POPULATION RISK f AND f (cid:48)
Revised: A. A NALYTIC F ORMULA FOR POPULATION RISK f AND f (cid:48)

Original: In this section we prove Theorem 2.1 and Theorem 2.2, which both follow from the following more general Theorem.
Revised: In this section we prove Theorem 2.1 and Theorem 2.2, which both follow from the following more general Theorem.

Original: Theorem A.4.
Revised: Theorem A.4.

Original: Let γ, σ ∈ L 2 ( R , e − x 2 / 2 ) , and ˆ y = a (cid:62) γ ( Bx ) with parameter a ∈ R (cid:96) and B ∈ R (cid:96) × d .
Revised: Let γ, σ ∈ L 2 ( R , e − x 2 / 2 ) , and ˆ y = a (cid:62) γ ( Bx ) with parameter a ∈ R (cid:96) and B ∈ R (cid:96) × d .

Original: Deﬁne the population risk f γ as
Revised: Deﬁne the population risk f γ as

Original: [Equation]
Revised: [Equation]

Original: Suppose B =   b (cid:62)
Revised: Suppose B =   b (cid:62)...

Original: [Equation]

Original: b (cid:62) [Equation]
Revised: b (cid:62) (cid:96) 

Original:   and B (cid:63) =   b (cid:63) 1 (cid:62) ...
Revised:  and B (cid:63) =   b (cid:63) 1 (cid:62) ...

Original: b (cid:63)m (cid:62) 
Revised: b (cid:63) m (cid:62) 

Original:  and b i ’s and b (cid:63)i ’s have unit (cid:96) 2 norm.
Revised:  and b i ’s and b (cid:63)i ’s have unit (cid:96) 2 norm.

Original: Then, 
Revised: Then, f ( a, B ) = (cid:88)

Original: [Equation]
Revised: k ∈ N (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ˆ σ k (cid:88) i ∈ [ m ] a (cid:63)i b (cid:63)i ⊗ k − ˆ γ k (cid:88) i ∈ [ (cid:96) ] a i b ⊗ k i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: F + const , where ˆ σ k , ˆ γ k are the k -th Hermite coefﬁcients of the function σ and γ respectively.
Revised: F + const , where ˆ σ k , ˆ γ k are the k -th Hermite coefﬁcients of the function σ and γ respectively.

Original: We can see that Theorem 2.1 follows from choosing γ = σ and Theorem 2.2 follows from choosing γ = ˆ σ 2 h 2 + ˆ σ 4 h 4 .
Revised: We can see that Theorem 2.1 follows from choosing γ = σ and Theorem 2.2 follows from choosing γ = ˆ σ 2 h 2 + ˆ σ 4 h 4 .

Original: The key intuition here is that we can decompose σ into a weighted combination of Hermite polynomials, and each Hermite polynomial inﬂuence the population risk more or less independently (because they are orthogonal polynomials with respect to the Gaussian measure).
Revised: The key intuition here is that we can decompose σ into a weighted combination of Hermite polynomials, and each Hermite polynomial inﬂuence the population risk more or less independently (because they are orthogonal polynomials with respect to the Gaussian measure).

Original: Proof of Theorem A.4.
Revised: Proof of Theorem A.4.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: Claim A.3)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: A. A NALYTIC F ORMULA FOR POPULATION RISK G
Revised: A. A NALYTIC F ORMULA FOR POPULATION RISK G

Original: In this section we show that the population risk G ( · ) (deﬁned as in equation (2.7)) has the following analytical formula:
Revised: In this section we show that the population risk G ( · ) (deﬁned as in equation (2.7)) has the following analytical formula:

Original: [Equation]
Revised: [Equation]

Original: The formula will be crucial for the analysis of the landscape of G ( · ) in Section
Revised: The formula will be crucial for the analysis of the landscape of G ( · ) in Section

Original: B. The formula follows straightforwardly from the following two theorems and the deﬁnition (2.7).
Revised: B. The formula follows straightforwardly from the following two theorems and the deﬁnition (2.7).

Original: Theorem A.5.
Revised: Theorem A.5.

Original: Let φ ( · , · , · ) be deﬁned as in equation (2.9) [Equation] we have that
Revised: Let φ ( · , · , · ) be deﬁned as in equation (2.9) , we have that

Original: [Equation]
Revised: [Equation]

Original: Theorem A.6.
Revised: Theorem A.6.

Original: Let ϕ ( · , · ) be deﬁned as in equation (2.8) [Equation] then we have that
Revised: Let ϕ ( · , · ) be deﬁned as in equation (2.8) , then we have that

Original: [Equation]
Revised: [Equation]

Original: In the rest of the section we prove Theorem A.5 and A.6.
Revised: In the rest of the section we prove Theorem A.5 and A.6.

Original: We start with a simple but fundamental lemma.
Revised: We start with a simple but fundamental lemma.

Original: Essentially all the result in this section follows from expanding the two sides of equation (A.1) below.
Revised: Essentially all the result in this section follows from expanding the two sides of equation (A.1) below.

Original: Lemma A.7.
Revised: Lemma A.7.

Original: Let u, v ∈ R d be two ﬁxed vectors and x ∼ N (0 , Id d × d ) .
Revised: Let u, v ∈ R d be two ﬁxed vectors and x ∼ N (0 , Id d × d ) .

Original: Then, for any s, t ∈ R ,
Revised: Then, for any s, t ∈ R ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the fact that E (cid:2) exp( v (cid:62) x ) (cid:3) =
Revised: Using the fact that E (cid:2) exp( v (cid:62) x ) (cid:3) =

Original: exp( 12 (cid:107) v (cid:107) 2 ) , we have that,
Revised: exp( 12 (cid:107) v (cid:107) 2 ) , we have that,

Original: [Equation]
Revised: [Equation]

Original: (by the formula E (cid:2) exp( v (cid:62) x ) (cid:3) = exp( 12 (cid:107) v (cid:107) 2 ) )
Revised: (by the formula E (cid:2) exp( [Equation] (cid:62) [Equation]

Original: =
Revised: =

Revised: [Equation]

Revised: ) =

Revised: [Equation]

Original: exp( (cid:104) u, v (cid:105) st ) .
Revised: ( (cid:104) u, [Equation] (cid:105) st ) .

Original: Next we extend some of the results in the previous section to the setting with different scaling (such as when v in Claim A.3 is no longer a unit vector.)
Revised: Next we extend some of the results in the previous section to the setting with different scaling (such as when v in Claim A.3 is no longer a unit vector.)

Original: Lemma A.8.
Revised: Lemma A.8.

Original: Let u be a ﬁxed unit vector and v be an arbitrary vector in R d .
Revised: Let u be a ﬁxed unit vector and v be an arbitrary vector in R d .

Original: Let ϕ ( v, x ) = 18 (cid:107) 
Revised: Let ϕ ( v, x ) = 18 (cid:107) v

Original: [Equation]
Revised: [Equation]

Original: As a sanity check, we can verify that when v is a unit vector, ϕ ( v, x ) = √ 24 h 4 ( v (cid:62) x ) and th Lemma reduces to a special case of Claim A.2.
Revised: As a sanity check, we can verify that when v is a unit vector, ϕ ( v, x ) = √ 24 h 4 ( v (cid:62) x ) and th Lemma reduces to a special case of Claim A.2.

Original: Proof.
Revised: Proof.

Original: Let A, B be formal power series in variable s, t deﬁned as A = exp( (cid:104) u, v (cid:105) st ) and B =
Revised: Let A, B be formal power series in variable s, t deﬁned as A = exp( (cid:104) u, v (cid:105) st ) and B =

Original: E (cid:2) exp( u (cid:62) xt − 12 (cid:107) u
Revised: E (cid:2) exp( u (cid:62) xt − 12 (cid:107) u

Original: (cid:107) [Equation] [Equation]
Revised: (cid:107) 2 t 2 ) exp

Original: [Equation] (cid:62) [Equation] 12 (cid:107) v (cid:107) 2 s 2 ) (cid:3) .
Revised: ( v (cid:62) xs − 12 (cid:107) v (cid:107) 2 s 2 ) (cid:3) .

Original: We refer the readers to Wikipedia (2017b) for more backgrounds of power series.
Revised: We refer the readers to Wikipedia (2017b) for more backgrounds of power series.

Original: For casual readers, one can just think of A as B as two power series obtained by expanding the exp ( · ) via Taylor expansion.
Revised: For casual readers, one can just think of A as B as two power series obtained by expanding the exp ( · ) via Taylor expansion.

Original: For a formal power series A in variable x , let [ x α ] A to denote coefﬁcient in front of the monomial x α .
Revised: For a formal power series A in variable x , let [ x α ] A to denote coefﬁcient in front of the monomial x α .

Original: By Lemma A.7, we have that A = B , and thus
Revised: By Lemma A.7, we have that A = B , and thus

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last line is by the fact that ϕ ( v, x ) =
Revised: where the last line is by the fact that ϕ ( v, x ) =

Original: [ s 4 ] exp( v (cid:62) xs − 12 (cid:107) v (cid:107) 2 s 2 ) .
Revised: [ s 4 ] exp( v (cid:62) xs − 12 (cid:107) v (cid:107) 2 s 2 ) .

Original: This can be veriﬁed by applying Claim A.1 with t =
Revised: This can be veriﬁed by applying Claim A.1 with t =

Original: s (cid:107) v (cid:107) and z = v T x (cid:107) v (cid:107) , and noting that H 4 ( x ) =
Revised: s (cid:107) v (cid:107) and z = v T x (cid:107) v (cid:107) , and noting that H 4 ( x ) =

Original: x 4 − 6 x 2 + 3 .
Revised: x 4 − 6 x 2 + 3 .

Original: Now we are ready prove Theorem A.6 using Lemma A.8.
Revised: Now we are ready prove Theorem A.6 using Lemma A.8.

Original: Proof of Theorem A.6.
Revised: Proof of Theorem A.6.

Original: Using the fact that σ ( v (cid:62) x ) = (cid:80) ∞
Revised: Using the fact that σ ( v (cid:62) x ) = (cid:80) ∞

Original: k
Revised: k

Original: =

Revised: =

Original: ˆ σ k h k( v (cid:62) x ) , we have that
Revised: ˆ σ k h k

Original: [Equation]
Revised: [Equation]

Original: (cid:63)i
Revised: v

Original: [Equation]
Revised: [Equation]

Original: (cid:63)i
Revised: x ) , we have that

Original: [Equation]
Revised: [Equation]

Original: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (by Lemma A.8 and h j = 1 √ j H j )
Revised: (by Lemma A.8 and h j = [Equation] j H j )

Original: Towards proving Theorem A.5, we start with the following Lemma.
Revised: Towards proving Theorem A.5, we start with the following Lemma.

Original: Inspired by the proofs above, we design a function φ ( v, w, x ) such that we can estimate (cid:104) u, v (cid:105) 2 (cid:104) u, w (cid:105) 2 by taking expectation of 
Revised: Inspired by the proofs above, we design a function φ ( v, w, x ) such that we can estimate (cid:104) u, v (cid:105) 2 (cid:104) u, w (cid:105) 2 by taking expectation of E (cid:2) σ [Equation] u

Revised: [Equation]

Revised: x

Original: E (cid:2) σ ( u (cid:62) x ) φ ( v, w, x ) (cid:3) .
Revised: [Equation] ( v, w, x [Equation] (cid:3) .

Original: Lemma A.9.
Revised: Lemma A.9.

Original: Let a be a ﬁxed unit vector in R d and v, w two ﬁxed vectors in R d .
Revised: Let a be a ﬁxed unit vector in R d and v, w two ﬁxed vectors in R d .

Original: Let ϕ ( · , · ) be deﬁned as in Lemma A.8.
Revised: Let ϕ ( · , · ) be deﬁned as in Lemma A.8.

Original: Deﬁne φ ( v, w, x ) as
Revised: Deﬁne φ ( v, w, x ) as

Original: [Equation]
Revised: [Equation]

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using the fact that(cid:104) u, v
Revised: Using the fact that

Original: [Equation]
Revised: [Equation]

Original: w
Revised: and

Revised: Lemma A.

Original: [Equation]
Revised: [Equation]

Original: + [Equation] [Equation] u
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: (cid:105){+Using the fact that σ ( u (cid:62) x ) = (cid:80) ∞ k =
 ˆ σ k h k ( u (cid:62) x )+}
Revised: {+Using the fact that σ ( u (cid:62) x ) = (cid:80) ∞ k =
 ˆ σ k h k ( u (cid:62) x )+}

Original: [Equation]
Revised: [Equation]

Original: − 2 (cid:104) u, w (cid:105) 4 = 12 (cid:104) u, v (cid:105) 2 (cid:104) u, w (cid:105) 2 and Lemma A.8, we have that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: [-Using the fact that σ ( u (cid:62) x ) = (cid:80) ∞ k =
 ˆ σ k-] hk ( u (cid:62) x ) , we conclude that
Revised: [-Using the fact that σ ( u (cid:62) x ) = (cid:80) ∞ k =
 ˆ σ k-]by Lemma A.8 and h

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to prove Theorem A.5 by using Lemma A.9 for every summand.
Revised: Now we are ready to prove Theorem A.5 by using Lemma A.9 for every summand.

Original: Proof of Theorem A.5.
Revised: Proof of Theorem A.5.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: B L ANDSCAPE OF P OPULATION R ISK G ( · )
Revised: B L ANDSCAPE OF P OPULATION R ISK G ( · )

Original: In this section we prove Theorem 2.3.
Revised: In this section we prove Theorem 2.3.

Original: Since the landscape property is invariant with respect to rotations of parameters [Equation] without loss of generality we assume B (cid:63) is the identity matrix Id throughout this section.
Revised: Since the landscape property is invariant with respect to rotations of parameters, without loss of generality we assume B (cid:63) is the identity matrix Id throughout this section.

Original: (See Section C for a precise statement for the invariance.)
Revised: (See Section C for a precise statement for the invariance.)

Original: Recall that by Theorem 2.6, the population risk G ( · ) in the case of B (cid:63)
Revised: Recall that by Theorem 2.6, the population risk G ( · ) in the case of B (cid:63)

Original: =
Revised: =

Original: Id is equal to
Revised: Id is equal to

Original: [Equation]
Revised: [Equation]

Original: In the rest of section we work with the formula above for G ( · ) instead of the original deﬁnition.
Revised: In the rest of section we work with the formula above for G ( · ) instead of the original deﬁnition.

Original: In fact, for future reference, we study a more general version of the function G .
Revised: In fact, for future reference, we study a more general version of the function G .

Original: For nonnegative vectors α, β and nonnegative number µ , let G α,β,µ be deﬁned asG α,β,µ ( B
Revised: For nonnegative vectors α, β and nonnegative number µ , let G α,β,µ be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: (B.2)

Original: Here e i denotes the i -th natural basis vector.
Revised: Here e i denotes the i -th natural basis vector.

Original: We see that G is sub-case of G α,β,µ
Revised: We see that G is sub-case of G α,β,µ

Original: and we prove the following extension of Theorem 2.3.
Revised: and we prove the following extension of Theorem 2.3.

Original: Let α max = max i
Revised: Let α max = max i

Original: α i and α min = min
Revised: α i and α min = min

Original: i α i .
Revised: i α i .

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Let κ α = α max /α min and c be a sufﬁciently small universal constant (e.g. c = 10 −sufﬁces).
Revised: Let κ α = α max /α min and c be a sufﬁciently small universal constant (e.g. c = 10 −sufﬁces).

Original: Suppose µ ≤ cα min /β max and λ ≥ 4 max( µβ max , α max ) .
Revised: Suppose µ ≤ cα min /β max and λ ≥ 4 max( µβ max , α max ) .

Original: Then, the function G α,β,µ ( B ) deﬁned as in equation (B.2) satisﬁes that
Revised: Then, the function G α,β,µ ( B ) deﬁned as in equation (B.2) satisﬁes that

Original: A matrix B is a local minimum of G α,β,µ if and only if B can be written as B = DP where P is a permutation matrix and D is a diagonal matrix with D ii ∈ (cid:110) ± (cid:113)
Revised: A matrix B is a local minimum of G α,β,µ if and only if B can be written as B = DP where P is a permutation matrix and D is a diagonal matrix with D ii ∈ (cid:110) ± (cid:113)

Original: 1 1 − µβ i /λ (cid:111) .
Revised: [Equation]

Original: Any saddle point B has strictly negative curvature in the sense that λ min ( ∇ 2 G α,β,µ ( B )) ≤ − τ 0 where τ 

Revised: Any saddle point B has strictly negative curvature in the sense that λ min ( ∇ 2 G α,β,µ ( B )) ≤ − τ 0 where τ

Original: =
Revised: =

Original: c min { µβ min / ( κ α d ) , µβ 2min /β max , λ } 3.
Revised: c min { µβ min / ( κ α d ) , µβ 2min /β max , λ } 3.

Original: Suppose B is an approximate local minimum in the sense that B satisﬁes(cid:107)∇ g
Revised: Suppose B is an approximate local minimum in the sense that B satisﬁes

Original: [Equation]
Revised: [Equation]

Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: ≤ ε and λ min ( ∇ 2 g ( B )) ≥ − τ

Original: Then B can be written as B = DP + E where P is a permutation matrix, D is a diagonal matrix with the entries satisfying
Revised: Then B can be written as B = DP + E where P is a permutation matrix, D is a diagonal matrix with the entries satisfying

Original: [Equation]
Revised: [Equation]

Original: and E is an error matrix satisfying
Revised: and E is an error matrix satisfying

Original: [Equation]
Revised: [Equation]

Original: As a direct consequence, B is O d ( ε ) -close to a global minimum in Euclidean distance, where O d ( · ) hides polynomial dependency on d and other parameters.
Revised: As a direct consequence, B is O d ( ε ) -close to a global minimum in Euclidean distance, where O d ( · ) hides polynomial dependency on d and other parameters.

Original: Here we recall that | E | ∞ denotes the largest entries in the matrix E .
Revised: Here we recall that | E | ∞ denotes the largest entries in the matrix E .

Original: Theorem 2.3 follows straightforwardly from Theorem B.1 by setting α = 2 √ 6 | ˆ σ 4 | a (cid:63) and β = | ˆ σ 4 | a (cid:63) / √ 6 .
Revised: Theorem 2.3 follows straightforwardly from Theorem B.1 by setting α = 2 √ 6 | ˆ σ 4 | a (cid:63) and β = | ˆ σ 4 | a (cid:63) / √ 6 .

Original: In the rest of the section we prove Theorem B.1.
Revised: In the rest of the section we prove Theorem B.1.

Original: Note that our variable B is a matrix of dimension d × d
Revised: Note that our variable B is a matrix of dimension d × d

Original: and we use b i to denote the rows of B , that is,B =  
Revised: and we use b i to denote the rows of B , that is,

Original: [Equation]
Revised: [Equation]

Original: b
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 

Original:  .
Revised:   .

Original: Naturally, towards analyzing the properties of a local minimum B , the ﬁrst step is that we pick a row b s of B and treat only b s as variables and others rows as ﬁxed.
Revised: Naturally, towards analyzing the properties of a local minimum B , the ﬁrst step is that we pick a row b s of B and treat only b s as variables and others rows as ﬁxed.

Original: We will show that local optimality of b s will imply that b s is equal to one of the basis vector e j up to some scaling factor.
Revised: We will show that local optimality of b s will imply that b s is equal to one of the basis vector e j up to some scaling factor.

Original: This step is done in Section B.1.
Revised: This step is done in Section B.1.

Original: Then in Section B.2 we show that the local optimality of all the variables in B implies that each of the rows of B corresponds to different basis vector, which implies that B is a permutation matrix (up to scaling of the rows).
Revised: Then in Section B.2 we show that the local optimality of all the variables in B implies that each of the rows of B corresponds to different basis vector, which implies that B is a permutation matrix (up to scaling of the rows).

Original: B. S TEP 1: A NALYSIS OF L OCAL O PTIMALITY OF A S INGLE R OW
Revised: B. S TEP 1: A NALYSIS OF L OCAL O PTIMALITY OF A S INGLE R OW

Original: Suppose we ﬁx b 1 , · · · , b s − 1 , b s +1 , · · · , b d , and optimize only over b s , we obtain the objective h of the following form:
Revised: Suppose we ﬁx b 1 , · · · , b s − 1 , b s +1 , · · · , b d , and optimize only over b s , we obtain the objective h of the following form:

Original: [Equation]
Revised: [Equation]

Original: We can see that setting α i = a (cid:63) i (cid:80) k (cid:54) =
Revised: We can see that setting α i = a (cid:63)i (cid:80) k (cid:54) =

Original: s ( b (cid:62) k
Revised: s ( b (cid:62) k

Original: e i ) 2 , β
Revised: e i ) 2 , β

Original: i = a (cid:63) i , and x = b s gives us the original objective
Revised: i = a (cid:63)i , and x = b s gives us the original objective

Original: G ( B ) .
Revised: G ( B ) .

Original: In this subsection, we will work with h ( · ) and analyze the properties of the local minima of h ( · ) .
Revised: In this subsection, we will work with h ( · ) and analyze the properties of the local minima of h ( · ) .

Original: The following lemma shows that a local minimum x of the objective h ( · ) must be a scaling of a basis vector.
Revised: The following lemma shows that a local minimum x of the objective h ( · ) must be a scaling of a basis vector.

Original: For a vector x , let | x | 2nd denotes the second largest absolute values of the entries for x .
Revised: For a vector x , let | x | 2nd denotes the second largest absolute values of the entries for x .

Original: We note that |·| 2nd is not a norm.
Revised: We note that |·| 2nd is not a norm.

Original: The lemma deals generally an approximate local minimum, though we suggest casual readers simply think of ε, τ = 0 in the lemma.
Revised: The lemma deals generally an approximate local minimum, though we suggest casual readers simply think of ε, τ = 0 in the lemma.

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let h ( · ) be deﬁned in equation (B.3) with non-negative vectors α and β in R d .
Revised: Let h ( · ) be deﬁned in equation (B.3) with non-negative vectors α and β in R d .

Original: Suppose parameters ε, τ ≥ 0 satisfy that ε ≤ (cid:112) τ 3 /β min .
Revised: Suppose parameters ε, τ ≥ 0 satisfy that ε ≤ (cid:112) τ 3 /β min .

Original: If some point x satisﬁes (cid:107)∇ h ( x ) (cid:107) ≤ ε and λ min ( ∇ 2 h ( x )) ≥ − τ , then we have [Equation]
Revised: If some point x satisﬁes (cid:107)∇ h ( x ) (cid:107) ≤ ε and λ min ( ∇ 2 h ( x )) ≥ − τ , then we have | x | 2nd ≤ (cid:114) τ β min .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we can take ε = (cid:112) τ 3 /β min which means τ = ε 2 / 3 β 1 / 3 min .
Revised: Without loss of generality, we can take ε = (cid:112) τ 3 /β min which means τ = ε 2 / 3 β 1 / 3 min .

Original: The gradient and Hessian of function h ( · ) are
Revised: The gradient and Hessian of function h ( · ) are

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where γ (cid:44) 4 λ ( (cid:107) x (cid:107) 2 − 1) .
Revised: where γ (cid:44) 4 λ ( (cid:107) x (cid:107) 2 − 1) .

Original: Let S
Revised: Let S

Original: = { i : | x i | ≥ δ } be the indices of the coordinates that are signiﬁcantly away from zero, where δ = (cid:16) ε β min (cid:17) 1 / 3 .
Revised: = { i : | x i | ≥ δ } be the indices of the coordinates that are signiﬁcantly away from zero, where δ = (cid:16) ε β min (cid:17) 1 / 3 .

Original: Since (cid:107)∇ h ( x ) (cid:107) ≤ ε , we have that |∇ h ( x ) i | ≤ ε for every i ∈ [ d ] , which implies that
Revised: Since (cid:107)∇ h ( x ) (cid:107) ≤ ε , we have that |∇ h ( x ) i | ≤ ε for every i ∈ [ d ] , which implies that

Original: [Equation]
Revised: [Equation]

Original: which further implies that
Revised: which further implies that

Original: [Equation]
Revised: [Equation]

Original: If | S | = 1 , then we are done because | x | 2nd ≤ δ .
Revised: If | S | = 1 , then we are done because | x | 2nd ≤ δ .

Original: Next we prove that | S | ≥ 2 .
Revised: Next we prove that | S | ≥ 2 .

Original: For the sake of contradiction, we assume that | S | ≥ 2 .
Revised: For the sake of contradiction, we assume that | S | ≥ 2 .

Original: Moreover, WLOG, we assume that | x | 1 ≥ | x | 2 are the two largest entries of | x | in absolute values.
Revised: Moreover, WLOG, we assume that | x | 1 ≥ | x | 2 are the two largest entries of | x | in absolute values.

Original: We take v ∈ R d such that v 1 = − x 2 / (cid:112) x 2 1 + x 2 2 , and v 2 = x 1 / (cid:112) x 2 1 + x 2 2 , and v j = 0 for j ≥ 2 .
Revised: We take v ∈ R d such that v 1 = − x 2 / (cid:112) x 21 + x 22 , and v 2 = x 1 / (cid:112) x 21 + x 22 , and v j = 0 for j ≥ 2 .

Original: Then we have that v (cid:62) x = 0 and (cid:107) v (cid:107) = 1 .
Revised: Then we have that v (cid:62) x = 0 and (cid:107) v (cid:107) = 1 .

Original: We evaluate the quadratic form and have that
Revised: We evaluate the quadratic form and have that

Original: [Equation]
Revised: [Equation]

Revised: x )

Revised: [Equation]

Revised: =

Revised: [Equation]

Original:  (cid:12)
Revised: ( β (cid:12)

Original: [Equation]
Revised: [Equation]

Original:  v (since v (cid:62) x = 0 )
Revised: (cid:12) 2 ) v (since v (cid:62) x = 0 )

Original: [Equation]
Revised: [Equation]

Original: (by equation (B.6) and (cid:107) v (cid:107) = 1 )
Revised: (by equation (B.6) and (cid:107) v (cid:107) = 1 )

Original: [Equation]
Revised: [Equation]

Original: Recall that δ = (cid:16) ε β min (cid:17) 1 / 3 .
Revised: Recall that δ = (cid:16) ε β min (cid:17) 1 / 3 .

Original: Then we conclude that
Revised: Then we conclude that

Original: [Equation]
Revised: [Equation]

Original: This contradicts with the assumption that λ min ( ∇ 2 h ( x )) ≥ − β 1 / 3 min ε 2 / 3 = τ and that (cid:107) v (cid:107) = 1 .
Revised: This contradicts with the assumption that λ min ( ∇ 2 h ( x )) ≥ − β 1 / 3 min ε 2 / 3 = τ and that (cid:107) v (cid:107) = 1 .

Original: Therefore we have | S | = 1 and
Revised: Therefore we have | S | = 1 and

Original: [Equation]
Revised: [Equation]

Original: using ε ≤ (cid:112) τ 3 /β min )

Original: For future reference, we can also show that for a sufﬁciently strong regularization term (sufﬁciently large λ ), the norm of a local minimum x should be bounded from below and above by 1 / 2 and 2 .
Revised: For future reference, we can also show that for a sufﬁciently strong regularization term (sufﬁciently large λ ), the norm of a local minimum x should be bounded from below and above by 1 / 2 and 2 .

Original: This are rather coarse bounds that sufﬁce for our purpose in this subsection.
Revised: This are rather coarse bounds that sufﬁce for our purpose in this subsection.

Original: In Section B.2 we will show that all the rows of a local minimum B of G have norm close to 1.
Revised: In Section B.2 we will show that all the rows of a local minimum B of G have norm close to 1.

Original: Lemma B.3.
Revised: Lemma B.3.

Original: In the setting of Lemma B.2, Suppose in
Revised: In the setting of Lemma B.2, 1.

Original:  addition that λ ≥ 4 max( β max , τ ) and ε ≤ 
Revised: Suppose in addition that λ ≥ 4 max( β max , τ ) and ε ≤ 0 .

Original: 1 β min d − 3 / 2 , then (cid:107) x (cid:107) 2 ≤ 2 .
Revised: 1 β min d − 3 / 2 , then [Equation]

Original: Let i (cid:63) = arg max i | x 
Revised: Let i (cid:63) = arg max i | x i | .

Original: i

Original: | In addition to the previous conditions in bullet 1, assume that 
Revised: In addition to the previous conditions in bullet 1, assume that λ ≥

Original: λ ≥ 4 α i (cid:63) .
Revised: [Equation] i (cid:63) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: We remark that we have to state the conditions for the upperbounds and lowerbounds separately since they will be used with these different conditions.
Revised: We remark that we have to state the conditions for the upperbounds and lowerbounds separately since they will be used with these different conditions.

Original: Proof.
Revised: Proof.

Original: Let S
Revised: Let S

Original: = { i : | x i | ≥ δ } be the indices of the coordinates that are signiﬁcantly away from zero, where δ = (cid:16) ε β min (cid:17) 1 / 3 .
Revised: = { i : | x i | ≥ δ } be the indices of the coordinates that are signiﬁcantly away from zero, where δ = (cid:16) ε β min (cid:17) 1 / 3 .

Original: We ﬁrst show that (cid:107) x (cid:107) 2 ≤ 2 .
Revised: We ﬁrst show that (cid:107) x (cid:107) 2 ≤ 2 .

Original: We divide into two cases: 1 S is empty.
Revised: We divide into two cases: 1 S is empty.

Original: Since ε ≤ 0 .
Revised: Since ε ≤ 0 .

Original: 1 β min d − 3 / 2 , then δ ≤ √ 2 √ d .
Revised: 1 β min d − 3 / 2 , then δ ≤ √ 2 √ d .

Original: We conclude that (cid:107) x (cid:107) 2 ≤ 2 .
Revised: We conclude that (cid:107) x (cid:107) 2 ≤ 2 .

Original: 2 S is non-empty.
Revised: 2 S is non- empty.

Original: For i ∈ S , recall equation (B.6) which implies that
Revised: For i ∈ S , recall equation (B.6) which implies that

Original: [Equation]
Revised: [Equation]

Original: Since λ ≥ 4 β max , so λ ≥ β 1 / 3 min ε 2 / 3 ≥ 3 ε 4 δ , and thus from the display above we have that (cid:107) x (cid:107) 2 ≤ 2 .
Revised: Since λ ≥ 4 β max , so λ ≥ β 1 / 3 min ε 2 / 3 ≥ 3 ε 4 δ , and thus from the display above we have that [Equation] x [Equation]

Original:  Next we show that (cid:107) x (cid:107) 2 ≥ 12 .
Revised: 2 ≤ 2 . Next we show that (cid:107) x (cid:107) 2 ≥ 12 .

Original: Again we divide into two cases: 1.
Revised: Again we divide into two cases: 1.

Original: S is empty.
Revised: S is empty.

Original: For the sake of contradiction, assume that (cid:107) x (cid:107) 2 ≤ 12 , then γ ≤ − 2 λ .
Revised: For the sake of contradiction, assume that (cid:107) x (cid:107) 2 ≤ 12 , then γ ≤ − 2 λ .

Original: We show that there is sufﬁcient negative curvature.
Revised: We show that there is sufﬁcient negative curvature.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: Choose index j (cid:63)
Revised: Choose index j (cid:63)

Original: so that α j (cid:63) =
Revised: so that α j (cid:63) =

Original: α min , then
Revised: α min , then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: This contradicts with the fact that λ min ( ∇ 2 h ( x )) ≥ − τ .
Revised: This contradicts with the fact that λ min ( ∇ 2 h ( x )) ≥ − τ .

Original: Thus when S is empty, (cid:107) x (cid:107) 2 ≥ 12 .
Revised: Thus when S is empty, (cid:107) x (cid:107) 2 ≥ 12 .

Original: S is non-empty.
Revised: S is non-empty.

Original: Recall that i (cid:63) = arg max i | x i | , and by deﬁnition i (cid:63) ∈ S .
Revised: Recall that i (cid:63) = arg max i | x i | , and by deﬁnition i (cid:63) ∈ S .

Original: Using Equation (B.6)
Revised: Using Equation (B.6)

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: Since λ ≥ 4 α i (cid:63) , and λ ≥ β 1 / 3 min ε 2 / 3 ≥ εδ , we conclude that (cid:107) x (cid:107) 2 ≥ 1 / 2 .
Revised: Since λ ≥ 4 α i (cid:63) , and λ ≥ β 1 / 3 min ε 2 / 3 ≥ εδ , we conclude that (cid:107) x (cid:107) 2 ≥ 1 / 2 .

Original: We have shown that a local minimum x of h should be a scaling of the basis vector
Revised: We have shown that a local minimum x of h should be a scaling of the basis vector

Original: e i (cid:63) .
Revised: e i (cid:63) .

Original: The following lemma strengthens the result by demonstrating that not all basis vector can be a local minimum — the corresponding coefﬁcient α i (cid:63) has to be reasonably small for e i (cid:63) being a local minimum.
Revised: The following lemma strengthens the result by demonstrating that not all basis vector can be a local minimum — the corresponding coefﬁcient α i (cid:63) has to be reasonably small for e i (cid:63) being a local minimum.

Original: The key intuition here is that if α i (cid:63) is very large compared to other entries of α , then if we move locally the mass of e i (cid:63) from entry i (cid:63) to some other index j , the objective function will be likely to decrease because α j x 2 j is likely to be smaller than α i (cid:63) x 2 i (cid:63) .
Revised: The key intuition here is that if α i (cid:63) is very large compared to other entries of α , then if we move locally the mass of e i (cid:63) from entry i (cid:63) to some other index j , the objective function will be likely to decrease because α j x 2 j is likely to be smaller than α i (cid:63) x 2 i (cid:63) .

Original: (Indeed, we will show that such movement will cause a second-order decrease of the objective function in the proof.)
Revised: (Indeed, we will show that such movement will cause a second-order decrease of the objective function in the proof.)

Original: Lemma B.4.
Revised: Lemma B.4.

Original: In the setting of Lemma B.2, let i (cid:63) =
Revised: In the setting of Lemma B.2, let i (cid:63) =

Original: arg max i | x i | .
Revised: arg max i | x i | .

Original: If (cid:107)∇ h ( x ) (cid:107) ≤ ε , and λ min ( ∇ 2 h ( x )) >
Revised: If (cid:107)∇ h ( x ) (cid:107) ≤ ε , and λ min ( ∇ 2 h ( x )) >

Original: − τ for 0 ≤ τ ≤ 0 .
Revised: − τ for 0 ≤ τ ≤ 0 .

Original: 1 β min /d and ε ≤ (cid:112) τ 3 /β min , then α i (cid:63) ≤ α min + 2 ε + 2 τ + 4 β i (cid:63) .
Revised: 1 β min /d and ε ≤ (cid:112) τ 3 /β min , then α i (cid:63) ≤ α min + 2 ε + 2 τ + 4 β i (cid:63) .

Original: Proof.
Revised: Proof.

Original: For the ease of notation, assume WLOG that
Revised: For the ease of notation, assume WLOG that

Original: i (cid:63) = 1 .
Revised: i (cid:63) = 1 .

Original: Let δ
Revised: Let δ

Original: = ( τ/β min ) 1 / 2 .
Revised: = ( τ/β min ) 1 / 2 .

Original: By the assumptions, we have that δ ≤ 1 √ 6 d .
Revised: By the assumptions, we have that δ ≤ 1 √ 6 d .

Original: By Lemma B.2, we have (cid:107) x (cid:107) 2 ≥ 12 , which implies that
Revised: By Lemma B.2, we have (cid:107) x (cid:107) 2 ≥ 12 , which implies that

Original: [Equation]
Revised: [Equation]

Original: Deﬁne v = − (cid:16) x k x 1 (cid:17) e 1 + e k .
Revised: Deﬁne v = − (cid:16) x k x 1 (cid:17) e 1 + e k .

Original: Since x 1 is the largest entry of x , we can verify that 1 ≤ (cid:107) v (cid:107) 2 = 1 + x 2 k x 21 ≤ 2 .
Revised: Since x 1 is the largest entry of x , we can verify that 1 ≤ (cid:107) v (cid:107) 2 = 1 + x 2 k x 21 ≤ 2 .

Original: By the assumption, we have that
Revised: By the assumption, we have that

Original: [Equation]
Revised: [Equation]

Original: On the other hand, recall the form of Hessian (equation (B.4)), by straightforward algebraic manipulation, we have that
Revised: On the other hand, recall the form of Hessian (equation (B.4)), by straightforward algebraic manipulation, we have that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Combining equation (B.8) and the equation above gives
Revised: Combining equation (B.8) and the equation above gives

Original: [Equation]
Revised: [Equation]

Original: Since k is arbitrary we complete the proof.
Revised: Since k is arbitrary we complete the proof.

Original: The previous lemma implies that it’s very likely that the local minimum x can be written as x
Revised: The previous lemma implies that it’s very likely that the local minimum x can be written as x

Original: = x i (cid:63) e i (cid:63) and the index i (cid:63) is also likely to be the argmin of α .
Revised: = x i (cid:63) e i (cid:63) and the index i (cid:63) is also likely to be the argmin of α .

Original: The following technical lemma shows that when this indeed happens, then we can strengthen Lemma B.2 in terms of the error bound’s dependency on ε and τ .
Revised: The following technical lemma shows that when this indeed happens, then we can strengthen Lemma B.2 in terms of the error bound’s dependency on ε and τ .

Original: In Lemma B.2, we have that | x | 2nd is bounded by a function of τ .
Revised: In Lemma B.2, we have that | x | 2nd is bounded by a function of τ .

Original: Here we strengthen the bound to be a function that only depends on ε .
Revised: Here we strengthen the bound to be a function that only depends on ε .

Original: Thus as long as τ be small enough so that we can apply Lemma B.2 and Lemma B.4 to meet the condition of the lemma below, then we get an error bound that goes to zero as ε goes to zero.
Revised: Thus as long as τ be small enough so that we can apply Lemma B.2 and Lemma B.4 to meet the condition of the lemma below, then we get an error bound that goes to zero as ε goes to zero.

Original: This translates to the error bound in bullet 3 of Theorem B.1 where the bound on E only depends on ε .
Revised: This translates to the error bound in bullet 3 of Theorem B.1 where the bound on E only depends on ε .

Original: For casual readers we suggest to skip this Lemma since its precise functionality will only be clearer in the proof of Theorem B.1.
Revised: For casual readers we suggest to skip this Lemma since its precise functionality will only be clearer in the proof of Theorem B.1.

Original: Lemma B.5.
Revised: Lemma B.5.

Original: In the setting of Lemma B.2, in addition we assume that i = argmin k | α k | and that x can be written as x
Revised: In the setting of Lemma B.2, in addition we assume that i = argmin k | α k | and that x can be written as x

Original: = x i e i + x − i satisfying
Revised: = x i e i + x − i satisfying

Original: [Equation]
Revised: [Equation]

Original: Then, we can strengthen the bound to
Revised: Then, we can strengthen the bound to

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: WLOG, let i
Revised: WLOG, let i

Original: =
Revised: =

Original: Let x j be the second largest entry of x in absolute value.
Revised: Let x j be the second largest entry of x in absolute value.

Original: Deﬁne v 1 = 4 β 1 x 21 − 2 α 1 − γ , and similarly v j = 

Revised: Deﬁne v 1 = 4 β 1 x 21 − 2 α 1 − γ , and similarly v j =

Original: β
Revised: β

Original: j x 2 j − 2 α j − γ .
Revised: j x 2 j − 2 α j − γ .

Original: Since (cid:107)∇ h ( x ) (cid:107) ≤ ε , by equation (B.6), we have that | v 1 | ≤ ε | x 1 | and | v 2 | = ε | x j | .
Revised: Since (cid:107)∇ h ( x ) (cid:107) ≤ ε , by equation (B.6), we have that | v 1 | ≤ ε | x 1 | and | v 2 | = ε | x j | .

Original: Subtracting 4 β 1 x 21 = 2 α 1 + γ + v 1 and 4 β j x 2 j = 

Revised: Subtracting 4 β 1 x 21 = 2 α 1 + γ + v 1 and 4 β j x 2 j =

Original: α j + γ + v j , we obtain,
Revised: α j + γ + v j , we obtain,

Original: [Equation]
Revised: [Equation]

Original: (since α j − α 1 ≥ 0 )
Revised: (since α j − α 1 ≥ 0 )

Original: Since (cid:107) x (cid:107) 2 ≥ 12 , then x 21 ≥ 12 − dδ 2 ≥ 13 .
Revised: Since (cid:107) x (cid:107) 2 ≥ 12 , then x 21 ≥ 12 − dδ 2 ≥ 13 .

Original: Since | x j | ≤ δ , 4 β j x 2 j ≤ 4 β max δ
Revised: Since | x j | ≤ δ , 4 β j x 2 j ≤ 4 β max δ

Original: Combining the above two displays,
Revised: Combining the above two displays,

Original: [Equation]
Revised: [Equation]

Original: (using x 21 ≥ 13 and δ ≤ (cid:112) β min / (6 β max ) ) ≥
Revised: (using x 21 ≥ 13 and δ ≤ (cid:112) β min / (6 β max ) ) ≥

Original: [Equation]
Revised: [Equation]

Original: min (B.9)
Revised: min (B.9)

Original: Since | v 1 | ≤ ε | x 1 | and | v 2 | = ε | x 2 | , 2 ε | x j | [Equation] 3 β min , (B.10) and re-arranging gives | x j | ≤ 3 ε β min .
Revised: Since | v 1 | ≤ ε | x 1 | and | v 2 | = ε | x 2 | , 2 ε | x j | ≥ 3 β min , (B.10) and re-arranging gives | x j | ≤ 3 ε β min .

Original: B. L OCAL O PTIMALITY OF A LL THE V ARIABLES
Revised: B. L OCAL O PTIMALITY OF A LL THE V ARIABLES

Original: In this section we prove Theorem B.1.
Revised: In this section we prove Theorem B.1.

Original: Results in Subsection B.1 have established that if B is a local minimum, then each row b s of B has to be a scaling of a basis vector.
Revised: Results in Subsection B.1 have established that if B is a local minimum, then each row b s of B has to be a scaling of a basis vector.

Original: In this section we show that these basis vectors need to be distinct from each other.
Revised: In this section we show that these basis vectors need to be distinct from each other.

Original: The following proposition summaries such a claim (with a weak error analysis).
Revised: The following proposition summaries such a claim (with a weak error analysis).

Original: Proposition B.6.
Revised: Proposition B.6.

Original: In the setting of Theorem B.1, suppose B satisﬁes (cid:107)∇ g ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 g ( B )) ≥ − τ for parameters τ, ε satisfying 0 ≤ τ ≤ c min { µβ min / ( κ α d ) , λ } and ε ≤ c min { α min , (cid:112) τ 3 /β min } .
Revised: In the setting of Theorem B.1, suppose B satisﬁes (cid:107)∇ g ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 g ( B )) ≥ − τ for parameters τ, ε satisfying 0 ≤ τ ≤ c min { µβ min / ( κ α d ) , λ } and ε ≤ c min { α min , (cid:112) τ 3 /β min } .

Original: Then, the matrix B can be written as
Revised: Then, the matrix B can be written as

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: where D is diagonal such that ∀ i,
Revised: where D is diagonal such that ∀ i,

Original: | D ii | ∈ [1 / 4 , 2] , and P is a permutation matrix, and | E | ∞ ≤ δ with δ = (cid:16) τ µβ min (cid:17) 1 / 2 .
Revised: | D ii | ∈ [1 / 4 , 2] , and P is a permutation matrix, and | E | ∞ ≤ δ with δ = (cid:16) τ µβ min (cid:17) 1 / 2 .

Original: As alluded before, in the proof we will ﬁrst apply the results in Section B.1 to show that when B is a local minimum, each row b s has a unique large entry.
Revised: As alluded before, in the proof we will ﬁrst apply the results in Section B.1 to show that when B is a local minimum, each row b s has a unique large entry.

Original: Then we will show that the largest entries of each row sit on different columns.
Revised: Then we will show that the largest entries of each row sit on different columns.

Original: The key intuition behind the proof is that if two rows, say row s, t , have their large entries on the same column, then it means that there exists a column— say column k — that doesn’t contain largest entry of any row.
Revised: The key intuition behind the proof is that if two rows, say row s, t , have their large entries on the same column, then it means that there exists a column— say column k — that doesn’t contain largest entry of any row.

Original: Then either row s or t will violate Lemma B.4.
Revised: Then either row s or t will violate Lemma B.4.

Original: Or in other words, either row s or t can move their mass into the column k to decrease the function value.
Revised: Or in other words, either row s or t can move their mass into the column k to decrease the function value.

Original: This contradicts the assumption that B is a local minimum.
Revised: This contradicts the assumption that B is a local minimum.

Original: Proof.
Revised: Proof.

Original: As pointed in the paragraph below equation (B.3), when we restrict our attention to a particular row of B and ﬁx the rest of the rows the function G α,β,µ reduces to the function h ( · ) in equation (B.3) so that we can apply lemmas in Section B.1.
Revised: As pointed in the paragraph below equation (B.3), when we restrict our attention to a particular row of B and ﬁx the rest of the rows the function G α,β,µ reduces to the function h ( · ) in equation (B.3) so that we can apply lemmas in Section B.1.

Original: Concretely, ﬁx an index s ∈ [ d ] and let x = b s .
Revised: Concretely, ﬁx an index s ∈ [ d ] and let x = b s .

Original: For all i ∈ [ d ] , let ¯ α i
Revised: For all i ∈ [ d ] , let ¯ α i

Original: = α i (cid:80) j (cid:54) =
Revised: = α i (cid:80) j (cid:54) =

Original: s ( b (cid:62) j e i ) 2 , and ¯ β i
Revised: s ( b (cid:62) j e i ) 2 , and ¯ β i

Original: = µβ i .
Revised: = µβ i .

Original: Then we have that
Revised: Then we have that

Original: [Equation]
Revised: [Equation]

Original: We view the function above as h ( x ) .
Revised: We view the function above as h ( x ) .

Original: Now we apply Lemma B.2 (by replacing α, β in Lemma B. by ¯ α, ¯ β ).
Revised: Now we apply Lemma B.2 (by replacing α, β in Lemma B. by ¯ α, ¯ β ).

Original: The assumption that λ min ( ∇ 2 g α,β,µ ( B )) ≥ − τ implies that λ min ( ∇ 2 h ( x )) ≥ − τ since ∇ 2 h ( x ) is a submatrix of ∇ 2 g ( B ) .
Revised: The assumption that λ min ( ∇ 2 g α,β,µ ( B )) ≥ − τ implies that λ min ( ∇ 2 h ( x )) ≥ − τ since ∇ 2 h ( x ) is a submatrix of ∇ 2 g ( B ) .

Original: Moreover, (cid:107)∇ h ( x ) (cid:107) ≤ (cid:107)∇ G ( B ) (cid:107) ≤ ε ≤ (cid:112) τ 3 / ( µβ min )
Revised: Moreover, (cid:107)∇ h ( x ) (cid:107) ≤ (cid:107)∇ G ( B ) (cid:107) ≤ ε ≤ (cid:112) τ 3 / ( µβ min )

Original: Hence by Lemma B.2, we have that the second largest entry of | b s | satisﬁes
Revised: Hence by Lemma B.2, we have that the second largest entry of | b s | satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where δ (cid:44) (cid:16)
Revised: where δ (cid:44) (cid:16)

Original: τ µβ min (cid:17) 1 / 2 for the ease of notation.
Revised: τ µβ min (cid:17) 1 / 2 for the ease of notation.

Original: We can check that δ ≤ {+
+}
Revised: We can check that δ ≤ {+
+}

Original: [Equation]
Revised: [Equation]

Original:  by the assumption.
Revised: κ α d by the assumption.

Original: Therefore, we have essentially shown that each row of B has only one single large entry, since the second largest entry is at most δ .
Revised: Therefore, we have essentially shown that each row of B has only one single large entry, since the second largest entry is at most δ .

Original: Next we show that each row of B has largest entries on distinct columns.
Revised: Next we show that each row of B has largest entries on distinct columns.

Original: For each row j ∈ [ d ] , let i j = arg max i | e (cid:62)
Revised: For each row j ∈ [ d ] , let i j = arg max i | e (cid:62)

Original: i
Revised: i

Original: b j | be the index of the largest entry of b j .
Revised: b j | be the index of the largest entry of b j .

Original: We will show that i 1 , . .
Revised: We will show that i 1 , . .

Original: , i d are distinct.
Revised: , i d are distinct.

Original: For the sake of contradiction, suppose they are not distinct, that is, there are two distinct rows s, t that have the same largest entries on column l , that is, we assume that i s = i t
Revised: For the sake of contradiction, suppose they are not distinct, that is, there are two distinct rows s, t that have the same largest entries on column l , that is, we assume that i s = i t

Original: = l .
Revised: = l .

Original: This implies that { i 1 , . . .
Revised: This implies that { i 1 , . . .

Original: , i d } (cid:54) =
Revised: , i d } (cid:54) =

Original: [ d ] and let k ∈ [ d ] be the index such that k / ∈ { i 1 , . .
Revised: [ d ] and let k ∈ [ d ] be the index such that k / ∈ { i 1 , . .

Original: , i d } .
Revised: , i d } .

Original: We note that by the assumption δ =(cid:16) τ µβ min (cid:17) 1 / 2 ≤
Revised: We note that by the assumption δ =

Original: [Equation]
Revised: [Equation]

Original: d ≤ 14 √ d .

Original: We ﬁrst bound from above ¯ α k¯
Revised: We ﬁrst bound from above ¯ α k

Original: [Equation]
Revised: [Equation]

Original: Assume in addition without loss of generality that | b (cid:62) s e l | ≤ | b (cid:62) t e l | .
Revised: Assume in addition without loss of generality that | b (cid:62) s e l | ≤ | b (cid:62) t e l | .

Original: Letz l
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: b [Equation] B.13) be the sum of squares of the entries on the column l without entry b (cid:62) j e l , and that ¯ α l = α l z l .
Revised: be the sum of squares of the entries on the column l without entry b (cid:62) j e l , and that ¯ α l = α l z l .

Original: We ﬁrst prove that z l ≥ 1 / 3 .
Revised: We ﬁrst prove that z l ≥ 1 / 3 .

Original: For the sake of contradiction, assume z l < 1 / 3 .
Revised: For the sake of contradiction, assume z l < 1 / 3 .

Original: Then we have that ¯ α l = α l z ≤ 13 α l .
Revised: Then we have that ¯ α l = α l z ≤ 13 α l .

Original: This implies that λ ≥ 4 max { ¯ α l , τ } , and since l is the index of the largest column of b s we can invoke Lemma B. and conclude that (cid:107) b s (cid:107) 2 ≥ 1 / 2 .
Revised: This implies that λ ≥ 4 max { ¯ α l , τ } , and since l is the index of the largest column of b s we can invoke Lemma B. and conclude that (cid:107) b s (cid:107) 2 ≥ 1 / 2 .

Original: This further implies that
Revised: This further implies that

Original: [Equation]
Revised: [Equation]

Original: Since we have assumed that | b (cid:62) s e l | ≤ | b (cid:62) t e l | .
Revised: Since we have assumed that | b (cid:62) s e l | ≤ | b (cid:62) t e l | .

Original: Then we obtain that
Revised: Then we obtain that

Original: [Equation]
Revised: [Equation]

Original: which contradicts the assumption.
Revised: which contradicts the assumption.

Original: Therefore, we conclude that z l ≥ 1 / 3 .
Revised: Therefore, we conclude that z l ≥ 1 / 3 .

Original: Then we are ready to bound ¯ α l from below:
Revised: Then we are ready to bound ¯ α l from below:

Original: [Equation]
Revised: [Equation]

Original: The display above and Equation (B.13) implies that
Revised: The display above and Equation (B.13) implies that

Original: [Equation]
Revised: [Equation]

Original: Note that l is the largest entry in absolute value in the vector b s .
Revised: Note that l is the largest entry in absolute value in the vector b s .

Original: We will apply Lemma B.4.
Revised: We will apply Lemma B.4.

Original: We ﬁx every row of B except b s and consider the objective as a function of b s only.
Revised: We ﬁx every row of B except b s and consider the objective as a function of b s only.

Original: Again let ¯ α
Revised: Again let ¯ α

Original: i
Revised: i

Original: = α i (cid:80) j (cid:54) =
Revised: = α i (cid:80) j (cid:54) =

Original: s ( b (cid:62) j e i ) 2 , and ¯ β i
Revised: s ( b (cid:62) j e i ) 2 , and ¯ β i

Original: = µβ i
Revised: = µβ i

Original: and we have the equation (B.11).
Revised: and we have the equation (B.11).

Original: (Note that now ¯ α depends on the choice of s which we ﬁxed.)
Revised: (Note that now ¯ α depends on the choice of s which we ﬁxed.)

Original: Lemma B.4 gives us that
Revised: Lemma B.4 gives us that

Original: [Equation]
Revised: [Equation]

Original: Since ε ≤ 150 α min , τ ≤ 150 α min and ¯ β l = µβ l ≤ 150 α min , we obtain that
Revised: Since ε ≤ 150 α min , τ ≤ 150 α min and ¯ β l = µβ l ≤ 150 α min , we obtain that

Original: [Equation]
Revised: [Equation]

Original: B.15) which contradicts equation (B.14).
Revised: (B [Equation] 15) which contradicts equation (B.14).

Original: Thus we have established that i 1 , . .
Revised: Thus we have established that i 1 , . .

Original: , i d are distinct.
Revised: , i d are distinct.

Original: Finally, let Q be the matrix that only contain the largest entries (in absolute value) of each columns of B .
Revised: Finally, let Q be the matrix that only contain the largest entries (in absolute value) of each columns of B .

Original: Since i 1 , .
Revised: Since i 1 , .

Original: , i d are distinct, we have that Q contains exactly one entry per row and per column.
Revised: , i d are distinct, we have that Q contains exactly one entry per row and per column.

Original: Therefore Q can be written as DP where P is a permutation matrix and D is a diagonal matrix.
Revised: Therefore Q can be written as DP where P is a permutation matrix and D is a diagonal matrix.

Original: Moreover, we have that (cid:107) b s (cid:107) 2 ∞ ≥ (cid:107)
Revised: Moreover, we have that (cid:107) b s (cid:107) 2 ∞ ≥ (cid:107)

Original: b s (cid:107) 2 − d | b s | 2 2nd ≥ 1 / 4 and (cid:107)
Revised: b s (cid:107) 2 − d | b s | 2 2nd ≥ 1 / 4 and (cid:107)

Original: b s (cid:107) 2 ≤ 2 .
Revised: b s (cid:107) 2 ≤ 2 .

Original: Therefore, the largest entry of each row has absolute value between 1 / 4 and 2 .
Revised: Therefore, the largest entry of each row has absolute value between 1 / 4 and 2 .

Original: Therefore | D | ii ∈ [1 / 4 , 2] .
Revised: Therefore | D | ii ∈ [1 / 4 , 2] .

Original: Let
Revised: Let

Original: E = B − PD .
Revised: E = B − PD .

Original: Then we have that | E | ∞ ≤ max s | b s | 2nd ≤ δ ,which completes the proof.
Revised: Then we have that | E | ∞ ≤ max s | b s | 2nd ≤ δ ,which completes the proof.

Original: Applying Lemma B.5, we can further strengthen Proposition B.6 with better error bounds and better control of the largest entries of each column.
Revised: Applying Lemma B.5, we can further strengthen Proposition B.6 with better error bounds and better control of the largest entries of each column.

Original: Proposition B.7 (Strengthen of Proposition B.6) .
Revised: Proposition B.7 (Strengthen of Proposition B.6) .

Original: In the setting of Proposition B.6.
Revised: In the setting of Proposition B.6.

Original: Suppose in addition that τ satisﬁes τ ≤ cµβ 2min /β max .
Revised: Suppose in addition that τ satisﬁes τ ≤ cµβ 2min /β max .

Original: Then, the matrix B can be written as
Revised: Then, the matrix B can be written as

Original: B = DP + E , where P is a permutation matrix, D is diagonal such that
Revised: [Equation] where P is a permutation matrix, D is diagonal such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Proposition B.6, we know that | E | ∞ ≤ δ = (cid:16) τ µβ min (cid:17) 1 / 2 .
Revised: By Proposition B.6, we know that | E | ∞ ≤ δ = (cid:16) τ µβ min (cid:17) 1 / 2 .

Original: Now we use Lemma B.5 to strength the error bound.
Revised: Now we use Lemma B.5 to strength the error bound.

Original: As we have done in the proof of Proposition B.6, we again ﬁx an arbitrary s ∈
Revised: As we have done in the proof of Proposition B.6, we again ﬁx an arbitrary s ∈

Original: [ d ] and all the rows except b s and view G α,β,µ as a function of b s .
Revised: [ d ] and all the rows except b s and view G α,β,µ as a function of b s .

Original: For all i ∈ [ d ] , let ¯ α i
Revised: For all i ∈ [ d ] , let ¯ α i

Original: = α i (cid:80) j (cid:54) =
Revised: = α i (cid:80) j (cid:54) =

Original: s ( b (cid:62) j e i ) 2 , and ¯ β i
Revised: s ( b (cid:62) j e i ) 2 , and ¯ β i

Original: = µβ i and view G α,β,µ as a function of the form h ( x ) with α, β replaced by ¯ α, ¯ β , namely,
Revised: = µβ i and view G α,β,µ as a function of the form h ( x ) with α, β replaced by ¯ α, ¯ β , namely,

Original: [Equation]
Revised: [Equation]

Original: We will verify the condition of Lemma B.5.
Revised: We will verify the condition of Lemma B.5.

Original: Let i be the index of the largest entry in absolute value of the vector b s .
Revised: Let i be the index of the largest entry in absolute value of the vector b s .

Original: Since we have shown that the largest entry in each row sits on different columns, and the second largest entry is always less than δ , we have that,
Revised: Since we have shown that the largest entry in each row sits on different columns, and the second largest entry is always less than δ , we have that,

Original: [Equation]
Revised: [Equation]

Original: For any k (cid:54) =
Revised: For any k (cid:54) =

Original: i , we know that the column k contains some entry ( k, j k ) which is the largest entry of some row, and we also have that j k (cid:54)
Revised: i , we know that the column k contains some entry ( k, j k ) which is the largest entry of some row, and we also have that j k (cid:54)

Original: =
Revised: =

Original: s since the largest entry of row s is on column i .
Revised: s since the largest entry of row s is on column i .

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: Therefore, ¯ α k ≥ ¯ α i for any k (cid:54) =
Revised: Therefore, ¯ α k ≥ ¯ α i for any k (cid:54) =

Original: i and thus i = argmin k
Revised: i and thus i = argmin k

Original: | ¯ α k | .
Revised: | ¯ α k | .

Original: By the fact that | E | ∞ ≤ δ , we have that (cid:107) x − i (cid:107) ∞ ≤ δ ≤ 0 .
Revised: By the fact that | E | ∞ ≤ δ , we have that (cid:107) x − i (cid:107) ∞ ≤ δ ≤ 0 .

Original: 1 min { 1 / √ d, (cid:112) β min / ( β max ) } .
Revised: 1 min { 1 / √ d, (cid:112) β min / ( β max ) } .

Original: Now we are ready to apply Lemma B.5 and obtain that | b s | 2nd ≤ 3 ε β min .
Revised: Now we are ready to apply Lemma B.5 and obtain that | b s | 2nd ≤ 3 ε β min .

Original: Applying the argument for every row s gives | E | ∞ ≤ 3 ε β min .
Revised: Applying the argument for every row s gives | E | ∞ ≤ 3 ε β min .

Original: Finally, we give the bound for the entires in D .
Revised: Finally, we give the bound for the entires in D .

Original: Let v be a short hand for ∇ h ( b s ) which is equal to the s -th column of ∇ G ( B ) .
Revised: Let v be a short hand for ∇ h ( b s ) which is equal to the s -th column of ∇ G ( B ) .

Original: Since B is an ε -approximate stationary point, then we have that (cid:107) v (cid:107) ≤ ε and by straightforward calculation of the gradient, we have
Revised: Since B is an ε -approximate stationary point, then we have that (cid:107) v (cid:107) ≤ ε and by straightforward calculation of the gradient, we have

Original: [Equation]
Revised: [Equation]

Original: Since x i (cid:54) = 0 , dividing by x i gives,
Revised: Since x i (cid:54) = 0 , dividing by x i gives,

Original: [Equation]
Revised: [Equation]

Original: Rearranging the equation above gives,
Revised: Rearranging the equation above gives,

Original: [Equation]
Revised: [Equation]

Original: To upper bound x 2 i , we note that | v i | < ε , ¯ α i > 0 , and (cid:80) j (cid:54) =
Revised: To upper bound x 2 i , we note that | v i | < ε , ¯ α i > 0 , and (cid:80) j (cid:54) =

Original: i x 2 j > 0 , so
Revised: i x 2 j > 0 , so

Original: [Equation]
Revised: [Equation]

Original: (since λ ≥4 µβ i )
Revised: i ≤ [Equation] since λ ≥

Original:  For the lower bound of x 2 i , we note that | E | ∞ ≤ δ = 3 ε β min implies (cid:80) j (cid:54) =
Revised: [Equation] i ) For the lower bound of x 2 i , we note that | E | ∞ ≤ δ = 3 ε β min implies (cid:80) j (cid:54) =

Original: i x 2 j ≤ dδ 2 .
Revised: i x 2 j ≤ dδ 2 .

Original: Moreover, we have proved that each rows has largest entry at different columns.
Revised: Moreover, we have proved that each rows has largest entry at different columns.

Original: Also note that the largest entry of row b s is on column i .
Revised: Also note that the largest entry of row b s is on column i .

Original: Therefore, we have ¯ α i
Revised: Therefore, we have ¯ α i

Original: = α i (cid:80) j (cid:54) =
Revised: = α i (cid:80) j (cid:54) =

Original: s ( b Tj e i ) 2 ≤ α max dδ 2 .
Revised: s ( b Tj e i ) 2 ≤ α max dδ 2 .

Original: Using these two estimates and δ = 3 ε β min , we have
Revised: Using these two estimates and δ = 3 ε β min , we have

Original: [Equation]
Revised: [Equation]

Original: Finally we are ready to prove Theorem B.1 by applying Proposition B.6.
Revised: Finally we are ready to prove Theorem B.1 by applying Proposition B.6.

Original: Proof of Theorem B.1.
Revised: Proof of Theorem B.1.

Original: By setting ε = 0 , τ = 0 in Proposition B.6, we have that any local minimum
Revised: By setting ε = 0 , τ = 0 in Proposition B.6, we have that any local minimum

Original: B satisﬁes that B = DP where P is a permutation matrix and D is a diagonal and the precise diagonal entries of D .
Revised: B satisﬁes that B = DP where P is a permutation matrix and D is a diagonal and the precise diagonal entries of D .

Original: It can be veriﬁed that all these points have the same function value, so that they are all global minimizers.
Revised: It can be veriﬁed that all these points have the same function value, so that they are all global minimizers.

Original: Towards proving the second bullet, we note that a saddle point B satisﬁes that ∇ G ( B ) = 0 .
Revised: Towards proving the second bullet, we note that a saddle point B satisﬁes that ∇ G ( B ) = 0 .

Original: We will prove that λ min ( ∇ 2 G ( B )) ≤ − τ 0 .
Revised: We will prove that λ min ( ∇ 2 G ( B )) ≤ − τ 0 .

Original: For the sake of contradiction, suppose λ min ( ∇ 2 G ( B )) ≥ − τ 0 .
Revised: For the sake of contradiction, suppose λ min ( ∇ 2 G ( B )) ≥ − τ 0 .

Original: Then setting ε = 0 and τ = τ 0 in Propostion B.7, we have that B = DP and D ii = (cid:110) ± (cid:113)[Equation] i /
Revised: Then setting ε = 0 and τ = τ 0 in Propostion B.7, we have that B = DP and D ii = (cid:110) ± (cid:113)

Original: [Equation]
Revised: [Equation]

Original: (cid:111) , which by bullet 1 implies that B is a local minimum.
Revised: (cid:111) , which by bullet 1 implies that B is a local minimum.

Original: This contradicts the assumption that B is a saddle point.
Revised: This contradicts the assumption that B is a saddle point.

Original: The 3rd bullet is a just a rephrasing of Proposition B.7.
Revised: The 3rd bullet is a just a rephrasing of Proposition B.7.

Original: C H ANDLING N ON -O RTHOGONAL W EIGHTS
Revised: C H ANDLING N ON -O RTHOGONAL W EIGHTS

Original: In this section, we ﬁrst show that when the weight vectors { b (cid:63)i } (cid:48) s are not orthonormal, the local optimum of a slight variant of G ( B ) still allow us to recover B (cid:63) .
Revised: In this section, we ﬁrst show that when the weight vectors { b (cid:63)i } (cid:48) s are not orthonormal, the local optimum of a slight variant of G ( B ) still allow us to recover B (cid:63) .

Original: The main observation is that the set of local minima are preserved (in a certain sense) by linear transformation of the variables.
Revised: The main observation is that the set of local minima are preserved (in a certain sense) by linear transformation of the variables.

Original: We design an objective function F ( B ) that is equivalent to G ( B ) up to a linear transformation.
Revised: We design an objective function F ( B ) that is equivalent to G ( B ) up to a linear transformation.

Original: This allows us to use Theorem 2.3 as a black box to characterize all the local minima of F .
Revised: This allows us to use Theorem 2.3 as a black box to characterize all the local minima of F .

Original: We use λ max ( · ) , λ min ( · ) to denote the largest and smallest eigenvalues of a square matrix.
Revised: We use λ max ( · ) , λ min ( · ) to denote the largest and smallest eigenvalues of a square matrix.

Original: Similarly, σ max ( · ) and σ 
Revised: Similarly, σ max ( · ) and σ min ( · ) are used to denote the largest and smallest singular values.

Original: C. L OCAL M INIMUM AFTER A L INEAR T RANSFORMATION
Revised: C. L OCAL M INIMUM AFTER A L INEAR T RANSFORMATION

Original: Given a function f ( y ) , we say function g ( · ) is a linear transformation of f ( · ) if there is a matrix W
Revised: Given a function f ( y ) , we say function g ( · ) is a linear transformation of f ( · ) if there is a matrix W

Original: such that g ( x ) =
Revised: such that g ( x ) =

Original: f ( Wx ) .
Revised: f ( Wx ) .

Original: If W has full rank, the local minima of f are closely related to the local minima of g .
Revised: If W has full rank, the local minima of f are closely related to the local minima of g .

Original: We recall some standard notation in calculus ﬁrst.
Revised: We recall some standard notation in calculus ﬁrst.

Original: We use ∇ f ( t ) to denote the gradient of f evaluated at t .
Revised: We use ∇ f ( t ) to denote the gradient of f evaluated at t .

Original: For example, ∇ f ( Wx ) is a shorthand for ∂f ( y ) ∂y | y
Revised: For example, ∇ f ( Wx ) is a shorthand for ∂f ( y ) ∂y | y

Original: =
Revised: =

Original: Wx , and similarly ∇ 2 f ( Wx ) is ∂
Revised: Wx , and similarly ∇ 2 f ( Wx ) is ∂

Original: 2 f ( y ) ( ∂y ) 2 | y
Revised: 2 f ( y ) ( ∂y ) 2 | y

Original: =
Revised: =

Original: Wx .
Revised: Wx .

Original: The following theorem then connects the gradients and Hessians of f ( Wx ) and g ( x ) .
Revised: The following theorem then connects the gradients and Hessians of f ( Wx ) and g ( x ) .

Original: Essentially, it shows that the set of local minima and saddle points have a 1-1 mapping between f and g , and the corresponding norms/eigenvalues only differ multiplicatively by quantities related to the spectrum of W .
Revised: Essentially, it shows that the set of local minima and saddle points have a 1-1 mapping between f and g , and the corresponding norms/eigenvalues only differ multiplicatively by quantities related to the spectrum of W .

Original: Theorem C.1.
Revised: Theorem C.1.

Original: Let W ∈ R d
Revised: Let W ∈ R d

Original: × m ( d ≥ m ) be a full rank matrix.
Revised: × m ( d ≥ m ) be a full rank matrix.

Original: Suppose g : R m → R and f : R d → R are twice-differentiable functions such that g ( x ) =
Revised: Suppose g : R m → R and f : R d → R are twice-differentiable functions such that g ( x ) =

Original: f
Revised: f

Original: ( Wx ) for any x ∈ R m .
Revised: ( Wx ) for any x ∈ R m .

Original: Then, for all x ∈ R m , the following three properties hold:
Revised: Then, for all x ∈ R m , the following three properties hold:

Original: σ min ( W ) (cid:107)∇ f ( Wx ) (cid:107) ≤ (cid:107)∇ g ( x ) (cid:107) ≤ σ max ( W ) (cid:107)∇ f ( Wx ) (cid:107) .
Revised: σ min ( W ) (cid:107)∇ f ( Wx ) (cid:107) ≤ (cid:107)∇ g ( x ) (cid:107) ≤ σ max ( W ) (cid:107)∇ f ( Wx ) (cid:107) .

Original: If λ min ( ∇ 2 g ( x )) < 0 , then σ max ( W ) 2 λ min ( ∇ 2 f ( Wx )) ≤ λ min ( ∇ 2 g ( x )) ≤ σ min ( W ) 2 λ min ( ∇ 2{+
+} f ( Wx )) .
Revised: If λ min ( ∇ 2 g ( x )) < 0 , then σ max ( W ) 2 λ min ( ∇ 2 f ( Wx )) ≤ λ min ( ∇ 2 g ( x )) ≤ σ min ( W ) 2 λ min ( ∇ {+
+} f ( Wx )) .

Original: The point x satisﬁes the ﬁrst and second order optimality condition for g iff y
Revised: The point x satisﬁes the ﬁrst and second order optimality condition for g iff y

Original: =
Revised: =

Original: Wx also satisfy the ﬁrst and second order optimality condition for f .
Revised: Wx also satisfy the ﬁrst and second order optimality condition for f .

Original: Proof.
Revised: Proof.

Original: The proof follows from the relationship between the gradients of g and the gradients of f .
Revised: The proof follows from the relationship between the gradients of g and the gradients of f .

Original: By basic calculus, we have
Revised: By basic calculus, we have

Original: [Equation]
Revised: [Equation]

Original: which immediately implies bullet 1.
Revised: which immediately implies bullet 1.

Original: Similarly, we can compute the second order derivative:
Revised: Similarly, we can compute the second order derivative:

Original: [Equation]
Revised: [Equation]

Original: To simplify notation, let A = ∇ 2 f ( Wx ) .
Revised: To simplify notation, let A = ∇ 2 f ( Wx ) .

Original: Let x = arg min (cid:107) x
Revised: Let x = arg min (cid:107) x

Original: (cid:107) =
Revised: (cid:107) =

Original: 1 x (cid:62) W (cid:62) AWx , and
Revised: 1 x (cid:62) W (cid:62) AWx , and

Original: y =
Revised: y =

Original: ( Wx ) / (cid:107) Wx (cid:107) .
Revised: ( Wx ) / (cid:107) Wx (cid:107) .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: On the other hand, let y be the unit vector that minimizes y (cid:62)
Revised: On the other hand, let y be the unit vector that minimizes y (cid:62)

Original: Ay , we know y is in column span of W because f is only deﬁned on the row span, so there must exist a unit vector x such that Wx = λy where λ ≥ σ min ( W ) .
Revised: Ay , we know y is in column span of W because f is only deﬁned on the row span, so there must exist a unit vector x such that Wx = λy where λ ≥ σ min ( W ) .

Original: For this x we have λ min ( W (cid:62) AW ) ≤ x (cid:62) W (cid:62) AWx =
Revised: For this x we have λ min ( W (cid:62) AW ) ≤ x (cid:62) W (cid:62) AWx =

Original: λ 2 λ min ( A ) ≤ σ 2 min ( W ) λ min ( A ) .
Revised: λ 2 λ min ( A ) ≤ σ 2 min ( W ) λ min ( A ) .

Original: This ﬁnishes the proof for 2.
Revised: This ﬁnishes the proof for 2.

Original: Finally, notice that W is full rank, so ∇ g ( x ) =
Revised: Finally, notice that W is full rank, so ∇ g ( x ) =

Original: W (cid:62) ∇ f ( Wx ) = 

Revised: W (cid:62) ∇ f ( Wx ) =

Original: iff ∇ f ( Wx )
Revised: iff ∇ f ( Wx )

Original: = 0 .
Revised: = 0 .

Original: Also, ∇ 2 g ( x ) =
Revised: Also, ∇ 2 g ( x ) =

Original: W (cid:62) [ ∇ 

Revised: W (cid:62) [ ∇

Original: f
Revised: f

Original: ( Wx )]
Revised: ( Wx )]

Original: W (cid:23) 

Revised: W (cid:23)

Original: iff ∇ 2 f ( Wx ) (cid:23) 0 .
Revised: iff ∇ 2 f ( Wx ) (cid:23) 0 .

Original: C. O BJECTIVE FOR N ON -O RTHOGONAL W EIGHTS
Revised: C. O BJECTIVE FOR N ON -O RTHOGONAL W EIGHTS

Original: Now we will design a new objective function that can be linearly transformed to the orthonormal case.
Revised: Now we will design a new objective function that can be linearly transformed to the orthonormal case.

Original: The main idea is to view the rows of B (cid:63) as the new basis that we work on (which is not necessarily orthogonal).
Revised: The main idea is to view the rows of B (cid:63) as the new basis that we work on (which is not necessarily orthogonal).

Original: Note that this is already the case for the ﬁrst two terms of the objective function G ( B ) , we change the objective function as follows: More concretely, we deﬁne[Equation]
Revised: Note that this is already the case for the ﬁrst two terms of the objective function G ( B ) , we change the objective function as follows: More concretely, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Note that the only change in the objective is the regularizer for the norm of b j .
Revised: Note that the only change in the objective is the regularizer for the norm of b j .

Original: It is now replaced by ( [Equation]{+(( (cid:80) mi =
+}
Revised: It is now replaced by {+(( (cid:80) mi =
+}

Original: , b [Equation]
Revised: α i (cid:104)

Original:  2 − 1) 2 − 1) 2 , which tries to ensure the “norm” of b j in the basis deﬁned by row of B (cid:63) to be 1.
Revised: b j , b (cid:63)i (cid:105) 2 − 1) 2 − 1) 2 , which tries to ensure the “norm” of b j in the basis deﬁned by row of B (cid:63) to be 1.

Original: The objective function that we will optimize corresponds to choosing α i = a (cid:63)i .
Revised: The objective function that we will optimize corresponds to choosing α i = a (cid:63)i .

Original: Similar as before, this function can be computed as expectations
Revised: Similar as before, this function can be computed as expectations

Original: [Equation]
Revised: [Equation]

Original: where ( x (cid:48) , y (cid:48) ) is an independent sample, and
Revised: where ( x (cid:48) , y (cid:48) ) is an independent sample, and

Original: φ 2 ( v, x ) = ( v (cid:62) x ) 2 − (cid:107) v (cid:107) 2 .
Revised: φ 2 ( v, x ) = ( v (cid:62) x ) 2 − (cid:107) v (cid:107) 2 .

Original: Intuitively, if we can ﬁnd a linear transformation that makes { b (cid:63)i } ’s orthonormal, that will reduce the problem to the orthonormal case.
Revised: Intuitively, if we can ﬁnd a linear transformation that makes { b (cid:63)i } ’s orthonormal, that will reduce the problem to the orthonormal case.

Original: This is in fact the whitening matrix:
Revised: This is in fact the whitening matrix:

Original: Let M = (cid:80) mi =1 a (cid:63)i b (cid:63)i ( b (cid:63)i ) (cid:62) be the weighted covariance matrix of b (cid:63)i ’s. Suppose the SVD of M is
Revised: Let M = (cid:80) mi =1 a (cid:63)i b (cid:63)i ( b (cid:63)i ) (cid:62) be the weighted covariance matrix of b (cid:63)i ’s. Suppose the SVD of M is

Original: UDU (cid:62) and let W =
Revised: UDU (cid:62) and let W =

Original: UD − 1 / 2 .
Revised: UD − 1 / 2 .

Original: We apply the transformation W (cid:62) to the vectors (cid:112) a (cid:63)i b i ’s and obtain that o i =
Revised: We apply the transformation W (cid:62) to the vectors (cid:112) a (cid:63)i b i ’s and obtain that o i =

Original: W (cid:62) (cid:112) a (cid:63)i b (cid:63)i .
Revised: W (cid:62) (cid:112) a (cid:63)i b (cid:63)i .

Original: We can verify that
Revised: We can verify that

Original: o i ’s are orthogonal vectors because
Revised: o i ’s are orthogonal vectors because

Original: [Equation]
Revised: [Equation]

Original: For notational convenience, let’s extend the deﬁnition of the G ( B ) in equation by using the putting the relevant information in the subscript
Revised: For notational convenience, let’s extend the deﬁnition of the G ( B ) in equation by using the putting the relevant information in the subscript

Original: [Equation]
Revised: [Equation]

Original: (That is, the index o denotes the ground-truth solution with respect to which G is deﬁned.)
Revised: (That is, the index o denotes the ground-truth solution with respect to which G is deﬁned.)

Original: The next Theorem shows that we can rotate the objective function F properly so that it matches the objective G with a ground-truth vector
Revised: The next Theorem shows that we can rotate the objective function F properly so that it matches the objective G with a ground-truth vector

Original: o i ’s.
Revised: o i ’s.

Original: Theorem C.2.
Revised: Theorem C.2.

Original: Let W be deﬁned as above, and let 1 /a (cid:63) be the vector whose i -th entry is 1 /a (cid:63)i .
Revised: Let W be deﬁned as above, and let 1 /a (cid:63) be the vector whose i -th entry is 1 /a (cid:63)i .

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: Note this can be interpreted as a linear transformation as in vector format B W (cid:62) is equal to B · ( W (cid:62) ⊗ Id d × d ) .
Revised: Note this can be interpreted as a linear transformation as in vector format B W (cid:62) is equal to B · ( W (cid:62) ⊗ Id d × d ) .

Original: Proof.
Revised: Proof.

Original: The equality can be obtained by straightforward calculation.
Revised: The equality can be obtained by straightforward calculation.

Original: We note that since B = 
Revised: We note that since B =   ¯ b (cid:62)...

Original: [Equation]
Revised: ¯ b (cid:62) m 

Original:  , the rows of B · ( W (cid:62) ⊗ Id d × d ) are W ¯ b 1 , . . 
Revised:  , the rows of B · ( W (cid:62) ⊗ Id d × d ) are W ¯ b 1 , . . .

Original: , W ¯ b m .
Revised: , W ¯ b m .

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: (cid:88) j,k ∈ [ d ] (cid:104) o

Original: i , ¯ b j (cid:105) 2 (cid:104) o

Original: i , ¯ b k (cid:105)− ˆ σ 4 µ √ (cid:88) i,j ∈

Original: [ d ] a (cid:63)i (cid:104)

Original: o i , ¯ b j (cid:105) 4 + λ m (cid:88)

Original: j = ( m (cid:88) i = (cid:104)

Original: ¯

Original: b j , o i (cid:105) 2 − 1) 2 .

Original: (by the deﬁnition of o i ’s)
Revised: (by the deﬁnition of o i ’s)

Original: From Theorem 2.3 we can immediately get the following Corollary (note that the only difference is that the coefﬁcients now are 1 /a
Revised: From Theorem 2.3 we can immediately get the following Corollary (note that the only difference is that the coefﬁcients now are 1 /a

Original: (cid:63)i instead of a (cid:63)i ).
Revised: (cid:63)i instead of a (cid:63)i ).

Original: Recall a (cid:63)max = max i a (cid:63)i and a (cid:63)min = min i a (cid:63)min , we have
Revised: Recall a (cid:63)max = max i a (cid:63)i and a (cid:63)min = min i a (cid:63)min , we have

Original: Corollary C.3.
Revised: Corollary C.3.

Original: Let κ a = a (cid:63)max /a (cid:63)min .
Revised: Let κ a = a (cid:63)max /a (cid:63)min .

Original: Let c be a sufﬁciently small universal constant (e.g. c = 0 .
Revised: Let c be a sufﬁciently small universal constant (e.g. c = 0 .

Original: 01 sufﬁces).
Revised: 01 sufﬁces).

Original: Assume µ ≤ c/κ a and λ ≥ ( ca (cid:63)min ) − 1 .
Revised: Assume µ ≤ c/κ a and λ ≥ ( ca (cid:63)min ) − 1 .

Original: The function G 1 /a (cid:63) ,µ,λ,o i ( · ) deﬁned as in
Revised: The function G 1 /a (cid:63) ,µ,λ,o i ( · ) deﬁned as in

Original: Theorem C.2 satisﬁes that
Revised: Theorem C.2 satisﬁes that

Original: A matrix B is a local minimum of G if and only if B can be written as B = PDO where O is a matrix whose rows are o
Revised: A matrix B is a local minimum of G if and only if B can be written as B = PDO where O is a matrix whose rows are o

Original: i ’s, P is a permutation matrix and D is a diagonal matrix with D ii ∈ {± 1 ± O ( µ/λa (cid:63)min ) } .
Revised: i ’s, P is a permutation matrix and D is a diagonal matrix with D ii ∈ {± 1 ± O ( µ/λa (cid:63)min ) } .

Original: Any saddle point B has a strictly negative curvature in the sense that λ min ( ∇ 2 G ( B )) ≥ − τ 0 where τ 

Revised: Any saddle point B has a strictly negative curvature in the sense that λ min ( ∇ 2 G ( B )) ≥ − τ 0 where τ

Original: = c min { µ/ ( κ a a (cid:63)max d ) , λ } 3.
Revised: = c min { µ/ ( κ a a (cid:63)max d ) , λ } 3.

Original: Suppose B is an approximate local minimum in the sense that B satisﬁes (cid:107)∇ g ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 g ( B )) ≥ − τThen B can be written as B = PDO + E where P is a permutation matrix, D is a diagonal matrix and | E | ∞ ≤ O ( εa (cid:63)max / ˆ σ 4 ) .
Revised: Suppose B is an approximate local minimum in the sense that B satisﬁes (cid:107)∇ g ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 g ( B )) ≥ − τThen B can be written as B = PDO + E where P is a permutation matrix, D is a diagonal matrix and | E | ∞ ≤ O ( εa (cid:63)max / ˆ σ 4 ) .

Original: Finally, we can combine the theorem above and Theorem B.1 to give a guarantee for optimizing F .
Revised: Finally, we can combine the theorem above and Theorem B.1 to give a guarantee for optimizing F .

Original: Let Γ be a diagonal matrix with Γ ii = (cid:112) a (cid:63) i .
Revised: Let Γ be a diagonal matrix with Γ ii = (cid:112) a (cid:63)i .

Original: Let M = B (cid:63) (cid:62) Γ 2 B (cid:63) and κ ( M )
Revised: Let M = B (cid:63) (cid:62) Γ 2 B (cid:63) and κ ( M )

Original: = (cid:107) M (cid:107) /σ min ( M ) .
Revised: = (cid:107) M (cid:107) /σ min ( M ) .

Original: Theorem C.4.
Revised: Theorem C.4.

Original: Let c be a sufﬁciently small universal constant (e.g. c = 0 .
Revised: Let c be a sufﬁciently small universal constant (e.g. c = 0 .

Original: 01 sufﬁces).
Revised: 01 sufﬁces).

Original: Let κ a = a (cid:63)max /a (cid:63)min .
Revised: Let κ a = a (cid:63)max /a (cid:63)min .

Original: Assume µ ≤ c/κ a and λ ≥ 1 / ( c · a (cid:63) min ) .
Revised: Assume µ ≤ c/κ a and λ ≥ 1 / ( c · a (cid:63) min ) .

Original: The function F ( · ) deﬁned as in Theorem C. satisﬁes that
Revised: The function F ( · ) deﬁned as in Theorem C. satisﬁes that

Original: A matrix B is a local minimum of F if and only if B satisfy B −(cid:62) =
Revised: A matrix B is a local minimum of F if and only if B satisfy B −(cid:62) =

Original: PD Γ B (cid:63) where P is a permutation matrix, Γ is a diagonal matrix with Γ ii =
Revised: PD Γ B (cid:63) where P is a permutation matrix, Γ is a diagonal matrix with Γ ii =

Original: (cid:112) a (cid:63)i , and D is a diagonal matrix with D ii ∈ {± 1 ± O ( µ/λa (cid:63)min ) } .
Revised: (cid:112) a (cid:63)i , and D is a diagonal matrix with D ii ∈ {± 1 ± O ( µ/λa (cid:63)min ) } .

Original: Furthermore, this means that all local minima of F are also global.
Revised: Furthermore, this means that all local minima of F are also global.

Original: Any saddle point B has a strictly negative curvature in the sense that λ min ( ∇ 2 F ( B )) ≥ − τ 0 where τ 

Revised: Any saddle point B has a strictly negative curvature in the sense that λ min ( ∇ 2 F ( B )) ≥ − τ 0 where τ

Original: = c min { µ/ ( κ a da (cid:63) max ) , λ } σ min ( M ) .
Revised: = c min { µ/ ( κ a da (cid:63) max ) , λ } σ min ( M ) .

Original: Suppose B is an approximate local minimum in the sense that B satisﬁes(cid:107)∇ F ( B ) (cid:107) ≤ ε and λ min ( ∇ 2 F ( B )) ≥
Revised: Suppose B is an approximate local minimum in the sense that B satisﬁes

Original: − τ
Revised: [Equation]

Original: Then B can be written as B −(cid:62) =
Revised: Then B can be written as B −(cid:62) =

Original: PD Γ B (cid:63) + E where Γ , D, P are as in 1, the error term 
Revised: PD Γ B (cid:63) + E where Γ , D, P are as in 1, the error term (cid:107) [Equation] md

Original: Proof.
Revised: [Equation] [Equation] / ˆ [Equation]

Revised: when

Revised: εa (cid:63)

Revised: max

Revised: √ md · κ (

Original:  Note that we can immediately apply Theorem 2.3 to G 1 /a (cid:63) ,µ,λ,o i ( B ) to characterize all its local minima.
Revised: M 2 / ˆ σ 4 < c ). Proof. Note that we can immediately apply Theorem 2.3 to G 1 /a (cid:63) ,µ,λ,o i ( B ) to characterize all its local minima.

Original: See Corollary C.3.
Revised: See Corollary C.3.

Original: Next we will transform the properties for local minima of G (stated in Corollary C.3) to F using
Revised: Next we will transform the properties for local minima of G (stated in Corollary C.3) to F using

Original: Theorem C.1.
Revised: Theorem C.1.

Original: First we note that the transformation matrix W and M are closely related:
Revised: First we note that the transformation matrix W and M are closely related:

Original: [Equation]
Revised: [Equation]

Original: This is because according to the deﬁnition of W , the SVD of M is M = UDU (cid:62) and W =
Revised: This is because according to the deﬁnition of W , the SVD of M is M = UDU (cid:62) and W =

Original: UD − 1 / 2 , so [Equation]
Revised: UD − 1 / 2 , so WW (cid:62) =

Original:  U [Equation]
Revised: UD − 1 U (cid:62) =

Original: = M − 1 .
Revised: M − 1 .

Original: The claims of the singular values follow immediately from the
Revised: The claims of the singular values follow immediately from the

Original: SVD of M and W .
Revised: SVD of M and W .

Original: As a result, all local minimum of F are of the form B W (cid:62) where B is a local minimum of G .
Revised: As a result, all local minimum of F are of the form B W (cid:62) where B is a local minimum of G .

Original: For
Revised: For

Original: B = B W (cid:62) , the gradient and Hessian of F ( B ) and G ( B ) are also related by Theorem C.1.
Revised: B = B W (cid:62) , the gradient and Hessian of F ( B ) and G ( B ) are also related by Theorem C.1.

Original: Let us ﬁrst prove 1.
Revised: Let us ﬁrst prove 1.

Original: By Corollary C.3, we know every local minimum of G is of the form B =
Revised: By Corollary C.3, we know every local minimum of G is of the form B =

Original: PDO .
Revised: PDO .

Original: According to the deﬁnition of O in Theorem C.2, we know each row vector
Revised: According to the deﬁnition of O in Theorem C.2, we know each row vector

Original: o i is equal to
Revised: o i is equal to

Original: W (cid:62) ( a (cid:63)i ) 1 / 2 b (cid:63)i , therefore O =
Revised: W (cid:62) ( a (cid:63) i ) 1 / 2 b (cid:63) i , therefore O =

Original: Γ B (cid:63) W .
Revised: Γ B (cid:63) W .

Original: As a result, all local minima of G are of the form B =
Revised: As a result, all local minima of G are of the form B =

Original: PD Γ B (cid:63) W .
Revised: PD Γ B (cid:63) W .

Original: By Theorem C.1 and Theorem C.2, we know all local minima of F must be of the formB =
Revised: By Theorem C.1 and Theorem C.2, we know all local minima of F must be of the form

Original: [Equation]
Revised: [Equation]

Original: Now we try to compute B −(cid:62) .
Revised: Now we try to compute B −(cid:62) .

Original: To do that observe that [Γ B (cid:63) ] M − 1 [Γ B (cid:63) ] (cid:62) =
Revised: To do that observe that [Γ B (cid:63) ] M − 1 [Γ B (cid:63) ] (cid:62) =

Original: I .
Revised: I .

Original: Therefore [Γ B (cid:63) M − 1 ] −(cid:62) =
Revised: Therefore [Γ B (cid:63) M − 1 ] −(cid:62) =

Original: Γ B (cid:63) , and for any local minimum B , we have
Revised: Γ B (cid:63) , and for any local minimum B , we have

Original: [Equation]
Revised: [Equation]

Original:  Note that P (cid:62) is still a permutation matrix, and D −(cid:62) is still a matrix whose diagonal entries are {± 1 ± O ( µ/λa (cid:63) min ) } , so this is exactly the form we stated in 1.
Revised: [Equation] Note that P (cid:62) is still a permutation matrix, and D −(cid:62) is still a matrix whose diagonal entries are {± 1 ± O ( µ/λa (cid:63) min ) } , so this is exactly the form we stated in 1.

Original: More concretely, the rows of B −(cid:62) are permutations of (cid:112) a (cid:63)i b (cid:63)i .
Revised: More concretely, the rows of B −(cid:62) are permutations of (cid:112) a (cid:63)i b (cid:63)i .

Original: For bullet 2, it follows immediately from Property 2 in Theorem C.1.
Revised: For bullet 2, it follows immediately from Property 2 in Theorem C.1.

Original: Note that by property 2,
Revised: Note that by property 2,

Original: [Equation]
Revised: [Equation]

Original: Finally we will prove 3.
Revised: Finally we will prove 3.

Original: Let B = BW −(cid:62) , so that G ( B ) =
Revised: Let B = BW −(cid:62) , so that G ( B ) =

Original: F ( B ) .
Revised: F ( B ) .

Original: We will prove properties of B using the properties of B from Corollary C.3.
Revised: We will prove properties of B using the properties of B from Corollary C.3.

Original: First we observe that by Theorem C.1,
Revised: First we observe that by Theorem C.1,

Original: [Equation]
Revised: [Equation]

Original: Therefore the second order condition for Claim 3 in Corollary C.3 is satisﬁed.Now when
Revised: Therefore the second order condition for Claim 3 in Corollary C.3 is satisﬁed.

Original: [Equation]

Original:  , we have [Equation] G ( B ) (cid:107) ≤ ε (cid:107) W (cid:107) =
Revised: Now when (cid:107)∇ F ( B ) (cid:107) ≤ ε , we have (cid:107)∇ G ( B ) (cid:107) ≤ ε (cid:107) W (cid:107) =

Original: ε/σ min ( M ) 1 / 2 .
Revised: ε/σ min ( M ) 1 / 2 .

Original: By Corollary C.3, we know B can be expressed as PDO + E (cid:48) where D is the diagonal matrix, P is a permutation matrix and | E (cid:48) | ∞ ≤ O ( εa (cid:63) max / (ˆ σ 4 σ min ( M ) 1 / 2 )) .
Revised: By Corollary C.3, we know B can be expressed as PDO + E (cid:48) where D is the diagonal matrix, P is a permutation matrix and | E (cid:48) | ∞ ≤ O ( εa (cid:63) max / (ˆ σ 4 σ min ( M ) 1 / 2 )) .

Original: We will apply perturbation Theorem C.9 for matrix inversion.
Revised: We will apply perturbation Theorem C.9 for matrix inversion.

Original: Since σ min ( PDO ) ≥ 1 / 2 , we know when (cid:107) E (cid:48) (cid:107) ≤ 1 / 4 ,
Revised: Since σ min ( PDO ) ≥ 1 / 2 , we know when (cid:107) E [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here (cid:107) E (cid:107) is bounded by (cid:107) E (cid:107) F ≤ √ md | E (cid:48) | ∞ ≤ O ( εa (cid:63) max √ md/ (ˆ σ 4 σ min ( M ) 1 / 2 )) , which is smaller than 1 / 4 when ε is small enough.
Revised: Here (cid:107) E (cid:107) is bounded by (cid:107) E (cid:107) F ≤ √ md | E (cid:48) | ∞ ≤ O ( εa (cid:63) max √ md/ (ˆ σ 4 σ min ( M ) 1 / 2 )) , which is smaller than 1 / 4 when ε is small enough.

Original: The corresponding point in F is B = B W (cid:62) , and in 1 we have already proved ( PDOW (cid:62) ) −(cid:62) is of the form we want, therefore we can deﬁne E = B −(cid:62) − ( PDOW
Revised: The corresponding point in F is B = B W (cid:62) , and in 1 we have already proved ( PDOW (cid:62) ) −(cid:62) is of the form we want, therefore we can deﬁne E = B −(cid:62) − ( PDOW

Original: [Equation]
Revised: [Equation]

Original: ) −(cid:62)
Revised: ) −(cid:62)

Original: =
Revised: =

Original: ( B − PDO ) −(cid:62)
Revised: ( B − PDO ) −(cid:62)

Original: W − 1 , and
Revised: W − 1 , and

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes the proof.
Revised: This ﬁnishes the proof.

Original: C. H ANDLE U NDERCOMPLETE C ASE
Revised: C. H ANDLE U NDERCOMPLETE C ASE

Original: The objective function F can handle the case when the weights b (cid:63)i ’s are not orthogonal, but still requires the number of components m to be equal to the number of dimensions d .
Revised: The objective function F can handle the case when the weights b (cid:63)i ’s are not orthogonal, but still requires the number of components m to be equal to the number of dimensions d .

Original: In this section we show how to use similar ideas for the case when the number of components is smaller than the dimension ( m < d ).
Revised: In this section we show how to use similar ideas for the case when the number of components is smaller than the dimension ( m < d ).

Original: Note that all the terms in F ( B ) only depends on the inner-products (cid:104) b j , b (cid:63)i (cid:105) .
Revised: Note that all the terms in F ( B ) only depends on the inner-products (cid:104) b j , b (cid:63) i (cid:105) .

Original: Let S be the span of { b (cid:63)i } ’s and P S be the projection matrix to this subspace, it is easy to see that F ( B ) satisﬁes
Revised: Let S be the span of { b (cid:63)i } ’s and P S be the projection matrix to this subspace, it is easy to see that F ( B ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: That is, the previous objective function only depends on the projection of B in the space S .
Revised: That is, the previous objective function only depends on the projection of B in the space S .

Original: Using similar argument as Theorem C.4, it is not hard to show the only local optimum in S satisﬁes the same conditions, and allow us to recover B (cid:63) .
Revised: Using similar argument as Theorem C.4, it is not hard to show the only local optimum in S satisﬁes the same conditions, and allow us to recover B (cid:63) .

Original: However, without modifying the objective, the local optimum of F ( B ) can have arbitrary components in the orthogonal subspace S ⊥ .
Revised: However, without modifying the objective, the local optimum of F ( B ) can have arbitrary components in the orthogonal subspace S ⊥ .

Original: In order to prevent the components from S ⊥ , we add an additional (cid:96) 2 regularizer: deﬁne F α,µ,λ,δ as follows:
Revised: In order to prevent the components from S ⊥ , we add an additional (cid:96) 2 regularizer: deﬁne F α,µ,λ,δ as follows:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, since the ﬁrst term F α,µ,λ ( B ) only cares about the projection BP S , minimizing (cid:107) B (cid:107) 2 F will remove the components in the orthogonal subspace of S .
Revised: Intuitively, since the ﬁrst term F α,µ,λ ( B ) only cares about the projection BP S , minimizing (cid:107) B (cid:107) 2 F will remove the components in the orthogonal subspace of S .

Original: We will choose δ carefully to make sure that the additional term does not change the local optima of F α,µ,λ ( B ) by too much, while still ensuring a small projection on S ⊥ .
Revised: We will choose δ carefully to make sure that the additional term does not change the local optima of F α,µ,λ ( B ) by too much, while still ensuring a small projection on S ⊥ .

Original: In this case we will consider pseudo-inverse instead of inverse.
Revised: In this case we will consider pseudo-inverse instead of inverse.

Original: In particular, for a m × d matrix B , deﬁne its pseudo-inverse B † to be the matrix such that BB † =
Revised: In particular, for a m × d matrix B , deﬁne its pseudo-inverse B † to be the matrix such that BB † =

Original: Id m × m and B † B is the projection to the row span of B .
Revised: Id m × m and B † B is the projection to the row span of B .

Original: Let M
Revised: Let M

Original: = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Theorem C.5.
Revised: Theorem C.5.

Original: For any desired accuracy ε 0 , we can choose parameters ε, δ, τ 0 , µ, λ , such that for the objective function F a (cid:63) ,µ,λ,δ ( B ) , for any B such that
Revised: For any desired accuracy ε 0 , we can choose parameters ε, δ, τ 0 , µ, λ , such that for the objective function F a (cid:63) ,µ,λ,δ ( B ) , for any B such that

Original: [Equation]
Revised: [Equation]

Original: we have [ B † ] (cid:62) =
Revised: we have [ B † ] (cid:62) =

Original: B (cid:63) D Γ P
Revised: B (cid:63) D Γ P

Original: + E where Γ is a diagonal matrix with entries (cid:112) a (cid:63)i , D is a diagonal matrix with entries close to 1, P is a permutation matrix and (cid:107) E (cid:107) ≤ ε 0 .
Revised: + E where Γ is a diagonal matrix with entries (cid:112) a (cid:63)i , D is a diagonal matrix with entries close to 1, P is a permutation matrix and (cid:107) E (cid:107) ≤ ε 0 .

Original: To choose the parameters, let c be a sufﬁciently small universal constant (e.g. c = 0 .
Revised: To choose the parameters, let c be a sufﬁciently small universal constant (e.g. c = 0 .

Original: 01 sufﬁces).
Revised: 01 sufﬁces).

Original: Assume µ ≤ c/κ (cid:63) and λ ≥ 1 / ( c · a (cid:63) min ) .
Revised: Assume µ ≤ c/κ (cid:63) and λ ≥ 1 / ( c · a (cid:63) min ) .

Original: Let τ 

Revised: Let τ

Original: = c min { µ/ ( κda (cid:63) max ) , λ } σ min ( M ) .
Revised: = c min { µ/ ( κda (cid:63) max ) , λ } σ min ( M ) .

Original: Let δ ≤ min {

Original: [Equation]

Original: [Equation] 0 a (cid:63)max · m √ dκ 1 / [-
-] M ) , τ 0 / 2 } , and ε = min { λσ min ( M ) 1 / 2 , cδ/ (cid:112) (cid:107) M (cid:107) , cε [Equation]
Revised: Let δ ≤ min { c ˆ σ 4 ε 0 a (cid:63)max · m √ dκ 1 / [-
-]2 ( M ) , τ 0 / 2 } , and ε = min { λσ min ( M ) 1 / 2 , cδ/ (cid:112) (cid:107) M (cid:107) , cε 0 δσ min ( M ) } .

Original: min [Equation] M ) } . We ﬁrst show that if the gradient is small, then the point cannot have a large component in S ⊥ .
Revised: We ﬁrst show that if the gradient is small, then the point cannot have a large component in S ⊥ .

Original: Lemma C.6.
Revised: Lemma C.6.

Original: If (cid:107)∇F a (cid:63) ,µ,λ,δ ( B ) (cid:107) ≤ ε , then (cid:107) P S ⊥ B (cid:107) F ≤ ε/δ .
Revised: If (cid:107)∇F a (cid:63) ,µ,λ,δ ( B ) (cid:107) ≤ ε , then (cid:107) P S ⊥ B (cid:107) F ≤ ε/δ .

Original: Proof.
Revised: Proof.

Original: Since F a (cid:63) ,µ,λ ( B ) only depends BP S , we know ∇ F a (cid:63) ,µ,λ ( B ) P S ⊥ = 0 .
Revised: Since F a (cid:63) ,µ,λ ( B ) only depends BP S , we know ∇ F a (cid:63) ,µ,λ ( B ) P S ⊥ = 0 .

Original: Therefore ε ≥ (cid:107)∇F
Revised: Therefore ε ≥ (cid:107)∇F

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: B

Original: [Equation]
Revised: [Equation]

Original: F , and we have (cid:107) BP S ⊥ (cid:107) F ≤ ε/δ as desired.
Revised: , and we have (cid:107) BP S ⊥ (cid:107) F ≤ ε/δ as desired.

Original: Next we show that if the gradient of F a (cid:63) ,µ,λ,δ ( B ) is small, and δ is also small, then the gradient of
Revised: Next we show that if the gradient of F a (cid:63) ,µ,λ,δ ( B ) is small, and δ is also small, then the gradient of

Original: F a (cid:63) ,µ,λ ( B ) can be bounded.
Revised: F a (cid:63) ,µ,λ ( B ) can be bounded.

Original: Lemma C.7.
Revised: Lemma C.7.

Original: In the setting of Theorem C.5, if (cid:107)∇F a (cid:63) ,µ,λ,δ ( B ) (cid:107) ≤ ε ≤ λσ min ( M ) 1 / 2 , then we have
Revised: In the setting of Theorem C.5, if (cid:107)∇F a (cid:63) ,µ,λ,δ ( B ) (cid:107) ≤ ε ≤ λσ min ( M ) 1 / 2 , then we have

Original: [Equation]
Revised: [Equation]

Original: Towards proving Lemma C.7, we ﬁrst bound the norm of B by the following claim:
Revised: Towards proving Lemma C.7, we ﬁrst bound the norm of B by the following claim:

Original: Claim C.8 .
Revised: Claim C.8 .

Original: If (cid:107)∇F a (cid:63) ,µ [Equation] B ) (cid:107) ≤ λσ min ( M ) 1 / 2 , then each row b i must satisfy b (cid:62) i Mb i ≤ 2 .
Revised: If (cid:107)∇F a (cid:63) ,µ,λ,δ ( B ) (cid:107) ≤ λσ min ( M ) 1 / 2 , then each row b i must satisfy b (cid:62) i Mb i ≤ 2 .

Original: Proof.
Revised: Proof.

Original: We prove by contradiction.
Revised: We prove by contradiction.

Original: Assume towards contradiction that there is a column b i such that b (cid:62) i
Revised: Assume towards contradiction that there is a column b i such that [Equation]

Original: Mb i ≥ 2 .
Revised: Mb i ≥ 2 .

Original: We consider the quantity,
Revised: We consider the quantity,

Original: [Equation]
Revised: [Equation]

Original: Note that F a (cid:63) ,µ,λ,δ has 4 terms:
Revised: Note that F a (cid:63) ,µ,λ,δ has 4 terms:

Original: [Equation]
Revised: [Equation]

Original: Among these 4 terms, the ﬁrst, third and forth terms all contribute positively to this innerproduct (because when b i is moved to (1 − ε )
Revised: Among these 4 terms, the ﬁrst, third and forth terms all contribute positively to this innerproduct (because when b i is moved to (1 − ε )

Original: b i all those terms clearly decrease).
Revised: b i all those terms clearly decrease).

Original: Term
Revised: Term

Original: [Table] [Equation] 
Revised: [Equation] [Table]

Original: [Table]
Revised: [Equation]

Original: 4 contribute negatively.
Revised: 4 contribute negatively.

Original: Therefore we can ignore terms 1 and 4:
Revised: Therefore we can ignore terms 1 and 4:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]

Original: (cid:104)

Original: [Equation]

Original: [Equation] b [Equation] (cid:105) 
Revised: 1) 2 ] , b i (cid:105) .

Original: [Equation]

Original: Let b (cid:62) i Mb i
Revised: Let b (cid:62) i Mb i

Original: =
Revised: =

Original: C ≥ 2 , we know (cid:80) i [Equation]
Revised: C ≥ 2 , we know (cid:80) i ∈

Original: [ m [Equation] [Equation] ≤
Revised: [ m ] a (cid:63)i (cid:104) b (cid:63)i , b j (cid:105) 4 ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:63)min (cid:80)

Original: i
Revised: i

Original: [Equation]

Original: [ m ]

Original: [Equation]

Original: a (cid:63)i

Original: [Equation]
Revised: [Equation]

Original: b

Original: [Equation]
Revised: [Equation]

Original: ≤ C

Original: [Equation]

Original: (cid:63)min .
Revised: (cid:63)min .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Revised: min

Original: On the other hand,
Revised: On the other hand,

Original: [Equation]
Revised: [Equation]

Original: By the choice of λ, µ , we can see that the negative term is negligible, and we know
Revised: By the choice of λ, µ , we can see that the negative term is negligible, and we know

Original: [Equation]
Revised: [Equation]

Original: Since b (cid:62) i Mb i =
Revised: Since b (cid:62) i Mb i =

Original: C , we have (cid:107) b
Revised: C , we have (cid:107) b

Original: i (cid:107) ≤ (cid:112) C/σ min ( M ) .
Revised: i (cid:107) ≤ (cid:112) C/σ min ( M ) .

Original: Therefore the norm of the gradient is at least 2 λC ( C − 1) / (cid:107) b i | ≥ 2 √ 2 λσ min ( M ) , this contradicts with the assumption.
Revised: Therefore the norm of the gradient is at least 2 λC ( C − 1) / (cid:107) b i | ≥ 2 √ 2 λσ min ( M ) , this contradicts with the assumption.

Original: The norm of the rows must all be bounded.
Revised: The norm of the rows must all be bounded.

Original: Proof of Lemma C.7.
Revised: Proof of Lemma C.7.

Original: We have that b (cid:62) i
Revised: We have that b (cid:62) i

Original: Mb i ≤ 2 implies (cid:107) b
Revised: Mb i ≤ 2 implies (cid:107) b

Original: i (cid:107) 2 ≤ 2 /σ min ( M ) .
Revised: i (cid:107) 2 ≤ 2 /σ min ( M ) .

Original: The norm of the whole matrix is bounded by (cid:107) B (cid:107) F ≤(cid:112)(cid:80) m i =
Revised: The norm of the whole matrix is bounded by (cid:107) B (cid:107) F ≤

Original: [Equation]
Revised: [Equation]

Original: b i (cid:107) 2 ≤ (cid:112) 2 m/σ min ( M ) , so by triangle inequality we have
Revised: 2 ≤ [Equation] 2 m/σ min ( M ) , so by triangle inequality we have

Original: [Equation]
Revised: [Equation]

Original: Finally we are ready to prove Theorem C.5.
Revised: Finally we are ready to prove Theorem C.5.

Original: Proof of Theorem C.5.
Revised: Proof of Theorem C.5.

Original: We will separate B into two components B S = BP S and B ⊥ =
Revised: We will separate B into two components B S = BP S and B ⊥ =

Original: BP S ⊥ .
Revised: BP S ⊥ .

Original: We will ﬁrst show that B S is close to the desirable solution.
Revised: We will ﬁrst show that B S is close to the desirable solution.

Original: To do that we will use Theorem C.4.
Revised: To do that we will use Theorem C.4.

Original: By the choice of ε, δ , we know from Lemma C.7 that (cid:107)∇ F a (cid:63) ,µ,λ ( B S ) (cid:107) ≤ 2 δ (cid:112) 2 m/σ min ( M ) .
Revised: By the choice of ε, δ , we know from Lemma C.7 that (cid:107)∇ F a (cid:63) ,µ,λ ( B S ) (cid:107) ≤ 2 δ (cid:112) 2 m/σ min ( M ) .

Original: Also, ∇ [Equation] ) − δ ≥ − τ 0 .
Revised: Also, ∇ 2 F a (cid:63) ,µ,λ ( B S ) ≥ ∇ 2 F a (cid:63) ,µ,λ ( B ) − δ ≥ − τ 0 .

Original: Therefore we know B S must be of the form
Revised: Therefore we know B S must be of the form

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) E 1 (cid:107) <
Revised: where (cid:107) E 1 (cid:107) <

Original: ε 0 / 2 .
Revised: ε 0 / 2 .

Original: Also at the same time from the proof of Theorem C.4 we know B S =
Revised: Also at the same time from the proof of Theorem C.4 we know B S =

Original: ( PDO +
Revised: ( PDO +

Original: E (cid:48) )
Revised: E (cid:48) )

Original: W (cid:62) where PDO + E (cid:48) has singular values close to 1.
Revised: W (cid:62) where PDO + E (cid:48) has singular values close to 1.

Original: Therefore σ min ( B S ) ≥ σ min ( W ) / 2 = 1 / 2 (cid:112) (cid:107) M
Revised: Therefore σ min ( B S ) ≥ σ min ( W ) / 2 = 1 / 2 (cid:112) (cid:107) M

Original: (cid:107) .
Revised: (cid:107) .

Original: By Lemma C.6 we know (cid:107) B ⊥ (cid:107) F ≤ ε/δ .
Revised: By Lemma C.6 we know (cid:107) B ⊥ (cid:107) F ≤ ε/δ .

Original: We apply inverse matrix perturbation (Theorem C.9) again, using B = B S + B ⊥ , therefore we know
Revised: We apply inverse matrix perturbation (Theorem C.9) again, using B = B S + B ⊥ , therefore we know

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) E 2 (cid:107) ≤ O ( (cid:107) B ⊥ (cid:107) F /σ 2 min ( B S )) ≤ ε 0 / 2 .
Revised: where (cid:107) E 2 (cid:107) ≤ O ( (cid:107) B ⊥ (cid:107) F /σ 2 min ( B S )) ≤ ε 0 / 2 .

Original: Combining these two perturbations we know[
Revised: Combining these two perturbations we know

Original: [Equation]
Revised: [Equation]

Original: and the error term E 1 + E (cid:62) 2 has spectral norm at most ε 0 .
Revised: and the error term E 1 + E (cid:62) 2 has spectral norm at most ε 0 .

Original: C. T OOLBOX : M ATRIX P ERTURBATION
Revised: C. T OOLBOX : M ATRIX P ERTURBATION

Original: In the proof we used the following theorem for the perturbation of matrices.
Revised: In the proof we used the following theorem for the perturbation of matrices.

Original: Theorem C.9 (Stewart and Sun Stewart & guang Sun (1990)) .
Revised: Theorem C.9 (Stewart and Sun Stewart & guang Sun (1990)) .

Original: Consider the perturbation of a matrix
Revised: Consider the perturbation of a matrix

Original: A : if B =
Revised: A : if B =

Original: A + E ,then we have
Revised: A + E ,then we have

Original: [Equation]
Revised: [Equation]

Original: As a corollary, if (cid:107) E (cid:107) ≤ σ min ( A ) / 2 , then we have
Revised: As a corollary, if (cid:107) E (cid:107) ≤ σ min ( A ) / 2 , then we have

Original: [Equation]
Revised: [Equation]

Original: D R ECOVERING THE L INEAR L AYER
Revised: D R ECOVERING THE L INEAR L AYER

Original: We will show in this section that if we have are given a δ -approximation of B (cid:63) , then it is easy to recover a (cid:63) .
Revised: We will show in this section that if we have are given a δ -approximation of B (cid:63) , then it is easy to recover a (cid:63) .

Original: The key observation here is that the correlation between the (cid:104) b (cid:63)i , x (cid:105) and the output y is exactly proportional to a (cid:63)i .
Revised: The key observation here is that the correlation between the (cid:104) b (cid:63)i , x (cid:105) and the output y is exactly proportional to a (cid:63)i .

Original: We also note that there could be multiple other ways to recover a (cid:63) , e.g., using linear regression with the σ ( Bx ) as input and the y as output.
Revised: We also note that there could be multiple other ways to recover a (cid:63) , e.g., using linear regression with the σ ( Bx ) as input and the y as output.

Original: We chose this algorithm mostly because of the ease of analysis.Algorithm
Revised: We chose this algorithm mostly because of the ease of analysis.

Original: Input: A matrix B with unit row norms that is row-wise δ -close to B (cid:63) in Euclidean distance.
Revised: Input: A matrix B with unit row norms that is row-wise δ -close to B (cid:63) in Euclidean distance.

Original: Return: Let a (cid:48)
Revised: Return: Let a (cid:48)

Original: i = 2 (cid:98) E
Revised: i = 2 (cid:98) E

Original: [ y (cid:104) x, b i (cid:105) ] where (cid:98) E means the empirical average.
Revised: [ y (cid:104) x, b i (cid:105) ] where (cid:98) E means the empirical average.

Original: Set a
Revised: Set a

Original: i ← | a (cid:48)
Revised: i ← | a (cid:48)

Original: i | and b i
Revised: i | and b i

Original: ← b i sgn ( a (cid:48) i )
Revised: ← b i sgn ( a (cid:48) i )

Original: Lemma (Restatement of Lemma 2.5) .
Revised: Lemma (Restatement of Lemma 2.5) .

Original: Given a matrix B whose rows are δ -close to B (cid:63) in Euclidean distance up to permutation and sign ﬂip with δ ≤ 1 / (2 κ (cid:63) ) .
Revised: Given a matrix B whose rows are δ -close to B (cid:63) in Euclidean distance up to permutation and sign ﬂip with δ ≤ 1 / (2 κ (cid:63) ) .

Original: Then, we can give estimates a, B (cid:48) (using e.g., Algorithm 1) such that there exists a permutation P where (cid:107) a − Pa (cid:63) (cid:107) ∞
Revised: Then, we can give estimates a, B (cid:48) (using e.g., Algorithm 1) such that there exists a permutation P where (cid:107) a − Pa (cid:63) (cid:107) ∞

Original: ≤ δa (cid:63) max and B (cid:48) is row-wise δ -close to PB (cid:63) .
Revised: ≤ δa (cid:63) max and B (cid:48) is row-wise δ -close to PB (cid:63) .

Original: To see why this simple algorithm works for recovering a (cid:63) , we need the following simple claim.
Revised: To see why this simple algorithm works for recovering a (cid:63) , we need the following simple claim.

Original: Claim D.1 .
Revised: Claim D.1 .

Original: For any vector v we have
Revised: For any vector v we have

Revised: [Equation]

Revised: i =

Revised: [Equation]

Revised: b (cid:63)i

Original: [Equation]
Revised: [Equation]

Original: The proof of this claim follows immediately from the property of Hermite polynomials.
Revised: The proof of this claim follows immediately from the property of Hermite polynomials.

Original: Now we are ready to prove the corollary.
Revised: Now we are ready to prove the corollary.

Original: Proof.
Revised: Proof.

Original: Without loss of generality we assume B is close to a sign ﬂip of B (cid:63) .
Revised: Without loss of generality we assume B is close to a sign ﬂip of B (cid:63) .

Original: The unknown permutation does not change the proof.
Revised: The unknown permutation does not change the proof.

Original: Since b i is δ close to B (cid:63)i , let u be the vector where u j = (cid:104) b (cid:63)j ,
Revised: Since b i is δ close to B (cid:63)i , let u be the vector where u j = (cid:104) b (cid:63)j ,

Original: b i − b (cid:63)i (cid:105) , we have a 
Revised: b i − b (cid:63)i (cid:105) , we have a (cid:48) i = m (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: i = m
Revised: ±

Original: [Equation]
Revised: [Equation]

Revised: δ.

Original: Therefore a (cid:48)
Revised: Therefore a (cid:48)

Original: i is always positive, a i is in the desirable range and (cid:107) B (cid:48)
Revised: i is always positive, a i is in the desirable range and (cid:107) B (cid:48)

Original: i − B (cid:63)i (cid:107) ≤ δ .
Revised: i − B (cid:63)i (cid:107) ≤ δ .

Original: Similarly, if − b i is δ close to B (cid:63)i , we have a (cid:48)
Revised: Similarly, if − b i is δ close to B (cid:63)i , we have a (cid:48)

Original: i ∈ − a (cid:63)i ± a (cid:63)max δ , and the conclusion still holds.
Revised: i ∈ − a (cid:63)i ± a (cid:63)max δ , and the conclusion still holds.

Original: For the settings considered in Section C, the vectors b (cid:63)i are not necessarily orthogonal.
Revised: For the settings considered in Section C, the vectors b (cid:63)i are not necessarily orthogonal.

Original: In this case we use the following algorithm:
Revised: In this case we use the following algorithm:

Original: Algorithm 

Revised: Algorithm

Original: Recovering a (cid:63) for general case
Revised: Recovering a (cid:63) for general case

Original: Input:  and sign ﬂip.
Revised: Input: A matrix B with unit row norms, and B is δ -close to B (cid:63)

Revised: in spectral norm up to permutation and sign

Revised: ﬂip

Original: Let u i =
Revised: Let u i =

Original: 2 (cid:98) E
Revised: 2 (cid:98) E

Original: [ y (cid:104) x, b i (cid:105) ] where (cid:98) E means the empirical average.
Revised: [ y (cid:104) x, b i (cid:105) ] where (cid:98) E means the empirical average.

Original: Let a (cid:48)
Revised: Let a (cid:48)

Original: = ( BB (cid:62) ) − 1 u .
Revised: = ( BB (cid:62) ) − 1 u .

Original: Return: Set a i ← | a (cid:48) i | and b i

Original: ← b i sgn ( a (cid:48) i )

Original: Lemma D.2.
Revised: Lemma D.2.

Original: Given a matrix B whose rows have unit norm, and (cid:107) B − SPB (cid:63) (cid:107) ≤ δ for some permutation matrix P and diagonal matrix S with ± 1 entries on diagonals.
Revised: Given a matrix B whose rows have unit norm, and (cid:107) B − SPB (cid:63) (cid:107) ≤ δ for some permutation matrix P and diagonal matrix S with ± 1 entries on diagonals.

Original: If σ 2 min ( B ) 4 √ 2 κ (cid:63) √ m , we can give estimates a, B (cid:48) (using e.g., Algorithm 2) such that (cid:107) a − Pa (cid:63) (cid:107) ≤ 2 √ 2 a (cid:63)max √ m σ − 2 min ( B ) · δ and (cid:107) B (cid:48) − PB (cid:63) (cid:107) ≤ δ .
Revised: If σ 2 min ( B ) 4 √ 2 κ (cid:63) √ m , we can give estimates a, B (cid:48) (using e.g., Algorithm 2) such that (cid:107) a − Pa [Equation] and [Equation]

Original: Proof.
Revised: B [Equation] − PB [Equation]

Original:  We again use Claim D.1: in this case we know the vector u satisﬁes u = B ( B (cid:63) ) (cid:62) a(cid:63) .
Revised: (cid:107) ≤ δ . Proof. We again use Claim D.1: in this case we know the vector u satisﬁes u = B ( B (cid:63) ) (cid:62) a

Revised: [Equation]

Original: As a result, for the vector a (cid:48) , we have
Revised: As a result, for the vector a (cid:48) , we have

Original: [Equation]
Revised: [Equation]

Original: By assumption we know B = SPB (cid:63) + E where (cid:107) E (cid:107) ≤ δ .
Revised: By assumption we know B = SPB (cid:63) + E where (cid:107) E (cid:107) ≤ δ .

Original: By the perturbation of matrix inverse (Theorem C.9), we know if (cid:107) E (cid:107) ≤ δ ≤ σ min ( B ) / 2 , then B † =
Revised: By the perturbation of matrix inverse (Theorem C.9), we know if (cid:107) E (cid:107) ≤ δ ≤ σ min ( B ) / 2 , then B † =

Original: ( B (cid:63) ) † P − 1 S − 1 + E (cid:48) where (cid:107) E (cid:48) (cid:107) ≤ 2 √ 2 σ min ( B ) − 2 δ .
Revised: ( B (cid:63) ) † P − 1 S − 1 + E (cid:48) where (cid:107) E (cid:48) (cid:107) ≤ 2 √ 2 σ min ( B ) − 2 δ .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: a (cid:63) + ( E (cid:48) )

Original: (cid:62) a (cid:63) =

Original: SPa + ( E (cid:48) )

Original: (cid:62) a (cid:63) .

Original: (Here the last equality is because for both permutation matrix P and sign ﬂip matrix S , P −(cid:62)
Revised: (Here the last equality is because for both permutation matrix P and sign ﬂip matrix S , P −(cid:62)

Original: =
Revised: =

Original: P and S −(cid:62)
Revised: P and S −(cid:62)

Original: = S .)
Revised: = S .)

Original: Therefore, coordinates of a (cid:48) are permutation and sign ﬂips of a (cid:63) , up to an error term ( E (cid:48) )
Revised: Therefore, coordinates of a (cid:48) are permutation and sign ﬂips of a (cid:63) , up to an error term ( E (cid:48) )

Original: (cid:62) a (cid:63) .
Revised: (cid:62) a (cid:63) .

Original: When δ ≤ σ 2 min ( B ) 4 √ 2 κ (cid:63) √ m , we know (cid:107) ( E (cid:48) ) (cid:62) a (cid:63) (cid:107) ≤ (cid:107) E (cid:48) (cid:107) a (cid:63)max √ m ≤ a (cid:63)min / 2 , therefore the signs are all recovered correctly.
Revised: When δ ≤ σ 2 min ( B ) 4 √ 2 κ (cid:63) √ m , we know (cid:107) ( E (cid:48) ) (cid:62) a (cid:63) (cid:107) ≤ (cid:107) E (cid:48) (cid:107) a (cid:63)max √ m ≤ a (cid:63)min / 2 , therefore the signs are all recovered correctly.

Original: After ﬁxing the sign, we have (cid:107) a − Pa (cid:107) ≤ (cid:107) ( E (cid:48) ) (cid:62) a (cid:63) (cid:107) ≤ 2 √ 2 δa (cid:63)max √ m σ − 2 min ( B ) , and (cid:107) B (cid:48) − PB (cid:63) (cid:107) ≤ δ .
Revised: After ﬁxing the sign, we have (cid:107) a − Pa (cid:107) ≤ (cid:107) ( E (cid:48) ) (cid:62) a (cid:63) (cid:107) ≤ 2 √ 2 δa (cid:63)max √ m σ − 2 min ( B ) , and (cid:107) B (cid:48) − PB (cid:63) (cid:107) ≤ δ .

Original: E S AMPLE C OMPLEXITY
Revised: E S AMPLE C OMPLEXITY

Original: In this section we will show that our algorithm only requires polynomially many samples to ﬁnd the desired solution.
Revised: In this section we will show that our algorithm only requires polynomially many samples to ﬁnd the desired solution.

Original: Note that we did not try to optimize the polynomial dependency.
Revised: Note that we did not try to optimize the polynomial dependency.

Original: Theorem E.1 (Theorem 2.7 Restated) .
Revised: Theorem E.1 (Theorem 2.7 Restated) .

Original: In the setting of Theorem 2.3, suppose we use N empirical samples to approximate G and obtain function (cid:98) G .
Revised: In the setting of Theorem 2.3, suppose we use N empirical samples to approximate G and obtain function (cid:98) G .

Original: There exists a ﬁxed polynomial such that if N ≥ poly ( d, a (cid:63)max /a (cid:63)min , 1 /ε ) , with high probability for any point B with λ min ( ∇ 2 (cid:98) G ( B )) ≥ − τ 0 / 2 and (cid:107)∇ 
Revised: There exists a ﬁxed polynomial such that if N ≥ poly ( d, a (cid:63)max /a (cid:63)min , 1 /ε ) , with high probability for any point B with λ min ( ∇ 2 (cid:98) G ( B )) ≥ − τ 0 / 2 and (cid:107)∇ (cid:98) G ( B ) (cid:107) ≤ ε/ 2 , then B can be written as B = DP + E where P is a permutation matrix, D is a diagonal matrix and | E | ∞ ≤ O ( ε/ (ˆ σ 4 a (cid:63) min )) .

Original: In order to bound the sample complexity, we will prove a uniform convergence result: we show that with polynomially many samples, the gradient and Hessian of (cid:98) G are point-wise close to the gradient and Hessian of G , therefore any approximate local minimum of (cid:98) G must also be an approximate local minimum of G .
Revised: In order to bound the sample complexity, we will prove a uniform convergence result: we show that with polynomially many samples, the gradient and Hessian of (cid:98) G are point-wise close to the gradient and Hessian of G , therefore any approximate local minimum of (cid:98) G must also be an approximate local minimum of G .

Original: However, there are two technical issues in showing the uniform convergence result.
Revised: However, there are two technical issues in showing the uniform convergence result.

Original: The ﬁrst issue is that when the norm of B is very large, both the gradient and Hessian of G and (cid:98) G are very large and we cannot hope for good concentration.
Revised: The ﬁrst issue is that when the norm of B is very large, both the gradient and Hessian of G and (cid:98) G are very large and we cannot hope for good concentration.

Original: We deal with this issue by showing when B has a large norm, the empirical gradient ∇ (cid:98) G ( B ) must also have large norm, and therefore it can never be an approximate local minimum (we do this later in Lemma E.5).
Revised: We deal with this issue by showing when B has a large norm, the empirical gradient ∇ (cid:98) G ( B ) must also have large norm, and therefore it can never be an approximate local minimum (we do this later in Lemma E.5).

Original: The second issue is that our objective function involves high-degree polynomials over Gaussian variables x, y , and is therefore not subGaussian or sub-exponential.
Revised: The second issue is that our objective function involves high-degree polynomials over Gaussian variables x, y , and is therefore not subGaussian or sub-exponential.

Original: We use a standard truncation argument to show that the function does not change by too much if we restrict to the event that the Gaussian variables have bounded norm.
Revised: We use a standard truncation argument to show that the function does not change by too much if we restrict to the event that the Gaussian variables have bounded norm.

Original: Lemma E.2.
Revised: Lemma E.2.

Original: Suppose P (cid:48)
Revised: Suppose P (cid:48)

Original: ( B ) + R ( B ) =
Revised: ( B ) + R ( B ) =

Original: E ( x,y )
Revised: E ( x,y )

Original: [ f ( x, y, B )] where f is a polynomial of degree at most 5 in x, y and at most 4 in B .
Revised: [ f ( x, y, B )] where f is a polynomial of degree at most 5 in x, y and at most 4 in B .

Original: Also assume that the sum of absolute values of coefﬁcients is bounded by Γ .
Revised: Also assume that the sum of absolute values of coefﬁcients is bounded by Γ .

Original: For any ε ≤ Γ / 2 , let R =
Revised: For any ε ≤ Γ / 2 , let R =

Original: Cd log( a (cid:63)max Γ /ε ) for a large enough constant C , let F be the event that (cid:107) x (cid:107) 2 ≤ R , and let G trunc =
Revised: Cd log( a (cid:63)max Γ /ε ) for a large enough constant C , let F be the event that (cid:107) x (cid:107) 2 ≤ R , and let G trunc =

Original: E ( x,y )
Revised: E ( x,y )

Original: [ f ( x, y, B )1 F ] .
Revised: [ f ( x, y, B )1 F ] .

Original: For any B such that (cid:107) b i (cid:107) ≤for all rows, we have
Revised: For any B such that (cid:107) b i (cid:107) ≤for all rows, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: By standard χ 2 concentration bounds, for large enough C and any z > R , the probability that (cid:107) x (cid:107) 2 ≥ z is at most exp( − 10 z ) .
Revised: By standard χ 2 concentration bounds, for large enough C and any z > R , the probability that (cid:107) x (cid:107) 2 ≥ z is at most exp( − 10 z ) .

Original: By simple calculation, it is easy to check that (cid:107)∇ B f ( x, y, B ) (cid:107) ≤ 4Γ d 1 .
Revised: By simple calculation, it is easy to check that (cid:107)∇ B f ( x, y, B ) (cid:107) ≤ 4Γ d 1 .

Original: 5 a (cid:63)max (cid:107) x (cid:107) 5 , and (cid:107)∇ 2 B f ( x, y, B ) (cid:107) ≤ 12Γ d 2 a (cid:63)max (cid:107) x (cid:107) 5 .
Revised: 5 a (cid:63)max (cid:63)

Original: We know(cid:107)∇ G ( B ) −∇G trunc ( B ) (cid:107) =
Revised: (cid:107) know ( B ) −∇ ( B ) (cid:107) =

Original: (cid:107) E [ ∇ B f
Revised: (cid:107) E [ ∇ B f